• Title/Summary/Keyword: Crawling

Search Result 371, Processing Time 0.028 seconds

Development of Social Data Collection and Loading Engine-based Reliability analysis System Against Infectious Disease Pandemic (감염병 위기 대응을 위한 소셜 데이터 수집 및 적재 엔진 기반 신뢰도 분석 시스템 개발)

  • Doo Young Jung;Sang-Jun Lee;MIN KYUNG IL;Seogsong Jeong;HyunWook Han
    • The Journal of Bigdata
    • /
    • v.7 no.2
    • /
    • pp.103-111
    • /
    • 2022
  • There are many institutions, organizations, and sites related to responding to infectious diseases, but as the pandemic situation such as COVID-19 continues for years, there are many changes in the initial and current aspects, and accordingly, policies and response systems are evolving. As a result, regional gaps arise, and various problems are scattered due to trust, distrust, and implementation of policies. Therefore, in the process of analyzing social data including information transmission, Twitter data, one of the major social media platforms containing inaccurate information from unknown sources, was developed to prevent facts in advance. Based on social data, which is unstructured data, an algorithm that can automatically detect infectious disease threats is developed to create an objective basis for responding to the infectious disease crisis to solidify international competitiveness in related fields.

Web crawling process of each social network service for recognizing water quality accidents in the water supply networks (물공급네트워크 수질사고인지를 위한 소셜네트워크 서비스 별 웹크롤링 방법론 개발)

  • Yoo, Do Guen;Hong, Seunghyeok;Moon, Gihoon
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2022.05a
    • /
    • pp.398-398
    • /
    • 2022
  • 최근 수돗물 공급과정에 있어 적수, 유충 발생 등 지역 단위의 수질문제로 국민의 직간접적인 피해가 발생된 바 있다. 수질문제 발생 시, 소셜네트워크서비스(SNS)에 게시되는 피해 관련 의견은 시공간적으로 빠르게 확산되며, 궁극적으로는 물공급과정 전체의 부정적 인식증가와 신뢰도 저하를 초래한다. 따라서, 물공급시스템에서의 수질사고 발생을 빠르게 인지하는 다양한 방법론의 적용을 통한 피해 최소화를 위한 노력이 반드시 필요하다. 일반적으로 수질사고는 다양한 항목의 실시간 계측기에서 획득되는 시계열자료의 변화양상을 통해 판단할 수 있으나, 이와 같은 방법론의 효율적 적용을 위해서는 선진계측인프라의 도입이 선행되어야 한다. 본 연구에서는 국내의 발달된 정보통신기술환경을 활용하여, 물공급네트워크 내 수질사고인지를 위한 SNS 별 웹크롤링 방법론을 제안하고, 적용결과를 분석하였다. 방법론의 구현에 앞서, 각종 SNS 별(트위터, 인스타그램, 블로그, 네이버 카페 등) 프로그래밍을 통한 웹크롤링 가능여부, 정보획득 기간 등을 확인하였으며, 과거 유사 수질사고 발생 시 영향력과 관련 게시글이 크게 나타난 네이버 카페와 트위터를 중심으로 웹 크롤링 절차를 제시하였다. 네이버 카페의 경우 대상급수구역 내의 시민들이 다수 참여하는 카페를 목록화하고, 지자체명과 핵심 키워드(수돗물, 유충, 적수) 조합을 활용한 웹크롤링을 수행하여, 관련 게시물 건수와 의미를 실시간으로 분석하는 절차를 마련하였다. 개발된 SNS 별 웹크롤링 방법론에 따라 과거 수질사고가 발생된 바 있는 2개 이상의 지자체에 대한 분석을 실시하였으며, SNS 별 결과에 있어 차이점을 확인하여 제시하였다. 향후 제안된 방법을 적용하여 시공간적 수질사고 정보의 전파 및 확산양상을 추가적으로 분석할수 있을 것으로 기대된다.

  • PDF

Media-based Analysis of Gasoline Inventory with Korean Text Summarization (한국어 문서 요약 기법을 활용한 휘발유 재고량에 대한 미디어 분석)

  • Sungyeon Yoon;Minseo Park
    • The Journal of the Convergence on Culture Technology
    • /
    • v.9 no.5
    • /
    • pp.509-515
    • /
    • 2023
  • Despite the continued development of alternative energies, fuel consumption is increasing. In particular, the price of gasoline fluctuates greatly according to fluctuations in international oil prices. Gas stations adjust their gasoline inventory to respond to gasoline price fluctuations. In this study, news datasets is used to analyze the gasoline consumption patterns through fluctuations of the gasoline inventory. First, collecting news datasets with web crawling. Second, summarizing news datasets using KoBART, which summarizes the Korean text datasets. Finally, preprocessing and deriving the fluctuations factors through N-Gram Language Model and TF-IDF. Through this study, it is possible to analyze and predict gasoline consumption patterns.

Detection Models and Response Techniques of Fake Advertising Phishing Websites (가짜 광고성 피싱 사이트 탐지 모델 및 대응 기술)

  • Eunbeen Lee;Jeongeun Cho;Wonhyung Park
    • Convergence Security Journal
    • /
    • v.23 no.3
    • /
    • pp.29-36
    • /
    • 2023
  • With the recent surge in exposure to fake advertising phishing sites in search engines, the damage caused by poor search quality and personal information leakage is increasing. In particular, the seriousness of the problem is worsening faster as the possibility of automating the creation of advertising phishing sites through tools such as ChatGPT increases. In this paper, the source code of fake advertising phishing sites was statically analyzed to derive structural commonalities, and among them, a detection crawler that filters sites step by step based on foreign domains and redirection was developed to confirm that fake advertising posts were finally detected. In addition, we demonstrate the need for new guide lines by verifying that the redirection page of fake advertising sites is divided into three types and returns different sites according to each situation. Furthermore, we propose new detection guidelines for fake advertising phishing sites that cannot be detected by existing detection methods.

Development of a Fake News Detection Model Using Text Mining and Deep Learning Algorithms (텍스트 마이닝과 딥러닝 알고리즘을 이용한 가짜 뉴스 탐지 모델 개발)

  • Dong-Hoon Lim;Gunwoo Kim;Keunho Choi
    • Information Systems Review
    • /
    • v.23 no.4
    • /
    • pp.127-146
    • /
    • 2021
  • Fake news isexpanded and reproduced rapidly regardless of their authenticity by the characteristics of modern society, called the information age. Assuming that 1% of all news are fake news, the amount of economic costs is reported to about 30 trillion Korean won. This shows that the fake news isvery important social and economic issue. Therefore, this study aims to develop an automated detection model to quickly and accurately verify the authenticity of the news. To this end, this study crawled the news data whose authenticity is verified, and developed fake news prediction models using word embedding (Word2Vec, Fasttext) and deep learning algorithms (LSTM, BiLSTM). Experimental results show that the prediction model using BiLSTM with Word2Vec achieved the best accuracy of 84%.

A Study on Traffic Vulnerable Detection Using Object Detection-Based Ensemble and YOLOv5

  • Hyun-Do Lee;Sun-Gu Kim;Seung-Chae Na;Ji-Yul Ham;Chanhee Kwak
    • Journal of the Korea Society of Computer and Information
    • /
    • v.29 no.1
    • /
    • pp.61-68
    • /
    • 2024
  • Despite the continuous efforts to mitigate pedestrian accidents at crosswalks, the problem persist. Vulnerable groups, including the elderly and disabled individuals are at a risk of being involved in traffic incidents. This paper proposes the implementation of object detection algorithm using the YOLO v5 model specifically for pedestrians using assistive devices like wheelchairs and crutches. For this research, data was collected and utilized through image crawling, Roboflow, and Mobility Aids datasets, which comprise of wheelchair users, crutch users, and pedestrians. Data augmentation techniques were applied to improve the model's generalization performance. Additionally, ensemble techniques were utilized to mitigate type 2 errors, resulting in 96% recall rate. This demonstrates that employing ensemble methods with a single YOLO model to target transportation-disadvantaged individuals can yield accurate detection performance without overlooking crucial objects.

Analysis for IT Trends in Korea and the United States using Big Data in IT-related Papers (IT 관련 논문 빅데이터를 활용한 한국과 미국의 IT 동향 분석)

  • Seung-Yeon Hwang;Seok-Woo Jang
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.24 no.3
    • /
    • pp.171-176
    • /
    • 2024
  • IT-related fields are very diverse. As of 2018, the IT revolution from the Fourth Industrial Revolution not only brought out the new fields that were different from the previous ones, but it also made a reexamination of various fields that had already been an issue in the past. Companies and public institutions have a great interest in understanding IT trends in this situation. Therefore, in this paper, IT trends are identified through the analyzation of keywords provided by domestic papers. Moreover, unlike previous industry trend analysis or economic analysis, this paper focuses on analyzing the keyword provided by the doctoral thesis or master's thesis about direct IT-related research, and grasps the more basic and direct IT trend. This analysis predicts and presents the vision based on the data of the analysis from the academic papers that researched in IT technology for IT related students or IT related educators.

Evaluation of communication effectiveness of cruelty-free fashion brands - A comparative study of brand-led and consumer-perceived images - (크루얼티 프리 패션 브랜드의 커뮤니케이션 성과 분석 - 브랜드 주도적 이미지와 소비자 지각 이미지에 대한 비교 -)

  • Yeong-Hyeon Choi;Sangyung Lee
    • The Research Journal of the Costume Culture
    • /
    • v.32 no.2
    • /
    • pp.247-259
    • /
    • 2024
  • This study assessed the effectiveness of brand image communication on consumer perceptions of cruelty-free fashion brands. Brand messaging data were gathered from postings on the official Instagram accounts of three cruelty-free fashion brands and consumer perception data were gathered from Tweets containing keywords related to each brand. Web crawling and natural language processing were performed using Python and sentiment analysis was conducted using the BERT model. By analyzing Instagram content from Stella McCartney, Patagonia, and Freitag from their inception until 2021, this study found these brands all emphasize environmental aspects but with differing focuses: Stella McCartney on ecological conservation, Patagonia on an active outdoor image, and Freitag on upcycled products. Keyword analysis further indicated consumers perceive these brands in line with their brand messaging: Stella McCartney as high-end and eco-friendly, Patagonia as active and environmentally conscious, and Freitag as centered on recycling. Results based on the assessment of the alignment between brand-driven images and consumer-perceived images and the sentiment evaluation of the brand confirmed the outcomes of brand communication performance. The study revealed a correlation between brand image and positive consumer evaluations, indicating that higher alignment of ethical values leads to more positive consumer assessments. Given that consumers tend to prioritize search keywords over brand concepts, it's important for brands to focus on using visual imagery and promotions to effectively convey brand communication information. These findings highlight the importance of brand communication by emphasizing the connection between ethical brand images and consumer perceptions.

A Study on Duplication Verification of Public Library Catalog Data: Focusing on the Case of G Library in Busan (공공도서관 목록데이터의 중복검증에 관한 연구 - 부산 지역 G도서관 사례를 중심으로 -)

  • Min-geon Song;Soo-Sang Lee
    • Journal of Korean Library and Information Science Society
    • /
    • v.55 no.1
    • /
    • pp.1-26
    • /
    • 2024
  • The purpose of this study is to derive an integration plan for bibliographic records by applying a duplicate verification algorithm to the item-based catalog in public libraries. To this, G Library, which was opened recently in Busan, was selected. After collecting OPAC data from G Library through web crawling, multipart monographs of Korean Literature (KDC 800) were selected and KERIS duplicate verification algorithm was applied. After two rounds of data correction based on the verification results, the duplicate verification rate increased by a total of 2.74% from 95.53% to 98.27%. Even after data correction, 24 books that were judged to be similar or inconsistent were identified as data from other published editions after receiving separate ISBN such as revised versions or hard copies. Through this, it was confirmed that the duplicate verification rate could be improved through catalog data correction work, and the possibility of using the KERIS duplicate verification algorithm as a tool to convert duplicate item-based records from public libraries into manifestation-based records was confirmed.

Analysis of YouTube Viewers' Characteristics and Responses to Virtual Idols (버추얼 아이돌에 대한 유튜브 시청자 특성과 반응 분석)

  • JeongYoon Kang;Choonsung Shin;Hieyong Jeong
    • Journal of Information Technology Services
    • /
    • v.23 no.3
    • /
    • pp.103-118
    • /
    • 2024
  • Due to the advancement of virtual reality technology, virtual idols are widely used in industrial and cultural content industries. However, it is difficult to utilize virtual idols' social perceptions because they are not properly understood. Therefore, this paper collected and analyzed YouTube comments to identify differences about social perception through comparative analysis between virtual idols and general idols. The dataset was constructed by crawling comments from music videos with more than 10 million views of virtual idols and more than 10,000 comments. Keyword frequency and TF-IDF values were derived from the collected dataset, and the connection centrality CONCOR cluster was analyzed with a semantic network using the UCINET program. As a result of the analysis, it was found that virtual idols frequently used keywords such as "person," "quality," "character," "reality," "animation," while reactions and perceptions were derived from general idols. Based on the results of this analysis, it was found that while general idols are mainly evaluated with their appearance and cultural factors, social perceptions of virtual idols' values are mixed with evaluations of cultural factors such as "song," "voice," and "choreography," focusing on technical factors such as "people," "quality," "character," and "animation." However, keywords such as "song," "voice," "choreography," and "music" are included in the top 30 like regular idols and appear in the same cluster, suggesting that virtual idols are gradually shifting away from minority tastes to mainstream culture. This study aims to provide academic and practical implications for the future expansion of the industry and cultural content industry of virtual idols by grasping the social perception of virtual idols.