• Journal of Digital Contents Society
• /
• v.4 no.2
• /
• pp.127-137
• /
• 2003

A Web crawler is an important Internet software technology used in a variety of Internet application software which includes search engines. As Internet continues to grow, implementations of high performance web crawlers are urgently demanded. In this paper, we study how to support dynamic scheduling for a multiprocess-based web crawler. For high performance, web crawlers are usually based on multiprocess in their implementations. In these systems, crawl scheduling which manages the allocation of web pages to each process for loading is one of the important issues. In this paper, we identify issues which are important and challenging in the crawl scheduling. To address the issue, we propose a dynamic crawl scheduling framework and subsequently a system architecture for a web crawler with dynamic crawl scheduling support. This paper presents the design of the Web crawler with dynamic scheduling support.

• Journal of Korea Society of Industrial Information Systems
• /
• v.8 no.4
• /
• pp.64-72
• /
• 2003

A Web crawler is an important Internet software technology used in a variety of Internet application software which includes search engines. As Internet continues to grow, implementations of high performance web crawlers are urgently demanded. In this paper, we study how to support dynamic scheduling for a multiprocess-based web crawler. For high peformance, web crawlers are usually based on multiprocess in their implementations. In these systems, crawl scheduling which manages the allocation of web pages to each process for loading is one of the important issues. In this paper, we identify issues which are important and challenging in the crawl scheduling. To address the issue, we propose a dynamic crawl scheduling framework and subsequently a system architecture for a web crawler with dynamic crawl scheduling support. And we analysed the behaviors of Web crawler. Based on the analysis result, we suggest the direction for the design of high performance Web crawler.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2014.05a
• /
• pp.582-584
• /
• 2014

Conventional Web crawlers are reducing overhead burden on the server to ensure the integrity of data optimization measures have been continuously developed. The amount of data growing exponentially faster among those data, then the data needs to be collected should be used to the modern web crawler is the indispensable presence. In this paper, suggested that the existing Web crawler and Web crawler approach efficiency comparison and analysis. In addition, based on the results, compared to suggest an optimized technique, Web crawlers, data collection cycle dynamically reduces the overhead of the server system was designed for. This is a Web crawler approach will be utilized in the field of the search system.

• Proceedings of the Korean Institute of Intelligent Systems Conference
• /
• 2008.04a
• /
• pp.79-82
• /
• 2008

웹크롤러는 웹페이지 내의 URL링크를 추적하여 다른 문서를 수집한다. 국내의 상당수 웹사이트는 웹 표준에 맞지 않는 링크방식으로 웹문서를 연결하고 있다. 일반적인 웹크롤러는 링크의 비표준적인 사용을 가정하지 않기 때문에 이러한 문서는 수집할 수 없다. 비표준적인 링크가 가능한 것은 사용자의 실수에 강인한 마크업 언어인 HTML에 자바스크립트 기능이 추가되면서 자바스크립트의 변칙적인 사용이 허용되었기 때문이다. 본 논문에서는 230여개의 웹사이트를 조사하여 기존 웹크롤러에서 해결하지 못한 링크 추출 문제를 찾아내고, 이를 수집하기 위한 알고리즘을 제안한다. 또한 자바스크립트 문제 해결을 위한 무거운 자바스크립트 엔진을 대신하여 필요한 기능만으로 구성된 모듈을 사용함으로써 효율적인 문서 수집기 모델을 제안한다.

• The Journal of the Korea Contents Association
• /
• v.13 no.6
• /
• pp.30-39
• /
• 2013

Web crawler should maintain fresh data with minimum server overhead for large amount of data in the web sites. The overhead in the server increases rapidly as the amount of data is exploding as in the big data era. The amount of web information is increasing rapidly with advanced wireless networks and emergence of diverse smart devices. Furthermore, the information is continuously being produced and updated in anywhere and anytime by means of easy web platforms, and smart devices. Now, it is becoming a hot issue how frequently updated web data has to be refreshed in data collection and integration. In this paper, we propose dynamic web-data crawling methods, which include sensitive checking of web site changes, and dynamic retrieving of web pages from target web sites based on historical update patterns. Furthermore, we implemented a Java-based web crawling application and compared efficiency between conventional static approaches and our dynamic one. Our experiment results showed 46.2% overhead benefits with more fresh data compared to the static crawling methods.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2019.10a
• /
• pp.1082-1084
• /
• 2019

인공지능 학습에 대한 관심이 증가하면서 학습에 필요한 데이터셋 구축에 필요한 많은 양의 데이터가 필요하다. 데이터셋 구축에 필요한 데이터들을 효과적으로 수집하기 위한 키워드 기반 웹크롤러를 제안한다. 구글 검색 API 를 기반으로 웹 크롤러를 설계하였으며 사용자가 입력한 키워드를 바탕으로 이미지의 URI 와 메타데이터를 지속적으로 수집하는 크롤러이다. 수집한 URI 와 메타데이터는 데이터베이스를 통해 관리한다. 향후 다른 검색 API 에서도 동작하고 다중 쓰레드를 활용하여 크롤링하는 속도를 높일 예정이다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2004.05a
• /
• pp.601-604
• /
• 2004

인터넷과 정보기술의 급속적인 발전으로 수천 또는 수억에 달하는 방대하고 다양한 정보들이 웹 상에서 존재하게 되었다. 이러한 다양한 정보들 중에서 이용자가 원하는 정보를 제공하기 위해 다양한 검색시스템들이 개발되었다. 웹 크롤러는 검색시스템의 중요한 부분 중의 하나로, 웹 서버를 순회하며 각 페이지에 있는 수많은 정보를 수집하는 프로그램이다. 본 논문에서는 웹 크롤러의 기본요건 중의 하나인 빠른 수집속도를 보장하기 위해 RTT(Round Trip Time) Scheduling 기법을 적용한 URL 스케줄러를 제시하고, 이러한 RTT Scheduling 기법을 이용한 웹 크롤러를 설계한다.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2021.01a
• /
• pp.281-282
• /
• 2021

본 논문에서는 항공 수요예측을 위한 빅데이터 기반 플랫폼의 설계 및 실증 결과를 제시한다. 항공 수요예측 통합 플랫폼은 항공산업 관련 데이터를 Open API, RSS Feed, 웹크롤러(Web Crawler) 등을 이용하여 수집 및 분석하여 자체 개발한 항공 수요예측 알고리즘을 기반으로 결과를 시각화하여 보여주도록 구현되어 있다. 또한, 제안하는 플랫폼의 사용자 인터페이스를 통해 변수 설정을 하여 단위별(Global, National 등), 기간별(단기, 중장기 등), 유형별(여객, 화물 등) 예측 통계 자료를 도출할 수 있다. 플랫폼의 성능 검증을 위해 정형화된 데이터를 비롯하여 소셜네트워크서비스(SNS), 검색엔진 등에서 수집한 비정형 데이터까지 활용하여 특정 키워드의 빈도와 특정 노선에 대한 항공 수요간 상관관계를 분석하였다. 개발한 통합 플랫폼의 지능형 항공 수요예측 알고리즘을 통해 전반적인 공항 운영 및 공항 운영 정책 수립에 기여할 것으로 예상한다.

• Journal of the Korean Institute of Intelligent Systems
• /
• v.20 no.3
• /
• pp.318-325
• /
• 2010

In this study, the wrapper database description language and model is suggested to collect product reviews from Korean shopping malls with multi-layer structures and are built in a variety of web languages. Above all, the wrapper based web crawlers have the website structure information to bring the exact desired data. The previously suggested wrapper based web crawler can collect HTML documents and the hierarchical structure of the target documents were only 2-3 layers. However, the Korean shopping malls in the study consist of not only HTML documents but also of various web language (JavaScript, Flash, and AJAX), and have a 5-layer hierarchical structure. A web crawler should have information about the review pages in order to visit the pages without visiting any non-review pages. The proposed wrapper contains the location information of review pages. We also propose a language grammar used in describing the location information.

• Journal of the Korea Convergence Society
• /
• v.13 no.4
• /
• pp.1-6
• /
• 2022

Real-time systems using web cralwing data must provide users with data from the same database as remote data. To do this, the web crawler repeatedly sends HTTP(HtypeText Transfer Protocol) requests to the remote server to see if the remote data has changed. This process causes network load on the crawling server and remote server, causing problems such as excessive traffic generation. To solve this problem, in this paper, based on user events, we propose a real-time web crawling technique that can reduce the overload of the network while securing the reliability of maintaining the sameness between the data of the crawling server and data from multiple remote locations. The proposed method performs a crawling process based on an event that requests unit data and list data. The results show that the proposed method can reduce the overhead of network traffic in existing web crawlers and secure data reliability. In the future, research on the convergence of event-based crawling and time-based crawling is required.