Browse > Article
http://dx.doi.org/10.22156/CS4SMB.2017.7.2.043

Multi-threaded Web Crawling Design using Queues  

Kim, Hyo-Jong (Department of Information Security, Tongmyong University)
Lee, Jun-Yun (School of Digital Media Engineering, Tongmyong University)
Shin, Seung-Soo (Department of Information Security, Tongmyong University)
Publication Information
Journal of Convergence for Information Technology / v.7, no.2, 2017 , pp. 43-51 More about this Journal
Abstract
Background/Objectives : The purpose of this study is to propose a multi-threaded web crawl using queues that can solve the problem of time delay of single processing method, cost increase of parallel processing method, and waste of manpower by utilizing multiple bots connected by wide area network Design and implement. Methods/Statistical analysis : This study designs and analyzes applications that run on independent systems based on multi-threaded system configuration using queues. Findings : We propose a multi-threaded web crawler design using queues. In addition, the throughput of web documents can be analyzed by dividing by client and thread according to the formula, and the efficiency and the number of optimal clients can be confirmed by checking efficiency of each thread. The proposed system is based on distributed processing. Clients in each independent environment provide fast and reliable web documents using queues and threads. Application/Improvements : There is a need for a system that quickly and efficiently navigates and collects various web sites by applying queues and multiple threads to a general purpose web crawler, rather than a web crawler design that targets a particular site.
Keywords
Web Crawling; Entity-Relationship Diagram(ERD); Multi-threaded; Bot; Uniform Resource Locator(URL);
Citations & Related Records
Times Cited By KSCI : 5  (Citation Analysis)
연도 인용수 순위
1 D. M. Seo and H. M. Jung, “Intelligent Web Crawler for Supporting Big Data Analysis Services,” Journal of the Korea Contents Association, Vol. 13, No. 12, pp. 575-584, Dec. 2013. DOI: 10.5392/JKCA.2013.13.12.575   DOI
2 H. W. Kim and Y. S. Han, "Web Crawler Design for the ARANES Search Engine," Korean Society For Internet Information, Vol. 2, No. 1, pp. 294-299. May. 2001.
3 H. Garcia-Molina and J. cho, "Parallel Crawler," Proceedings of the 11th International World Wide Web Conference, pp. 124-135, 2002. DOI: 10.1145/511446.511464   DOI
4 H. C. Kim and S. H. Chae, "Design and Implementation of a High Performance Web Crawler," Journal of Digital Contents Society, Vol. 4, No. 2, pp. 127-137, 2003.
5 A. Heydon and M. Najork, "Mercator : A Scalable : Extensible Web Crawler," Journal World Wide Web, Vol. 2, Issue. 4, 1999. DOI: 10.1023/A:1019213109274   DOI
6 K. Y. Kim, W. G. Lee, H. M. Yoon, S. H. Shin and M. . Lee, "Development of Web Crawler for Archiving Web Resources," Journal of the Korea Contents Association, Vol. 11, No. 9, pp. 9-16, Sep. 2011. DOI: 10.5392/JKCA.2011.11.9.009   DOI
7 D. Y. Kim and J. T. Kim, “Efficient Design of Web Searching Robot Engine Using Distributed Processing Method with Javascript Function,” Journal of the Korea Institute of Information and Communication Engineering, Vol. 13, No. 12, pp. 2595-2602, Dec. 2009.
8 K. H. Kim and J. H. Lee, “A Methodology for Performance Evaluation of Web Robots,” Information Processing Society, Vol. 11, No. 3, pp. 563-570, 2006.
9 M. Najork and A. Heydon, High-Performance Web Crawling, SRC Research Report 173, Compaq Systems Research Center, 2001.
10 W. S. Cho, J. E. Lee and C. H. Choi, "Refresh Cycle Optimization for Web Crawlers," Journal of the Korea Contents Association, Vol. 13, No. 6, pp. 30-39, Jun. 2013. DOI: 10.5392/JKCA.2013.13.06.030   DOI
11 M. S. Choi, A Study on the Improvement of the Web-Crawler Performance based on Weighted Directed Graph, Department of Computer Science, Graduate School, Kyungpook National University, 2010.
12 H. H. Kim, Y. W. Kim and P. W. Lee, “A Method of GridIR System Configuration over Distributed Experiment of Web Crawler,” Korean Society For Internet Information, Vol. 8, No. 2, pp. 281-286, Nov. 2007.
13 D. K. Jung and S. G. Min, “A study of Client Response Time Collection Method by changing the Dynamic HTML in the Web Application Server System,” Korea Information Science Society, Vol. 39, No. 1, pp. 305-307, Jun. 2012.