Browse > Article
http://dx.doi.org/10.33778/kcsa.2019.19.3.021

Distribute Parallel Crawler Design and Implementation  

Jang, Hyun Ho (숭실대학교/IT정책경영학)
jeon, kyung-sik (숭실대학교/IT정책경영학)
Lee, HooKi (건양대학교/사이버보안공학과)
Publication Information
Convergence Security Journal / v.19, no.3, 2019 , pp. 21-28 More about this Journal
Abstract
As the number of websites managed by organizations or organizations increases, so does the number of web application servers and containers. In checking the status of the web service of the web application server and the container, it is very difficult for the person to check the status of the web service after accessing the physical server at the remote site through the terminal or using other accessible software It. Previous research on crawler-related research is hard to find any reference to the processing of data from crawling. Data loss occurs when the crawler accesses the database and stores the data. In this paper, we propose a method to store the inspection data according to crawl - based web application server management without losing data.
Keywords
crawler; web Crawler; server;
Citations & Related Records
연도 인용수 순위
  • Reference
1 http://www.tta.or.kr
2 Berners-Lee, Tim. "HyperText Transfer Protocol". World Wide Web Consortium. Retrieved 31 August 2010.
3 Curbera Francisco et al., "Unraveling the Web Services Web:An Introduction to SOAP, SDL, and UDDI," IEEE Internet computing, Vol.6 No.2, pp.86-93, 2002.   DOI
4 Castillo, C., "Effective Web Crawling," ACM SIGIR Forum 55, Vol.39, No.1, pp. 55-56, June 2005.   DOI
5 "HTTP/1.1". Webcom.com Glossary entry. Archived from the original on 2001-11-21. Retrieved 2009-05-29.
6 Heydon, A. and Najork, M., "Mercator: A Scalable, Extensible Web Crawler," In Proc. 2nd Int'l Conf. on World Wide Web, pp.219-229, Dec. 1999.
7 Tim Berners-Lee. "The Original HTTP as defined in 1991". World Wide Web Consortium. Retrieved 24 July 2010.
8 V. Shkapenyukn, T. Suel, "Design and Implementation of a High-performance Distributed Web Crawler," In Processings of the 18th International Conference on Data Engineering, San Jose, California, 2002.
9 신은정, 김이론, 허준석, 황규영, "오디세우스 용량 검색 엔진을 한 병렬 웹 크롤러의 구현" 정보과학회 논문지 : 컴퓨팅의 실제 및 레터, Vol. 14, No 6, 2008년 8월