Browse > Article
http://dx.doi.org/10.5392/JKCA.2013.13.12.575

Intelligent Web Crawler for Supporting Big Data Analysis Services  

Seo, Dongmin (한국과학기술정보연구원 소프트웨어연구센터 컴퓨터지능연구실)
Jung, Hanmin (한국과학기술정보연구원 소프트웨어연구센터 컴퓨터지능연구실)
Publication Information
The Journal of the Korea Contents Association / v.13, no.12, 2013 , pp. 575-584 More about this Journal
Abstract
Data types used for big-data analysis are very widely, such as news, blog, SNS, papers, patents, sensed data, and etc. Particularly, the utilization of web documents offering reliable data in real time is increasing gradually. And web crawlers that collect web documents automatically have grown in importance because big-data is being used in many different fields and web data are growing exponentially every year. However, existing web crawlers can't collect whole web documents in a web site because existing web crawlers collect web documents with only URLs included in web documents collected in some web sites. Also, existing web crawlers can collect web documents collected by other web crawlers already because information about web documents collected in each web crawler isn't efficiently managed between web crawlers. Therefore, this paper proposed a distributed web crawler. To resolve the problems of existing web crawler, the proposed web crawler collects web documents by RSS of each web site and Google search API. And the web crawler provides fast crawling performance by a client-server model based on RMI and NIO that minimize network traffic. Furthermore, the web crawler extracts core content from a web document by a keyword similarity comparison on tags included in a web documents. Finally, to verify the superiority of our web crawler, we compare our web crawler with existing web crawlers in various experiments.
Keywords
Web Crawler; Content Extraction; Big-data; RMI; NIO;
Citations & Related Records
Times Cited By KSCI : 5  (Citation Analysis)
연도 인용수 순위
1 성원경, 이상환, 정한민, 박경석, 이승우, 김선태, 황미녕, 조민희, 과학기술 빅데이터 추진과제 발굴 및 활용 극대화를 위한 추진전략 마련 기획연구, 교육과학기술부, 2013.
2 김정숙, "빅 데이터 활용과 관련기술 고찰", 한국콘텐츠학회논문지, 제10권, 제1호, pp.34-40, 2012.
3 http://www.1000genomes.org
4 http://www.smartinsight.co.kr
5 http://www.energy.gov
6 M. K. Lee, S. W. Lee, J. Y. Kim, D. M. Seo, P. K. Ki, H. M. Jung, J. H. Lee, T. H. Kim, H. K. Koo, and W. K. Sung, "InSciTe Advanced: Strategic Decision-Making Support Service based on Technology Opportunity Discovery Model," J. of Information An International Interdiscipli- nary, Vol.16, No.1(B), pp.639-698, 2013.
7 이진희, 김태홍, 이승우, 김평, 이미경, 서동민, 김진형, 정한민, "정보 가치 향상을 위한 정보 분석 기술의 테크놀로지 인텔리전스 적용", 정보과학회논문지 : 컴퓨팅의 실제 및 레터, 제18권, 제2호, pp.153-157, 2012.   과학기술학회마을
8 http://en.wikipedia.org/wiki/Web_crawler
9 김영광, 이원구, 윤화묵, 신성호, 이민호, "웹 자원 아카이빙을 위한 웹 크롤러 연구 개발", 한국콘텐츠학회논문지, 제11권, 제9호, pp.9-16, 2011.   과학기술학회마을   DOI   ScienceOn
10 J. H. Cho and H. G. Molina, "Parallel Crawlers," In Proc. of International World Wide Web Confe- rence, pp.124-135, 2002.
11 http://en.wikipedia.org/wiki/RSS
12 https://developers.google.com/custom-search
13 J. H. Cho, H. G. Molina, and L. Page, "Efficient Crawling through URL Ordering," J. of Computer Networks and ISDN Systems, Vol.30, pp.161-171, 1998.   DOI   ScienceOn
14 A. Heydon and M. Najork, "Mercator: A Scalable Extensible Web Crawler," In Proc. of International World Wide Web Conference, pp.219-229, 1999.
15 김진일, 권유진, 김성렬, 박근수, "그래프 탐색 기법을 이용한 효율적인 웹 크롤링 알고리즘", 한국정보과학회 가을학술발표논문집, 제35권, 제2호, pp.299-300, 2008.
16 D. Cai, S. Yu, J. R. Wen, and W. Y. Ma, "VIPS: a Vision-based Page Segmentation Algorithm," Microsoft Technical Report, 2003.
17 송문원, 김우승, 김명원, "텍스트 블록 주변의 문맥을 이용한 HTML 문서 본문 추출", 정보과학회논문지 : 소프트웨어 및 응용, 제40권, 제3호, pp.155-163, 2013.   과학기술학회마을
18 C. Kohlschutter, P. Fankhauser, and W. Nejdl, "Boilerplate Detection using Shallow Text Features," In Proc. of ACM International Conference on Web Search and Data Mining, pp.441-450, 2010.
19 방승준, 안진호, "소켓 및 RMI 기반 자바 메시지 전달 시스템의 구현 및 성능평가", 한국 인터넷정보학회지, 제8권, 제5호, pp.11-20, 2007.   과학기술학회마을
20 http://www.cnet.com
21 http://www.infoworld.com
22 http://www.computerworld.com