Browse > Article
http://dx.doi.org/10.3745/KIPSTD.2004.11D.3.563

A Methodology for Performance Evaluation of Web Robots  

Kim, Kwang-Hyun (숭실대학교 대학원 컴퓨터학과)
Lee, Joon-Ho (숭실대학교 컴퓨터학부)
Publication Information
The KIPS Transactions:PartD / v.11D, no.3, 2004 , pp. 563-570 More about this Journal
Abstract
As the use of the Internet becomes more popular, a huge amount of information is published on the Web, and users can access the information effectively with Web search services. Since Web search services retrieve relevant documents from those collected by Web robots we need to improve the crawling quality of Web robots. In this paper, we suggest evaluation criteria for Web robots such as efficiency, continuity, freshness, coverage, silence, uniqueness and safety, and present various functions to improve the performance of Web robots. We also investigate the functions implemented in the conventional Web robots of NAVER, Google, AltaVista etc. It is expected that this study could contribute the development of more effective Web robots.
Keywords
Information Retrieval; Web Robot; Performance Evaluation;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 A. Heydon and M. Najork, 'Mercator : A Scalable, : Extensible Web Crawler,' InRecordings of the 8th World Wide Web Conference, Toronto, Canada, 1999
2 M. Koster, 'A Method for Web Rotots Control,' Network Working Group, Internet Draft, Dec. 1996, http://www.robotstxt.org/wc/norobots-rfc.html
3 J. Cho and H. Garcia-Molina, 'Parallel Crawler,' In Proceedings of the 11th Interational World Wide Web Conference, Hawaii, USA, 2002
4 S. Raghavan and H. Garcia-Molina, 'Crawling the Hidden Web,' Proceedings of the 27th International Conference on Very Large Databases, Rome, Italy, 2001
5 J. Cho, N. Shivakumar and H. Garcia-Molina, 'Finding Replicated Web Collections,' In Proceedings of the ACM SIGMOD International Conference on Management of Data, Dallas, Texas, 2000   DOI
6 M. Gray, 'Internet Growth and Statistics: Credits and Background,' http://www.mit.edu/people/mkgray/net/background.html
7 M. Najork and A. Heydon, 'High-Performance Web Crawling,' SRC Research Report 173, Compaq Systems Research Center, 2001
8 S. Brin and L. Page, 'The Anatomy of a Large-Scale Hypertextual Web Search Engine,' In Proceedings of the 7th International World Wide Web Conference, Brisbane, Australia, 1998
9 J. Cho and H. Garcia-Molina, 'The Evolution of the Web and Implications for an Incremental Crawler,' In Proceedings of the 26th International Conference on Very Large Databases, Cairo, Egypt, 2000
10 V. Shkapenyukn and T. Suel, 'Design and Implementation of a High-performance Distributed Web Crawler,' In Proceedings of the 18th International Conference on Data Engineering, San Jose, California, 2002