1 |
Mctacrawler : search the search engines, http://www.metacrawler.com/info.metac/dog/index.html
|
2 |
Debrifing Meta Search Engine : fast and accurate, http://www.debriefing.com/
|
3 |
Berners-Lee,T., Masinter,L. and M. McCahill, 'Uniform Resource Locators (URL),' RFC 1738
|
4 |
R. Fielding, J. Gettys, J. C. Mogul, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee, 'Hypertext Transfer Protocol-HTTP/1.1,' RFC 2068
|
5 |
Martijn Koster, 'Robots in the Web : threat or treat?,' April, 1995
|
6 |
Koster, M., 'A Standard for Robot Exclusion,' http://info.webcrawler.com/mak/projects/robots/exclusion.html
|
7 |
'The Web Robots FAQ,' http://www.robotstxt.org/wc/robots.html
|
8 |
D. Wessels, K. Claffy, 'Internet Cache Protocol (ICP), version 2,' RFC 2186
|
9 |
Search Engine Robots that visit your web site, http://www.jafsoft.com/searchengines/webbots.html
|