Browse > Article

Has Retrieval Technology in Vertical Site Search Systems Improved over the Years? A Holistic Evaluation for Real Web Systems  

Mandl, Thomas (University of Hildesheim, Department of Information Science and Natural Language Processing (IWIST), Hildesheim, Germany)
Womser-Hacker, Christa (University of Hildesheim, Department of Information Science and Natural Language Processing (IWIST), Hildesheim, Germany)
Gatzke, Natalia (TUV NORD, Hannover, Germany)
Publication Information
Journal of Information Science Theory and Practice / v.3, no.4, 2015 , pp. 19-34 More about this Journal
Evaluation of retrieval systems is mostly limited to laboratory settings and rarely considers changes of performance over time. This article presents an evaluation of retrieval systems for internal Web site search systems between the years 2006 and 2011. A holistic evaluation methodology for real Web sites was developed which includes tests for functionality, search quality, and user interaction. Among other sites, one set of 20 Web site search systems was evaluated three times in different years and no substantial improvement could be shown. It is surprising that the communication between site and user still leads to very poor results in many cases. Overall, the quality of these search systems could be improved, and several areas for improvement are apparent from our evaluation. For a comparison, Google’s site search function was also tested with the same tasks.
Site Search; Information Retrieval; Evaluation;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Gätzke, N. (2011). Verbessert sich die Suchfunktion auf Internetseiten im Laufe der Zeit? Eine diachrone Analyse der Qualität von Sitesuche auf deutschen Unternehmens-Webseiten (Bachelors thesis). University of Hildesheim.
2 Jansen, B., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: A study and analysis of user queries on the web. Information Processing and Management, 36, 207-227.   DOI
3 Järvelin, K. (2009). Explaining user performance in information retrieval: Challenges to IR evaluation. Proceedings of the 2nd International Conference on the Theory of Information Retrieval, 2009 (pp. 289-296). Heidelberg: Springer, Lecture Notes in Computer Science vol. 5766.
4 Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2), 1-224.
5 Kemp, C., & Ramamohanarao, K. (2002). Long-term learning for Web search engines. In Principles and practice of knowledge discovery in databases (PKDD) [LNAI 2431] (pp. 263-274). Springer: Berlin Heidelberg. pp. 263-274.
6 Mandl, T., & Womser-Hacker, C. (2015). Information retrieval. In Encyclopedia of information science and technology (3rd ed.) (pp. 3923-3931). Hershey, PA: Idea Group Reference.
7 Adar, E., Teevan, J., & Dumais, S. T. (2008). Large scale analysis of web revisitation patterns. Proc. ACM Conf. on Human Factors in Computing Systems (CHI) (pp. 1197-1206). ACM Press: New York.
8 Robertson, S. (2008). On the history of evaluation in IR. Journal of Information Science, 34(4), 439-456.   DOI
9 Shaikh, D. A., & Lenz, K. (2006). Where’s the search? Re-examining user expectations of Web objects. Usability News.
10 Wittenberg, R. (1998). Grundlagen computerunterstützter Datenanalyse (2nd ed.). Stuttgart: Lucius und Lucius.
11 Zobel, J., Williams, H. E., & Kimberley, S. (2000). Trends in retrieval system performance. 23rd Australasian Computer Science Conference (ACSC), Jan 31 - Feb 3, Canberra, Australia (pp. 241-249).
12 Borlund, P. (2013). Interactive information retrieval: An introduction. Journal of Information Science Theory and Practice (JISTAP), 1(3), 12-32.   DOI
13 Uhl, J. (2010). Information Retrieval-Studie zur Evaluierung von Site-Search-Systemen (Masters thesis). University of Hildesheim.
14 Armstrong, T. G., Alistair, M., Webber, W., & Zobel, J. (2009). Improvements that don't add up: Ad-hoc retrieval results since 1998. Conference on Information and Knowledge Management (CIKM) (pp. 601-610). ACM Press: New York.
15 Armstrong, T. G., Moffat, A., Webber, W., & Zobel, J. (2009). Has Adhoc retrieval improved since 1994? Proc. Annual Intl. SIGIR Conf., Jul 19-23. ACM Press: New York.
16 Braschler, M., Herget, J., Pfister, J., Schäuble, P., Steinbach, M., & Stuker, J. (2006). Evaluation der Suchfunktion von Schweizer Unternehmens-Websites. Churer Schriften zur Informationswissenschaft. Switzerland: HTW Chur. Retrieved from
17 Furtner, K., Mandl, T., & Womser-Hacker, C. (2015). Effects of auto-suggest on the usability of search in eCommerce. Proc. 14th International Symposium on Information Science (ISI 2015), Zadar, Croatia, May 2015 (pp. 178-190). Glückstadt.
18 Braschler, M., Heuwing, B., Mandl, T., Womser-Hacker, C., Herget, J., Schäuble, P., & Stuker, J. (2007). Evaluation der Suchfunktion deutscher Unternehmens-Websites. Proceedings Wissensorganisation 09: "Wissen - Wissenschaft - Organisation" 12. Tagung der Deutschen ISKO (International Society for Knowledge Organization), Bonn, Oct. 19-21.
19 Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.   DOI