Browse > Article
http://dx.doi.org/10.3743/KOSIM.2015.32.3.277

A Study on the Utility of Relevance/Non-relevance Information in Homogeneous Documents  

Moon, Sung-Been (연세대학교 문헌정보학과)
Publication Information
Journal of the Korean Society for information Management / v.32, no.3, 2015 , pp. 277-293 More about this Journal
Abstract
This study examined the relative retrieval effectiveness after relevance feedback between two systems (Title/Abstract and Full-text) using four different sets of relevance judgment. Four relevance levels (not relevant, marginally relevant, relevant, highly relevant) are also used, each of which is determined by referees giving a relevance score to documents. This study also investigated how much the average precision was improved after relevance feedback when "marginally relevant" documents are included in the relevant class with the Title/Abstract system, and with the Full-text retrieval system as well. It is found that the Title/Abstract system benefited from relevance feedback with the marginally relevant documents. In case of the Title/Abstract system, the higher percentage of improvement was consistently obtained when including the marginally relevant documents in the relevance class, however the result was vice versa in case of the Full-text retrieval system. It implied that the marginally relevant documents in the relevant class had caused noises in the Full-text retrieval system.
Keywords
relevance; relevance judgment; relevance feedback; full-text retrieval system; retrieval effectiveness;
Citations & Related Records
연도 인용수 순위
  • Reference
1 문성빈 (1993). 적합성피드백을 이용한 전문검색시스템의 효율성 증진을 위한 연구. 정보관리학회지, 10(2), 43-67. (Moon, Sung-Been (1993). Enhancing performance of full-text retrieval systems using relevance feedback. Journal of the Korean Society for Information Management, 10(2), 43-67.)
2 문성빈 (1997). 상이한 적합성 판정과 전문검색시스템의 평가에 관한 연구. 정보관리학회지, 14(2), 123-141. (Moon, Sung-Been (1997). Variations in relevance assessments and evaluation of the performance of full-text retrieval system. Journal of the Korean Society for Information Management, 14(2), 123-141.)
3 Amati, G., & Crestani, F. (1999). Probabilistic learning for selective dissemination of information. Information Processing and Management, 35(5), 633-654.   DOI
4 Belkin, N.J. (1984). Cognitive models and information transfer. Social Science Information Studies, 4, 111-129.   DOI
5 Belkin, N.J., Cool, C., Kelly, D., Lin, S.J., Park, S.Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing and Management, 37(3), 403-434.   DOI
6 Blair, D.C., & Maron, M.E. (1990). Full-text information retrieval: further analysis and clarification. Information Processing and Management, 26(3), 437-447.   DOI
7 Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54(10), 913-925.   DOI   ScienceOn
8 Burgin, R. (1992). Variations in relevance judgements and evaluation of retrieval performance. Information Processing and Management, 28(5), 619-627.   DOI
9 Dang, E.K.F., Luk, R.W.P., Allan, J., Ho, K.S., Chan, S.C.F., Chung, K.F.L., & Lee, D.L. (2010). A new context-dependent term weight computed by boost and discount using relevance information. Journal of the American Society for Information Science and Technology, 61(12), 2514-2530.   DOI
10 Eisenberg, M.B. (1988). Measuring relevance judgments. Information Processing and Management, 24(4), 373-389.   DOI
11 Eisenberg, M.B., & Barry, C. (1988). Order effect: A study of the possible influence of presentation order on user judgments of document relevance. Journal of the American Society for Information Science, 39(1), 37-49.
12 Greisdorf, H. (2003). Relevance threshold: A multi-stage predictive model of how users evaluate information. Information Processing and Management, 39(3), 403-423.   DOI
13 Harter, S.P. (1992). Psychological relevance and information science. Journal of the American Society for Information Science, 43(9), 602-615.   DOI
14 Harter, S.P. (1996). Variation in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47(1), 37-49.   DOI
15 Lopez-Pujalte, C., Guerrero Bote, V. P., & Moya Anegon, F. (2002). A test of genetic algorithms in Relevance Feedback. Information Processing and Management, 38(6), 793-805.   DOI
16 Hjorland, B. (2010). The Foundation of the concept of relevance. Journal of the American Society for Information Science and Technology, 61(2), 217-237.   DOI
17 Huang, X., & Soergel, D. (2013). Relevance: An improved framework for explicating the notion. Journal of the American Society for Information Science and Technology, 64(1), 18-35.   DOI   ScienceOn
18 Kekalainen, J., & Jarvelin, K. (2002). Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology, 53(13), 1120-1129.   DOI
19 Lopez-Pujalte, C., Guerrero Bote, V.P., & Moya Anegon, F. (2003). Order-based fitness functions for genetic algorithms applied to relevance feedback. Journal of the American Society for Information Science and Technology, 54(2), 152-160.   DOI
20 Maglaughlin, K.L., & Sonnenwald, D.H. (2002). User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not relevant. Journal of the American Society for Information Science and Technology, 53(5), 327-342.   DOI
21 Maron, M.E. (1988). Probabilistic design principles for conventional and full-text retrieval systems. Information Processing and Management, 24(3), 249-255.   DOI
22 Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48(9), 810-832.   DOI
23 Quiroga, L.M., & Mostafa, J. (2002). An experiment in building profiles in information filtering: The role of context of user relevance feedback. Information Processing and Management, 38(5), 671-694.   DOI
24 Saracevic, T. (2007b). Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58(13), 2126-2144.   DOI
25 Salton. G., & Buckley, C. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4), 288-297.   DOI
26 Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26(6), 321-343.   DOI
27 Saracevic, T. (2007a). Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, 58(13), 1915-1933.   DOI
28 Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29(1), 3-48.
29 Schamber, L., Eisenberg, M.B., & Nilan, M.S. (1990). A re-examination of relevance: Toward a dynamic, situational, definition. Information Processing & Management, 26(6), 755-776.   DOI
30 Shaw, W.M. Jr., Wood, R.E., & Tibbo, H.R. (1991). The cystic fibrosis database: Content and research opportunities. LISR, 13, 347-366.
31 Sormunen, E. (2002). Liberal relevance criteria of TREC-counting on negligible documents? In M. Beaulieu, R. Baeza-Yates, S. Myaeng, and K. Jarvelin (Eds.), Proceedings of the SIGIR 2002 (pp. 324-330). New York: ACM.
32 Spink, A., Greisdorf, H., & Bateman, J. (1998). From highly relevant to nonrelevant: Examining different regions of relevance. Information Processing and Management, 34(5), 599-622.   DOI
33 Sormunen, E., Kekalainen, J., Koivisto, J., & Jarvelin, K. (2001). Document text characteristics affect the ranking of the most relevant documents by expanded structured queries. Journal of Documentation, 57(3), 358-376.   DOI
34 Spink, A., & Greisdorf, H. (2001). Regions and levels: Measuring and mapping users' relevance judgments. Journal of the American Society for Information Science and Technology, 52(2), 161-173.   DOI
35 Spink, A., & Losee, R.M. (1996). Feedback in information retrieval. Annual Review of Information and Science and Technology, 31, 33-78.
36 Swanson, D.R. (1986). Subjective versus objective relevance in bibliographic retrieval system. The Library Quarterly, 56, 389-398.   DOI
37 Tang, R., Shaw, W.M., & Vevea, J.L. (1999). Towards the identification of the optimal number of relevance categories. Journal of the American Society for Information Science, 50(3), 254-264.   DOI
38 Vakkari, P., & Sormunen, E. (2004). The influence of relevance levels on the effectiveness of interactive information retrieval. Journal of the American Society for Information Science and Technology, 55(11), 963-969.   DOI
39 Voorhees, E. (2001). Evaluation by highly relevant documents. In W. Croft, D. Harper, D. Kraft, and J. Hobel (Eds.), Proceedings of the SIGIR 2001 (pp. 74-82). New York: ACM.
40 Voorhees, E.M., & Harman, D.K. (Eds.) (2005). TREC: Experiment and evaluation in information retrieval. Cambridge: MIT Press.
41 Xu, Y., & Chen, Z. (2006). Relevance judgment: What do information users consider beyond topicality. Journal of the American Society for Information Science and Technology, 57(7), 961-973.   DOI
42 Xu, Y., & Wang, D. (2008). Order effect in relevance judgment. Journal of the American Society for Information Science and Technology, 59(8), 1264-1275.   DOI