• Journal of the Korean Society for information Management
• /
• v.13 no.2
• /
• pp.143-163
• /
• 1996

Relevance judgments play a very important role in evaluation of information systems since the degree of success of the information retrieval depends on the relevance judgments. This article reviews the theoretical background of the concept of 'relevance' associated with information retrieval evaluation and tries to identify whether there is any factor that affects relevance judgments. By reviewing previous researches done in the information retrieval evaluation field, four variables have been identified as impacting factors, such as document surrogates presented to judges, the order of presentation, measuring devices of relevance judgments and judges.

• Journal of Internet Computing and Services
• /
• v.1 no.2
• /
• pp.89-96
• /
• 2000

Relevance feedback is a technique to learn the user's subjective perception of similarity between images, and has recently gained attention in Content Based Image Retrieval. Most relevance feedback methods assume that the individual features that are used in similarity judgments do not interact with each other. However, this assumption severely limits the types of similarity judgments that can be modeled In this paper, we explore a more sophisticated model for similarity judgments based on fuzzy measures and the Choquet Integral, and propose a suitable algorithm for relevance feedback, Experimental results show that the proposed method is preferable to traditional weighted- average techniques.

• Journal of the Korean BIBLIA Society for library and Information Science
• /
• v.7 no.1
• /
• pp.231-246
• /
• 1994

A major problem for information retrieval research in the past three decades has been methodology, even though some progress has been made in obtaining useful results from methodologically sound experiments. Within a methodology, potential problems include artificial data generated by the researcher, small sample size interpretation of findings. Critics have pointed out that some room exists for improving methodology of information retrieval research; using existing data, having big enough sample size, including large numbers of search queries, introducing more control in relation to variables, utilizing more appropriate performance measures, conducting rests carefully and evaluating findings properly. Relevance judgments depend entirely on the perception of the user and on the situation of the moment. In an experiment, the best judge of relevance is a user with a well defined information need. Normally more than two categories for relevance judgments are desirable becase there are degrees of relevance. In experimental design, careful control of variables is meeded for internal validity. When no single database exists for comparison, existing operational databases should be used cautiously, Careful control for the variations of search queries, inter-searcher sonsistency, intra-searcher consistency and search strategies is necessary. Parametric statistics requiring rigid assumptions are not appropriate in information retrieval research and non-parametric statistics requiring few assumptions are necessary. Particularly, the sign test and the Wilcoxon test are good alternatives.

• Journal of the Korean Society for information Management
• /
• v.14 no.2
• /
• pp.123-141
• /
• 1997

This study examined the extent to which variations in relevance assessments affect the evaluation of the performance of full-text retrieval system. Four sets of relevance judgments obtained by examining the full-text of documents were used to test the retrieval effectiveness. There was no noticeable difference in retrieval performance among the four relevance judgment sets. It implies that a variety of definitions of relevance has no effect on the evaluation of the performance of the full-text retrieval system. Furth r retrieval experiments on this topic incorporating relevance feedback, which is one of the sophisticated retrieval techniques using relevance information, are suggested.

• Asia Marketing Journal
• /
• v.14 no.2
• /
• pp.23-37
• /
• 2012

Individuals often make their judgments on the basis of the ease or difficulty with which information comes to mind (for reviews, see Greifeneder, Bless, and Pham 2010; Schwarz 1998, 2004). Recent research, however, has documented that variables known to determine the degree of cognitive resources invested in information processing such as personal relevance (Grayson and Schwarz 1999; Rothman and Schwarz 1998), accuracy motivation (Aarts and Dijksterhuis 1999), and processing capacity (Menon and Raghubir 2003) can affect the extent to which individuals draw on metacognitive difficulty in making their judgments. The primary aim of this research is thus to investigate whether individuals with substantial cognitive resources or those with lack of cognitive resources are more likely to draw on metacognitive difficulty when making their product evaluations. The findings from two laboratory experiments indicate that individuals who perceive a greater level of fit between their self-regulatory orientation and temporal construal (Experiment 1), and between their self-construal and the type of product benefit appeal (Experiment 2) are more likely than those who perceive the lack of such fit to evaluate a target product less positively after thinking of many rather than a few positive reasons. The findings provide supporting evidence for the two-stage backward inference process involved with the effect of metacognitive difficulty on consumer judgments in that consumer judgments based on metacognitive difficulty may require greater cognitive resources than those based on the content of information generated. Also, the current research documents further empirical evidence for the relationship between self-regulatory orientation-construal level fit and cognitive resources such that perceived regulatory-construal level fit can increase consumer willingness to invest cognitive resources into their judgment tasks. Last, the findings can help marketers differentiate purchase situations where asking consumers to think of many positive benefits from purchase situations where asking consumers to think of a few key benefits is relatively more beneficial.

• Journal of the Korean Society for information Management
• /
• v.20 no.4 s.50
• /
• pp.23-48
• /
• 2003

In this study an information filtering system was implemented and a series of relevance feedback experiments were conducted using the system. For the relevance feedback, the original queries were searched against the database and the results were reviewed by the researchers. Based on users' online relevance judgements a pair of 17 refined queries were generated using two methods called 'co-occurrence exclusion method' and 'lower frequencies exclusion method,' In order to generate them, the original queries, the descriptors and category codes appeared in either relevant or irrelevant document sets were applied as elements. Users' relevance judgments on the search results of the refined queries were compared and analyzed against those of the original queries.

• Journal of the Korean Society for Library and Information Science
• /
• v.35 no.3
• /
• pp.173-189
• /
• 2001

This paper presents a framework for observable behavior that can be used as a basis for user modeling, and it reports the results of a pair of user studies that examine the joint utility of two specific behaviors. User models can be constructed by hand, or they can be teamed automatically based on feedback provided by the user about the relevance of documents that they have examined. By observing user behavior, it is possible to obtain implicit feedback without requiring explicit relevance judgments. Four broad categories of potentially observable behavior are identified : examine, retain, reference, and annotate, and examples of specific behaviors within a category are further subdivided based on the natural scope of information objects being manipulated . segment object, or class. Previous studies using Internet discussion groups (USENET news) have shown reading time to be a useful source of implicit feedback for predicting a user's preferences. The experiments reported in this paper extend that work to academic and professional journal articles and abstracts, and explore the relationship between printing behavior and reading time. Two user studies were conducted in which undergraduate students examined articles or abstracts from the telecommunications or pharmaceutical literature. The results showed that reading time can be used to predict the user's assessment of relevance, that the mean reading time for journal articles and technical abstracts is longer than has been reported for USENET news documents, and that printing events provide additional useful evidence about relevance beyond that which can be inferred from reading time. The paper concludes with a brief discussion of the implications of the reported results.

• Journal of the Korean Society for information Management
• /
• v.23 no.2
• /
• pp.97-111
• /
• 2006

The purpose of this study is to investigate what level of difference in precision would be significantly perceived by a human user of an information retrieval system. Not many researches have been conducted with regards to this issue in information retrieval field. Despite the non-significant results, there were several interesting findings in recognizing different levels of precision rates. The correctness of relevance task had little to do with the taken time for the task. In addition, the strong relationship between the subjects' topic familiarity and rate of correct judgments is one of the most interesting results in this study. It turned out that the subjects have more difficulty in a situation they have to judge between the two lists having more non-relevant documents than in a situation they do between the lists haying more relevant documents. Finally, the serious influence from the first top N documents in a list for relevance judgment task has been confirmed.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.14 no.5
• /
• pp.2122-2140
• /
• 2020

Proper and precise embedding of commercial ads within Webpages requires Ad-hoc analysis and understanding of their content. By the successful implementation of this step, both publishers and advertisers gain mutual benefits through increasing their revenues on the one hand, and improving user experience on the other. In this research work, we propose a novel multi-level context-based ads serving approach through which ads will be served at generic publisher websites based on their contextual relevance. In the proposed approach, knowledge encoded in domain-specific and generic semantic repositories is exploited in order to analyze and segment Webpages into sets of contextually-relevant segments. Semantically-enhanced indexes are also constructed to index ads based on their textual descriptions provided by advertisers. A modified cosine similarity matching algorithm is employed to embed each ad from the Ads repository into one or more contextually-relevant segments. In order to validate our proposal, we have implemented a prototype of an ad serving system with two datasets that consist of (11429 ads and 93 documents) and (11000 documents and 15 ads), respectively. To demonstrate the effectiveness of the proposed techniques, we experimentally tested the proposed method and compared the produced results against five baseline metrics that can be used in the context of ad serving systems. In addition, we compared the results produced by our system with other state-of-the-art models. Findings demonstrate that the accuracy of conventional ad matching techniques has improved by exploiting the proposed semantically-enhanced context-based ad serving model.

• Journal of the Korean Society for Library and Information Science
• /
• v.27
• /
• pp.265-289
• /
• 1994

Databases publicly available for online searching permit both citation and subject searching, however, subject searching has dominated the online search environment. Despite the power of citation searching, it may be underutilized This study explored the relationship between the number of cited references used in a citation search and information retrieval effectiveness, a relatively unstudied phenomenon. Three articles in the library and information science literature were chosen to represent sample questions. Cited reference searches were conducted for each article and each of its references. All searches were conducted in Social Scisearch and Scisearch on DIALOG. Relevance judgments on the retrieved citations were obtained from the authors of the original articles. This research focused on analyzing, in terms of information retrieval effectiveness, the overlap among postings sets retrieved by various combinations of cited references. The findings from the three case studies clearly showed that the more cited references used for the citation search, the better the performance, in terms of retrieving more relevant documents, up to a point of diminishing retums. In addition, generally the overall level of overlap among relevant documents sets was found to be low. Therefore, if only some of the cited references among many candidates are used for a citation search, a significant proportion of relevant documents may be missed. The analysis of the characteristics of cited references provided the ways to predict which cited refereces would be useful to improve information retrieval. The findings of this comprehensive exploratory study are of interest for both theoretical and practical reasons. They contribute to the development of a theoretical model for the effective use of the citation search. This model might also be implemented in operational online systems. In addition, the findings potentially will help online searchers improve their search strategies using the citation search so that they can better achieve their information retrieval goals: the retrieval of items relevant to a given question and the suppression of nonrelevant items.