Relationship between index term specificity and relevance judgment
Introduction
Specificity has been one of the core concepts in librarianship and information science from its earliest days. The concern about specificity emerged as a practical issue for cataloguing, especially in allocating subject headings or descriptors to each documentary unit. In 1876, Cutter wrote, “Enter a work under its subject heading, not under the headings of a class, which includes that subject” (p. 37). Since then, specificity has been discussed usually as a criterion for identifying index terms or descriptors. A common focus of the previous research on specificity is on its discriminating ability, which is the ability to distinguish a class of documents from other classes of documents, or one subclass from another. Such a distinguished class should be a class of documents related to a user’s criteria for her/his relevant judgment, which is based on her/his information need. Therefore, a salient research question emerges: What is the impact of specificity on a user’s relevance judgment? Because relevance judgment is a basis for evaluation of effectiveness of an information retrieval system, the impact of specificity, if it exists, is also a factor for effective information retrieval. Although some previous research has focused on the impact of specificity on some notions based on relevance judgment, such as recall and precision (Soergel, 1994, Sparck Jones, 1972, Svenonius, 1971), no research has been identified which addresses the impact of specificity on relevance judgment.
However, several concepts of specificity have been used in research, but these have not been consistent as the list below indicates:
- (1)
‘Indexing’ is more specific than ‘information retrieval’ to a book titled ‘indexing’, because the book is about indexing (term-document specificity).
- (2)
‘Cat’ is more specific than ‘domestic animal’, because ‘cat’ is semantically narrower than ‘domestic animal’ (hierarchical specificity).
- (3)
‘Cancer-Chemotherapy-Congresses’ is more specific than ‘Cancer’, because the longer statement describes more closely the desired document (statement specificity).
- (4)
‘Thesaurus’ is more specific than ‘indexing’ in a bibliographic database named Library and Information Science Abstracts, because ‘thesaurus’ has fewer postings than ‘indexing’ (posting specificity).
This diversity in the meaning of the term has affected research on specificity, and shows different characteristics between the concepts. For instance, Weinberg and Cunningham, 1984, Weinberg and Cunningham, 1985 point out that hierarchical specificity is not always positively correlated with posting specificity when using two groups of terms: core terms and peripheral terms in a subject area. Also, Khosh-Khui (1987) indicates that statement specificity is not related to posting specificity at all. The existence of various concepts of index term specificity allows us to refine and expand the research question, as follows (1) Do the above concepts of specificity affect users’ relevance judgments? (2) What are the relationships among the concepts? (3) Does a combination of the concepts affect relevance judgments more than any single concept?
In spite of the diversity, researchers generally agree on the characteristics of specificity as a relationship in terms of subject, topic, meaning, and/or theme, and as a degree. This means that a specific term indicates that the term is involved in a topical relationship with a document, another term, or a set of documents, and that one term may be more or less specific than another term. This helps to set a framework for the discussion of specificity.
The main purpose of this study is the identification of the significant relationship between relevance judgment and specificity. Also, reliable and acceptable ways to measure types of specificity are explored. This study is purposely done using selected subject areas with a limited number of queries in each area. The unit of analysis is the term and this provides for a large enough sample to suggest that results might be generalizable; however, it is not intended that this, in fact, be done. In effect, this is a methodological study focusing on relevance and specificity. Four concepts of specificity are identified in the next section and, following that, is the report of an experiment investigating relationships between specificity and relevance judgments of users.
Section snippets
Concepts of specificity
It is important to distinguish among the four concepts of specificity: term-document specificity, hierarchical specificity, statement specificity, and posting specificity. Term-document specificity focuses on the topical relationship between an index term and a document indexed with the term. The question regarding this specificity is how accurately the term represents the topic of the document which is indexed with the term. Due to the semantic characteristics of term-document specificity,
Data description
The current data uses data collected in an experiment by Saracevic and his colleagues (Saracevic, 1989, Saracevic et al., 1991) for investigating interaction between users and intermediaries. The original study design called for users to search 40 questions in various subject areas; a total of 6225 documents were retrieved. The relevance of each retrieved document to the users’ information need was judged by the users using a three-point measurement scale: non-relevant, partially-relevant, and
Results
All statistical analyses were done using the Statistical Package of the Social Sciences. The relationships between three types of specificity (t-spec, p-spec, and h-spec) were analyzed using Pearson product–moment correlations. To control Type I error across the three correlations, the Bonferroni approach was used, and the required p-value was set at less than .016 (.05/3). The results of the correlation analyses are presented in Table 4.
The only statistically significant correlation is between
Discussion
According to the results of correlation analysis, the three types of specificity have their own characteristics and it is potentially useful if these are distinguished from each other. The results of ANOVA, LOGIT, and DA show that t-spec is most significant in relation to relevance, and h-spec is not significant at all. t-spec can identify non-relevant documents from partially-relevant and relevant ones, but it does not distinguish between relevant and partially-relevant documents. It shows
Conclusions
This experimental study is the first attempt to investigate the impact of specificity of index terms on users’ relevance judgment, and eventually on effective information retrieval. In previous research on specificity, several concepts of specificity have existed concurrently. Therefore, the concept of specificity is investigated on the basis of previous research. The identified types of specificity are term-document, posting, hierarchical, and statement specificity. Of these, statement
Acknowledgements
Many thanks to Professor James D. Anderson in SCILS, Rutgers University for his assistance. Thanks also to Professor Tefko Saracevic for his warm provision of the raw data for our study, and to Professor Daniel O. O’Connor for his advice on statistics.
References (18)
- Anderson, J. D., & Pérez-Carballo, J. (2000). Information retrieval database design. Preliminary Draft. School of...
Specificity
Rules for a printed dictionary catalogue
(1876)On the distribution of relevance judgement
Proceedings of the ASIS Annual Meeting
(1993)Effects of subject specificity. Part I: Specificity of LC subject heafings and depth of subject analysis in monographic records
Technical Services Quarterly
(1987)Seeking meaning: A process approach to library and information services
(1993)Relevance: The whole history
Journal of American Society for Information Science
(1997)Modeling and measuring user-intermediary-computer interaction in online searching: Design of a study
- et al.
A study of information seeking and retrieving. I: Background and methodology”
Journal of American Society for Information Science
(1988)
Cited by (13)
On the impact of domain expertise on query formulation, relevance assessment and retrieval performance in clinical settings
2017, Information Processing and ManagementOn domain expertise-based roles in collaborative information retrieval
2014, Information Processing and ManagementCitation Excerpt :One main difference remains on the fact that domain experts are more familiar with technical vocabulary (Allen, 1991; Hölscher & Strube, 2000; Kang & Fu, 2010; White et al., 2009) unlike domain novices who need more term suggestion for getting a better insight of the domain (Hsieh-yee, 1993; Hölscher & Strube, 2000). Accordingly, the expertise may be estimated through term specificity (Kim, 2006) but other authors assimilate the expertise level to the document reading level (Kim, Collins-Thompson, Bennett, & Dumais, 2012) which they integrate in a personnalisation model relying either on a language model (Collins-Thompson & Callan, 2004) or a predictive one (Collins-Thompson, Bennett, White, de la Chica, & Sontag, 2011). As well as query terms are more sophisticated for experts, their queries are longer and more keyword-oriented considering their facilities to formulate a well-know information need (White et al., 2009; Kang & Fu, 2010).
Aspects of 'relevance' in the alignment of curriculum with educational standards
2010, Information Processing and ManagementApproaches to the concepts of exhaustivity and specificity in ISKO International meeting proceedings: 2000-2017
2018, Advances in Knowledge OrganizationOn the collaboration support in information retrieval
2017, ACM Computing SurveysAnalysis of biomedical and health queries: Lessons learned from TREC∗ and CLEF∗∗ evaluation benchmarks
2015, Journal of the Association for Information Science and Technology