Relationship between index term specificity and relevance judgment

https://doi.org/10.1016/j.ipm.2005.12.004Get rights and content

Abstract

Concurrent concepts of specificity are discussed and differentiated from each other to investigate the relationship between index term specificity and users’ relevance judgments. The identified concepts are term-document specificity, hierarchical specificity, statement specificity, and posting specificity. Among them, term-document specificity, which is a relationship between an index term and the document indexed with the term, is regarded as a fruitful research area. In an experiment involving three searches with 175 retrieved documents from 356 matched index terms, the impact of specificity on relevance judgments is analyzed and found to be statistically significant. Implications for index practice and for future research are discussed.

Introduction

Specificity has been one of the core concepts in librarianship and information science from its earliest days. The concern about specificity emerged as a practical issue for cataloguing, especially in allocating subject headings or descriptors to each documentary unit. In 1876, Cutter wrote, “Enter a work under its subject heading, not under the headings of a class, which includes that subject” (p. 37). Since then, specificity has been discussed usually as a criterion for identifying index terms or descriptors. A common focus of the previous research on specificity is on its discriminating ability, which is the ability to distinguish a class of documents from other classes of documents, or one subclass from another. Such a distinguished class should be a class of documents related to a user’s criteria for her/his relevant judgment, which is based on her/his information need. Therefore, a salient research question emerges: What is the impact of specificity on a user’s relevance judgment? Because relevance judgment is a basis for evaluation of effectiveness of an information retrieval system, the impact of specificity, if it exists, is also a factor for effective information retrieval. Although some previous research has focused on the impact of specificity on some notions based on relevance judgment, such as recall and precision (Soergel, 1994, Sparck Jones, 1972, Svenonius, 1971), no research has been identified which addresses the impact of specificity on relevance judgment.

However, several concepts of specificity have been used in research, but these have not been consistent as the list below indicates:

  • (1)

    ‘Indexing’ is more specific than ‘information retrieval’ to a book titled ‘indexing’, because the book is about indexing (term-document specificity).

  • (2)

    ‘Cat’ is more specific than ‘domestic animal’, because ‘cat’ is semantically narrower than ‘domestic animal’ (hierarchical specificity).

  • (3)

    ‘Cancer-Chemotherapy-Congresses’ is more specific than ‘Cancer’, because the longer statement describes more closely the desired document (statement specificity).

  • (4)

    ‘Thesaurus’ is more specific than ‘indexing’ in a bibliographic database named Library and Information Science Abstracts, because ‘thesaurus’ has fewer postings than ‘indexing’ (posting specificity).

This diversity in the meaning of the term has affected research on specificity, and shows different characteristics between the concepts. For instance, Weinberg and Cunningham, 1984, Weinberg and Cunningham, 1985 point out that hierarchical specificity is not always positively correlated with posting specificity when using two groups of terms: core terms and peripheral terms in a subject area. Also, Khosh-Khui (1987) indicates that statement specificity is not related to posting specificity at all. The existence of various concepts of index term specificity allows us to refine and expand the research question, as follows (1) Do the above concepts of specificity affect users’ relevance judgments? (2) What are the relationships among the concepts? (3) Does a combination of the concepts affect relevance judgments more than any single concept?

In spite of the diversity, researchers generally agree on the characteristics of specificity as a relationship in terms of subject, topic, meaning, and/or theme, and as a degree. This means that a specific term indicates that the term is involved in a topical relationship with a document, another term, or a set of documents, and that one term may be more or less specific than another term. This helps to set a framework for the discussion of specificity.

The main purpose of this study is the identification of the significant relationship between relevance judgment and specificity. Also, reliable and acceptable ways to measure types of specificity are explored. This study is purposely done using selected subject areas with a limited number of queries in each area. The unit of analysis is the term and this provides for a large enough sample to suggest that results might be generalizable; however, it is not intended that this, in fact, be done. In effect, this is a methodological study focusing on relevance and specificity. Four concepts of specificity are identified in the next section and, following that, is the report of an experiment investigating relationships between specificity and relevance judgments of users.

Section snippets

Concepts of specificity

It is important to distinguish among the four concepts of specificity: term-document specificity, hierarchical specificity, statement specificity, and posting specificity. Term-document specificity focuses on the topical relationship between an index term and a document indexed with the term. The question regarding this specificity is how accurately the term represents the topic of the document which is indexed with the term. Due to the semantic characteristics of term-document specificity,

Data description

The current data uses data collected in an experiment by Saracevic and his colleagues (Saracevic, 1989, Saracevic et al., 1991) for investigating interaction between users and intermediaries. The original study design called for users to search 40 questions in various subject areas; a total of 6225 documents were retrieved. The relevance of each retrieved document to the users’ information need was judged by the users using a three-point measurement scale: non-relevant, partially-relevant, and

Results

All statistical analyses were done using the Statistical Package of the Social Sciences. The relationships between three types of specificity (t-spec, p-spec, and h-spec) were analyzed using Pearson product–moment correlations. To control Type I error across the three correlations, the Bonferroni approach was used, and the required p-value was set at less than .016 (.05/3). The results of the correlation analyses are presented in Table 4.

The only statistically significant correlation is between

Discussion

According to the results of correlation analysis, the three types of specificity have their own characteristics and it is potentially useful if these are distinguished from each other. The results of ANOVA, LOGIT, and DA show that t-spec is most significant in relation to relevance, and h-spec is not significant at all. t-spec can identify non-relevant documents from partially-relevant and relevant ones, but it does not distinguish between relevant and partially-relevant documents. It shows

Conclusions

This experimental study is the first attempt to investigate the impact of specificity of index terms on users’ relevance judgment, and eventually on effective information retrieval. In previous research on specificity, several concepts of specificity have existed concurrently. Therefore, the concept of specificity is investigated on the basis of previous research. The identified types of specificity are term-document, posting, hierarchical, and statement specificity. Of these, statement

Acknowledgements

Many thanks to Professor James D. Anderson in SCILS, Rutgers University for his assistance. Thanks also to Professor Tefko Saracevic for his warm provision of the raw data for our study, and to Professor Daniel O. O’Connor for his advice on statistics.

References (18)

  • Anderson, J. D., & Pérez-Carballo, J. (2000). Information retrieval database design. Preliminary Draft. School of...
  • J. Balnaves

    Specificity

  • C.A. Cutter

    Rules for a printed dictionary catalogue

    (1876)
  • J.W. Janes

    On the distribution of relevance judgement

    Proceedings of the ASIS Annual Meeting

    (1993)
  • A. Khosh-Khui

    Effects of subject specificity. Part I: Specificity of LC subject heafings and depth of subject analysis in monographic records

    Technical Services Quarterly

    (1987)
  • C.C. Kuhlthau

    Seeking meaning: A process approach to library and information services

    (1993)
  • S. Mizzaro

    Relevance: The whole history

    Journal of American Society for Information Science

    (1997)
  • T. Saracevic

    Modeling and measuring user-intermediary-computer interaction in online searching: Design of a study

  • T. Saracevic et al.

    A study of information seeking and retrieving. I: Background and methodology”

    Journal of American Society for Information Science

    (1988)
There are more references available in the full text version of this article.

Cited by (13)

  • On domain expertise-based roles in collaborative information retrieval

    2014, Information Processing and Management
    Citation Excerpt :

    One main difference remains on the fact that domain experts are more familiar with technical vocabulary (Allen, 1991; Hölscher & Strube, 2000; Kang & Fu, 2010; White et al., 2009) unlike domain novices who need more term suggestion for getting a better insight of the domain (Hsieh-yee, 1993; Hölscher & Strube, 2000). Accordingly, the expertise may be estimated through term specificity (Kim, 2006) but other authors assimilate the expertise level to the document reading level (Kim, Collins-Thompson, Bennett, & Dumais, 2012) which they integrate in a personnalisation model relying either on a language model (Collins-Thompson & Callan, 2004) or a predictive one (Collins-Thompson, Bennett, White, de la Chica, & Sontag, 2011). As well as query terms are more sophisticated for experts, their queries are longer and more keyword-oriented considering their facilities to formulate a well-know information need (White et al., 2009; Kang & Fu, 2010).

View all citing articles on Scopus
View full text