Skip to main content
SpringerLink
Log in

A fuzzy set theory based computational model to represent the quality of inter-rater agreement

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

In this paper we present a method to evaluate the quality of a rater’s judgement, which can integrate and enrich the use of inter-rater agreement as a reliability measure. Our proposal is an integrative one and evaluates the quality of a rater’s performance through an analysis of the profile of that individual rater’s performance. We discuss its rationale on the basis of the interpretation of inter-rater agreement, highlighting some critical issues. For this purpose, we adopt a computational model based on fuzzy set theory, demonstrating its main characteristics with an exemplary case study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. A strategy that tries to avoid such problems is the centralization of the training, namely only a specific research group, usually the one that has developed the instrument, is recognized as legitimated to train the raters, and this is often formalized in terms of a specific program to be attended for formal certificating (e.g., the procedure for obtaining the certification required for using the Adult Attachment Interview). Yet this solution is only partial, and has more costs than benefits. First, the centralization of the training does not solve the problem of the application of the instrument to clinical material reflecting a different cultural context. Thus, rater B can be trained by rater A whose way of coding is taken as normative; and so B can get a satisfactory level of agreement. Yet such agreement concerns specific objects, the ones used in the training and it is not obvious that it can be generalized to Bs cultural or research contexts. Secondly, due to its cost, the centralization of the training is a mechanism that can treat only a very limited subset of raters and instruments. Above all, the centralization of the training, as with any form of monopoly, reduces the free circulation and exchange of knowledge that is the ground of any scientific community.

  2. The use of the PCA for basic dimension characterization and the description of a clinical process is a procedure widely used in studies using such methods.

  3. Briefly, we set up a specific algorithm based on the concepts of data information and entropy; in particular, this procedure allowed us to reconstruct an histogram related to a dataset with a specific fuzzy set (e.g., a triangular fuzzy set) capturing the whole information, as measured by the entropy formula, that was present in the empirical data. Strictly speaking, if we have a data vector represented by a specific histogram, we can express it by the best fuzzy set capturing the information present in the vector (using a specific formula, De Luca and Termini’s fuzzy weighted entropy), thus the fuzzy set obtained in this way best describes the empirical data vector. For the purposes of this paper, we do not report other details, which can be requested directly from the authors.

  4. The fuzziness computed by the system is related to the definition of the overlap space between the fuzzy sets, and thus the greater is the overlap space, the greater is the possibility of having judgement scores with an high degree of fuzziness.

References

  • Ablon, J., Jones, E.: How expert clinicians’ prototypes of an ideal treatment correlate with outcome in psychodynamic and cognitive-behavioral therapy. Psychother. Res. 8(1), 71–83 (1998)

    Article  Google Scholar 

  • Ablon, J., Jones, E.: Psychotherapy process in the national institute of mental health treatment of depression collaborative research program. J. Consult. Clin. Psychol. 67(1), 64 (1999)

    Article  Google Scholar 

  • Ablon, J., Jones, E.: Validity of controlled clinical trials of psychotherapy: findings from the nimh treatment of depression collaborative research program. Am. J. Psychiatr. 159(5), 775–783 (2002)

    Article  Google Scholar 

  • Ablon, J., Levy, R., Katzenstein, T.: Beyond brand names of psychotherapy: identifying empirically supported change processes. Psychother. Theory Res. Pract. Train. 43(2), 216 (2006)

    Article  Google Scholar 

  • Auletta, A., Salvatore, S., Metrangolo, R., Monteforte, G., Pace, V., Puglisi, M.: The grid of the models of interpretations (gmi): a trans-theoretical method to study therapist interpretive activity. J. Psychother. Integr. 22(2), 61 (2012)

    Article  Google Scholar 

  • Block, R., Reed, M.: Remembered duration: evidence for a contextual-change hypothesis. J. Exp. Psychol. Hum. Learn. Mem. 4(6), 656 (1978)

    Article  Google Scholar 

  • Ciavolino, E.: General distress as second order latent variable estimated trough PLS-PM approach. Electron. J. Appl. Stat. Anal. 5(3), 458–464 (2012)

    Google Scholar 

  • Ciavolino, E., Dahlgaard, J.: Ecsi-customer satisfaction modelling and analysis: a case study. Total Qual. Manag. 18(5), 545–554 (2007)

    Article  Google Scholar 

  • Coppi, R., Gil, M., Kiers, H.: The fuzzy approach to statistical analysis. Comput. Stat. Data Anal. 51(1), 1–14 (2006)

    Article  Google Scholar 

  • Corbetta, P.: Metodologia e tecniche della ricerca sociale. Il Mulino, Bologna (1999)

  • Fleiss, J., Levin, B., Paik, M.: The measurement of interrater agreement. In: Statistical Methods for Rates and Proportions, 3rd edn. pp. 598–626 (2004)

  • Gigerenzer, G., Todd, P.: Fast and frugal heuristics: the adaptive toolbox (1999)

  • Gwet, K.: Handbook of inter-rater reliability. Adv Anal LLC (2001)

  • Hesketh, B., Pryor, R., Gleitzman, M.: Fuzzy logic: toward measuring gottfredson’s concept of occupational social space. J. Couns. Psychol. 36(1), 103 (1989)

    Article  Google Scholar 

  • Hesketh, T., Pryor, R., Hesketh, B.: An application of a computerized fuzzy graphic rating scale to the psychological measurement of individual differences. Int. J. Man Mach. Stud. 29(1), 21–35 (1988)

    Article  Google Scholar 

  • Jaynes, E.: Prior probabilities. Syst. Sci. Cybern. IEEE Trans. 4(3), 227–241 (1968)

    Article  Google Scholar 

  • Jones, E.: Psychoterapy Process Q-set Coding Manual. University of California, Berkeley (1985)

    Google Scholar 

  • Jones, E., Hall, S.A., Parke, L.A.: The process of change: the Berkeley Psychotherapy Research Group. In: Beutler L.E., Crago M. (eds.) Psychotherapy Research: An International Review of Programmatic Studies, pp. 99–106. American Psychological Association, Washington (1991)

  • Jones, E., Pulos, S.: Comparing the process in psychodynamic and cognitive-behavioral therapies. J. Consult. Clin. Psychol. 61(2), 306 (1993)

    Article  Google Scholar 

  • Klir, G., Yuan, B.: Fuzzy Sets and Fuzzy Logic. Prentice Hall, New Jersey (1995)

    Google Scholar 

  • Mannarini, T., Ciavolino, E., Nitti, M., Salvatore, S.: The role of affects in culture-based interventions: implications for practice. Psychology 3(8), 569–577 (2012)

    Article  Google Scholar 

  • Marradi, A.: Misurazione e scale: qualche riflessione e una proposta. Quaderni di sociologia 29, 595–639 (1981)

    Google Scholar 

  • Medasani, S., Kim, J., Krishnapuram, R.: An overview of membership function generation techniques for pattern recognition. Int. J. Approx. Reason. 19(3), 391–417 (1998)

    Article  Google Scholar 

  • Moscovici, S.: La psychanalyse: Son image et son public. Presses Universitaires de France-PUF, Paris (1976)

  • Rampone, S., Russo, C.: A fuzzified BRAIN algorithm for learning DNF from incomplete data. Electron. J. Appl. Stat. Anal. 5(2), 256–270 (2012)

    Google Scholar 

  • Ross, T.: Fuzzy Logic with Engineering Applications. Wiley, New York (2009)

    Google Scholar 

  • Salvatore, S., Freda, M.: Affect, unconscious and sensemaking. A psychodynamic, semiotic and dialogic model. New Ideas Psychol. 29(2), 119–135 (2011)

    Article  Google Scholar 

  • Valsiner, J.: Culture in minds and societies: foundations of cultural psychology. Psychol. Stud. (September 2009) 54, 238–239 (2007)

  • Verkuilen, J., Smithson, M.: Fuzzy Set Theory: Applications in the Social Sciences, vol. 147. Sage Publications, Incorporated (2006)

  • Weick, K.: Sensemaking in Organizations, vol. 3. Sage Publications, Incorporated (1995)

  • Zadeh, L.: Fuzzy sets. Inf. Control 8, 338–353 (1965)

    Article  Google Scholar 

  • Zadeh, L.: Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets Syst. 100, 9–34 (1999)

    Article  Google Scholar 

  • Zadeh, L.: Toward a generalized theory of uncertainty (GTU)–an outline. Inf. Sci. 172, 1–40 (2005)

    Google Scholar 

  • Zétényi, T.: Fuzzy Sets in Psychology, vol. 56. North Holland, Amsterdam (1988)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of History, Society and Human Studies, University of Salento, Palazzo Parlangeli, Via V.M. Stampacchia, 45, 73100 , Lecce, Italy

    Enrico Ciavolino & Sergio Salvatore

  2. Department of Psychology and Cognitive Science, University of Trento, Corso Bettini, 84, 38068 , Rovereto, TN, Italy

    Antonio Calcagnì

Authors
  1. Enrico Ciavolino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sergio Salvatore
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonio Calcagnì
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enrico Ciavolino.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ciavolino, E., Salvatore, S. & Calcagnì, A. A fuzzy set theory based computational model to represent the quality of inter-rater agreement. Qual Quant 48, 2225–2240 (2014). https://doi.org/10.1007/s11135-013-9888-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-013-9888-3

Keywords

Search

Navigation