Skip to main content
SpringerLink
Log in

A rapid non-parametric estimate of multi-judge reliability

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

A technique is presented for obtaining a rapid estimate of reliability between judges, with special reference to qualitative judgments. It is shown that reliability and discrimination are independent and that estimates of both are needed. A method of obtaining an independent estimate of multi-judge discrimination is developed. It is shown that the size of item-samples is specified by the latter method. Tests of significance for both reliability and discrimination are described.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bliss, C. I. A chart of the chi-square distribution.J. Amer. statist. Assoc., 1944,39, 246–248.

    Google Scholar 

  2. Fisher, R. A., and Yates, F. Statistical tables for biological, agricultural and medical research. (4th Ed.), New York: Hafner, 1953.

    Google Scholar 

  3. Friedman, M. The use of ranks to avoid the assumption of normality.J. Amer. statist. Assoc., 1937,32, 675–701.

    Google Scholar 

  4. Garner, W. R., and Hake, H. W. The amount of information in absolute judgments.Psychol. Rev., 1951,58, 446–459.

    Google Scholar 

  5. Guetzkow, H. Unitizing and categorizing problems in coding qualitative data.J. clin. Psychol., 1950,6, 47–58.

    Google Scholar 

  6. Kogan, L. S., and Hunt, J. McV. Problems of multi-judge reliability.J. clin. Psychol., 1950,6, 16–19.

    Google Scholar 

  7. Pitman, E. J. G. Significance tests which may be applied to samples from any populations. III. The analysis of variance test.Biometrika, 1937,29, 322–335.

    Google Scholar 

  8. Tukey, J. W. Discussion of symposium.J. clin. Psychol., 1950,6, 61–74.

    Google Scholar 

  9. Whitworth, W. A. Choice and chance. New York: Stechert, 1934.

    Google Scholar 

  10. Wilcoxon, F. Some rapid approximate statistical procedures. New York: American Cyanamid Co., 1949.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Chicago, USA

    Desmond S. Cartwright

Authors
  1. Desmond S. Cartwright
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

This technique was developed in connection with research at the Counseling Center, University of Chicago. This investigation was supported by a research grant (PHS M 903) from the National Institute of Mental Health, of the National Institutes of Health, Public Health Service. Acknowledgment is made to Dr. Lyle V. Jones, University of Chicago, for valuable discussions relevant to the technique. Responsibility rests, however, entirely with the writer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cartwright, D.S. A rapid non-parametric estimate of multi-judge reliability. Psychometrika 21, 17–29 (1956). https://doi.org/10.1007/BF02289084

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02289084

Keywords

Search

Navigation