Abstract
Portable electronic data collection devices permit investigators to collect large amounts of observational data in a form ready for computer analysis. These devices are particularly efficient for gathering continuous data on multiple behavior categories. We expect that the increasing availability of these devices will lead to greater use of continuous data collection methods in observational research. This paper addresses the difficulties encountered when calculating traditional interobserver agreement statistics for continuous, multiple-code scoring. Two alternative strategies are described that yield interobserver agreement values based on the exact time of behavior code entries by the primary and secondary observers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Baer, D. M. (1977). Reviewers comment: Just because it's reliable doesn't mean that you can use it.Journal of Applied Behavior Analysis, 10, 117–119.
Cohen, J. (1960). A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20, 37–46.
Deni, R., Szijarto, K., Eisler, A., & Fantauzzo, C. (1983). BASIC programs for observational research using the TRS-80 Model 100 Portable and Model 4 computers.Behavior Research Methods and Instrumentation, 15, 616.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa.Psychological Bulletin, 77, 323–327.
Hartmann, D. P. (1977). Considerations in the choice of interobserver reliability estimates.Journal of Applied Behavior Analysis, 10, 103–116.
Hollenbeck, A. R. (1978). Problems of reliability in observational research. In G. P. Sackett (Ed.),Observing behavior, Vol. 2. Data collection and analysis methods. Baltimore: University Park Press.
Johnson, S. M., & Bolstad, O. D. (1973). Methodological issues in naturalistic observation: Some problems and solutions for field research. In L. A. Hamerlynck, L. C. Handy, & E. J. Mash (Eds.),Behavior change: Methodology, concepts, and practice. Champaign, IL: Research Press.
Jones, R. R., Reid, J. B., & Patterson, G. R. (1975). Naturalistic observation in clinical assessment. In P. McReynolds (Ed.),Advances in psychological assessment, 3. San Francisco: Jossey-Bass.
Kalsher, M. J., Iwata, B. A., Kraufman, D. T., Wuster, R. M., & Criswell, E. L. (1984).Free operant and event based data recording with a hand held computer. Paper presented at the Tenth Annual Convention of the Association for Applied Behavior Analysis, Nashville, TN.
Kazdin, A. E. (1977). Artifact, bias, and complexity of assessment: The ABC's of reliability.Journal of Applied Behavior Analysis, 10, 141–150.
Kratochwill, T. R., & Wetzel, R. J. (1977). Observer agreement, credibility, and judgment: Some considerations in presenting observer agreement data.Journal of Applied Behavior Analysis, 10, 133–139.
Romanczyk, R. G. (1984).Behavior observation in the treatment of severe developmental disability. Paper presented at the Tenth Annual Convention of the Association for Behavior Analysis, Nashville, TN.
Sackett, G. P. (1979). The lag sequential analysis of contingency and cyclicity in behavioral interaction research. In J. D. Osofsky (Ed.),Handbook of infant development. New York: Wiley, pp. 623–649.
Sackett, G. P., Stephenson, E., & Ruppenthal, G. G. (1973). Digital acquistion systems for observing behavior in laboratory and field settings.Behavior Research Methods and Instrumentation, 5, 344–348.
Sanson-Fisher, R. W., Poole, A. D., Small, G. A., & Fleming, I. R. (1979). Data acquisition in real time-an improved system for naturalistic observations.Behavior Therapy, 10, 543–554.
Wampold, B. E., & Holloway, E. L. (1983). A note on interobserver reliability for sequential data.Journal of Behavioral Assessment, 5, 217–226.
Yelton, A. R., Wildman, B. G., & Erickson, M. T. (1977). A probability-based formula for calculating interobserver agreement.Journal of Applied Behavior Analysis, 10, 127–131.
Author information
Authors and Affiliations
Additional information
Work on this paper was supported in part by NICHD Grants P01HD15051 and R01HD17650 and Office of Special Education and Rehabilitation Services Grant G008302980.
Rights and permissions
About this article
Cite this article
MacLean, W.E., Tapp, J.T. & Johnson, W.L. Alternate methods and software for calculating interobserver agreement for continuous observation data. J Psychopathol Behav Assess 7, 65–73 (1985). https://doi.org/10.1007/BF00961847
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF00961847