Abstract
The effects of prior exposure to a subject on three measures of observer performance (interobserver agreement, intraobserver consistency, and accuracy) were investigated in an analogue study. Seventy one undergraduate students who served as experimental observers were randomly assigned to one of four conditions. Observation material consisted of two sets of videotapes, each featuring a child with developmental delay and a therapist. Observers were trained to observe and record three target behaviors. During actual test observations, two observer groups continued to observe the subjects they had been exposed to during training, while the other two groups changed over to the unknown subject pairs. Outcome hypotheses were based on a model of implicit/explicit behavior definitions by Hawkins and Dobes (1977). The data were analyzed by three-factor ANOVAs, one for each of the dependent measures and post hoc analyses. As predicted, prior exposure to subjects resulted in significantly higher interobserver agreement than the observation of novel subjects, while intraobserver consistency was not influenced by prior exposure. Contrary to the expectations, however, was the finding that accuracy was not affected by prior exposure either. The findings were discussed with respect to observer training, particularly as it applies to studies with groups of subjects.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bakeman, R., & Gottman, J. M. (1986).Observing interaction: An introduction to sequential analysis. New York: Cambridge University Press.
Boykin, R. A., & Nelson, R. O. (1981). The effects of instructions and calculation procedures on observers' accuracy, agreement, and calculation correctness.Journal of Applied Behavior Analysis, 14, 479–489.
Cohen, J. (1960). A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20, 37–46.
Dorsey, B. L., Nelson, R. O., & Hayes, S. C. (1986). The effect of code complexity and of behavioral frequency on observer accuracy and interobserver agreement.Behavioral Assessment, 8, 349–363.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face. New Jersey: Prentice-Hall.
Forehand, R. L., & McMahon, R. J. (1981).Helping the noncompliant child. New York: Guilford.
Hahlweg, K., Reissner, L., Kohli, G., Vollmer, M., Schindler, L., & Revensdorf, D. (1984). Development and validity of a new system to analyze interpersonal communication: Kategoriensystem fuer partnerschaftliche Interaktion. In K. Hahlweg & N. S. Jacobson (Eds.),Marital interaction: Analysis and modification. New York: Guilford.
Hawkins, R. P., & Dobes, R. W. (1977). Behavioral definitions in applied behavioral analysis: Explicit or implicit definitions. In B. C. Etzel, J. M. LeBlanc, & D. M. Baer (Eds.),New developments in behavioral research: Theory, methods and applications (pp. 167–188). Hillsdale, NJ: Lawrence Erlbaum.
Johnson, S. M., & Bolstad, O. D. (1973). Methodological issues in naturalistic observation: Some problems and solutions for field research. In L. A. Hamerlinck, L. C. Handy, & E. J. Mash (Eds.),Behavior change: Methodology, concepts and practice (pp 7–67). Champaign, IL: Research Press.
Johnson, S. M., & Lobitz, G. K. (1974). Parental manipulation of child behavior in home observations.Journal of Applied Behavior Analysis, 7, 23–32.
Johnston, J. M., & Pennypacker, H. S. (1980).Strategies and tactics of human behavioral research. New Jersey: Lawrence Erlbaum.
Kazdin, A. E. (1977). Artifact, bias, and complexity of assessment: the ABCs of reliability.Journal of Applied Behavior Analysis, 10, 141–150.
Kent, R. N. (1972).Expectancy bias in behavioral observation. Unpublished doctoral dissertation, Stony Brook: State University of New York.
Kent, R. N., & Foster, S. L. (1977). Direct observational procedures: Methodological issues in naturalistic settings. In A. R. Ciminero, K. S. Calhoun, & H. E. Adams (Eds.),Handbook of behavioral assessment (pp. 279–328) New York: John Wiley & Sons.
Mash, E. J., & Makohoniuk, G. (1975). The effects of prior information and behavioral predictability on observer accuracy.Child Development, 46, 513–519.
Mash, E. J., & McElwee, J. D. (1974). Situational effects on observer accuracy: Behavioral predictability, prior experience, and complexity of coding categories.Child Development, 45, 367–377.
Mercatoris, M., & Craighead, W. E. (1974). The effects of nonparticipant observation on teacher and pupil classroom behavior.Journal of Educational Psychology, 66, 512–519.
O'Leary, K. D., Kent, R. N., & Kanowitz, J. (1975). Shaping data collection congruent with experimental hypothesis.Journal of Applied Behavior Analysis, 8, 43–51.
Powell, J., Martindale, A., & Kulp, S. (1975). An evaluation of time-sample measures of behavior.Journal of Applied Behavior Analysis, 8, 563–570.
Reid, J. B. (1978).A social learning approach to family intervention. Volume 2: Observation in home settings. Eugene, OR: Castaglia.
Rojahn, J. (1978). Validity and reliability of data from naturalistic observational studies—problems and alternatives.Behavioural Analysis and Modification, 2, 296–305.
Rojahn, J., & Wool, R. (1979). Inter- and intra-observer agreement as a function of explicit behavior definitions in direct observation.Behavioural Analysis and Modification, 3, 211–228.
Sackett, G. P. (Ed.) (1978a).Observing behavior Volume I: Theory and applications in mental retardation. Baltimore, MD: University Park Press.
Sackett, G. P. (Ed.) (1978b).Observing behavior Volume II: Data collection and analysis methods. Baltimore, MD: University Park Press.
SAS Institute Inc. (1985).SAS's user's guide: Statistics (5th ed.). Cary, NC: SAS Institute.
Wahler, R. G., House, A. E., & Stambaugh, E. E. (1976).Ecological assessment of child problem behavior. New York: Pergamon Press.
Weick, K. E. (1985). Systematic observational methods. In G. Lindzey, & E. Aronson (Eds.),The handbook of social psychology (pp. 567–634). New York: Random House.
Whalen, C. K., Henker, B., Collins, B. E., Finck, D., & Dotemoto, S. (1979). A social ecology of hyperactive boys: Medication effects in structured classroom environments.Journal of Applied Behavior Analysis, 12, 65–81.
Wolf, V. V., Conger, J. D., & Wolf, D. A. (1986). Social and solipsistic observer training: Effects on agreement with a criterion.Journal of Psychopathology and Behavioral Assessment, 8, 211–226.
Author information
Authors and Affiliations
Additional information
Preparation of the manuscript was supported in part by grants awarded to The Ohio State University Nisonger Center for Mental Retardation and Developmental Disabilities from the U.S. Department of Health and Human Services, Administration on Developmental Disabilities (Grant 07DD0270/16), and Bureau of Maternal and Child Health and Resources Development, Division of Maternal and Child Health (Grant MCJ 922). This paper is based on an experiment conducted in partial fulfillment of the requirements for the first author's Master's degree. The study was presented at the 98th Annual Meeting of the American Psychological Association, Boston (1990).
Rights and permissions
About this article
Cite this article
Rojahn, J., Warren, V.J. The effects of prior exposure to a subject on measures of observer performance. J Psychopathol Behav Assess 13, 389–403 (1991). https://doi.org/10.1007/BF00960449
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF00960449