Abstract
Indirect assessments are a commonly used component of functional behavior assessment by behavior analysts in practice who work with individuals with severe behavior disorders. Although used frequently, closed-ended indirect assessments have repeatedly been shown to have low to moderate interrater reliability and poor concurrent validity with functional analysis. Recently, the use of open-ended interviews has become more commonly adopted in applied clinical practice, despite no studies evaluating the psychometric properties of such assessments. In the present study, we evaluated the interrater reliability and concurrent validity of an open-ended functional assessment interview. We compared the results of two open-ended indirect assessments conducted with a common caregiver and subsequently conducted functional analyses in an attempt to validate hypotheses generated from the interviews. Interrater agreement for the open-ended interviews was higher than previous research on closed-ended interviews (75%); however, concurrent validity with functional analysis was relatively poor (50%). We discuss these findings in the context of assessment and treatment for severe behavior disorders, as well as best practice methods during functional behavior assessment.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Barton-Arwood, S. M., Wehby, J. H., Gunter, P. L., & Lane, K. L. (2003). Functional behavior assessment rating scales: Intrarater reliability with students with emotional or behavioral disorders. Behavioral Disorders, 28, 386–400. https://doi.org/10.1177/019874290302800402
Beaulieu, L., Van Nostrand, M. E., Williams, A. L., & Herscovitch, B. (2018). Incorporating interview-informed functional analyses into practice. Behavior Analysis in Practice, 11, 385–389. https://doi.org/10.1007/s40617-018-0247-7.
Beavers, G. A., & Iwata, B. A. (2011). Prevalence of multiply controlled problem behavior. Journal of Applied Behavior Analysis, 44, 593–597. https://doi.org/10.1901/jaba.2011.44-593.
Beavers, G. A., Iwata, B. A., & Lerman, D. C. (2013). Thirty years of research on the functional analysis of problem behavior. Journal of Applied Behavior Analysis, 46, 1–21. https://doi.org/10.1002/jaba.30.
Bell, M. C., & Fahmie, T. A. (2018). Functional analysis screening for multiple topographies of problem behavior. Journal of Applied Behavior Analysis, 51, 528–537. https://doi.org/10.1002/jaba.462.
Campbell, J. M. (2003). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: A quantitative synthesis of single-subject research. Research in Developmental Disabilities, 24, 120–138. https://doi.org/10.1016/S0891-4222(03)00014-3.
Carr, E. G., Horner, R. H., Turnbull, A. P., Marquis, J. G., McLaughlin, D. M., McAtee, M. L., et al. (1999). Positive behavior support for people with developmental disabilities: A research synthesis. Washington, DC: American Association on Mental Retardation.
Crawford, J., Brockel, B., Schauss, S., & Miltenberger, R. G. (1992). A comparison of methods for the functional assessment of stereotypic behavior. Journal of the Association for Persons With Severe Handicaps, 17, 77–86. https://doi.org/10.1177/154079699201700202.
Didden, R., Duker, P. C., & Korzilius, H. (1997). Meta-analytic study on treatment effectiveness for problem behaviors with individuals who have mental retardation. American Journal on Mental Retardation, 101, 387–399.
Dracobly, J. D., Dozier, C. L., Briggs, A. M., & Juanico, J. F. (2018). Reliability and validity of indirect assessment outcomes: Experts versus caregivers. Learning and Motivation, 62, 77–90. https://doi.org/10.1016/j.lmot.2017.02.007.
Ellingson, S. A., Miltenberger, R. G., Stricker, J., Galensky, T. L., & Garlinghouse, M. (2000). Functional assessment and intervention for challenging behaviors in the classroom by general classroom teachers. Journal of Positive Behavior Interventions, 2, 85–97. https://doi.org/10.1177/109830070000200202.
Fisher, W. W., Greer, B. D., Romani, P. W., Zangrillo, A. N., & Owen, T. M. (2016). Comparisons of synthesized and individual reinforcement contingencies during functional analysis. Journal of Applied Behavior Analysis, 49, 596–616. https://doi.org/10.1002/jaba.314.
Fisher, W. W., Piazza, C. C., Bowman, L. G., & Amari, A. (1996). Integrating caregiver report with a systematic choice assessment to enhance reinforcer identification. American Journal on Mental Retardation, 101, 15–25.
Fryling, M. J., & Baires, N. A. (2016). The practical importance of the distinction between open and closed-ended indirect assessments. Behavior Analysis in Practice, 9, 146–151. https://doi.org/10.1007/s40617-016-0115-2.
Green, G., & Striefel, S. (1988). Response restriction and substitution with autistic children. Journal of the Experimental Analysis of Behavior, 50, 21–32. https://doi.org/10.1901/jeab.1988.50-21.
Gresham, F. M., Watson, T. S., & Skinner, C. H. (2001). Functional behavioral assessment: Principles, procedures, and future directions. School Psychology Review, 30, 156–172.
Hagopian, L. P., Rooker, G. W., & Zarcone, J. R. (2015). Delineating subtypes of self-injurious behavior maintained by automatic reinforcement. Journal of Applied Behavior Analysis, 48, 523–543. https://doi.org/10.1002/jaba.236.
Hanley, G. P. (2012). Functional assessment of problem behavior: Dispelling myths, overcoming implementation obstacles, and developing new lore. Behavior Analysis in Practice, 5, 54–72. https://doi.org/10.1007/BF03391818.
Hanley, G. P., Iwata, B. A., & McCord, B. E. (2003). Functional analysis of problem behavior: A review. Journal of Applied Behavior Analysis, 36, 147–185. https://doi.org/10.1901/jaba.2003.36-147.
Hanley, G. P., Jin, C. S., Vanselow, N. R., & Hanratty, L. A. (2014). Producing meaningful improvements in problem behavior of children with autism via synthesized analyses and treatments. Journal of Applied Behavior Analysis, 47, 16–36. https://doi.org/10.1002/jaba.106.
Herman, C., Healy, O., & Lydon, S. (2018). An interview-informed synthesized contingency analysis to inform the treatment of challenging behavior in a young child with autism. Developmental Neurorehabilitation, 21, 202–207. https://doi.org/10.1177/019874290302800402
Heyvaert, M., Saenen, L., Campbell, J. M., Maes, B., & Onghena, P. (2014). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: An updated quantitative synthesis of single-subject research. Research in Developmental Disabilities, 35, 2463–2476. https://doi.org/10.1016/j.ridd.2014.06.017.
Horner, R. H., Carr, E. G., Strain, P. S., Todd, A. W., & Reed, H. K. (2002). Problem behavior interventions for young children with autism: A research synthesis. Journal of Autism and Developmental Disorders, 32, 423–446. https://doi.org/10.1023/A:1020593922901.
Iwata, B. A., DeLeon, I. G., & Roscoe, E. M. (2013). Reliability and validity of the functional analysis screening tool. Journal of Applied Behavior Analysis, 46, 271–284. https://doi.org/10.1002/jaba.31.
Iwata, B. A., & Dozier, C. L. (2008). Clinical application of functional analysis methodology. Behavior Analysis in Practice, 1, 3–9. https://doi.org/10.1007/BF03391714.
Iwata, B. A., Pace, G. M., Dorsey, M. F., Zarcone, J. R., Vollmer, T. R., Smith, R. G., et al. (1994). The functions of self-injurious behavior: An experimental-epidemiological analysis. Journal of Applied Behavior Analysis, 27, 215–240. https://doi.org/10.1901/jaba.1994.27-215.
Iwata, B. A., Wong, S. E., Riordan, M. M., Dorsey, M. F., & Lau, M. M. (1982). Assessment and training of clinical interviewing skills: Analogue analysis and field replication. Journal of Applied Behavior Analysis, 15, 191–203. https://doi.org/10.1901/jaba.1982.15-191.
Jessel, J., Hanley, G. P., & Ghaemmaghami, M. (2016). Interview-informed synthesized contingency analyses: Thirty replications and reanalysis. Journal of Applied Behavior Analysis, 49, 576–595. https://doi.org/10.1002/jaba.316.
Jessel, J., Ingvarsson, E. T., Metras, R., Kirk, H., & Whipple, R. (2018). Achieving socially significant reductions in problem behavior following the interview-informed synthesized contingency analysis: A summary of 25 outpatient applications. Journal of Applied Behavior Analysis, 51, 130–157. https://doi.org/10.1002/jaba.436.
Kelley, M. E., LaRue, R. H., Roane, H. S., & Gadaire, D. (2011). Indirect behavioral assessments: Interviews and rating scales. In W. W. Fisher, C. C. Piazza, & H. S. Roane (Eds.), Handbook of applied behavior analysis (pp. 182–190). New York, NY: Guilford Publications.
Kuhn, D. E., DeLeon, I. G., Fisher, W. W., & Wilke, A. E. (1999). Clarifying an ambiguous functional analysis with matched and mismatched extinction procedures. Journal of Applied Behavior Analysis, 32, 99–102. https://doi.org/10.1901/jaba.1999.32-99.
Lennox, D. B., & Miltenberger, R. G. (1989). Conducting a functional assessment of problem behavior in applied settings. Journal of the Association for Persons With Severe Handicaps, 14, 304–311. https://doi.org/10.1177/154079698901400409.
Lewis, T. J., Mitchell, B. S., Harvey, K., Green, A., & McKenzie, J. (2015). A comparison of functional behavioral assessment and functional analysis methodology among students with mild disabilities. Behavioral Disorders, 41, 5–20. https://doi.org/10.17988/0198-7429-41.1.5.
Matson, J. L., & Nebel-Schwalm, M. (2007). Assessing challenging behaviors in children with autism spectrum disorders: A review. Research in Developmental Disabilities, 28, 567–579. https://doi.org/10.1016/j.ridd.2006.08.001.
Newcomer, L. L., & Lewis, T. J. (2004). Functional behavioral assessment: An investigation of assessment reliability and effectiveness of function-based interventions. Journal of Emotional and Behavioral Disorders, 12, 168–181. https://doi.org/10.1177/10634266040120030401.
Oliver, A. C., Pratt, L. A., & Normand, M. P. (2015). A survey of functional behavior assessment methods used by behavior analysts in practice. Journal of Applied Behavior Analysis, 48, 817–829. https://doi.org/10.1002/jaba.256.
Paclawskyj, T. R., Matson, J. L., Rush, K. S., Smalls, Y., & Vollmer, T. R. (2001). Assessment of the convergent validity of the Questions About Behavioral Function Scale with analogue functional analysis and the Motivation Assessment Scale. Journal of Intellectual Disability Research, 45(6), 484–494. https://doi.org/10.1046/j.1365-2788.2001.00364.x.
Querim, A. C., Iwata, B. A., Roscoe, E. M., Schlichenmeyer, K. J., Ortega, J. V., & Hurl, K. E. (2013). Functional analysis screening for problem behavior maintained by automatic reinforcement. Journal of Applied Behavior Analysis, 46, 47–60. https://doi.org/10.1002/jaba.26.
Reid, R., & Nelson, J. R. (2002). The utility, acceptability, and practicality of functional behavioral assessment for students with high-incidence problem behaviors. Remedial and Special Education, 23, 15–23. https://doi.org/10.1177/074193250202300103.
Roscoe, E. M., Phillips, K. M., Kelly, M. A., Farber, R., & Dube, W. V. (2015). A statewide survey assessing practitioners’ use and perceived utility of functional assessment. Journal of Applied Behavior Analysis, 48, 830–844. https://doi.org/10.1002/jaba.259.
Roscoe, E. M., Rooker, G. W., Pence, S. T., Longworth, L. J., & Zarcone, J. (2009). Assessing the utility of a demand assessment for functional analysis. Journal of Applied Behavior Analysis, 42, 819–825. https://doi.org/10.1901/jaba.2009.42-819.
Saini, V., Fisher, W. W., & Betz, A. M. (in press). Functional analysis: History and methods. In W. W. Fisher, C. C. Piazza, & H. S. Roane (Eds.), Handbook of applied behavior analysis (2nd ed., pp. ). New York, NY: Guilford.
Saini, V., Fisher, W. W., & Retzlaff, B. J. (2018). Predictive validity and efficiency of ongoing visual-inspection criteria for interpreting functional analyses. Journal of Applied Behavior Analysis, 51, 303–320. https://doi.org/10.1002/jaba.450.
Sigafoos, J., Kerr, M., & Roberts, D. (1994). Interrater reliability of the Motivation Assessment Scale: Failure to replicate with aggressive behavior. Research in Developmental Disabilities, 15, 333–342. https://doi.org/10.1016/0891-4222(94)90020-5.
Slaton, J. D., Hanley, G. P., & Raftery, K. J. (2017). Interview-informed functional analyses: A comparison of synthesized and isolated components. Journal of Applied Behavior Analysis, 50, 252–277. https://doi.org/10.1002/jaba.384.
Sugai, G., Lewis-Palmer, T., & Hagan-Burke, S. (2000). Overview of the functional behavioral assessment process. Exceptionality, 8, 149–160. https://doi.org/10.1207/S15327035EX0803_2.
Umbreit, J. (1996). Functional analysis of disruptive behavior in an inclusive classroom. Journal of Early Intervention, 20, 18–29. https://doi.org/10.1177/105381519602000104.
Zarcone, J. R., Rodgers, T. A., Iwata, B. A., Rourke, D. A., & Dorsey, M. F. (1991). Reliability analysis of the Motivation Assessment Scale: A failure to replicate. Research in Developmental Disabilities, 12, 349–360. https://doi.org/10.1016/0891-4222(91)90031-M.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical Approval
This study was approved by the Research Ethics Board at Surrey Place Centre.
Informed Consent
Informed consent for assessment, treatment, and publication of findings was provided in writing by the substitute decision maker.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Research Highlights
• This study is valuable for behavior analysts in practice who routinely conduct functional behavior assessments or work with individuals with disabilities who engage in severe problem behavior.
• Functional analysis is the gold-standard method for identifying the environmental conditions that maintain severe problem behavior.
• Recently, the use of open-ended indirect assessments to inform functional analysis design has received greater attention in the extant literature on severe behavior disorders.
• This study examined the interrater reliability and concurrent validity of open-ended indirect assessments with functional analysis.
• Interrater reliability and concurrent validity for open-ended interviews were moderate and poor, respectively.
• This study highlights the value and importance of conducting functional analyses as opposed to using less reliable and valid methods, such as indirect assessments, to inform behavioral interventions.
Rights and permissions
About this article
Cite this article
Saini, V., Ubdegrove, K., Biran, S. et al. A Preliminary Evaluation of Interrater Reliability and Concurrent Validity of Open-Ended Indirect Assessment. Behav Analysis Practice 13, 114–125 (2020). https://doi.org/10.1007/s40617-019-00364-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40617-019-00364-3