Abstract
Measurement feedback systems (MFSs) that routinely collect and report client progress to mental health therapists have demonstrated beneficial impact on outcomes in numerous studies, with evidence that there is a dose–response relationship related to the implementation of the MFS. The current study examined the impact of MFS implementation (Implementation Index) on youth symptom outcomes separately by caregiver and youth self-report. Additionally, we tested the extent to which Implementation Index rates varied by individual therapists and clients, and whether therapist and client characteristics predicted MFS implementation. Methods: Administrative data (client charts, youth- and caregiver-reported Youth Outcome Questionnaires) for 229 youth (52.83% Latinx, 42.79% girls, M age = 10.33) treated during a 1-year period at a community mental health organization in Central Texas were analyzed using multi-level modeling. Caregiver-reported symptoms decreased faster for those with a higher MFS Implementation Index. Between-group differences among therapists accounted for a significant proportion of variance in the Implementation Index for caregiver report, whereas client differences accounted for most of the variance in the Implementation Index for youth self-report. Therapist trainee status predicted a significant increase in the Implementation Index for caregiver-report data. Youth symptom improvement as reported by caregivers varied with the extent of MFS implementation fidelity, and MFS implementation fidelity was higher for clients treated by trainees relative to staff therapists for caregiver report of symptoms.
Similar content being viewed by others
References
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61–74.
Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.
Amble, I., Gude, T., Ulvenes, P., Stubdal, S., & Wampold, B. E. (2016). How and when feedback works in psychotherapy: Is it the signal? Psychotherapy Research, 26(5), 545–555. https://doi.org/10.1080/10503307.2015.1053552.
American Psychological Association. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285. https://doi.org/10.1037/0003-066X.61.4.271.
Bearman, S. K., & Weisz, J. R. (2015). Review: Comprehensive treatments for youth comorbidity—Evidence-guided approaches to a complicated problem. Child and Adolescent Mental Health, 20(3), 131–141. https://doi.org/10.1111/camh.12092.
Bergman, H., Kornør, H., Nikolakopoulou, A., Hanssen-Bauer, K., Soares-Weiser, K., Tollefsen, T. K., et al. (2018). Client feedback in psychological therapy for children and adolescents with mental health problems (Review). Cochrane Database of Systematic Reviews, 8, CD011729–CD011729. https://doi.org/10.1002/14651858.CD011729.
Bickman, L., Douglas, S. R., De Andrade, A. V., Tomlinson, M., Gleacher, A., Olin, S., et al. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. https://doi.org/10.1007/s10488-015-0647-8.
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429. https://doi.org/10.1176/appi.ps.002052011.
Bickman, L., Rosof-Williams, J., Salzer, M. S., Summerfelt, W. T., Noser, K., Wilson, S. J., et al. (2000). What information do clinicians value for monitoring adolescent client progress and outcomes? Professional Psychology: Research and Practice, 31(1), 70–74. https://doi.org/10.1037/0735-7028.31.1.70.
Bjaastad, J. F., Jensen-Doss, A., Moltu, C., Jakobsen, P., Hagenberg, H., & Joa, I. (2019). Attitudes toward standardized assessment tools and their use among clinicians in a public mental health service. Nordic Journal of Psychiatry, 73(7), 387–396. https://doi.org/10.1080/08039488.1642383.
Brookman-Frazee, L., Haine, R. A., Baker-Ericzén, M., Zoffness, R., & Garland, A. F. (2010). Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 254–269.
Brown, G. S., & Jones, E. R. (2005). Implementation of a feedback system in a managed care environment: What are patients teaching us? Journal of Clinical Psychology/ In Session, 61(2), 187–198. https://doi.org/10.1002/jclp.20110.
Burlingame, G. M., Dunn, T., Cox, J., Wells, G., Lambert, M. J., & Brown, G. S. (2004). Administration and scoring manual for the Youth Outcome Questionnaire-30 (Y-OQ-30). Salt Lake City, UT: Outcome Questionnaire Measures.
Burlingame, G. M., Mosier, J. I., Wells, G. M., Atkin, Q. G., Lambert, M. J., Whoolery, M., et al. (2001). Tracking the influence of mental health treatment: The development of the Youth Outcome Questionnaire. Clinical Psychology & Psychotherapy, 8(5), 361–379.
Cannon, J. N., Warren, J. S., Nelson, P. L., & Burlingame, G. M. (2010). Change trajectories for the Youth Outcome Questionnaire Self-Report: Identifying youth at risk for treatment failure. Journal of Clinical Child and Adolescent Psychology, 39(3), 289–301. https://doi.org/10.1080/15374411003691727.
Cheron, D. M., Chiu, A. A. W., Stanick, C. F., Stern, H. G., Donaldson, A. R., Daleiden, E. L., et al. (2019). Implementing evidence based practices for children’s mental health: A case study in implementing modular treatments in community mental health. Administration and Policy in Mental Health and Mental Health Services Research, 46(3), 391–410. https://doi.org/10.1007/s10488-019-00922-5.
Chorpita, B. F., Daleiden, E. L., & Bernstein, A. D. (2016). At the intersection of health information technology and decision support: Measurement feedback systems… and beyond. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 471–477.
Chorpita, B. F., Daleiden, E. L., Park, A. L., Ward, A. M., Levy, M. C., Cromley, T., … Krull, J. L. (2017). Child STEPs in California: A cluster randomized effectiveness trial comparing modular treatment with community implemented treatment for youth with anxiety, depression, conduct problems, or traumatic stress. Journal of Consulting and Clinical Psychology, 85(1), 13–25.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.
Cooper, M., Stewart, D., Sparks, J., & Bunting, L. (2013). School-based counseling using systematic feedback: A cohort study evaluating outcomes and predictors of change. Psychotherapy Research, 23(4), 474–488. https://doi.org/10.1080/10503307.2012.735777.
Costello, E. J., He, J., Sampson, N. A., Kessler, R. C., & Merikangas, K. R. (2014). Services for adolescents with psychiatric disorders: 12-month data from the National Comorbidity Survey—Adolescent. Psychiatric Services, 65(3), 359–366. https://doi.org/10.1176/appi.ps.201100518.
Costello, E. J., Mustillo, S., Erkanli, A., Keeler, G., & Angold, A. (2003). Prevalence and development of psychiatric disorders in childhood and adolescence. Archives of General Psychiatry, 60, 837–844. https://doi.org/10.1001/archpsyc.60.8.837.
De Haan, A. M., Boon, A. E., De Jong, J. T. V. M., Hoeve, M., & Vermeiren, R. R. J. M. (2013). A meta-analytic review on treatment dropout in child and adolescent outpatient mental health care. Clinical Psychology Review, 33(5), 698–711.
Delgadillo, J., Overend, K., Lucock, M., Groom, M., Kirby, N., McMillan, D., & ... de Jong, K. (2017). Improving the efficiency of psychological treatment using outcome feedback technology. Behaviour Research and Therapy, 99, 89–97.https://doi.org/10.1016/j.brat.2017.09.011.
Douglas, S. R., Jonghyuk, B., de Andrade, A. R. V., Tomlinson, M. M., Hargraves, R. P., & Bickman, L. (2015). Feedback mechanisms of change: How problem alerts reported by youth clients and their caregivers impact clinician-reported session content. Psychotherapy Research, 25(6), 678–693.
Edbrooke-Childs, J., Wolpert, M., & Deighton, J. (2016). Introduction to the special section on implementing feedback from outcome measures in child and adult mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 277–278. https://doi.org/10.1007/s10488-016-0726-5.
Finch, A. E., Lambert, M. J., & Schaalje, B. G. (2001). Psychotherapy quality control: The statistical generation of expected recovery curves for integration into an early warning system. Clinical Psychology and Psychotherapy, 8(1), 231–242.
Gleacher, A. A., Olin, S. S., Nadeem, E., Pollock, M., Ringle, V., Bickman, L., & ... Hoagwood, K. (2016). Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 426–440.https://doi.org/10.1007/s10488-015-0642-0.
Goldberg, S. B., Babins-Wagner, R., Rousmaniere, T., Berzins, S., Hoyt, W. T., Whipple, J. L., & ... Wampold, B. E. (2016). Creating a climate for therapist improvement: A case study of an agency focused on outcomes and deliberate practice. Psychotherapy, 53(3), 367–375.https://doi.org/10.1037/pst0000060.
Gondek, D., Edbrooke-Childs, J., Fink, E., Deighton, J., & Wolpert, M. (2016). Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: A systematic review. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 325–343. https://doi.org/10.1007/s10488-015-0710-5.
Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., et al. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61(2), 155–163.
Hatfield, D., McCullough, L., Frantz, S. B., & Krieger, K. (2010). Do we know when our clients get worse? An investigation of therapists’ ability to detect negative client change. Clinical Psychology & Psychotherapy, 17(1), 25–32.
Hawton, K., Saunders, K. E., & O’Connor, R. C. (2012). Self-harm and suicide in adolescents. The Lancet, 379(9834), 2373–2382. https://doi.org/10.1016/S0140-6736(12)60322-5.
Hedges, L. V. (2009). Effect sizes in nested designs. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 337–356). New York: Russell Sage Foundation.
Hutson, J., Hooke, G. R., & Page, A. C. (2020). Progress monitoring and feedback delivered in routine psychiatric care: Beneficial but not reaching those thought to need it most. Psychotherapy Research. https://doi.org/10.1080/10503307.2019.1710305.
Ionita, G., & Fitzpatrick, M. (2014). Bringing science to clinical practice: A Canadian survey of psychological practice and usage of progress monitoring measures. Canadian Psychology/Psychologie canadienne, 55(3), 187–196. https://doi.org/10.1037/a0037355.
Ionita, G., Fitzpatrick, M., Tomaro, J., Chen, V. V., & Overington, L. (2016). Challenges of using progress monitoring measures: Insights from practicing therapists. Journal of Counseling Psychology, 63(2), 173–182. https://doi.org/10.1037/cou0000122.
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19.
Janse, P. D., De Jong, K., Van Dijk, M. K., Hutschemaekers, G., & Verbraak, M. (2017). Improving the efficiency of cognitive behavioural therapy by using formal client feedback. Psychotherapy Research, 27(5), 525–538. https://doi.org/10.1080/10503307.2016.1152408.
Jensen-Doss, A., Haimes, E. B., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., et al. (2016). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-016-0763-0.
Jensen-Doss, A., Smith, A. M., Becker-Haimes, E. M., Ringle, V. M., Walsh, L. M., Nanda, M., ... & Lyon, A. R. (2018). Individualized progress measures are more acceptable to clinicians than standardized measures: Results of a national survey. Administration and Policy in Mental Health and Mental Health Services Research, 45(3), 392–403.
Kelley, S. D., de Andrade, A. R. V., Bickman, L., & Robin, A. V. (2012). The Session Report Form (SRF): Are clinicians addressing concerns reported by youth and caregivers. Administration and Policy in Mental Health and Mental Health Services Research, 39(1–2), 133–145.
Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L., & ... Moore, M. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. The Cochrane Database of Systematic Reviews.https://doi.org/10.1002/14651858.CD011119
Knaup, C., Koesters, M., Schoefer, D., Becker, T., & Puschner, B. (2009). Effect of feedback of treatment outcome in specialist mental healthcare: Meta-analysis. The British Journal of Psychiatry, 195(1), 15–22.
Lambert, M., Hansen, N., & Harmon, S. (2010). Outcome questionnaire system (The OQ System): Development and practical applications in healthcare settings. In M. Barkham, G. E. Hardy, & J. Mellor-Clark (Eds.), Developing and delivering practice-based evidence: A guide for the psychological therapies (pp. 141–154). Chichester: Wiley.
Lambert, M. J., Whipple, J. L., Bishop, M. J., Vermeersch, D. A., Gray, G. V., & Finch, A. E. (2002). Comparison of empirically-derived and rationally-derived methods for identifying patients at risk for treatment failure. Clinical Psychology & Psychotherapy, 9(3), 149–164. https://doi.org/10.1002/cpp.333.
Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018). Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy (Chicago, Ill.), 55(4), 520–537. https://doi.org/10.1037/pst0000167.
Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., … Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335.
Lutz, W., Rubel, J., Schiefele, A., Zimmermann, D., Böhnke, J. R., & Wittmann, W. W. (2015). Feedback and therapist effects in the context of treatment outcome and treatment length. Psychotherapy Research, 25(6), 647–660. https://doi.org/10.1080/10503307.2015.1053553.
Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 441–466. https://doi.org/10.1007/s10488-016-0719.
Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Knaster Wasse, J., & McCauley, E. (2019). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child & Adolescent Psychology, 48(S1), S168–S179. https://doi.org/10.1080/15274416.2017.1280808.
Merikangas, K. R., He, J. P., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., ... & Swendsen, J. (2010). Lifetime prevalence of mental disorders in US adolescents: Results from the National Comorbidity Survey Replication–Adolescent Supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 49(10), 980–989.
Milette-Winfree, M., Nakamura, B. J., Kotte, A., & Higa-McMillan, C. (2019). Multilevel predictors of case managers’ assessment administration behavior in a precursor to a measurement feedback system. Administration and Policy in Mental Health and Mental Health Services Research, 46(5), 636–648.
Mosier, J., Burlingame, G. M., Wells, M. G., Ferre, R., Latkowski, M., Johansen, J., & Walton, E. (2001). In-home, family-centered psychiatric treatment for high-risk children and youth. Children’s Services: Social Policy, Research, and Practice, 4(2), 51–68.
Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2016). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 379–393.
Norcross, J. C., & Wampold, B. E. (2011). What works for whom: Tailoring psychotherapy to the person. Journal of Clinical Psychology, 67(2), 127–132.
Norman, S., Dean, S., Hansford, L., & Ford, T. (2014). Clinical practitioner’s attitudes towards the use of routine outcome monitoring within child and adolescent mental health services: A qualitative study of two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 19(4), 576–595. https://doi.org/10.1177/1359104513492348.
Pinsof, W. M. (1994). An integrative systems perspective on the therapeutic alliance: Theoretical, clinical and research implications. In A. Horvath & L. Greenberg (Eds.), The working alliance: Theory, research and practice (pp. 173–195). New York: Wiley.
Pinsof, W. M., & Catherall, D. R. (1986). The integrative psychotherapy alliance: Family, couple and individual therapy scales. Journal of Marital and Family Therapy, 2(2), 137–151.
Plener, P. L., Schumacher, T. S., Munz, L. M., & Groschwitz, R. C. (2015). The longitudinal course of non-suicidal self-injury and deliberate self-harm: A systematic review of the literature. Borderline Personality Disorder and Emotion Dysregulation, 2(1), 2. https://doi.org/10.1186/s40479-014-0024-3.
Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., Congdon, R., & Du Toit, M. (2011). HLM statistical software: Version 7. Computer software. Lincolnwood: Scientific Software International Inc.
Rye, M., Rognmo, K., Aarons, G. A., & Skre, I. (2019). Attitudes towards the use of routine outcome monitoring of psychological therapies among mental health providers: The EBPAS–ROM. Administration and Policy in Mental Health and Mental Health Services Research, 46(6), 833–846.
Schiefele, A. K., Lutz, W., Barkham, M., Rubel, J., Böhnke, J., Delgadillo, J., ... & Lambert, M. J. (2017). Reliability of therapist effects in practice-based psychotherapy research: A guide for the planning of future studies. Administration and Policy in Mental Health and Mental Health Services Research, 44(5), 598–613.
Smith, A. M., & Jensen-Doss, A. (2017). Youth psychotherapy outcomes in usual care and predictors of outcome group membership. Psychological Services, 14(1), 66–76. https://doi.org/10.1037/ser0000115.
Walfish, S., McAlister, B., O’Donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports, 110(2), 639–644. https://doi.org/10.2466/02.07.17.PR0.110.2.639-644.
Warren, J. S., Nelson, P. L., Burlingame, G. M., & Mondragon, S. A. (2012). Predicting patient deterioration in youth mental health services: Community mental health vs. managed care settings. Journal of Clinical Psychology, 68(1), 24–40. https://doi.org/10.1002/jclp.20831.
Warren, J. S., Nelson, P. L., Mondragon, S. A., Baldwin, S. A., & Burlingame, G. M. (2010). Youth psychotherapy change trajectories and outcomes in usual care: Community mental health versus managed care settings. Journal of Consulting and Clinical Psychology, 78(2), 144–155. https://doi.org/10.1037/a0018544.
Weisz, J. R., Chorpita, B. F., Palinkas, L. A., Schoenwald, S. K., Miranda, J., Bearman, S. K., ... & Gray, J. (2012). Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: A randomized effectiveness trial. Archives of General Psychiatry, 69(3), 274–282.
Whitcomb, K. E., Woodland, S. C., & Burlingame, G. M. (2018). Do clinicians really use feedback-monitoring systems? A qualitative analysis of 16 group leaders. Psychotherapy, 55(2), 132–143.
Worthen, V. E., & Lambert, M. J. (2007). Outcome oriented supervision: Advantages of adding systematic client tracking to supportive consultations. Counselling & Psychotherapy Research, 7(1), 48–53. https://doi.org/10.1080/14733140601140873.
Acknowledgements
We are grateful to the leadership, therapists, and staff at the Austin Child Guidance Center, and give particular thanks to Julia Hoke, Ph.D., for her support of this study and tireless efforts on behalf of children and families.
Author information
Authors and Affiliations
Contributions
Authors RS and SKB contributed to the study conception and design and developed the initial outline and first draft of the manuscript. All authors contributed to data analysis and subsequent manuscript drafts.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethics Approval
This retrospective chart review study involving human participants was in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The study was reviewed and deemed exempt (classified as a program evaluation) by the University of Texas’s Institutional Review Board (Study No. 2018-01-0164).
Informed Consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sale, R., Bearman, S.K., Woo, R. et al. Introducing a Measurement Feedback System for Youth Mental Health: Predictors and Impact of Implementation in a Community Agency. Adm Policy Ment Health 48, 327–342 (2021). https://doi.org/10.1007/s10488-020-01076-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-020-01076-5