Abstract
This study evaluated whether community therapists delivering family therapy for adolescent behavior problems in usual care achieved performance benchmarks established in controlled trials for treatment fidelity and outcomes, with particular focus on individual differences in therapist performance. The study contained N = 38 adolescents (50 % male; mean age 15 years) whose self-reported race/ethnicity was Hispanic (74 %), African American (11 %), multiracial (11 %), and other (4 %). Clients were treated by 13 therapists in one community mental health clinic that delivered family therapy as the routine standard of care. Therapists provided self-report data on adherence to core family therapy techniques; these scores were inflation-adjusted based on concordance with observer reports. Results showed that community therapists surpassed the fidelity benchmark for core family therapy techniques established by research therapists during a controlled trial. Regarding change in client functioning at 6-month follow-up, community therapists were equivalent to the benchmark for internalizing symptoms and superior for externalizing symptoms and delinquent acts. Community therapists also demonstrated a high degree of performance uniformity: Each one approximated the fidelity benchmark, and only two produced relatively weak outcomes on any of the client change indicators. Caveats for interpreting therapist performance data, given the small sample size, are described. Recommendations are made for developing therapist-report fidelity measures and utilizing statistical process control methods to diagnose therapist differences and enhance quality assurance procedures.
Similar content being viewed by others
References
Achenbach, T. M., & Rescorla, L. A. (2001). ASEBA School Age Forms and Profiles. Burlington, Vt.: ASEBA.
Adelson, J. L., & Owen, J. (2012). Bringing the psychotherapist back: Basic concepts for reading articles examining therapist effects using multilevel modeling. Psychotherapy, 49, 152.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th edn., text revision). Washington, DC: APA.
Baldwin, S. A., Christian, S., Berkeljon, A., Shadish, W., & Bean, R. (2012). The effects of family therapies for adolescent delinquency and substance abuse: A meta-analysis. Journal of Marital and Family Therapy, 38, 281–304.
Barber, J. P., Sharpless, B., Klostermann, S., & McCarthy, K. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Research and Practice, 38, 493–500.
Barth, R. P., Kolivoski, K. M., Lindsey, M. A., Lee, B. R., & Collins, K. S. (2014). Translating the common elements approach: Social work’s experiences in education, practice, and research. Journal of Clinical Child & Adolescent Psychology, 43(2), 301–311.
Bearman, S. K., Weisz, J. R., Chorpita, B. F., Hoagwood, K., Ward, A., & Research Network on Youth Mental Health. (2013). More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Administration and Policy in Mental Health and Mental Health Services Research, 40, 518–529.
Beidas, R. S., & Kendall, P. C. (2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17, 1–30.
Blatt, S. J., Sanislow, C. A. III, Zuroff, D. C., & Pilkonis, P. A. (1996). Characteristics of effective therapists: Further analyses of data from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical psychology, 64, 1276.
Bond, G. R., Becker, D. R., & Drake, R. E. (2011). Measurement of fidelity of implementation of evidence-based practices: Case example of the IPS Fidelity Scale. Clinical Psychology: Science and Practice, 18, 126–141.
Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-evaluation of cognitive therapy performance: Do therapists know how competent they are? Behavioural and Cognitive Psychotherapy, 36, 581–587.
Callahan, C. D., & Barisa, M. T. (2005). Statistical process control and rehabilitation outcome: The single-subject design reconsidered. Rehabilitation Psychology, 50, 24–33.
Carroll, K. M., Nich, C., & Rounsaville, B. J. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8, 307–320.
Carroll, K. M., Nich, C., Sifry, R., Nuro, K. F., Frankforter, T. L., et al. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug & Alcohol Dependence, 57, 225–238.
Chorpita, B., Daleiden, E., Ebesutani, C., Young, J., Becker, K., Nakamura, B., et al (2011). Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science and Practice, 18, 154–172.
Chorpita, B. F., Becker, K. D., & Daleiden, E. L. (2007). Understanding the common elements of evidence-based practice: Misconceptions and clinical examples. Journal of the American Academy of Child & Adolescent Psychiatry, 46, 647–652.
Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35, 114–123.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Hillsdale, NJ: Erlbaum.
Crits-Christoph, P., Gibbons, M. B. C., Hamilton, J., Ring-Kurtz, S., & Gallop, R. (2011). The dependability of alliance assessments: The alliance-outcome correlation is larger than you might think. Journal of Consulting and Clinical Psychology, 79, 267–278.
Crits-Christoph, P., & Mintz, J. (1991). Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology, 59, 20.
Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100.
Curtis, N. M., Ronan, K. R., Heiblum, N., & Crellin, K. (2009). Dissemination and effectiveness of multisystemic treatment in New Zealand: a benchmarking study. Journal of Family Psychology, 23, 119.
Delgadillo, J., McMillan, D., Leach, C., Lucock, M., Gilbody, S., & Wood, N. (2014). Benchmarking routine psychological services: a discussion of challenges and methods. Behavioural and Cognitive Psychotherapy, 42, 16–30.
Deming, W. E. (1986). Out of the Crisis. Cambridge, MA: MIT Press.
Dey, M. L., Sluyter, G. V., & Keating, J. E. (1994). Statistical process control and direct care staff performance. Journal of Mental Health Administration, 21, 201–209.
Elkin, I. (1999). A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical Psychology: Science and Practice, 6, 10–32.
Elliott, D. S., Huizinga, D., & Ageton, S. S. (1985). Explaining delinquency and drug use. Beverly Hills, CA: Sage Publications.
Erickson, S. J., Tonigan, J. S., & Winhusen, T. (2012). Therapist effects in a NIDA CTN intervention trial with pregnant substance abusing women: Findings from a RCT with MET and TAU conditions. Alcoholism Treatment Quarterly, 30, 224–237.
Fals-Stewart, W., & Birchler, G. R. (2002). Behavioral couples therapy with alcoholic men and their intimate partners: The comparative effectiveness of bachelor’s and master’s level counselors. Behavior Therapy, 33, 123–147.
Flicker, S. M., Waldron, H. B., Turner, C. W., Brody, J. L., & Hops, H. (2008). Ethnic matching and treatment outcome with Hispanic and Anglo substance-abusing adolescents in family therapy. Journal of Family Psychology, 22, 439.
Garland, A. F., Bickman, L., & Chorpita, B. F. (2010). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 37, 15–26.
Garland, A. F., Haine-Schlagel, R., Brookman-Frazee, L., Baker-Ericzen, M., Trask, E., & Fawley-King, K. (2013). Improving community-based mental health care for children: Translating knowledge into action. Administration and Policy in Mental Health and Mental Health Services Research, 40, 6–22.
Garland, A. F., Hawley, K. M., Brookman-Frazee, L. I., & Hurlburt, M. (2008). Identifying common, core elements of evidence-based practice for children with disruptive behavior disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 47, 505–514.
Green, R. S. (1999). The application of statistical process control to manage global client outcomes in behavioral healthcare. Evaluation and Program Planning, 22, 199–210.
Haley, J. (1987). Problem-solving therapy. San Francisco: Jossey-Bass.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic.
Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. (1999). Multisystemic treatment of substance-abusing and-dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1, 171–184.
Henggeler, S. W., & Sheidow, A. J. (2012). Empirically supported family-based treatments for conduct disorder and delinquency in adolescents. Journal of Marital and Family Therapy, 38, 30–58.
Hill, C. E. (1991). Almost everything you ever wanted to know about how to do process research on counseling and psychotherapy but didn’t know who to ask. In C. E. Hill & L. J. Schneider (Eds.), Research in Counseling (pp. 85–118). Hillsdale, NJ: Erlbaum.
Hill, C. E., O’Grady, K. E., & Elkin, I. (1992). Applying the collaborative study psychotherapy rating scale to rate therapist adherence in cognitive-behavior therapy, interpersonal therapy, and clinical management. Journal of Consulting and Clinical Psychology, 60, 73–79.
Hoagwood, K. E. (2005). Family-based services in children’s mental health: A research review and synthesis. Journal of Child Psychology and Psychiatry, 46, 690–713.
Hoagwood, K. E. (2013). Don’t mourn: Organize. Reviving mental health services research for healthcare quality improvement. Clinical Psychology: Science and Practice, 20, 120–126.
Hogue, A., & Dauber, S. (2013). Assessing fidelity to evidence-based practices in usual care: The example of family therapy for adolescent behavior problems. Evaluation and Program Planning, 37, 21–30.
Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C. E., et al. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35, 137–147.
Hogue, A., Dauber, S., & Henderson, C. E. (2014). Therapist self-report of evidence-based practices in usual care for adolescent behavior problems: Factor and construct validity. Administration and Policy in Mental Health and Mental Health Services Research, 41, 126–139.
Hogue, A., Dauber, S., Henderson, C. E., Bobek, M., Johnson, C., Lichvar, E., & Morgenstern, J. (2015). Randomized trial of family therapy versus non-family treatment for adolescent behavior problems in usual care. Journal of Clinical Child and Adolescent Psychology, 44, 954–969.
Hogue, A., Dauber, S., Henderson, C. E., & Liddle, H. A. (2014). Reliability of therapist self-report on treatment targets and focus in family-based intervention. Administration and Policy in Mental Health and Mental Health Services Research, 41, 697–705.
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42, 229–243.
Hogue, A., Dauber, S., Samuolis, J., & Liddle, H. A. (2006). Treatment techniques and outcomes in multidimensional family therapy for adolescent behavior problems. Journal of Family Psychology, 20, 535–543.
Hogue, A., Henderson, C. E., Ozechowski, T. J., & Robbins, M. S. (2014). Evidence base on outpatient behavioral treatments for adolescent substance use: Updates and recommendations 2007–2013. Journal of Clinical Child and Adolescent Psychology, 43, 697–720.
Hogue, A., Henderson, C. E., & Schmidt, A. T. (2016). Multidimensional predictors of treatment outcome in usual care for adolescent conduct problems and substance use. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-016-0724-7.
Hogue, A., & Liddle, H. A. (2009). Family-based treatment for adolescent substance abuse: Controlled trials and new horizons in services research. Journal of Family Therapy, 31, 126–154.
Hogue, A., Ozechowski, T. J., Robbins, M. R., & Waldron, H. B. (2013). Making fidelity an intramural game: Localizing quality assurance procedures to promote sustainability of evidence-based practices in usual care. Clinical Psychology: Science and Practice, 20, 60–77.
Hoyer, R. W., & Ellis, W. C. (1996). A graphical exploration of SPC: Part 1-SPC’s definitions and procedures. Quality Progress, 29, 65–72.
Hunsley, J., & Lee, C. M. (2007). Research-informed benchmarks for psychological treatments: Efficacy studies, effectiveness studies, and beyond. Professional Psychology: Research and Practice, 38, 21.
Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–244.
Institute of Medicine. (2006). Improving the quality of healthcare for mental and substance-use conditions. Washington, D.C.: National Academy Press.
Laska, K. M., Smith, T. L., Wislocki, A. P., Minami, T., & Wampold, B. E. (2013). Uniformity of evidence-based treatments in practice? Therapist effects in the delivery of cognitive processing therapy for PTSD. Journal of Counseling Psychology, 60, 31.
Löfholm, C. A., Eichas, K., & Sundell, K. (2014). The Swedish implementation of multisystemic therapy for adolescents: Does treatment experience predict treatment adherence? Journal of Clinical Child & Adolescent Psychology, 43, 643–655.
Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19, 181–193.
McHugh, R., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatment: A review of current efforts. American Psychologist, 65, 73–84.
McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodríguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20, 14–32.
Minami, T., Serlin, R. C., Wampold, B. E., Kircher, J. C., & Brown, G. S. (2008). Using clinical trials to benchmark effects produced in clinical practice. Quality and Quantity, 42, 513–525.
Minami, T., Wampold, B. E., Serlin, R. C., Hamilton, E. G., Brown, G. S., & Kircher, J. C. (2008). Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment: A preliminary study. Journal of Consulting and Clinical Psychology, 76, 116–124.
Minuchin, S., & Fishman, H. C. (1981). Family therapy techniques. Cambridge, MA: Harvard University Press.
Muthen, B., & Muthen, L. (2012). Mplus User’s Guide. Los Angeles, CA: Muthen & Muthen.
Noyez, L. (2009). Control charts, cusum techniques, and funnel plots: A review of methods for monitoring performance in healthcare. Interactive Cardiovascular and Thoracic Surgery, 9, 494–499.
Ozechowski, T. J., & Waldron, H. B. (2010). Assertive outreach strategies for narrowing the adolescent substance abuse treatment gap: Implications for research, practice, and policy. Journal of Behavioral Health Services & Research, 37, 40–63.
Project MATCH Research Group (1998). Therapist effects in three treatments for alcohol problems. Psychotherapy Research, 8, 455–474.
Schoenwald, S. K. (2011). It’s a bird, It’s a plane, It’s… fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18, 142–147.
Schoenwald, S. K., Carter, R. E., Chapman, J. E., & Sheidow, A. J. (2008). Therapist adherence and organizational effects on change in youth behavior problems one year after multisystemic therapy. Administration and Policy in Mental Health and Mental Health Services Research, 35, 379–394.
Schoenwald, S. K., Chapman, J. E., Sheidow, A. J., & Carter, R. E. (2009). Long-term youth criminal outcomes in MST transport: The impact of therapist adherence and organizational climate and structure. Journal of Clinical Child & Adolescent Psychology, 38, 91–105.
Schoenwald, S. K., Henggeler, S. W., Brondino, M. J., & Rowland, M. D. (2000). Multisystemic therapy: Monitoring treatment fidelity. Family Process, 39, 83–103.
Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. (2005). Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child and Adolescent Psychology, 34(4), 658–670.
Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E., et al. (1998). The Mini International Neuropsychiatric Interview (MINI): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of Clinical Psychiatry, 59, 22–33.
Southam-Gerow, M. A., McLeod, B. D., Arnold, C. C., Rodríguez, A., Cox, J. R., Reise, S. P., … & Kendall, P. C., et al. (2016). Initial development of a treatment adherence measure for cognitive–behavioral therapy for child anxiety. Psychological Assessment, 28, 70–80.
Spilka, M. J., & Dobson, K. S. (2015). Promoting the internationalization of evidence-based practice: Benchmarking as a strategy to evaluate culturally transported psychological treatments. Clinical Psychology: Science and Practice, 22, 58–75.
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2012). The comparative effectiveness of outpatient treatment for adolescent substance abuse: A meta-analysis. Journal of Substance Abuse Treatment, 44, 145–158.
Wampold, B. E., & Serlin, R. C. (2000). The consequence of ignoring a nested factor on measures of effect size in analysis of variance. Psychological Methods, 5, 425.
Weersing, R. V., & Weisz, J. R. (2002). Community clinic treatment of depressed youth: Benchmarking usual care against CBT clinical trials. Journal of Consulting and Clinical Psychology, 70, 299–310.
Weersing, R. V., Weisz, J. R., & Donenberg, G. R. (2002). Development of the Therapy Procedures Checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology, 31, 168–180.
Weersing, V. R. (2005). Benchmarking the effectiveness of psychotherapy: Program evaluation as a component of evidence-based practice. Journal of the American Academy of Child & Adolescent Psychiatry, 44, 1058–1062.
Weisz, J. R., Chorpita, B. F., Palinkas, L. A., Schoenwald, S. K., Miranda, J., Bearman, S. K., et al. (2012). Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: A randomized effectiveness trial. Archives of General Psychiatry, 69, 274–282.
Wiborg, J. F., Knoop, H., Wensing, M., & Bleijenberg, G. (2012). Therapist effects and the dissemination of cognitive behavior therapy for chronic fatigue syndrome in community-based mental health care. Behaviour Research and Therapy, 50, 393–396.
Zucker, D. M. (1990). An analysis of variance pitfall: The fixed effects analysis in a nested design. Educational and Psychological Measurement, 50, 731–738.
Acknowledgments
Preparation of this article was supported by Grant R01DA023945 from the National Institute on Drug Abuse. The authors are extremely grateful to Takuya Minami for providing technical assistance in completing benchmarking analyses. We acknowledge the dedicated contributions of Molly Bobek, Jaqueline Horan Fisher, Candace Johnson, and Emily Lichvar in supporting this work. We also thank the therapists working at the Roberto Clemente Center who generously agreed to submit regular self-reports of therapeutic interventions in the hopes of advancing the clinical science in their field.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
Dr. Hogue declares that he has no conflict of interest. Dr. Dauber declares that she has no conflict of interest. Dr. Henderson declares that he has no conflict of interest.
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed Consent
Informed consent was obtained from all individual participants included in the study.
Results Involving Animal Studies
This article does not contain any studies with animals performed by any of the authors.
Rights and permissions
About this article
Cite this article
Hogue, A., Dauber, S. & Henderson, C.E. Benchmarking Family Therapy for Adolescent Behavior Problems in Usual Care: Fidelity, Outcomes, and Therapist Performance Differences. Adm Policy Ment Health 44, 626–641 (2017). https://doi.org/10.1007/s10488-016-0769-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-016-0769-7