Abstract
Conceptual models of implementation posit contextual factors and their associations with evidence-based practice (EBP) use at multiple levels and suggest these factors exhibit complex cross-level interactions. Little empirical work has examined these interactions, which is critical to advancing causal implementation theory and optimizing implementation strategy design. Mixed effects regression examined cross-level interactions between clinician (knowledge, attitudes) and organizational characteristics (culture, climate) to predict cognitive-behavioral and psychodynamic therapy use with youth (N = 247 clinicians across 28 agencies). Results indicated several interactions, highlighting the importance of attending to interactions between variables at multiple levels to advance multilevel implementation theory and strategies.
Similar content being viewed by others
Notes
As the high number of predictors and interactions may stretch the limits of the sample size and increases risk for multicollinearity, we also tested each interaction independently. Results were largely unchanged, suggesting we retain the overall models for parsimony. Inter-correlations between subscales of the EBPAS were small to moderate (all r < .33) further supporting the decision to retain the overall models. Inter-correlations between organizational proficiency and implementation climate were high (r = .50), suggesting each organizational variable be examined separately.
References
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61–74. https://doi.org/10.1023/b:mhsr.0000024351.12294.65.
Aarons, G. A. (2005). Measuring provider attitudes toward evidence-based practice: Consideration of organizational context and individual differences. Child and Adolescent Psychiatric Clinics of North America, 14(2), 255–271. https://doi.org/10.1016/j.chc.2004.04.008.
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015). Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10(1), 11. https://doi.org/10.1186/s13012-014-0192-y.
Aarons, G. A., Glisson, C., Green, P. D., Hoagwood, K., Kelleher, K. J., & Landsverk, J. A. (2012). The organizational social context of mental health services and clinician attitudes toward evidence-based practice: A United States national study. Implementation Science, 7(1), 56. https://doi.org/10.1186/1748-5908-7-56.
Aarons, G. A., Glisson, C., Hoagwood, K., Kelleher, K., Landsverk, J., & Cafri, G. (2010). Psychometric properties and US National norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychological Assessment, 22(2), 356. https://doi.org/10.1037/a0019188.
Abbass, A. A., Rabung, S., Leichsenring, F., Refseth, J. S., & Midgley, N. (2013). Psychodynamic psychotherapy for children and adolescents: A meta-analysis of short-term psychodynamic models. Journal of the American Academy of Child and Adolescent Psychiatry, 52(8), 863–875. https://doi.org/10.1016/j.jaac.2013.05.014.
Becker, E. M., Smith, A. M., & Jensen-Doss, A. (2013). Who’s using treatment manuals? A national survey of practicing therapists. Behaviour Research and Therapy, 51(10), 706–710. https://doi.org/10.1016/j.brat.2013.07.008.
Beidas, R. S., Aarons, G., Barg, F., Evans, A., Hadley, T., Hoagwood, K., et al. (2013). Policy to implementation: Evidence-based practice in community mental health - study protocol. Implementation Science. https://doi.org/10.1186/1748-5908-8-38.
Beidas, R. S., Marcus, S., Aarons, G. A., Hoagwood, K. E., Schoenwald, S., Evans, A. C., … & Adams, D. R. (2015). Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatrics, 169(4), 374-382. https://doi.org/10.1001/jamapediatrics.2014.3736
Beidas, R., Skriner, L., Adams, D., Wolk, C. B., Stewart, R. E., Becker-Haimes, E., et al. (2017). The relationship between consumer, clinician, and organizational characteristics and use of evidence-based and non-evidence-based therapy strategies in a public mental health system. Behaviour Research and Therapy, 99, 1–10. https://doi.org/10.1016/j.brat.2017.08.011.
Beidas, R. S., Williams, N. J., Green, P. D., Aarons, G. A., Becker-Haimes, E. M., Evans, A. C., … & Marcus, S. C. (2018). Concordance between administrator and clinician ratings of organizational culture and climate. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 142-151. https://doi.org/10.1007/s10488-016-0776-8
Benjamin Wolk, C., Marcus, S. C., Weersing, V. R., Hawley, K. M., Evans, A. C., Hurford, M. O., et al. (2016). Therapist-and client-level predictors of use of therapy techniques during implementation in a large public mental health system. Psychiatric Services, 67(5), 551–557.
Brookman-Frazee, L., Haine, R. A., Baker-Ericzén, M., Zoffness, R., & Garland, A. F. (2010). Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 254–269. https://doi.org/10.1007/s10488-009-0244-9.
Brookman-Frazee, L., & Stahmer, A. C. (2018). Effectiveness of a multi-level implementation strategy for ASD interventions: Study protocol for two linked cluster randomized trials. Implementation Science, 13(1), 66. https://doi.org/10.1186/s13012-018-0757-2.
Busse, C., Kach, A. P., & Wagner, S. M. (2016). Boundary conditions: What they are, how to explore them, why we need them, and when to consider them. Organizational Research Methods, 20(4), 574–609. https://doi.org/10.1177/1094428116641191.
Creed, T. A., Wolk, C. B., Feinberg, B., Evans, A. C., & Beck, A. T. (2016). Beyond the label: Relationship between community therapists’ self-report of a cognitive behavioral therapy orientation and observed skills. Administration and Policy in Mental Health and Mental Health Services Research, 43(1), 36–43. https://doi.org/10.1007/s10488-014-0618-5.
Czincz, J., & Romano, E. (2013). Childhood sexual abuse: Community-based treatment practices and predictors of use of evidence-based practices. Child and Adolescent Mental Health, 18(4), 240–246. https://doi.org/10.1111/camh.12011.
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. https://doi.org/10.1186/1748-5908-4-50.
De Nadai, A. S., & Storch, E. A. (2013). Design considerations related to short-term psychodynamic psychotherapy. Journal of the American Academy of Child and Adolescent Psychiatry, 52(11), 1213–1214. https://doi.org/10.1016/j.jaac.2013.08.012.
Dorsey, S., McLaughlin, K. A., Kerns, S. E., Harrison, J. P., Lambert, H. K., Briggs, E. C., et al. (2017). Evidence base update for psychosocial treatments for children and adolescents exposed to traumatic events. Journal of Clinical Child & Adolescent Psychology, 46(3), 303–330. https://doi.org/10.1080/15374416.2016.1220309.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. https://doi.org/10.1007/s10464-008-9165-0.
Egger, N., Konnopka, A., Beutel, M. E., Herpertz, S., Hiller, W., Hoyer, J., … & Wiltink, J. (2016). Long‐term cost‐effectiveness of cognitive behavioral therapy versus psychodynamic therapy in social anxiety disorder. Depression and Anxiety, 33(12), 1114-1122. https://doi.org/10.1002/da.22540
Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 157. https://doi.org/10.1186/s13012-014-0157-1.
Ehrhart, M. G., Torres, E. M., Wright, L. A., Martinez, S. Y., & Aarons, G. A. (2016). Validating the implementation climate scale (ICS) in child welfare organizations. Child Abuse and Neglect, 53, 17–26. https://doi.org/10.1016/j.chiabu.2015.10.017.
Garland, A. F., Brookman-Frazee, L., Hurlburt, M. S., Accurso, E. C., Zoffness, R. J., Haine-Schlagel, R., et al. (2010). Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services, 61(8), 788–795. https://doi.org/10.1176/ps.2010.61.8.788.
Glisson, C. (2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review, 5(4), 233–253. https://doi.org/10.1023/a:1020972906177.
Glisson, C., Green, P., & Williams, N. J. (2012). Assessing the organizational social context (OSC) of child welfare systems: Implications for research and practice. Child Abuse and Neglect, 36(9), 621–632. https://doi.org/10.1016/j.chiabu.2012.06.002.
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., et al. (2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 98. https://doi.org/10.1007/s10488-007-0148-5.
Glisson, C., & Williams, N. J. (2015). Assessing and changing organizational social contexts for effective mental health services. Annual Review of Public Health, 36(1), 507–523. https://doi.org/10.1146/annurev-publhealth-031914-122435.
Glisson, C., Williams, N. J., Hemmelgarn, A., Proctor, E., & Green, P. (2016). Increasing clinicians’ EBT exploration and preparation behavior in youth mental health services by changing organizational culture with ARC. Behaviour Research and Therapy, 76, 40–46. https://doi.org/10.1016/j.brat.2015.11.008.
Henggeler, S. W., Chapman, J. E., Rowland, M. D., Halliday-Boykins, C. A., Randall, J., Shackelford, J., et al. (2008). Statewide adoption and initial implementation of contingency management for substance-abusing adolescents. Journal of Consulting and Clinical Psychology, 76(4), 556. https://doi.org/10.1037/0022-006x.76.4.556.
Higa-McMillan, C. K., Francis, S. E., Rith-Najarian, L., & Chorpita, B. F. (2016). Evidence base update: 50 years of research on treatment for child and adolescent anxiety. Journal of Clinical Child & Adolescent Psychology, 45(2), 91–113. https://doi.org/10.1080/15374416.2015.1046177.
Higa-McMillan, C. K., Nakamura, B. J., Morris, A., Jackson, D. S., & Slavin, L. (2015). Predictors of use of evidence-based practices for children and adolescents in usual care. Administration and Policy in Mental Health and Mental Health Services Research, 42(4), 373–383. https://doi.org/10.1007/s10488-014-0578-9.
Hofmann, S. G., Asnaani, A., Vonk, I. J., Sawyer, A. T., & Fang, A. (2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research, 36(5), 427–440. https://doi.org/10.1007/s10608-012-9476-1.
Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 230–244. https://doi.org/10.1007/s10488-009-0251-x.
Jacobs, S. R., Weiner, B. J., & Bunger, A. C. (2014). Context matters: Measuring implementation climate among individuals and groups. Implementation Science, 9(1), 46. https://doi.org/10.1186/1748-5908-9-46.
Jensen-Doss, A., Hawley, K. M., Lopez, M., & Osterberg, L. D. (2009). Using evidence-based treatments: The experiences of youth providers working under a mandate. Professional Psychology: Research and Practice, 40(4), 417. https://doi.org/10.1037/a0014690.
Kam, C.-M., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4(1), 55–63. https://doi.org/10.1023/a:1021786811186.
Kolko, D. J., Cohen, J. A., Mannarino, A. P., Baumann, B. L., & Knudsen, K. (2009). Community treatment of child sexual abuse: A survey of practitioners in the National Child Traumatic Stress Network. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 37–49. https://doi.org/10.1007/s10488-008-0180-0.
LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852. https://doi.org/10.1177/1094428106296642.
Lewis, C. C., Klasnja, P., Powell, B., Tuzzio, L., Jones, S., Walsh-Bailey, C., et al. (2018). From classification to causality: Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6, 136. https://doi.org/10.3389/fpubh.2018.00136.
Lim, A., Nakamura, B. J., Higa-McMillan, C. K., Shimabukuro, S., & Slavin, L. (2012). Effects of workshop trainings on evidence-based practice knowledge and attitudes among youth community mental health providers. Behaviour Research and Therapy, 50(6), 397–406. https://doi.org/10.1016/j.brat.2012.03.008.
Lyon, A. R., Ludwig, K., Romano, E., Leonard, S., Vander Stoep, A., & McCauley, E. (2013). “If it’s worth my time, I will make the time”: School-based providers’ decision-making about participating in an evidence-based psychotherapy consultation program. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 467–481. https://doi.org/10.1007/s10488-013-0494-4.
McCart, M. R., & Sheidow, A. J. (2016). Evidence-based psychosocial treatments for adolescents with disruptive behavior. Journal of Clinical Child & Adolescent Psychology, 45(5), 529–563. https://doi.org/10.1080/15374416.2016.1146990.
McCrae, R. R., & Costa, P. T. (1987). Validation of the five-factor model of personality across instruments and observers. Journal of Personality and Social Psychology, 52(1), 81–90. https://doi.org/10.1037/0022-3514.52.1.81.
Midgley, N., O’Keeffe, S., French, L., & Kennedy, E. (2017). Psychodynamic psychotherapy for children and adolescents: an updated narrative review of the evidence base. Journal of Child Psychotherapy, 43(3), 307–329. https://doi.org/10.1080/0075417x.2017.1323945.
Nelson, T. D., & Steele, R. G. (2007). Predictors of practitioner self-reported use of evidence-based practices: Practitioner training, clinical setting, and attitudes toward research. Administration and Policy in Mental Health and Mental Health Services Research, 34(4), 319–330. https://doi.org/10.1007/s10488-006-0111-x.
Niven, D. J., Mrklas, K. J., Holodinsky, J. K., Straus, S. E., Hemmelgarn, B. R., Jeffs, L. P., et al. (2015). Towards understanding the de-adoption of low-value clinical practices: A scoping review. BMC Medicine, 13(1), 255. https://doi.org/10.1186/s12916-015-0488-z.
Novins, D. K., Green, A. E., Legha, R. K., & Aarons, G. A. (2013). Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. Journal of the American Academy of Child and Adolescent Psychiatry, 52(10), 1009–1025. https://doi.org/10.1016/j.jaac.2013.07.012.
Olin, S. S., Williams, N., Pollock, M., Armusewicz, K., Kutash, K., Glisson, C., et al. (2014). Quality indicators for family support services and their relationship to organizational social context. Administration and Policy in Mental Health and Mental Health Services Research, 41(1), 43–54. https://doi.org/10.1007/s10488-013-0499-z.
Powell, B. J., Beidas, R. S., Rubin, R. M., Stewart, R. E., Wolk, C. B., Matlin, S. L., et al. (2016). Applying the policy ecology framework to Philadelphia’s behavioral health transformation efforts. Administration and Policy in Mental Health and Mental Health Services Research, 43(6), 909–926. https://doi.org/10.1007/s10488-016-0733-6.
Powell, B. J., Mandell, D. S., Hadley, T. R., Rubin, R. M., Evans, A. C., Hurford, M. O., et al. (2017). Are general and strategic measures of organizational context and leadership associated with knowledge and attitudes toward evidence-based practices in public behavioral health settings? A cross-sectional observational study. Implementation Science, 12(1), 64. https://doi.org/10.1186/s13012-017-0593-9.
Prasad, V., & Ioannidis, J. P. (2014). Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implementation Science, 9(1), 1. https://doi.org/10.1186/1748-5908-9-1.
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.
Schoenwald, S. K., & Garland, A. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25(1), 146. https://doi.org/10.1037/a0029715.
Stumpf, R. E., Higa-McMillan, C. K., & Chorpita, B. F. (2009). Implementation of evidence-based services for youth: Assessing provider knowledge. Behavior Modification, 33(1), 48–65. https://doi.org/10.1177/0145445508322625.
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337–350. https://doi.org/10.1016/j.amepre.2012.05.024.
Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., … Saul, J. (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3-4), 171-181. https://doi.org/10.1007/s10464-008-9174-z
Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child and Adolescent Psychology, 31(2), 168–180. https://doi.org/10.1207/153744202753604458.
Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., … & Weersing, V. R. (2017). What five decades of research tells us about the effects of youth psychological therapy: A multilevel meta-analysis and implications for science and practice. American Psychologist, 72(2), 79. https://doi.org/10.1037/a0040360
Whetten, D. A. (1989). What constitutes a theoretical contribution? Academy of Management Review, 14(4), 490–495.
Williams, N. J., Ehrhart, M. G., Aarons, G. A., Marcus, S. C., & Beidas, R. S. (2018). Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: Cross-sectional and lagged analyses from a 2-year observational study. Implementation Science, 13(1), 85.
Williams, N. J., & Glisson, C. (2013). Reducing turnover is not enough: The need for proficient organizational cultures to support positive youth outcomes in child welfare. Children and Youth Services Review, 35(11), 1871–1877. https://doi.org/10.1016/j.childyouth.2013.09.002.
Williams, N. J., & Glisson, C. (2014a). The role of organizational culture and climate in the dissemination and implementation of empirically supported treatments for youth. In R. Beidas & P. Kendall (Eds.), Dissemination and implementation of evidence based practices in child and adolescent mental health (pp. 61–81). New York: Oxford University Press.
Williams, N. J., & Glisson, C. (2014b). Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: A United States national study. Child Abuse and Neglect, 38(4), 757–767. https://doi.org/10.1016/j.chiabu.2013.09.003.
Williams, N. J., Glisson, C., Hemmelgarn, A., & Green, P. (2017). Mechanisms of change in the ARC organizational strategy: Increasing mental health clinicians’ EBP adoption through improved organizational culture and capacity. Administration and Policy in Mental Health and Mental Health Services Research, 44(2), 269–283. https://doi.org/10.1007/s10488-016-0742-5.
Zhou, X., Hetrick, S. E., Cuijpers, P., Qin, B., Barth, J., Whittington, C. J., et al. (2015). Comparative efficacy and acceptability of psychotherapies for depression in children and adolescents: A systematic review and network meta-analysis. World Psychiatry, 14(2), 207–222. https://doi.org/10.1002/wps.202.
Zyzanski, S. J., Flocke, S. A., & Dickinson, L. M. (2004). On the nature and analysis of clustered data. The Annals of Family Medicine, 2(3), 199–200.
Funding
Funding for this research project was supported by the National Institute of Mental Health (NIMH) Grant K23MH099179 to Dr. Beidas.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Dr. Beidas receives royalties from Oxford University Press and has served as a consultant for Merck, Dohme, & Sharp. All other authors have nothing to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Becker-Haimes, E.M., Williams, N.J., Okamura, K.H. et al. Interactions Between Clinician and Organizational Characteristics to Predict Cognitive-Behavioral and Psychodynamic Therapy Use. Adm Policy Ment Health 46, 701–712 (2019). https://doi.org/10.1007/s10488-019-00959-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-019-00959-6