Abstract
As the most common setting where youth access behavioral healthcare, the education sector frequently employs training and follow-up consultation as cornerstone implementation strategies to promote the implementation of evidence-based practices (EBPs). However, these strategies alone are not sufficient to promote desirable implementation (e.g., intervention fidelity) and youth behavioral outcomes (e.g., mitigated externalizing behaviors). Theory-informed pragmatic pre-implementation enhancement strategies (PIES) are needed to prevent the lackluster outcomes of training and consultation. Specifically, social cognitive theory explicates principles that inform the design of PIES content and specify mechanisms of behavior change (e.g., “intentions to implement” (ITI)) to target increasing providers’ responsiveness to training and consultation. This triple-blind parallel randomized controlled trial preliminarily examined the efficacy of a pragmatic PIES based on social cognitive theories (SC-PIES) to improve implementation and youth behavioral outcomes from universal preventive EBPs in the education sector. Teachers from a diverse urban district were recruited and randomly assigned to the treatment (SC-PIES; ntreatment = 22) or active control condition (administrative meeting; ncontrol = 21). Based on the condition assigned, teachers received the SC-PIES or met with administrators before their EBP training. We assessed teachers’ ITI, intervention fidelity, and youth behavioral outcome (academic engagement as an incompatible behavior to externalizing disorders) at baseline, immediately after training, and 6 weeks afterward. A series of ANCOVAs detected sizeable effects of SC-PIES, where teachers who received SC-PIES demonstrated significantly larger improvement in their ITI, intervention fidelity, and youth behaviors as compared to the control. Conditional analyses indicated that teachers’ ITI partially mediated the effect of SC-PIES on intervention fidelity, which in turn led to improved youth behaviors. Findings suggest that theory-informed pragmatic PIES targeting providers’ ITI can boost their responsiveness to implementation strategies, as reflected in improved implementation behaviors and youth behavioral outcomes. The results have implications for targeting motivational mechanisms of behavior change and situating preventive implementation strategies at the intersection between the preparation and active implementation stages of an implementation process. Limitations and implications for research and practice are discussed.
Clinicaltrials.gov: NCT05240222. Registered on: 2/14/2022. Retrospectively registered. https://clinicaltrials.gov/show/NCT05240222
Similar content being viewed by others
Availability of Data and Material
The de-identified datasets are available in the Open Science Framework repository, (osf.io/d5t4m/).
Abbreviations
- PIES:
-
Pre-implementation enhancement strategies
- SC-PIES:
-
Social cognitive theory-informed pre-implementation enhancement strategies
- EBP:
-
Evidence-based practice
- TPB:
-
Theory of planned behavior
- ITI:
-
Intentions to implement
- AET:
-
Academic engaged time
References
Abry, T., Rimm-Kaufman, S. E., Larsen, R. A., & Brewer, A. J. (2013). The influence of fidelity of implementation on teacher–student interaction quality in the context of a randomized controlled trial of the Responsive Classroom approach. Journal of School Psychology, 51(4), 437–453.
Altikriti, S. (2022). Examining the relationship between cognitive ability and arrest using a differential offenses hypothesis: Evidence of inconsistent mediation. Crime & Delinquency, 00111287211057862.
Aronson, J., Fried, C. B., & Good, C. (2002). Reducing the effects of stereotype threat on African American college students by shaping theories of intelligence. Journal of Experimental Social Psychology, 38(2), 113–125.
Anderson, S. F., & Maxwell, S. E. (2017). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52(3), 305–324.
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173.
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (methodological), 57(1), 289–300.
Bevilacqua, L., Hale, D., Barker, E. D., & Viner, R. (2018). Conduct problems trajectories and psychosocial outcomes: A systematic review and meta-analysis. European Child & Adolescent Psychiatry, 27(10), 1239–1260.
Birken, S. A., Haines, E. R., Hwang, S., Chambers, D. A., Bunger, A. C., & Nilsen, P. (2020). Advancing understanding and identifying strategies for sustaining evidence-based practices: A review of reviews. Implementation Science, 15(1), 1–13.
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246–263.
Brownson, R. C., Eyler, A. A., Harris, J. K., Moore, J. B., & Tabak, R. G. (2018). Research full report: Getting the word out: New approaches for disseminating public health science. Journal of Public Health Management and Practice, 24(2), 102.
Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2017). Dissemination and implementation research in health: Translating science to practice. Oxford University Press.
Chafouleas, S. M. (2011). Direct behavior rating: A review of the issues and research in its development. Education and Treatment of Children, 34(4), 575–591.
Christofferson, M., & Sullivan, A. L. (2015). Preservice teachers’ classroom management training: A survey of self-reported training experiences, content coverage, and preparedness. Psychology in the Schools, 52(3), 248–264.
Cialdini, R. B., & Goldstein, N. J. (2004). Social influence: Compliance and conformity. Annual Review of Psychology, 55, 591–621.
Cialdini, R. B. (2001). The science of persuasion. Scientific American, 284(2), 76–81.
Collier-Meek, M., Fallon, L. M., & DeFouw, E. R. (2017). Toward feasible implementation support: E-mailed prompts to promote teachers’ treatment integrity. School Psychology Review, 46(4), 379–394.
Cook, C. R., Fiat, A., Larson, M., Daikos, C., Slemrod, T., Holland, E. A., & Renshaw, T. (2018). Positive greetings at the door: Evaluation of a low-cost, high-yield proactive classroom management strategy. Journal of Positive Behavior Interventions, 20(3), 149-.
Cook, C. R., Frye, M., Slemrod, T., Lyon, A. R., Renshaw, T. L., & Zhang, Y. (2015). An integrated approach to universal prevention: Independent and combined effects of PBIS and SEL on youths’ mental health. School Psychology Quarterly, 30(2), 166.
Cook, C. R., Lyon, A. R., Locke, J., Waltz, T., & Powell, B. J. (2019). Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science, 20(6), 914–935.
Diaz, A., Eisenberg, N., Valiente, C., VanSchyndel, S., Spinrad, T. L., Berger, R., & Southworth, J. (2017). Relations of positive and negative expressivity and effortful control to kindergarteners’ student–teacher relationship, academic engagement, and externalizing problems at school. Journal of Research in Personality, 67, 3–14.
Danga, L., & Korb, K. A. (2014). The effect of treatment diffusion on educational experimental designs. Benin J Educ Stud, 23, 29–37.
Dickie, J., & Shuker, M. J. (2014). Ben 10, superheroes and princesses: Primary teachers’ views of popular culture and school literacy. Literacy, 48(1), 32–38.
Duong, M. T., Bruns, E. J., Lee, K., Cox, S., Coifman, J., Mayworm, A., & Lyon, A. R. (2021). Rates of mental health service utilization by children and adolescents in schools and other common service settings: A systematic review and meta-analysis. Administration and Policy in Mental Health and Mental Health Services Research, 48(3), 420–439.
Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20(3), 308–313.
Edmunds, J. M., Beidas, R. S., & Kendall, P. C. (2013). Dissemination and implementation of evidence–based practices: Training and consultation as implementation strategies. Clinical Psychology: Science and Practice, 20(2), 152.
Elston, D. M. (2021). Participation bias, self-selection bias, and response bias. Journal of the American Academy of Dermatology.
Eisman, A. B., Hutton, D. W., Prosser, L. A., Smith, S. N., & Kilbourne, A. M. (2020). Cost-effectiveness of the Adaptive Implementation of Effective Programs Trial (ADEPT): Approaches to adopting implementation strategies. Implementation Science, 15(1), 1–13.
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191.
Fazel, M., Hoagwood, K., Stephan, S., & Ford, T. (2014). Mental health interventions in schools in high-income countries. The Lancet Psychiatry, 1(5), 377–387.
Filter, K. J., & Brown, J. (2019). Validation of the PBIS-ACT full: An updated measure of staff commitment to implement SWPBIS. Remedial and Special Education, 40(1), 40–50.
Fiorilli, C., Buonomo, I., Romano, L., & Pepe, A. (2020). Teacher confidence in professional training: The predictive roles of engagement and burnout. Sustainability, 12(16), 6345.
Fishman, J., Yang, C., & Mandell, D. (2021). Attitude theory and measurement in implementation science: A secondary review of empirical studies and opportunities for advancement. Implementation Science, 16(1), 1–10.
Glasgow, R. E. (2013). What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Education & Behavior, 40(3), 257–265.
Godin, G., Bélanger-Gravel, A., Eccles, M., & Grimshaw, J. (2008). Healthcare professionals’ intentions and behaviours: A systematic review of studies based on social cognitive theories. Implementation Science, 3(1), 1–12.
Godin, G., & Kok, G. (1996). The theory of planned behavior: a review of its applications to health-related behaviors. American journal of health promotion, 11(2), 87–98.
Hayes, A. F., & Cai, L. (2007). Using heteroskedasticity-consistent standard error estimators in OLS regression: An introduction and software implementation. Behavior Research Methods, 39(4), 709–722.
Hayes, A. F. (2017). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. Guilford publications.
Kazdin, A. E. (2007). Mediators and mechanisms of change in psychotherapy research. Annual Review of Clinical Psychology, 3, 1–27.
Kincaid, D., Childs, K., Blase, K. A., & Wallace, F. (2007). Identifying barriers and facilitators in implementing schoolwide positive behavior support. Journal of Positive Behavior Interventions, 9(3), 174–184.
Kortteisto, T., Kaila, M., Komulainen, J., Mäntyranta, T., & Rissanen, P. (2010). Healthcare professionals’ intentions to use clinical guidelines: A survey using the theory of planned behaviour. Implementation Science, 5(1), 1–10.
Larson, M., Cook, C. R., Brewer, S. K., Pullmann, M. D., Hamlin, C., Merle, J. L., & Lyon, A. R. (2021). Examining the effects of a brief, group-based motivational implementation strategy on mechanisms of teacher behavior change. Prevention Science, 22(6), 722–736.
Lee, P., & Bierman, K. L. (2015). Classroom and teacher support in kindergarten: Associations with the behavioral and academic adjustment of low-income students. Merrill-Palmer quarterly (Wayne State University. Press), 61(3), 383.
Leflot, G., Van Lier, P. A., Verschueren, K., Onghena, P., & Colpin, H. (2011). Transactional associations among teacher support, peer social preference, and child externalizing behavior: A four-wave longitudinal study. Journal of Clinical Child & Adolescent Psychology, 40(1), 87–99.
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., & Weiner, B. (2018). From classification to causality: Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6, 136.
Low, S., Smolkowski, K., & Cook, C. (2016). What constitutes high-quality implementation of SEL programs? A latent class analysis of second step implementation. Prevention Science, 17(8), 981–991.
Lyon, A. R., & Bruns, E. J. (2019). From evidence to impact: Joining our best school mental health practices with our best implementation strategies. School Mental Health, 11(1).
Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., & Aarons, G. A. (2018). Assessing organizational implementation context in the education sector: confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science, 13(1), 1–14.
Lyon, A. R., Cook, C. R., Duong, M. T., Nicodimos, S., Pullmann, M. D., Brewer, S. K., & Cox, S. (2019). The influence of a blended, theoretically-informed pre-implementation strategy on school-based clinician implementation of an evidence-based trauma intervention. Implementation Science,14(1), 1–16.
Lyon, A. R., Pullmann, M. D., Walker, S. C., & D’Angelo, G. (2017). Community-sourced intervention programs: Review of submissions in response to a statewide call for “promising practices.” APMH,44(1), 16–28.
MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000). Equivalence of the mediation, confounding and suppression effect. Prevention Science,1(4), 173–181.
Mangurian, C., Niu, G., Schillinger, D., Newcomer, J., Dilley, J., & Handley, M. (2017). Utilization of the behavior change wheel framework to develop a model to improve cardiometabolic screening for people with severe mental illness. Imp Sci, 12(1), 1–16.
McLeod, B. D., Cox, J. R., Jensen-Doss, A., Herschell, A., Ehrenreich-May, J., & Wood, J. J. (2018). Proposing a mechanistic model of clinician training and consultation. Clinical Psychology: Science and Practice, 25(3)
Mitchell, M. M., & Bradshaw, C. P. (2013). Examining classroom influences on student perceptions of school climate: The role of classroom management and exclusionary discipline strategies. Journal of School Psychology, 51(5), 599–610.
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., & Altman, D. G. (2012). CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. International Journal of Surgery, 10(1), 28.
Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364–386.
Nagro, S. A., Fraser, D. W., & Hooks, S. D. (2019). Lesson planning with engagement in mind: Proactive classroom management strategies for curriculum instruction. Intervention in School and Clinic, 54(3), 131–140.
Office of the Surgeon General. (2022). Protecting Youth Mental Health: US Surgeon General’s Advisory. Retrieved October 25, 2022, from https://www.hhs.gov/surgeongeneral/reports-and-publications/youth-mental-health/index.html
Phillippi, S., Beiter, K., Thomas, C., & Vos, S. (2020). Identifying gaps and using evidence-based practices to serve the behavioral health treatment needs of medicaid-insured children. Children and Youth Services Review, 115, 105089
Pence, B., Gaynes, B. , Thielman, N. Heine, A., Mugavero, M., Turner, E., & Quinlivan, E. (2015). Balancing contamination and referral bias in a randomized clinical trial: An application of pseudo-cluster randomization. American Journal of Epidemiology,
Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S. (2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services & Research, 44(2), 177–194.
Pratkanis, A. R. (2011). Social influence analysis: An index of tactics. In The science of social influence (pp. 17–82). Psychology Press.
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 1–11.
Rathvon, N. (2008). Effective school interventions: Evidence-based strategies for improving student outcomes. Guilford Press.
Regan, T., Tubman, J. G., & Schwartz, S. J. (2020). Relations among externalizing behaviors, alcohol expectancies and alcohol use problems in a multi-ethnic sample of middle and high school students. Substance abuse: research and treatment, 14.
Robertson, R. E., Buonomo, K., Abdellatif, H., & DeMaria, S. (2021). Results of a “Psychologically Wise” professional development to increase teacher use of proactive behavior management strategies. Psychology in the Schools, 58(9), 1724–1740.
Rodwell, L., Romaniuk, H., Nilsen, W., Carlin, J. B., Lee, K. J., & Patton, G. C. (2018). Adolescent mental health and behavioral predictors of being NEET: A prospective study of young adults not in employment. Psychological Medicine, 48(5), 861–871.
Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445.
Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai, G. (2008). Evidence-based practices in classroom management: Considerations for research to practice. Education and treatment of children, 351–380.
Sohl, S. J., & Moyer, A. (2007). Tailored interventions to promote mammography screening: A meta-analytic review. Preventive Medicine, 45(4), 252–261.
Spangenberg, E. R., & Greenwald, A. G. (2001). Self-prophecy as a behavior modification technique in the United States.
Splett, J. W., Garzona, M., Gibson, N., Wojtalewicz, D., Raborn, A., & Reinke, W. M. (2019). Teacher recognition, concern, and referral of children’s internalizing and externalizing behavior problems. School Mental Health, 11(2), 228–239.
Steinmetz, H., Knappstein, M., Ajzen, I., Schmidt, P., & Kabst, R. (2016). How effective are behavior change interventions based on the theory of planned behavior?. Zeitschrift für
Taljaard, M., Goldstein, C. E., Giraudeau, B., Nicholls, S. G., Carroll, K., Hey, S. P., & Weijer, C. (2020). Cluster over individual randomization: Are study design choices appropriately justified? Review of a random sample of trials. Clinical Trials, 17(3), 253-.
Teerenstra, S., Melis, R. J. F., Peer, P. G. M., & Borm, G. F. (2006). Pseudo cluster randomization dealt with selection bias and contamination in clinical trials. Journal of Clinical Epidemiology, 59(4), 381–386.
Urbaniak, G. C., & Plous, S. (2013). Research Randomizer (Version 4.0) [Computer software]. Retrieved June 22, 2013, from http://www.randomizer.org/
Vandenberghe, S., Vansteelandt, S., & Loeys, T. (2017). Boosting the precision of mediation analyses of randomised experiments through covariate adjustment. Statistics in Medicine, 36(6), 939–957.
Walton, G. M., & Cohen, G. L. (2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92(1), 82.
Wan, F. (2021). Statistical analysis of two arm randomized pre-post designs with one post-treatment measurement. BMC Medical Research Methodology, 21(1), 1–16.
Wason, J. M., & Robertson, D. S. (2021). Controlling type I error rates in multi-arm clinical trials: A case for the false discovery rate. Pharmaceutical Statistics, 20(1), 109–116.
Weist, M. D., Bruns, E. J., Whitaker, K., Wei, Y., Kutcher, S., Larsen, T., & Short, K. H. (2017). School mental health promotion and intervention: Experiences from four nations. School Psychology International, 38(4), 343–362.
Westerlund, A., Nilsen, P., & Sundberg, L. (2019). Implementation of implementation science knowledge: The research-practice gap paradox. Worldviews on EB Nursing, 16(5), 332.
Williams, N. J., & Beidas, R. S. (2019). Annual research review: The state of implementation science in child psychology and psychiatry: A review and suggestions to advance the field. Journal of Child Psychology and Psychiatry, 60(4), 430–450.
Williams, N. J. (2015). Assessing mental health clinicians’ intentions to adopt evidence-based treatments: Reliability and validity testing of the evidence-based treatment intentions scale. Implementation Science, 11(1), 1–13.
Wittrup, A. R., Hussain, S. B., Albright, J. N., Hurd, N. M., Varner, F. A., & Mattis, J. S. (2019). Natural mentors, racial pride, and academic engagement among black adolescents: Resilience in the context of perceived discrimination. Youth & Society, 51(4), 463–483.
Wood, W. (2000). Attitude change: Persuasion and social influence. Annual review of psychology, 51(1).
Wu, H. K., & Huang, Y. L. (2007). Ninth‐grade student engagement in teacher‐centered and student‐centered technology‐enhanced learning environments. Science education, 91(5).
Yeager, D. S., & Dweck, C. S. (2012). Mindsets that promote resilience: When students believe that personal characteristics can be developed. Educational Psychologist, 47(4), 302–314.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics Approval and Consent to Participate
Informed consent was obtained from all individual participants in the study. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee (University of Washington Human Subject Division, HSD52139EG) and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Conflict of Interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, Y., Cook, C.R., Azad, G.F. et al. A Pre-Implementation Enhancement Strategy to Increase the Yield of Training and Consultation for School-Based Behavioral Preventive Practices: a Triple-Blind Randomized Controlled Trial. Prev Sci 24, 552–566 (2023). https://doi.org/10.1007/s11121-022-01464-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-022-01464-3