Skip to main content
SpringerLink
Log in

Using a Meta-analytic Technique to Assess the Relationship between Treatment Intensity and Program Effects in a Cluster-Randomized Trial

  • Original Paper
  • Published:
Journal of Behavioral Education Aims and scope Submit manuscript

Abstract

School bullying and delinquent behaviors are persistent and pervasive problems for schools, and have lasting effects for all individuals involved (Copeland et al., JAMA Psychiatry 70:419–426, 2013; Espelage et al., J Res Adolesc 24(2):337–349, 2013a). As a result, policymakers and practitioners have attempted to thwart these ill-effects using school-based interventions. Recent meta-analyses have found, however, that these programs produce only moderate effects (Ttofi and Farrington, J Exp Criminol 7:27–56, 2011). Consequently, it is important to investigate further the reasons for such findings. One promising analysis is to assess the relation between treatment intensity variables and program outcomes. Unfortunately, few treatment intensity variables have been utilized in the school-based prevention literature, and it is often cumbersome to model the relation between treatment intensity and outcomes. The purpose of this project, therefore, is to explicate novel measures of treatment intensity and delineate a relatively new meta-analytic technique to model the relation between the variables and program effects. The context for this project is a large-scale, multi-site, cluster-randomized trial; 36 schools and 3,616 students participated in three waves of data collection. The results indicated that, for the second wave of data collection, stronger treatment effects were found when teachers and program implementers spent a greater amount of time prepping lessons, provided additional financial resources, and received outside consultation and support.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • American Educational Research Association. (2013). Prevention of bullying in schools, colleges, and universities: Research report and recommendations. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. West Sussex: Wiley.

    Book  Google Scholar 

  • Brown, E. C., Low, S., Smith, B. H., & Haggerty, K. P. (2011). Outcomes from a school-randomized controlled trial of STEPS to RESPECT: A Bullying Prevention Program. School Psychology Review, 40, 423–443.

    Google Scholar 

  • Committee for Children. (2008). Second step: Student Success through Prevention Program. Seattle, WA: Author.

    Google Scholar 

  • Copeland, W. E., Wolke, D., Angold, A., & Costello, E. J. (2013). Adult psychiatric outcomes of bullying and being bullies by peers in childhood and adolescence. JAMA Psychiatry, 70, 419–426.

    Article  PubMed Central  PubMed  Google Scholar 

  • Cordray, D. S. (2014). Fidelity of implementation.IES summer training institute for cluster-randomized control trials. Lecture conducted at Northwestern University, Evanston, IL.

  • Cordray, D. S., & Pion, G. M. (2006). Treatment strength and integrity: Models and methods. In R. R. Bootzin & P. E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103–124). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Crowley, D. M., Coffman, D. L., Feinberg, M. E., Greenberg, M. T., & Spoth, R. L. (2014). Evaluating the impact of implementation factors on family-based prevention programming: Methods for strengthening causal inference. Prevention Science, 15(2), 246–255.

    Article  PubMed  Google Scholar 

  • Daly, E. J., Martens, B. K., Barnett, D., Witt, J. C., & Olson, S. C. (2007). Varying intervention delivery in response to intervention: Confronting and resolving challenges with measurement, instruction, and intensity. School Psychology Review, 36(4), 562–581.

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.

    Article  PubMed  Google Scholar 

  • Espelage, D. L. (2013). Why are bully prevention programs failing in U.S. schools? Journal of Curriculum and Pedagogy, 10, 121–123.

    Article  Google Scholar 

  • Espelage, D. L., Basile, K. C., & Hamburger, M. E. (2012). Bullying experiences and co-occurring sexual violence perpetration among middle school students: Shared and unique risk factors. Journal of Adolescent Health, 50, 60–65.

    Article  PubMed  Google Scholar 

  • Espelage, D. L., & Holt, M. L. (2001). Bullying and victimization during early adolescence: Peer influences and psychosocial correlates. Journal of Emotional Abuse, 2, 123–142.

    Article  Google Scholar 

  • Espelage, D. L., Holt, M. K., & Henkel, R. R. (2003). Examination of peer-group contextual effects on aggression during early adolescence. Child Development, 74, 205–220.

    Article  PubMed  Google Scholar 

  • Espelage, D. L., Low, S., Polanin, J. R., & Brown, E. C. (2013a). The impact of a middle school program to reduce aggression, victimization, and sexual violence. Journal of Adolescent Health, 53(2), 180–186.

    Article  PubMed  Google Scholar 

  • Espelage, D. L., Low, S., Rao, M. A., Hong, J. S., & Little, T. (2013b). Family violence, bullying, fighting, and substance use among adolescents: A longitudinal transactional model. Journal of Research on Adolescence, 24(2), 337–349.

    Article  Google Scholar 

  • Fisher, Z., & Tipton, E. (2014). robumeta: Robust variance meta-regression (Version 1.0). Retrieved from: http://cran.r-project.org/web/packages/robumeta/robumeta.pdf.

  • Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.

    Article  Google Scholar 

  • Higgins, J., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558.

    Article  PubMed  Google Scholar 

  • Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110.

    Article  Google Scholar 

  • Inc, Survey Monkey. (2014). SuveryMonkey [computer software]. Palo Alto, CA: SurveyMonkey Inc.

    Google Scholar 

  • Kazdin, A. E. (2009). Understanding how and why psychotherapy leads to change. Psychotherapy Research, 19(4), 418–428.

  • Kochenderfer, B. J., & Ladd, G. W. (1996). Peer victimization: Cause or consequence of school maladjustment? Child Development, 67, 1305–1317.

    Article  PubMed  Google Scholar 

  • Konstantopoulos, S. (2011). How consistent are class size effects? Evaluation Review, 35, 71–92.

    Article  PubMed  Google Scholar 

  • Merrell, K. W., Gueldner, B. A., Ross, S. W., & Isava, D. M. (2008). How effective are school bullying intervention programs? A meta-analysis of intervention research. School Psychology Quarterly, 23, 26–42.

    Article  Google Scholar 

  • Musher-Eizenman, D. R., Boxer, P., Danner, S., Dubow, E. F., Goldstein, S. E., & Heretick, D. M. (2004). Social-cognitive mediators of the relation of environmental and emotion regulation factors to children’s aggression. Aggressive Behavior, 30, 389–408.

    Article  Google Scholar 

  • Olweus, D. (2005). A useful evaluation design, and effects of the Olweus bullying prevention program. Psychology, Crime & Law, 11, 389–402.

    Article  Google Scholar 

  • Paunonen, S. V. (1984). Optimizing the validity of personality assessments: The importance of aggregation and item content. Journal of Research in Personality, 18, 411–431.

    Article  Google Scholar 

  • Pigott, T. D. (2012). Advances in meta-analysis. New York: Springer.

    Book  Google Scholar 

  • Pituch, K. A., Whittaker, T. A., & Stapleton, L. M. (2005). A comparison of methods to test for mediation in multisite experiments. Multivariate Behavioral Research, 40(1), 1–23.

    Article  Google Scholar 

  • Polanin, J. R., Espelage, D. L., & Pigott, T. D. (2012). A meta-analysis of school-based bullying prevention programs’ effects on bystander intervention behavior. School Psychology Review, 41, 47–65.

    Google Scholar 

  • Poteat, V. P., & Espelage, D. L. (2005). Exploring the relation between bullying and homophobic verbal content: The homophobic content agent target (HCAT) scale. Violence and Victims, 20, 513–528.

    Article  PubMed  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Rigby, K. (2001). Health consequences of bullying and its prevention in schools. In J. Juvonen & S. Graham (Eds.), Peer harassment in school: The plight of the vulnerable and victimized (pp. 310–331). New York: Guilford Press.

    Google Scholar 

  • Rigby, K., & Slee, P. T. (1993). Dimensions of interpersonal relation among Australian children and implications for psychological well-being. The Journal of Social Psychology, 133, 33–42.

    Article  PubMed  Google Scholar 

  • Rivers, I., Poteat, V. P., Noret, N., & Ashurst, N. (2009). Observing bullying at school: The mental health implications of witness status. School Psychology Quarterly, 24, 211–223.

    Article  Google Scholar 

  • Ross, S. W., Horner, R. H., & Higbee, T. (2009). Bully prevention in positive behavior support. Journal of Applied Behavior Analysis, 42, 747–759.

    Article  PubMed Central  PubMed  Google Scholar 

  • Smith, P. K. (1997). Bullying in schools: The UK experience and the Sheffield anti-bullying project. The Irish Journal of Psychology, 18, 191–201.

    Article  Google Scholar 

  • Srabstein, J. C., & Leventhal, B. L. (2010). Prevention of bullying-related morbidity and mortality: A call for public health policies. Bulletin of the World Health Organization, 77, 403–404.

    Article  Google Scholar 

  • Sweeting, H., Young, R., West, P., & Der, G. (2006). Peer victimization and depression in early–mid adolescence: A longitudinal study. British Journal of Educational Psychology, 76, 577–594.

    Article  PubMed  Google Scholar 

  • Ttofi, M. M., & Farrington, D. P. (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56.

    Article  Google Scholar 

  • Ttofi, M. M., Farrington, D. P., & Lösel, F. (2012). School bullying as a predictor of violence later in life: A systematic review and meta-analysis of prospective longitudinal studies. Aggression and Violent Behavior, 17, 405–418.

    Article  Google Scholar 

  • Unlu, F., Bozzi, L., Layzer, C., Smith, A., Price, C., & Hurtig, R. (2013, March). Linking implementation fidelity to impacts in an RCT. Paper presented at the annual meeting of Society for Research on Educational Effectiveness, Washington, DC. Abstract retrieved from https://www.sree.org/conferences/2013s/program/downloads/abstracts/826.pdf.

  • U.S. Department of Education. (2011). Analysis of state bullying laws and policies. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service.

    Google Scholar 

  • vanGeel, M., Vedder, P., & Tanilon, J. (2014). Relationship between peer victimization, cyberbullying, and suicide in children and adolescents: A meta-analysis. JAMA pediatrics, Advanced Online Publication. doi:10.1001/jamapediatrics.2013.4143.

  • Warren, S. F., Fey, M. E., & Yoder, P. J. (2007). Differential treatment intensity research: A missing link to creating optimally effective communication interventions. Mental Retardation and Developmental Disabilities Research Reviews, 13(1), 70–77.

    Article  PubMed  Google Scholar 

  • Whitaker, D. J., Rosenbluth, B., Valle, L. A., & Sanchez, E. (2004). Expect Respect: A school-based intervention to promote awareness and effective responses to bullying and sexual harassment. In D. L. Espelage & S. M. Swearer (Eds.), Bullying in American schools: A social-ecological perspective on prevention and intervention (pp. 327–350). Mahwah, NJ: Taylor & Francis.

    Google Scholar 

  • Wickham, H. (2009). ggplot2: Elegant graphics for data analysis. New York: Springer.

    Book  Google Scholar 

Download references

Acknowledgments

Research for the current study was supported by the Centers for Disease Control & Prevention (#1U01/CE001677) to Dorothy Espelage (PI) at the University of Illinois at Urbana-Champaign. Opinions expressed herein do not necessarily reflect those of the Centers for Disease Control & Prevention, or related offices within.

Author information

Authors and Affiliations

  1. Peabody Research Institute, Vanderbilt University, 230 Appleton Place, Nashville, TN, 37203, USA

    Joshua R. Polanin

  2. Department of Educational Psychology, University of Illinois at Urbana-Champaign, Urbana-Champaign, IL, USA

    Dorothy L. Espelage

Authors
  1. Joshua R. Polanin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dorothy L. Espelage
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joshua R. Polanin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Polanin, J.R., Espelage, D.L. Using a Meta-analytic Technique to Assess the Relationship between Treatment Intensity and Program Effects in a Cluster-Randomized Trial. J Behav Educ 24, 133–151 (2015). https://doi.org/10.1007/s10864-014-9205-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10864-014-9205-9

Keywords

Search

Navigation