Abstract
Recent studies suggest that the reported effect sizes of prevention and intervention trials in criminology are considerably larger when program developers are involved in a study than when trials are conducted by independent researchers. This paper examines the possibility that these differences may be due to systematic bias related to conflict of interest. A review of the evidence shows that the possibility of a substantial problem cannot be currently rejected. Based on a theoretical model about how conflict of interest may influence research findings, the paper proposes several strategies to examine empirically the extent of systematic bias related to conflict of interest. It also suggests that, in addition to improved standards for conducting and publishing future experimental studies, more research is needed on the extent of systematic bias in the existing body of literature.
Similar content being viewed by others
References
Al-Marzouki, S., Evans, S., Marshall, T., & Roberts, I. (2005a). Are these data real? Statistical methods for the detection of data fabrication in clinical trials. BMJ, 331, 267–270.
Al-Marzouki, S., Roberts, I., Marshall, T., & Evans, S. (2005b). The effect of scientific misconduct on the results of clinical trials: a Delphi survey. Contemporary Clinical Trials, 26(3), 331–337.
Altman, D. G. (1996). Better reporting of randomised controlled trials: the consort statement. BMJ, 313(7057), 570–571.
Babcock, L., Loewenstein, G., Issacharoff, S., & Camerer, C. (1995). Biased judgments of fairness in bargaining. The American Economic Review, 85(5), 1337–1343.
Bauer, N. S., Lozano, P., & Rivara, F. P. (2007). The Effectiveness of the Olweus bullying prevention program in public middle schools: a controlled trial. The Journal of Adolescent Health: Official Publication of the Society for Adolescent Medicine, 40(3), 266–274.
Beelmann, A., & Lösel, F. (2006). Child social skills training in developmental crime prevention: effects on antisocial behavior and social competence. Psycothema, 18(3), 603–610.
Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. JAMA, 273(14), 1106–1112.
Campbell, M. K., Elbourne, D. R., & Altman, D. G. (2004). Consort statement: extension to cluster randomised trials. BMJ, 328(7441), 702–708.
Cho, H., Hallfors, D. D., & Sánchez, V. (2005). Evaluation of a high school peer group intervention for at-risk youth. Journal of Abnormal Child Psychology, 33(3), 363–374.
Cornish, D. B., & Clarke, R. V. G. (Eds.). (1986). The reasoning criminal: Rational choice perspectives on offending. New York: Springer.
Dana, J., & Loewenstein, G. (2003). A social science perspective on gifts to physicians from industry. JAMA, 290(2), 252–255.
de Graaf, I., Speetjens, P., Smit, F., de Wolff, M., & Tavecchio, L. (2008). Effectiveness of the Triple P positive parenting program on behavioral problems in children: a meta-analysis. Behav Modif, 32(5), 714–735.
Diekmann, A. (2007). Not the first digit! Using Benford’s Law to detect fraudulent scientific data. Journal of Applied Statistics, 34(3), 321–329.
DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002a). Effectiveness of mentoring programs for youth: a meta-analytic review. American Journal of Community Psychology, 30, 157–197.
DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002b). Effectiveness of mentoring programs for youth: a meta-analytic review. American Journal of Community Psychology, 30(2), 157–197.
Eggert, L. L., Seyi, C. D., & Nicholas, L. J. (1990). Effects of a school-based prevention program for potential high school dropouts and drug abusers. Substance Use & Misuse, 25(7), 773–801.
Eggert, L. L., Thompson, E. A., Herting, J. R., Nicholas, L. J., & Dickers, B. G. (1994). Preventing adolescent drug abuse and high school dropout through an intensive social network development program. American Journal of Health Promotion, 8(2), 202–215.
Eisner, M., & Ribeaud, D. (2008). Markt, Macht und Wissenschaft; Kritische Überlegungen zur Deutschen Präventionsforschung. In E. Marks & W. Steffen (Eds.), Starke Jugend - Starke Zukunft (Ausgewählte Beiträge des 12. Deutschen Präventionstages, 18. und 19. Juni 2007) (pp. 173–191). Mönchengladbach: Forum Verlag Godesberg.
Eisner, M., Ribeaud, D., Jünger, R., & Meidert, U. (2007). Frühprävention von Gewalt und Aggression; Ergebnisse des Zürcher Interventions- und Präventionsprojektes an Schulen. Zürich: Rüegger.
Ellickson, P. L. (1998). Preventing adolescent substance abuse: lessons from the project alert program. In J. Crane (Ed.), Social programs that work (pp. 201–257). New York: Russell Sage.
Ellickson, P. L., & Bell, R. M. (1990). Drug prevention in junior high: a multi-site longitudinal test. Science, 247, 1299–1305.
Ellickson, P. L., McCaffrey, D. F., Ghosh-Dastidar, B., & Longshore, D. L. (2003). New inroads in preventing adolescent drug use: results from a large-scale trial of project ALERT in middle schools. American Journal of Public Health, 93(11), 1830–1836.
Elliott, D. S., & Mihalic, S. F. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5(1), 47–53.
Etzkowitz, H. (2003). Research groups as ‘Quasi-Firms’: the invention of the entrepreneurial university. Research Policy, 32(1), 109–121.
Farrington, D. P. (2003). Methodological quality standards for evaluation research. Annals of the American Academy of Political and Social Sciences, 587, 49–68.
Farrington, D. P., & Welsh, B. (2003). Family-based prevention of offending: a meta-analysis. Australian and New Zealand Journal of Criminology, 36(2), 127–151.
Felson, M. (1994). Crime and everyday life; insight and implications for society. Thousand Oaks: Pine Forge Press.
Friedman, L. S., & Richter, E. D. (2004). Relationship between conflicts of interest and research results. Journal of General Internal Medicine, 19(1), 51–56.
Gandhi, A. G., Murphy-Graham, E., Petrosino, A., Chrismer, S. S., & Weiss, C. H. (2007). The devil is in the details: examining the evidence for “proven” school-based drug abuse prevention programs. Evaluation Review, 31(1), 43–74.
Gilbert, N. (1997). Advocacy research and social policy. Crime and Justice: A Review of Research, 22, 101–148.
Gorman, D. M. (2003). Alcohol & drug abuse: the best of practices, the worst of practices: the making of science-based primary prevention programs. Psychiatric Services, 54(8), 1087–1089.
Gorman, D. M. (2005a). Does measurement dependence explain the effects of the life skills training program on smoking outcomes? Preventive Medicine, 40(4), 479–487.
Gorman, D. M. (2005b). Drug and violence prevention: rediscovering the critical rational dimension of evaluation research. Journal of Experimental Criminology, 1(1), 39–62.
Gorman, D. M. (2006). Conflicts of interest in the evaluation and dissemination of drug use prevention programs. In J. Kleinig, & S. Einstein (Eds.), Intervening in drug use: Ethical challenges (pp. 171–187). Huntsville: Office of International Criminal Justice.
Gorman, D. M., & Conde, E. (2007). Conflict of interest in the evaluation and dissemination of “Model” school-based drug and violence prevention programs. Evaluation and Program Planning, 30(4), 422–429.
Heinrichs, N., Hahlweg, K., Bertram, H., Kuschel, A., Naumann, S., & Harstick, S. (2006). Die langfristige Wirksamkeit eines Elterntrainings zur universellen Prävention kindlicher Verhaltensstörungen: Ergebnisse aus Sicht der Mütter und Väter. Zeitschrift für klinische Psychologie und Psychotherapie, 35(2).
Henggeler, S. W., Cinningham, P. B., Pickrel, S. G., Schoenwald, S. K., & Brondino, M. J. (1996). Multisystemic therapy: an effective violence prevention approach for serious juvenile offenders. Journal of Adolescence, 19(1), 47–61.
Hewitt, C. E., Mitchell, N., & Torgerson, D. J. (2008). Listen to the data when results are not significant. BMJ, 336(7634), 23–25.
Holyoak, K. J., & Simon, D. (1999). Bidirectional reasoning in decision making by constraint satisfaction. Journal of Experimental Psychology; General, 128(1), 3–31.
Kaptchuk, T. J. (2003). Effect of interpretive bias on research evidence. BMJ, 326(7404), 1453–1455.
Lipsey, M. W. (1995). What do we learn from 400 research studies on the effectiveness of treatment with juvenile delinquents? In J. McGuire (Ed.), What works? Reducing reoffending (pp. 63–78). New York: Wiley.
Lipsey, M. W., & Cullen, F. T. (2007). The effectiveness of correctional rehabilitation: a review of systematic reviews. Annual Review of Law and Social Science, 3, 297–320.
Lipsey, M. W., Petrie, C., Weisburd, D., & Gottfredson, D. (2006). Improving evaluation of anti-crime programs: summary of a national research council report. Journal of Experimental Criminology, 2(3), 271–307.
Littell, J. (2005). Lessons from a systematic review of effects of multisystemic therapy. Children and Youth Services Review, 27(4), 445–463.
Littell, J., Popa, M., & Forsythe, B. (2005). Multisystemic Therapy for Social, Emotional, and Behavioral Problems in Youth Aged 10–17 (Report for the Campbell Collaboration) (Electronic Version). Accessed 14 January 2008 from http://www.sfi.dk/graphics/Campbell/Dokumenter/MST_Review/MULTISYSTEMIC%20TERAPHY%20-%20REVIEW.pdf.
Lock, S., Wells, F., & Farthing, M. (Eds.). (2001). Fraud and misconduct in medical research. London: BMJ Publishing Group.
Lösel, F., & Beelmann, A. (2003). Effects of child skills training in preventing antisocial behavior: a systematic review of randomized evaluations. The ANNALS of the American Academy of Political and Social Science, 587(1), 84–109.
Lösel, F., & Koferl, P. (1989). Evaluation research on correctional treatment in West Germany: A meta-analysis. In H. Wegener, F. Lösel, & J. Haisch (Eds.), Criminal behavior and the justice system: Psychological perspectives. New York: Springer.
Lösel, F., & Beelmann, A. (2003). Effects of child skills training in preventing antisocial behavior: a systematic review of randomized evaluations. The ANNALS of the American Academy of Political and Social Science, 587(1), 84–109.
Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., et al. (1999). The researcher’s own therapy allegiances: a “Wild Card” in comparisons of treatment efficacy. Clinical Psychology: Science and Practice, 6(1), 95–106.
Lundahl, B., Risser, H. J., & Lovejoy, M. C. (2006). A meta-analysis of parent training: moderators and follow-up effects. Clinical Psychology Review, 26(1), 86–104.
MacCoun, R. J. (1998). Biases in the interpretation and use of research statistics. Annual Review of Psychology, 49, 259–287.
MacCoun, R. J. (2005). Conflicts of interest in public policy research. In D. Moore, D. M. Cain, G. Loewenstein, & M. H. Bazerman (Eds.), Conflicts of interest; challenges and solutions in business, law, medicine, and public policy (pp. 233–262). Cambridge: Cambridge University Press.
Martinson, B. C., Anderson, M. S., & de Vries, R. (2005). Scientists behaving badly. Nature, 435(9 June 2005), 737–738.
Messick, D. M., & Sentis, K. P. (1979). Fairness and preference. Journal of Experimental Social Psychology, 15(4), 418–434.
Moore, D., Tetlock, P. E., Tanlu, L., & Bazerman, M. H. (2006). Conflicts of interest and the case of auditor independence: moral seduction and strategic issue cycling. Academy of Management Review, 31(1), 10–29.
Nowak, C., & Heinrichs, N. (2008). A comprehensive meta-analysis of Triple P-positive parenting program using hierarchical linear modeling: effectiveness and moderating variables. Clinical Child and Family Psychology Review, 11(3), 114–144.
Ogden, T., & Halliday-Boykins, C. A. (2004). Multisystemic treatment of antisocial adolescents in Norway: replication of clinical outcomes outside of the us. Child and Adolescent Mental Health, 9, 77–83.
Ogden, T., & Hagen, K. A. (2006). Multisystemic treatment of serious behaviour problems in youth: sustainability of effectiveness two years after intake. Child and Adolescent Mental Health, 11(3), 142–149.
Okike, K., Kocher, M. S., Mehlman, C. T., & Bhandari, M. (2007). Conflict of interest in orthopaedic research. an association between findings and funding in scientific presentations. J Bone Joint Surg Am, 89(3), 608–613.
Olweus, D. (1994). Bullying at school: basic facts and effects of a school based intervention program. Journal of Child Psychology and Psychiatry, 35(7), 1171–1190.
Perlis, R. H., Perlis, C. S., Wu, Y., Hwang, C., Joseph, M., & Nierenberg, A. A. (2005). Industry sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. American Journal of Psychiatry, 162(10), 1957–1960.
Petrosino, A., & Soydan, H. (2005). The impact of program developers as evaluators on criminal recidivism: results from meta-analyses of experimental and quasi-experimental research. Journal of Experimental Criminology, 1(4), 435–450.
Ranstam, J., Buyse, M., George, S. L., Evans, S., Geller, N. L., Scherrer, B., et al. (2000). Fraud in medical research: an international survey of biostatisticians. Controlled Clinical Trials, 21(5), 415–427.
Resnik, D. B. (2000). Financial interests and research bias. Perspectives on Science, 8(3), 255–285.
Russo, J. E., Carlson, K. A., Meloy, M., & Yong, K. (2007). The goal of consistency as a cause of information distortion. Johnson School Research Paper Series No. 04-07.
Sanchez, V., Steckler, A., Nitirat, P., Hallfors, D., Cho, H., & Brodish, P. (2007). Fidelity of implementation in a treatment effectiveness trial of reconnecting youth. Health Education Research, 22(1), 95–107.
Sanders, M. R. (1992). Every parent: A positive guide to children’s behavior. Sydney: Addison-Wesley.
Sanders, M. R. (1999). Triple P-positive parenting program: towards an empirically validated multilevel parenting and family support strategy for the prevention of behaviour and emotional problems in children. Clinical Child and Family Psychology Review, 2(2), 71–89.
Shadish, W. R., Cook, T. D., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.
Sherman, L. W. (2006). “to develop and test:” the inventive difference between evaluation and experimentation. Journal of Experimental Criminology, 2(3), 393–406.
Sherman, L. W., Farrington, D. P., Welsh, B. C., & MacKenzie, D. L. (Eds.). (2002). Evidence-based crime prevention. London: Routledge.
St Pierre, T. L., Osgood, D. W., Mincemoyer, C. C., Kaltreider, D. L., & Kauh, T. J. (2006). Results of an independent evaluation of project alert delivered in schools by cooperative extension. Prevention Science, 6(4), 305–317.
Sundell, K., Hansson, K., Löfholm, C. A., Olsson, T., Gustle, L. H., & Kadasjo, C. (2008). The transportability of multisystemic therapy to Sweden: short-term results from a randomized trial of conduct disordered youth. Journal of Family Psychology, 22(4), 550–560.
Wilson, S. J., & Lipsey, M. W. (2007). School-based interventions for aggressive and disruptive behavior: update of a meta-analysis. American Journal of Preventive Medicine, 33(2, Supplement 1), S130–S143.
Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003a). The effects of school-based intervention programs on aggressive behavior: a meta-analysis. Journal of Consulting and Clinical Psychology, 71(1), 136–149.
Wilson, S. J., Lipsey, M. W., & Soydan, H. (2003b). Are mainstream programs for juvenile delinquency less effective with minority youth than majority youth? a meta-analysis of outcomes research. Research on Social Work Practice, 13(1), 2–26.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Eisner, M. No effects in independent prevention trials: can we reject the cynical view?. J Exp Criminol 5, 163–183 (2009). https://doi.org/10.1007/s11292-009-9071-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11292-009-9071-y