Skip to main content
SpringerLink
Log in

Reflections on the Reproducibility Project in Psychology and the Insights It Offers for Clinical Psychology

  • Chapter
  • First Online:
Avoiding Questionable Research Practices in Applied Psychology

Abstract

The overarching goal of this chapter is to discuss the Reproducibility Project in Psychology (RP:P), the resulting credibility movement, and its implications for scientific practices in clinical psychology and beyond. We start by introducing the RP:P and then describe how the RP:P led to other replication projects and the development of improved research practices. We conclude with a discussion of the replicability in clinical psychology, with attention to the challenges and opportunities of replicating clinical psychological findings in different places, with different people, and at different times.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Anderson, S. F., & Maxwell, S. E. (2016). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods, 21(1), 1–12. https://doi.org/10.1037/met0000051

    Article  PubMed  Google Scholar 

  • Baker, M., & Dolgin, E. (2017). Cancer reproducibility project releases first results. Nature News, 541(7637), 269.

    Article  Google Scholar 

  • Barba, L. A. (2018). Terminologies for reproducible research. arXiv preprint arXiv:1802.03311.

    Google Scholar 

  • Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531–533.

    Article  Google Scholar 

  • Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407–425. https://doi.org/10.1037/a0021524

    Article  PubMed  Google Scholar 

  • Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376.

    Article  Google Scholar 

  • Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., … Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436.

    Article  Google Scholar 

  • Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., … Wu, H. (2018). Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644.

    Article  Google Scholar 

  • Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7–18. https://doi.org/10.1037/0022-006X.66.1.7

    Article  PubMed  Google Scholar 

  • Crocker, J. (2011). The road to fraud starts with a single step. Nature, 479, 151.

    Article  Google Scholar 

  • Cuijpers, P., Cristea, I. A., Karyotaki, E., Reijnders, M., & Huibers, M. J. (2016). How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence. World Psychiatry, 15(3), 245–258.

    Article  Google Scholar 

  • Cybulski, L., Mayo-Wilson, E., & Grant, S. (2016). Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals. Journal of Consulting and Clinical Psychology, 84(9), 753–767. https://doi.org/10.1037/ccp0000115

    Article  PubMed  Google Scholar 

  • Duyx, B., Urlings, M. J., Swaen, G. M., Bouter, L. M., & Zeegers, M. P. (2017). Scientific citations favor positive results: A systematic review and meta-analysis. Journal of Clinical Epidemiology, 88, 92–101.

    Article  Google Scholar 

  • Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., ... & Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. https://doi.org/10.1016/j.jesp.2015.10.012

  • Editorial. (2018). What next for registered reports. Nature Human Behaviour, 2, 789–790. https://doi.org/10.1038/s41562-018-0477-2

    Article  Google Scholar 

  • Frank, M. C., Bergelson, E., Bergmann, C., Cristia, A., Floccia, C., Gervain, J., … Yurovsky, D. (2017). A collaborative approach to infant research: Promoting reproducibility, best practices, and theory-building. Infancy, 22(4), 421–435.

    Article  Google Scholar 

  • Grant, S. P., Mayo-Wilson, E., Melendez-Torres, G. J., & Montgomery, P. (2013). Reporting quality of social and psychological intervention trials: A systematic review of reporting guidelines and trial publications. PLoS One, 8(5), e65442. https://doi.org/10.1371/journal.pone.0065442

    Article  PubMed  PubMed Central  Google Scholar 

  • García-Lizana, F., & Muñoz-Mayorga, I. (2010). What about telepsychiatry? A systematic review. The Primary Care Companion for CNS Disorders, 12(2), 26919. https://doi.org/10.4088/PCC.09m00831whi

  • Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

  • Inbar, Y. (2016). Association between contextual dependence and replicability in psychology may be spurious. Proceedings of the National Academy of Sciences of the United States of America, 113(34), E4933–E4934. https://doi.org/10.1073/pnas.1608676113

    Article  PubMed  PubMed Central  Google Scholar 

  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

  • Kazdin, A. E., & Blase, S. L. (2011). Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspectives on Psychological Science, 6(1), 21–37.

    Article  Google Scholar 

  • Kendall, P. C. (1998). Empirically supported psychological therapies. Journal of Consulting and Clinical Psychology, 66(1), 3–6. https://doi.org/10.1037/0022-006X.66.1.3

    Article  PubMed  Google Scholar 

  • Kim, D. M., Wampold, B. E., & Bolt, D. M. (2006). Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychotherapy Research, 16(02), 161–172. https://doi.org/10.1080/10503300500264911

    Article  Google Scholar 

  • Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., ... & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456. https://doi.org/10.1371/journal.pbio.1002456

  • Kleinman, A. (2004). Culture and depression. New England Journal of Medicine, 351(10), 951–953. https://doi.org/10.1056/NEJMp048078

    Article  PubMed  Google Scholar 

  • Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., . . . Nosek, B. A. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152. https://doi.org/10.1027/1864-9335/a000178

  • Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., … Sowden, W. (2018). Many labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490.

    Article  Google Scholar 

  • Kravitz, D. J., Mitroff, S. R., & Bauer, P. J. (2020). Practicing good laboratory hygiene, even in a pandemic. Psychological Science, 31, 483.

    Article  Google Scholar 

  • Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537–542.

    Article  Google Scholar 

  • McKay, K. M., Imel, Z. E., & Wampold, B. E. (2006). Psychiatrist effects in the psychopharmacological treatment of depression. Journal of Affective Disorders, 92(2–3), 287–290. https://doi.org/10.1016/j.jad.2006.01.020

    Article  PubMed  Google Scholar 

  • Morrison, S. J. (2014). Reproducibility project: Cancer biology: Time to do something about reproducibility. eLife, 3, e03981.

    Article  Google Scholar 

  • Mobley, A., Linder, S. K., Braeuer, R., Ellis, L. M., & Zwelling, L. (2013). A survey on data reproducibility in cancer research provides insights into our limited ability to translate findings from the laboratory to the clinic. PLoS One, 8(5), e63221. https://doi.org/10.1371/journal.pone.0063221

  • Mullinix, K., Leeper, T., Druckman, J., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(2), 109–138. https://doi.org/10.1017/XPS.2015.19

    Article  Google Scholar 

  • Munro, K. J., & Prendergast, G. (2019). Encouraging pre-registration of research studies. International Journal of Audiology, 58(3), 123–124. https://doi.org/10.1080/14992027.2019.1574405

    Article  PubMed  Google Scholar 

  • Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A. (2019). Open science practices in clinical psychology journals: An audit study. Journal of Abnormal Psychology, 128(6), 510–516. https://doi.org/10.1037/abn0000414

    Article  PubMed  Google Scholar 

  • Nosek, B. A., & Errington, T. M. (2017). Reproducibility in cancer biology: Making sense of replications. eLife, 6, e23383.

    Article  Google Scholar 

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 6251.

    Article  Google Scholar 

  • Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–789.

    Article  Google Scholar 

  • Reardon, K. W., Smack, A. J., Herzhoff, K., & Tackett, J. L. (2019). An N-pact factor for clinical psychological research. Journal of Abnormal Psychology, 128(6), 493–499.

    Article  Google Scholar 

  • Rees, C. S., & Maclaine, E. (2015). A systematic review of videoconference‐delivered psychological treatment for anxiety disorders. Australian Psychologist, 50(4), 259–264. https://doi.org/10.1111/ap.12122

    Article  Google Scholar 

  • Ryder, A. G., Yang, J., Zhu, X., Yao, S., Yi, J., Heine, S. J., & Bagby, R. M. (2008). The cultural shaping of depression: Somatic symptoms in China, psychological symptoms in North America? Journal of Abnormal Psychology, 117(2), 300–313.

    Article  Google Scholar 

  • Sakaluk, J. K., Williams, A. J., Kilshaw, R. E., & Rhyner, K. T. (2019). Evaluating the evidential value of empirically supported psychological treatments (ESTs): A meta-scientific review. Journal of Abnormal Psychology, 128(6), 500–509.

    Article  Google Scholar 

  • Schimmack, U. (2016). The Replicability-Index: Quantifying statistical research integrity. https://wordpress.com/post/replication-index.wordpress.com/920.

  • Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014). An introduction to registered replication reports at perspectives on psychological science. Perspectives on Psychological Science, 9(5), 552–555.

    Article  Google Scholar 

  • Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341

  • Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242

    Article  Google Scholar 

  • Strack, F., Martin, L. L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: A nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology, 54(5), 768–777. https://doi.org/10.1037/0022-3514.54.5.768

  • Tackett, J. L., Brandes, C. M., King, K. M., & Markon, K. E. (2019). Psychology’s replication crisis and clinical psychological science. Annual Review of Clinical Psychology, 15, 579–604.

    Article  Google Scholar 

  • Tolin, D. F., McKay, D., Forman, E. M., Klonsky, E. D., & Thombs, B. D. (2015). Empirically supported treatment: Recommendations for a new model. Clinical Psychology: Science and Practice, 22(4), 317–338.

    Google Scholar 

  • Turgoose, D., Ashwick, R., & Murphy, D. (2018). Systematic review of lessons learned from delivering tele-therapy to veterans with post-traumatic stress disorder. Journal of Telemedicine and Telecare, 24(9), 575–585. https://doi.org/10.1177/1357633X17730443

    Article  PubMed  Google Scholar 

  • Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences, 113(23), 6454–6459.

    Article  Google Scholar 

  • Wagenmakers, E. J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Jr., … Zwaan, R. A. (2016). Registered replication report: Strack, martin, & stepper (1988). Perspectives on Psychological Science, 11(6), 917–928.

    Article  Google Scholar 

  • Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432. https://doi.org/10.1037/a0022790

    Article  PubMed  Google Scholar 

  • Yen, S., Robins, C. J., & Lin, N. (2000). A cross-cultural comparison of depressive symptom manifestation: China and the United States. Journal of Consulting and Clinical Psychology, 68(6), 993–999.

    Article  Google Scholar 

  • Zhou, X., Snoswell, C. L., Harding, L. E., Bambling, M., Edirippulige, S., Bai, X., & Smith, A. C. (2020). The role of telehealth in reducing the mental health burden from COVID-19. Telemedicine and e-Health, 26(4), 377–379. https://doi.org/10.1089/tmj.2020.0068

    Article  PubMed  Google Scholar 

Download references

Author information

Author notes

  1. Elizabeth W. Chan and Johnny Wong contributed equally with all other contributors.

Authors and Affiliations

  1. Department of Psychology, University of Toronto, Toronto, ON, Canada

    Elizabeth W. Chan, Johnny Wong & Felix Cheung

  2. Department of Psychology, University of Hong Kong, Hong Kong, SAR, China

    Christian S. Chan

Authors
  1. Elizabeth W. Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Johnny Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian S. Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Felix Cheung
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felix Cheung .

Editor information

Editors and Affiliations

  1. Department of Psychology, University of Nevada, Reno, NV, USA

    William O'Donohue

  2. Department of Psychology, University of Hawaii at Manoa, Honolulu, HI, USA

    Akihiko Masuda

  3. Department of Psychology, Emory University, Atlanta, GA, USA

    Scott Lilienfeld

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Chan, E.W., Wong, J., Chan, C.S., Cheung, F. (2022). Reflections on the Reproducibility Project in Psychology and the Insights It Offers for Clinical Psychology. In: O'Donohue, W., Masuda, A., Lilienfeld, S. (eds) Avoiding Questionable Research Practices in Applied Psychology. Springer, Cham. https://doi.org/10.1007/978-3-031-04968-2_18

Download citation

Publish with us

Policies and ethics

Search

Navigation