Computational reproducibility: Examining verification errors and frictions

Data archives, libraries, and publishers are extending their services to support computational reproducibility of results reported in manuscripts. Computational reproducibility is having enough information about the data, code, and compute environment to re-run and reproduce analyses. While archives and publishers are adopting policies and audit workflows to verify the results in a manuscript, many opponents express concerns about the additional effort, time, and specialized expertise being placed on authors. What are the challenges that researchers face in complying with computational reproducibility and transparency policies? The American Journal of Political Science (AJPS) adopted a verification policy requiring authors to make available all research materials needed to reproduce the results in a manuscript. As an independent third party, the Odum Institute performs the computational reproducibility check on the submitted materials with quantitative analyses. The Odum Institute has collected detailed data about the verification process for all reviewed manuscripts including errors encountered, number of resubmissions, and submission package characteristics, providing a novel dataset to explore challenges that authors face. This paper will report results from qualitative coding and analysis of verification reports for 105 AJPS manuscripts. Twenty-three errors that authors commonly make in replication packages were identified, representing 7 categories: code, data, documentation, file, methods, results, and technology. Together, these sets of errors provide a more holistic picture of challenges faced by authors in making their research reproducible. These challenges range in terms of causes, scope, significance, persistence, and repetition. These findings will be discussed in relation to questions of services, tools, and trainings. How can archives and libraries help researchers comply with reproducibility policies? Which categories of errors should we prioritize? These questions will lead to a discussion on translating reproducibility policies and recommendations into actionable, local practices and services.