Preregistration of Modeling Exercises May Not Be Useful

Abstract

This is a commentary on Lee et al.’s (2019) article encouraging preregistration of model development, fitting, and evaluation. While we are in general agreement with Lee et al.’s characterization of the modeling process, we disagree on whether preregistration of this process will move the scientific enterprise forward. We emphasize the subjective and exploratory nature of model development, and point out that “under-modeling” of data (relying on black-box approaches applied to data without data exploration) is as big a problem as “over-modeling” (fitting noise, resulting in models that generalize poorly). We also note the potential long-run negative impact of preregistration on future generations of cognitive scientists. It is our opinion that preregistration of model development will lead to less, and to less creative, exploratory analysis (i.e., to more under-modeling), and that Lee at al.’s primary goals can be achieved by requiring publication of raw data and code. We conclude our commentary with suggestions on how to move forward.