Making ERP research more transparent: Guidelines for preregistration

A combination of confirmation bias, hindsight bias, and pressure to publish may prompt the (unconscious) exploration of various methodological options and reporting only the ones that lead to a (statistically) significant outcome. This undisclosed analytic flexibility is particularly relevant in EEG research, where a myriad of preprocessing and analysis pipelines can be used to extract information from complex multidimensional data. One solution to limit confirmation and hindsight bias by disclosing analytic choices is preregistration: researchers write a time-stamped, publicly accessible research plan with hypotheses, data collection plan, and the intended preprocessing and statistical analyses before the start of a research project. In this manuscript, we present an overview of the problems associated with undisclosed analytic flexibility, discuss why and how EEG researchers would benefit from adopting preregistration, provide guidelines and examples on how to preregister data preprocessing and analysis steps in typical ERP studies, and conclude by discussing possibilities and limitations of this open science practice.

These cognitive biases find fertile ground in complex and multifaceted intellectual endeavors like empirical sciences. Data collected in an experimental or observational study are rarely interpreted in their raw form. Instead, researchers typically apply a series of transformations to deal with outliers and missing data (Enders, 2010;Hawkins, 1980), combine or discretize variables into composite indices, change the unit of measurement, and so on. In other words, "data are to a certain extent actively constructed" (Steegen et al., 2016, p. 702). Moreover, there are countless statistical techniques that can be chosen to analyze the preprocessed data, including classical null hypothesis tests (Field et al., 2012;Judd et al., 2017a) and their robust counterparts (Wilcox, 2016), Bayesian parameter estimation (Kruschke, 2014;McElreath, 2018), and more. This myriad of choices that researchers have to make during the research process is referred to as analytic flexibility. 2 Often, the rationale behind the selection of data preprocessing and analysis pipelinese.g., the selection of cut-off values when identifying outliers or the choice of a particular statistical technique 3 -is not properly described. This is not necessarily due to blind following of "statistical rituals" (Gigerenzer, 2004), because there may very well be multiple reasonable processing steps that can be applied to the same dataset (Steegen et al., 2016). Thus, analytic flexibility per se does not necessarily lead to unverifiable or incorrect knowledge (see also Devezer et al., 2020). Instead, problems arise when methodological choices on preprocessing pipelines and statistical analysis are not transparently reported. Below we describe how undisclosed analytic flexibility may influence the interpretation of results in human electrophysiology research.

Undisclosed analytic flexibility in human electrophysiology research
Magneto-and electroencephalographic (M/EEG) signals are complex and multidimensional: space, time, and frequencyassessed via indices such as activity magnitude, connectivity, and network properties (Kida et al., 2016) interact with experimental designs of various complexity, often resulting in a large number of independent and dependent variables. The raw signal recorded by electrodes (and magnetometers) must undergo a series of preprocessing steps that magnify cerebral activity against environmental noise (Cohen, 2014;Hansen et al., 2010;Luck, 2014). Offline modifications of the continuous EEG signal include: (i) rereferencing to the activity of specific electrodes or the average activity of all electrodes on the scalp; (ii) interpolation of noisy channels; (iii) high-, low-, or band-pass filtering; (iv) correcting or rejecting physiological artifacts (e.g., blinks, muscular activity); (v) removal of baseline activity; and (vi) segmentation into epochs around the event(s) of interest (Luck, 2014). Needless to say, there is considerable flexibility at each of these steps: (i) popular reference methods include vertex, linked mastoids or ears, average reference, and Reference Electrode Standardization Technique (for reviews, see Dong et al., 2019;Liu et al., 2015), and their choice is not always obvious with respect to the experimental design or dependent variables of interest; (ii) channel interpolatione.g., nearest neighbor (Shepard, 1968), thin-plate spline (Harder and Desmarais, 1972), spherical spline (Perrin et al., 1989), 3-D spline (Law et al., 1993) is also a potential source of stochastic error (Fletcher et al., 1996), and its choice is often left to the software used for preprocessing; (iii) there are many different filter types available and considerable flexibility in setting the exact parameters for the filter, e.g., the cut-off frequency, transition width, etc.; moreover, common filtering techniques can severely distort the signal (e.g., Kappenman and Luck, 2010), which even led some to propose their exclusion from preprocessing pipelines in specific experimental designs (VanRullen, 2011; but see Rousselet, 2012;Widmann and Schröger, 2012); (iv) there is a large number of artifact detection, correction, and rejection techniques (for a review, see Jiang et al., 2019), each with its own expected user input (e.g., from tweaking a few parameters in a fully automated algorithm to visual inspection of epochs for manual removal); (v) for baseline correction, the selected time window can vary in length and location (i.e., more proximal or distal from the event of interest); also, traditional baseline correction can bias scalp topographies (Urbach and Kutas, 2006), which may lead researchers to favor other techniques, for example including the baseline interval as a predictor in a GLM-based statistical approach (Alday, 2019). Finally, the order in which some of the above mentioned steps are performed may distort the resulting waveforms, e.g., filtering epoched instead of continuous EEG data may create edge artifacts, particularly when using inappropriate filter types or cut-off values (Luck, 2014, pp. 247-248; see also Widmann et al., 2015).
Recent papers directly demonstrated that analytic flexibility may influence the results and interpretation of electrophysiological data. Robbins et al. (2020) applied four preprocessing pipelines (Bigdely-Shamlo et al., 2020a;Chang et al., 2020;Winkler et al., 2011) to a large and heterogeneous EEG dataset containing 7.8 million event-related epochs (Bigdely-Shamlo et al., 2020a). There were differences in the spectral characteristics of the processed signals, attributable to the different artifact correction procedures across preprocessing pipelines. In addition, small parameter deviations in otherwise very similar artifact correction algorithms were shown to distort the signal, especially in low frequency bands. Calculation of event-related epochs was also affected by specific steps in the selected preprocessing pipeline: for example, outlier detection algorithms may be incorporated in some pipelines (e. g., Bigdely-Shamlo et al., 2020b) but not in others.
Another example pertains to the error-related negativity (ERN). This ERP component of negative polarity peaks ~80-150 milliseconds after an erroneous motor response in speeded tasks, is largest at midline frontal and central electrode sites, and originates from the anterior cingulate cortex (Falkenstein et al., 1991;Gehring et al., 1993). A recent paper (Sandre et al., 2020) highlighted cross-study variability in the selection of reference location, baseline correction, and electrode site from which signal amplitudes were measured. The authors systematically compared 72 preprocessing pipelines to examine their effects on the resulting ERN amplitude. Results showed that different preprocessing choices had a remarkable influence on the within-and betweensubject effects typically assessed in ERN researchi.e., post-error slowing and gender differences -, with mastoid reference, distal baseline correction periods (i.e., further away from the time-locked response), single electrode site, and peak-to-peak amplitude measures leading to larger estimated differences between conditions (see also Klawohn et al., 2020;Š oškić et al., 2020).
Thus, EEG researchers routinely deal with a large number of "forking paths", which are seldom constrained for theoretical reasons: they have at their disposal a considerably long list of data transformation steps (each with its own challenges and complexities) which can lead to a different interpretation of the results. Quoting Sandre et al. (2020): "[...] different ways of processing the same data can lead researchers to different conclusions, demonstrating yet again that transparency of all processing decisions is a necessity." (p. 35). We concur: transparently reporting all analytic choices would increase study reproducibility and, more generally, the trustworthiness of the electrophysiological literature.
As mentioned before, researchers (just like other human beings) tend to shape their analytic choices with the (largely implicit) aim to confirm their prior beliefs, and post-hoc justification of said choices is rationalized under the "illusion of objectivity" (Kunda, 1990;Pyszczynski and Greenberg, 1987). EEG researchers are not immune to the pitfalls of confirmation bias and hindsight bias: for example, they may be thinking that all preprocessing and analysis choices were determined a priori while they may have been at least partly based on seeing the data. Sandre et al. (2020) suggest that "a single processing stream should be 2 Readers may be familiar with other terms, such as researcher degrees of freedom (Simmons et al., 2011) or garden of forking paths (Gelman and Loken, 2013), which refer to all choices that researchers make throughout their workflow, including hardware and software selected for data collection and analysis, the type and number of stimuli presented to participants, and much more. These choices can have tangible consequences on study results and interpretation. For instance, the same analysis pipeline on the same dataset can lead to quantifiably different results when run with different software (Bowring et al., 2019; see also Eklund et al., 2016). Typically, larger and more homogeneous samples of both participants and stimuli increase statistical power (Judd et al., 2017b), and ignoring these sources of variability in the applied statistical model has a direct impact on the generalizability of the results from a particular dataset to other (hypothetically similar) scenarios (Yarkoni, 2020). Throughout this paper, we limit our discussion to analytic flexibility during the preprocessing and analysis phases of the research cycle.
3 Although more robust methods are often justified in comparison to traditional methods.
finalized before any analyses are undertaken" (p. 35). In other words, confirmation bias and hindsight bias cannot take place if analytic choices are not only determined before the data are collected but also transparently reported. This practice is called preregistration.

Preregistration
Preregistrations are time-stamped, (eventually) publicly accessible documents with hypotheses, data collection plan, and/or intended preprocessing and statistical analyses, written before the start of a research project. 4 In other words, researchers commit to one among many ways in which the study can be conducted and analyzed. This document is uploaded on a trusted online repositorye.g., Open Science Framework (OSF; https://osf.io/), ClinicalTrials.gov (https://clinicaltr ials.gov/), American Economic Association's registry for randomized controlled trial (AEA RCT Registry; https://www.socialscienceregistry. org/) -, which assigns it a date and time. The protocol is made public immediately or after an embargo period. Date and time of submission ensure that the research plan was devised before starting the study.
The popularity of preregistration has skyrocketed in recent years: for example, the number of publicly available preregistrations uploaded on the OSF went from 38 in 2012 to 36,675 in 2019 . Many journals now explicitly encourage this practice by awarding "preregistration badges" (https://osf.io/tvyxz/; see Kidwell et al., 2016), including Psychological Science and Cortex (for a full list, see https://tinyurl.com/COS-badges).

Advantages of preregistration
The advantages of preregistering research plans are manyfold. First and foremost, preregistration can be seen as an additional tool to effectively achieve as much transparency as possible (see also Navarro, 2020), ultimately increasing verifiability at all stages of the research cycle (Resnik, 2005;Lupia and Elman, 2014; see also Merton, 1942). Researchers are expected to abide by ethical principles that are functional to the epistemic goals of science: advancing human knowledge by describing nature, developing theories and hypotheses that allow the generation of reliable predictions, and eliminating errors and biases (Resnik, 2005). Openness is one of these foundational principles: "Scientists should share data, results, methods, ideas, techniques, and tools. They should allow other scientists to review their work and be open to criticism and new ideas." (Resnik, 2005, p. 52). Other open science practicese.g., sharing study protocols, materials, raw data, and analysis codedirectly follow from this principle. Preregistration additionally offers the possibility to document the rationale behind theoretical and methodological choices, useful not only in quantitative but also qualitative disciplines (Haven and Van Grootel, 2019). In addition, deviations from the original design (e.g., discrepancies between planned and actual sample size, unforeseen moderators, flexible exclusion criteria) can be more easily identified, effectively counteracting selective outcome reporting (Goldacre et al., 2019;John et al., 2012;Simmons et al., 2011). Furthermore, the presence of public, traceable evidence of the original plan exposes (and possibly mitigates) confirmation bias and hindsight bias.
Preregistration allows researchers to specify the rationale and hypotheses of the study while also maintaining flexibility with respect to additional analyses conducted after seeing the data, provided that they are included in a different section of the final manuscript. This precludes presenting any hypotheses generated after observing the data as if they were a priori, or "hypothesizing after the results are known" (HARKing; Kerr, 1998). 5 This practice is particularly difficult to identify in published papers because readers can only access what the authors reported after collecting, analyzing, and interpreting the data, without knowing whether the hypotheses described in the introduction were originally unanticipated (or even considered implausible) until reassessed in light of the collected empirical evidence. This problem is magnified by the fact that, at least in some research fields, theoretical frameworks and hypotheses are often underspecified, which decreases their explanatory power and predictive utility (Meehl, 1967;Muthukrishna and Henrich, 2019;Szollosi and Donkin, 2021;van Rooij and Baggio, 2021).
It has also been argued (Nosek et al., 2019(Nosek et al., , 2018) that preregistration can contribute to mitigating publication bias in the academic literature (Nissen et al., 2016;Rosenthal, 1979;Scargle, 2000), since research plans are discoverable regardless of whether the final report is ultimately published in peer-reviewed journals. Yet, in our opinion, publication bias can only be effectively mitigated when results are published regardless of study outcome, that is, via Registered Reports (see Section 4.1) or journals that publish studies based on scientific rigor rather than their outcome (see Section 4.2). Nonetheless, discoverability of research plans is a useful step in making the entire research process discoverable. Importantly, preregistration is not only helpful when hypotheses are tested or p-values are reported (McPhetres, 2020), but also for exploratory 6 and qualitative research (Dirnagl, 2020;Haven and Van Grootel, 2019) and when using other statistical procedures (e.g., specify and justify in advance what priors will be used in a planned Bayesian analysis; see Depaoli and van de Schoot, 2017).
Correlational evidence accumulated over the past 20 years in several disciplines suggests that preregistration may facilitate the publication of non-significant findings, thus providing a more accurate representation of available knowledge. For instance, Kaplan and Irvin (2015) reviewed a sample of randomized clinical trials funded by the National Heart, Lung, and Blood Institute evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Of the 55 selected studies, 30 were published before and 25 after the year 2000, when study registration on ClinicalTrials.gov became compulsory in the U.S. following the Food and Drug Administration Modernization Act in 1997. Results showed that 57% of the studies published before 2000 showed a significant benefit of the intervention, as opposed to only 8% of trials published after 2000. 7 Similar results were reported in a (preregistered) meta-analysis of meta-analyses of orthodontics and dentofacial orthopedics studies: registered trials reported less favorable intervention effects compared to unregistered trials (Papageorgiou et al., 2018). Preregistration can also help identify whether funding sources are correlated with study outcome, potentially uncovering questionable practices due to (undisclosed) conflicts of interest. For instance, a review of studies of safety and efficacy trials for a wide array of drugs (Bourgeois et al., 2010) revealed that trials funded by industry were less likely to be published within 2 years from study completion and most likely to report a positive outcome (85%, as opposed to 50% for governmentfunded trials). 4 Preregistration at a later point in time is also possible, as long as authors transparently report at which stage of the study they crafted the protocol and declare that they are not yet aware of any results. Another possibility is to preregister analysis plans of data that have already been collected but not accessed, i.e. secondary data analysis (Mertens and Krypotos, 2019; Van den Akker et al., 2019). 5 It should be noted that the results of statistical tests can still be valid (i.e., expected false positives close to nominal α) assuming proper statistical conditioning, e.g., by building the conditional reference distribution of the test statistic via data permutation (for details, see Devezer et al., 2020, Box 2). 6 A preregistration can serve as a 'log' for exploratory research, to make the many choices during the research process transparent: "Methodological and analytic flexibility is maintained but disclosed." (Dirnagl, 2020, p. 4). 7 The clinical studies considered in Kaplan and Irvin (2015) were specifically chosen to be large, well-funded projects, likely to get published even if results were not statistically significant. Thus, their work does not directly show that preregistered studies are easier to get published. Yet, it does suggest that, if studies are preregistered, non-significant findings are more likely to be reported as such, instead of being p-hacked to chase publication.
Recently, Adda et al. (2020) analyzed the distribution of p-values of primary outcomes for phase II and phase III drug trials registered on ClinicalTrials.gov between 2010 and 2019 and found no indication of selective outcome reporting, suggesting that such registries may successfully disincentivize the (conscious or unconscious) use of suboptimal reporting practices and, consequently, improve the credibility of published research.

Benefits for individual researchers
Besides being advantageous for whole research fields, anecdotal experience and preliminary evidence suggest that preregistration can be beneficial for individual researchers as well (Allen and Mehler, 2019;McKiernan et al., 2016;Toth et al., 2020;Wagenmakers and Dutilh, 2016). Generally speaking, drafting a thorough preregistrationpreferably with the help of useful templates and checklists (see Section 3.1)can improve the experimental design not only because authors are stimulated to think more carefully about the research plan, but also because feedback from peers can be solicited 8 early and incorporated when most valuable, that is, when there is still time to make changes. Early-career researchers (ECRs) may benefit even more from learning this skill, since they are often directly involved with the ideation and development of the research project, data preprocessing and analysis, and writing of the final report. Preregistering a study as an ECR can also give a stronger sense of ownership over ideas that were originally conceptualized by their supervisors, for example by having a clearer overview on the different steps of the workflow, making informed decisions about the rationale, experimental design, and planned analyses early on in the project.
Given an increasing interest in transparency, we expect ECRs to be working in an environment that valuesand might even requirea certain level of commitment to open science practices, of which preregistration is an example. As mentioned earlier, a growing number of journals encourage preregistration, e.g., by means of badges. In academia, 9 funding agencies appreciate the importance of study preregistration in medical and non-medical disciplines: for example, the recent COVID-19 Programme by the Dutch funder ZonMw includes specific open science guidelines 10 for prospective applicants, among which mandatory preregistration of animal studies and "strongly recommended" preregistration for all other studies. In addition, preregistration may lead editors and reviewers to more easily trust authors when reporting certain methodological choices, such as sequential testing and one-sided tests (Lakens, 2017, Study 1). Last but not least, researchers who preregister their studies may be perceived as more trustworthy, because they are willing to open all products of their workflow to their peers for scrutiny. However, a recent registered report investigating whether preregistration increases peers' trust in the final publication revealed inconclusive evidence either in favor or against this hypothesis (Field et al., 2020), leaving this question open for future examinations.
The advantages of preregistration in neuroimaging and electrophysiology have not yet systematically been evaluated. Nonetheless, the data accumulated in other disciplines provide a number of insights, practical examples, and learned lessons that can guide a widespread and informed implementation of this practice in our research field. When done properly, preregistration works as intended.

Recommendations for preregistration of ERP research
In this section, we provide guidelines on how to transparently document the planned analytic choices in a preregistration of a prototypical ERP study. We focus on ERPs because of their widespread use in cognitive and clinical research (Hajcak et al., 2019;Helfrich and Knight, 2019), although most of these recommendations can still be useful when using other signal processing techniques (e.g., ERP and time-frequency analyses have many preprocessing steps in common). Furthermore, we only include sections that would decrease researchers' flexibility during signal preprocessing and statistical analysis. As discussed earlier, these steps are complex and multifaceted, with many reasonable choices that can lead to qualitatively different interpretations of the data; therefore, transparently documenting them would have a significant impact on the verifiability of the results. However, other aspects of a study should also be carefully planned and included in the preregistration protocol, e.g., the rationale behind the chosen sample size (including a power analysis; for recent guidelines, see Baker et al., 2020;Boudewyn et al., 2018), inclusion and exclusion criteria, and stimulus details and characteristics (e.g., to ensure that items sampled from all planned conditions are reported in the published manuscript).
We encourage researchers to craft a document that is specific, precise, and exhaustive (Veldkamp, 2017, chap. 6;Wicherts et al., 2016). A preregistration is specific when it includes a detailed description of all phases of the research workflow, from the initial design of the study to the information reported in the final manuscript; precise when the research plan is interpretable in only one way (e.g., there is no ambiguity regarding the intended preprocessing pipeline); and exhaustive when the research plan states that only the mentioned analyses will be considered as diagnostic to confirm or falsify predictions, thereby clarifying that other analyses have been conducted after seeing the data (see also McPhetres, 2020, on adding underspecified secondary analyses).
In our experience, it is very useful to run a pilot study before drafting the preregistration document. Advantages include: (i) gauge the feasibility of recruitment, randomization, and assessment procedures, especially when testing clinical populations and/or evaluating a novel treatment (e.g., Leon et al., 2011); (ii) ensure that task instructions are clear for participants; (iii) confirm that the target ERP component(s) are elicited; (iv) test preprocessing and analysis pipelines for possible bugs, errors, and/or computational feasibility. We emphasize that small scale pilot studies should not be used to estimate effect sizes to inform a priori power analysis (for details, see Albers and Lakens, 2018;Kraemer et al., 2006).

Preregistration templates
In principle, any time-stamped, accessible protocol with a clear study plan can serve as a preregistration. However, ready-made templates can greatly facilitate the inclusion of preregistration in researchers' workflows by providing a list of bullet points Wicherts et al., 2016). In addition, hosting preregistrations on online platforms that are popular among the research community (rather than, for example, personal websites) can improve accessibility. One of the most popular platforms is the OSF, which offers several preregistration templates 11 differing on topic, length, and specificity. While less extensive templates (e.g., AsPredicted: https://aspredicted.org/) are typically used by newcomers for their first preregistration, we would rather recommend the standard OSF Prereg template, whose increased level of detail facilitates the creation of specific, precise, and exhaustive preregistrations that more effectively decrease the risk of undisclosed analytic flexibility. A template specifically for preregistration of EEG studies was started during a hackathon at the annual meeting of the Society for the Improvement of Psychological Science (SIPS) in 2019 8 For example on platforms like OSF (in the comment section) or Peer Community In (https://peercommunityin.org/2020/01/15/submit-your-prere gistration-to-peer-community-in-for-peer-review/). 9 Solid project management skills are also extremely valuable outside of academia, where careful planning can help prioritize goals in a fast-paced environment (see Powell, 2018). 10 https://tinyurl.com/ZonMw-COVID19-OS. 11 https://osf.io/zab38/wiki/home/. (Algermissen et al., 2019) and is currently being developed online by an active community of volunteers. Readers are welcome to contribute to (and use) the current draft at https://tinyurl.com/eegprereg. Below we provide some examples on how to preregister typical preprocessing and analysis steps in an ERP study. We do not intend to recommend one preprocessing step or statistical method over another, but rather give examples on how commonly used preprocessing and analysis steps can be transparently reported. Please note that a good preregistration should also be explicit about the order of preprocessing steps. Again, the examples below are for illustrative purposes: their order is not meant to be prescriptive, and should be adjusted based on the pipeline that is appropriate for the specific study. 12 It can also be advantageous to include analysis scripts in the preregistration (see, for instance, Nunez et al., 2017). In any case, the preregistration should be specific about the software and (standardized) pipelines that will be used to carry out the preprocessing steps. If researchers plan to use the default settings of a given software, they should also include its version number (these settings might change with different versions) and clearly state in the preregistration that default parameters will be used.

Re-referencing
A preregistration should specify which electrodes will be chosen for the offline re-referencing. Common offline reference channels include the linked mastoids, ears, vertex, or an average reference. An example could be: "The continuous data is [...] re-referenced to the average of the left and right mastoid." (preregistration: Nieuwland et al., 2018a;publication: Fleur et al., 2020).

Filtering
Preregistering the parameter values of the filters that will be applied to the recorded EEG data should be detailed enough to theoretically allow readers to completely reproduce each filter (see Widmann et al., 2015). This includes specifying not only the filter cut-off frequency, but also the type (e.g., butterworth, finite impulse response, infinite impulse response), transition width, passband edge, the order for the transition bandwidth, at what point during the preprocessing pipeline the filter was applied (e.g., to continuous or segmented data), andin case of multiple filtersthe order in which the filters are applied. For example: "The continuous EEG data will be filtered with separate Hamming windowed sinc finite impulse response (FIR) filters (Widmann, 2006): (1) high-pass: passband edge 0.5 Hz, filter order 1,690, transition bandwidth 0.5 Hz, cutoff frequency (− 6 dB) 0.25 Hz; (2) low-pass: passband edge 30 Hz, filter order 114, transition bandwidth 7.4 Hz, cutoff frequency (− 6 dB) 33.71 Hz." (preregistration: Schettino et al., 2017;publication: Schindler et al., 2018).

Trial segmentation and time-locking
For trial segmentation, it is especially important to specify when the continuous EEG data are segmented, because this has implications for other preprocessing steps. 13 When preregistering trial segmentation, it is also important to specify what the trial will be time-locked toe.g., stimulus onset, participant's motor responseand how long the pre-and post-stimulus period will be. For example: "For the ERPs, we [..] epoch the data from -500 to 1500 ms relative to [critical word] onset." (preregistration: Coopmans andNieuwland, 2018, publication: Coopmans and.

Interpolation
With interpolation, the EEG signal recorded from noisy channels is replaced with estimated activity from neighboring electrodes. In the preregistration protocol, one should prespecify the criteria used to identify noisy channels as well as the algorithm that will be used for the interpolation. For example: "Bad channels with a voltage > 2 SD of the EEG voltage will be interpolated with a spline interpolation [...]. The interpolation algorithm [...] is implemented in the EEGLAB toolbox." (preregistration: Duma et al., 2018, publication: Duma et al., 2019.

Artifact rejection and correction
Approaches to artifact rejection can be roughly divided into three categories: (1) automatic; (2) manual; and (3) semi-automatic, that is, a combination of automatic and manual approaches. Many different algorithms for automatic artifact rejection are available and implemented in different software and toolboxes. For example, the FASTER algorithm (Nolan et al., 2010) calculates various statistical parameters of the signal and defines data as artifactual when exceeding a pre-specified z-score (e. g., ±3). This approach can be additionally integrated by identifying the frequency bands in which one would expect artifacts to occur, e.g., 110 to 140 Hz for muscle artifacts (see Delorme et al., 2007). From a computational perspective, fully automatic approaches are more reproducible, although their sensitivity and specificity can vary. Semiautomatic and manual approaches are more subjective and dependent on the researcher's skills, but they may be necessary when working with special populations (e.g., infants 14 ) whose signal shows less typical artifacts, difficult for automatic approaches to detect. Therefore, to increase transparency and reproducibility, the final report should include the list of epochs marked for rejection (or correction; see below). In addition, source code could be referenced in-text and be made publicly available for inspection upon publication, to provide additional information (e.g., on some parameter values) while decluttering the Methods section. An example of a planned automatic artifact rejection procedure pertaining specifically to ocular artifact rejection could read: "All trials will be checked for eye and muscle activity related artifacts. To detect eye movements and blinks, the EOG signals will be combined to derive bipolar vertical and horizontal channels that will be passed through a set of artifact detection steps. Any trials containing amplitude change larger than 80 μV in the vertical bipolar EOG channel and 120 μV in any other channel within a moving window of 200 ms will be removed to avoid any contamination of data by eye blinks or muscle movements. Additionally, any trials with potential eye-movement activity, i.e., amplitude changes larger than 25 μV in the horizontal bipolar EOG channel detected by a step function, will also be rejected. Participants with rejection rates larger than 30% of the total trials in any of the experimental conditions will be excluded from further analyses." (preregistration: Johnson, 2017, publication: Bocincova andJohnson, 2019).
In addition to artifact rejection procedures, there are a variety of techniques available to correct artifacts in order to reduce data loss. Methods for artifact correction include independent component analysis (ICA), regression-based methods, and wavelet-transforms (see Jiang et al., 2019). Each of these methods require different parameter values to be preregistered. For example, for ICA, at least the following information should be added (see, for instance, Keil et al., 2014): the method used to compute the ICAe.g., fastICA (Hyvärinen, 1999) or infomax ICA (Bell and Sejnowski, 1997) -, the electrodes included, the number of 12 Other examples of preregistration can be retrieved from a public Zotero Group library maintained by the Center for Open Science (https://tinyurl.com/ OSF-Zotero-group) as well as a spreadsheet started during the aforementioned hackathon at SIPS2019 (https://tinyurl.com/SIPS2019-prereg-list). 13 As already mentioned, high-pass filters may produce edge artifacts when applied to non-continuous signal; thus, segmentation is typically performed after these filters are applied. Other methods are also effective, e.g., extending epochs with zeroes (zero-padding) so that edge artifacts do not affect the actual signal.
14 Recently, though, automatic artifact rejection pipelines for infant EEG data have been developed, e.g., MADE (Debnath et al., 2020) and HAPPE (Gabard-Durnam et al., 2018). computed components, the method by which artifactual components will be identifiede.g., using templates (Campos Viola et al., 2009) or manually -, and which type of artifactual components will be removed (e.g., only ocular components, ocular and heart beat components, etc.). For example, a preregistration for artifact correction using ICA could read: "Data are then subjected to independent component analysis using single-order blind identification [...]. This is achieved by transforming the weight matrix for components into z-scores across all electrodes, and identifying those that have a z-score greater than 4.0. This is an arbitrary large value which has been determined in previous studies to identify signals due to blinks or to other artefact. Components whose activity is heavily focused on a single electrode are then subtracted from the signal." (preregistration: Bishop, 2014, publication: Hobson andBishop, 2016).

Baseline correction
A preregistration should also clarify whether or not the data will be baseline-corrected by setting the scalp distribution to zero during a preset period before the onset of the event of interest. Important information include the time window that is used as the baseline, and at which point in the analysis pipeline the baseline correction will be applied. For example: "Epochs extending from − 200 ms to +1000 ms timelocked to word onset will be created, and baseline correction will be applied using the pre-stimulus interval." (preregistration: Schettino et al., 2017;publication: Schindler et al., 2018).

Statistical analysis
Analytic flexibility does not only occur at the level of preprocessing. Rich M/EEG data sets can be analyzed in multiple different wayse.g., event-related potentials (ERPs; Luck, 2014), EEG microstates (Michel and Koenig, 2018), time-frequency (Cohen, 2014), functional connectivity (Bastos and Schoffelen, 2016), steady-state evoked potentials (Regan, 1977), source localization (Michel and He, 2019) -, with different objectives, dependent variables, and levels of analytic sophistication. Moreover, researchers have multiple valid options for statistical tests, such as ANOVAs and t-tests (Luck, 2014, chap. 10), clusterbased permutation tests (Maris and Oostenveld, 2007), Bayes factors (Keysers et al., 2020), linear mixed effects models (Frömer et al., 2018), threshold-free cluster enhancement (Mensen and Khatami, 2013), and more. 15 An example of ERP experiment may involve measurements from 64 electrodes and a sampling rate of 256 Hz, with a trial length of 1,000 ms after stimulus onset (e.g., Schindler et al., 2018). After averaging over trials, this would result in 16,384 data points for each participant and condition. For statistical analysis, this leads to a large number of potential comparisons, often referred to as the multiple comparison problem (MCP). Standard statistical correction procedures operating at the level of single electrode-time pairs would yield hyper-conservative results (increased Type II error); on the other hand, failing to correct for multiple comparisons can easily lead to spurious statistically significant results (increased Type I error). Therefore, statistical plans of ERP studies should always include a strategy on how to deal with the MCP. Several solutions are available (Luck, 2014, chap. 10;Luck and Gaspelin, 2017), including: (i) a priori definition of electrode sites and time windows based on previous studies; (ii) collapsed localizers, i.e., averaging all trials of all conditions of all participants to identify electrode clusters and time windows, thereby avoiding condition-specific biases; and (iii) mass univariate statistics, i.e., computing statistical tests at each electrode and time point and applying appropriate multiple comparisons correction techniques (Fields and Kuperberg, 2020;Groppe et al., 2011).
In the following sections, we provide examples on how to preregister three common ERP analyses: (1) Analysis of Variance (ANOVA); (2) cluster-based permutation tests; and (3) Bayes factors. We hope these examples will be useful for readers planning to preregister similar statistical analyses and give an idea of the preferred level of detail for other analysis methods not listed here.

ANOVA
ANOVAs are a popular statistical technique to analyze ERP data: it is not uncommon to read published studies including several within-and between-subject factors for various spatial and temporal ROIs. However, a growing number of factors comes at a cost, namely an increase in false positive rate: up to 50% chance to find at least one false positive effect with 4 factors, and up to 100% chance with 8 factors (Luck and Gaspelin, 2017). Therefore, ERP researchers should carefully plan appropriate corrections for multiple testing not only for the follow-up tests to an ANOVA, but also as a function of the number of factors included in the ANOVA itself. Alternatively, researchers can limit the number of factors included in the ANOVA if they have a specific hypothesis about the spatial and temporal region of interest or only run planned (paired) contrasts on the relevant comparisons. Preregistration can help make these decisions beforehand, without being biased by seeing the data.
In the following example, researchers interested in the N400 ERP component (Kutas and Hillyard, 1980;Kutas and Federmeier, 2011; see also the work by Nieuwland and colleagues for a series of preregistered N400 studies, e.g., Nieuwland et al., 2018b;Coopmans and Nieuwland, 2020) identify the time window and region of interest a priori, and aim to test two hypotheses: (1) larger N400 component for semantically incongruent compared to semantically congruent sentences (i.e., an incongruency effect); and (2) larger incongruency effect for native speakers compared to non-native speakers. In the Analysis section, the preregistration could read: "We will analyze the amplitude 16 of the N400 by means of a mixed ANOVA with 2 factors: congruency (semantically congruent sentences vs. semantic violations; within-subject) and language experience (native speaker vs. non-native speaker; between-subject)." As outlined above, researchers should also consider correcting for multiple comparisons as a function of the number of factors in the ANOVA. In this example, three p-values are computed: one for the main effect of congruency, one for the main effect of language experience, and one for their interaction. Researchers could preregister the correction in the following way: "The significance level for the main effect and interaction terms will be Bonferroni-corrected for the number of tests computed in the ANOVA: 0.05/3 = 0.0167." Of course, the alpha level of the test should also be explicit (α = 0.05 in this example). If researchers also plan follow-up comparisons, they could add: "In case the interaction between congruency and language experience is statistically significant, we will compute two paired t-tests comparing congruent vs incongruent sentences, separately for native speakers and non-native speakers. For these two planned comparisons, we will set α = 0.025 (Bonferroni-corrected for two planned tests with an uncorrected alpha of 0.05)."

Cluster-based permutation tests
Cluster-based permutation tests (CBPT; Maris and Oostenveld, 2007) are another popular statistical approach to analyze EEG data. In the following example, researchers have a hypothesis about a difference in ERP responses between two conditions, but not about specific electrodes and time points. The preregistration should include whether the test is 15 Other factors, including context-dependent psychometric properties of brain measures (e.g., Clayson and Miller, 2017) and suboptimal reporting of methodological details in published papers (e.g., Clayson et al., 2019;Larson and Carbine, 2017), further complicate the picture and warrant caution on the reliability of the psychophysiological literature (Baldwin, 2017). 16 It is also important to clearly specify the time window and electrode cluster from which the ERP component is scored. This information should be included in the section Measured variables of the preregistration protocol, e.g.: We will analyze the mean amplitude value in a time window from 300 to 500 ms after stimulus onset, averaged across a cluster of centro-parietal electrodes (C3, Cz, C4, CP5, CP6, P3, Pz, P4, P7, and P8).
one-tailed or two-tailed, within-subject, between-subject, or mixed, as well as the alpha level. In addition, there are several parameters that are more specific to the CBPT, such as the alpha at the cluster-level, the method for computing cluster statistics, the minimum number of electrodes that can form a cluster, how neighboring relations between electrodes will be computed, and the number of randomizations. For example, a preregistration using a CBPT could read: "To test withinsubject differences between congruent and incongruent sentences, we will compute a two-tailed cluster-based permutation test using 'ft_statfun_indep-samplesT' in Fieldtrip (Oostenveld et al., 2011) with α = 0.025 for each tail (i.e., the overall alpha is 0.05). The alpha at the cluster-level will be set at 0.05. Cluster statistics will be computed with a 'maxsum' approach and clusters will require a minimum of two neighboring electrodes. Neighboring electrodes will be defined via the 'triangulation' method implemented in Fieldtrip. Like on the test-level, the clusters will be tested with a two-tailed statistic. One thousand randomizations will be computed via Montecarlo method to estimate the p-value under the permutation distribution."

Bayes factors
Problems inherent in accepting the null hypothesis with classical frequentist procedures (e.g., Wagenmakers, 2007) and common misinterpretations of p-values (Colquhoun, 2017;Wasserstein and Lazar, 2016) are leading an increasing number of researchers to explore Bayesian approaches (Etz and Vandekerckhove, 2018;Kruschke and Liddell, 2017). Bayesian inference allows to incorporate prior knowledge into statistical tests, quantify evidence in favor of the null hypothesisthus discriminating between "absence of evidence" and "evidence of absence" (e.g., Keysers et al., 2020) -, and monitor the evidence as the data accumulate (Rouder, 2014; but see de Heide and Grünwald, 2020). In particular, Bayes factors -"the extent to which the data sway our relative belief from one hypothesis to the other" (Etz and Vandekerckhove, 2018, p. 10) -have gained considerable popularity, also thanks to the development of user-friendly software that facilitate their calculation (e.g., JASP; https://jasp-stats.org/).
Researchers planning to analyze their data using Bayes factors should clarify the software and procedure used for the estimation, a description of the prior specification (i.e., the type of distribution and its parameter values), and an assessment of the robustness of the results under different prior specifications (see also van Doorn et al., 2020). A preregistered description of planned comparisons using Bayes factors could read as follows: "We will analyze the amplitude values of the N1 ERP component using Bayes Factors (BFs; Kass and Raftery, 1995). Two-tailed Bayesian t-tests (Rouder et al., 2009) will be calculated to estimate the degree of evidence in favor of a model assuming differences between conditions relative to a model assuming no differences. The null hypothesis will be specified as a point-null prior (Dirac distribution, standardized effect size δ = 0), whereas the alternative hypothesis will be defined as a Jeffrey-Zellner-Siow (JZS) prior, a folded Cauchy distribution centered around δ = 0 with scaling factors of r = 1, r = 0.707, and r = 0.5, to verify the robustness of the results as a function of changes in the prior (Schönbrodt et al., 2017). Participants will be included as random factors, and their variance considered nuisance. The threshold to identify the winning model is set at BF ≥ 10 or BF ≤ 0.1, typically considered "strong" evidence in favor of the model in the numerator or denominator, respectively (Kass and Raftery, 1995). BFs will be estimated via the R package BayesFactor v0.9.12-2 (Morey et al., 2015) using Markov Chain Monte Carlo sampling (10,000 iterations)." (preregistration: Schettino et al., 2017;publication: Schindler et al., 2018).

General considerations
In Section 2 we clarified how preregistration can mitigate some of the issues related to undisclosed analytic flexibility. In Section 3 we provided guidelines and examples on how to preregister common preprocessing and statistical analysis steps in ERP studies. In Section 4 we discuss several considerations that EEG researchers may want to take into account when critically evaluating whether to preregister their studies.

Preregistration vs. Rregistered Reports
Throughout this manuscript we have described unreviewed preregistrations (see van't Veer and Giner-Sorolla, 2016), that is, the protocols uploaded on public repositories are not formally peer-reviewed. A preregistered study can still be rejected by scientific journals for a number of reasonsfor example, lack of interest in "negative" (nonsignificant) findings (Fanelli, 2010), unprofessional peer-review (Gerwing et al., 2020), or submission in the "wrong" day of the week (Boja et al., 2018) -, thus limiting its discoverability (although the experimental protocol may still be publicly accessible). Conversely, reviewed preregistrationscommonly referred to as Registered Reports (Chambers and Tzavella, 2020) are alternative article formats in which the study proposal is peer-reviewed and conditionally accepted for publication (in-principle acceptance, or IPA), provided that the original plan is followed and deviations are properly documented. Publication is thus independent from study outcome. Preliminary research has shown that this format seems to effectively mitigate publication bias and reduce the prevalence of selective outcome reporting (Scheel et al., 2020; see also Wiseman et al., 2019). 17 Therefore, we consider Registered Reports the state-of-the-art article format for confirmatory research and recommend them over preregistrations. At the time of writing, more than 250 journals 18 from various scholarly disciplines offer Registered Reports alongside traditional submissions, including Psychophysiology (Keil et al., 2020) and the International Journal of Psychophysiology (Larson, 2016). Please note that, while the current manuscript focuses on preregistrations, our recommendations also hold for Registered Reports.
Despite these desirable properties, researchers should take into account the relatively strict submission criteria (e.g., expected statistical power of 90% or higher 19 ) as well as the time necessary to review the study plan (typically 2-4 months; Chambers, 2020), during which the project cannot start. For these reasons, preregistration can be seen as an easier, entry-level practice that is advantageous in itself and helps researchers familiarize with the steps required for a future (recommended) Registered Report submission.

Potential disadvantages of preregistration
Crafting a comprehensive preregistration protocol requires time. A recent survey (Toth et al., 2020) revealed that respondents with previous experience in preregistration invested, on average, around 4 hours to create the initial draft. Our experience suggests that this could be considered a lower bound: the multidimensional nature of EEG data, coupled with the high level of specificity recommended to effectively avoid selective reporting, requires documenting a large number of preprocessing and analysis steps (see Section 3), planned sample size (with a priori power analysis), sampling strategy, inclusion and exclusion criteria, and more. As Allen and Mehler (2019) point out, planning analyses based on existing data is likely easier, because a preregistration involves anticipating possible outcomes, for example in a decision tree, that depend on seeing the data (e.g., whether the assumptions of the planned statistical model are fulfilled). These authors further suggest 17 Due to heterogeneous journal policies, IPA protocols may not always be publicly available or easily verifiable (Hardwicke and Ioannidis, 2018), which sometimes makes it difficult for readers to compare registered plans and published papers. Recent developments have tackled the lack of transparency and standardized protocol registration by updating recommended editorial policy templates (Chambers and Mellor, 2018), but we advise to read the target journal's specific guidelines before submission. 18 The updated list of journals offering Registered Reports can be found at https://cos.io/rr/. 19 As an example, see Registered Reports Submission Guidelines at Cortex that implementing open science practices (including preregistration) during a project increases its duration ("In our experience, these additional requirements can easily double the duration of a project."; p. 4), which might be especially difficult for ECRs on short-term contracts. However, to the best of our knowledge, this is based only on anecdotal evidence and preregistration could also, at least for some projects, save some time. 20 We also note that most of the information included in a comprehensive preregistration is also required in the final manuscript, not only to facilitate communication between the authors and other relevant parties (editors, reviewers, and readers) but also to ensure that the methods leading to the conclusions advertised in the paper are reproducible. Publication guidelines for M/EEG studies (Gross et al., 2013;Keil et al., 2014;Pernet et al., 2018) emphasize the need to describe equipment, study materials, preprocessing steps, dependent variables, and analysis pipelines, and also provide a checklist that authors can consult while writing the manuscript (Keil et al., 2014, section Appendix). Here we propose to anticipate this time investment, with the advantage that carefully thinking about these methodological details before data collection may lead to improvements in the study design when still useful.
Some researchers might also be worried that the time invested in writing the preregistration would be wasted if results do not pan out as expected and, consequently, the final manuscript would be more difficult to publish. Indeed, current incentive structures (be it hiring practices, journals, or funders) usually value quality over quantity (Allen and Mehler, 2019) and exert pressure to publish novel, groundbreaking, positive results (Fanelli, 2012(Fanelli, , 2010Ioannidis et al., 2014;Jennings and Van Horn, 2012;Nissen et al., 2016;Scargle, 2000). Nonetheless, many academic journals accept manuscripts with non-significant findings if the methodology is robust, with the aim to mitigate the pervasive problem of publication bias. 21 Similarly, hiring practices are starting to reward open science practices (Schönbrodt et al., 2020). We are highly sympathetic to Allen and Mehler's (2019) call to align incentive structures more with open and transparent research practices that value quality over quantity (see also Flier, 2017).
Unforeseen circumstancese.g., problems recruiting the planned number of participants due to a pandemicmay require reasonable deviations from the original preregistered plan. This is acceptable as long as it is transparently documented in the published paper. Regrettably, recent evidence shows that undisclosed protocol deviations are common. An analysis of preregistered studies in the journal Psychological Science (Claesen et al., 2019) showed that none of them had perfectly anticipated every step of the research project: differences between preregistration protocols and final manuscripts were observed, for instance, in sample size, exclusion criteria, and statistical models, with only one study transparently reporting all discrepancies. Partial or lack of disclosure of deviations from pre-study plans is a well-known problem not only in social sciences (e.g., Franco et al., 2016) but also in clinical trials (Goldacre et al., 2019), for which registration is compulsory or strongly encouraged in many countries (see the World Health Organization Registry Network; https://www.who.int/ictrp/network/en/). We emphatically recommend to clearly report any deviations from the preregistered plan, preferably in a separate section in the main manuscript or in Supplementary Materials. A useful checklist can be found on the OSF (https://osf.io/yrvcg/).

Preregistration is not a silver bullet
As argued in Section 2.1, preregistration can strengthen the evidential value of studies by increasing transparency, disclosing selective outcome reporting, and increasing the number of publications with nonsignificant findings. However, adopted in isolation, preregistration is not sufficient to increase scientific rigor; for example, it may not necessarily prompt researchers to carefully examine whether their chosen statistical models are appropriate for the experimental question (Guest and Martin, 2021;Szollosi et al., 2020), improve statistical inferences (Devezer et al., 2020;Navarro, 2020), strengthen the link between theories and their mathematical representations (Szollosi and Donkin, 2019), or develop more precise, consistent, and "hard-to-vary" theories altogether (Szollosi and Donkin, 2021;van Rooij and Baggio, 2021). In other words, preregistration in itself does not necessarily improve the quality of the research, and it might even be harmful if it grants statistically invalid or theoretically weak research an unwarranted higher status ("a superficial veneer of rigor"; Devezer et al., 2020, p. 19) compared to nonpreregistered, but otherwise solid and transparently documented, research. In fact, when other conditions are fulfillede.g., a strong theoretical framework that warrants precise predictions; a clear justification of analytical choices; open data, materials, and code; and/or convergent results via multiverse analysis -, preregistration does not necessarily lead to more robust and trustworthy conclusions (Rubin, 2020). Unfortunately, these conditions are rarely fulfilled in electrophysiology and psychophysiology, even when investigating popular topics with a long research tradition. As an example, a recent review on the electrophysiological correlates of early word prediction (Nieuwland, 2019) analyzed available evidence for the sensory hypothesis as opposed to the recognition hypothesis. This analysis revealed that current published evidence is often obtained via novel tasks with unclear specificity and sensitivity, in samples which might not be sufficient for a precise estimation of small or medium effect sizes, using statistical analyses that do not allow to accurately partition between different sources of variance, and whose data cleaning and analysis procedures are unavailable for scrutiny. As argued in the current paper, preregistration may help researchers think about all of these steps in the planning stage of their projects, with clear advantages in avoiding confirmation and hindsight bias. We thus believe that preregistration serves the important goal of increasing transparency (Navarro, 2020) by offering a window into the research workflow, an often messy and non-linear process that is far from the flawless stories recounted in academic papers. Accepting these imperfections may promote a work culture that normalizes errors, acknowledges the depth of domain-specific knowledge, and fosters intraand interdisciplinary collaborations (see also . Moreover, preregistration offers the opportunity to evaluate whether the chosen tests support or falsify theoretical predictions, i.e., their severity (Lakens, 2019).
Finally, just like other research practices, preregistration can in principle be used unethically (Yamada, 2018). For example, one could preregister a large number of similar experiments and keep them under embargo. Whenever one of the studies turns out to be "successful" (i.e., statistically significant), the resulting paper would only refer to the corresponding preregistration and all the other "unsuccessful" ones could be withdrawn. However, metadata and a justification for each withdrawal would still be publicly available and therefore raise suspicion. Another unethical practice has been termed "preregistering after the results are known" (PARKing; Yamada, 2018), i.e., drafting and publishing the preregistration of a study that has already been completed and whose results conveniently fit within the narrative of the "preregistered" document. Complementary open science practices, e.g., data and code sharing, can effectively mitigate this risk. Journal editors and reviewers are invited to carefully compare the preregistered document with the manuscript during submission and evaluation, as well as request raw data and analysis code by default (Morey et al., 2016). Having said that, we prefer to think of fellow researchers in more 20 https://antonio-schettino.com/post/2019-07-23-prereg-challenge/. 21 See, for instance, two recent Nature (2020, 2017) editorials, the PLOS ONE article collection Missing Pieces (2015), the editorial by Munafò and Neill (2016) in the Journal of Psychopharmacology, as well as the submission guidelines of Meta-Psychology (https://open.lnu.se/index.php/metapsychology/about) and Royal Society Open Science (https://royalsocietypublishing.org/rsos/for-autho rs). optimistic terms, motivated by higher goals than simply publishing as much as possible (although we might, admittedly, be hopelessly naive; Chapman et al., 2019;DeDeo, 2020). We also point out that premeditated approaches to exploit the vulnerabilities of a system can hardly be reconciled with claims that mistakes were made in good faith due to ignorance or procedural complexity: researchers engaging in such behaviors (if proven) would consciously commit fraud, and responsible institutions should be contacted and deliver appropriate sanctions.

Conclusion
The adoption of a new procedure can be met with resistance, particularly if the benefits are unclear, the amount of work is perceived as too onerous, and training and guidance are lacking. In this paper, we argue that preregistering EEG projects can effectively facilitate the transparent reporting of data preprocessing and analysis choices, thereby improving study replicability and the verifiability of published knowledge. The time spent writing a preregistration is saved at later stages, because the information included in a comprehensive protocol is required at the time of publication. Ready-made templates can serve as useful guidelines and facilitate the implementation of this practice in the research workflow. Combined with other open science practicese.g., sharing study protocols, materials, raw data, and analysis code -, preregistration increases transparency in the research process and trustworthiness of the scholarly products not only for academic peers, but also other stakeholders in society (Jamieson et al., 2019).

Contributions
AS supervised the project. All authors wrote the initial draft, reviewed, edited, and approved the final version of the manuscript.

Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.