A primer for choosing, designing and evaluating registered reports for qualitative methods

Registered reports are a publication format that involves peer reviewing studies both before and after carrying out research procedures. Although registered reports were originally developed to combat challenges in quantitative and confirmatory study designs, today registered reports are also available for qualitative and exploratory work. This article provides a brief primer that aims to help researchers in choosing, designing, and evaluating registered reports, which are driven by qualitative methods.


Introduction
Registered reports ("RRs", Chambers & Tzavella, 2022) are a modification of the scientific publication process that aims to shift publication decisions from being based on the nature of the results of the study and towards rigorous conceptualization and design.In so doing, RRs improve transparency and the timeliness of peer review.The FORTT (2022) glossary defines RRs as a two-stage process, where peer review occurs at study design (Stage 1) and, if 'in-principle-accepted' (IPA), again at study completion (Stage 2).While RRs were originally developed for research following the hypotheticodeductive method (Chambers, 2013), today they are also open to explorative and qualitative studies (Branney et al., 2022).In fact, some qualitative RRs are already emerging (e.g., Stage 1, Topor et al., 2022) and at least two have already been published (Stage 2: Karhulahti et al., 2022;Xiao, 2022).
As RRs are likely to be unfamiliar to those using qualitative methods, this is a brief primer on choosing, designing and evaluating RRs for qualitative research.This primer is principally for people who design and conduct research using qualitative methods, although it may also be useful for editors and undergraduate and graduate students.As a rule of thumb, we propose the following: the more qualitative researchers can commit to research decisions before the study, the more benefits they will harvest by using the RR format.In other words, not all qualitative designs benefit from using the RR format equally, but some may benefit significantly.

Choosing RRs for qualitative research
Frontloading peer review Choosing to carry out a research project in the RR format allows one to receive peer review feedback in the design phase.Feedback alone can be a good reason for choosing the RR format if the study timeline allows.Projects that are carried out under severe time pressure should carefully consider whether they wish to use the RR format, as reviewing the study design usually takes time (see Evans et al., 2023).On the other hand, RRs submitted via Peer Community in Registered Reports (PCI RR) can schedule their review to save time, or even start generating data before completing Stage 1 by lowering the reported control level (PCI RR, 2022).

Negotiation…
…with fellow researchers and stakeholders In frontloading the design, an RR gives legitimacy to researchers and stakeholders spending time transparently negotiating the study design, including uncertainties and delayed decision making.This could mean that the design is collaborative and each person involved has agency in decision making.For example, in Button et al.'s (2020) model for consortium based empirical undergraduate dissertations, the academics collaboratively write the study protocol and main research question; once the new academic year commences, the dissertation students join the consortium and they collaboratively develop the protocol further.This process, developed in the context of quantitative research, is just as applicable to qualitative research.Qualitative RRs allow editors, reviewers, and authors to negotiate research questions and the study design, empowering all those involved.

…with journal or review platform
The RR format allows negotiating research decisions with editors before carrying out the study.As journals and review platforms have many explicit and implicit policies regarding the studies they publish, using the RR format allows pursuing an agreement with them at Stage 1.This can save valuable time and resources compared to traditional publication processes, which sometimes produce month/year-long (desk) rejection loops.Again, this process was developed with quantitative methods in mind, but is equally relevant to qualitative methods.

Transparency
Due to the flexible nature of decision-making in qualitative research, authors may often need to apply for permission to make changes between Stage 1 and 2.Although such changes may also need to be applied when encountering problems in hypothesis-testing RRs, in qualitative RRs it is a natural means for documenting the expected and unexpected events in the research process.

Designing qualitative RRs
Analysis Methods of analysis should be written in such detail that editors and reviewers understand what will be done (Stage 1) and what was done (Stage 2).Researchers should also clearly explain how they plan to report the data and findings.In qualitative studies, methods tend to include flexible elements (see Haven & Van Grootel, 2019).Such elements should be identified and planned, i.e. if researchers cannot decide between multiple alternative analyses, rules of decision making can be stated instead (e.g., if X then we Y).If biases are of epistemological concern, it is also possible for teams to apply masked analyses (Dutilh et al., 2021).

Data access & stewardship
Qualitative research can include a wide range of different data types such as audio, images, videos, and written text.All data types are suitable for RRs.Qualitative data are likely their Stage 1 proposal to minimize misunderstandings in peer review.

Hypotheses
It is uncommon (but not impossible) for qualitative research designs to test hypotheses.On the other hand, it is also possible to set hypotheses without testing them.These "qualitative hypotheses" (QHs) can be used in a similar way as positionality statements, i.e. to report the team's prior beliefs and hypothetical biases, which can affect the study design and its procedures.So far, at least two RRs (Karhulahti et al., 2022;Topor et al., 2022) have used qualitative hypotheses.Unlike positionality statements, QHs are based on previous data, literature, and theory, which have influenced the study design and may influence data interpretation, thus being akin to 'priors' in Bayesian statistical designs (see Andrews & Baguely, 2017).

Positionality
Because qualitative research is often highly interpretive and reflective, it is important to disclose the position(s) from which the interpretations and reflections are made.It may be useful to have separate positionality statements for data generation (e.g., interviews) and analysis (e.g., coding).The APA Journal Article Reporting Standards for qualitative research, for example, recommend describing the researchers, how their perspectives were used in methodological integrity of the data collection and analysis, and their understanding of the conclusions (APA, 2020;Levitt et al., 2018).

Research questions
As qualitative research is usually nonconfirmatory, one of the most important parts of study design is the formation of useful research questions.Specific types of qualitative data and analyses are often suitable for producing answers only to specific types of research questions, for which authors should carefully assess these relationships in their design.In general, good research questions ensure that the produced answers will contribute to the field.

Sample (size) and participants
There are no universal sampling rules for qualitative studies, but the selected participants or other data should always be justified.Different justifications apply to different types of data and methods.When justifying the nature and size of a sample, authors might also apply e.g., saturation, where the analytic process defines the sample during data generation e.g., Low, 2019).In such cases, however, it can still be useful to preregister estimations, which facilitate editorial work and increase transparency.

Evaluating qualitative RRs
Evaluating cost-benefit (Stage 1) Successful RRs receive in-principle acceptance after Stage 1, which means that one of the special features in evaluating them is to assess whether the study is worth carrying out (in contrast to evaluating completed work for which authors seek a publisher).Importantly, this is not the same as "impact", to present 'legitimate sensitivities' to participants' privacy (Branney et al., 2019), even through the 'innocent collection of details ' (Branney et al., 2017).Therefore, it is important to outline and negotiate data sharing plans at an early stage, and RRs facilitate this process (Karhulahti, 2022a).Two current trends contribute to the importance of these efforts.
The first trend is the digitization of our research and data protection legislation, which means that data can have a short life (Fang et al., 2013) and will be impossible to access without clear consent and data sharing agreements between all organizations (compared to reading through letters or documents in a filing cabinet, for example where researchers had only a single copy of their data but still shared it, see Craig & Reese, 1973).Organizations, for example, may be unable to grant access to archeologists of the future.The second trend is the range of data sharing and open data policies of research bodies, such as funders and professional bodies (see Riley et al., 2019).From 2013, the UK Economic and Social Research Council had a policy that data should be available for reuse within three months of grant completion (ESRC, 2013).In turn, the British Psychological Society has a 'as open as possible; as closed as necessary' position statement on open data (BPS, 2020).
Due to various pragmatic difficulties of anonymizing qualitative datasets and the significant amounts of labor typically involved in such processes, it is less common for qualitative studies to share data openly, especially in human research.On the other hand, some qualitative approaches like conversation analysis have a long open data tradition (see Joyce et al., 2022) and data sharing is also possible for other types of qualitative studies (for an overview, see DuBois et al., 2018).The Qualitative Data Repository was developed for precisely this purpose, but authors can also consult national and international repositories for assistance in sharing procedures.Anonymization (or pseudonymization), consent procedures, and controlling data reuse are preferably carried out in collaboration with archives and experts (Karhulahti, 2022b).Clear reasons should be provided if data are non-shareable.Notably, some qualitative data is not collected but rather generated (e.g., interviews), in which case the researchers' own involvement should be recognized, for instance, via positionality statements.

Ethics approval and IRBs
Different countries and universities apply different processes for reviewing research ethics, and each Stage 1 submission should communicate how they meet the standards of their country or institution.When needed, ethics approval for RRs should generally be obtained before Stage 1 submission, but parallel and post-IPA approvals are negotiable as well in case the committee is inflexible (see Figure 3 in Chambers & Tzavella, 2022; also, PCI RR, 2022).Authors can discuss with journal (or review platform) representatives when applying for approval, being explicit to each about the amendments recommended by each process.If an ethics approval sets critical limits to a planned design, authors can also consult RR representatives before submission and report those limits in but rather the degree to which a study can contribute, given the available resources.The value of expected findings needs to be assessed against the current knowledge, e.g.do the findings have potential to produce useful contributions to knowledge or improve the understanding of a phenomenon?In most cases, this means carefully evaluating the research questions and their match with the generated data as well as applied analyses.Many authors are committed to asking specific research questions (e.g., for funding bodies, community partnerships, etc.), for which even small contributions can be worth pursuing.Primarily, reviewers should help authors to maximize the contribution potential of their resources.

Evaluating interpretations (Stage 2)
As authors must clearly spell out how they will report the data and results at Stage 1, reviewers should start by assessing whether the plan was followed and if not, are possible deviations pre-approved or otherwise justified.A key difference between qualitative and other RRs is the former's strong affiliation with interpretative and reflective methodology.This means that reviewers have less control over the production of findings compared to e.g., statistically driven studies.Because interpretations may not always be fully reproducible (recall positionality), reviewers balance their assessment between ensuring that authors communicate their interpretations clearly, back up the interpretations with data, and draw reasonable conclusions from the analysis (see Josselson, 2004).

Evaluating language (Stage 1 & 2)
Qualitative studies, which are typically exploratory by nature, should rarely use confirmatory language.Likewise, as qualitative studies less commonly reach high generalizability, or even seek to generalize at all, language should be used with care when discussing the findings in general contexts.Reviewers should pay special attention to the language of discussion and conclusions at Stage 2.

Evaluating data/materials (Stage 1 & 2)
Following the BPS principle of as 'open as possible, as closed as necessary' (BPS, 2020), peer reviewers should generally be provided secure access to data at Stage 2. Materials used or produced in analysis should be included as an appendix to the shared data whenever possible; for instance, coding manuals and documentation for establishing reliability/trust should always be shared (when applicable).Use of supplemental materials can allow for additional reporting of qualitative results and findings that may not fit within the confines of a journal article.Importantly, data sharing is not a binary question, but usually parts of the data can be shared whereas other parts (e.g., with potential personal identifiers) cannot (Syed, 2022).These dimensions must be assessed case by case and researchers may apply the FAIR principles as useful guide in thinking about how data is shared, not shared or 'stewarded' (see 'Step 3: opening up your (meta) data' in Branney et al., 2022).Agreeing upon a clear plan for data and material sharing at Stage 1 and assessing them at Stage 2 are an important part of the evaluation process.

Use cases and discussion
Registered reports are becoming increasingly popular across fields, and many of their benefits also apply to qualitative methods.So far, our own experiences as authors and evaluators serve as practical use cases, which have demonstrated the potential of registered reports for qualitative methods.The present primer, in fact, was originally inspired by the dialogues we had during two RR review processes: in the first one, PB served as a reviewer of a manuscript where VMK/ MSI were authors (Karhulahti et al., 2022), and in the second one MSY served as a reviewer and VMK as a recommender (Topor et al., 2022).Afterwards, these review processes led us to collectively discuss the limits and possibilities of qualitative RRs.
As authors (Karhulahti et al., 2022), we witnessed a significant improvement of our phenomenologically motivated research design at Stage 1 due to valuable review feedback.For example, the feedback introduced us to new literature and perspectives, which further helped us to plan analysis and data sharing procedures, consider various reflexive elements in the process, and better distinguish our interpretive phenomenological approach from epistemologically different phenomenology.Moreover, the feedback also improved the longitudinal design by replacing unnecessary components (e.g., contextualizing interviews) with more focused member checking-a change that we did not realize ourselves due to the heavy time pressure set by the funding scheme (Box 1).Overall, the

Box 1.
The study was funded as part of a project that took place 1.1.2021-31.12.2022.Funding had been received to run a longitudinal qualitative study where the experiences of both healthy and treatment-seeking videogame players were followed for 12 months, with two interview rounds by a one-year interval.Because of the double interview design, two separate manuscripts were planned.We utilized the "programmatic" feature at PCI RR, which allows registering and reviewing multiple analysis plans via one Stage 1 manuscript (in our case: two rounds of interviews by one-year interval).The manuscript was submitted to PCI RR in June 16 (2022) and three reviews were received July 16.After one more review round, the Stage 1 plan received in-principle acceptance on September 24.Because of the fixed project timeline, we decided to start the first round of interviews before in-principle acceptance in order to ensure that our follow-up a year later remained possible within the project deadline.This meant lowering the confidence level (we had generated some data before the Stage 1 review was completed)-we were not worried about maintaining the highest confidence level because pre-study confidence plays a less meaningful role in exploratory qualitative research anyway.The first Stage 2 outcome was published a year later (September 21, 2022) and the longitudinal follow-up Stage 2 is currently in review.registered report format did not conflict with the flexibility characteristic of qualitative research, but we were also allowed to make non-registered choices, such as reporting themes through a case format that turned out optimal after final analysis.
As evaluators (Karhulahti et al., 2022;Topor et al., 2022;Xiao, 2022), we witnessed exceptional motivation to support the authors because feedback at Stage 1 can have a fundamental positive effect on the study design and the quality of the research-this cannot happen to the same degree in other publication formats, which are reviewed after the study has already been carried out.A key manifestation of this process is working with study authors to align their research questions with their methods and analysis plans to ensure a coherent, informative final product.In sum, based on our experience, registered reports allow authors and evaluators to "play in the same team," of which the present primer is also a practical example: authors and their evaluators having teamed up for follow-up collaboration.
We hope this brief primer is helpful for authors, editors, and reviewers involved in qualitative registered reports, and we look forward to updating these recommendations along with our accumulating experience and knowledge.

Comments
First, I wonder about the relevance of Figure 1?The content of the figure is nicely summarised in paragraph 2 as "the more qualitative researchers can commit to research decisions before the study, the more benefits they will harvest by using the RR format".This makes the figure somewhat redundant.Also, the figure suggests a simple direct linear relationship between the benefits of using RR and commitments to prior decisions, which might be a little misleading, unless there is evidence that can back up this particular type of relationship.

○
In the first paragraph of the section "Frontloading peer review" the authors state that "Choosing to carry out a research project in the RR format allows one to receive peer review feedback in the design phase.This can be helpful especially for early career researchers and teams with limited qualitative expertise".It would be useful to provide some backup for this last claim, in the form of, for example, citations, concrete examples etc.If no evidence for this exists yet, then perhaps the claim should be reformulated as a hypothetical statement.

○
The section on "Negotiation…with fellow researchers" features a nice example of how RR allow multiple parties to participate in negotiating the design of a QUANTITATIVE study.
Given that the audience for this article may be interested mainly in how this applies to QUALITATIVE research, an example of how negotiating works in a qualitative study would ○ be welcome.
In the section on "Data access & stewardship", the claim is made that "It is less common for qualitative studies to share data openly, especially in human research".This is certainly true, but it would be beneficial to include some context for why this is the case.Also, it might be worth highlighting that some qualitative approaches, such as conversation analysis, include data sharing practices embedded in their methodologies.In fact, in conversation analysis, data have always been shared in many different ways and that is a defining characteristic of this approach -for details, see Often methodological discussions (including those about Open Science) are written up in terms that apply mainly to quantitative research methods and that is problematic because those terms mobilise some assumptions about research that do not apply to all qualitative methodologies.Therefore, it may be relevant for the authors to go through the paper and consider some of the vocabulary choices made and their alternatives.I can see that this has already been considered on page 4 when the issue of data "collection" vs "generation" is mentioned.Other terms that may benefit from reflection are: "sample", "results" (vs analysis or findings) etc. Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Open Science in qualitative research; qualitative methodologies; conversation analysis I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Corina Logan
Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany This is a timely "How to" article for researchers conducting qualitative research in how to write, evaluate, update, and use registered reports to their advantage.Registered reports are new for qualitative research and the experiences of these authors who have been involved in the process as authors and evaluators is particularly useful.I am sure this will primer will be welcomed widely.
I have only a few minor comments: Transparency > change "apply for a permission" to "apply for permission" Transparency > "Although this is generally against the philosophy of hypothesis-testing RRs".It is my understanding that authors with Stage 1 IPA at various journals can contact their editor after IPA to discuss changes they might need to make, even for RRs that involve hypothesis testing.
Ethics approval > change "ethics approval for RRs should generally be applied before Stage 1 submission" to "ethics approval for RRs should generally be obtained before Stage 1 submission" Ethics approval > change "but parallel and post-IPA approvals are negotiable as well in case in inflexible committees" to "but parallel and post-IPA approvals are negotiable as well in case the committee is inflexible" Hypotheses > what a great idea to liken QHs to priors!Evaluating cost-benefit (stage 1) > I love the distinction between usefulness and impact.Change "study can contribute, given resources" to "study can contribute, given the available resources" Evaluating interpretations (Stage 2) > "are possible deviations justified".Perhaps also add whether these deviations were pre-approved by the editor, as you state above.
Use cases and discussion > "Stage 1 due valuable review feedback" to "Stage 1 due to valuable review feedback" Use cases and discussion > change "same degree in publication formats" to "same degree in other publication formats"

All my best, Corina Logan
Is the rationale for developing the new method (or application) clearly explained Yes

Is the description of the method technically sound? Yes
Are sufficient details provided to allow replication of the method development and its use by others?

○ References 1 .
Joyce J, Douglass T, Benwell B, Rhys C, et al.: Should we share qualitative data?Epistemological and practical insights from conversation analysis.International Journal of Social Research Methodology.2022.1-15 Publisher Full Text Is the rationale for developing the new method (or application) clearly explained Yes Is the description of the method technically sound?Yes Are sufficient details provided to allow replication of the method development and its use by others?Yes If any results are presented, are all the source data underlying the results available to ensure full reproducibility?No source data required Joyce et al. (2022).I found the section on 'Use cases and discussion' particularly valuable because if briefly presented concrete examples of RR.I was wondering whether the authors could elaborate this section, and, in particular, give concrete examples of how the feedback received at Stage 1 has helped improve the articles.
○ ○Lastly, I have a general observation about the use of methodological terms in this paper.