Evaluating Registered Reports Funding Partnerships: a feasibility study [version 1; peer review: awaiting peer review]

Background: We studied a novel initiative – Registered Reports Funding Partnerships (RRFPs) – whereby research funders and journals partner in order to integrate their procedures for funding applications and Registered Reports submissions into one process. We investigated the feasibility of conducting a randomised controlled trial (RCT) of the impact of RRFPs on (1) research quality and (2) the efficiency of the research process, from funding to publication. Methods: We conducted 32 semi-structured interviews and follow-up questionnaires with stakeholders (funders, editors, authors, and reviewers) across six different RRFPs. Results: A RCT of RRFPs appears to be feasible in principle. The partnership concept seems worthwhile to pursue further and is adaptable to the needs of various funders and publishers, and across disciplines. Three primary outcomes of interest should be measurable, and participant randomisation could conceivably be done in a number of ways. In practice, however, the current volume of submissions going through existing partnerships is too low to support a full trial. Conclusions: Although a RCT of RRFPs is conceptually feasible, it will only be possible if organisations are willing to form new partnerships, scale up existing ones, and incorporate a trial (i.e., randomisation) into these partnerships.


Introduction
Registered Reports (RRs) are a publishing format that aims to improve the research and publishing process by having manuscripts undergo two stages of peer-review (Chambers, 2013). Stage One review occurs before data collection, with study protocols evaluated based on their methodological rigour and the importance of the research question. Successful submissions are granted 'in-principle acceptance' (IPA) by the journal. This means the journal agrees to publish the final manuscript if it passes Stage Two peer review. At this stage, reviewers check the final manuscript, ensuring that the authors have adhered to the research plan, justified any deviations from it, and made reasonable conclusions based on their results.
The RR format is intended to combat various issues in research conduct and publishing, such as publication bias, selective reporting of results, inadequate statistical power, and undisclosed analytical flexibility (Chambers et al., 2014;Hardwicke & Ioannidis, 2018;Nosek & Lakens, 2014;Wilkinson et al., 2019). Proponents argue that pre-study peer review should improve study designs, thereby reducing the amount of time and resources wasted on flawed research (Kiyonaga & Scimeca, 2019;Parker et al., 2019;Probst & Hagger, 2015). Although most of what has been written on RRs is theoretical, evidence is emerging which suggests that they are meeting some of these aims, such as combatting publication bias (Scheel et al., 2021) and improving research quality (Soderberg et al., 2020).
Funding applications are very similar to Stage One RR submissions, in that they also focus on the importance of the research question and the robustness of the proposed methodology. Because of this, some research funders and journals have begun collaborating on joint initiatives that we call Registered Reports Funding Partnerships (RRFPs) -that attempt to streamline the grant and Stage One application procedures into one process (Chambers & Tzavella, 2020;Global Research Awards for Nicotine Dependence (GRAND), 2018;Munafò, 2017;Murray, 2020;PLOS One Editors, 2017). Most RRFPs entail collaboration between a funder and a journal, but some initiatives differ from this model. For example, the RRFP at the journal Politics and The Life Sciences (PLS) has been funded by the learned society associated with the journal, the Association for Politics and The Life Sciences (APLS). Consequently, we define the term RRFP as describing instances in which money is awarded to applicants for conducting a Registered Report and submitting it to the partner journal, although in some cases the opportunity to submit a RR to the partner journal is optional. These funding partnerships represent a distinct change from the norm, insofar as funders rarely encourage their grantees to publish in specific outlets, nor do they tend to communicate with the journal that their grantees submit work to.
The handful of active RRFPs have adopted a variety of models. The journal Nicotine & Tobacco Research (NTR), for example, has been involved in two partnerships, one with Cancer Research UK's (CRUK) Tobacco Advisory Group, and the other with the Pfizer-sponsored Global Research Awards for Nicotine Dependence (GRAND) grant scheme. Both partnerships have used the same design. The funder reviews grant applications as normal, but also invites applicants to opt into the partnership and submit a Stage One RR to the journal, which then reviews this submission as normal. In contrast, the journal PLOS One and the Children's Tumour Foundation (CTF) have created a workflow that integrates their processes more closely. In this scheme, CTF independently reviews letters of intent, sending successful applications, along with detailed reviews, to PLOS One for Stage One review.
The other two existing partnerships -see Table 1, partnerships four and five -are more closely integrated still. Applicants have no contact with the funding bodies, submitting directly to the journal. Both workflows consist largely of the standard RR process, but with the additional benefit that applicants receive money to help fund their research. A partnership -currently still in development -is being considered the journal Because RRFPs are new, no research has been conducted to evaluate their impact on researchers, research quality, and the efficiency of the research process, from funding to publication (henceforth, funding-to-publication process). A randomised controlled trial (RCT), in which typical funding and publication processes are compared against RRFPs, would provide a strong means of evaluating them. Given the complexity of conducting such a trial, coupled with the fact that RRFPs are so novel that very little has been written about them, we conducted a feasibility study with two main aims. First, we wanted to understand the experiences of the stakeholders (authors, reviewers, journal editors, funders) involved in the various partnership models. This aim was primarily addressed by a thematic analysis, reported elsewhere (Drax et al., 2021a). Second, we wanted to understand whether a RCT of RRFPs is feasible and, if so, how best to design such a trial (e.g., what outcome measures would be valid, reliable, acceptable, feasible, and yield high completion rates). We report the results of this second aim here.

Partnerships
We identified seven potential RRFPs by means of informal communications, web searches, and the cos.io/rr website (see Table 1). We excluded one of these, the Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative (PCFMFRI), from our final sample. They did not meet our definition of a RRFP, as studies were conducted as RRs by the authors' choice, rather than being incentivised by the research funders. This left a final sample of six partnerships.

Study design
We conducted semi-structured interviews via a video-call with stakeholders (funders, editors, authors, and reviewers) involved in existing and planned partnerships. These were used to understand how and why the partnerships were set up, how they work, ways they could be improved, and general thoughts on these initiatives. Follow-up questionnaires were also sent to some funders and editors following their interviews. These were used to understand what data are accessible and shareable that could help to measure the efficiency of the funding-topublication process. The study protocol was pre-registered on the Open Science Framework (Drax et al., 2021b). Ethics approval for the study was obtained from the School of Psychological Science Research Ethics Committee at the University of Bristol (reference number: 06022098163). Participants gave written informed consent to participate in the study using an online consent form, prior to the interview.

Participants
We interviewed 32 people, stratified by partnership and stakeholder role (see Table 2). We pre-specified a desired sample of at least one person of each stakeholder category, for each partnership (6 partnerships × 4 stakeholder categories = 24). We achieved this in all cases, except for a reviewer at the journal Politics and the Life Sciences (PLS), which we decided was not necessary. Our rationale was that it would not tell us anything unique about reviewing for a partnership because their model essentially mimics the standard RR workflow. Anyone with experience of a partnership was eligible to participate.
Participants were recruited via email. We emailed editors and funders who had been involved in these partnerships, informing them of our study and inviting them to participate in our interviews. No compensation was given for participation. All the funders and editors we contacted agreed to participate. Through these connections we were also able to invite authors and reviewers to participate. We followed up on non-responses, waiting at least a week, sending a maximum of three emails. Of the 39 authors and reviewers contacted, 14 (36%) agreed to participate, 19 (49%) never replied, 2 (5%) stopped responding to emails, 2 (5%) asked to follow up much later, and 2 (5%) declined. Author 1 4 2 n/a 1 n/a Reviewer 3 1 2 n/a 0 n/a 'Matchmaker' n/a n/a n/a 2 n/a n/a Note: The numbers add up to 39, greater than our total sample size, because some individuals are represented more than once. The editor of NTR was the same in both instances. Some of the editors at PLOS spoke on behalf of multiple partnerships. The PLS editors sat on the APLS council which decided to award money for RRs, thereby serving the role of funders. The two 'matchmakers' were representatives at the Center for Open Science. Data collection RC, JT and KD conducted the semi-structured interviews remotely using BlueJeans video conferencing software between 19 th March and 4 th August 2020. All three were present in almost every interview, with one person leading and the other two observing with their cameras switched off until the end, at which point they asked any questions they had. Interviewer characteristics can be found in the Extended data (Drax et al., 2021b).
Interviews lasted about an hour on average, occasionally limited by participant availability. Audio was recorded, transcribed, and then anonymised as much as possible. All identifiers were removed except those relevant to the analysis, such as stakeholder role and partnership. The interviews loosely followed a topic guide, tailored to the stakeholder role and any prior information pertaining to the interviewee. We piloted and refined the topic guides over 14 practice interviews. The content and sequencing remained similar throughout the piloting; changes mainly related to wording, tone, potential follow-up probes, and other reminders to help us with the interviews.

Analysis
RC performed a multi-stage analysis of the interview transcripts in order to extract information relating to the topics 'Acceptability', 'Demand', 'Implementation', 'Practicality', 'Adaptation', 'Integration', and 'Expansion' ('Limited-efficacy testing' was deemed not applicable). This set of topics was informed by the guidelines for conducting feasibility studies described by Bowen & colleagues (2009). Using these topics as a framework, RC created a comprehensive set of field notes, listing all potentially useful information from each participant's transcript, either paraphrasing or directly copying extracts from the transcripts into a separate document. Next, he organised these notes by stakeholder category. He further divided these into 'generic' or 'partnership-specific' groupings, and categorised them based on their content, e.g., 'Positives, 'Negatives', 'Overall opinions', 'Improvements'. The result was an organised list of all the main points from each interviewee, which enabled RC to identify evident trends and quantify key points. JT and MM then provided feedback on the analytical conclusions. RC gained additional insights into stakeholders' experiences from KD's thematic analysis (Drax et al., 2021a).

Efficiency questionnaires
One of the key outcome measures of interest in a trial is whether funding partnerships improve the efficiency of the fundingto-publication process (i.e., the time from grant application to manuscript publication). Between August and September 2020, we sent follow-up questionnaires to three funders and three editors from the existing partnerships. These representatives from CRUK, GRAND, CTF, NTR, PLOS One, and PLS represented all possible stakeholders with data to answer these questions.
The questionnaires were used to ascertain the feasibility of gathering data with which to measure efficiency. Specifically, we asked funders and editors whether they could report key processing dates for individual manuscripts -both RRs and typical submissions -across timepoints spanning the funding-to-publication process (e.g., the date of initial submission, sent for Stage One review, etc.) We also asked about reporting aggregated data for all submissions to the scheme, such as the total number of Stage One submissions, successful submissions, withdrawals, etc. We asked whether these data were available, easy to access, and could be shared (publicly or privately) for a trial. Two editors and one funder completed the form. The third editor did not have the time, and two of the funders did not reply. Exemplar questionnaires can be found in our Extended data (Drax et al., 2021b) (each was slightly different, tailored to the partnership workflow).

Results
We report our results following guidelines for reporting feasibility studies (Bowen et al., 2009), and include the following sections: Acceptability, Demand, Implementation, Practicality, Adaptation, Integration, and Expansion. All references to 'the intervention' should be understood to mean the RRFP process, as compared with usual practice.
Acceptability of the intervention Acceptability to funders and editors. The funders and editors involved in creating or overseeing the partnerships were largely satisfied with how they were proceeding. The majority reported that the workload to create and maintain the partnerships was not too great and did not outweigh their theoretical benefits.
[F6]: "Plus it didn't take so much of our time. There weren't big downsides on our side. I mean I had to spend quite a good number of hours with lawyers, convincing everybody and so on. Still, in the grand scheme of things, I think it was worthwhile." [F2]: "I would say that three main areas that were a bit more of a pinch point for us were probably around like the legal -just getting a contract in place -generally it takes a bit of time for that to happen. But I think, in terms of once we were implementing the scheme, apart from the going out and contacting people, it was very, it was quite minimal and low impact from a resource point of view for us to do." However, in all cases interviewees considered it too early to make concrete judgements on the overall success of the partnerships, as only one author had made it all the way through to publication at the time of interview.
[E8]: "I mean it's worked fairly well. It's very early to tell, and the problem with experimenting with Registered Reports is that I think we've only had one study completed… So, it's too early to draw any robust conclusions from that." Therefore, although the present evidence is that those who created partnerships have found them acceptable, evidence may emerge to the contrary.

Acceptability to authors.
It is similarly difficult to draw strong conclusions regarding acceptability of RRFPs to authors, because only three of the eight authors had completed Stage One peer review at the time of interview, and only one, A8, had completed the entire process (i.e., published a paper through a RRFP). A8 had had a positive experience; they would apply to a similar scheme again, provided that the study was a good fit for the journal. They had even recommended the initiative to someone else, who subsequently took part in a later round of the RRFP scheme. They also noted a few benefits which were specific to RRs: [A8]: "Well, it was useful to get the reviews at the start.
And it was helpful to know that, once it was approved, Stage One was accepted, it was good to know that we were kind of likely to get a publication in [the journal]. Obviously not guaranteed but it was, yeah, that was quite reassuring." These benefits of pre-study peer review and IPA-guaranteed publication were incentives for many authors. A few authors also mentioned how money operated as an incentive.

[A5]: "Basically [my supervisor] said you won't have to worry about getting into a journal, it's already pre-approved and we get money out of it as well, obviously that's a big thing, we get funding and so, yeah, [they were] very, very keen on the idea to have heaps of money as well as a pre-approved publication."
For authors in general, whether the journal was a 'good fit' for the study appeared to be the most important consideration governing acceptability. When appraising journal fit, authors often considered not only the research topic, but also the results obtained. This could be a concern, as several authors mentioned they might be motivated to withdraw their submission, should they obtain results which they felt could be submitted to a higher impact journal.
[A1]: "I suspect that, if we had a positive finding and we think that the quality of the study is high, the temptation always is to submit it to something like [a higher impact journal] rather than to [the partnership journal], to be very honest with you." Two authors discussed 'high impact' in terms of both the scientific impact their work would have, as well as the career advantages of prestigious journals. They noted that getting publications in high impact journals is a key research metric, linked with "how we survive in science" [A6], because favourable funding and hiring decisions depend on them. This dynamic did not worry funders, as they only cared whether manuscripts got published, not where. However, the prospect of manuscript withdrawals had concerned some editors, as they saw it as inevitable, but something which fundamentally undermines the reason for doing RRs.
[ Two authors from one scheme withdrew their manuscripts after receiving IPA at the journal, one of whom subsequently resubmitted. Both had opted into the scheme thinking that the journal would publish their Stage One submissions as separate publications, as it is common practice in their field of clinical research to publish study protocols separately. After realising that their Stage One submissions would not be published in the journal, both withdrew due to logistical necessities regarding having these protocols published. Indeed, all four authors participating in this scheme would have preferred the journal to have published their Stage One protocols separately. These withdrawals are not inherently a concern for the acceptability of partnerships. Rather, they demonstrate the need to clearly inform prospective applicants about these new processes and guidelines. Moreover, they highlight that an understanding of the norms and wishes of researchers from the given field is vital for a partnership's success.
In summary, the evidence suggests that RRFPs are generally viewed favourably, but that it is premature to draw strong conclusions. The main threat to acceptability of partnerships was the spectre of manuscript withdrawal; acceptability could increase if this is addressed, for instance through RRFP schemes that take advantage of journal-independent peer review in which one centralised RR review process leads to automatic acceptance by multiple journals (see Peer Community in Registered Reports: https://rr.peercommunityin.org/about/about).
Demand for the intervention Demand from funders and journals. The demand from funders and journals to continue and/or start new partnerships is mostly positive. Three of the five partnerships will continue for future rounds. Of the remaining two, F6 said that "The only shame was that it was the last year of the programme.", referring to the fact that the grant programme associated with the partnership was ending. The other funder had not definitively discontinued the partnership, but rather had paused it for two reasons. First, they were restructuring internal funding mechanisms. Second, they wanted to survey author opinions on the two pilot rounds, before committing to anything further. In summary, extraneous logistical factors and the desire for a formal evaluation were the reasons behind two partnerships not definitely continuing.
Demand from applicants. The demand from applicants is somewhat mixed, but generally positive. Aggregating optin rates for all rounds across the two NTR partnerships, 81% (43/53) of grant applicants were willing to participate in the scheme and submit a RR to the journal. Similarly, E4 at PLS told us, "… I mean, we've had to turn down proposals, so it's not as if we're having to scrim to get proposals. So in that sense I guess it's been positive". In contrast to this positive outlook, the funder CTF estimated that the number of grant applications they receive has dropped by roughly a half since mandating the adoption of RRs. Whether this is due to the new requirements is purely speculative, however, and they have no intention to discontinue the partnership.

Implementation of a trial
This section addresses three key factors regarding the conceptual feasibility of implementing a trial.
A successful trial will rely on successful partnership implementation. There have been a number of different partnerships, each successfully implementing a new and unique workflow (see Figure 1 for simplified workflows for each existing partnership). This has required the coordination of interests and processes across independent organisations. The different models resulted from the varied needs and interests of those involved. This suggests that the partnership concept is flexible and robust in different contexts. While there have been lessons and scope for improvement, there has been "nothing really major" [E5]. This bodes well for a trial, as the adequate implementation of partnerships will be necessary to avoid confounding the outcome measurements.

Randomisation is logistically feasible -how it occurs will depend on the model adopted.
Participant randomisation appears to be logistically feasible, at least for certain partnership designs. This conclusion is based on the fact that, in their partnership with NTR, CRUK randomly selected grant applicants who they invited to opt into the partnership (meaning, invited to submit a RR to NTR). CRUK's willingness and ability to do this in a way that did not bias funding decisions serves as evidence that this is both possible and acceptable to a funder. However, their random selection process did not operate in exactly the same way that randomisation would in a trial. CRUK randomly selected five applicants and invited them to opt into the partnership. By contrast, in a trial, invitees who agree to take part would have an equal chance of being randomised to one of two arms (intervention or usual practice). Consequently, these pilot schemes offer no evidence about whether it is acceptable to applicants to be randomised, either to the RRFP condition, or control condition.
Randomisation has to be tailored to the specific partnership model because different models employ varied grant and RR submission processes. Careful consideration as to where randomisation should occur is critical in order to avoid unnecessary attrition. We present here two potential ways that participants could be randomised as an illustration of feasibility. Note, because a trial may involve multiple partnerships with different designs, the strongest design may involve stratified randomisation within partnerships. Workflow A is modelled on the two partnerships involving the journal NTR. Workflow B is modelled on the CTF-PLOS One partnership.
In Workflow A (see Figure 2.a) there are two evident points at which participants could be randomised: at the point of application, or after funding has been awarded. CRUK randomly invited applicants to 'opt in' before making funding decisions. This limited the number of people who went through the scheme, as only 3/7 of those who opted in were then awarded funding.
[ In light of F2's reflections, we recommend randomising participants after funding decisions have been made. However, we believe the best time to invite people to participate in the trial (i.e., opt in) is at the point of application. First, this will yield the largest sample with which to understand how acceptable the trial is to applicants. Second, it will give applicants greater time to consider and learn about the RR scheme.
Workflow B (Figure 2.b) involves modifying the more integrated workflow of CTF and PLOS One. In the current model, CTF reviews letters of intent, passing successful applicants to PLOS to submit a Stage One RR. IPA and funding are then awarded simultaneously for successful submissions. In this workflow, randomisation would have to occur before funding is awarded, because there is only one main round of review.
The feasibility of measuring outcomes: efficiency, research quality, and researcher attitudes. To measure how RRFPs affect the efficiency of the funding-to-publication process, it will be necessary to calculate the time various aspects of the research projects take. This will include the time from funding application submission to publication, and ideally also subsidiary measures such as the length and number of rounds of peer review. We emailed three editors and three funders a questionnaire on whether it would be possible to gather data to measure these timelines. Two editors and one funder responded. The unanimous response was that dates for almost every stage of an individual submission are easily accessible. Similarly, summary data for the whole scheme -such as total number of submissions -are also easily accessible. Two responders felt that most, if not all, of these data could be shared publicly, whereas the third was more hesitant to claim that granular dates for individual submissions could be publicly shared. Nevertheless, they believed that the key dates outlining the process, such as Stages One and Two submission dates and full manuscript publication, could be publicly shared. In summary, it appears feasible to collect all necessary dates for the efficiency measurement and may be possible to collect other dates that would add to a richer analysis of efficiency. Figure 3 presents a flow diagram of the RCT process and where efficiency could be tracked.
By contrast, our results reveal only limited insights into the feasibility of measuring research quality. We asked interviewees how they thought the partnerships may have affected research quality, in an effort to identify both what aspects of quality a trial ought to measure, and whether these aspects were measurable in any way. Consistent with at least our first aim, interviewees were often able to speculate as to how the partnership may have influenced quality. Most frequently mentioned were the benefits of additional pre-study peer review, as well as the increased level of consideration and rigour Similarly, speaking for both partnerships involving NTR: [E5]: "I mean, there's certainly nothing I could say at this stage, apart from the reasons why one might expect to see higher quality outputs from research that has been through this process." As more authors go through the entire process, it will be worthwhile to conduct follow-up interviews with stakeholders to glean their insights on how the partnerships may have influenced other important aspects of quality. This will ultimately help to design a better measure of quality for a trial. Timepoints 1-4 indicate points for which dates could be recorded and shared. They would be same on both arms. Three additional timepoints for which data would be available are noted on arm 2. These could be recorded to better understand the distribution of work across the RR process. Points at which attrition is possible are labelled in red. This diagram is modelled on CRUK/NTR workflow.
Lastly, we identified an unanticipated yet crucial third outcome to measure in a trial: researchers' attitudes towards these new initiatives. Speaking with funders and editors in interviews, it became clear that one of their primary concerns is how these partnerships affect and are received by researchers and wider communities. Authors were typically well equipped to articulate their views on this, although in some cases memory faltered due to the time lag between the interviews and the experiences we asked them to recall. Consequently, we believe that gaining a comprehensive understanding of these attitudes is both feasible and ought to be one of the central measures in a trial.

Practicality of a trial
Although a RCT may be feasible in principle, the scale of throughput and time required may mean that a trial is not feasible in practice. Sample size and timescale will be key considerations of practicality. Crucially, the trial will involve tracking submissions from grant to publication. As seen in these pilot partnerships, projects take several years to complete. Over such a period, delays and attrition are inevitable.
Many of the research studies in these partnerships experienced delays. In addition to the disruption caused by coronavirus disease 2019 (COVID-19), we heard of natural disasters, recruitment issues, novel study ideas, conflicts of priority with other projects, and other circumstances delaying or derailing studies. In one instance, an author reported being delayed by a reviewer who did not fully understand the RR review process. At Stage Two review the reviewer wrote "two paragraphs on how much he still hates our theory and doesn't understand it… and then [brought]  Beyond this, however, there was no evidence to suggest that delays were caused by the RR format. These inevitable but unpredictable delays will lead to greater-than-anticipated variability in research completion times, adding noise to the efficiency measurement. Although we expect such variability to be similar across both arms of a trial, mitigating this will require greater sample sizes than one might expect when looking at projected project completion timelines alone.
In addition to delay, one must also consider attrition when calculating sample size. We encountered attrition most commonly in the form of author withdrawals. This can likely be minimised by better communicating the RRFP procedures, and by understanding the desires of applicants in that field. Still, our results should be interpreted cautiously, as further withdrawals could happen among the cohort we interviewed. Notably, several authors mentioned the possible attraction of withdrawing to publish in a high impact journal if their final results were striking enough. Attrition rates are difficult to assess from our feasibility study, yet attrition could greatly impact the success of a trial. Therefore, it would be wise to learn more about researchers' attitudes and the various reasons they might consider withdrawing. A full trial should also measure rates and reasons of withdrawal as a secondary outcome.
In summary, the central question of practicality is that scale, including both sample size and time, cannot be overlooked. It is difficult to estimate the potential length of a trial at this stage, due to insufficient data on project completion times, as well as differences in project timelines across partnerships. These differences across partnerships are due to the varied scope of projects within the partnerships' remit, as well as the divergent workflows of the partnerships. Consequently, key questions in a pilot RCT will be to understand partnership-specific timelines, as well as the number of rounds required to achieve sufficient throughput, will be Adaptation of the intervention As already mentioned, multiple RRFPs have shown that the concept is adaptable and can be successfully implemented using a number of designs and in different contexts. This begs the question: how much can the partnership concept be adapted before it no longer meets the inclusion criteria for a trial? This question is key, because the way in which partnerships are defined and implemented could affect the conclusions of a trial. The definition needs to be specific enough for the term RRFP to mean something, but also generalisable enough to have external validity to the variety of RRFP formats that exist (or will likely exist). We have used the principle that any initiative in which money is awarded to applicants for doing RRs, linked to a particular journal, can be considered a partnership; however, the design of a future pilot or RCT should carefully consider this question in light of the criteria above.

Integration of the intervention
The ease with which new partnerships can be integrated into organisational contexts will be contingent on the organisations themselves and the models they choose to adopt. For example, the two partnerships involving NTR required very little change at both the journal and funder. For the funders, the partnership added only two touchpoints with applicants onto their existing workflows: the invitation to opt in, and the handover to the journal. These partnership models were deliberately "light touch" [F2], allowing both parties to retain both their autonomy and existing procedures.
By contrast, CTF and PLOS One created a workflow which streamlined the application procedure, integrating the decisionmaking processes of the two organisations more closely. They did this while still retaining autonomy, which was crucial for both parties.
[E3]: "I mean, the editorial decision is independent of the funding decision. Obviously, we're not going to have a word on what they're going to fund, and likewise they're not going to have a say in our scientific assessment on the journal's side." They achieved this by having CTF review the letters of intent (LOI), passing successful applications to PLOS One, who independently conducted Stage One peer review. By passing the LOI reviews to the PLOS reviewers, as well as handling all correspondence with applicants, CTF were able to remain involved in the Stage One review process. If PLOS One decided not to accept an application, CTF "… still had the opportunity to fund anyway." [F1], thereby retaining the ultimate final decision on their end. This workflow resulted in a more streamlined application procedure, containing just one main review (the Stage One review), as opposed to distinct grant and Stage One reviews in the NTR partnerships. Consequently, this model required the creation of entirely new workflows, and greater integration between organisations.
The level of integration required to implement RRs at the journal is also important to consider. One of the most common difficulties editors faced was that their online manuscript handling infrastructure did not cater for RRs, resulting in manual handling and occasional delays. At one journal, "initial proposals and reviews were all [sent] straight to the managing editor's individual email". This was clearly inefficient, even at a very small scale. It will be vital for journals to have appropriate manuscript handling infrastructure for RRs in order to manage the volume of submissions necessary for a full trial.
Expansion of the intervention A number of interviewees mentioned the importance of advocacy and education as RRs are introduced into new fields.
[M2]: "It's kind of all, I used the word hearts and minds before, I think. But we were trying to figure out how to reach the most members of, identify the community, reach them, speak to them, and convince them about the appropriateness of this." Several of the current partnerships were set up in research fields in which RRs were not widely known. Their uptake indicates that unfamiliarity of the RR format in a given field is not necessarily a barrier to successfully introducing a RRFP. Based on one author's interview, there is reason to believe that partnerships might see better participant uptake in fields where funding is harder to come by.
[A7]: "You know, I don't have any money! I have all these ideas and I have no money, even though I apply to grants all the time." By contrast, some noted that partnerships are not suited to all research areas and funding contexts.
[E8]: "I don't think that this is going to be applicable to any and all grant applications that are out there. Because of the requirement for predetermining a number of things, it's very difficult to do that on a multi-year grant about something that is going to be a very ambitious and large and diverse study." Partnerships may be more difficult to implement in disciplines where the norm is to fund research programmes involving a series of multiple projects, and/or more exploratory research.

Discussion
In summary, many of the necessary aspects for a RCT of RRFPs seem achievable. In practice, however, the volume of submissions going through existing partnerships is too low to support a full trial. Scaling up existing partnerships or creating new ones will therefore be necessary. While we have seen evidence that funders and editors are interested in forming new partnerships, it remains uncertain when, if at all, this might happen. Below, we elaborate on some central lessons from our study, as well as questions which remain unanswered.
Three measurable outcomes: efficiency, research quality, and researcher attitudes We found that it would be feasible to measure the efficiency of the funding-to-publication process, mainly by requesting data from funders and journals. This gives us confidence that this key primary outcome measure could be measured in a trial. Nevertheless, because the funding-to-publication process is so long and highly variable, a large sample will likely be required in order to achieve adequate statistical power.
Although our data could not fully answer whether it would be feasible to measure research quality in a trial, a recent study demonstrated one way this could be done. Soderberg & colleagues (2020) asked academics to peer review a matched pair of published RRs and non-RRs. They found that reviewers were able to evaluate papers on 19 measures of research quality, such as methodological rigour and overall paper quality. Their implementation of these methods attests to the feasibility of measuring research quality (or at least the perception of research quality) in a similar way for a trial.
Researchers' attitudes towards these new initiatives emerged as a third and very important outcome measure. The heart of funders and journals' motivations to form partnerships was ultimately to change and improve aspects of how researchers work. Reasons included developing awareness of RRs in different research communities; incentivising the uptake of RRs; facilitating RR adoption through financial awards; improving the efficiency of research; and reducing burden on peer reviewers. Consequently, understanding researcher attitudes towards partnerships should be one of the central goals of a trial. Attitudes should arguably be tracked over time, providing surveys at various intervals throughout the whole process. This should avoid issues with memory recall. It would also help to uncover and promptly resolve any issues in the partnerships. This in itself is important, as funders and editors quite often mentioned minor teething problems in implementation, which were more reflective of the infancy of their partnership workflow, than the weaknesses of the partnership concept itself.
The role of multiple partnerships in a trial A trial may require involvement from multiple partnerships in order to get enough throughput to power the outcome measures. If the partnerships we studied are any indication, each partnership is likely to differ slightly from the others in its design, determined by the desires and logistical circumstances of its creators. This may make it difficult to compare 'RRFPs' versus 'the norm', because combining data from a number of heterogeneously designed partnerships would increase variability in the intervention arm. However, it would also enable comparisons across partnerships.
Contrasting the effects of different partnership designs would enable the strengths and weaknesses of various models to be compared. Given just how diverse RRFP workflows can be, there are intuitive reasons to expect different designs to affect the three main outcome measures in various ways. For example, models that preserve existing workflows at the funder and journal -having two separate reviews and a handover phase -are likely to be less efficient than models which adopt a more integrated workflow. Conversely, having two independent reviews may confer greater benefit to the overall quality of the research than a single, integrated review. Again, the feasibility of meaningfully drawing comparisons between partnerships will hinge upon the throughput achievable by each uniquely designed partnership. This is therefore an important consideration for those considering involvement in a trial.

Unresolved questions
In this section, we first discuss outstanding questions relating to the review and randomisation procedures in a trial. Then, we consider how more evidence is required to give a full account of RRFP acceptability and demand.
Blinding reviewers to trial participation. As much as possible, people involved in funding or publication decisions should be blinded as to whether the submissions they are reviewing are part of a trial. Without blinding, the quality and outcome of the reviews could be biased, as reviewers may have their own predispositions for or against RRFPs, or expectations about whether RRFPs affect quality. It is highly desirable that these opinions of reviewers do not affect the decision process, both for the integrity of the trial, and for ethical considerations.
Interviewees from the funder CRUK mentioned how important it was not to introduce bias into the grant review process. For them it was crucial that the grant review board were unaware which submissions were involved in the RRFP pilot scheme. They were able to ensure blinding, as all submissions had the same format, regardless of whether they were invited to participate and whether they opted in. By contrast, blinding is trickier for journal review, and for funding decisions in some other workflows where decision makers see submitted manuscripts; if reviewers are aware of the nature of the trial, it will be obvious which treatment group the submission was assigned to, due to the clear differences in format between RRs and typical submissions. In these cases, the only way to ensure unbiased reviews is to blind them to the fact that the submissions are part of a trial at all. To see this, consider the example workflows shown in Figures 2a and 2b. Randomisation is set to occur before key review stages, such as Stage One and grant review. A reviewer who is aware of the trial will therefore know which arm the submission belongs to, based on whether it is a Stage One RR or grant submission.
Within such workflows, it is therefore vital to determine how funding or publication decisions could be made without reviewers, funders, and editors being aware of trial participation.

Acceptability of randomisation.
Although it appears conceptually feasible to randomise applicants to either arm of a trial, whether it would be acceptable to stakeholders remains unknown. In the example workflows, applicants would be invited to participate before knowing which arm they would be in. The uncertainty of having an even chance of being asked to conduct either a RR or a 'traditional' manuscript may inhibit trial participation. Without pilot data, it is difficult to say whether and to what extent this may be so. Similarly, we have only anecdotal evidence to suggest that randomisation is acceptable to funders. While CRUK were willing to randomly select applicants for the pilot partnership scheme, it is unclear whether they would be willing to randomise submissions earlier in the process, which is how randomisation would ideally occur in a trial. Consequently, a preliminary step when designing a pilot should be to approach funders and journals about their willingness to randomise applicants.
Insufficient evidence. Because our interviews occurred at a very early stage in the partnerships' lifecycles, they gave a limited insight into the whole process. As discussed, only one author we interviewed had completed the entire publication process, while six had yet to go through Stage Two peer review. For the Flu Lab initiative, no authors were interviewed as none had finished Stage One review. Consequently, funders and editors were hesitant to say anything concrete about the effects of the schemes, often referring to the theoretical and expected benefits in the absence of data. Our account of acceptability is therefore only a preliminary sense. A more nuanced picture will be possible to create as more authors complete Stage Two review, including shedding light onto the frequency of withdrawals and other forms of attrition.
In a similar vein, our account of demand lacked information on why some researchers chose not to opt in, as we did not interview applicants who declined to participate in RRFPs. Understanding peoples' reservations towards partnerships would highlight well-founded concerns, misconceptions, potential improvements, and barriers which are inhibiting uptake. This information would be particularly useful for the CTF-PLOS One partnership, in order to better understand whether (and how) reservations about RRFPs might have caused the observed decline in applications they have received. Knowing what turns potential participants away from RRFPs would allow partnerships to address these issues through messaging and outreach, and consequently increase participation in a trial. Moreover, it might uncover contexts in which new partnerships are unlikely to work.

Conclusion
In conclusion, the results of our study suggest that a RCT of RRFPs is feasible, at least in principle. The partnership concept seems worthwhile to pursue further and is adaptable to different organisational needs and various research disciplines. Moreover, the various partnership models appear to all be largely acceptable to their stakeholders. In a trial, all three primary outcomes of interest appear to be measurable, and randomisation could conceivably occur in a number of ways, although only one (imperfect) randomisation has yet been tried. To avoid bias, a trial should ensure that those involved in funding and publication decisions are blinded as to whether submissions are part of a trial. The key unknown is whether sufficient throughput could be achieved to statistically power the desired outcomes. A "consortium" model, whereby multiple journals and/or funders could work together within a single partnership could help to address this, although it would also introduce new challenges (see Box 1). The success of a RCT will therefore hinge on institutions forming new partnerships and agreeing to participate in a trial.

Box 1. A Registered Reports Funding Partnerships (RRFP) consortium involving multiple journals and funders
Although none exist at this stage, funders and editors alike expressed interest in the idea of a consortium, whereby multiple journals and funders form a single partnership.
Having multiple journals within a partnership would allow applicants more choice in terms of their eventual target journal, and would allow more types of study and research question to be accommodated within a partnership. It could also in principle make it easier to achieve the sample size necessary for a trial.
Discussions of consortia took up a considerable amount of time during our interviews. A possible barrier to starting a consortium is the prospect of finding a singular model or submission flow which would be agreeable to all parties.
If a single framework cannot be agreed upon, the alternative is to have multiple options within the consortium, adopting varied workflows, divergent instructions, and potentially inconsistent terminology. This would add unnecessary layers of complexity for applicants, as noted by one editor in our interviews.
Given the importance of user experience for applicants, simplicity and consistency in implementation will likely be vital for a consortium's success.

Underlying data
The study data for the interviews are hosted on the University of Bristol's online data repository (data.bris) as controlled data at: https://doi.org/10.5523/bris.1m38wyz9gvzo52i2 kpecr8w6kb.
It was essential for our analysis to link each interviewee with the partnership in which they participated and their role within it. Therefore, this stringent level of data control was chosen because some interviewees may be identifiable from their transcripts.
To access the data in data.bris, bona fide researchers will need to secure a Data Access Agreement from their host institution.
With their host institution's approval, a request for access will be judged by the repository's Data Access Committee.
More information about Controlled Data access requests is available at: https://www.bristol.ac.uk/staff/researchers/data/accessingresearch-data/.
This project contains the following extended data: • Protocol_RRFM_v1.0.pdf (the study protocol) • coded-extracts-sample.csv (examples of codes and their relevant text.) • interviewer-characteristics.csv (characteristics of the three interviewers, KD, JT, and RC, such as their credentials, occupation, gender, etc.) • The 'Code' folder contains 4 files: ○ README-code.txt (instructions of how to use and understand the code) ○ 0-nvivo-export.txt (directions for exporting from NVIVO) ○ 1-collate-coded-text.R (code to collate files exported from NVIVO into a single csv file) • The 'Efficiency Questionnaires' folder contains 2 files: ○ Efficiency_questionnaire_for_funder_CRUK_ GRAND.pdf (the blank questionnaire sent to the funders at CRUK and GRAND, used to understand what data are accessible and shareable that could help to measure the efficiency of the funding-to-publication process.) ○ Efficiency_questionnaire_for_journal_PLOS.pdf (the blank questionnaire sent to editors at PLOS, used to understand what data are accessible and shareable that could help to measure the efficiency of the funding-to-publication process.) • The 'Ethics' folder contains 3 files: ○ consent-form.pdf (the consent form used to obtain informed consent before the interview.) ○ debrief-sheet.pdf (the debriefing information given to participants after the interview.) ○ participant-information.pdf (the participant information document given to participants before the interview.)