The Luck of the Draw: Wellcome's Institutional Fund for Research Culture

Wellcome's Institutional Fund for Research Culture (IFRC) closed call is an invite-only grant call in 2023. It is a departure from Wellcome's previous methods of institutional funding, providing institutions with up to £1m of grant funding to take on ambitious projects that advance research cultures and research environments that are equitable, diverse and supportive. Recognising the broad range of topics and ideas for advancing positive research cultures, IFRC is the first ever Wellcome-funding scheme to use partial randomisation to allocate funding. Applications were grouped by a funding committee into three categories (Gold, Silver and Bronze), with the applications selected for Gold being directly recommended for funding and all applications in the Silver group being set for funding by the randomiser. Applications grouped into Bronze were not funded. To ensure that this activity Wellcome's principles for openness and transparency, we have included the Python script for the call here. IFRC comes when efforts to fund positive and inclusive research cultures are mainstream. Similar efforts to support research culture activities at scale have come from the Research England Development (RED) Fund, and the next iteration of the Research Excellence Framework (REF 2028) will also mark 25% of the assessment criteria for people, culture and environment. IFRC was not designed with the national picture in mind but is a testament to Wellcome's values as an inclusive funder. The range of projects and geographies that IFRC will fund is exciting.Still, it also threw up several interesting social and philanthropic research questions we want to explore in future community-facing activities following the call. We hope that findings from IFRC projects become a valuable resource for institutions wishing to improve their research cultures and a catalyst for future change and discussion within the sector that makes academic careers more inclusive.


Amendments from Version 1
Updated text to include details about the proportion of applicants in each grouping and to note that further unnamed institutions will receive some of the funding on behalf of the named institutions.
We have added additional authorship and acknowledgement details.
Any further responses from the reviewers can be found at the end of the article

Disclaimer
The views expressed in this article are those of the author(s).Publication in Wellcome Open Research does not imply endorsement by Wellcome.

What is the Institutional Fund for Research Culture?
Wellcome's Research Environment (RE) approach cuts across Wellcome's strategy i (Figure 1), considering how Wellcome funds research.It is grounded in three values: • The research that Wellcome supports is strengthened by being ethical, open, and engaged.
• The people Wellcome funds thrive in equitable, diverse, and supportive cultures.
• Wellcome is an innovative, efficient, and inclusive partner and organisation.
Wellcome's ongoing commitment to encouraging positive and inclusive research cultures ii and communities where everyone can thrive caters mainly to the second value.We utilise Wellcome's money, values, and influence to change the academic research sector, enabling the researchers' Wellcome funds to thrive in positive and inclusive research environments.
Wellcome has a history of using "strategic" funding to support universities and other research institutions across the UK with activities such as supporting early career researchers, diversity and inclusion and public engagement iii .However, our approach in the past to allocating the funding was not strategic, lacked specific outcome measures and favoured institutions with a strong track record of gaining Wellcome funding (you probably know who they are).This got us thinking.
• Can we fund research culture activities at scale?
• How can we do this equitably, recognising that every research environment differs with various challenges, contexts, and available resources between institutions?
• How can we influence further positive developments in the research culture space, build trust and partner with the community to create solutions?
And so, the Institutional Fund for Research Culture (IFRC) was created.
We invited 43 institutions from the UK and the Republic of Ireland to apply for grants of up to £1m over two years.The invited organisations had to have held at least ten Wellcome grants in the last five years to be eligible.Funders, pharmaceutical companies, and other not-for-profit organisations were excluded.This shift from Wellcome's previous approach meant we were targeting research institutions most likely to conduct research aligned with Wellcome's strategic priorities iv and hosting researchers who could apply for Wellcome funding in the future.It also extended our reach of eligible institutions to all parts of the UK, which, viewed from an equity standpoint, broadened the opportunity for institutions with fewer resources to progress their research cultures as future centres of excellence.

How did we review the applications?
Applications were assessed according to the following criteria, guidance for which was shared online: • Identification of the barriers to a positive research culture and the potential to overcome them (25%) • Breadth of impact (25%) • Evaluation (20%) • Institutional Commitment (15%) • Team skills and experience (15%) The review process we used for this scheme is summarised in Figure 2. We recruited a funding committee of internal Wellcome colleagues and additional experts from other funders, research centres and higher education institutions in the UK, US, and mainland Europe.The committee scored applications as typical for a usual funding process, but rather than a ranked list, applications were sorted into groups of Gold, Silver, or Bronze, as agreed by the committee.Gold applications were exceptional and the top priority for funding.Applications in the Bronze group were not considered a priority for funding.Silver applications were fundable but with some risks or minor flaws.A randomisation (lottery) process using a Python script (a high-level programming language), the first of its kind in Wellcome history, decided which applications in this Silver group to fund.For transparency in this process, the code used for the randomisation can be accessed at the end of this article v .
From the outset, we knew how difficult the challenge of assessing research culture would be.An institution that is typically well-funded could have more resources and tools to understand its research cultures than an institution with less funding.This could bias our process towards institutions with 'bigger' names or to projects claiming research culture change at "scale" but with not necessarily the most inclusive, brave, transformative, or thoughtful ideas.Now, suppose you have two institutions that got the same score, for example.How could iii Wellcome, Institutional Strategic Support Fund: https://wellcome.org/grantfunding/funded-people-and-projects/institutional-strategic-support-fundiv Wellcome, What we do: https://wellcome.org/what-we-dov Boylan-Toomey, J., Smith, D. and Goodwin, Z. ( 2023) "IFRC Award we decide between Institution A's project looking at research leadership capabilities and Institution B's project looking at anti-ableism?Our intention for IFRC was not to tell the sector what "Wellcome" finds essential for positive research cultures.It was to help the sector find the answers for themselves, otherwise known as "equipping the problem solvers" vi .
Within an application process, there are time constraints and limits to the information we could reasonably collect to base our assessment on.We also tried to be thoughtful and mindful of the questions that exist in the sector for how to devise suitable measures for research culture that consider context, the "distance travelled", vii,viii and that this was a closed call to institutions with an already 'warm' relationship to Wellcome.
In this regard, we chose partial randomisation as the fairest funding mechanism, recognising that there is no "right" answer for research culture but plenty of solutions.
Who did the funding go to?
The committee grouped 16% of applications into Gold, 56% Silver, and 28% Bronze.We funded 24 institutions in total.These are shown in

Organisation
collaborative grant between The University of Glasgow, the University of Edinburgh and the University of St Andrews was the only joined application of the call, some of the awardees will share their funding with institutions that were not invited to apply.These further beneficiaries are not named.

What we hope to achieve with IFRC
We are excited about the range of projects and geographies that IFRC will fund.We hope the funding is viewed as another strong example of Wellcome using its money, values, and influence to affect positive change in the external research sector.For example: • Money: Investment in diverse research culture ideas and institutions helps Wellcome and the research sector develop and share evidence about approaches that work or don't.The projects benefit researchers who may apply to Wellcome for funding of their own in future.
• Values: Wellcome's brave, transformative, inclusive, and thoughtful use of partial randomisation adds to the evidence base for where it makes sense to use this funding mechanism strategically.In this case, using a randomiser helped remove bias and the need for consensus in the final recommendations.We would probably not find the use of a randomiser appropriate for an open-mode grant going to an individual where Wellcome wants to have more of a say on what it considers the most 'bold creative and high-quality ix ' applications to be.Nonetheless, a trial comparing the success of applicants selected by a standard process versus a randomiser would be fascinating, as discussed in this article x and this blog xi .
• Influence: The projects succeed and influence more positive research culture and practice change in the UK and beyond.Wellcome enables idea and knowledge sharing as a partner through research culture community events, report-writing, and tool development, which transmits good practice beyond the lifetime of the grants.
The call has also raised several exciting social and philanthropic research questions we want to explore, evaluate, and engage the sector with.We will invite everyone who applied to join a "community of practice" (CoP) over the next two years.Within this community, we want to examine several themes and questions, such as: • How do we promote collaboration, sharing and transparency of different ideas and approaches within the research culture space?
• What does success look like for the call, and how should we evaluate this?
• What commonalities and cross-cutting themes exist between the awarded applications?
• What can we learn from institutions about the existential challenges of implementing culture change at scale?
• When the funding finishes, what has worked and what hasn't, and what, if anything, does this say about the decision-making we did and didn't do for this scheme?
• How can we utilise the outputs of this call to create a permanent shift in institutional research cultures for the future?
We look forward to sharing more soon as we build this exciting programme.If you have any questions, please reach out to researchenvironment@wellcome.org 1.The authors wrote: "our approach in the past to allocating the funding was not strategic, lacked specific outcome measures and favoured institutions with a strong track record of gaining Wellcome funding"Is this the view of Wellcome or the authors' assessment of some areas of opportunity for the funding organisation?If the former, a reference would be helpful to add.If the latter, it may be important to add detail of the evidence that supports the statement.This is because lacking specific outcome measures in funding allocation would be a vital area of development for an organisation, and I find it difficult to understand how these were not in place.
In addition, one could argue that publishing figures of how funds were distributed in terms of number of awards/grants, number of researchers, etc. are a form of outcome measure, which are available for most funding organisations.

2.
On the eligibility criteria for the IFRC and fairness.What is the likelihood that institutions with fewer resources have had at least ten Wellcome grants in the last five years?Might this criterion have inadvertently excluded some institutions from being eligibility for the call?The use of partial randomisation was geared towards increasing fairness.Was this a consideration throughout the decision process?Were all institutions that met the criteria of at least 10 Wellcome grants in the last five years invited?Or was there a selection process?Providing information about the invitation process would answer some of the questions.

3.
A number of factors supporting the adoption of partial randomisation to allocate research funds have been discussed, some of them being about removing potential bias that may affect decisions and others being about increasing efficiencies.The reasons to use partial randomisation provided in the text ("In this regard, we chose partial randomisation as the fairest funding mechanism, recognising that there is no "right" answer for research culture but plenty of solutions") cover the aspect of increasing fairness, which can be argued addresses bias.For the second factor, was there any insight on the use of the IFRC Award Allocator addressing time constraints and increased efficiencies?This would be of great interest considering that the process reported a more traditional review of applications that required a committee scoring and sorting proposals.Furthermore, some additional information on the criteria for scoring and sorting would be valuable as it is part of the partial randomisation process, and would need to be reported as was the python code.

4.
Referencing differing views and opinions.Random allocation in funding has been implemented by various organisations (e.g., the Health Research Council of New Zealand (HRC), the Volkswagen Foundation in Germany and the Swiss National Science Foundation).The article by Liu et al ( 2020) 2 report the views of researchers on the use of this method at (HRC), and as we could anticipate, not everybody agrees with its use.As such, it would be good to see the way that the IFRC addressed the challenges faced when implementing this method, as I believe there must have been diverse opinions in the use of partial randomisation.I personally would find it highly valuable to learn the opinion of researchers, funding committees and the general public on the use of this method and how it is appropriate in some funding calls for reasons of fairness and efficiency.Furthermore, it would be of interest to learn how the method ensures that the best research (from a scientific, relevance, innovation, and potential for delivering benefit points of view), is still supported when using partial randomisation.I recognise that the first stage in selecting awards addresses these areas, but it would be helpful to have more details on this reported in the letter.
I look forward to reading further reports, letters and articles on this topic.This is an exciting initiative and the authors are to be commended both on launching the initiative and in particular for publishing the methodology and sharing their experience.This is excellent to see.

References
The rationale for the Open Letter is clear and in sufficient detail 1.

2.
The articles cited by Gadd et al (vi) and Curry et al (viii) provide useful background and the opening paragraphs of the letter set the scene for the principles and rational for the call.These are clear.However, this is not as wide a contextual setting as might be helpful.In setting out how they might draw conclusions from this very welcome experimental approach some additional context would be helpful to: illustrate both the nature of culture as a concept and differences in how this is experienced.It is important to set out the logic for the role that policy and funding may play in change (both positive and negative) and to understand how reviewers were briefed, trained and supported to deliver this vision; there is no justification as to why the call was invite only.
○ frame how learnings from related Wellcome Trust investments e.g.institutional translational partnerships, PhD programmes and by those of other funders might have informed the design of the call and the overall approach including for example work included in links 1, 2, 3 and 4 at the end of this report.
○ acknowledge that the system in which this call is operating is inherently dynamic and complex, assumptions are hard to generalise.The operating context across the UK is not equal, nor as the authors highlight is it equal at an institutional level.For example: Whilst the authors are correct that 'typically well-funded (institutions) could have more resources to understand their research cultures the assertion that this means change it is easier to affect may not be correct, with more traditional operating models in place.

○
Whilst it is absolutely true that discussion of culture and the importance of a thriving research ecosystem is more mainstream than it was across the UK 5 years ago, baseline operating contexts are very different across the devolved nations.It is important to acknowledge this in particular in relation as to how applications are reviewed against different baselines, an extremely difficult task, hence the importance of experiments like this!For example, whilst its absolutely true that RED(F) funding has delivered significant opportunities this is England only and that whilst REF has a UK wide 4 nation approach at a process level, funding allocations are allocated/calculated differently by the various devolved governments at a National level.

○ ○
The authors state that they hope that findings from IFRC projects become a valuable resource for institutions wishing to improve their research cultures and a catalyst for future change and discussion within the sector that makes academic careers more inclusive.We hope also that this will be the case.We would suggest also that experiments like this are crucially important for Funders to reflect on their own ways of working, to understand how the way they work drive behaviours and cultures.Experiments such as this are, we would argue, a crucially important opportunity for co-creation of future research and policy at a Funder, national and institutional level The commitment to building a community of practice to explore the next steps is incredibly valuable and we fully support this, a few points of clarification would be helpful in terms of: Whether by scale the authors mean breadth of impact rather than / or in addition to scale of funding?

○
In terms of equitable ways of working it would be helpful to acknowledge that it would be almost impossible to understand all nuances, or to remove all bias -but stress the importance of understanding and acknowledging baseline positions.Institutional ways of working and operational factors play a significant role in equity.Work by the University of Oxford from 2022 on equity in research funding is a valuable and useful read and sets out the issues well.

○
Influencing further positive developments.The applicants state that from the outset, they knew how difficult the challenge of assessing research culture would be and that they want to understand what has worked and what hasn't.To an extent this will always be context dependent and always messy.Setting the right conditions and behaviours that encourage learning and continual improvement across all parts of the system are likely to be as (if not more) important than policy.How will Wellcome itself build a learning approach into their own ways of working and make the best use of this and other experimental approaches?

○
In terms of influence, are Wellcome collaborating with other funders UK or globally to join up across the funder community of practice.

○
In conclusion we are delighted to see this experimental approach and commend Shomari Lewis-Wilson, Sonya Towers and Harriet Wykeham for submitting this open letter.
A few references that may be helpful: ref

Figure 1 .
Figure 1.A schematic diagram presenting the Wellcome Trust Research Programme strategy that the Research Environment team cuts across.Adapted from Wellcome, Corporate Affairs by Carleigh Krubiner.

©
2024 Coriat A et al.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Anne-Marie Coriat Registrar, College of Medicine and Veterinary Medicine, The University of Edinburgh (Ringgold ID: 3124), Edinburgh, Scotland, UK Jennifer Cusiter Head of Research Office, College of Medicine and Veterinary Medicine, The University of Edinburgh (Ringgold ID: 3124), Edinburgh, Scotland, UK

○ 3 .
Is the article written in open and accessible language?Yes 4. Next steps.This is an exciting experiment, we are delighted to see it happen, and for the letter to be shared so openly.In terms of the questions the authors posed before creating the IFC Can they fund at scale? ○ Can they fund equitably?○ Can they influence further positive developments?○ T, Adams E: Setting the right tone.Elife.2020; 9. PubMed Abstract | Publisher Full Text Is the rationale for the Open Letter provided in sufficient detail?Yes Does the article adequately reference differing views and opinions?Partly Are all factual statements correct, and are statements and arguments made adequately supported by citations?Yes Is the Open Letter written in accessible language?Yes Where applicable, are recommendations and next steps explained clearly for others to follow?Yes

Table 1 . List of awarded institutions for Wellcome's Institutional Fund for Research Culture 2023.
Table 1 in no particular order.Though the vi Rayner C & Bonnici F. The Systems Work of Social Change: How to Harness Connection, Context, and Power to Cultivate Deep and Enduring Change (Chapter 5).Oxford University Press 2021.ISBN: 0198857454, 9780198857457

Is the rationale for the Open Letter provided in sufficient detail? Yes Does the article adequately reference differing views and opinions? No Are all factual statements correct, and are statements and arguments made adequately supported by citations? Yes Is the Open Letter written in accessible language? Yes Where applicable, are recommendations and next steps explained clearly for others to follow? Yes Competing Interests:
1. Avin S: Mavericks and lotteries.Stud Hist Philos Sci.2019; 76: 13-23 PubMed Abstract | Publisher Full Text 2. Liu M, Choy V, Clarke P, Barnett A, et al.: The acceptability of using a lottery to allocate research funding: a survey of applicants.Res Integr Peer Rev. 2020; 5: 3 PubMed Abstract | Publisher Full Text No competing interests were disclosed.

have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Reviewer Report 09 February 2024 https://doi.org/10.21956/wellcomeopenres.22730.r69985