Measuring bias, burden and conservatism in research funding processes

Search strategy

We identified relevant literature through three routes:

Screening strategy

Using the search strings described above, the Scopus database yielded 1,741 results. We performed an initial test of our strategy by checking that specific key papers we were already aware of appeared in the results, for example Guthrie et al. (2018). Once satisfied the search strategy was performing effectively, we implemented a filtering process to determine the inclusion or exclusion of articles based on their relevance to address the primary objectives of this task as set out in Figure 1.

Figure 1. Screening process for articles pre-extraction.

Data extraction

Data extraction was performed using a data extraction template—a pre-determined coding framework based on the study aims (i.e. bias, burden, and conservatism). The headers of this template against which data was extracted for each article (where available) were:

Three researchers performed the full extraction of 100 articles in parallel. During this process, each researcher was instructed to add key references to a ‘snowballing database’. The snowballing database was populated with 15 articles, which were passed through the filtering processes described above, yielding an additional eight papers that were fully extracted. We also considered additional articles using a combination of targeted web searches and suggestions from our key informant interviews. These methods yielded an additional 18 articles that were included in our REA.

Key informant interviews

We conducted key informant interviews with one representative from each research funding organisation in Table 2 in order to understand how evaluation methods are employed in practice and to explore evaluation approaches that may not be documented in the literature. We identified respondents with relevant expertise at key biomedical and health research funders internationally and contacted them by email to request their participation. We focused on developed research systems that may be comparable with the Australian system, primarily in Europe and North America. We also interviewed researchers working on the analysis of peer review and grant funding approaches and their challenges. Initially 12 individuals were contacted. Of those, 6 agreed to participate; 5 did not respond to our request; 1 declined to participate; 2 further identified colleagues to participate in their place, who were contacted by email and in both cases accepted.

Interviews were conducted by telephone and lasted for approximately one hour. Interviews were recorded and field notes taken. One interview was conducted per participant. Interviews were conducted following a semi-structured protocol (see Table 3) to enable consistent evidence collection while providing the opportunity to explore emerging issues. As the interviews were designed to be semi-structured, we encouraged the interviewees to explore areas they thought were important that may not have been directly covered in our interview protocol. To protect the anonymity of the interviewees, the analysis that we report does not make any specific reference to individuals; we use the interview identifiers INT01, INT02, etc. to make references to specific interviews in our analysis.

Data analysis

The analysis took a framework analytic approach, aiming to capture information on processes and metrics used in practice across organisations in relation to the aims of this study to identify how bias, burden and innovation in funding process can be measured. Data from each interview was coded into an excel template by each individual conducting interviews (GM, DRR), with one row per interview (and hence organisation). The column headers were as follows:

Analysis was primarily focused on capturing information on practice at each of these organisations to provide a picture of the methods currently being used by research funding organisations to measure and to alleviate burden, bias and conservatism in peer review-based funding processes. However, we also reviewed evidence on challenges, strengths and weaknesses to identify any information to inform our wider analysis and discussion.

Ethical approval

This study was recommended for exemption by the RAND Human Subjects Protection Committee. Participant consent was obtained orally at the start of each interview. The precise detail of consent sought is set out in the interview protocol (Table 3).

Bias in the funding process

Robustness of procedure and efficiency of funding distribution are the two pillars supporting the legitimacy of peer review (Gurwitz et al., 2014), which has been described as a cornerstone of the scientific method (Tomkins et al., 2017) and the backbone of modern science (Tamblyn et al., 2018). A robust peer review process must be fair and objective in the distribution of grants. However, an increasing number of studies suggest that systematic bias occurs in a range of forms across grant peer review processes.

Measuring bias in the funding process

While there are an increasing number of studies examining bias in grant peer review, there is still deemed to be a lack of comparable, quantitative studies in the area (Bromham et al., 2016; Gurwitz et al., 2014; Guthrie et al., 2018). The main body of work has been performed by analysing historic data made available by funding agencies to academic researchers, though funding bodies themselves have undertaken work in the area (DFG, 2016; Ranga et al., 2012). A challenge in identifying and evaluating sources of bias is the lack of generalisability of findings, as funding programs have highly variable structures and funding bodies collect and make available different datasets. Table 4 lists the measurement approaches employed in the literature, indicates which areas of potential bias were explored and provides key references. A table listing each item identified during the literature review is available as Underlying data (Guthrie, 2019).

Applicant characteristics. Applicant characteristics collected during grant application processes may include gender, age, race, ethnicity and nationality. While reviewers do not usually see information about all characteristics, for instance race and ethnicity (Erickson, 2011), it may be apparent due to name, affiliation or prior knowledge.

Gender bias has been the primary area of study within applicant characteristics, perhaps having gained significant visibility in an early study that showed that females needed to be 2.5-fold more productive to achieve the same scores as males in the Swedish Medical Research Council’s peer review process (Wennerås & Wold, 1997).

Following this initial study, gender bias has been explored in several different countries. In The Netherlands, researchers funded by The Netherlands Organisation for Scientific Research (NWO) examined 2,823 applications between 2010 and 2012 from early career scientists, analysed gender as a statistical predictor of funding rate, and examined the success rate throughout the process (application, pre-selection, interview, award) (Van Der Lee et al., 2015) The authors found that there was a gender disparity with males receiving higher scores in ‘quality of researcher’ evaluations but not ‘quality of proposal’ evaluations, particularly in disciplines with equal gender distribution among applicants.

Another study in the US looked at bias in the Research Project (R01) grants from the National Institutes of Health (NIH), and found this grant program exhibited gender bias in Type 2 renewal applications (Kaatz et al., 2016). The authors analysed 739 critiques of both funded and unfunded applications, using text analysis and regression models. The study found that reviewers gave worse scores to female applicants even though they used standout adjectives in more of their critiques. A second piece of work from the same authors employed more state-of-the art text mining algorithms to discover linguistic patterns in the critiques (Malikireddy et al., 2017). The algorithms showed that male investigators were described in terms of leadership and personal achievement while females were described in terms of their working environments and ‘expertise’—potentially suggesting an implicit bias where reviewers more easily view males as scientific leaders, which is a criterion of several grant funding programs.

In a longitudinal study, researchers followed the careers of an elite cohort of PhDs who started postdoctoral fellowships between 1992 and 1994 (Levitt, 2010). The study found that 16 years after the fellowships, although 9 per cent of males had stopped working in a scientific field, compared with 28 per cent of females, there was no significant difference in the fractions obtaining associate or full professorships. However, females whose mentors had an h-index in the top quartile were almost three times more likely to receive grant funding – males’ success had no such correlation with their mentors’ publication record.

In a Canadian Institutes of Health Research (CIHR) funded study, researchers evaluated all grant applications submitted to CIHR in the years 2012–2014 (Tamblyn et al., 2018). Descriptive statistics were used to summarise grant applications, along with applicant and reviewer characteristics. The dataset was then interrogated with a range of statistical approaches (2-tailed F-test, Wald χ² test), which showed that higher scores were associated with having previously obtained funding and the applicant’s h-index and lower scores with applicants who were female or working in the applied sciences.

Some funding agencies do not detect gender bias in their grant programs. For example, the Austrian Science Fund performed an analysis of 8,496 research proposals from the years 1999–2009 using a multilevel regression model, and found no statistically significant association between application outcome and gender (Mutz et al., 2012). Meta-analyses of gender bias have reported on both sides of the debate, with claims that applicant gender has little (Ceci & Williams, 2011) or substantial (Bornmann et al., 2007) effect on receiving grants.

Exploration in relation to racial bias has also been performed, though there is a smaller body of work than on gender bias. In 2011 researchers funded by the NIH showed that black applicants were ten percentage points less likely to obtain R01 funding than their white peers, after extensively controlling for external factors (educational background, country of origin, training, previous research awards, publication record and employer characteristics) (Ginther et al., 2011). A funding gap between white/mixed-race applications and minority applicants has been a persistent feature of NIH grant funding between 1985 and 2013 (Check Hayden, 2015). According to a preprint article from mid-2018, racial bias in the NIH system may have diminished (Forscher et al., 2018). The researchers report on an experiment where 48 NIH R01 proposals were modified to contain white male, white female, black male and black female names before being sent for review by 412 scientists. The authors found no evidence—at the level of ‘pragmatic importance’—of white male names receiving better evaluations than any other group; however, they note there may be bias present at other stages of the granting process.

Career stage. Career stage is another potential source of bias in the peer review process. An ageing population of researchers may cause problems because it may crowd-out early-career researchers from funding, thus preventing them from establishing their careers (Blau & Weinberg, 2017). In the US, the average career age of new NIH grantees, defined as years elapsed since award of the highest doctoral degree, increased dramatically between 1965 and 2005, rising from 7.25 years to 12.8 years (Azoulay et al., 2013). The cause of the shift is uncertain. Proposed theories include an increased burden of knowledge due to an expanding scientific frontier; the use of post-doctoral positions as a ‘holding tank’ for cheap, skilled labour; and the move to awarding grants as prizes for substantial preliminary results rather than to fund new research (Azoulay et al., 2013).

Measuring bias regarding career stage is problematic due to the challenges associated with defining career stage. Career age—the years elapsed since award of the highest doctoral degree—is one commonly used description (Azoulay et al., 2013). While this approach is suitable for identifying strong trends, such as the near doubling of career age of new NIH grantees discussed above, it does not take into account factors such as teaching commitments, changing research topics, clinical work or career breaks (Spaan, 2010).

There are other approaches to defining career stage, for example focusing on necessary competences rather than time elapsed. The European Framework for Research Careers has four stages—first stage researcher, recognised researcher, established researcher and leading researcher—and provides a classification system that is independent of career path or sector (EC-DGRI, 2011).

Research field. There may be biases between research fields and also against research that falls between, or combines, those fields. While interdisciplinary research is often considered fertile ground for innovation, there is a belief among researchers that interdisciplinary proposals are less likely to receive funding (Bromham et al., 2016). Defining and identifying interdisciplinary research is a challenge that has hindered the evaluation of this potentially damaging belief. A recent study sought to address this challenge by developing a biodiversity metric, the interdisciplinary distance (IDD) metric, to capture the relative representation of different research fields and the distance between them (Bromham et al., 2016). Using data from 18,476 proposals submitted to the Australian Research Council’s Discovery Program over a five-year period, the authors found that the greater the degree of interdisciplinarity, the lower the probability of an application being funded.

Institution. There is also some evidence that characteristics of the institution may be a source of bias in the grant application process. For example, a 2016 study of Canada’s Natural Sciences and Engineering Research Council (NSERC) Discovery Grant program found that funding success and quantity were consistently lower for applicants from small institutions, and that this finding persisted across all levels of applicant experience as well as three different scoring criteria (Murray et al., 2016). The authors analysed 13,526 proposal review scores, using logistical regression to determine patterns of funding success and developing a forecasting model that was parameterised using the dataset. The authors note that some differences between institutions may be due to differences in merit and differences in research environments; they recommend that more needs to be done to ensure funds are distributed appropriately and without bias.

Reviewers. Reviewers may have overt or implicit biases that can affect their scoring of grant proposals, some of which are noted above. The level of expertise that reviewers have relating to an application can affect their evaluations, with studies finding both advantageous and disadvantageous effects. Li examined this issue by constructing and analysing a dataset of almost 100,000 applications evaluated in over 2,000 meetings (Li, 2017). The study found an applicant was 2.2 per cent more likely to receive funding, the equivalent of one-quarter of the standard deviation, if evaluated by an intellectually closer reviewer as measured by the number of permanent reviewers who had cited the applicant’s work in the five years prior to the meeting. Conversely, another study found that reviewers with more expertise in an applicant’s field, as measured by a self-assessment of their level of expertise relating to an application, were harsher in their evaluations (Gallo et al., 2016).

The characteristics of reviewers have also been shown to affect their evaluations. Jayasinghe, Marsh and Bond have published several studies based on collaboration with the Australian Research Council (Marsh et al., 2008) exploring the peer review of grant applications. One finding was that the nationality of peer reviewers affected the ratings they gave, with Australian reviewers scoring similarly to European reviewers, but harsher than those from other countries and specifically North America. The authors were unable to determine if the cause was awareness that Australian reviewers were likely competing with applicants for funding, or purely a result of different nationalities.

Even in the absence of bias, reviewers may not always agree on the quality of a proposal. The concept of inter-rater reliability—the degree of agreement among raters—is central to peer-review, yet has not been thoroughly examined in this context (Clarke et al., 2016). Three studies over the last half century have shown quite consistent levels of disagreement between reviewers, ranging from 24–35 per cent disagreement (Cole et al., 1981; Fogelholm et al., 2012; Hodgson, 1997). A more recent randomised trial study considered 60 applications to NHMRC’s Early Career Fellowship program, which were duplicated by NHMRC secretariat and reviewed by two grant panels (Clarke et al., 2016). The study found inter-rater reliability to be 83 per cent, which is comparable to the previous studies. The authors suggest that the slight reduction in disagreement may be due to the nature of early career applications or differences in the scoring and assessment criteria.

Strategies for reducing bias

As the research community has gained an increasing awareness of bias, steps have been taken to develop fairer processes and programs. Table 5 below provides a summary of approaches used by a range of international funders both to reduce bias in their funding processes, and to evaluate bias across their funding streams.

Many funders now have targeted grant streams to support applicants who were found to be disadvantaged by biases in peer review or program structure, such as early career researchers. For example, the NIH K02 Independent Research Scientist Development Award, the Medical Research Council (MRC) New Investigator Research Grant, and the European Research Council (ERC) Starting Grants, are a small selection of funding aimed at early career researchers.

There is some emerging evidence that training can reduce bias and increase the inter-rater reliability of reviewers. The CIHR introduced a reviewer training program following the discovery that its new grant system focusing on applicants’ track records was disadvantaging women, while a program focusing on the research proposal was not. In the grant cycle following the introduction of a training module on unconscious biases, female and male scientists had equal success rates (Guglielmi, 2018). Additionally, an online training video was found to increase the inter-rater reliability for both novice and experienced NIH reviewers, with correlation scores rising from 0.61 to 0.89 following training (Sattler et al., 2015).

Blinding the identity of applicants from reviewers has been studied as a mechanism for increasing the fairness of peer review systems. In the context of journal peer review, the journal Behavioural Ecology found that its introduction of double-blind review increased the representation of female authors by 33 per cent, to reach a level that reflects the composition of the life sciences academic workforce (Budden et al., 2008). The US National Science Foundation (NSF) has trialled a blinded application process called ‘The Big Pitch’, which involves applicants submitting an anonymised two-page research proposal alongside a full conventional proposal (Bhattacharjee, 2012). The NSF reported that there was only ‘a weak correlation’ between the success outcomes of the full and the brief, anonymous applications.

Burden in the funding process

The number of grant applications for research submitted for review is increasing in the majority of countries and disciplines (De Vrieze, 2017). However, funding for research is being reduced, leading to a decrease in the success rate of applications. The grant application process is time-consuming and costly, with the burden falling on those applying for the funding (Bolli, 2014; Gurwitz et al., 2014; Guthrie et al., 2018; Kulage et al., 2015), those reviewing the applications submitted (Bolli, 2014; Kitt et al., 2009; Schroter et al., 2010; Snell, 2015), the funding agency (Liaw et al., 2017; Schroter et al., 2010; Snell, 2015) and the research institutions (Kulage et al., 2015; Ledford, 2014; Specht, 2013).

Measuring burden in the funding process. The burden of the grant application process has been measured for applicants, reviewers, funders and research institutions using a variety of methods. A list of the different approaches used to evaluate burden of the application process can be found in Table 6.

Surveys have frequently been used to assess the burden of grant preparation (Herbert et al., 2013; Herbert et al., 2014; Kulage et al., 2015; Wadman, 2010) and grant reviewing (Schroter et al., 2010; Wadman, 2010). These surveys enquire on the time spent on average in the application and review process, as well as the distribution of time among the various activities of the grant application process.

Another approach has been to use an observational study of proposals for health services research funding to measure time spent preparing proposals, the use of simplified scoring of grant proposals (reject, revise or invite to interview), and progression from submission to outcome (Barnett et al., 2015).

Others have estimated the cost of grant application by tracking the time spent preparing a proposal and combining the data with nationwide salaries, fringe rates, and facilities and administrative (F&A) cost recovery rates for personnel (Kulage et al., 2015).

The minimum number of reviewers has also been assessed in order to reduce the burden on reviewers. (Liaw et al., 2017; Snell, 2015). One study focused on NHMRC and looked at the agreement on funding decisions on applications between different numbers of panellists and different lengths of applications (Liaw et al., 2017). A second study evaluated the review process from a CIHR post-doctoral fellowship competition, bootstrapping replicates of scores from different reviewers to determine the minimum number of reviewers required to obtain reliable and consistent scores (Snell, 2015).

Strategies for reducing burden

In recent years, different strategies have been developed to try to reduce the burden of grant applications. Table 7 provides a summary of approaches used by a range of international funders, both to reduce burden in their funding processes, and to measure the level of burden. These measures could allow researchers to focus on their research, save reviewers time, and potentially reduce the cost of grant review by reducing the labour required to review grant applications (Bolli, 2014).

Application limits. In 2009, the NIH incorporated a clause into their grant policy limiting applicants to two submissions of a research proposal (Rockey, 2012). Although this policy was not well received by the research community (Benezra, 2013), analysis by the NIH revealed a higher proportion of first-time applications were being awarded, along with a reduction in the average time to award from submission (Rockey, 2012). Restricting resubmission has also been adopted by the European Research Council (ERC) (Council, 2017).

Two-stage application process. The ERC has combined the application limit with a two-stage application that involves awarding project proposals a score during the first stage of the application process (A, B or C). If the project is awarded an A, the proposal will proceed onto the next assessment stage. If the proposal received a B, the applicant must wait one year before reapplying. And if the proposal is graded with a C, the applicant must wait two years before reapplying to any of the ERC-funded programs. This approach has led to a decrease in the number of applications received for evaluation by the ERC (INT04). The NIHR have also adopted a two-stage application process that has led to a decrease in the number of applications sent for peer review and a shorter time between application submission and outcome notification (INT01).

Multiple calls per year. Moreover, the NIHR has multiple calls for proposals throughout the year, which reduces not only the burden on reviewers by decreasing the number of applicants to review per round (INT01), but also on the applicants by having ongoing grant applications (Herbert et al., 2014).

Grant application length. In 2012, the NIH reduced the length of most grant applications from 25 pages to 12 pages, with the aim of reducing the administrative burden on both applicants and reviewers, and to focus on the concept and impact of the proposed research rather than the details of the methodological approach (Wadman, 2010). However, a study by Barnet et al. found that shortening the length of an application slightly increased the time spent by applicants preparing the proposal (Barnett et al., 2015).

Funding period. Extending the funding period to five years (Bolli, 2014) has also been suggested to reduce the burden of grant application.

Conservatism in research funding: identifying and defining innovation and creativity

What constitutes ‘innovative research’ is difficult to define. Approaches to identifying and defining innovation are varied and include defining innovation based on expert opinion (Marks, 2011), as research that does not require extensive preliminary results (Spier, 2002) (INT04), and (in the field of clinical research) as efforts that lead to improvements in patient care and progress (Benda & Engels, 2011).

Although there is a belief that quality research should be innovative and lead to new understanding of science (Benda & Engels, 2011), the current process of reviewing grant applications, mainly peer review, has been defined as ‘anti-innovation’, providing strong incentives for incremental research and discouraging research into new unexplored approaches (Azoulay et al., 2013; Fang & Casadevall, 2016; Guthrie et al., 2018).

Innovation and productivity are driven by diversity (Magua et al., 2017); therefore, advancing women or other underrepresented groups to institutional leadership (Magua et al., 2017) as well as promoting interdisciplinary research (Bromham et al., 2016) should have a positive impact on promoting innovative research.

Another feature of innovative research is its uncertain and potentially controversial nature. While many funding agencies aim to support innovative research, the body of work on peer review suggests it is an inherently conservative process (Langfeldt & Kyvik, 2010).

Measuring innovation and creativity

Since defining and identifying innovative, creative research is challenging, it can be difficult to measure the level of innovation and creativity within a research portfolio, or the extent to which a research funding program fosters innovation. However, there are examples in the literature of potential approaches to measuring innovation, as set out in Table 8.

By definition, new ideas are likely not to be met with consensus. It has been suggested that innovation could be measured through a metric based on lack of agreement between reviewers, measuring controversy as a surrogate for innovation, with new metrics, including variance or negative kurtosis, the degree to which observations occur in the tails of the grading distribution (Kaplan, 2007).

Productivity is another potential approach to measuring innovation. One study assessed the careers of researchers funded by two distinct mechanisms, investigator-initiated R01 grants from the NIH and the investigator program from the Howard Hughes Medical Institute (HHMI), with the aim of determining whether HHMI-style incentives result in higher rate of production of valuable ideas (Azoulay et al., 2011). The authors estimated the effect of the program by comparing the outputs of HHMI‐funded scientists with that of the NIH-funded scientists within the same area of research, who received prestigious early career awards. Using a combination of propensity-score weighting and difference-in-differences estimation strategies, the authors found that HHMI investigators produced more high-impact journal articles than the NIH-funded researchers, and that their research was more prone to changes.

Another study looked at the relation between the knowledge contained in an application proposal and a reviewer’s expertise and the outcome of proposals focusing on innovative research and area of expertise (Boudreau et al., 2016). In this study, the authors designed and executed a grant proposal process for research, and randomised how proposals and reviewers were assigned, generating 2,130 evaluator-proposal pairs. The authors found that evaluators give lower scores to research proposals that are highly novel, evaluated as new combinations of MeSH terms in a proposal relative to MeSH terms in the existing scientific literature, and to proposals in their area of expertise (Boudreau et al., 2016). However, another study focusing on public health proposals found that reviewers favour their own fields (Gerhardus et al., 2016).

A few organisations conduct peer review, with some unique practices placing higher value on innovation (Liaw et al., 2017). Criteria for assessing innovation are determined by the different organisations.

Strategies for improving the assessment of innovation and creativity

In recent years, different strategies have been developed to improve the assessment of innovation in grant review. Table 9 provides a summary of approaches used by a range of international funders, both to increase innovation and creativity and to evaluate the level of innovation and creativity across their funding streams.

To ensure innovative research is being funded some agencies, including the NIH, adopt an ‘out of order funding’ approach (Lindner & Nakamura, 2015). In this approach, a number of applications for innovative research are chosen for funding despite receiving lower scores than other funded research based purely on the peer review process. In the NIH, this strategy has led to approximately 15 per cent of applications selected ‘out of order’.

The NIH has also made additional changes to the peer review process in order to increase the emphasis on innovation and decrease the focus on methodological detail (Lindner et al., 2016). These changes included reducing the length of the methodological description (from 25 to 12 pages), with guidance to focus away from routine methodological details towards describing how their application is innovative. Including innovation as a criterion for grant assessment could incentivise researchers to include innovative ideas and new approaches into their proposals (Guthrie et al., 2018).

Many funding agencies have also adopted the strategy of having a separate scheme to fund innovative research, allocating smaller funds with a shorter time frame to these specific streams. The NIH has developed the New Innovator Award, committing $80 million to the award, and two others that specifically encourage innovation, the Pioneer and Transformative R01 Awards (Alberts, 2009). ZonMW have designed an ‘off-road’ program aimed at high-risk, high-reward projects, providing €100,000 for 1.5 years (INT02). NIHR has also designed different funding tiers to promote funding for innovative projects, providing £150,000 funding for 18 months (INT01). However, this strategy could include longer funding periods to encourage a culture of innovation among young researchers who remain reluctant to take the risk of pursuing ambitious ideas, acknowledging the need for preliminary results to obtain funding for most research (Alberts, 2009).

Underlying data

Figshare: Articles on bias burden and conservatism in grant peer review. https://doi.org/10.6084/m9.figshare.8184113.v1 (Guthrie, 2019).

This project contains a list of all publications, with URLs, identified during the literature review.

Consent sought and confidentiality assured in the interview process means that the interview transcripts from this study cannot be made publicly available (see interview protocol Table 3). This is due to the high risk of identifying individuals from the small sample of interviews conducted.