Gender Systematics in the NRAO Proposal Review System

Access to a national facility such as the National Radio Astronomy Observatory (NRAO) is often critical to the success of a scientist's research program. The NRAO therefore allocates telescope time through a competitive peer review process as do many other observatories. Studies show that unconscious bias can play a significant role when evaluating proposals, but the situation is complex and other factors can explain systematic trends in proposal success rates (e.g., see Johnson & Kirk 2020, and references within). A broad representation of our user community should produce the most compelling science, and therefore identifying systematic trends in the proposal review process is important.

Reid (2014) found that male ⁴ PIs had a higher success rate than female PIs for Hubble Space Telescope (HST) observing proposals from cycles 11–21. The gender imbalance in any single cycle was subtle and not immediately obvious, but a pattern was noticeable when several successive proposal cycles were combined. These differences do not depend on the geographic location of the PI nor on the gender distribution of the reviewers, but review panels with a higher average seniority produced the lower success rates for proposals with female PIs. This led the Space Telescope Science Institute (STScI) to adopt a dual-anonymous proposal review process starting in cycle 26. Johnson & Kirk (2020) investigated reviewer bias and found that female PIs were ranked lower than male PIs by male reviewers before, but not after, the dual-anonymous process was adopted. Additional data are needed to confirm this trend.

The findings by Reid (2014) prompted studies at other observatories. Patat (2016) analyzed gender systematics in telescope time allocation at the European Southern Observatory (ESO) over a period of eight years. Similar to HST, Patat (2016) found that male PIs had a higher success rate than female PIs, but this difference could mostly be attributed to seniority. For example, only 34% of female PIs are professional astronomers, which do not include post-docs or students, compared with 53% for male PIs. Nevertheless, after accounting for seniority there remained a small, but significant, gender imbalance.

Lonsdale et al. (2016) studied gender systematics in the facilities operated by Associated Universities Inc. (AUI). These include the Atacama Large Millimeter/submillimeter Array (ALMA) ⁵ and AUI's North American (NA) facilities: the Very Large Array (VLA), the Very Long Baseline Array (VLBA), and the Green Bank Telescope (GBT). ⁶ ALMA cycles 2–4 and NRAO semesters 12A–17A were analyzed. Lonsdale et al. (2016) found a significant gender imbalance where male PIs were favored over female PIs. For ALMA the rank distributions between male and female PIs were not the same with a high reliability. A similar result was found for AUI's NA facilities but with less significance. The was no correlation between gender rankings and the gender distribution on the review panels. The data to study the seniority of PIs was not available.

Carpenter (2020) expanded the gender systematics study of ALMA to cycles 0–6. The analysis considered both review stage 1, which includes preliminary scores, and review stage 2, which are final scores after a face-to-face discussion within each panel. Regardless of regional demographics, male PIs were found to have a higher proposal success rate than female PIs for each cycle. In any given cycle, however, the result is not significant (>3σ). Comparison of stage 1 and stage 2 results revealed no significant changes in the distribution of rank by gender, region, or seniority. He concluded that any systematics in the proposal ranking is not introduced from the face-to-face discussions.

Hunt et al. (2019, 2021) extended the analysis of Lonsdale et al. (2016) for AUI's NA facilities to semesters 12A–21A. Using the same procedures they initially found similar trends, that male PIs were favored over female PIs, but in later semesters this gender imbalance appeared to be ameliorated. During semesters 17B–21A there appears to be a slight imbalance that favors female PIs over male PIs. Here we perform a more rigorous analysis of the data for AUI's NA facilities over semesters 12A–21A to help understand the significance of these results.

The NRAO runs the proposal process for AUI's NA facilities which includes the VLA, VLBA, and GBT (see Schwab et al. 2015). Observing proposals are submitted twice per year with nominal deadlines of August 1 (semester A), for observations to be made the following year from February to July, and February 1 (semester B), for observations to be made from August to the following January. Proposals are evaluated based on their scientific merit and technical feasibility using a panel-based review system. There are currently nine science review panels (SRPs) which are divided into different areas of science. Each SRP consists of at least 5 reviewers plus a chair. Proposals are initially reviewed and assigned an individual raw score from 0.1–9.9 (lower scores are better). Once the individual reviews are complete, the scores are normalized for each reviewer to have a mean of 5.0 and a standard deviation of 2.0. Each SRP will then meet remotely to discuss the proposals and form a consensus review. At this stage a technical review of the feasibility of the proposed observations, performed by observatory staff, is made available to the SRP. During the SRP meeting, scores may be adjusted based on the discussion. After the consensus reviews are completed, the proposals in each SRP are rank-ordered and normalized to obtain what we call a linear rank. The linear rank for the ith proposal is 10 r_i /n, where r_i is the rank associated with that proposal and n is the number of proposals reviewed by the SRP.

The TAC consists of the SRP chairs, with one serving as the TAC chair, and meets face-to-face after the SRP meetings. ⁷ The TAC considers the linear ranks, together with any technical, resource, or scheduling constraints to assign a scheduling priority. The TAC does not reevaluate the scientific merit of the proposal. Priority A is the highest scheduling priority and the observations will almost certainly be scheduled. Priority B is the next highest scheduling priority and the observations will be scheduled on a best effort basis. Priority C is the lowest scheduling priority (e.g., filler time). Priority N will not be scheduled. The TAC does not allocate telescope time but makes recommendations to the observatory Directors. A Directors' review is held each semester to evaluate the TAC's recommendations. A final program is then approved. Therefore, a successful proposal might be defined as a proposal that receives a scheduling priority A or B. But this depends on resource and scheduling constraints and therefore here we use the linear rank as the measure of success.

In response to the findings in Lonsdale et al. (2016) two significant administrative changes were made to the review process starting with semester 17B. (1) At the beginning of each SRP and TAC meeting the status of the gender imbalance is discussed and the reviewers are reminded that rankings and decisions must reflect only scientific merit, technical feasibility, and operational constraints. (2) The NRAO set goals to populate the SRPs and TAC with a gender distribution that reflects the community demographics (∼30%–35% female membership for NRAO users) and furthermore to arrange that there should be at least 2 female reviewers on each SRP and the TAC. Figure 1 shows the female fraction of SRPs and TAC by semester. These goals were achieved after only a few semesters.

Zoom In Zoom Out Reset image size

The data consist of the linear ranks and the PI gender for each proposal. ⁸ We also track the gender distribution of the SRPs and TAC. The PI gender is taken from the user account profile. Historically the user was able to specify their gender as "male" or "female." More recently we have allowed the user to specify their gender as "male," "female," or "self-identify." Regardless, the selection of gender is optional since we are not permitted to request the gender of any person using our instruments. Only ∼65% of users specify their gender. Most other PI gender assignments were determined by performing literature or web searches; a few were established by asking colleagues who had been collaborators. ⁹ Adding these genders to the database increases the percentage of PIs with a defined gender to 99.6%. A few genders are thus obviously still not determined, and the corresponding proposals are not included in our analysis. This can also lead to a slight inconsistency with time, since registrants may choose to add their gender to their profiles and others' genders may be determined later.

We divide the data into 32 different data subsets for analysis. We consider each semester (19 subsets); semesters 12A–17A and 17B–21A (2 subsets); each year (10 subsets); and all years (1 subset). For each data subset we calculate both the cumulative distribution function (CDF) and the 25th, 50th, and 75th percentiles for the male and female PI linear ranks. These percentiles are commonly referred to as the "quartile values" (i.e., Q1, Q2, and Q3). To assess the uncertainty in the linear rank distributions, we apply bootstrap resampling (Efron 1982). To do this we generate 10,000 simulated distributions of the male and female PI linear ranks. Each distribution consists of N data points which are randomly drawn from the original data with replacement, where N is the number of linear ranks in the sample. So the simulated distributions will miss some linear rank values from the original data and have some duplicates, triplicates, etc.

We use the Anderson–Darling (AD) test to evaluate the probability that the observed linear rank distributions for male and female PIs are drawn from the same parent sample. The AD statistic is considered to be more reliable than the Kolmogorov–Smirnov or Cramér-von Mises tests (cf., Babu & Feigelson 2006). The AD p-value is denoted as ρ_AD(x, y), where x is the list of female proposers' linear ranks and y the list of the male proposers'. To assess the uncertainty in the AD test results we again apply bootstrap resampling where ${\tilde{x}}^{(i)}$ represents the ith of m = 10,000 bootstrap resampling of the x values, and similarly ${\tilde{y}}^{(i)}$ represents the ith resampling of the y values. Then ${\rho }_{\mathrm{AD}}\left({\tilde{x}}^{(i)},{\tilde{y}}^{(i)}\right)$ is the AD p-value of the ith resample, and ${\rho }_{\mathrm{AD}}^{\mathrm{boot}}\equiv \displaystyle \frac{1}{m}{\sum }_{i=1}^{m}{\rho }_{\mathrm{AD}}({\tilde{x}}^{(i)},{\tilde{y}}^{(i)})$ is the mean of the bootstrap AD p-values.

A low ρ_AD(x, y) indicates that the two distributions are not from a common parent distribution, whereas a higher ρ_AD(x, y) suggests there is a degree of commonality. To broadly determine if there are gender imbalances in the NRAO proposal system, we assign an imbalance key (IK) indicating which gender, if any, is favored using the following criteria:

• 1.  
  Upper Case Letter (M or F): if ρ_AD(x, y) ≤ 0.1 (very suggestive).
• 2.  
  Lower Case Letter (m or f): if ρ_AD(x, y) ≤ 0.2 (suggestive).
• 3.  
  Blank: if ρ_AD(x, y) > 0.2.

When there is an imbalance the ρ_AD(x, y) does not indicate which distribution has the lower (better) linear ranks. This is determined by inspecting the male and female PI linear rank distributions.

We generate a series of plots for each data subset that includes histograms, CDFs, and quartile distributions of the male and female PI linear ranks. Figures 2 and 3 show the results for semesters 12A–17A and 17B–21A, respectively. The smoothed distributions are produced using kernel density estimation (KDE, Silverman 1996). ¹⁰ We use Mathematica functions SmoothHistogram and SmoothKernelDistribution with the default parameters: smoothing with a Gaussian kernel and an automatic choice of kernel bandwidth. Hereafter, in Mathematica parlance, we call these "smooth kernel histograms" or "smooth kernel CDFs." The smooth kernel distributions capture major differences, if any, by ignoring spurious detail.

Zoom In Zoom Out Reset image size

For semesters 12A–17A, male PIs clearly have better linear ranks than female PIs. This is obvious in the histogram plots of the linear rank distributions, but the CDF and quartile plots better demonstrate their significance via bootstrapping. For semesters 17B–21A, the trend is reversed with female PIs having better linear ranks than male PIs, but with less significance. This is only true for linear ranks near the 50th percentile (Q2). Plots for the remaining 30 data subsets are shown in the Appendix. For most semesters and years male PIs have lower linear ranks than female PIs, but the trend is reversed in some cases.

To help quantify these trends, we list the AD p-values for the 32 data subsets in Tables 1–2. The results by semester are shown in Table 1 where we list the semester, the imbalance key, ρ_AD(x, y), both ${\rho }_{\mathrm{AD}}^{\mathrm{boot}}$ and the 68% confidence interval from bootstrap resampling, and the percent of female PIs, SRP members, and TAC members. The ${\rho }_{\mathrm{AD}}\left({\tilde{x}}^{(i)},{\tilde{y}}^{(i)}\right)$ modeled distributions are not very Gaussian and therefore ${\rho }_{\mathrm{AD}}^{\mathrm{boot}}$ is not particularly useful. The 68% confidence interval, however, provides a useful measure of the uncertainty in ρ_AD(x, y). The horizontal line divides the data into two parts: before and after we informed our reviewers of the Lonsdale et al. (2016) results and started to recruit more female reviewers. The results for all semesters between 12A–17A and 17B–21A are given in the last two rows of the table and shown in Figures 2 and 3. Male PIs are clearly favored over female PIs for the earlier semesters. After the administrative changes the trend has reversed to some extent but with less significance. The ρ_AD(x, y) numbers are sensitive to small changes in the data. This is demonstrated by the large range in some of the 68% confidence intervals. For example, in semester 12A the AD p-value is ρ_AD(x, y) = 0.86, but the mean AD p-value from bootstrap resampling is ρ_AD ^boot = 0.34 with a 68% confidence interval of 0.099–0.58.

Zoom In Zoom Out Reset image size

Table 1. Gender Results by Semester

			AD Bootstrapping		Percent Female
Semester	IK	ρAD(x, y)	${\rho }_{\mathrm{AD}}^{\mathrm{boot}}$	68% Conf.	PI	SRP	TAC
12A	⋯	0.86	0.34	(0.099–0.58)	26.9	25.0	12.5
12B	⋯	0.40	0.25	(0.028–0.50)	26.2	25.0	12.5
13A	⋯	0.34	0.21	(0.026–0.41)	25.5	25.0	12.5
13B	⋯	0.56	0.29	(0.042–0.56)	24.5	27.1	25.0
14A	⋯	0.51	0.23	(0.050–0.43)	29.7	29.2	25.0
14B	M	0.049	0.056	(0.0030–0.10)	30.3	25.0	25.0
15A	⋯	0.73	0.31	(0.069–0.56)	24.8	18.8	12.5
15B	⋯	0.54	0.28	(0.045–0.54)	27.0	25.0	0.0
16A	⋯	0.65	0.27	(0.065–0.49)	30.2	25.0	12.5
16B	M	0.019	0.036	(0.00083–0.065)	22.6	17.0	11.1
17A	m	0.12	0.15	(0.0057–0.32)	32.7	20.8	12.5
17B	⋯	0.25	0.13	(0.027–0.24)	30.6	31.5	25.0
18A	⋯	0.22	0.16	(0.017–0.32)	32.1	38.9	25.0
18B	F	0.064	0.085	(0.0033–0.17)	33.6	48.1	37.5
19A	⋯	0.73	0.30	(0.061–0.56)	32.0	48.1	50.0
19B	⋯	0.71	0.33	(0.057–0.62)	28.7	43.6	77.8
20A	f	0.13	0.15	(0.0048–0.35)	36.5	39.3	66.7
20B	⋯	0.83	0.32	(0.074–0.57)	33.6	43.9	55.6
21A	F	0.097	0.12	(0.0037–0.26)	31.9	43.6	55.6
12A–17A	M	0.0084	0.015	(0.00038–0.025)	27.3	23.9	14.6
17B–21A	f	0.11	0.11	(0.0057–0.24)	32.5	42.1	49.1

Note. There are typically about 300 proposals per semester.

Download table as:  ASCIITypeset image

Following Lonsdale et al. (2016), we show the results by year by averaging over two semesters (see Table 2). This increases the sensitivity and allows us to show the imbalance key by telescope. The smaller number of proposals submitted to the VLBA, however, is still not statistically significant and therefore this column is blank and only included for completeness. Inspection of the imbalance keys shows the transition from male to female after the administrative changes.

Table 2. Gender Results by Year

	Imbalance Key					AD Bootstrapping		Percent Female
Year	GBT	VLA	VLBA	All	ρAD(x, y)	${\rho }_{\mathrm{AD}}^{\mathrm{boot}}$	68% Conf.	PI	SRP	TAC
2012	⋯	⋯	⋯	⋯	0.39	0.21	(0.033–0.41)	26.6	25.0	12.5
2013	F	⋯	⋯	⋯	0.20	0.18	(0.011–0.38)	25.0	26.0	18.8
2014	M	⋯	⋯	M	0.040	0.046	(0.0025–0.085)	30.0	27.1	25.0
2015	⋯	⋯	⋯	⋯	0.71	0.28	(0.073–0.49)	25.7	21.9	6.2
2016	⋯	m	⋯	m	0.13	0.11	(0.0086–0.22)	26.5	21.0	11.8
2017	⋯	⋯	⋯	M	0.086	0.087	(0.0047–0.17)	31.6	26.2	18.8
2018	m	f	⋯	⋯	0.23	0.13	(0.023–0.23)	32.7	43.5	31.2
2019	⋯	⋯	⋯	⋯	0.68	0.32	(0.048–0.63)	30.8	45.9	63.9
2020	⋯	⋯	⋯	⋯	0.61	0.28	(0.044–0.54)	35.0	41.6	61.1
2021	f	⋯	⋯	F	0.097	0.12	(0.0041–0.25)	31.9	43.6	55.6
All	⋯	f	⋯	M	0.031	0.029	(0.0022–0.052)	29.3	43.6	29.2

Note. Statistics for year 2021 only includes semester A.

Download table as:  ASCIITypeset image

There is mounting evidence of systematic, gender-related trends in the review of astronomical observing proposals. This was first discovered in HST observing proposals where male PIs had better success rates than female PIs (Reid 2014). Similar trends have been found at ESO (Patat 2016), ALMA (Lonsdale et al. 2016; Carpenter 2020), NRAO (Lonsdale et al. 2016), and the Canada–France–Hawaii Telescope and Gemini Observatory (Spekkens et al. 2018).

Unconscious bias may be the cause of the measured gender imbalance. But unconscious bias is complex, unintended, and multifaceted. Other effects such as prestige bias complicate the interpretation of the data. For example, some studies have noted a dependence on the gender-based success rate and PI seniority (Spekkens et al. 2018), the seniority of the review panels (Patat 2016), and the potential privilege of a proposer being a member of the TAC (Greaves 2018)—although Carpenter (2020) did not find any correlation. Numerous studies have noted that both males and females evaluate work by males higher than females for similar results. This bias has been discussed in several professional fields (see Spekkens et al. 2018; Johnson & Kirk 2020, and references within). This would lead to the conclusion that changing the gender distribution of the review panels would not by itself rectify the situation. Lonsdale et al. (2016) found no evidence that the panel gender composition was causing the gender imbalance, but here we do see a correlation between the gender distribution on the review panels and linear ranks. But we also started briefing the SRPs about the gender imbalance at the same time and therefore we cannot associate this correlation with the improved female PI linear ranks.

The STScI has adopted a dual-anonymous proposal review process for the HST where the identity of both the authors and reviewers are hidden. This requires that the proposal be anonymized and thus proposers have to consciously remove any information in the proposal that points to their team. Many other observatories are following this approach (e.g., ALMA, ESO, most NASA programs, etc.). Johnson & Kirk (2020) argue that unconscious bias is difficult to overcome. Many of the procedures used to combat unconscious bias, such as training programs, are not effective. Moreover, there can be a backlash of a perceived advantage. So instead of "fixing" the problem, a dual-anonymous approach reduces the likelihood of a bias from entering into the process. For these reasons the NRAO has decided to adopt a dual-anonymous proposal review process for the new telescope time allocation software currently being developed.

Studies reveal gender-related systematic trends in the peer review of observing proposals from astronomical observatories (Reid 2014; Lonsdale et al. 2016; Patat 2016; Spekkens et al. 2018). The exact cause of the measured gender imbalance is not certain and in some cases there are other factors at play (e.g., seniority). Here we investigate gender systematic effects from semesters 12A–21A in AUI's NA facilities: VLA, VLBA, and GBT. We use the linear ranks that are finalized after the SRP has formed their consensus review. To interpret these data we produce histograms, CDFs, and the quartile values of the male and female PI linear ranks. We use bootstrap resampling to assess the uncertainty in the CDFs and to produce a distribution of quartile values. To determine if the distribution of male and female PI linear ranks are from the same parent distribution, we calculate the Anderson–Darling statistic.

Over all semesters we find that male PIs are favored over female PIs with ρ_AD(x, y) = 0.031. The 68% confidence interval via bootstrap resampling is 0.0022–0.052. Starting in semester 17B, the NRAO made two administrative changes: (1) we informed SRP and TAC members of the gender imbalance found by Lonsdale et al. (2016), and (2) we increased the female membership on the SRP and TAC to reflect the community demographics. These changes appear to have ameliorated the gender imbalance. Between semesters 12A–17A male PIs are favored over female PIs with ρ_AD(x, y) = 0.0084, whereas between semesters 17B–21A female PIs are slightly favored over male PIs with ρ_AD(x, y) = 0.11. The gender imbalance may have merely been reversed, however, but at a lower significance.

Many observatories have adopted a dual-anonymous approach to proposal review to reduce the possibility of a bias, instead of trying to control biases with training or other methods. The NRAO plans to adopt dual-anonymous review with the new software being developed for our telescope time allocation system.

We thank the following people for constructive comments on the manuscript: John Carpenter, Allison Costa, Jeff Kern, Adele Plunkett, and Lyndele von Schill. The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc.

Software: Mathematica (Wolfram Research Inc., 2021).

Here we summarize the results of our analysis for 30 data subsets that include each semester (Figures 4–22), each year (Figures 23–32), and all years (Figure 33). Plotted are the histograms, CDFs, and quartile value distributions of the linear ranks for male and female PIs.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size