Investigation of potential gender bias in the peer review system at Reproduction

This study examined whether publication outcome was affected by the gender of author, handling associate editor (AE), or reviewer, and whether there was gender bias in reviewer selection, in the journal Reproduction. Analyses were carried out on 4289 original research manuscripts submitted to the journal between 2007 and 2019. Both female and male AEs appointed more male reviewers than female reviewers, but female AEs were significantly more likely to appoint female reviewers than male AEs were (p < 0.001). When examining the gender of either first or last author manuscripts, those with female authors that were reviewed by female reviewers received better scores than those with male authors that were reviewed by female reviewers (p < 0.05): where the reviewer was male, no such effect was observed. Acceptance rates of manuscripts were similar for both female and male authors, whether first or last, regardless of AE gender. Overall, there was no significant correlation between gender of first or last author, or of AE, on the likelihood of acceptance of a research paper. These data suggest no bias against female authors during the peer review process in this reproductive biology journal.


INTRODUCTION
Equality, diversity and inclusion are key to promoting innovative and collaborative research (Campbell et al., 2013). A major component of this is representation of females in publishing and peer review. Historically, females have been underrepresented as published authors (Larivière et al., 2013;van den Besselaar & Sandström, 2017). This representation gap has been linked to gender bias in the peer review system. For example, a study of Frontiers journal papers published between same-gender preference (homophily) in reviewer selection (Helmer et al., 2017), with females selected as reviewers overall less often than males in the geophysical sciences (Lerback & Hanson, 2017).
Similarly, studies of several journals in the ecological sciences have demonstrated significant gender homophily in reviewer selection (Buckley et al., 2014;Fox et al., 2016;Murray et al., 2019).
Because of a lack of comprehensive analyses, whether author gender bias affects outcomes of manuscript submission is still not clear. Some studies suggest that there is no difference in acceptance rates between manuscripts authored by males or females (Buckley et al., 2014;Fox et al., 2016), or even that papers authored by females have higher acceptance rates than those authored by males (Squazzoni et al., 2021). Other reports do suggest that papers authored by females have lower acceptance rates than those authored by males, although these studies are smaller in scope (Fox & Paine, 2019;Hagan et al., 2020;Murray et al., 2019). Exploration of the full impact of gender bias in publishing is needed to resolve these discrepancies.
As suggested by Lerback and Hanson (2017), it is important that journals assess whether there is a gender bias within their own peer review process, since the bias could be journal-and/or field-specific. The objective of this study was to determine whether publication outcome has been affected by the gender of the author, handling associate editor (AE), or reviewer, and also whether there has been gender bias in reviewer selection, in the journal Reproduction.

MATERIALS AND METHODS
Analyses were carried out on a total of 4289 original research manuscripts submitted to the journal Reproduction between 2007 and 2019.
The journal Reproduction is a society journal, owned by the Society of Reproduction and Fertility. The journal falls in the Science Citation Index category 'Reproductive biology' and publishes studies of reproductive biology, primarily in mammals including humans. Typical author order in the field is trainee as first author and principal investigator as last (senior) author. Manuscripts are assigned to an AE by the editorial staff: AEs are then responsible for selecting two or three reviewers using a single-anonymized review system: AE and reviewers are anonymous to authors. Reviewer selection is made based on a combination of the following: the AEs knowledge of experts in the field; PubMed searches; and a journal-specific reviewer database on the ScholarOne platform, with AEs instructed as to the use of these. AEs did not themselves act as reviewers. Upon receipt of external reviews, the AE makes a recommendation to the editor-in-chief (EIC), who then makes the final decision. In rare situations where the EIC does not concur with the AE, the two discuss the paper to arrive at a final decision. During the period covered by this study, there was always a single male EIC, with three different EICs in that role across the study period. At any one time, the editorial board consisted of one male EIC and a mean of 46 AEs, with individuals retired from and recruited onto the board in a continuous manner.
Information was obtained for the 4289 original research manuscripts submitted to Reproduction over the study period, with each submission providing an independent entry to the dataset irrespective of the identity of the handling AE. Information gathered included inferred gender of first author, inferred gender of last author, country of origin, gender of the AE who handled the manuscript, inferred gender of agreed reviewers, reviewer scores of manuscripts, and outcome of the submission. The gender of the AEs was provided by the publisher based upon first-hand knowledge of the individuals. For all other cases, gender was inferred using the genderize.io API tool with country-specific parameters: this assigned a binary 'male' or 'female' label. Where assumed gender could not be predicted with greater than 95% reliability, or where authors were from a country where overall confidence of prediction was below 55% (Fig. 1), gender prediction information was not included in the statistical analyses but are given in the Tables for completeness, shown as 'Gender not classified'. Specific cut-off levels of 95% and 55% were used to reduce ambiguous inferences to a minimum, bearing in mind that the ScholarOne system used by this journal required authors names to be given in English which can lead to prediction errors for some countries; see for example Sebo (2022).
Data were analysed using R, and results considered significant where p < 0.05. The raw data file used for all analyses is provided, showing inferred gender where predicted, but without names for reasons of confidentiality (File S1).

Demographics
Over the period examined by this study, the journal's editorial board contained a significantly higher proportion of male AEs; $17-24% of AEs were female between 2007 and 2013, with that proportion increasing to 44% during the last 2 years analysed. In addition, males acted as reviewers more often than females (Table 1:

Key points
• Data from 4289 original research manuscripts submitted to the journal Reproduction were examined to determine if publication outcome was affected by gender.
• Female Associate Editors were significantly more likely to appoint female reviewers than male Associate Editors.
• Overall, there was no significant correlation between gender of first or last author, or of Associate Editor, on the likelihood of acceptance of a research paper. n = 4235; χ 2 p < 0.001 for all). The submitted manuscripts had a significantly higher proportion of female first authors and male last authors. These findings are in agreement with other studies (Helmer et al., 2017) which demonstrate that the underrepresentation of females in senior authorship and in editorial positions occurs in the reproductive sciences. Percentage of reliable unique name predictions per country. The darker the bar, the more unique names from a given country were present in the data (exact figure shown next to bar). To avoid ambiguous predictions, data were used only where predictions could be made for at least 55% of the unique names from each country.
Gender of author and of AE had no overall effect on likelihood of publication As the main concern is whether gender of authors affects publication outcome, we started by assessing whether there was a correlation between the gender of first author, last author or AE on the likelihood of acceptance of a research paper; no significant correlation was found (Table 2: total n = 4235; χ 2 p > 0.2).

Impact of AE gender on reviewer selection
We then assessed whether AE gender correlated with the gender of reviewers that they recruited. Both female and male AEs appointed more male reviewers than female reviewers, but female AEs were significantly more likely to appoint female reviewers than male AEs (Table 3: n = 9282; χ 2 p < 0.001). Again, this outcome is consistent with the literature (Buckley et al., 2014;Fox et al., 2016;Pinho-Gomes et al., 2022). We note that these data take into account only those reviewers who accepted an invitation to review, with no information available about those who were approached but then declined.

Impact of reviewer gender on manuscript recommendations
To determine whether reviewers treated male and female authors differently, we ranked the recommendation that reviewers gave to the original, first-submission versions of manuscripts.
Reviewers' recommendations were scored as: one for acceptance; two for minor revisions; three for major revisions; or four for rejection. Separate analyses were carried out to determine whether reviewers' recommendations were affected by the gender of either the first authors or the last authors.
For first authors, there was no correlation between the overall reviewer's score (i.e. the average reviewer's score regardless of gender) and the gender of the first author (  Note: Data were not used for analysis where gender could not be predicted with sufficient reliability ('Gender not classified'). *** p < 0.001.  Note: Data were not used for analysis where gender could not be predicted with sufficient reliability ('Gender not classified'). *** p < 0.001. Note: Data were not used for analysis where gender could not be predicted with sufficient reliability ('Gender not classified'). * p < 0.05.
Armitage p = 0.155, Mann-Whitney p = 0.157). However, when data for female and male reviewers were analysed separately, manuscripts with female first authors that were reviewed by female reviewers received better (lower) scores than those with male first authors that were reviewed by female reviewers ( Cochran-Armitage p < 0.05, Mann-Whitney p < 0.05): where the reviewer was male, no such effect was observed (Table 7: n = 3611, Cochran-Armitage p = 1.0, Mann-Whitney p = 0.99).
When considering gender of last authors, for the overall reviewer's score (i.e. the average reviewer's score regardless of gender), manuscripts with female last authors received better (lower) scores than those with male last authors ( Cochran-Armitage p < 0.05, Mann-Whitney p < 0.05). When subdividing the data according to reviewer gender, the correlation between last author gender and score remained significant when the reviewer was female, with female last authors receiving better (lower) scores than male last authors from female reviewers (Table 5: n = 2056; Cochran-Armitage p < 0.05, Mann-Whitney p < 0.05), but not when the reviewer was male ( Cochran-Armitage p = 0.837, Mann-Whitney p = 0.836).

Impact of AE gender on manuscript outcomes
Acceptance rates of manuscripts were similar for both female and male authors, whether first or last, regardless of AE gender. In detail, manuscripts with female first authors had the same likelihood of acceptance as those with male first authors, whether handled by female AEs (Table 6: n = 707; χ 2 p = 0.625) or by male AEs (Table 6: n = 2156; χ 2 p = 0.071): likewise, manuscripts with female last authors had the same likelihood of acceptance as those with male last authors, whether handled by female AEs (Table 7: n = 731; χ 2 p = 0.145) or by male AEs (Table 7: n = 2228; χ 2 p = 0.894). Note: Data were not used for analysis where gender could not be predicted with sufficient reliability ('Gender not classified'). * p < 0.05. Note: Data were not used for analysis where gender could not be predicted with sufficient reliability ('Gender not classified').

Limitations
This analysis did not include country of origin, ethnicity or age of authors; some of these parameters would require authors to selfidentify and therefore cannot be analysed in retrospective studies such as this. Only one assignment tool was used to predict gender; the inclusion of other(s) may have reduced the proportion of nonclassified genders (Sebo, 2021). Another interesting question is whether gender of a reviewer biased AE decisions in the case of opposing reviewer recommendation, for example if a male reviewer recommended reject and a female reviewer recommended accept. This is difficult to analyse in the present study with this journal's editorial system as manuscripts have four recommendation categories.
In addition, the numbers of cases with completely opposing opinions from mixed-sex reviewers was too small to be meaningful. Further, we analysed individuals who agreed to act as reviewer and not those who were invited but declined; this is a different research question that deserves further attention.

CONCLUSION
Reports on the impact of author gender on the peer review process, and particularly manuscript acceptance rates, vary with discipline. This is, to our knowledge, the first report from a reproductive biology journal, and, encouragingly, suggests no bias against female authors during the peer review process. Although this did not result in a significant effect on outcome, female reviewers appear to be more favourable towards female authors than to male authors: this could be the result of a conscious or unconscious bias, potentially a reaction to the historic underrepresentation of females in the review process. It should be noted that at the time of writing, the editorial board is more balanced than it was towards the end of the study period, currently with 14 men and 12 women (AEs); from January 2023 the journal has one female co-EIC and one male co-EIC. For the period during which data was collected for this study, it is unknown whether AEs received any institutional training on unconscious bias: we suggest that journals encourage AEs to complete unconscious bias training. Determining the full impact of gender bias in publishing will take many such studies: we encourage other journals to conduct and report similar analyses.

AUTHOR CONTRIBUTIONS
All authors participated in design of the study. MB carried out statistical analyses. NS coordinated the study. MB, CAP and NS prepared figures/tables and wrote the manuscript. All authors helped draft the manuscript, and read and approved the final manuscript.