Predictive Value of Preresidency Academic Metrics on Resident Publication Potential

Abstract Objective Otolaryngology residency is highly competitive, and applicant academic metrics are scrutinized. The predictive value of preresidency academic metrics on applicants’ future research productivity and career aspirations remains largely undefined. Study Design Retrospective cohort study Setting Academic otolaryngology department, 2014 to 2015. Methods Applicant demographics, publication history, and United States Medical Licensing Examination (USMLE) scores were downloaded from Electronic Residency Application Service archives. Publications during residency were tallied from all PubMed articles indexed between July 1, 2015 and June 30, 2020. Postresidency career paths were examined by 2 investigators (D.J.C. and L.X.Y.) using Google searches with an emphasis on program websites, Doximity, and LinkedIn profiles. Associations with publication potential and postresidency positions were evaluated with Spearman rank correlation coefficients and Kruskal‐Wallis, Wilcoxon rank sum, and χ 2 tests. Results Of 321 applicants, 226 (70%) matched, and 205 (64%) completed residency by June 2020. Matched residents published a median of 4 (range: 0‐41) manuscripts during residency. USMLE scores, Alpha Omega Alpha status, and the number of preresidency publications did not significantly correlate with publication potential during residency. The number of research experiences had a significant positive correlation with publications during residency (p < 0.001). Asian race (p = 0.002) and geographical region of residency (p < 0.001) also had significant associations with publication potential. Of the 205 graduates, 118 (58%) enrolled in fellowship. Age and female sex (74% vs 48%; p = 0.002) were the only factors significantly associated with pursuing a fellowship. Conclusion In otolaryngology, not all preresidency academic metrics are associated with publication potential during residency or propensity for fellowship training. Programs should not use academic metrics alone to predict an applicant's future research productivity or career trajectory.

Methods. Applicant demographics, publication history, and United States Medical Licensing Examination (USMLE) scores were downloaded from Electronic Residency Application Service archives. Publications during residency were tallied from all PubMed articles indexed between July 1, 2015 and June 30, 2020. Postresidency career paths were examined by 2 investigators (D.J.C. and L.X.Y.) using Google searches with an emphasis on program websites, Doximity, and LinkedIn profiles. Associations with publication potential and postresidency positions were evaluated with Spearman rank correlation coefficients and Kruskal-Wallis, Wilcoxon rank sum, and χ 2 tests.
Results. Of 321 applicants, 226 (70%) matched, and 205 (64%) completed residency by June 2020. Matched residents published a median of 4 (range: 0-41) manuscripts during residency. USMLE scores, Alpha Omega Alpha status, and the number of preresidency publications did not significantly correlate with publication potential during residency. The number of research experiences had a significant positive correlation with publications during residency (p < 0.001). Asian race (p = 0.002) and geographical region of residency (p < 0.001) also had significant associations with publication potential. Of the 205 graduates, 118 (58%) enrolled in fellowship. Age and female sex (74% vs 48%; p = 0.002) were the only factors significantly associated with pursuing a fellowship.
Conclusion. In otolaryngology, not all preresidency academic metrics are associated with publication potential during residency or propensity for fellowship training. Programs should not use academic metrics alone to predict an applicant's future research productivity or career trajectory. O tolaryngology is consistently one of the most competitive surgical subspecialties in the residency match. The popularity of the field, the limited availability of training positions, and the anxiety of going unmatched have created a bottleneck effect in the application process. 1,2 There has been a consistent increase in the number of prospective applications per program, which strains the ability of training programs to provide each application with a truly holistic review-a crisis that has only partially been mitigated with the initiation of a signaling system. 3 To efficiently vet applicants, many programs have used the United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha (AOA) status, medical school ranking, and prior research experience. 4 Nearly half of US otolaryngology programs admit to using USMLE Step 1 cutoff scores when reviewing applications. 5 Medical school tier is often associated with resident performance, although lower-ranked schools have been shown to correlate with higher performance. 6 AOA status and publications are significantly associated with odds of matching into otolaryngology residency. 7 Yet the reliance on these traditional academic markers of success may be misguided, as studies have shown these may not accurately predict clinical success or acquisition of surgical skills during training. [8][9][10] The impact of these traditional selection metrics on performance during residency and beyond has not been adequately studied through the lens of publication potential and the pursuit of fellowship after graduation. Research has become a required component of otolaryngology training 11 and acts as both a measure of academic success after training. 12 The expectation among otolaryngology programs for applicant research experience 4 has fueled the significant rise in preresidency research activity and participation in a research gap year. 13 It remains unclear whether this reflects the dedication to research and future career aspiration, or simply a short-term goal based on the perception that research experience is a prerequisite to a successful match.
In this study, we aim to explore associations between academic metrics in residency applications and the publication potential and career aspirations of matched applicants in otolaryngology. We hypothesize that USMLE Step scores and AOA status do not predict an applicant's propensity to publish during residency, enroll in fellowship, or enter academic or private practice after graduation.

Participant Identification
All applicants to the authors' Accreditation Council for Graduate Medical Education accredited 5-year otolaryngology training program in the 2014 to 2015 application cycle were identified using the Electronic Residency Application Service (ERAS). This study was deemed exempt by the Mayo Clinic Institutional Review Board (IRB 20-001640). Full applications including sections on applicant demographics, publication and presentation history, work experience, academics, and USMLE Step 1 and Step 2 scores were downloaded from the ERAS archives.

ERAS Data Extraction
Demographics extracted from ERAS included birthdate, sex, race and ethnicity, birthplace, participation in the couples match, and citizenship status. Race and ethnicity were categorized as Asian, black, Hispanic, Middle Eastern, Mixed, Native American, white, or "prefer not to say" as reported in ERAS. Birthplace was categorized as within the United States or in a foreign country. Within the United States, birthplace was further categorized as regions in the Northeast, Southeast, Southwest, Midwest, or West. Medical schools were classified as US allopathic, US osteopathic, or foreign. Applicants from US medical schools were further subdivided by the regions summarized above. USMLE board scores were extracted.
Applicant AOA status was categorized as AOA, non-AOA, or AOA is not announced/awarded at the applicant's home school. Advanced degrees were noted. An applicant's undergraduate field of study was categorized as sciences and health, arts and humanities, business, multiple majors, or unknown. Preresidency research was measured as the number of posters, oral presentations, research experiences, total publications, first-author publications, and submitted publications reported on ERAS.

Publication Potential and Career Trajectory
All matched applicants were followed to determine the residency program they attended and their career placement immediately after graduation. Several internet sources were used to determine an applicant's ultimate training program and postresidency career trajectory, including Google searches with an emphasis on Doximity, LinkedIn, published fellowship match results, and academic department websites. These searches were performed by 2 independent reviewers (D.J.C. and L.X.Y.). Residency programs were categorized by US region as summarized above. Residency programs were ranked using the Doximity Residency Navigator function, sorting programs by "research output." Of note, this methodology for finding career trajectory only confirms which applicants ultimately matched and enrolled into fellowship programs, but does not provide information on which applicants applied to but were not selected for fellowships, nor does it provide information on which applicants ultimately graduated from fellowship training. Publication potential during residency was defined as the number of PubMed-indexed publications for each applicant between July 1, 2015 and June 30, 2020. PubMed searches were conducted by 2 independent reviewers (D.J.C. and L.X.Y.) by querying each applicant by Last Name, First Name as well as Last Name, FM where F is the first name initial and M is the middle name initial. Publications were then reviewed in detail to identify the authors' institutional affiliations, confirming each PubMed-indexed study as authored by the applicant rather than by another author with a similar name. An attempt was made to identify the H-index of applicants using Google Scholar, but most applicants did not have profiles and therefore the H-index could not be used as a metric to measure publication potential in this cohort.

Statistical Analysis
Continuous features were summarized using medians, interquartile ranges, and ranges, and categorical features were summarized using frequencies and percentages. Features of matched and unmatched applicants were compared using Wilcoxon rank sum and χ 2 tests. Associations between the number of publications during residency and ERAS features were evaluated using Spearman rank correlation coefficients, Kruskal-Wallis tests, or Wilcoxon rank sum tests. Associations between postresidency career path (private practice, academic general practice, or fellowship) and ERAS features were evaluated using Kruskal-Wallis, χ 2 , or Fisher exact tests. Statistical analyses were conducted using SAS version 9.4 (SAS Institute). All tests were 2-sided and p < 0.05 were considered statistically significant.

Applicant Demographics and Matched Applicants
In the 2014 to 2015 application cycle, the residency program received 337 applications; 16 were excluded due to incomplete information received from ERAS, leaving 321 applicants for study ( Table 1). Most applicants selfidentified as male (64%) and white (58%). The majority were born in the United States (77%) and current US citizens (93%), and 226 (70%) matched into otolaryngology ( Table 2).

Academic Potential of Matched Applicants
During residency, matched applicants published a median of 4 papers (range: 0-41; Table 3). After 5 years of residency training, just over half of matched otolaryngology residents enrolled in further subspecialty training in fellowship (n = 118, 52%). An additional 71 (31%) went into private practice, 16 (7%) entered general academic or academic-affiliated practice, and 13 (6%) remained in training as T32 research track residents. Eight (4%) of matched residents did not graduate or complete residency by the end of the academic year in 2020 ( Figure 1).

Associations Between ERAS Application Features and Future Academic Potential
Associations between publication potential during residency and preresidency applicant features from ERAS are summarized in Table 4. Asian residents published significantly more than white residents (median 6 vs 4 publications, p = 0.002). Applicants who attended residency in the Northeast or West Coast and at the top 50 research output programs according to Doximity Residency Navigator published significantly more than residents at other programs (p < 0.001 for both). USMLE scores, AOA status, and the presence of advanced degrees were not significantly associated with increased publication potential during residency. The number of poster presentations, research experiences, and unpublished submissions before residency showed a statistically significant but weakly positive correlation with publication potential during residency.
Associations between future career paths and preresidency applicant features are shown in Table 5. The only features significantly associated with postresidency career plans were the age and sex of the applicant before residency. Graduates going straight into general academic or academic-affiliated practice were younger (p = 0.026).
The majority of female (n = 56, 74%) graduates entered fellowship, which was a significantly higher proportion compared to men (p = 0.002).

Discussion
This is the first study examining the association between academic metrics provided on ERAS applications and the future publication potential and career trajectory of residents matching into otolaryngology. Overall, USMLE Step scores and AOA status were not predictive of resident publication records during training or enrollment in advanced fellowship after graduation. The number of research experiences and poster presentations before residency was associated with publication potential after the match. Asian residents and those who attended residency on the Northeast or West Coast had significantly more publications than other demographic groups. The only features that were significantly associated with enrollment in advanced fellowship training after residency graduation were the age and sex of the applicant before residency, with women significantly more likely to enter fellowship training than men. These findings indicate that traditional academic metrics used to screen applicants by many selection committees do not predict future publication potential.
Otolaryngology is a competitive specialty. In 2021, the overall match rate in otolaryngology was only 62%, with a match rate among US MD seniors of only 77%. 1 In 2021, the average applicant applied to more than half of all US otolaryngology training programs. 3 This is partially due to the decreased financial burden of the virtual interview interface adopted during the COVID-19 pandemic. With the surge of applications per program, detailed application review is more time-consuming. Often, it is easier for selection committees to fall back on traditional academic metrics to predict resident performance. A review in 2016 revealed that otolaryngology applicants had a higher USMLE Step 1 score (248 vs 233) and a higher proportion of AOA members (45% vs 17%) compared to all other specialties. 4 However, the reliance on these academic metrics can in fact create biases in the selection process. While USMLE Step scores appear to predict performance on written boards, 14 most academic otolaryngology faculty do not believe that it correlates with patient care or surgical skills. 5 The poor discriminating ability of USMLE scores may lie in the high and narrow score range seen among matched applicants. While significantly low scores (ie, below 200) may predict a lower propensity for success in residency, the narrow range of scores among most applicants into otolaryngology makes this a nondiscriminating academic metric. In fact, the test was not designed or validated to be used to discriminate applicants performing at the highest end of the score distribution. Furthermore, when used for resident selection, USMLE scores can create sex and racial biases-women and  underrepresented minorities on average have lower USMLE Step scores compared to their demographic counterparts. 15 Similarly, only 2.2% of AOA members identify as black, and white students are much more likely to be selected for AOA membership than black students. 16 As a result, an emphasis on these standard metrics could lead to further underrecruitment of women and underrepresented minorities into our field-one that  (17)  struggles more than most surgical subspecialties in its recruitment of Black trainees. 17 The search for other metrics to predict resident success is even more pressing in the setting of USMLE Step 1 going to a pass/fail system. 5 Our results suggest that the volume of research experiences, not publications, is associated with research success during residency. These results may reflect the challenges that medical students face in seeing research projects to full publication. Longitudinal projects can take years to complete, and placing undue emphasis on the number of publications students should achieve may deter students from joining in on long-term projects in favor of shorter retrospective studies or case reports. When it comes to research, it appears that the effort students put in, as measured by the number of experiences, is a better predictor of enduring success than the end product of publications.
Research experience too must be used conservatively in the selection process. In 2020, matched applicants into otolaryngology had an average of over 6 research experiences and over 13 abstracts, presentations, and publications. 18 Yet research productivity early in training does not necessarily lead to a career as a clinicianscientist. Clinical responsibilities during residency are time-demanding, and there are many perceived obstacles to pursuing a career in research including a low salary, high competition for funding, and lack of departmental support. 19 Outside of clinical responsibilities, residents may choose to engage in other academic pursuits such as   quality improvement or leadership positions within their institution or national societies. With limited free time, residents may also choose to spend any time away from clinical responsibilities with family-caring for children or elders. Greater research productivity early in training may lead to greater burnout, which decreases research productivity later on. Moreover, a focus on research experience prior to residency can further discriminate against underrepresented minorities, as most academic centers do not offer funding to trainees seeking to take a research gap year and underrepresented minority students may already have less robust academic networks and financial resources. 17 In fact, underrepresented minority students report significant difficulty in finding mentorship, particularly race-concordant mentorship, within otolaryngology. 17 Without mentorship, these students may face even greater challenges in gaining the preresidency research experience that some training programs may expect. The association between demographic features and research productivity during otolaryngology residency appears to be novel to this study. Asian race was associated with a significantly higher number of publications during residency compared to Whites. Unlike other surgical subspecialties, Asians are overrepresented within the field of otolaryngology. 20 As such, the drive for Asian residents to publish could be related to the "model minority" mentality 21 and the perceived notion and pressure that career advancement in the field requires further academic success compared to their White counterparts. In reality, however, applicants who selfidentify as Asian via ERAS can come from culturally diverse and heterogeneous backgrounds-some of which may remain underrepresented. We also found that residents who trained or attended medical school in the Northeast or West had a significantly higher number of publications compared to those in other regions. This is likely related to the density of highly funded and productive research universities along the two coasts, as studies have shown that faculty at academically competitive universities are more productive than those in less competitive environments. 22 In fact, in this cohort, residents at the top 50 research output training programs, per Doximity Residency Navigator, published significantly more than residents at other programs.
Finally, our results showed that women were significantly more likely than men to enroll in fellowship training, contradicting other studies on the representation of women in otolaryngology. This finding could simply be because more women than men are pursuing careers in academia. On the other hand, in 2017, although over 35% of trainees were women, women represented only 14.5% of practicing otolaryngologists. 23 In 2020, there were only 5 female chairs in otolaryngology, representing only 5.1% of chair positions. 24 The discrepancy between the poor representation of women in otolaryngology compared to the high proportion of women pursuing additional training could be attributed to the perception among women that additional training is necessary to gain respect from their male counterparts and to advance their careers. Nearly one-fifth of women feel that their gender can hinder their career advancement. 23 Given the low number of women in academic otolaryngology, fellowship training could be 1 avenue for women to pursue additional career mentorship. Finally, genderbased discrepancies in surgical case volumes among otolaryngology residents may lead some women to feel that they did not gain sufficient experience during residency. 25 The future of resident selection in otolaryngology must take on a more holistic approach. In addition to not predicting an applicant's clinical success, our results show that USMLE scores and AOA status also do not predict publication potential or future career trajectory. Furthermore, many of these academic requirements can discriminate against underrepresented minority students 17 and widen the gap in gender and racial representation seen in this field. In recent years, many residency programs have changed their approach to the selection process, including "unblinded" application reviews to  limit implicit bias and the adaptation of algorithms that do not solely rely on traditional metrics to identify strong diverse trainees 26 This study is not without limitations. This is a retrospective study examining a single application cycle into otolaryngology, with no long-term follow-up and details regarding career trajectory after fellowship. Publication potential was measured only in the number of publications, and not in the impact of those publications, as the H-index of most applicants could not be found. In addition, we recognize that publication potential only represents 1 facet of resident academic success during training. Success in research may not, and likely does not predict resident performance on board examinations, acquisition of surgical skills, clinical judgment, or bedside manner-all of which are equally important measures of competency in surgical training. We also do not advocate that future career trajectories in fellowship, academia, or private practice should influence a training program's decision to select applicants. In reality, the decision to enter a certain career trajectory is multifaceted, and mixed methods studies are likely needed to fully understand the motivations behind a trainee's career choice. Ultimately, programs should aim to select a diverse group of trainees with individualized career aspirations and provide excellent training to residents who should graduate well-equipped to enter any practice environment that they desire. Finally, it is important to acknowledge that the retrospective associations between demographic factors and academic performance discovered in this study do not imply causality. They should not be used by programs as a reliable predictor of future success or used in selection algorithms going forward.

Conclusion
In otolaryngology, not all preresidency academic metrics are associated with publication potential during residency or propensity for postresidency fellowship training. Research experience prior to residency is associated with greater research output during training, but USMLE scores, the total number of publications and AOA status are not. Programs should not use academic metrics alone to predict an applicant's future academic success or career trajectory. A diverse range of academic and life experiences should be considered in the selection of the next generation of otolaryngologists.