If You Build It, Will They Come? An Analysis of Candidate Attitudes Toward A New Residency Program

This article was migrated. The article was marked as recommended. Background: To address looming healthcare workforce shortages, the Association of American Medical Colleges (AAMC) has recommended a substantial increase in residency positions. However, new residency programs face unique recruitment challenges. Objective: To identify new program characteristics and recruitment practices that most influence candidate-ranking decisions. Methods: In a post-match survey, applicants who interviewed during the first two recruitment seasons of a new program were asked to rate program characteristics and recruitment techniques regarding the effects they had on ranking decisions, and to describe the most attractive and concerning features. Somers’ D directional coefficients were calculated to determine the correlation between applicants’ ranking of the program and ranking of survey items. Qualitative responses were analyzed as word clouds. Results: 163/349 surveys were returned (47%). The strongest correlating items included the opportunities to work closely with faculty, to help shape a new program, and to be one of the first graduates. Influential recruitment practices included program communications and website vividness. Concerns included lack of senior residents, fear of unforeseen difficulties, and no evidence of outcomes. Conclusion: New programs have unique features that influence candidates’ ranking decisions. Recruitment practices should be designed to identify candidates who embrace the unique opportunities of a new program.


Introduction
The international physician workforce is already strained to meet the needs of patients, described recently as a "global crisis" (Sirili et al., 2017), and there have been increasing calls for expanded medical training opportunities globally (Ibrahim et al., 2016, Sirili et al., 2017, Domagala and Klich, 2018).In the United States, the Association of American Medical Colleges (AAMC) estimates a shortage of 61,700 -94,700 physicians by 2025(Dall et al., 2016)).The AAMC reported in 2015 that first-year medical student enrollment is projected to grow 30% between 2002 and 2018 (AAMC Center for Workforce Studies, 2016); however, the subsequent need for more graduate medical training positions has not been addressed.One solution is to develop 3,000 Medicare-funded resident training positions per year between 2017 and 2021 (AAMC Center for Workforce Studies, 2016).Since most existing programs are limited in the growth they can support, new residency programs are needed and have specific logistics for being created (Barajaz and Turner, 2016).
It can be challenging to attract applicants to a program with no proven track record.An 8-year study of applicants to programs at Massachusetts General Hospital revealed that two key attractive features were resident morale and preparation for future training/career (Phitayakorn et al., 2015).In the 2015 National Residency Matching Program (NRMP) applicant survey, seven of the top 16 factors important to ranking decisions (reputation, resident quality, work/ life balance, house staff morale, future fellowship opportunities, career paths of graduates, and preparation for fellowship training) cannot be assessed in new training programs (NRMP, 2015), and outcomes data such as board passage rates are likewise unavailable.Candidates for a new program might be apprehensive of the untried nature of the program, lack of upper-level residents, and level of institutional support.Conversely, a new program offers unique opportunities that can prove attractive (e.g., close working relationships with faculty, the ability to shape and influence a new program).Identifying these factors and determining their roles in ranking decisions can inform both the recruitment and selection processes.
Business literature offers cognitive frameworks surrounding acceptance decisions with regard to the features that attract or concern applicants.The Cornell Center for Advanced Human Resources Studies (Rynes, 1989) offers two theories related to job selection: 1) The Objective Factor Theory (Maclean and Pato, 2011) which states that applicants are influenced by the attributes of the job, how well that job is "sold" to them, and the organization's reputation and stature, and 2) the Critical Contact Theory, which states that applicants are influenced by comments and behaviors of the interviewer as an organizational representative.Both frameworks demonstrate that the organization itself greatly influences the recruitment process.Hence, identifying the qualities that are most important to applicants in the unique situation of interviewing at a new program is critical to the quality of recruitment outcomes (Schreurs and Syed, 2011).For instance, in one psychiatry program, a change in recruitment process yielded a sustained increase in fill rates and median test scores (Maclean and Pato, 2011).
The factors that are important to applicants at a new residency program have not been reported.We surveyed interviewees at the new Baylor College of Medicine (BCM) Pediatrics Residency Program at The Children's Hospital of San Antonio (CHofSA) during the first two recruitment seasons.This is an ACGME accredited program that accepted 10 residents per year in the first two recruitment cycles (2015 and 2016).Our primary training site is a new free-standing children's hospital with 190 inpatient beds.It includes a 60 bed level 4 Neonatal Intensive Care Unit, a 40 bed combined Pediatric Intensive Care/Intermediate Care Unit, and a 48 bed emergency room.It is staffed with approximately 200 faculty across general pediatrics and the medical and surgical subspecialties.Applicants rated the impact of specific program characteristics and recruitment practices on their decisions to rank the program.The results revealed distinct features of a new program that strongly correlated with applicants' ranking decisions.This study should assist leaders of new residency programs in designing materials and practices to optimize their recruitment.

Study participants and eligibility
Institutional Review Board approval was obtained from both BCM and Christus Santa Rosa Health System, the parent entity of CHofSA.Eligible participants interviewed at CHofSA for the 2015 and 2016 categorical pediatrics residency match.An anonymous survey was sent electronically (https://www.surveymonkey.com) in May-June 2015 to the 201 applicants who interviewed at CHofSA in its inaugural recruitment season, and in April-May 2016 to the 148 applicants who interviewed during the second recruitment season.

Post-interview survey
The survey requested basic demographic information (Table 1), including sex, medical degree, location of medical school (in state = Texas, out of state, or international medical school), number of interviews the applicant attended, number of programs the applicant ranked, and how the applicant ranked our program.
The applicants were then asked to rate specific program characteristics on a 7-point Likert scale as to the effect they had on the applicant's ranking decisions for our program (Tables 2 and 3).Program characteristics included items common to all programs, such as curriculum, benefits, and patient volume, as well as items specific to a new program such as lack of upper level residents, newness of the program, the opportunity to work one-on-one with faculty, and the opportunity to help shape the program.The survey also assessed items related to recruitment practices (Table 4), including the vividness and early readiness of a program website, availability of a pre-interview season open house, and opportunities to meet with faculty during the interview day.
Additionally, a free response section on the survey solicited respondents to express their top concerns and motivators while considering this new program and the recruitment practices that did help or which would have helped to alleviate any concerns.

Statistical analysis
For each Likert scale response, a Somers' D directional coefficient was calculated based on how respondents ranked the program to determine the correlation between each characteristic or practice and an applicant's likelihood of ranking the program highly.Program characteristics and recruitment practices were assigned a value of 1 (large or moderate positive effect), 2 (slight positive, no effect, or slight negative effect), or 3 (moderate or large negative effect) on ranking the program.The respondents' rankings of the program were given a value of 1 (first preference), 2 (second preference), 3 (ranked between 3 and 5), 4 (ranked below 5), and 5 (not ranked).SAS software v9.4 (Cary, NC) was used to generate the Somers' D values, 95% confidence intervals, and p-value for each survey item.We accepted two-sided α ≤ 0.05 as indicative of statistical significance.

Qualitative analysis
Free text responses on the survey were analyzed by an online word cloud generator (https://www.wordclouds.com)with the frequency of words within responses graphically represented in larger text to illustrate popularity.Questions included in this analysis were: (1) "What were the greatest attractive features of joining a new program?"(2) "What were your biggest concerns about joining a new program?"and ( 3) "What aspects of your interview day were most helpful in your evaluation of Baylor San Antonio as a new program?"

Results/Analysis
In 2015, 72 of 201 (35.8%) surveys were returned by the first class of interviewees.In 2016, 91 of 148 (61.5%) surveys were returned by the second class of interviewees.The overall response rate was 163/349 (46.7%).
Each program characteristic (Tables 2 and 3) and recruitment practice (Table 4) response in the survey is accompanied by a Somers' D coefficient, 95% confidence interval, and p-value.A coefficient closer to 1 indicates that the item was more likely to be ranked positively by high rankers and negatively by low rankers.A coefficient closer to 0 indicates that the item exhibited poor correlation between positive and negative responses among high and low rankers.Negative coefficients suggest negative responses among high rankers and positive responses among low rankers; no statistically significant negative coefficients were observed.Most of these characteristics also showed significant correlations, although several program features did not have a significant correlation with ranking decisions.These less-impactful characteristics included: recommendation by residents and/or students at the home institution, recommendation by faculty mentor, intern night shift system, and the department chair.
The recruitment practices assessed in our survey are shown in Table 4.Although none of these practices are limited to new residency programs, several practices describing written and in-person communications related to the interview had significant positive correlations.We note that the strength of correlation (i.e., magnitude of Somers' D coefficient) is lower among the recruitment practices than was seen among the most impactful program characteristics in Tables 2 and 3.
The free responses in the survey were analyzed qualitatively by word cloud generation.The first question, "What were the greatest attractive features of joining a new program?"yielded 41 responses (Figure 1A).Similar to the quantitative data (Table 2), responses frequently included the desire to shape a new program and aid in its growth.By contrast, the second question, "What were your biggest concerns about joining a new program?"yielded 43 responses (Figure 1B) and reflected an uncertainty about outcomes including fellowship match data, lack of upper-level residents, and experience of program faculty.To assess recruitment practices, the third question asked, "What aspects of your interview day were most helpful in your evaluation of Baylor San Antonio as a new program?"and yielded 38 responses (Figure 1C).Reflecting some of the positive communication themes from the quantitative data (Table 4), responses included the interviews and interactions with faculty and program leadership.Respondents to the 2016 (second year) survey also reported meeting with the first-year residents during the interview day as beneficial.

Discussion
This study identified the characteristics of a new residency program and the recruitment practices that appear to have the greatest influence on applicant rank order.Although applicants have reported to NRMP the factors that are most important in determining their ranking decisions (NRMP, 2015), several of these factors cannot be measured in new programs.Conversely, certain characteristics unique to new programs are not included on the NRMP standard survey.
Our survey results are the first to provide specific information on characteristics and recruitment practices that may make a new residency program attractive to applicants.
Several characteristics unique to a new program correlated highly with attracting certain applicants.They included helping to build a new program and working closely with faculty without the traditional learner hierarchy (Table 2 and Figure 1A).Interestingly, the lack of senior residents was also listed as a chief concern by some respondents (Figure 1B), illustrating one difference between applicants who might or might not be a good fit for a new program.
Our survey response rate (46.7%) from applicants to our program is comparable to the overall response rate of 47.5% reported nationally for the 2015 NRMP Applicant Survey (NRMP, 2015).Approximately 75% of our respondents interviewed at and ultimately ranked 5-15 programs (Table 1), congruent with data of U.S. medical school seniors who matched to pediatrics in 2015 (median number of interviews/candidate = 12; median number of programs ranked/ candidate = 11).We noted a higher response rate in the second recruiting year (62% in 2016 vs. 36% in 2015).We attribute this difference to sending the 2016 survey to interviewees four weeks earlier than we did in 2015, which may have influenced the likelihood of completing the survey.Our study has several limitations.There may have been an association bias, such that respondents who were favorably disposed to our program may have answered all questions more favorably, and vice versa.It is difficult to assess this effect, although we note multiple non-significant results in Tables 3 and 4 that counter suspicion of this bias.We initially considered the possibility of response bias in our survey, such that applicants who ranked us higher would be more likely to respond.Instead, the observed result (Table 1) was that 25% of the respondents ranked us first or second, whereas the remainder ranked us lower or not at all, indicating that our respondents were more broadly representative of the total pool of interviewees.Our findings represent a single new training program and may not directly generalize to all new programs.For example, one of the survey items specifically asked about our academic affiliation with BCM (Table 3) and garnered a significant positive response.It is possible that applicants to our new program had different overall application patterns than those that applied only to established programs; our study was not designed to identify these potential differences.Since each established and new training program has inherent differences, and since nationally-available data does not adequately capture the qualities that could make new programs distinctively attractive to applicants, we share our findings with the intention of identifying themes that could be used to emphasize the value of new programs during recruitment.

Conclusion
This study demonstrates that the unique qualities of a new program and communication-focused recruitment practices were attractive to a distinct subset of applicants to a new residency program.As the gap between applicants in the NRMP and available PGY-1 positions continues to widen, with 42,370 applicants for 27,860 positions in the 2016 match (NRMP, 2016), the need for new residency training opportunities becomes more acute with each passing year.Our findings will help leaders of future new programs develop recruitment materials and interview practices to highlight the benefits of joining a new program, to address the concerns that applicants might have with the unproven nature of a new program, and to attract applicants who are the best fit for the opportunities and challenges of a new residency program.We look forward to more systematic evaluation of recruitment practices across new programs to help guide this important approach to solving the global healthcare workforce dilemma.

Take Home Messages
New residency programs are an essential tool to help meet projected physician workforce shortages, but face unique recruitment challenges.
Certain factors related to a new program influence ranking decisions significantly.
A subset of applicants may find attributes of a new program attractive.
Communication-based recruitment practices can help identify and attract best-fit applicants to a new program.Notes On Contributors Drs.Barajaz, Denniston, and Wolfe (ORCID 0000-0002-3113-2298)

Permphan Dharmasaroja Faculty of Science, Mahidol University
This review has been migrated.The reviewer awarded 4 stars out of 5 The authors identified the characteristics of the new BCM pediatrics residency program and the recruitment practices that might influence the applicants' ranking decision by using an online survey that involved items common to all programs and also those specific to the new program.
Furthermore, an open-ended question section was additionally included in order to identify major concerns and motivators.The results provide useful information that could be applied to other residency training programs to recruit potential applicants.However, there are some points that the authors may be able to clarify prior to further implementation, as follows.1.The specific items asked in the survey regarding the new program should be clearly described.All programs that are claimed to be new usually have some new aspects.What does the term "newness of the program" mean?An example of "the opportunity to help shape the program" should also be given.These details will make the questions asked in the survey more objective, and the applicants who are interviewed will have a similar understanding.2.For the free responses in the survey, is it possible that the answers were influenced by the list of the characteristics of the new program provided in the section before, resulting in the responses being similar to the quantitative data?3.In the discussion section, in order to make the authors' statement that the characteristics of the new program correlate highly with attracting a distinct subset of applicants more

Figure 1 .
Figure 1.Word cloud analysis of free responses on survey.Larger text size correlates with frequency of the word's use in the survey responses.A, responses to question, "What were the greatest attractive features of joining a new program?"based on 41 responses.B, responses to question, "What were your biggest concerns about joining a new program?"based on 43 responses.C, responses to question, "What aspects of your interview day were most helpful in your evaluation of Baylor San Antonio as a new program?"based on 38 responses.

Table 1 .
Demographics of survey respondents

Table 2
lists the survey items that assess program characteristics intrinsic to our new training program.All of these items showed significant positive correlations, indicating that they were considered more appealing program features to high rankers and more concerning to low rankers.Table 3 lists the survey items that assess characteristics not distinctive of a new training program.

Table 2 .
Impact of characteristics of a new program on applicant rank decision *p ≤ 0.05

Table 3 .
Impact of characteristics not distinctive of a new program on applicant rank decision *p ≤ 0.05

Table 4 .
Impact of recruitment practices on applicant rank decision *p ≤ 0.05 work in educational leadership for the Baylor Pediatric Residency Program at The Children's Hospital of San Antonio.Ms.Shelley Kumar is a statistician and researcher for the Center for Research, Innovation, and Scholarship in Medical Education at Texas Children's Hospital.West, T., Chakrabarti, R., Reynolds, R., et al. (2016) The Complexities of Physician Supply and Demand 2016 Update: Projections from 2014 to 2025.Prepared for the Association of American Medical Colleges.Available at:Thankyou for this submission I found it really interesting.New programmes attract different groups for differing reasons and this study helps us understand what those reasons may be.The text reads easily and although you bring in multiple tables you have made the article clear.The tables should make things clearer but table 1,3 and 4 are complex and not clear on quick view.I would suggest simplifying and grouping the areas so they have greater impact.Table2is simple.You do not need to asterisk and quote the p value, one or the other is fine.Figure1is very clear and has a good impact.Your discussion seems rushed and does not, to my mind, cover the key themes of the study.Overall this is an important piece of work for anyone implementing a new programme but could be re-written to simplify the results, expand the discussion and grouping of the themes.No conflicts of interest were disclosed.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.