Pearls and pitfalls in letters of recommendation for neurology residency applications

Letters of recommendation are a cornerstone of residency applications. Variability and bias in letters exists across specialties, neurology being no exception. Studies done in other specialty fields assessing nuanced language uncovered key attention points for improvement and mitigation of bias, lessons from which should be applied in the field of neurology. We review common pearls and pitfalls in the letter solicitation, writing and reading process, with suggested best-practices for residency applicants, letter writers, and program faculty reviewers. We advocate for the thoughtful selection of writers, emphasis on highlighting professional skills, and attention to implicit bias. This discussion focuses on recommendations for US advanced or categorical neurology programs, but elements of this guidance may apply more broadly to fellowship and faculty promotion letters as well.


Context
In light of the diminishing availability of objective metrics for assessing residency candidates in the USA, including the recent transition to pass-fail grading for the USMLE Step 1, the suspension of USMLE Step 2 Clinical Skills, and an increasing number of medical colleges with pass-fail clerkship grading, residency programs increasingly rely on the subjective written components of applications [1].These subjective sections are namely letters of recommendation (LORs), Medical Student Performance Evaluations (MSPE) and personal statements.The MSPE has objective intention but is not without bias and subjective language content, demonstrated to vary by writer [2].Personal statements are limited to the applicant perspective and may feature common or contrived themes.As there is a great deal of variance in neurology education among medical schools, both domestic and international, a student's purported understanding of the field may be limited by only brief exposure and not be well reflected in their personal essays [3].This is where LORs serve as the keystone of an application.A letter writer with a strong sense of the intended discipline, typically a practicing clinician and/or researcher, is well-suited to assessing a candidate's aptitude for a given specialty.Writers should leverage their knowledge of what it takes to practice successfully in neurology in the crafting of applicant LORs and understand the importance of their professional perspective.
In the 2021 NRMP Program Director Survey, LORs (93.1%) and perceived commitment to specialty (93.1%) were the top two personal characteristics of applicants valued by residency program directors in making interview offers [4].This importance presents a unique opportunity as commitment to specialty as well as other highly-valued personal qualities can be effectively emphasized in a strong letter highlighting an applicant's strengths.As an authorship team comprised of a first-year neurology resident, a neurology residency program director, and two medical-education experts in neurology, we represent the major stakeholders (applicants, letter writers, and letter readers) in the residency application process and offer our guidance on best practices.

Guidance
As a first step, applicants are encouraged to speak to mentors or advisors about appropriate prospective letter writers.Applicants may initially desire letters from "high profile" neurologists despite not having worked with them closely, however such letters are often generic and lacking in specifics about the individual's skills and attributes [5].As such, these letters should not be advised.Once appropriately identified, potential writers should request the applicant's CV and personal statement, as well as meet with them to discuss career interests, goals, and potential letter highlights.Writers are advised to be candid with applicants as early as possible if they feel unable to provide an honest endorsement, have had insufficient exposure to the individual in an academic setting, or are not available to give a thoughtful letter adequate time.Writers must remain mindful that their professionalism in LORs ultimately reflects on the applicant.The invitation to submit a letter is precisely thatan invitation.The opportunity to speak on an applicant's character is a privilege and represents an extension of trust that must be shown due gravitas.Mindful usage of an applicant's preferred name, spelling, and pronouns throughout the letter is also paramount.
The format and presentation of a successful LOR should adhere to a standard general structure [6,7].High-quality letters should be written unhurriedly, use professional letterhead, and avoid impersonal, copypasted statements.Beginning with "Dear Program Director" is advised over nonspecific "Dear Sir/Madam" or "To whom it may concern."The introductory paragraph should highlight the writer's relationship to the student, state the position for recommendation, and firmly qualify the strength of this recommendation for neurology residency.The letter body (~3 paragraphs) should utilize a narrative structure to detail the experience of the writer with the applicant in praiseworthy terms.A strong letter blends professional attributes (clinical skills, work ethic, teamwork, accountability) with specific anecdotal evidence of an applicant's work and performance over time.For neurology residency letters, this may include specific comments with examples reflecting neurologic history taking, performance of the neurologic examination, and clinical reasoning.The letter should focus on these professional qualities and skills over personality, and may do well to frame a candidate's strengths and areas for potential growth in positive terms relating directly to the ACGME core competencies.As with personal statements, any anecdotes or patient stories should respect privacy protections and must directly emphasize applicant qualities or growth, not be added gratuitously without explanation of impact [8].The LOR is also an opportunity for writers to provide crucial context regarding the applicant's standing within their home program and performance in comparison to other candidates (for example, state the home program's desire to keep the candidate if possible).Writers should also feel empowered to privately solicit and include feedback on the student's performance from colleagues, other trainees, staff members, or relevant research personnel who know the applicant well.An effective LOR concludes with a brief summary (1-2 sentences) and definitive restatement of the writer's endorsement.Writers may invite further inquiry and make themselves available to programs who wish to clarify an applicant's attributes.Program directors should similarly feel empowered to reach out to letter writers if they are seeking further insight into an applicant's demonstration of program-specific core values.
(Table 1.Pearls & Pitfalls in Letters of Recommendation) Attention to word choice is critical.Studies unpacking the 'coded' language used by letter writers in the fields of pediatrics, internal medicine, surgery, and obstetrics & gynecology have explored the nuances of diction in LORs [9][10][11].Subtleties in phraseology, such as "I recommend" versus "I give my highest recommendation," or "showed improvement" versus "exceeded expectations" were shown to be differentially interpreted by program directors across specialties [10].While such studies have not been performed for neurology letters specifically, these results emphasize the importance of precise, deliberate language.Writers must strive for clarity while maintaining awareness of how terms may differ in connotation and interpretation.
Sensitivity to implicit bias is of particular import for program faculty reading LORs [1].Not only must readers consider the depth of the working relationship between a writer and applicant, but they must also actively filter for inappropriate bias, no matter how unwitting.The invocation of racial or gendered stereotypes (whether in describing an applicant's appearance, work ethic, home life, or qualifications) is lamentably pervasive [9,12,13].Women and underrepresented minorities in medicine are often the targets of doubt-raising language and are more likely to receive 'letters of minimal assurance' (15% for women compared to 6% for men) wherein half-hearted or bland words of support are employed (ex."solid student") [14].References to what writers may perceive as challenges overcome by the applicant specifically pertaining to their upbringing, identity, values, or financial means run the risk of reflecting implicit bias and are not appropriate fodder for LORs unless discussed with the student in advance and agreed upon.The rise and implementation of implicit bias training in academic medicine may help combat this issue in the future.Participation in such trainings for letter writers and readers may be encouraged at an institutional level [1].

Future directions & conclusions
To address potential variance and implicit bias in narrative LORs, an alternative (termed "Structured Evaluation Letters" or SELs) was proposed by the Coalition for Physician Accountability in their 2021 Undergraduate -Graduate Medical Education Review Committee report [15].The coalition advocated for the standardization of LORs using specialty-specific templates that directly assess observed core competencies.SELs would ideally serve to reduce bias and provide explicit performance-linked reviews using a convenient, readily-completable form.This initiative would also facilitate research and transparent external review of the admissions process.However, SELs are not yet in use by most specialty residency programs and more focused development is needed prior to implementation in neurology.A recent study addressing standardized letters of recommendation in a neurosurgery cohort reported low levels of interrater reliability and significant grade inflation (90% of letters ranked candidates in the top 25%), results which are comparable across other specialties that use structured letters [16].It is evident that further training of evaluators is needed prior to broad implementation of SELs to limit undue bias and variability.Medical colleges may consider publishing internal guidance on reading and writing the narrative standard LORs to facilitate consistency within their institutions in the interim [13].
It is prudent to also acknowledge the advent of Artificial Intelligence (AI) tools, such as ChatGPT, and their anticipated impact on LORs.Though these resources may tempt with the promise of efficient and polished-appearing outputs, caution is advised.AI-generated LORs should still be carefully edited and customized to each applicant.Original work is always preferrable.The onus is on letter writers to provide authentic, personalized letters that applicants universally deserve.
With the tools and systems available to applicants, letter writers, and program directors, strong LORs require attention to implicit bias and phraseology.We advocate for thoughtful solicitation, construction, and interpretation of LORs for neurology residency applicants.While our focus has been on the residency application process in the USA, this guidance may be translatable to an international audience as well as to fellowship or faculty LORs.With cognizance of the pearls and pitfalls described here, we hope that applicants can leverage meaningful relationships with mentors through mindfully-crafted, impactful letters to best position them for future success in neurology.