Twelve Tips on Implementing Multiple Mini- Interviews in a Hybrid Admissions Model

Background: Multiple mini-interviews (MMI) were proposed almost a decade ago as an alternate means of assessing medical school applicants. MMIs are similar to an OSCE (Objectively Structured Clinical Examination) in that applicants complete a series of brief encounters focused on non-clinical topics. Aim: To assist those considering the addition of MMIs with standard interviews in a hybrid admissions model. Methods: The tips are derived from personal experience with a hybrid model at one academic institution. Results: The twelve tips describe approaches for obtaining early faculty buy-in, designing a floor plan and timetable, aligning MMI content with the program mission and goals, conducting faculty training, developing contingency plans, managing applicants’ expectations, encouraging committee participation, promoting holistic review of each applicant, utilizing the benefits the model allows, changing scenarios periodically, studying the model, and sharing the lessons learned. Conclusion: It is hoped these suggestions are helpful to those considering a hybrid model for their institution.


Introduction
Multiple mini-interviews (MMI) were proposed over a decade ago (Eva et al. 2004a) as an alternate means of assessing medical school applicants. The traditional admissions process typically includes a standard interview of Bing-You R, Linder J, White P, Neumeyer D MedEdPublish https://doi.org/10. 15694/mep.2016.000164 Page | 2 30-60 minutes with a small number of individual faculty members. The MMI differs from a standard interview by having applicants complete a series of brief stations very similar to an OSCE-like format. These brief MMI encounters, which may only last 8-10 minutes, usually focus on non-clinical topics (e.g., listening skills, critical thinking skills).
MMIs have been shown to be reliable (Eva et al. 2004b) in attempting to assess the non-cognitive aspects of applicants beyond the traditional academic metrics of grade point averages and MCAT (Medical College Admission Test) scores. A recent systematic review (Patterson et al. 2016) provided evidence that MMIs, aptitude tests, and situational judgment tests are more effective and generally fairer than traditional interviews, references and personal statements. MMIs are able to assess a broad array of applicants' attributes, with a reduction in bias and errors due to the use of multiple interviewers. The use of MMIs has been extended to the residency interview process (Dore et al. 2010, Hormeister et al. 2008) and also to other graduate level schools like pharmacy (Kelsch & Friesner 2012). In terms of predicting performance measurements later in clinical training, MMI scores have been found to correlate with future OSCE scores (Eva et al. 2004c) and national licensing examinations (Eva et al. 2012).
We implemented the MMI for the "Maine Track" program five years ago. The Maine Track program is a partnership between Maine Medical Center (MMC) and Tufts University School of Medicine (TUSM) which began in 2008 (Bing-You et al. 2010). This partnership was formed in part to address the workforce needs of Maine, particularly in rural areas. A separate admissions subcommittee of Maine-based TUSM faculty enrolls up to forty students for this unique program. Students spend the majority of the first and second years in Boston, and the remaining third and fourth years in Maine. For the initial two recruitment years we utilized two 1-hour long standard interviews. Because we occasionally found the two interviewers had diametrically opposing viewpoints of the same candidate, and with the growing literature describing MMIs, we decided to explore the use of MMIs in our admissions process.
An organization can decide whether their process will only consist of MMI stations (Eva et al. 2004a) or include standard faculty interviews. Others may choose to simply increase the number of standard interviews (Axelson & Kreiter 2009). We chose a hybrid model of combining both MMI stations along with traditional interviews partly due to concerns expressed by applicants during our pilot year regarding the MMI's inability to expand on their complete background without a standard interview. Our faculty members were also concerned that solely using MMI stations would not allow for exploration of specific issues found in the applicant's file (e.g., institutional actions, withdrawal from courses, or other unique experiences). We began utilizing this hybrid model in 2011 when there were no published descriptions of this approach, and since then, similar models have been described (Zaidi et al. 2016). We designed a hybrid model with seven 10-minute MMI stations, and two 20-minute standard interviews. This paper offers suggestions to those considering the use of MMIs in a hybrid admissions process.

Twelve Tips Tip 1 -Get faculty buy-in early
Faculty in most medical schools or other professional programs are likely accustomed to the traditional format of standard interviews. As such, they may have strong beliefs in the usefulness of this method, though the reliability of interviews, with particularly unstructured formats, has been questioned (Morris J 1999;Patterson et al. 2016).
Several approaches can be used to get faculty buy-in early. Review of the literature to date provides persuasive data Bing-You R, Linder J, White P, Neumeyer D MedEdPublish https://doi.org/10. 15694/mep.2016.000164 Page | 3 that the MMI is reliable, and may be a better predictor of medical student clinical performance than the traditional cognitive tests. We engaged our admissions subcommittee members during an annual retreat to help determine the focus of the MMI stations, aligned with our mission goals, and design the content of specific stations. For the first year, we agreed as a subcommittee to pilot a smaller version of the MMI with a volunteer group of applicants. The members were also involved with crafting the evaluation scoring sheets for the MMI stations.
In recruiting subcommittee members, address expectations as committee members early on. Besides asking faculty to commit to a minimum number of interview days, and periodic training, there needs to a strong emphasis on arriving on time for the interview day. Similar to keeping all applicant information confidential, faculty are asked not share the MMI station content.

Tip 2 -Design a floor plan and time-table
One of the first helpful steps is to design a "blueprint" of the admissions day with faculty input. This plan outlines the number of stations involved, and a physical lay-out of rooms for the MMI stations, the standard interviews and where the applicant and faculty break areas will be. We were fortunate to have a 10-room Standardized Patient (SP) Center available on interview days. Although these rooms are modeled after patient examination outpatient clinic rooms, this did not detract from the MMI interactions. Most SP Centers are hard-wired for audio (e.g., overhead speakers, bells), which is a requirement to signal applicants to move to each subsequent station. Using several offices with a common hallway is another floor plan option to consider. A time-table of the interview day needs to be created, with specific time slots outlined, which will drive decisions as to how long each station will be (e.g., will each station be 8 minutes in duration with 2 minutes transition time between stations?; will a break be incorporated?).

Tip 3 -Align the MMI station content with the program's mission and goals
There are many options as to what the MMI station content could be. MMIs are designed to help assess specific knowledge, skills, and abilities in response to specific scenarios. Stations can be focused on personal characteristics like honesty or compassion (Hofmeister 2008) or they could be developed to assess competencies important for students' success in medical school (Koenig et al. 2013). We believe including scenarios that reflect the mission of the educational program will assist in finding the best candidates for the specific program. For example, one programmatic goal of the Maine Track is to attract applicants for eventual practice in Maine, which is a predominantly rural state. An MMI station could focus on areas such as resilience while coping with limited resources or dealing with confidentiality issues in a small community.

Tip 4 -Conduct training
Once the faculty agree to the MMI approach, there should be training in the MMI process and how to conduct the stations. Practice runs are helpful for faculty to gain a sense of the shortened time frame for interactions. Faculty can also view videotaped examples of stations. Practicing with more than one candidate per MMI station is helpful. Our faculty members came to realize that preparation time before the interview day is significantly shortened because they are no longer reviewing multiple, lengthy applicant files. Faculty training should include conducting Bing-You R, Linder J, White P, Neumeyer D MMI stations with applicants whose English is not their first language, handling cultural differences in interpreting station scenarios, or interacting with applicants with disabilities.
After the floor plan is established, create a draft of the entire interview day. The time-table outlines each station length (e.g., 9:30 -9:40 a.m.) and rooms assigned to each MMI station. Keeping the same MMI station in the same room throughout the interview season simplifies the process. Applicant instructions for each room door needs to be designed. We used laminated cards which could be removed after the interview day had concluded. Other SP facilities may use a sliding cover mechanism on the door or instructions on a computer or tablet immediately outside the door. If an overhead announcer is used, a written timed script works best.
Anticipating Murphy's law (i.e., anything that can go wrong, will go wrong) conduct practice runs for timing, familiarization with the scripted announcements, and to see where transition flow problems may arise. Staff can role play the faculty interviewers, which also allows another perspective as to the clarity of the station instructions for the faculty. Although the scenario content is now available on the computer in each room, we also provide a laminated a copy of each station's instructions for the faculty in the room each interview day.

Tip 5 -Develop contingency plans
Worst-case scenario planning is a helpful exercise to complete as part of the MMI implementation process. Because an MMI is conducted similarly to an OSCE (i.e., a group starts all at the same time, and moves from room to room at the same time), the level of complexity is magnified compared to standard faculty interviews. There are potentially many unforeseen situations. Consider how to handle the following situations: a faculty interviewer or an applicant does not arrive on time (e.g., traffic accidents, storms, or other delays); a fire alarm sounds during the MMI day; power is lost to the audio system; if instructions or evaluations are online, internet connection is temporarily lost; a faculty interviewer is called away for a patient emergency; an applicant is not feeling well and needs to take a break in the process; the faculty interviewer does not follow the MMI station instructions or chronically goes over the time. We have encountered some of these scenarios and having contemplated beforehand how they would be handled was very helpful. For example, we have one or two back-up interviewers standing by every interview day. These faculty members have enough familiarity with the applicants' backgrounds or the MMI scenario to quickly step in anytime during the interviewing period.

Tip 6 -Manage applicants' expectations
As part of the invitation notice to applicants, they are informed that an MMI will be a component of their interview day. We emphasize that the hybrid model allows us to gather additional information about their attributes by way of the MMI, which is an accepted and validated assessment tool, while the standard interviews allow us to address specific areas in their application. The majority of applicants appear to be well aware of what an MMI is, based on available information in scholarly publications, medical school websites, prehealth advisors, the lay press, and applicant websites (e.g., www.studentdoctor.net). During the interview day orientation, we review the MMI schedule in detail. To help address the applicants' anxiety, we emphasize that the MMI stations are not clinical scenarios but every day situations, and that there are no right or wrong answers. The applicants are advised to carry a copy of their individual schedule and a bottle of water if they wish to. We utilize a computer-based station for a reflection exercise so no writing instruments are required. We have observed that typically after the first MMI station is completed, the applicants appear much more relaxed and many have stated the whole process was actually "fun" for Bing-You R, Linder J, White P, Neumeyer D MedEdPublish https://doi.org/10. 15694/mep.2016.000164 Page | 5 them.

Tip 7 -Encourage committee participation
We strongly emphasize that admissions committee members should attend the committee meeting when their applicants are being reviewed. If our members cannot attend, there is a "warm hand-off" to one of the two co-chairs of our subcommittee (i.e., the completed evaluation form is reviewed in-depth and the likely voting category discussed). We continue to have a high level of participation we think in part because with a hybrid model we have nine committee members who have interacted with each applicant, whereas previously when we conducted two standard interviews only two interviewers had a personal interaction with an applicant. Previously, when two standard interviewers had very conflicting perspectives of an applicant, finding common ground could be challenging. As a result, subcommittee discussions of applicants have become much richer, more insightful and holistic (Kirsh 2012).

Tip 8 -Ensure holistic review of each interviewed applicant's file
In the complex admissions process, it is important to consider the entire applicant when making an admissions decision (Monroe 2013), and we believe the hybrid model allows us to do this more easily. We suggest assigning two committee members to review an applicant's file. The full review must include a system to clarify inconsistencies in the applicant's record and gather additional information from the applicant. By design MMI stations look at specific attributes of an applicant, raising a concern that MMIs alone do not consider an applicant holistically. MMI interviewers do not review the applicants file prior to their encounter. Standard interviewers review the entire file for each applicant they are assigned to interview and conduct a traditional 20-minute interview. Each applicant is assigned to two standard interview stations and all of the MMI stations on the interview day. The applicants are presented to the subcommittee by the individuals who conducted the standard interviews with additional comments provided by the MMI interviewers.

Tip 9 -Utilize other benefits of the hybrid model
We found that shifting from a standard interview-only model to the hybrid model actually decreased the total interview time and allowed us to spend more time with applicants describing our medical school program. Although getting to know your applicants is the primary focus of the interview day, we suggest including time to highlight specific attributes of your program to help solidify applicant understanding of your program and increase your yield of accepted offers. Marketing materials include websites and information sent directly to applicants with an invitation to interview. On the interview day, we include a presentation by an admissions subcommittee leader, a financial aid overview, as well as tours of the educational facility. A student hosts lunch with the applicant group and leads a tour of Maine Medical Center's primary teaching site at the end of the interview day. During each session, time is allotted for questions and answers. Each invited applicant is given a packet of materials for reference prior to the interview day which can also be used for future reference.
Tip 10 -Review scenarios at least annually Bing-You R, Linder J, White P, Neumeyer D MedEdPublish https://doi.org/10.15694/mep.2016.000164 Page | 6 We suggest that the scenarios be reviewed annually as part of a quality improvement process and changed if necessary. Scores should be reviewed to see if enough variability is being attained. Faculty feedback can be collected so that the station content can be modified if needed. Faculty can also review whether they think the MMI stations are still aligned with the program's mission and goals as outlined above in Tip 3. We have used our annual subcommittee retreat to debrief the interview season and also determine what components of the MMI scenarios and process will be changed. In general, we would not suggest changing MMI stations during the interview season. Although test security has been raised for OSCEs (Niehaus et al. 1996), having prior information about MMI station content does not appear to impact the outcomes (Reiter et al. 2005).

Tip 11 -Assess what you're doing
The MMI approach for admissions remains fairly new in its use. The majority of US medical schools continue to use the standard interview as the main method of assessing candidates. As you develop the MMI process, consider studying as many aspects as possible. Potential areas of scholarly investigation include: are better admissions decisions being made with a hybrid model (i.e., combination of both MMI stations and standard interviews) versus only MMI stations? What characteristics of MMI faculty or applicants, such as age, affect MMI scores? The gender of applicants has been shown to impact MMI ratings (Ross et al. 2016). Does having senior medical students, or lay individuals, participating in the MMI impact scores? What is the long-term relationship between MMI scores and measures of clinical performance (e.g., patient satisfaction surveys)? Do MMI scores predict social and interpersonal problems of medical students? Many questions still remain about the MMI and we would encourage multiinstitutional studies going forward. As MMIs share some similar features with OSCEs, guidelines for measuring the quality of OSCEs may be useful for further assessment of MMIs (Pell et al. 2010).

Tip 12 -Share lessons learned
Collect summary data as an ongoing quality assessment, especially when first starting out with an MMI model (e.g., ranges of scores for each station, scores for individual faculty). Share results with committee members, Deans, and faculty periodically, as a way to increase buy-in with the process as well as to improve the process. Sharing these results with local stakeholders (e.g., medical education leaders including clerkship directors, Board members) is important to generate ongoing support. Like any admissions committee, we need to recruit additional members from time to time, and sharing our lessons learned helps generate significant interest and participation. Lastly, as mentioned in Tip 11, sharing more MMI results and outcomes with the wider medical education community helps develop better methods to select the best candidates to become future physicians.

Conclusion
Designing and organizing a hybrid admissions process combining MMIs and standard interviews is a complex activity involving many logistical details. Use of a hybrid model for high-stakes decisions emphasizes the need for a smooth and efficient process. We provide 12 tips to implement the MMI into the traditional model of standard interviews. Though the 12 tips do not guarantee success, we hope faculty in the selection and admissions field find them relevant and useful for their specific organizational situation.