Development and Implementation of a New OSCE for Assessing Resident Competency in Weight Loss Counseling

Background: Despite the need to provide weight management counseling to patients who are overweight or obese, resident physicians receive little relevant education and practice in performing this skill. As a result, residents report low levels of self-efficacy and often demonstrate limited performance competency. Structured educational interventions providing opportunities for residents to practice weight loss counseling along with objective assessment are lacking. Methods: This article describes the development and implementation of an objective structured clinical examination (OSCE) for the assessment of resident physician weight loss counseling. A total of 30 PGY-2 and 3 internal medicine residents participated. The OSCE included a brief (10-15 minute) educational session on the 5As counseling framework, followed by an 18-minute encounter with a standardized patient (SP) who was overweight or obese. Resident performance in the OSCE was measured by use of a skills checklist composed of 23 pre-specified counseling skills. Each skill represents a specific element of communication used in effective weight loss counseling, and is based on the 5As counseling framework. The checklist was completed by the SP immediately following the encounter, and the percentage of skills performed was calculated. Results: Residents performed a mean of 68.6% of the pre-specified counseling skills (SD = 17.2%). No significant differences in OSCE performance based on gender, PGY year, SP rater, or career plan (specialty vs. primary care) emerged. Conclusion: The authors developed and implemented an OSCE for the assessment of resident weight loss counseling. The skills checklist allowed for the identification of individual and overall counseling skills in need of Herzog J, Cameron K, Badaracco M, Kushner R MedEdPublish https://doi.org/10.15694/mep.2018.0000015.1 Page | 2 improvement, and allowed for targeted feedback.


Introduction
Despite increasing awareness of the obesity epidemic in America, more than two in three adults are considered to be overweight or obese, and more than one in three are classified as having obesity (Ogden et al. 2014). Obesity is associated with an increased risk of comorbid conditions such as type 2 diabetes, cardiovascular disease, and cancer, and increased mortality (Eyre et al. 2004, McGee et al. 2005. Multiple studies have shown that counseling by primary care physicians is positively associated with patient weight loss and behavior change (Leblanc et al. 2011, Dansinger et al. 2007, Rose et al. 2013, Huang et al. 2004. As a result, many health organizations have published evidenced-based guidelines and recommendations for primary care physicians to screen patients for obesity and to provide weight loss counseling (Moyer& Force 2012, Jensen et al. 2014, NIH 1998. However, despite these recommendations, weight loss counseling occurs infrequently during primary care visits (Antognoli et al. 2014, Ma et al. 2009, McAlpine & Wilson 2007. Furthermore, the rates of weight related counseling have declined in recent years (Kraschnewski et al. 2013). Lack of skills and insufficient training are commonly cited barriers to delivering weight loss counseling (Huang et al. 2004, Jay et al. 2008, Hebert et al. 2012, Kushner 1995.
To address these concerns, many have called for improvements in primary care residency education (Lenders et al. 2014, Colbert & Jangi 2013. Multiple studies demonstrate residents lack competency in weight loss counseling, and report low self-efficacy in their counseling skills (Jay et al. 2008, Hebert et al. 2012, Park et al. 2005, Ruser et al. 2005, Vetter et al. 2008, Block et al. 2003, Davis et al. 2008, Fluker et al. 2010, Smith et al. 2015. There are few published studies examining the current state of residency education on weight loss counseling, but the current evidence suggests minimal curriculum time is spent on obesity, nutrition, and physical activity counseling (Antognoli et al. 2016). Program directors and residents have called for a greater emphasis on teaching specific counseling techniques, additional opportunities to practice counseling skills, and assessment tools that facilitate feedback and identify areas in need of improvement (Smith et al. 2015, Antognoli et al. 2016. While interventions incorporating such practices have been associated with improved counseling behaviors by residents in addition to patient weight loss (Burton et al. 2011, Jay et al. 2013, Burton et al. 2016 most require multiple hours to implement. As time availability during residency is a commonly cited barrier in implementing a curriculum on weight loss counseling (Antognoli et al. 2016, Goff et al. 2010 lengthy interventions may be difficult to incorporate. Hence, the need exists for more condensed educational interventions that provide an objective assessment of resident counseling skills. In this study we sought to develop and implement an objective structured clinical examination (OSCE) for the assessment of weight loss counseling based on the 5As framework, and to assess residents' performance in the OSCE using a skills checklist completed by a standardized patient (SP). Secondary study objectives included providing residents the opportunity to practice counseling in a simulated environment, and educating residents on the use of the 5As framework.

Participants
PGY-2 and PGY-3 Internal Medicine residents were invited to participate in the study OSCE during a one-month Herzog J, Cameron K, Badaracco M, Kushner R MedEdPublish https://doi.org/10.15694/mep.2018.0000015.1 Page | 3 outpatient medicine rotation. The study took place from September 2015 through July 2016. Resident participation in the study OSCE was encouraged by the residency program director and a chief resident, but participation was voluntary. Every month, residents eligible to participate (i.e., were engaged in a one-month outpatient medicine rotation) were recruited via email by an internal medicine resident involved with the study (MB). Thirty out of 46 total PGY-2 and PGY-3 residents (65.2%) elected to participate. A total of eleven OSCEs occurred throughout the study period. Northwestern University Institutional Review Board approved the study, designating it exempt.

OSCE Content and Format
The SP case was developed by the principal investigator (RK) along with another clinical medicine faculty member. The case is also used to assess medical student motivational interviewing competence in the primary care clerkship of the Feinberg School of Medicine (FSM).
For training, four SPs (1 male and 3 females) were asked to participate from the general pool of SPs who work in the medical school simulation curriculum. The SPs were trained for two hours on the scenario. They were expected to elaborate with responses that seemed natural to them as they already had experience with the subject.
Two months prior to initiation of the study, the principal investigator (RK) gave a 1-hour didactic noon-time lecture on weight loss counseling for which resident attendance was optional. After agreeing to participate in the OSCE, residents were instructed, via email, to review a short article (Vallis et al. 2013) and a 5-minute video (Youtube) describing the 5As method for weight loss counseling to better prepare them for the OSCE. The 5As (Assist, Advise, Agree, Assist, Arrange) framework for weight loss counseling is endorsed by the Centers for Medicare and Medicaid Services and the United States Preventative Services Task Force (CMS 2011). It has been shown to be a brief and effective method for delivering weight loss counseling (Alexander et al. 2011, Ockene et al. 1999, Sherson et al. 2014, Goldstein et al. 2004). Prior to their participation in the OSCE, residents also were asked to fill out the Nutrition, Exercise, and Weight Management (NEW) attitudes scale to measure weight bias (Ip et al. 2013).
Each OSCE took place at FSM's Clinical Education Center. Upon arrival, all residents participating in the OSCE met with investigators (JH, RK, MB) over the lunch hour where the purpose of the study, the 5As counseling method, and the structure of the OSCE were reviewed. A brief description of the OSCE case was provided, explaining that residents were expected to perform weight loss counseling during an 18-minute encounter with an SP who was overweight or obese. Residents were advised to use the stopwatch function of their cell-phone to ensure adequate time for counseling. The 5As counseling framework was reviewed through use of a one-page handout which provided a description of each A alongside sample questions they may wish to ask. Investigators used a preformed, standardized script to deliver all information on the 5As and OSCE structure to the residents. Explanation and presentation of the information took 10-15 minutes to complete.
Residents were given a folder containing materials available for use during the SP encounter. Residents had previously been randomized into two groups, differing only in the materials included in this folder. All folders included the handout of the 5As with sample questions, and after visit summary (AVS) patient handouts describing healthy diet and exercise habits. The AVS handouts were obtained from the electronic medical record used in the outpatient setting at this institution. One of the groups, comprised of 14 residents, was also given materials to aid in scoring a patient-completed behavior questionnaire, as well as additional handouts to aid in counseling based on the score of the questionnaire. Residents were advised all materials were available for their use, but using any or all of the materials was not required. As there were no significant differences between the two groups in any outcome measures, data from both groups were combined for this study. Residents were informed their performance would be assessed, and were offered the opportunity of reviewing their performance evaluation at a later date.
The SP portrayed a patient with a chief complaint of continued weight gain. Residents were provided the SP's height, weight, body mass index (BMI), and results of an elevated HbA1c and elevated lipid panel identified as having been completed prior to the encounter. The SP was motivated to make dietary changes for weight loss. Residents were notified via intercom when the 18-minute encounter was over. Following the encounter, each resident completed a self-assessment questionnaire.

Evaluation
Performance in the OSCE was measured using a skills checklist that was completed by the SP immediately following the encounter. The checklist includes 23 items, divided into 5 categories representing the 5As. The checklist was adapted from a similar assessment instrument used in a study by Jay et al. to measure resident competence in weight loss counseling ). Jay et al. reported that the items used in the instrument showed adequate internal consistency (Cronbach's alpha >0.75) when used in OSCEs to evaluate physician performance . For the new checklist, the "Agree" section was expanded from three to eight items in order to capture more robust assessment regarding shared decision making. Added items specifically assess if both diet and physical activity/exercise changes were discussed, the degree to which the SP was involved in the discussion, and the SP's rating of the clarity and adequacy of diet and physical activity/exercise changes recommended. The new checklist used in this study and the instrument used by Jay et al. are included in the attached appendix.
Following the SP encounter, each resident completed the 9-item physician version of the patient perception of patient-centeredness (PPPC) questionnaire as a self-assessment of their communication during the SP encounter (Rider et al. 2007). Residents rated nine distinct aspects of communication, each of which had four response options (e.g., completely, mostly, a little, and not at all). Final scores comprise the sum of the nine individual items with of a maximum score of 36. We asked residents to complete the NEW attitudes scale (Ip et al. 2013) prior to the encounter as we anticipated that residents' attitudes and beliefs about obesity may affect their counseling. The NEW attitudes scale provides a score from -118 to +118, with more negative numbers indicating more weight bias.

Analyses
Resident performance in the OSCE was determined by the percent of 23 pre-specified 5As counseling skills performed. For analysis, we rated each skill dichotomously, as "performed" or "not performed." Five of the 23 items included four response options (not at all, only a little, somewhat, very). These five skills were rated as "performed" if the SP responded "only a little, somewhat, or very" to the scale, and as "not performed" if the SP responded "not at all." Data from all 30 resident SP encounters were used in determining the mean and standard deviation of the percent of skills performed. Two-sample t-tests were used to assess differences in the percent of 5As skills performed based on gender, PGY level, SP rater, and career plans of specialty vs. primary care, with p<0.05 as a cutoff for statistical significance.
Resident responses to the physician version of the PPPC questionnaire were tallied. The NEW attitudes scale was scored using the method described by Ip  al. 2013). The Pearson correlation coefficient between self-assessment of communication and percent of 5As skills performed, as well as between weight bias and percent of 5As skills performed was calculated.

Results
A total of 30 residents participated, including 17 PGY-2 residents (out of 22 eligible), and 13 PGY-3 residents (out of 24 eligible). Fifty percent of participants were female, and 86% had plans to pursue a subspecialization fellowship. Seven residents attended the noon-time lecture on weight loss counseling. While all residents were asked to review a short article and watch a 5 minute video on the 5As in preparation for the OSCE, 16 out of 30 reported reviewing the article, 12 reported watching the video, and 6 reported doing neither. A total of four SPs performed the 30 encounters throughout the study period, each performing 15, 11, 3, and 1 encounters, respectively.
The mean percent of counseling skills performed was 68.6% (SD = 17.2%). There were no significant differences in the percentage of 5As skills performed based on gender, PGY year, SP rater, completion of preparation materials, or career plan (specialty vs. primary care). The percentage of residents who performed each individual skill is shown in Table 1. Resident performance by skill ranged from 10% to 100%. Few residents asked the SP how important it was for them to lose weight (26.7%), or their confidence in losing weight (10%). Only 20% of residents recommended referral resources such as a dietitian, commercial or internet weight loss program, or a health psychologist. In contrast, 100% of residents discussed making dietary changes, 93.3% informed the SP that weight loss will improve health, and 93.3% provided clear recommendations for change.
The NEW attitudes scale was completed by 29 of 30 residents. The mean score was 22.9 (SD = 23.7). Scores ranged from -26 to 59. The mean score on the PPPC self-assessment questionnaire was 25.3 (SD = 3.59). There were no significant differences in scores on the NEW attitudes scale or the PPPC self-assessment questionnaire based on gender, PGY year, completion of preparation materials, or career plan (specialty vs. primary care). There was no significant relationship between self-assessment of communication and the percent of 5As skills performed (r=0.23, p=0.21), or between weight bias and the percent of 5As skills performed (r=0.15, p=0.51).

Discussion
In this pilot study, we were able to develop and test an obesity counseling OSCE for internal medicine residents and assess their performance using SPs. We chose to adapt a previously validated OSCE to provide a more robust performance assessment. The 5As counseling framework was chosen based on the general applicability of this model and its adaption by CMS for the delivery of obesity counseling by primary care providers. Overall, residents performed slightly over two-thirds of the pre-specified counseling skills. However, resident performance varied considerably on the individual skills. All residents provided some dietary counseling and clear dietary goals for the SP. This finding may reflect a focus on dietary counseling in medical education coupled with residents' level of confidence from clinical practice. In contrast, physical activity/exercise counseling was performed less often. This difference in counseling types also may have been influenced by the written case description provided to the SPs, which emphasized a willingness to make dietary changes over physical activity/exercise changes.
Residents appeared to be less likely to perform more directive counseling skills, such as asking the SP specifically about how important it was to try to lose weight, and how confident they were that they could lose weight. Only onefifth of residents referred the SP to outside resources, which may represent a lack of knowledge of such resources.
The residents in our study performed a higher overall percentage of counseling skills than residents in two studies by Jay et al, which used similar instruments to measure resident competency in weight loss counseling . In one study, the mean percent of counseling skills performed by 152 internal medicine residents was 34% (SD=27%) of 19 skills (Jay & Sclair et al. 2010). In another study 23 primary care residents performed an average of 29.4% of 18 relevant counseling skills . However, there were important differences between our study and those performed by Jay et al. We used trained SPs to rate resident performance whereas Jay et al. asked patients to conduct the ratings. We also made several changes to the instrument used in the studies by Jay et al. These changes included removing two skills requiring the resident to ask the patient a specific question ("Did your doctor ask if you're trying to lose weight" and "Did your doctor ask you to recall all the food and drinks you've had in the past 24 hours"), and adding several more general skill prompts (i.e. "The recommended changes provided were clear," "I received adequate guidance on what to do with my diet, see appendix 1 for full comparison). In addition the residents in our study were provided additional resources: a review of the 5As counseling framework immediately prior to the SP encounter, and had a handout with sample questions for each of the 5As which was available to them during the interview.
One strength of our study was the ability of the skills checklist to identify specific areas of weight loss counseling in need of improvement. As noted, resident performance on each of the 5As was highly variable with as few as 10% and as many as 100% of residents performing a particular counseling skill. This information could be used to facilitate targeted group and individual feedback. Instruments that identify areas of counseling in greatest need of improvement would allow residency programs to use curriculum time most effectively, which is critical, given time availability is a commonly cited barrier in implementing or improving curricula on weight loss counseling (Goff et al. 2010, Antognoli et al. 2016. While the primary objectives of our study were to develop an OSCE for the assessment of weight loss counseling based on the 5As framework, and to assess residents' performance using a skills checklist, the study also allowed residents to practice counseling in a simulated environment. There are few studies examining the current state of residency education on weight loss counseling, but the current evidence suggests most programs rely mainly on didactic sessions rather than practicing with SPs (Antognoli et al. 2016). Several studies have identified concerns related to teaching weight loss counseling topics through didactic instruction alone. Indeed, research demonstrates that didactic instructional style is negatively associated with attitudes and perceived professional norms toward counseling (Antognoli et al. 2016) and the rate of obesity documentation and weight loss counseling appeared unchanged following a didactic presentation (Ren et al. 2016). Program directors have called for additional opportunities for residents to practice and receive feedback on counseling skills (Antognoli et al. 2014, Smith et al. 2015. Furthermore, evidence suggests practicing weight loss counseling with SPs improves competency and confidence in counseling, and can clinically lead to patient weight loss , Jay et al. 2013, Ockene et al. 1999, Burton et al. 2016).
Our study has several limitations. It included a small sample (n=30) of internal medicine residents at one institution. Not all internal medicine residents participated, which may have introduced selection bias. In this pilot study, we did not assess the degree of improvement in counseling associated with taking part in the OSCE. Similarly, as the study was not designed to be longitudinal, we are unable to assess the effects the OSCE had on counseling competency, resident self-efficacy, or outcomes relevant to patients with obesity.

Conclusions
This article has described the initial development, implementation, and testing of an OSCE for the assessment of resident physician weight loss counseling skills. The skills checklist used for assessing performance in the OSCE identified specific strengths and weaknesses in the resident cohort's counseling technique, and the OSCE provided residents with an opportunity to practice counseling in a simulated environment. To address the current obesity epidemic, resident physicians need to be well trained in the delivery of brief and effective weight loss counseling. Given the time constraints of residency curriculum, targeted interventions that allow for practice and meaningful feedback on counseling skills are needed. It will be important for future studies to determine what types of educational interventions are most efficient in improving resident competency in counseling, and ultimately, what interventions lead to improvements in behavioral and weight loss outcomes.

Notes On Contributors
Jackson Herzog -Jackson Herzog is a fourth year medical student at the Northwestern University Feinberg School of Medicine with an interest in medical education. He plans to pursue a radiology residency upon graduation. Kenzie Cameron -Dr. Cameron is a research professor in the department of medicine at Northwestern University Feinberg School of Medicine. She is a health services researcher who has a background in communication and health behavior theory, with a particular focus on health communication and social influence. Her interests include addressing racial and ethnic disparities, as well as using mixed methodologies to increase individuals' use of preventive services such as adult vaccinations and cancer screenings.
Maria Badaracco -Dr. Badaracco earned her medical degree from Boston University and completed her residency in internal medicine at Northwestern Memorial Hospital. She is currently practicing in California. Her interests include preventive health and education, exercise, nutrition, and weight management.
Robert Kushner -Dr. Robert Kushner is a Professor of Medicine at Northwestern University Feinberg School of Medicine and Director of the Center for Lifestyle Medicine in Chicago, IL. He is an international expert in the treatment of obesity and actively engaged in curricular development for undergraduate and postgraduate medical education.