End-of-Rotation Examinations in Canadian Obstetrics and Gynaecology Residency Programs: The Perspectives of Faculty Members and Residents

Abstract

The Royal College Competence by Design curriculum in obstetrics and gynaecology will launch in 2019, and it will depend heavily on multiple tools for accurate resident assessment. Several Canadian obstetrics and gynaecology residency programs use rotation-specific examinations at the end of various rotations for formative feedback. The obstetrics and gynaecology residency program at the University of Toronto adopted end-of-rotation examinations (EOREs) in 2014. We conducted a national survey to assess the current use of EOREs across Canada and to examine the attitudes and beliefs of residents and program directors regarding their use. We discuss faculty and resident experiences with EOREs and their perceptions of them. We also consider the role and benefit of these examinations in the context of the educational literature, and how they may integrate with future competency-based medical education frameworks.

Résumé

Le programme de la compétence par conception (CPC) en obstétrique et gynécologie du Collège royal sera lancé en 2019. Il dépendra largement de plusieurs outils pour évaluer de façon adéquate les résidents. De nombreux programmes canadiens de résidence en obstétrique et gynécologie ont recours à des examens ciblés à la fin de chaque stage afin d’offrir une rétroaction formative. Le programme de résidence en obstétrique et gynécologie de l’Université de Toronto a d’ailleurs adopté cette façon de faire en 2014. Nous avons mené une étude à l’échelle du pays dans le but d’évaluer l’utilisation de ces examens au Canada et d’examiner les croyances et l’attitude des résidents et des directeurs de programmes à l’égard de ces examens et de leur emploi. Nous parlons ici de la perception qu’ont les résidents et les enseignants de ces examens ainsi que de leur expérience avec ces derniers. Nous examinons également les avantages de ces examens et la place qu’ils occupent dans la littérature universitaire, et étudions leur intégration potentielle dans d’éventuels programmes de formation fondés sur les compétences.

Introduction

The assessment of knowledge is crucial to physician training, and the new Royal College Competence by Design curriculum in obstetrics and gynaecology will depend heavily on multiple tools for accurate resident assessment.¹ In educational theory, Miller’s pyramid of clinical assessment places knowledge as the foundation for all other competencies.² A poor knowledge foundation impedes the development of higher-level clinical skills, including competence in daily practice.³ In recognition of the fundamental importance of clinical knowledge in training, the CanMEDS framework of essential physician competencies places “Medical Expert” as the central role.⁴

Inadequate assessment of knowledge carries several consequences. Residency programs have a public responsibility to meet accreditation standards, and residents are thus required to meet specific learning objectives for each rotation. Many Canadian obstetrics and gynaecology training programs rely heavily on in-training evaluation reports to determine rotation outcomes. These consist of supervisors’ assessments of a resident’s clinical performance, often in a rubric-based subdivision of CanMEDS roles using a 5-point Likert-type scale of defined behaviours.

Despite widespread use, ITERs have considerable limitations in assessing the medical expert role.⁵ Ratings are subjective, creating concerns about inter-rater and intra-rater consistency.6, 7 Raters may consistently mark high or low or be influenced by character judgements. Awareness of the identity of a rater may lead to falsely inflated scores.⁸ Assessments may be based on a limited range of encounter types,⁸ and detailed feedback on knowledge deficits may not be provided if there is a time lapse between the encounter and completion of the ITER.⁹

Another approach to knowledge assessment is the use of annual written in-training examinations (e.g., the Canadian Association of Academic Professionals in Obstetrics and Gynaecology and the American Council in Resident Education in Obstetrics and Gynaecology examinations). These examinations have well-established metrics including excellent criterion validity,10, 11 construct validity,¹² and face validity.¹³ However, these examinations broadly cover all topics in obstetrics and gynaecology and therefore are not aligned with a resident’s current clinical learning environment. In addition, they do not provide specific correct-answer feedback of knowledge deficits, and they provide limited indication of residents who are at risk of failure. Whereas higher CREOG scores are predictive of candidates likely to pass the American Board of Obstetrics and Gynecology licensing examination,10, 11, 14 lower scores are poor predictors of examination failure.¹⁴ Additional assessment tools may be important for predicting residents at risk of failing licensure examinations.

Testing in medical education is predominantly considered an assessment tool, but it can also be a beneficial educational tool.¹⁵ Cognitive psychology research shows that repeated testing of information produces retention that is superior to repeated study, a phenomenon known as test-enhanced learning.16, 17 Testing also provides many indirect benefits, such as formative feedback, improved motivation, better study strategies, and efficient use of study time.17, 18 Early identification and remediation of residents with inadequate knowledge are essential to prevent failure in summative examinations.¹⁹ It is well-established in evaluation research that individuals are poor at self-assessment,²⁰ and the lowest performers are most likely to have inaccurate self-assessments.²¹ This highlights the importance of introducing formative objective examinations throughout training.

Several Canadian obstetrics and gynaecology residency programs have introduced EOREs for various general or subspecialty rotations, using them for evaluation or to provide formative feedback for determining readiness for summative examinations. The use of EOREs as both assessment and educational tools in obstetrics and gynaecology residency programs has not been reported. To allow for repeated use, the questions used in EOREs have not been published and likely vary across programs in format, frequency, type of feedback provided, and purpose.

The obstetrics and gynaecology residency program at the University of Toronto introduced EOREs in 2014 in the subspecialty rotations (gynaecologic oncology, maternal-fetal medicine, reproductive endocrinology and infertility/pediatric adolescent gynaecology, and urogynaecology) for both feedback and evaluative purposes. Faculty members at nine clinical sites readily engaged and accepted the use of EOREs.

We conducted surveys in July 2015 to obtain feedback about EOREs from residents and to better understand the current usage and formats in Canadian obstetrics and gynaecology residency programs. We aimed to evaluate the perspectives and experiences of residents and program directors among the 13 English-language obstetrics and gynaecology residency programs regarding use of EOREs.

Three different online surveys were distributed to Canadian obstetrics and gynaecology residency program directors, current residents (“pre-Royal College Examination”), and residents or graduates who had completed the RCE within the previous two years (“post-RCE”). Survey questions and results are provided in eTable 1, eTable 2, eTable 3. Survey questions focused on current EORE practices and experiences and on the perceptions of residents and faculty members, inquired about barriers to using EOREs, and asked for open-ended comments. The survey was sent to 560 residents and 13 program directors. Residents from all invited programs responded. The overall response rate was 32% (129/441 pre-RCE residents, 41/119 post-RCE residents, and 13/13 program directors).

We recognize that this low response rate creates a high risk of responder bias and prohibits us from drawing firm conclusions. Furthermore, responses may not reflect the opinions of programs not surveyed and are limited due to the mostly quantitative nature of the survey. However, in the absence of any literature on the use of EOREs in Canadian obstetrics and gynaecology training programs, these responses serve as a starting point for discussion of EORE use across Canada and highlight the need for further research.

The majority of Canadian obstetrics and gynaecology residency programs (9/13) use EOREs, typically as a combination of oral case-based examinations, short-answer questions, and multiple-choice questions. According to respondents, programs were more likely to use EOREs in subspecialties such as gynaecologic oncology, reproductive endocrinology and infertility, and urogynaecology (Figure). Although obstetrics and gynaecology residents typically spend significant portions of training in general gynaecology and ambulatory clinics, these rotations were rarely evaluated using EOREs (two programs evaluated general gynaecology; no programs evaluated ambulatory clinics). In comparison, obstetrics, gynaecology, and ambulatory care comprise 60% of the CREOG examination, whereas subspecialties such as gynaecologic oncology and reproductive endocrinology and infertility comprise only 20%.¹² Because topics in general obstetrics and gynaecology comprise a large part of clinical practice after residency, residency programs might consider using EOREs for these rotations; conversely, some may argue that further testing is not required, given the extensive clinical exposure to these topics during residency training, as well as on Association of Academic Professionals in Obstetrics and Gynaecology and CREOG examinations.

The University of Toronto program uses EOREs to (1) provide residents with general feedback, (2) contribute to the rotation pass/fail, (3) identify site differences, and (4) determine if rotations are meeting specified learning objectives. We found that all programs with EOREs used them to provide residents with feedback, and most provided feedback in a similar manner (general comments with or without a grade score) and time frame (< 1 month).

In our survey, only one residency program provided residents with specific feedback on weaknesses after EOREs, using comments or a scoring key. Yet cognitive psychology research suggests that the type of feedback plays an important role in test-enhanced learning.16, 17 Learners perform better on subsequent testing when they are provided specific, correct-answer feedback on their errors, compared with re-studying the material without feedback.²² Although general comments on EOREs may provide learners with a sense of their overall performance, they do not identify or correct specific knowledge deficits.

However, the potential benefits of providing specific feedback must be weighed against the practical considerations regarding the provision of answer keys to residents. Our survey found that faculty time is perceived as a major barrier to implementation of EOREs, and two thirds of residency programs repeat their EORE questions in subsequent examinations. Correcting errors when learners are motivated and immersed in the material might have beneficial educational effects.²³

All program directors using EOREs believed that use of EOREs had a positive impact on residency education. Most post-RCE residents (84%) believed that EOREs raised their motivation to study throughout residency, and 53% believed that they also improved their level of preparedness and organization for the RCE. However, they believed that these examinations had no effect on their final performance or their level of anxiety about the RCE. Most pre-RCE residents (91%) believed that EOREs improved or would improve their motivation to study and organize notes and their learning and understanding of the material (80%). Perceptions were similar among residents with or without EORE experience.

The perceptions of residents and program directors regarding the impact of EOREs on clinical rotation experience were generally divided between a positive effect (55% of program directors) and no effect (63% of pre-RCE residents and 52% of post-RCE residents). However, a minority of post-RCE residents (5%) and pre-RCE residents (7%) believed that EOREs detracted from their clinical experience. Some commented that time spent studying for EOREs negatively affected their work-life balance and stress levels and detracted from the time available to read about clinical cases. Providing a syllabus and study time could make undergoing EOREs a more positive experience by balancing the heightened service requirements of some rotations with an educational focus.

Most pre-RCE residents (65%) felt that EOREs should be used only to provide formative feedback. Some residents commented that if used purely for formative feedback rather than evaluation, EOREs would be a more positive learning experience. Other residents believed that, although EOREs would likely impose additional stress, they were “probably worth it” because they would help solidify the learning objectives for rotations and help them practise the skills required to take the RCE. Although pre-RCE residents may be wary of EOREs, most program directors and post-RCE residents agreed that EOREs should be used for both evaluation and formative feedback.

Testing has long been known to be an unpopular strategy among learners at all stages compared with studying because it highlights errors.²⁴ In educational theory, desirable difficulties refer to techniques, such as testing, that promote long-term retention even though they might make the learning task more challenging.17, 25 Although testing tends to highlight more errors and reduce confidence compared with repeated studying, it is known to lead to improved long-term retention.¹⁶ Training programs that focus on creating desirable difficulties for residents through testing may actually enhance learning. This is particularly true if educators emphasize that challenges encountered during testing should be considered positive learning opportunities.

Although some programs (22%) are currently using EOREs as the sole assessment to determine rotation pass or fail, no program director or resident surveyed supported this practice. Both faculty members and residents commented on the importance of considering EORE testing metrics, and noted that to be useful they must avoid testing obscure details. We argue that if EOREs are being used as the sole or primary assessment tool for a rotation, it is important to ensure that they are reliable and valid assessment tools.

In comparison to the traditional ITERs, EOREs might be more a reliable assessment of knowledge.⁷ Programs can bolster the reliability of EOREs by including many questions from a broad range of knowledge domains and by using standardized scoring keys to reduce subjective influences on scoring.26, 27 It is also important to ensure that EOREs are valid assessment tools; that is, that their score interpretation truly reflects the residents’ knowledge and leads to appropriate remediation plans as necessary. Programs can enhance the validity of EOREs by creating examination “blueprints” that map questions to the primary Royal College learning objectives to ensure that assessment is based on relevant domains.27, 28, 29 Programs can ensure validity by testing only essential information, ensuring adequate length, sampling content domains equally, and incorporating items that test higher-order cognitive loads.²⁹

Establishing valid and reliable assessment tools is time-consuming. It is therefore not surprising that, although over 90% of residents and program directors favoured the introduction of EOREs in Canadian obstetrics and gynaecology residency programs, they currently are used in only 69% of programs. The main perceived barriers to implementation were the effort required by faculty members to create and proctor examinations, and the fact that EOREs would take time away from clinical duties for all staff members. Some noted that, given the administrative burden and the absence of supportive metrics, they did not think EOREs would be useful. The educational literature indicates that the utility of an assessment tool depends on its performance across five domains: reliability, validity, educational impact, acceptability, and cost.²⁶ If EOREs are used mainly as an educational tool and to provide formative feedback, then programs may be willing to tolerate lower reliability or validity in favour of their positive educational impact. However, if the results will contribute substantially to determining rotation pass or fail, programs must dedicate additional resources to ensure basic testing standards are met.

CBME promotes a graduated, trainee-focused approach based on observation and assessment of clinical competencies.1, 30 Using a CBME approach for education will shift the focus of the residency training environment towards one of multiple, formative clinical assessments. It promotes assessment of skills that residents perform in practice, relying on direct observation in the clinical setting.

With an increase in the number of observed clinical encounters required for residents to be allowed to progress, there will be a corresponding increase in workload for faculty members. EOREs that consist solely of oral case-based examinations may be redundant if similar encounters are frequently evaluated in CBME. However, during CBME implementation, there may be significant heterogeneity among clinical assessments of essential medical expert content. Continuing to offer EOREs might help to ensure that faculty assessments consistently meet a minimum standard. EOREs might also remain a valuable means of assessing knowledge of rare but important subjects less commonly encountered or evaluated in clinical practice. In the Dutch obstetrics and gynaecology CBME model, summative assessments (including an annual national examination) continue to be used.¹ However, it is formative assessment that becomes pivotal to progression within the training program.

Programs may benefit from collaborating on EORE design. Several survey respondents expressed an interest in standardizing and unifying examinations across the country, such as development of a question bank. A pooled initiative among stakeholders in postgraduate medical education to create a high-quality question bank might enhance EORE validity and reliability. Ultimately, collaboration and improvement in question design and quality may bolster the educational impact and assessment quality of EOREs and help to align them with new CBME curricula. Future research will be important for identifying whether we are meeting these goals.