Abstract
In Canada, high stakes objective structured clinical examinations (OSCEs) administered by the Medical Council of Canada have relied exclusively on physician examiners (PEs) for scoring. Prior research has looked at using SPs to replace PEs. This paper reports on two studies that implement and evaluate a standardized patient (SP) scoring tool to augment PE scoring. The unique aspect of this study is that it explores the benefits of combining SP and PE scores. SP focus groups developed rating scales for four dimensions they labelled: Listening, Communication, Empathy/Rapport, and Global Impression. In Study I, 43 SPs from one site of a national PE-scored OSCE rated 60 examinees with the initial SP rating scales. In Study II, 137 SPs used slightly revised rating scales with optional narrative comments to score 275 examinees at two sites. Examinees were blinded to SP scoring and SP ratings did not count. Separate PE and SP scoring was examined using descriptive statistics and correlations. Combinations of SP and PE scoring were assessed using pass-rates, reliability, and decision consistency and accuracy indices. In Study II, SP and PE comments were examined. SPs showed greater variability in their scoring, and rated examinees lower than PEs on common elements, resulting in slightly lower pass rates when combined. There was a moderate tendency for both SPs and PEs to make negative comments for the same examinee but for different reasons. We argue that SPs and PE assess performance from different perspectives, and that combining scores from both augments overall reliability of scores and pass/fail decisions. There is potential to provide examinees with feedback comments from each group.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Boulet, J. R., Smee, S., Dillon, G. F., & Gipel, J. R. (2009). The use of standardized patient assessments for certification and licensure decisions. Simulation in Healthcare: Journal of the Society for Simulation in Healthcare, 4, 35–42.
Byrne, A., Tweed, N., & Halligan, C. (2014). A pilot study of the mental workload of objective structured clinical examination examiners. Medical Education, 48, 262–267.
Chong, L., Taylor, S., Haywood, M., Adelstein, B. A., & Shulruf, B. (2017). The sights and insights of examiners in objective structured clinical examinations. Journal of Educational Evaluation in the Health Professions, 14, 34. https://doi.org/10.3352/jeehp.2017.14.34.
De Champlain, A. F., Margolis, M. J., King, A., & Klass, D. J. (1997). Standardized patients’ accuracy in recording examinees’ behaviours using checklists. Academic Medicine, 72, 9–23.
Donnelly, M. B., Sloan, D., Plymale, M., & Schwartz, R. (2000). Assessment of residents’ interpersonal skills by faculty proctors and standardized patients: A psychometric analysis. Academic Medicine, 75(Supplement), S93–95.
Eva, K. W., Bordage, G., Campbell, C., Galbraith, R., Ginsburch, S., Holmboe, E., et al. (2016). Towards a program of assessment for health professionals: From training into practice. Advances in Health Science Education: Theory and Practice, 21, 897–913.
Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the “black box” differently: Assessor cognition from three research perspectives. Medical Education, 48, 1055–1068.
Ginsburg, S., Kogan, J. R., Gingerich, A., Lynch, M., & Watling, C. J. (2019). Taken out of context: Hazards in the interpretation of written assessment comments. Academic Medicine. https://doi.org/10.1097/ACM.0000000000003047.
Han, J. J., Kreiter, C. D., Park, H., & Ferguson, K. J. (2006). An experimental comparison of rater performance on an SP-based clinical skills exam. Teaching and Learning in Medicine, 18, 304–309.
Howley, L. D. (2004). Performance assessment in medical education: Where we’ve been and where we’re going. Evaluation and the Health Professions, 27, 285–303.
Hauer, K. E., Hodgson, C. S., Kerr, K. M., Teherani, A., & Irby, D. M. (2005). A national study of medical student clinical skills assessment. Academic Medicine, 80(Suppl), S25–S29.
Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179–197.
Lockyer, J., Sargeant, J., Campbell, J. L., Richards, S. H., & Rivera, L. A. (2017). Multisource feedback and narrative comments: Polarity, specificity, actionability, and CanMEDS roles. Journal of Continuing Education in the Health Professions. https://doi.org/10.1097/CEH.0000000000000183.
Reznick, R. K., Blackmore, D. E., Dauphinee, W. D., Rothman, A. I., & Smee, S. (1996). Large-scale High Stakes Testing with an OSCE: Report from the Medical Council of Canada. Academic Medicine, 71(Supplement), S19–21.
Swanson, D. B., & Norcini, J. J. (1989). Factors influencing the reproducibility of tests using standardized patients. Teaching and Learning in Medicine, 1, 158–166.
Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18, 291–303.
Taveres, W., Sadowski, A., & Eva, K. W. (2018). Asking for less and getting more: The impact of broadening a rater’s focus in formative assessment. Academic Medicine, 93, 1584–1590.
Thistlethwaite, J. E. (2002). Developing an OSCE station to assess the ability of medical students to share information and decisions with patients: Issues relating to interrater reliability and the use of simulated patients. Education for Health, 15, 170–179.
Touchie, C., & Streefkerk, C. (2014). Blueprint project—Qualifying examinations blueprint and content specifications. Retrieved from https://mcc.ca/media/Blueprint-Report-1.pdf. Accessed 25 Aug 2019.
van Zanten, M., Boulet, J. R., Norcini, J. J., & McKinley, D. (2005). Using a standardised patient assessment to measure professional attributes. Medical Education, 39, 20–29.
Weidner, A. C., Gimple, J. R., Boulet, J. R., & Solomon, M. (2010). Using standardized patients to assess the communication skills of graduating physicians for the comprehensive osteopathic medical licencing examination (COMLEX) level 2- performance evaluation (Level 2-PE). Teaching and Learning in Medicine, 22, 8–15.
Whelan, G. P., Boulet, J. R., McKinley, D. W., Norcini, J. J., vanZanten, M., Hambleton, R. K., et al. (2005). Scoring standardized patient examinations: Lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA). Medical Teacher, 27, 200–206.
Williams, R. G. (2004). Have standardised patient examinations stood the test of time and experience? Teaching and Learning in Medicine, 16, 215–222.
Acknowledgements
We would like to acknowledge the contributions of Dr. Claire Touchie and Anthony King for reviewing this manuscript and providing insightful comments. In addition, we would like to thank Dr. Gordon Page for sharing his knowledge of the literature on SP scoring and Dr. Andrea Gotzmann for her statistical advice.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical standards
Examinees registering for examinations with the Medical Council of Canada (MCC) sign an agreement allowing the collected data to be used for quality assurance studies. This study used MCC data to inform OSCEs scoring methods in order to ensure fair and reliable examinations. As per Privacy and Confidentiality of the World Medical Association Declaration of Helsinki (2013)—Ethical Principles for Medical Research Involving Human Subjects, every precaution was taken to protect the privacy of the examinees and the confidentiality of their personal information. When the final research dataset was assembled, all identifiers were deleted; only aggregate summaries are presented. This quality assurance study was reviewed by the Research Advisory Committee and the Central Examination Committee to the Medical Council of Canada and follows the principles of the Tri-council Policy Statement regarding the ethical conduct of research.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Roy, M., Wojcik, J., Bartman, I. et al. Augmenting physician examiner scoring in objective structured clinical examinations: including the standardized patient perspective. Adv in Health Sci Educ 26, 313–328 (2021). https://doi.org/10.1007/s10459-020-09987-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-020-09987-6