Assessment of Surgery Residents’ Interpersonal Communication Skills: Validation Evidence for the Communication Assessment Tool in a Simulation Environment

doi:10.1016/j.jsurg.2016.04.016

Journal of Surgical Education

Volume 73, Issue 6, November–December 2016, Pages e19-e27

https://doi.org/10.1016/j.jsurg.2016.04.016 Get rights and content

Objectives

Although development of trainees’ competency in interpersonal communication is essential to high-quality patient-centered surgical care, nontechnical skills present assessment challenges for residency program directors. The Communication Assessment Tool (CAT) demonstrated internal reliability and content validity for general surgery residents, though the tool has not yet been applied in simulation. The study provides validation evidence for using the CAT to assess surgical residents’ interpersonal communication skills in simulation scenarios.

Design

Simulations of delivering bad news were completed by 21 general surgery residents during a mandatory communication curriculum. Upon completion of the 10-minute scenario, standardized participants (SPs) assessed performance using the 14-item CAT rating scale and individually provided feedback to residents. Discrete communication behaviors were recorded on video review by a trained blinded observer. The traits emotional intelligence questionnaire short form (TEIQue-SF) was completed by the residents 6 months later. SP-CAT ratings are evaluated with respect to learner characteristics, observed behaviors, and TEIQue results.

Setting

Surgical simulation center in a 900-bed tertiary care hospital.

Participants

General surgery residents were targeted learners. Trauma survivors network volunteers served as SPs, acting as a family member of a patient who developed an intracerebral hemorrhage following a small bowel procedure.

Results

Discrete communication behaviors were reliably assessed by the observer (interrater reliability with trainer: 89% agreement, κ = 0.77). SP-CAT ratings ranged from 34 to 61. Higher SP-CAT ratings were correlated with positive communication behaviors (Spearman ρ = 0.42, p = 0.056). Total TEIQue was positively related to SP-CAT ratings (ρ = 0.42, p = 0.061). The TEIQue emotionality factor was strongly correlated with SP-CAT ratings (ρ = 0.52, p = 0.016).

Conclusions

The CAT demonstrates content validity in a simulation environment with former patients acting as SPs. This study provides validation evidence relating the SP-CAT to discrete observations of communication behaviors by a trained, reliable observer as well as residents’ self-reported emotional intelligence traits.

Introduction

Surgeons’ interpersonal and communication skills are essential to the delivery of high-quality care. The Institute of Medicine’s “Crossing the Quality Chasm” notes that providing patient-centered care requires effective interpersonal communication that is customized to varied and complex clinical situations,¹ and educators acknowledge that successful integration of nontechnical skills training into graduate medical education may benefit from theoretical constructs of emotional intelligence, such as emotional awareness, empathy, and self-management.2, 3, 4, 5 Accordingly, the Accreditation Council for Graduate Medical Education (ACGME) recognizes communication skills as 1 of 6 essential competencies and provides a framework for measuring the development of these skills across competencies. The ACGME milestones identify specific communication behaviors that residents should aim to master, including the ability to customize emotionally difficult information and manage conflict with or among patients and families.⁶

Although the ACGME milestones provide clarification of targeted behaviors, assessment of surgery residents’ interpersonal communication skills remains challenging. First, standardized and validated metrics are needed. Second, opportunities to observe trainee communication with patients in the clinical setting may be limited. Difficult conversations with patients and family members are often reserved for attending surgeons, leaving residents to model communication skills through the “hidden curriculum.”⁷

We approached this challenge through simulation-based education and by testing assessment tools previously developed for patient use. Development of nontechnical skills related to emotional intelligence training may be especially suited for simulation,8, 9, 10, 11 as it empowers trainees to sharpen their skills before interacting with patients and creates opportunities for real-time feedback and evaluation.12, 13, 14, 15, 16, 17, 18 We chose the Communication Assessment Tool (CAT), previously developed for patients to assess physician communication skills across a breadth of specialties¹⁹ and recently validated for use in assessing general surgery resident communication.²⁰ The purpose of this study is to provide validation evidence for usage of the CAT in simulation-based communication scenarios to assess general surgery residents’ interpersonal communication skills.

Section snippets

Setting

Communication scenarios were completed by 21 general surgery residents as part of a mandatory patient-centered communication curriculum.²¹ Standardized participants (SPs) rated general surgery residents’ performance delivering bad news in a simulated scenario. The Institutional Review Board determined that the research met criteria for exemption to study standard educational practices. Residents were given the opportunity to opt out of contributing their data for research; none elected to opt

Communication Scores

Residents’ mean SP-CAT rating was 45.6 (median: 43). The total SP-CAT rating scores ranged from 34 to 61. The average percentage of items rated as “Excellent” was 5%. Only 24% of residents (5/21) received any rating of “Excellent”; among those, the proportion of “Excellent” ratings ranged from 7% to 35% of SP-CAT items. Mean ratings by item are presented in Table 3. The highest rated items were “paid attention to me (looked at me, listened carefully)” (mean: 3.62) and “let me talk without

Conclusions

The need for formal communication skill assessments in surgical training programs has only recently been recognized. For most of the history of surgical training, the question of communication skill assessment has either been ignored or evaluated through the apprenticeship system of an attending surgeon informally observing trainees. The increasingly recognized importance of nontechnical skill assessment and related structured curricula, and the creation of mandated milestone reporting have

References (32)

S.K. McKinley et al.
Emotional intelligence and simulation
Surg Clin North Am
(2015)
S.K. McKinley et al.
Are there gender differences in the emotional intelligence of resident physicians?
J Surg Educ
(2014)
H. Wehbe-Janek et al.
An evaluation of the usefulness of the standardized patient methodology in the assessment of surgery residents’ communication skills
J Surg Educ
(2011)
A.C. Larkin et al.
Human Emotion and Response in Surgery (HEARS): a simulation-based curriculum for communication skills, systems-based practice, and professionalism in surgical residency training
J Am Coll Surg
(2010)
S.E. Raper et al.
Improving communication skills: a course for academic medical center surgery residents and faculty
J Surg
(2015)
G. Makoul et al.
Measuring patient views of physician communication skills: development and testing of the Communication Assessment Tool
Patient Educ Couns
(2007)
J.M. Stausmire et al.
Measuring general surgery residents’ communication skills from the patient’s perspective using the Communication Assessment Tool (CAT)
J Surg Educ
(2015)
N. Sevdalis et al.
Trait emotional intelligence and decision-related emotions
Pers Individ Dif
(2007)
D. Nelis et al.
Increasing emotional intelligence: (how) is it possible?
Pers Individ Dif
(2009)
P.A. Harris et al.
Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support
J Biomed Inform
(2009)

Crossing the Quality Chasm: A New Health System for the 21st Century

(2001)

D. Grewal et al.

Emotional intelligence and graduate medical education

J Am Med Assoc

(2008)

S. Arora et al.

Emotional intelligence in medicine: a systematic review through the context of the ACGME competencies

Med Educ

(2010)

M.K. Swami et al.

Emotional intelligence, perceived stress and burnout among resident doctors: an assessment of the relationship

Natl Med J India

(2013)

C. Taylor et al.

Perspective: Can emotional intelligence training serve as an alternative approach to teaching professionalism to residents? Academic Medicine

(2011)

The General Surgery Milestone Project

J Grad Med Educ

(2014)

Cited by (29)

Defining Practice Readiness in Pediatric Surgery Fellowship: A Qualitative Analysis
2024, Journal of Surgical Education
While competency-based training is at the forefront of educational innovation in General Surgery, Pediatric Surgery training programs should not wait for downstream changes. There is currently no consensus on what it means for a pediatric surgery fellow to be “practice-ready”. In this study, we aimed to provide a framework for better defining competency and practice readiness in a way that can support the Milestones system and allow for improved assessment of pediatric surgery fellows.
For this exploratory qualitative study, we developed an interview guide with nine questions focused on how faculty recognize competency and advance autonomy among pediatric surgery fellows. Demographic information was collected using an anonymous online survey platform. We iteratively reviewed data from each interview to ensure adequate information power was achieved to answer the research question. We used inductive reasoning and thematic analysis to determine appropriate codes. Additionally, the Dreyfus model was used as a framework to guide interpretation and contextualize the responses. Through this method, we generated common themes.
A total of 19 pediatric surgeons were interviewed. We identified four major themes from 127 codes that practicing pediatric surgeons associate with practice-readiness of a fellow: skill-based competency, the recognition and benefits of struggle, developing expertise and facilitating autonomy, and difficulties in variability of evaluation. While variability in evaluation is not typically included in the concept of practice readiness, assessment and evaluation were described by study participants as essential aspects of how practicing pediatric surgeons perceive practice readiness and competency in pediatric surgery fellows. Competency was further divided into interpersonal versus technical skills. Sub-themes within struggle included personal and professional struggle, benefits of struggle and how to identify and assist those who are struggling. Autonomy was commonly stated as variable based on the attending.
Our analysis yielded several themes associated with practice readiness of pediatric surgery fellows. We aim to further refine our list of themes using the Dreyfus Model as our interpretive framework and establish consensus amongst the community of pediatric surgeons in order to define competency and key elements that make a fellow practice-ready. Further work will then focus on establishing assessment metrics and educational interventions directed at achieving such key elements.
Assessment of surgeon communication skills from the patient perspective: A national evaluation using the Communication Assessment Tool
2022, Patient Education and Counseling
Citation Excerpt :
Recently, the Communication Assessment Tool (CAT), an instrument that measures patient perceptions of the extent to which physicians accomplish essential interpersonal and communication skills, was translated and culturally adapted to the Italian context in a surgical outpatient clinic [13,14]. The CAT has been used to evaluate surgical residents’ patient-communication skills [15–17], but few data are available on the skills of practicing surgeons. The aims of this study are to validate the CAT at national level in surgical outpatient clinics across Italy and to evaluate the communication skills of practicing surgeons from the patient perspective.
The Communication Assessment Tool (CAT) has previously been translated and adapted to the Italian context. This national study aimed to validate the CAT and evaluate communication skills of practicing surgeons from the patient perspective.
CAT consists of 14 items associated with a 5-point scale (5 = excellent); results are reported as the percent of ‘‘excellent’’ scores. It was administered to 920 consenting outpatients aged 18–84 in 26 Italian surgical departments.
The largest age group was 45–64 (43.8%); 52.2% of the sample was male. Scores ranged from 44.6% to 66.6% excellent. The highest-scoring items were “Treated me with respect” (66.6%), “Gave me as much information as I wanted” (66.3%) and “Talked in terms I could understand” (66.0%); the lowest was “Encouraged me to ask questions” (44.6%). Significant differences were associated with age (18–24 year old patients exhibited the lowest scores) and geographical location (Northern Italy had the highest scores).
CAT is a valid tool for measuring communication in surgical settings.
Results suggest that expectations of young people for communication in surgical settings are not being met. While there is room to improve communication skills of surgeons across Italy, patients highlighted the greatest need in the Central and Southern regions.
Reflections on the difficult patient
2021, Medicina Clinica
Training Surgical Residents to Communicate with Their Patients: A Scoping Review of the Literature
2021, Journal of Surgical Education
Citation Excerpt :
Systematic efforts to address deficiencies provide evidence that interventions promoting communication skills lead to significant increases in patient-centered practice.9 Educators and researchers have developed tools for understanding important communication tasks during patient interviews and these have been variously applied in surgical education and assessment.10-12 The importance of communication skills training in surgical specialties is gaining momentum.13
Effective communication is critical in surgical practice and the Accreditation Council on Graduate Medical Education requires a focus on interpersonal and communication skills. Absent a national communication skills curriculum for surgical residents, individual programs have designed and implemented their own curricula. This scoping review explores communication training in North American surgical residency programs.
The review was conducted according to PRISMA guidelines, and included articles identified through searches of 6 publication databases conducted on June 25, 2019 and updated on April 2, 2020. Eligible studies described patient or family communication skills education for surgical residents in North America. Published abstracts were excluded. Article abstracts were screened by 2 reviewers. For articles meeting criteria, data on study setting, participants, curriculum design, educational techniques, and skills focus were extracted and analyzed.
A total of 2724 articles were identified after removing duplicates; 84 articles were reviewed in full text. Thirty-three met study criteria including 4 from 2006 to 2009 and 29 after 2010. The most common skills education focused on breaking bad news, followed by a range of topics, such as empathy, end of life and/or palliative care discussions, medical error disclosure, shared decision making, and informed consent. Some of the training was part of larger curriculum including team training or professionalism. Over half of the studies involved general surgery learners. The most common form of education included didactics followed by practice with peers or simulated patients. Only half of the programs described curricula involving multiple training sessions which is important for ongoing development and skill reinforcement.
Effective communication skills are important, necessary, and increasingly incorporated into surgical training programs. While the literature is expanding, few surgical residency programs have described communication curricula. This review serves as a guide for programs interested in developing their own communication curricula for surgical residents.
Multi-Disciplinary Trauma Evaluation and Management Simulation (MD-TEAMS) training for emergency medicine and general surgery residents
2021, American Journal of Surgery
Citation Excerpt :
The ability to study the link between communication and essential task completion in a simulation setting is an attractive method to assess communication-based curricula and creates important data for the ability to assess the link between communication and patient safety. Several communication tools have been developed to score the communication skills of residents in patient care settings.27,28 Many of these address resident communication with a patient and not within a team, and thus, cannot be used for trauma scenarios.
Successful trauma resuscitation relies on multi-disciplinary collaboration. In most academic programs, general surgery (GS) and emergency medicine (EM) residents rarely train together before functioning as a team.
In our Multi-Disciplinary Trauma Evaluation and Management Simulation (MD-TEAMS), EM and GS residents completed manikin-based trauma scenarios and were evaluated on resuscitation and communication skills. Residents were surveyed on confidence surrounding training objectives.
Residents showed improved confidence running trauma scenarios in multi-disciplinary teams. Residents received lower communication scores from same-discipline vs cross-discipline faculty. EM residents scored higher in evaluation and planning domains; GS residents scored higher in action processes; groups scored equally in team management. Strong correlation existed between team leader communication and resuscitative skill completion.
MD-TEAMS demonstrated correlation between communication and resuscitation checklist item completion and communication differences by resident specialty. In the future, we plan to evaluate training-related resident behavior changes and specialty-specific communication differences by residents.
Development and assessment of a simulation-based curriculum in pediatric surgical education: Conventional wisdom and lessons learned from the national training program in France
2020, Seminars in Pediatric Surgery
Traditional training in pediatric surgery has focused on knowledge acquisition and technical skills. Although recognized as important, the skills relating to communication, collaboration, leadership, management, health advocacy and professionalism have received less formal attention. Recognition of the importance of these softer skills in combination with the greater use of simulation in education prompted us to develop a national simulation-based curriculum in pediatric surgery. This paper describes the development, implementation and three-year follow-up of the mandatory, simulation-based curriculum for pediatric surgery residents in France. We review the available literature on the topic and present our results.

View all citing articles on Scopus

View full text

2016 APDS SPRING MEETINGAssessment of Surgery Residents’ Interpersonal Communication Skills: Validation Evidence for the Communication Assessment Tool in a Simulation Environment

Objectives

Design

Setting

Participants

Results

Conclusions

Introduction

Section snippets

Setting

Communication Scores

Conclusions

Surg Clin North Am

J Surg Educ

J Surg Educ

J Am Coll Surg

J Surg

Patient Educ Couns

J Surg Educ

Pers Individ Dif

Pers Individ Dif

J Biomed Inform

Crossing the Quality Chasm: A New Health System for the 21st Century

Emotional intelligence and graduate medical education

J Am Med Assoc

Emotional intelligence in medicine: a systematic review through the context of the ACGME competencies

Med Educ

Emotional intelligence, perceived stress and burnout among resident doctors: an assessment of the relationship

Natl Med J India

Perspective: Can emotional intelligence training serve as an alternative approach to teaching professionalism to residents? Academic Medicine

The General Surgery Milestone Project

J Grad Med Educ

2016 APDS SPRING MEETING
Assessment of Surgery Residents’ Interpersonal Communication Skills: Validation Evidence for the Communication Assessment Tool in a Simulation Environment