Developing a team-based assessment strategy: direct observation of interprofessional team performance in an ambulatory teaching practice [version 1; peer review: awaiting peer review]

Background: High functioning interprofessional teams may benefit from understanding how well (or not so well) a team is functioning and how teamwork can be improved. A team-based assessment can provide team insight into performance and areas for improvement. Though individual assessment via direct observation is common, few residency programs in the United States have implemented strategies for interprofessional team (IPT) assessments. Methods: We piloted a program evaluation via direct observation for a team-based assessment of an IPT within one Internal Medicine residency program. Our teams included learners from medicine, pharmacy, physician assistant and psychology graduate programs. To assess team performance in a systematic manner, we used a Modified McMaster-Ottawa tool to observe three types of IPT encounters: huddles, patient interactions and precepting discussions with faculty. The tool allowed us to capture team behaviors across various competencies: roles/responsibilities, communication with patient/family, and conflict resolution. We adapted the tool to include qualitative data for field notes by trained observers that added context to our ratings. Results: We observed 222 encounters over four months. Our results support that the team performed well in measures that have been iteratively and intentionally enhanced – role clarification and conflict resolution. However, we observed a lack of consistent incorporation of patient-family preferences into IPT discussions. Our qualitative results show that team collaboration is fostered when we look for Open Peer Review Reviewer Status AWAITING PEER REVIEW Any reports and responses or comments on the article can be found at the end of the article. MedEdPublish Page 1 of 9 MedEdPublish 2021, 11:6 Last updated: 06 DEC 2021


Introduction
Team-based care can be found in a variety of practice settings, including primary and acute care; commonly this model incorporates the expertise of multiple professions into a health care team 1 . Benefits of team-based care include reduced errors, improvement in communication, clinical outcomes and patient satisfaction 2-4 . Given the need to manage complex chronic illnesses, interprofessional (IP) teams in primary care have become standard practice 5,6 , as reflected in the emphasis on care teams in the Patient Centered Medical Home model 7 . High-functioning teams must assess how the team members work together using clearly defined competencies. Accreditation bodies have well-defined individual competencies via direct observation such as standardized patients and the mini-Clinical Evaluation Exercise (CEX) 8 and various evaluation structures are used such as 360 degree-evaluations. However, residency programs must develop strategies to train and assess students and residents as individuals and also as members of a team. Competencies of team performance in outpatient settings is a growing area of study.
Recognition that well-functioning teams require development 9 , and that teams are not simply a collection of "soloists" 5 , led to a call to action to develop interprofessional education (IPE) models 1,10 . In its "Framework for Action on Interprofessional Education & Collaborative Practice", the World Health Organization states interprofessional education "occurs when students from two or more professions learn about, from and with each other to enable effective collaboration and improve health outcomes" 1 . IPE is a necessary step to develop practice-ready teams comprised of health care workers who are competent in IP interactions. This competency allows true interprofessional collaborative (IPC) practice when "multiple health workers from different professional backgrounds work together with patients, families, care givers and communities to deliver the highest quality of care" 1 .
As more training programs enter the IPE and IPC realm, new challenges arise, which include lack of IP expertise and role models, resistance to change, and moving from educational and clinical silos to shared practice. This transition from solo practice to team-based practice necessitates the development of new skills: teamwork, identifying roles and responsibilities, communication among team members, critical reflection, and working collaboratively in the best interest of the patient 1,11 . Likewise, evaluation models must move from assessment of performance of a single profession to that of the team 12,13 .
The goal of this study was to pilot an IP team assessment of these competencies via direct observation of a team of trainees from different professions in an Internal Medicine teaching practice. Prior studies evaluating IP primary care teams have assessed time speaking 14 , the personal experiences of team members and team configuration models [15][16][17] . Still, we need strategies that will help evaluate team behaviors and processes beyond self-reported surveys 18 . Direct observation provides objective data that cannot be garnered from self-report: the approach is viewed as a continuous quality improvement process for our team. We adapted a standardized data collection tool, Modified McMaster-Ottawa Scale 19 , to document observer field notes to elaborate on the behaviors observed to add context to our ratings. This tool was selected to help us evaluate our team processes as we were more interested in evaluating our processes versus outcomes at this phase in the program.

Methods
This manuscript describes an evaluation and educational needs assessment of the Improving Patient Access, Care and Cost through Training (IMPACcT) program, which is a clinic-within-a-clinic model of a large academic medical center in the Northeast United States that covers suburban and urban regions. The IMPACcT program is a general internal medicine/ primary care clinic that focuses on patient-centered care, quality improvement, and population health in an IP, team-based setting 20,21 . We considered this a pilot as we planned to apply lessons learned to future team assessment evaluations for the entire IM residency program.

Ethics statement
The Northwell IRB determined this evaluation was not research involving human subjects and therefore the study was exempt from full IRB review. We obtained verbal consent for participation.

Program structure
The IP team composition consisted of the following trainees: students from medicine, pharmacy and physician's assistant programs as well as a psychology intern. The faculty included medical preceptors, pharmacy preceptors and a clinical psychologist. The Internal Medicine residents followed a 4+1 schedule 22 . The teams would fluctuate daily as each professional school followed varying schedules.
The clinic session began with a huddle and then followed standard resident clinic workflows. Huddles are brief group meetings that occurred at the beginning of every session where the team reviewed the patient's medical problems and preventive screening needs 21 . Patient interactions occurred between the medical care providers and patient in private exam rooms. Precepting occurred after the learner (i.e. resident, student) returned to the team office to discuss the patient with the attending preceptor and other team members.

Data collection
Observations were performed by a rating team consisting of a medical attending, a medical student, and two volunteer undergraduate students. The medical attending trained the student raters to observe three types of clinic encounters: huddles, precepting discussions, and patient interactions. The rating team was independent of the IMPACcT program in that they were not participants in providing direct care in the IP clinic. If an observer was present in the room during a patient interaction, the care team would first introduce the observer to the patient and ask for the patient's permission to have the observer present. Raters attended a two-hour training session led by the medical attending to standardize use of the assessment tool.
Through iterative discussions and review of the competencies measured in the tool, they were taught to evaluate the team process, which included noticing the language and behaviors of the team members. For the first two weeks of the study period, onsite experiential training continued as observers would overlap sessions in order to check for consistency in ratings. The team would then debrief after the sessions to understand differences in observations and to ensure some similarity in understanding how behaviors were measured.
Observations occurred between March and July 2018. Observation of a team encounter (i.e. huddle, precepting discussions, or patient interaction) by a rater represented a unit of analysis. Patient encounters that involved more than one team member were prioritized for observations. For example, if an Internal Medicine resident was conducting an annual physical exam without another student then we would not observe such a visit. An observation schedule was developed to ensure that the various teams led by each of the five attending physicians were observed at least once. The team prioritized observations on a day when the complete IP team was present. The team did not necessarily seek to observe the same team over time. Debrief was not provided after each encounter. Instead, the team was provided with a presentation of overall suggestions for improvement for the entire IMPACcT program.

Evaluation tool
We used the Modified McMaster-Ottawa Scale, which was developed to assess the performance of healthcare teams according to six competencies: communication, collaboration, roles and responsibilities, collaborative patient-family centered approach, conflict management and resolution, and teamwork/team functioning 23,24 . Each competency, as well as a global score, is rated as "below expected"; "at expected"; or "above expected" according to specific criteria. We simplified the rating scale to assist raters to detect key competencies for each type of team encounter. The back of the rating sheet outlined behaviors to remind the raters of how the competencies are defined. The professional composition of the team and the duration of the team interaction were also recorded. Raters were also trained to keep field notes to record any qualitative information about the interaction not reflected in the objective rating. Finally, raters were also asked to detail their justification for rating an encounter when assessed as below or above expectations. The evaluation team agreed that some competencies would not be present in some interactions. For example, "communication with patient" is not expected to be observed during the huddle as patients are not present during the huddle.

Data analysis
We analyzed the data via quantitative and qualitative methods. First, we calculated average duration of the IP team interactions and team members present by each encounter type. Though significance testing of each of the six competencies by the three encounter types was attempted via Chi Square, some counts were too small to obtain valid results. Subgroup analyses looked at the presence of pharmacy and behavioral health to the differences in competency ratings. In addition, we calculated correlation coefficients among raters who completed observations on the same day. Our analyses were conducted using SAS software 9.4. Qualitative analysis of field notes from the 222 observation rating sheets that described behaviors of the IP team interactions was performed using thematic analysis as described by Braun and Clarke 25 . An inductive approach was used to generate themes based on the comments themselves using constant comparative methods. The field notes were separated into two rating categories: encounters that had a "below expected" rating or "at or above expected" on any competency or global rating. The data were independently coded by hand by two raters (LM and AD). The authors agreed on the coding scheme within the analytical framework, discussed findings, and recoded the data until saturation was met. All coding was reviewed by the co-authors. In the event that there was disagreement on a code, the issue was discussed until a consensus was reached. A third researcher (AF) served as arbiter of disputes, who also read the final themes for consistency with the original research question. A description of the themes, as well as representative quotes are included.

Quantitative results
Our raters observed 233 total encounters, 222 of which were labeled and used for analysis (i.e. we excluded the 11 with missing labels). Huddles lasted on average 23 minutes and included a mean of six members. Table 1 summaries the IP team encounters that were observed. Five of the six competencies yielded responses for which the majority were "at" or "above expected." With a total of 149 IP precepting encounters, this represented the majority of our observations. The average duration was 11.5 minutes with a mean of four team members present. For precepting, the majority of the following competencies were observed to be "at expected": team collaboration, conflict management, and responsibilities and roles. We note that in 41% of encounters, roles and responsibilities were rated "above expected". In comparison, collaborative patient-family approach was rated "below expected" in 32% of encounters and "at expected" in 41% (see Figure 1-Figure 3); this domain represented the area in need of greatest improvement.
The episodes that observed patient interactions represent the encounters in which the competency for communication with patient was observed. The program noted that 69% of the Precepting discussions 149 11.5 4  encounters were rated "above expected." In regard to collaboration with team, roles and responsibilities, conflict management, team functioning and global scale, the majority of episodes were "at expected." Collaborative patient family approach showed a wide distribution with 23% of encounters being rated "below expected"; 38% "at expected" and 33% "above expected." Results of interrater reliability resulted in a range of correlation coefficients from 0.15 to 0.7.

Qualitative analysis
Of the 222 IP encounters, there were 76 (34%) that had a "below expected" rating in at least one of the six competencies or the global rating scale. The common themes for those that were rated below expected include: 1) lack of inclusive language; 2) limited attempt to engage team members; and 3) failure to incorporate patient's social situation in the care delivery. We were able to identify process concerns, communication gaps, as well as behaviors that can be improved upon to improve team performance. Table 2 outlines additional themes with associated notes for encounters that were "below expected" and "at or above expected" among the three types of IP encounters observed.

Discussion
In our team-based primary care training program, we observed the team members during huddles, precepting discussions, and patient interactions. Observers recorded team collaboration, role clarity, patient-centered approach, conflict resolution, and team functioning; competencies that reflect concepts at the core of IPE 2 . For each of the settings observed, all domains except for patient-family centered approach were rated at or above expected for at least 90% of observations. This competency, patient-family centered approach, had the most room for improvement: ~13%, 24%, and 33% of huddles, precepting discussions, and patient interactions were rated "below expectations." While the team scored well in remaining domains, qualitative data did permit feedback on more specific opportunities for improvement. For example, the need to use more inclusive language, the timeliness of huddles and opportunities to use various team members to avoid a physician-centric culture. This type of team assessment is able to provide feedback on IP team dynamic and processes in an ambulatory training setting.
IP education has been recommended by multiple organizations to prepare healthcare professionals to work collaboratively and is associated with improvements in patient-related outcomes 26 . A collaborative approach to primary care is associated with positive outcomes for patients and providers, but has not been widely adapted in training settings 27,28 . Our results demonstrate that training settings can create learning spaces that teach IPE competencies. As our team had over a year to define roles, work together across educational and clinical settings, and navigate obstacles, these domains were observed to function well. Team members were noted to teach each other, plan patient care activities to maximize efficiency, and provide input using their own areas of expertise. These processes represent collaborative learning and care delivery.
Collaborative patient-family centered approach was noted to be the weakest of the domains observed. Several reasons for this lower rating exist. First, perhaps this competency was not built into the established team workflows. Huddles are conducted before the sessions take place, when the patient voice may not yet be known. Second, as the healthcare setting operates under time constraints, this aspect of care may be the one that is most neglected. Patient and precepting interactions may be time pressured with the need to deliver care and teach; thus visits remain medically−rather than socially−focused. Consistent with quantitative data, qualitative comments included a lack of comment on social and family structure during patient interactions and precepting discussions. Providers can display more patient-centered care by emphasizing communication skills that show empathy and respect 29 . Last, we can also debate if this competency is expected to be present in all the observed encounters.
Strengths of this approach included observations across the team rather than specific individuals. A large number of observations was performed across various patient care activities to facilitate a comprehensive assessment of team functioning. This broad assessment was able to capture the deliberate approach required in training the professions to work together across all aspects of our care delivery model. An alternative team was created for the direct observation evaluation rather than using office staff or faculty who would interact with or evaluate the trainees. A study period was chosen over a year after establishment of the team-based clinic, after which workflows and expectations had been set, to allow for observation during the "performing" stage of group development 9 . An evidence-based tool was used that had been validated across health professions and can be used efficiently within fast-paced clinical settings with behavior-specific anchors 19,23,24 . Attending still seeing own patients The other resident is not here. (Staff) was unaware but his name is up on the white board.
There was a lot of "I" will solve, not "we" will solve...

Precepting
At/above expected Teaching across the professions; between pharmacy and medicine teams Spatial arrangement -facilitates the discussion when group sits in a circle Resident provides lots of teaching to PA student...great interaction between the two but could also pull in the pharmacy student… Lots of back and forth between team members when discussing plan; sharing of perspective

Patient interaction
At/above expected Plan of action prior to entering the room, who will be examining the patient, when pharmacy will do med rec. This allows things go more smoothly.
Benefits of having pharmacy present, expertise is good to have for the patient in real time Though one tool will never be perfect in team assessment, understanding the competencies that are important in team dynamics and that can potentially impact patient care and outcomes is important to determine for leaders in ambulatory training programs. As we added qualitative data to our tool, this helped to reveal trends and allow for reflection and strategies for improvement on these trends.
There were several limitations in this single-institution evaluation, conducted at a single clinical site with well-established faculty and trainees. Assessment of inter-rater reliability was limited, but we did attempt to find correlations between observers completing ratings on the same dates. We did note an increase in correlation over time, suggesting the raters increasingly were seeing and interpreting behaviors similarly. Data were collected on team functioning, which was done to limit individual blame, but also limited individual feedback. A limited dataset of observations did not allow for a robust assessment of whether below expected competencies were spread across faculty or limited to specific faculty or team members. While six key domains of team functioning were addressed, any single tool cannot be a comprehensive assessment of team functioning, and the patient perspective was not addressed in this evaluation. Future iterations would focus on specific teams, provide detailed timely feedback and evaluate for improvement over time in team performance. Though the raters had experience working in teams, they did not have formal healthcare team assessment experience prior to this evaluation. In addition, we acknowledge that team assessment can be a challenge for those with limited resources. Last, the possibility of persons changing their behavior as a result of being observed (i.e. Hawthorne effect) is possible, and it remains unclear the extent to which this affects the documented findings.
In addition to planned educational workshops on patient-centered communication and workflows, these data offered additional next steps to improve team functioning. Structured time for the patient access coordinator or other team members to address patient and family dynamics can be added to huddle and precepting workflows. Scheduling patients and key educational activities around the schedules of pharmacy and psychology team members can maximize their input and integration into the team. Protocols for streamlining language for patients with limited English proficiency can ensure understanding among all team members, as our qualitative data noted communication gaps specifically with patients who spoke languages other than English.
Team-based assessment is becoming as important as individual assessment in training programs. Individual assessment−required throughout one's training−is common via mini-CEX and other modalities. Prior studies in primary care on team performance show that training in teamwork can improve team performance 30 . We demonstrate the need to include direct observation of a team in order to improve the competencies of teamwork; beyond individual performance.

Conclusion
While key features of high performing primary care teams are known, optimizing the performance of an existing team is not straightforward 31 . Particularly in a residency-based practice, team-based care may be limited by rotating trainee and faculty schedules, continuity of care, and time pressures 32 . Direct observation using a standardized tool offers an option to highlight areas to improve team functioning. In our team-based residency practice, direct observation revealed that while teambased aspects of care, including role clarification and conflict resolution worked well, an opportunity existed to integrate a patient and family-centered approach. Direct observation offers a viable process for continuous quality improvement in team collaboration 33 .