Group Trust in the Setting of Competency Committee Decisions: A Qualitative Observational Study

Introduction: Trust is an important foundational component of a competency committee. Little work has been done in health professions education to look at trust and group decisions. A recent scoping review defined "group trust," examined factors that may influence group trust, and proposed a model to illustrate the relationship between trust at the individual and group levels. The purpose of this study was to test this model in the context of competency committee decisions and assignment of the ACGME Milestones. Methods: The authors conducted structured observations of competency committee meetings in internal medicine and pediatrics residency and subspecialty programs within the National Capital Consortium. Data were analyzed using a constant comparison method. Results: The authors observed six meetings from May to June 2018 (15 hours) reaching thematic saturation by the final meeting. The proposed model served as an organizing framework for reporting findings into the following categories: individual and group level factors, interpersonal interactions, and environmental factors. Results suggest that diversity of opinion promotes a more complete understanding of trainee competence. Varied faculty experiences, interactions, and actions by committee chairs helped to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones. Meetings generally had similar structure based upon the size of the training program, and groupthink was more of a risk for larger training programs, especially when assigning individual Milestones. An important environmental factor identified was decision making fatigue, which was most prevalent towards the end of committee meetings. Conclusions: As training programs develop policies, procedures, and determine the membership of competency committees, utilizing these findings may help improve the design and execution of these groups, enhancing the translation of evaluation data and individual opinions into Milestone and competency decisions. Sapp J, Larsen K, Torre D, Holmboe E, Durning S


Introduction
Developing a process to assess clinical competence is an essential component of all graduate medical education (GME) programs. Built upon the fundamentals of competency-based medical education, the Accreditation Council for Graduate Medical Education (ACGME) Milestones help GME programs determine which of their trainees are on appropriate paths towards unsupervised clinical practice and which are not. Within a GME program, the clinical competency committee (CCC) forms a fundamental part of determining trainees' progress in the context of the Milestones, and some recent studies describe data sources that CCCs use and how they weigh this information in their collective decisions (Choe et al., 2016;Shumway et al., 2015). Several other studies have also looked at group decision making processes and how these concepts may be utilized by CCCs (Chahine et al., 2017;Hauer et al., 2016;Hemmer and Kelly, 2017).
Given the important decisions shouldered by the CCC (e.g. overall progress, trainee promotion, and remediation), trust is an important foundational component of these committees, enabling open and collaborative discussions of strengths and areas of improvement, to include recommendations of remediation and dismissal. In the context of individual trainee-supervisor relationships, health professions education research has explored trust in decisions that faculty make about the level of supervision granted for different clinical responsibilities and the factors that influence those decisions (Damodaran, Shulruf, and Jones, 2017;Hauer, 2015;Cate et al., 2016). Entrustable Professional Activities (EPAs) are an example of these decisions and address tasks that trainees are permitted to execute once they attain a sufficient level of competence; the level of supervision for each EPA depends upon how much a supervisor trusts a trainee to complete that specific task (Cate, 2013). While individual decisions of trust are important, little work has been done in health professions education to look at trust as it relates to group decisions, such as trust within a CCC, trust in the context of how characteristics and processes of these committees might impact collective decisions, and group member trust in these decisions.
In a recent scoping review, we defined group trust (which encompasses both trust within a group as an aggregate or combination of individual trust and trust in the context of group decisions) as a group-directed willingness to accept vulnerability to actions of the members based on the expectation that members will perform a particular action important to the group, encompassing social exchange, collective perceptions, and interpersonal trust. We also examined literature outside of health professions education to determine what factors may influence trust from the perspective of those who rely on decision making groups and proposed a model (Figure 1) to illustrate the relationship between trust at the individual and group levels (Sapp et al., 2019). In this model, we delineate individual level factors, group level factors (i.e. group structure and processes), interpersonal interactions, and environmental factors. As empirical data in a health professions education context has not been collected for this literature-informed model, we conducted the current study with two primary purposes: a) test our proposed model in the context of CCC collective decisions and assignment of the Milestones in primary care (i.e. internal medicine and pediatrics) and subspecialty settings and b) assess what factors are important to individual faculty members regarding trust in the setting of CCC decisions.

Theoretical Framework
For decision making groups to reach a consensus, communication and interactions between and among people, objects (those items that extend human capabilities in a learning or task-oriented setting, such as learning management systems), and the environment are believed to be essential. We therefore used situated cognition as the Sapp J, Larsen K, Torre D, Holmboe E, Durning S MedEdPublish https://doi.org/10. 15694/mep.2019.000066.1 Page | 3 underlying theoretical framework for our model (Sapp et al., 2019). Situated cognition contends that thinking is viewed as situated (or located) within the larger social and physical context of the environment (Durning and Artino, 2011). This theory suggests that a group setting is highly complex with multiple components (i.e. social, cultural, and physical) and opportunities for interactions between these components. It recognizes the complex interplay between participants, objects that augment individuals' cognitive capabilities (i.e. artifacts), and the environment. Situated cognition places emphasis on how these various interactions lead to thinking and decisions, and this framework informed our model.

Methods
We conducted a qualitative observational study involving GME program CCCs at our institution to determine how collective decisions about trainee progress within these programs are made in the context of the Milestones. We focused on internal medicine and pediatrics residency and subspecialty fellowship programs within the National Capital Consortium because they represent the largest population of GME trainees within our institution.
Given the social and group dynamics underpinning CCCs and their decisions, along with the multiple inputs to our model, we believe that a qualitative approach was optimal for this study. Our qualitative approach also allows for iterative construction of the results based upon the emergence of conceptual categories and descriptive themes.

Data Collection
Before collecting data we developed a structured observation guide (included as a supplemental file) based on our literature review and proposed model (Figure 1) to measure various components of trust and to focus data collection on a CCC's structure and processes. We also held conversations with subject matter experts who informed revisions to the guide. The structured guide included the following headings along with a sample of items within those sensitizing concepts: individual-level factors (i.e. observed behaviors that may affect interactions amongst group members, group decisions, etc.), group structure (e.g. group size and diversity), group processes (e.g. group conflict, procedures utilized by the CCC, overall climate of the group, and leadership), information sharing/interpersonal interactions, and environmental factors (e.g. task complexity, uncertainty, and threats). Next we pilot tested the guide by observing four CCC meetings in internal medicine and adult endocrinology (amounting to 10.5 hours of observation) from September 2017 to May 2018 and made revisions based upon observations in these meetings. Observers were part of the research team, had the opportunity to practice observing during these four CCC meetings, and helped to revise the observation guide.
We sent out requests to a purposive sample of programs to observe their CCC meetings during the June 2018 endof-year Milestone reporting period. For programs that elected to participate, we conducted structured observations of the CCC meetings using the revised observation guide. We had two observers in the room, and they observed as spectators (as opposed to a participant observer). To minimize interference in the CCC meeting, observers sat outside the group circle along the perimeter of the room. Observations were from an outside perspective, and the inquiry of the observation was directed entirely by the observers (i.e. there was no collaboration with those being studied). Observers provided full disclosure of their role to those being observed and independently recorded field notes utilizing the structured observation guide. At the end of each CCC, the lead observer asked the committee, "what do you think contributes to trust in CCC decisions?" Observers collected information provided by individual committee members on their field notes. Field notes were independently read and coded by the two CCC meeting observers (JS, KL). To code and analyze the data, we used a constant comparison method (Fram, 2013) to identify and confirm themes and/or topics that emerged from the observations and discussions. We looked at published approaches to group decision making (i.e. individual and group level factors, environmental factors, the interactions between all of these elements, and measurable components of trust), how CCCs operationalize this information, components of trust in the setting of decisions made by these groups, and the emergence of new concepts or themes. We sought counter-examples to determine how new concepts or categories may vary from the published literature and how this impacts the process of group decision making. Data analysis occurred simultaneously with the collection of data in order to monitor for thematic saturation and the need for any new directions in data collection. The research team (JS, KL, DT, SD) met regularly to discuss coding and resolve all differences. During analysis, memoranda and diagrams were used to keep written records of the process. Techniques to enhance trustworthiness of the qualitative analysis process per the Standards of Reporting Qualitative Research included member checking, maintaining an audit trail, and collection techniques to allow triangulation of the data (O'Brien et al., 2014).

Ethics
Our research was approved by the Walter Reed National Military Medical Center Institutional Review Board (IRB; WRNMMC-2018-0137). All participating training program directors and individual CCC faculty members provided written consent prior to their participation in the study.

Results/Analysis
We observed six CCC meetings in the following specialties from May to June 2018: general pediatrics, pediatric endocrinology, neonatal-perinatal medicine, internal medicine, and adult cardiology and hematology-oncology. During this time, we observed a total of 15 hours' worth of meetings amounting to 36 pages of data, and we reached thematic saturation before the sixth CCC meeting. Our proposed model served as an organizing framework for reporting our findings. See Appendix 1 for themes that were observed during these meetings, with representative quotes, which are organized under various components of the model (Figure 1) and discussed below. We aggregated data from the observations and the debrief discussions into the sections below and contend that the different components of the model may affect group trust.

Group Structure
The size of the committees varied from 5 to 11 individuals (average 8, standard deviation 2.8). All assigned faculty members were present for the meeting with two exceptions. An absent faculty member on one CCC provided her scoring sheet to the CCC chair in advance of the meeting. On another CCC, the committee had to table the discussion of an absent faculty member's trainees because he had not provided recommendations about the updated Milestone levels for these individuals. Besides physician faculty members, some of the CCCs also included a diabetic nurse educator, nurse practitioner, chief residents who had completed a core residency program, and training program administrative personnel. CCC members commonly reported that a diversity of opinions from different faculty members contributes to a more complete collective understanding/picture of competence for each trainee.

Group Conflict
In all of our observations, we did not witness any instances of affective (relationship) conflict. Rather, during each meeting we observed at least one episode of task (cognitive) conflict, which tends to arise from individual differences in viewpoints, ideas, or opinions when group members participate in a shared undertaking. Observers did not specify how frequently these episodes occurred during a single meeting, and CCC faculty members in general seemed to offer personal experiences that were largely confirmatory and agreeable relative to points about learners raised by the CCC chair or other members. However, "individuals at times volunteered differing comments, but [these experiences] were usually treated as apples-and-oranges differences rather than conflicting experiences Sapp J, Larsen K, Torre D, Holmboe E, Durning S MedEdPublish https://doi.org/10.15694/mep.2019.000066.1 Page | 6 (KL)." When assigning individual Milestones, some differences of opinion occurred. "Occasionally, after committee members suggested scores, other committee members would inquire how they saw the [trainee] as satisfying that score; the majority of inquiries were collegial, open-ended, and were interpreted as clarifications rather than challenges (KL)." Some faculty members felt that dominant voices/personalities may negatively impact a CCC. One member remarked, "It is difficult to achieve a confident consensus when there is a forceful personality in the group, especially if other members of the committee may be more conflict averse." In one CCC, however, a faculty member consistently emerged as a dominant personality in both volume and quantity of commentary. Unless other CCC members were specifically addressed, this individual volunteered, and at times interrupted, conversation an average of four times per each trainee discussion (a contrast to other members generally volunteering thoughts zero to two times per trainee). These comments consisted of generally confirming/restating the point of others, injected with personal experiences from working with various trainees. The other CCC members never attempted to prevent this individual from speaking, nor did they give any indication of annoyance or frustration (rather, there was an indication that they simply were comfortable and did not mind having the forceful personality present).

Group Procedural Fairness
Regarding the assignment of individual Milestones, procedures varied with each training program observed. For smaller programs, such as pediatric endocrinology (3 trainees), the group generally went through each individual Milestone in greater detail with each trainee, and individual trainees were not assigned to faculty members. In cases where trainees had previously achieved high scores on their prior Milestone assignments, committee members would both implicitly assume that those Milestones did not need to be considered ("let's talk globally… I don't want to go line by line…") and at times would explicitly state that those Milestones did not need to be considered ("if it's mostly 4's then we don't really have to do it [as in depth as for others]"). However, this was not universally true; on another CCC, several trainees had their Milestone levels decreased given some faculty concerns about professionalism and completing administrative tasks on time.
For training programs with a larger number of trainees (i.e. internal medicine and pediatrics with 74 and 29 trainees respectively), faculty members were assigned individual trainees and utilized a standardized CCC data collection form that they filled out in advance of the meeting, which included proposing updated Milestone levels for each trainee. During the meeting, faculty members took turns presenting the information off of these forms, which seemed to help standardize the process of discussing trainees and assigning Milestones. Faculty presentations were followed by varying degrees of discussion. In some circumstances, "the group seemed reluctant to make changes to the assigned faculty recommendations (JS)," and the discussion was more likely to involve groupthink (i.e. group members refrained from expressing doubts and judgments or disagreeing with the consensus).
In other CCCs, there also seemed to be less discussion for certain Milestones, especially in areas where the faculty members may not have any observational experience or evaluation data for specific trainees. In some committees, "very rarely were specific Milestone numbers discussed; rather the discussions focused almost entirely on [trainee] skill-sets and general scoring (KL)." Multiple CCC members also indicated that they felt they should not progress trainees in the context of the Milestones just for the sake of progression. Members stated that they should be comfortable keeping trainees at their same level if no clear evidence exists to suggest that the learner has progressed in that specific Milestone. The atmosphere of the meetings was "collegial and positive (KL)" and "comfortable (JS)." Each meeting seemed to be a safe environment where all opinions were welcome and openly discussed. One member stated that "respect for each other is important to build trust, which includes listening to other members when they are talking and sharing." In one CCC, the discussion repeatedly made use of the term "we" and referenced what the team could do as a collective to improve the trainees' performance, which is the hallmark of a common group identity. Further, the cardiology CCC chair stated that consensus in this setting helped to support "[cardiology] community standards." The group agreed that determining when fellows are ready for independent practice provides reassurance to society that these trainees graduate with a basic level of skills to practice cardiology in any setting. Finally, it was clear during some of the group discussions (especially in smaller programs with fewer than 15 trainees) that all of the faculty members knew the trainees well and had worked with them frequently.

Group Leadership
The two most important leader behaviors that we observed were the abilities of the CCC chair to conduct the meeting and to facilitate a shared mental model. In most of the meetings we observed, the chair provided an overview of the Milestones to the group and the structure of the session. "The pace was immediately established by the chair as quick; he explicitly stated the order in which [trainees] would be evaluated (third years first, 'anticipat[ing] it will go quickly'); the pertinent information (timeline details, when Milestones are due, etc), and a brief reminder of the core of the Milestones' meaning (deficiencies, ongoing progress, and aspirations) (KL)." Two different meeting formats emerged during our observations. Some CCCs went through all of the trainees in order (e.g. by post-graduate year, alphabetically, or based upon other external factors) discussing the trainee in general terms and assigning the Milestones at the same time. "It seemed like the faculty who had other competing schedule requirements were given priority to present their [trainees] early (JS)." For other CCCs, meetings generally had two distinct phases. In phase 1 the chair led a discussion to generate formative comments for each trainee, and in phase 2 the group assigned the Milestones for each trainee.
Generating a shared mental model seemed to be one of the most important parts of making group decisions and assigning Milestones, and the CCC chair appeared critical in facilitating this. In CCCs of smaller programs (i.e. fewer than 15 trainees), the chair usually introduced and provided some information about each trainee. "[The CCC chair] began by summarizing the past activities and future plans of each of the [trainees], in addition to comments about exam performances and most recent one-on-one experiences with the [trainee] (KL)." In larger programs, assigned faculty members generally provided this information as discussed above. In all of the meetings, group members tended to freely share their opinions; for instance, in one meeting "the CCC chair did not need to engage individual faculty members because all freely participated when it seemed like they had something to contribute to the discussion (JS)." When one of the members commented about a trainee's performance, other members tended to jump into the conversation to provide their individual experience with the trainee.

Individual Level Factors
Several faculty members felt that individuals on a CCC should have some baseline knowledge and experience in medical education. One CCC member noted that "individuals who serve on the committee should have a basic level of competence as a physician within their specialty and a basic understanding of evaluating and discussing trainee performance." Additional areas identified by the observers include buy in ("faculty should recognize that they bought into training these residents/fellows, and the overall agenda of the group should be to make the learners successful, to the maximum extent possible"); having an open mind ("members should come to the meeting with no preconceived notions or 'labels' for specific trainees"); and advocacy for the trainees ("the group felt that they were Sapp J, Larsen K, Torre D, Holmboe E, Durning S MedEdPublish https://doi.org/10.15694/mep.2019.000066.1 Page | 8 advocates for the trainees, and the group setting allowed them to identify mentors for individual trainees who may be struggling (JS)"). We did not observe any specific comments related to patient advocacy and if any tension may result from advocacy for patients versus trainees.

Imported Information
Information brought to the meeting by individual faculty members is important to helping build a shared mental model. Two main themes that emerged from the observations include information obtained from faculty pre-work assignments and individual experiences with trainees. For CCC decisions, the latter seemed to be the most frequently relied upon source of information. Especially for smaller programs, CCCs frequently used specific individual stories and/or experiences with trainees as a means of evaluation. During the general discussion of each trainee, CCC members would most frequently offer specific experiential anecdotes to illustrate their opinions, both to offer positive and negative evidence regarding trainee performance.

Shared Mental Model
During our observations we noted three general themes whereby CCCs build a shared mental model: assignment of the Milestones, perceptions regarding the developmental level and overall clinical competence of various trainees, and the work and processes of the CCC. When building a shared mental model, CCCs discussed a number of different topics to include trainee global strengths/weaknesses (e.g. praises of bedside manner, maturity and concerns of professionalism, and personality issues), instances in which feedback was given to trainees (and whether they did/did not respond to it), and general discussions of what the CCC members could do and actions they could take in the future in order to aid the trainees and foster additional growth. Indeed, when discussing solutions for questions and/or concerns about certain trainees, CCC members either described specific instances in which they attempted to correct behavior one-on-one or discussed team actions in abstract terms (e.g., "we have to somehow make it clear to [the trainee]…" or "we might need to consider program-level remediation…"). A sample of other topics discussed are listed in Appendix 1.
When there was quick group consensus regarding the Milestones, CCC chairs would generally move the discussion forward to the next trainee. In instances when a member disagreed with any of the proposed numbers, he or she would usually either implicitly ask the CCC member who offered the number why ("really?") or offer a contradictory opinion ("I'd say [the trainee] is still probably at a x…"), at which point CCC members would offer specific examples, explanations, or evaluations that explain their points. Additionally, there was an occasional voiced reluctance to offer "perfect" scores, as the committee generally agreed that trainees may never achieve some aspirational competencies.
Conversations seemed to be key to the process of discussing trainee competence and building a shared mental model. Members used the analogy of a puzzle in which each faculty member had pieces of data about experience with trainees, and together through discussion and information sharing a more complete picture of trainee competence emerged during the meeting. It was generally felt that members should share concrete, specific examples that can be interpreted by the group in a relevant context. Individual participation is important for this to occur, and discussion amongst the faculty members led to additional insights and comments about each trainee. The discussion uncovered information not written on global or other evaluation forms. One faculty member noted that written evaluations are most useful for the highest and lowest performing trainees. He stated that "97% of the time my evaluation is the same for trainees in between those two extremes and in the middle of the competency spectrum." He also felt that Likert scales are not useful and that group discussion is a much more effective way to share information about group members' experiences with individual trainees. Some members reported feeling that it was easier to identify troubling trends with trainees based upon conversations with faculty members who have worked with these individuals. Oftentimes they said that it is hard to write negative comments on an evaluation form, especially if that form goes back to the trainee. As a result, these "neutral" evaluations may not identify trends in a way that conversation does. For example, if more concerning faculty observations arise with trainees and there has been no record of borderline performance or other concerns, it may be harder to justify things like program level remediation, etc.
To facilitate a shared mental model, some CCCs had additional resources available to the group. In most meetings, the group had access to a copy of the Milestones. In others the CCC chair had complete evaluation packets for each faculty member and asked the group to read the entire packet prior to discussing trainees. The latter led to notable periods of silence as committee members read the evaluation summaries, as well as specific Milestone criteria.

Standing in a Group
In all of the CCCs, participation in the meeting did not seem to be influenced by military rank, seniority, specialty, or whether one was a voting member of the group. In two CCCs chief residents attended as non-voting members of the committee, and they freely provided input from a near peer perspective about their experiences with most of the residents. These individuals had unique insight into most of the trainees that other faculty members may not be in a position to observe, and their input on the CCC provided information the committees considered very valuable. One CCC also solicited input from two program administrators, and their comments were valuable regarding trainee accomplishment of administrative tasks.

Task Complexity/Uncertainty
We identified the following themes under this heading: specific role of the CCC (e.g. two members in one meeting actively debated what regulative power the committee had when a trainee may need additional supervision); faculty member roles (e.g. should the members be more of a coach/mentor versus an evaluator or should trainees be assigned to faculty members based upon aligned interests?); developing remediation plans; lack of faculty member experience with certain trainees; and the Milestones themselves. In regards to the Milestones, members at times expressed frustration with the Milestone definitions, especially those with lengthier anchors and what the milestones don't capture (e.g., burnout). In several instances groups noted the difficulty in determining numerical milestone assignments given how the Milestones were written/phrased and commented on the difficulty in fitting individuals into the specifically defined categories. Some groups acknowledged the difficulty and/or worry in selecting only one demonstrative experience as a justification for a number assignment. For programs with a larger number of trainees (i.e. greater than 15 individuals), information and evaluation gaps complicated the ability of the CCC to make informed assignments of some individual trainees' Milestone levels. Additionally, a large number of trainees coupled with the number of Milestones typically led to longer meetings, which seemed to truncate the discussion and the competency evaluation process compared to CCCs with fewer trainees.

Threats
One of the biggest environmental threats identified was decision making fatigue. While members of one CCC admitted that they come to the meeting knowing what needs to be accomplished, almost all acknowledged that they Sapp J, Larsen K, Torre D, Holmboe E, Durning S MedEdPublish https://doi.org/10.15694/mep.2019.000066.1 Page | 10 get more fatigued as the meeting continues and tend to make more brusque decisions. Observations near the end of the meetings included members increasingly stepping out for short intermittent periods, multiple members checking their cell phones and fidgeting, group members openly asking how many trainees they had left to evaluate, and/or comments such as "I'm fading," "homestretch for milestones!" and "last one!," to CCC cheers. Other threats identified during the observations included scheduling conflicts, technical difficulties, and institutional structures/policies (e.g. on several occasions one CCC considered not just how the trainee did/did not perform, but how the institutional structures may/may not be at fault).

Discussion
Based upon a synthesis of the information uncovered in our previous scoping review, we proposed a model ( Figure  1) utilizing situated cognition to help better understand trust within a group and the various factors that may influence trust in group decisions from individuals either within or outside of the group. In our current study, we observed 15 hours of CCC meetings from multiple different GME training programs using a structured observation guide developed from a review of the literature and this proposed model. We explored how members characterize trust in the setting of a CCC and observed how these committees make decisions about trainee clinical competence and Milestone-based performance in the context of previously identified group trust elements: individual and group level factors, environmental factors, and the interactions between all of these elements. Our proposed model captured the themes that emerged at the observed CCC meetings.
Our results suggest that diversity of opinion promotes a more complete collective understanding of competence for each trainee. Varied faculty experiences provide a diversity of information available to the CCC, and interactions among faculty members help to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones. Further, CCC chairs were instrumental in facilitating a shared mental model by communicating an agenda, determining a process to assign the Milestones, moving along the conversation, engaging individual faculty members as needed, etc. These findings were consistent with our expectations based on the proposed model.
Additionally, CCC meetings generally had similar structure and agendas based upon the size of the training program. Faculty in smaller training programs tended to have more familiarity with all of the trainees and discussed them in more detail compared to larger programs. Larger programs also assigned faculty members to individual trainees, and it was their responsibility to complete standardized pre-work on their trainees prior to the meeting. Groupthink was more of a risk for larger training programs, especially when faculty members did not know their individual trainees well and when assigning individual Milestones.
Finally, environmental factors we identified that can compromise trust were task complexity, uncertainty, and threats to the process of group decision making. One of the most important threats was decision making fatigue. Almost universally, increasing distractions and observed lapses of concentration seemed to occur the longer the meetings ran.
Our research was limited to CCC meeting observations and brief group question sessions at the end of the meetings. While we were able to collect data about easily observable components of our proposed model such as group size, diversity, and communication patterns, we were unable to obtain data on individual faculty member thoughts and feelings. As a result, we were unable to test all components of the model, such as an individual's propensity to trust, vulnerability, relational identity orientation, need for affiliation, etc. We also did not collect data about what types of assessments each CCC used and had access to. Given that observers were present in the same room as CCCs, the results may have been influenced by the Hawthorne effect (i.e. CCC members knew they were being watched, which Sapp J, Larsen K, Torre D, Holmboe E, Durning S MedEdPublish https://doi.org/10.15694/mep.2019.000066.1 Page | 11 may have impacted their behavior and/or group discussions). Observations were also limited to meetings occurring during one Milestone reporting cycle at the end of the academic year at one institution.
Future research might gather more validity evidence and explore faculty member thoughts and feelings regarding trust in a group setting. We also suggest additional research looking at trust in group decisions from a non-CCC member stakeholder perspective. This might involve examining how department leaders, program directors, and individual faculty members who are not on a CCC define trust in this context.
Developing a deeper understanding of our trust in CCCs may help these committees implement a more effective and meaningful process to make collective decisions. Our model attempts to provide leaders within GME training programs a way to view and attend to the various factors that may impact trust. The model might also serve as a way to "diagnose" why a CCC goes awry by looking at the CCC discussions through various components of the model (e.g. if environmental or leadership factors may be perceived as "ineffective," what can be done to change or correct this?).

Conclusion
This observational study should allow program leaders and educators to better understand what factors may contribute to trust in group settings (e.g. CCCs). As programs develop policies, procedures, and determine the membership of these committees, utilizing our findings may help improve the design and execution of groups like CCCs, enhancing the translation of evaluation data and individual opinions into Milestone and competency decisions.

Take Home Messages
Little work has been done in health professions education to look at trust and group decisions. Group trust can be broken down into individual and group level factors, interpersonal interactions, and environmental factors. A diversity of opinion promotes a more complete understanding of trainee competence. Varied faculty experiences, interactions, and actions by committee chairs helped to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones. Groupthink and decision making fatigue are two potential threats to competency committee group trust.  [trainees]." (KL) "At other times, individuals explicitly acknowledge that they did not have experience with a specific [trainee] and did not feel they had the capacity to make an assessment (P18: 'I haven't been here so I don't think I have that much to add'.)" (KL) "On the whole, the CCC relied frequently upon specific individual stories/experiences with [trainees] as a means of evaluation, with members typically describing one specific personal interaction for one or more [trainees]." (KL) "It was clear that one CCC member (P13) was relied upon to volunteer/respond with commentary specific to the [trainees'] 'personal style,' offering input on the relationships between [trainees] and patients." (KL) Faculty knowledge requirements CCC faculty member comment: "Members need to have a basic level of competence as a physician within their specialty and a basic understanding of evaluating and discussing trainee performance." (JS) "Prior to the start of the meeting, the Chair (P32) explicitly indicated that it was his/her first time chairing the CCC meeting and that he/she was unsure how long it would go (or how best to execute the meeting)." (KL) "[The Chair] described prior meetings' procedural organization, noting that in the past the CCC possibly 'arbitrarily threw out numbers' and that P32 wanted this specific CCC meeting to be 'thoughtful.'" (KL) "There seemed some hesitancy or confusion about next steps (e.g., P44: 'how do we do this?' and P46 restating the goals of the CCC)." (KL) Help trainees "Even in the case of a trainee described as the 'lowest [first-year trainee] scores we've ever had,' the full group conversation turned to what would be offered in the future in order to aide his/her progress (e.g. P01 stated 'we could do X to make [the trainee] aware.')." (KL) "It seemed like the group wanted to generate summative feedback comments (recorded by the Chair) to be provided at a later counseling date with the trainees." (JS)

Conflict
Task conflict "Occasionally, after committee members suggested scores, other committee members would inquire how they saw the [trainee] as satisfying that score; inquiries were collegial, open-ended, and seemed to be interpreted as clarifications rather than challenges." (KL) "No one clearly seemed to disagree with other CCC faculty members, and the group discussion seemed to focus more on brainstorming ideas for how best to support the trainees." (JS) Dominant voice/ personality "P06 volunteered that it is difficult to achieve a confident consensus when there is a forceful personality in the group; other members agreed, and all commented on an example of a colleague with a forceful personality with whom they would not want to serve on such a committee." (KL) "At this point, P16 would consistently emerge as a dominant personality in both volume and quantity of commentary; unless other CCC members were specifically addressed, P16 volunteered-and at times interruptedconversation on average four times per each [trainee] discussion (a contrast to other members generally volunteering thoughts zero to two times per [trainee])." (KL) "At times, P15 would ask general questions about changing the Milestones (e.g., "has the [trainee] moved up to a 3.5?") and would get no responses at all, to which P15 would point out (e.g., say "Crickets") and members (most often P16) would volunteer thoughts." (KL) Groupthink "Other CCC faculty members either concurred with the assigned faculty member [Milestone] recommendations (most of the time) or they didn't (rarely). There was often minimal group discussion/disagreement on the Milestones as the group seemed reluctant to make changes to assigned faculty recommendations. Only on a handful of occasions did the discussion about an individual [trainee] spark a discussion about adjusting respective Milestones from the faculty member recommendations" (JS) "If faculty members had no evaluation data to change the existing Milestone level, they left the level from earlier in the year alone." (JS) "For certain Milestones, there seemed to be less discussion, especially for areas where the faculty members may not have any observational experience with the trainees." (JS)

Procedural Fairness
Voting "Though it was clear that P39 was in charge of the group, over time P46 kept formally requesting a vote and appeared to try to institute more formal order to the process." (KL) "The group followed with a procedural hand-raise vote, with all except P31 voting in favor of 'graduation with reservations.'" (KL) "At several points CCC members successfully altered the initial numerical assessment and/or P32 would casually ask for a vote from the group." (KL) Length of discussion "During the meeting there was an obvious difference in the amount of time that some [trainees] were discussed as opposed to others. Struggling [trainees] received significantly more committee discussion than highly performing [trainees] or those with average performance." (JS) During the CCC meetings observed, the average length of discussion for each trainee varied from 1-3 minutes all the way up to 24 minutes with discussion about one trainee lasting 56 minutes.
Meeting structure "As the meeting continued a clear structure emerged, divided according to (Phase 1) a general recounting and discussing of [trainees'] performance and (Phase 2) a specific Milestone assessment of each [trainee]." (KL) "P02 began by assessing each [trainee] according to specific evaluation data, including numerical rankings and formally reading evaluation comments (word-for-word, even when lengthy) while other participants attentively listened." (KL) "In order of years (second year [trainees], then first year [trainees]), the following pattern was generally employed: P15 would summarize the [trainee's] schedule, test scores, and evaluation scores (very detailed summaries, but only rarely involving word-for-word reading of specific comments), offer some personal opinions on the strengths/weaknesses of the [trainee], open the discussion to commentary from the CCC members, and then most often end discussion with a summary consensus question (e.g., 'do we agree?' 'other comments?')." (KL) "After the CCC faculty presented one of their individual interns, a group discussion followed. During this discussion, the end-of-year Milestones spreadsheet was populated." (JS) "The order in which each CCC member introduced their [trainees] to the room was determined according to two central criteria: personal time constraints of the member, and whether certain [trainees] had particularly pressing/significant issues to address (indeed P44 began the discussion in respect of personal time constraints). Such cases were introduced earlier in the meeting, whereas satisfactory/typical [trainees] were introduced later in the meeting." (KL) Ability to make decisions "P38 wondered 'if our only options are to graduate or not…' and shared that 'I think we're at least obligated to share this information' and 'I don't think we have enough information to not graduate [the trainee].'" (KL) "There seemed to be an explicit recognition that the program would be under increased scrutiny for failing to promote someone (e.g. P44 indicated it would look bad for the program, and P46 noted 'this is going to draw attention when the [trainee] has never failed a rotation')." (KL) Milestone assignment "P02 began each Milestone by explicitly stating the past scores of each of the [trainees], followed by silence while the committee members read the qualifications. Often, the first comment volunteered from any of the committee members would be a 'new' score, either the same or marginally higher than prior scores, at which point the group would consider whether they found that suggested score appropriate (e.g. for one Milestone P02 offered 'I think the [trainee] has reached a 3.' P03 specifically noted the importance that they all have the same 'outlook'-an implicit understanding that they don't want to progress a [trainee] just for the sake of progressing them and that each CCC member buys into that mentality." (KL) "At this stage there was little to no explicit discussion of numerical Milestone assignments; nearly all the discussion was general about [trainees'] behavior and test performance (with a seemingly implicit assumption that P32 would take those comments and independently assign Milestones as appropriate). In cases where [trainees] had previously achieved high scores (e.g., 4) on their prior Milestone assignments, committee members would both implicitly assume that those Milestones did not need to be considered (P32: 'let's talk globally… I don't want to go line by line…') and at times would explicitly state that those Milestones did not need to be considered ( Conduct of meeting agenda "The pace was immediately established by the Chair as quick; he/she explicitly stated the order in which [trainees] would be evaluated (third years first, 'anticipat[ing] it will go quickly'); the pertinent information (timeline details, when Milestones are due, etc), and a brief reminder of the core of the Milestones' meaning (deficiencies, ongoing progress, and aspirational targets." (KL) "As members showed up and there was a quorum, P32 began by discussing the guidelines and aims of the meeting, emphasizing that he/she would discuss the [trainees] according to their Milestones and overall progress. P32 distributed information packets on each of the Milestones at the start of the meeting and laid out the organization of the meeting (intending to start from 3 rd year [trainees] and progress to 1 st year [trainees]." (KL) Agility "While the first few [trainees] were initially being evaluated, the Chair was actively entering specific information into the digital Milestone spreadsheet; however, over time he/she appeared to recognize that it was not an effective use of time and that CCC members could simply contact him/her directly with the data to be entered." (KL) Facilitating a shared mental model "As the Chair would transition from his/her own introduction to the discussion, he/she would typically open the conversation by asking, 'anyone have specific concerns?' He/she summarized the process of 'how we do Milestones,' telling the group the plan to progress according to each individual Milestone and assess all [trainees] within that Milestone at that time (at which point he/she asked for, and received, the group's consensus agreement on this plan). At times, he/she also directly asked specific CCC members for their own evaluations/experiences with certain [trainees]." (KL) "For the first part of the meeting, the Chair let the group discussion play out without much intervention, and all CCC faculty members seemed to be involved equally in the discussion. He/she did not really need to engage individual faculty members because all freely participated when it seemed like they had something to contribute to the discussion." (JS)

Imported Information
Pre-work assignments "The discussion centered around the CCC Evaluation Form that was supposed to be filled out by assigned faculty members in advance of the meeting." (JS) "For each [trainee], faculty members gave some brief comments from the evaluations that they reviewed prior to the meeting along with a general synopsis of the electronically generated numerical values from the evaluation system that was prepopulated on the middle column of the CCC Evaluation Form." (JS) "There was no set format for how CCC faculty members presented his/her interns. Some utilized the program's CCC data collection sheet and some had information on other documents. However, all of the faculty members covered mostly the same information." (JS) Experiences with trainees "During the general discussion of each [trainee], CCC members would most frequently use specific experiential anecdotes to illustrate their opinions, both to offer positive and negative evidence regarding [trainee] performance." (KL) "Each time P12 raised concerns over the course of this specific conversation, other members (P07, P08, P10, P11) volunteered small, specific instances of praise of the same [trainee], and in some instances (P10) explicitly disagreed with P12's critique. For another [trainee], P08 raised a personality-related critique; other CCC members (P07, P10, P11) offered specific experiences and praise for the [trainee's] growth." (KL) "Members also noted that they sometimes could not think of multiple examples to invoke beyond singular memories, with P32 at one point noting 'I should have taken more notes during cases I staffed with the [trainee].'" (KL) "P16 added that it was the level of interaction with each of the [trainees] that mattered most in building an effective CCC." (KL)

Shared Mental Model
Discussion patterns "Most of the information shared at the meeting was via informal group discussion. Faculty members discussed areas such as medical knowledge, in-training exam scores, procedural competence, mentorship ability (for more senior [trainees]), ability to complete administrative tasks, teaching skills, scholarly activity, medical decision making, patient care, response to feedback, intrinsic motivation, patient presentation skills, and interpersonal communication skills and ability to interact with other [trainees], staff, patients, etc." (JS) "These discussion patterns largely considered the [trainee] according to global skills/traits (e.g., professionalism) the [trainee] currently exhibits that were then supported with singular experiences of interaction the CCC members had with the [trainees]." (KL) "As the meeting progressed, P33 began to assert a more dominant and organized tone in the meeting, at times ending the initial reading silence faster by offering opening comments (e.g. 'I can speak to clinic experience with the [trainee]…') and by explicitly wrapping up conversations (e.g. 'Who's next?' when discussions had devolved into unrelated chatter)." (KL) Development of a shared mental model CCC faculty member comments: "Members felt that group input from a personal perspective and discussion is a very important part of this process. They felt that written evaluations don't provide nearly the information about an individual [trainee's] performance than conversation with someone who has worked with them and has a personal opinion about their performance." (JS) "It was clear that P32 was the final arbiter of the number chosen-per the use of numerous "I" statements ('I'm fine with 3 or 4;' 'I say leave the [trainee] at 3.5,' etc.)." (KL) Resources available to the group "When formally beginning the evaluation of [trainees], P32 asked that all members of the CCC read the entire evaluation packets and comments. This consistently led to notable periods of silence as committee members read the evaluation summaries, as well as when committee members read each individual Milestone criteria silently to themselves during the Milestone assignments." (KL) Disparate opinions "Disparate opinions seemed to be constructive and productive although the degree of disagreement or dissention in the group was low. Different CCC faculty members brought up additional/supplementary information, and for the most part consensus occurred fairly easily within the group." (JS) "After generally agreeing with a hybrid solution-graduating the [trainee] with noted reservations-P31 asked if they could formally record P31's dissent from that decision (with P31 explaining that they recommended graduation without reservations, noting the number of cases the [trainee] had been a part of and emphasizing that 'if the [trainee] has made it this far without remediation, the CCC's role should be to graduate the "During the CCC discussion it became clear that group members were unsure of what the process was should they not want to recommend the [trainee] for independent practice; P36: 'the alternative is what?'" (KL) "There seemed some hesitancy or confusion about next steps (e.g., P44: 'how do we do this?' and P46 restating the goals of the CCC)." (KL) "At this point, P33 and P36 actively debated how the CCC assessment information would be shared and what regulative power the committee actually had in an instance in which the [trainee] may need additional supervision." (KL) "During the meeting, it was unclear from an observer standpoint exactly what decisions the group was making." (JS) CCC faculty member role "Various members discussed the role of the individual CCC faculty member. Should members be more of a coach/mentor versus an evaluator? Should trainees be aligned to faculty members based upon aligned interests, specialties, etc.?" (JS) Milestones "On several occasions, the group noted difficulty in determining numerical Milestone assignments given how the Milestones were written/phrased, and commented on the difficulty in fitting individuals into the specifically defined categories (e.g., at one point P16 noted the 'insanity' of anyone being able to achieve the highest ranking for one specific Milestone)." (KL) "At times, the CCC members expressed frustration with the Milestone definitions, specifically to what the Milestones are unable to capture (e.g., burnout)." (KL) "It seemed to me that there are so many Milestones and so many [trainees] that there were a number of gaps in information to make informed changes to the existing Milestone levels, and faculty members were comfortable leaving Milestones alone in these circumstances or going along with the assigned faculty member recommendations when there was no data to change the Milestone level." (JS) Developing remediation plans "Rarely, comments were made to the effect of not knowing what to do (e.g., P02 stated for one [trainee] that they 'don't know how to help.'" (KL) Outside Group Memberships/ Social Ties N/A Not observed Threats Decision-making fatigue "The meeting ran several minutes past its scheduled time, and there was some small indication that time pressure and decision-making fatigue were setting in among the group-for instance, the group was quicker to both deliver evaluations and come to consensus on those evaluations over time." (KL) "[Faculty members] nearly unanimously acknowledged that they get more fatigued as the meeting continued and often end up brusquely assessing [trainees] on the final Milestones." (KL) Schedule conflicts "At one point P08 took a phone call and left the room, and by 8:30 P07 had to leave to attend another meeting." (KL) "In one instance a CCC member (P36) needed to complete outside grading work-but rather than leave the CCC meeting, he/she simply asked the committee if it was ok for him/her to focus on the computer and submitting grades while the committee continued its discussion and that P36 would chime in when necessary (the group agreed; P36 did redirect focus away from the meeting but occasionally volunteered comments in line with [trainee] discussions)." (KL) Institutional structures/ policies "Several times the CCC considered not just how the [trainee] did/did not perform, but how the institutional structures were/were not at fault." (KL) Technical difficulties "At one point, P02 sought to locate additional electronic materials to show the group using the available screen, but due to technical difficulties he/she was unable to have the materials at their disposal." (KL) Staffing Levels N/A Not observed

Declarations
The author has declared the conflicts of interest below.