Feedback Process in The Mini Clinical Evaluation Exercise (Mini-CEX): an Exploratory Study

In the midst of clinical environment unpredictability, feedback helps students to make the most out of clinical learning opportunities. Mini clinical evaluation exercise (Mini-CEX) is considered to be appropriate for providing feedback to students. This study aims to explore the feedback process within the Mini-CEX at a large Australian medical school. Between year 2011-2012, 23 Mini-CEX sessions, involved 18 final year medical students, were observed, followed by 15-minutes structured interviews. 7 experienced clinical teachers were also interviewed. A questionnaire administered to 274 final year medical students to seek their views on feedback provided in the Mini-CEX. Most feedback has complied with the characteristics of constructive feedback, however the amount of feedback decreased as the quality level of feedback increased. These findings matched students’ perceptions recorded on the questionnaire. The themes derived from the interviews showed that for feedback to be useful, each of feedback process components (feedback provider, types of feedback, feedback recipient and action plans following feedback) need to be attended to. Both tutors and students need to focus on the process of incorporating feedback to inform students’ learning, not only on the feedback characteristics. One of the means to achieve this is by providing reflective feedback where students are considered as active recipients.


Introduction
Clinical medical education brings its own challenges to the provision of effective feedback. In a busy clinical environment, patient care comes first and this may limit the chances for students to obtain feedback from clinical teachers. Learning opportunities may also occur unexpectedly, in an unpredictable timing, and students may rely on habitual action without actually try to reflect on learning. Feedback from clinical teachers, especially facilitate students' reflection, may assist them in making the most out of the clinical learning opportunities. [1][2][3][4] Van de Ridder et al 5 reviewed the literature on feedback in the clinical education area and found that in most of the literature, feedback is viewed as information transmitted from teachers to students. Despite the argument that reflection requires feedback, 6,7 the analysis of both feedback cycle and its characteristics has not provided an adequate account on a particular type of feedback that relates to reflection.
Holmboe et al 7 categorize feedback into four levels which are giving recommendations, enabling learner reaction, asking for self-assessment and agreeing on action plans. The category describes the level of feedback recipient's involvement in delivery process. The lowest level does not acknowledge the roles of the recipient in receiving and responding feedback, while the second level starts to take into account the students' reaction. The third and fourth level involve self-assessment and formulation of an action plan respectively.
The above categories correspond to Hattie and Timperley's four levels of feedback 8 i.e. self level, task level, process level, and self-regulation level, of which each level differs in its effectiveness in affecting students' learning. The highest level, the self-regulation level feedback (similar to fourth level feedback in Holmboe's category), functions to modify the self-regulation of learning mechanism. It then stimulates the creation of internal feedback and also encourages students to seek and incorporate external feedback to improve the learning. Therefore, it is likely that this type of feedback will enable students to become more effective learners, since effective learning requires reflection and feedback. 6,9 The Mini-CEX is considered to be a highly appropriate and feasible setting to examine immediate feedback provided to students. Jackson et al 10 in their study on Mini-CEX in a foundation program have found that feedback is perceived to be the most important aspect, however the trainees have not received it in an adequate amount. Another study found that the feedback delivered was not specific 11 and students were not optimally involved in formulating plans for improvement. 12 An exploration on how medical students perceive feedback and its influencing factors and also to what extent students make use of it is then deemed essential in order to understand more about the value of Mini-CEX as a learning tool. Therefore, this study aims to explore the feedback provision process during Mini-CEX from different point of views. Proper understanding of feedback process enables the development of optimal model of feedback delivery in medical education.

Methods
Multiple perspectives in this study were obtained from three different approaches, i.e. non-participant observation of Mini-CEX sessions, structured interviews with students and administration of questionnaire on feedback. The information was then triangulated through structured interviews with a group of clinical teachers.

Non-participant Observation
Twenty-three Mini-CEX observations were conducted in three metropolitan clinical schools of an Australian university, involving 18 final year Bachelor of Medicine, Bachelor of Surgery (MBBS) students (nine females and nine males) and seven clinical tutors/assessors. These students were in their second rotation of clinical clerkship and had become familiar with the Mini-CEX process. Training sessions for assessors was provided at each clinical school but there is likely to have been some variations in the approach to these sessions taken by individual leader.
In each Mini-CEX session, the researcher, who acted as a non-participant observer, observed the interaction between the student and the clinical tutor. The observation focused on feedback provision using a field note template developed by the observer, based on the theoretical framework of feedback including its characteristics, category and the delivery process. The data reached saturation after 20 observations.
The data from the field notes for each student was transcribed by the researcher and analysed qualitatively to identify the characteristics and types of feedback in each Mini-CEX session. A coding manual was developed based on relevant literature to assist with the analysis. It consisted of the main features of feedback, which were characteristics (timely, specific, descriptive, clarified and "sandwich model" [(positive-negative-positive) feedback] 13 and hierarchical types of feedback according to Holmboe et al. 7

Structured Interviews
A 15-minutes structured interview was conducted with each student participant, who was assessed in the 23 observed Mini-CEX sessions, to explore the immediate response of the student to the Mini-CEX encounters and feedback. Seven experienced clinical teachers, of which five were male from two metropolitan clinical schools were also interviewed. They had the roles of teaching medical students during the clinical attachments and assessing them in the Mini-CEX sessions. None of them had specific formal training in education. Each interview lasted for about 15-30 minutes.
A set of interview questions was prepared based on the theoretical framework of the feedback cycle, such as credibility of teachers, 13 students' emotional state, 14 availability of guidance in responding to feedback. 15 The interviews were conducted by the researcher and were audio-recorded. Following each interview, a verbatim transcript was produced. A thematic analysis approach was used as a method of analysis in order to reduce and make sense of the data. 16 The transcripts were reviewed for the first time to identify specific patterns of responses. After all patterns identified from the analysis were coded, they were translated into relevant themes. The themes were categorized and grouped to arrive at an interpretable conclusion. 16,17

Questionnaire Administration
All final year MBBS students attached to three metropolitan clinical schools (n=274) were given a questionnaire, during a large class session, to complete within ten minutes. The questionnaire was developed based on theoretical analysis and previous studies, 18,19 seeking students' views on the type and benefits of feedback provided in the Mini-CEX. It also explores students' general perceptions on feedback. Data was analysed using descriptive statistical methods.
Ethics approval from the University Human Research Ethics Committee was obtained prior to the commencement of this study. Both students and clinical teachers provided their consents before participating in the study.

Non-participant Observation of Mini-CEX Sessions
Most observations were conducted during general medicine rotations in the in-patient setting. All assessors were consultant physicians. Most of the cases were noted as of medium complexity on the Mini-CEX marking sheet (e.g. epigastric pain, metastatic rectal cancer and type 2 diabetes mellitus).
Verbal feedback given within all 23 Mini-CEX observations was analysed and coded against the characteristics of constructive feedback and types of feedback. The summary of results is provided in Table 1. There was no feedback that had all of the five feedback characteristics. The amount of feedback decreased as the level of feedback increased, with "agreeing on action plan" feedback observed only once. In eleven Mini-CEX observations, the lowest level of feedback recommendations was the only type of feedback given.

Structured Interviews of Medical Students and Clinical Teachers
The thematic analysis resulted in a list of themes and sub-themes. The data was saturated after 23 structured interviews with students and seven interviews with clinical teachers. The themes derived included characteristics of feedback provider, types of feedback, roles of feedback recipient and further plans following the feedback provision. These themes well matched with the essential factors involved in feedback delivery cycle, which were feedback provider, type of feedback, feedback recipient and responses to feedback.

Characteristics of Feedback Provider
There were several characteristics of the clinical tutors, according to the students' perspectives, which influenced the process of delivering feedback. Tutors' friendly attitude was the attribute that most often occurred in the comments. Students considered that tutors' approaches in providing the feedback affected the way students think about the feedback and how they respond to it.
I think it was good because RD [the tutor's name] was very personable…as…character…so he was very good with his interaction with us… (S3) There was still variability between tutors in terms of their stringency and leniency.
It is still variable between examiner as well, one examinee does the same examination, you get hundred percent and excellent feedback, and do the same examination with another consultant, and failed it (S13) From the perspectives of clinical teachers, most responses identified in the analysis dealt with the need of support for teachers in providing feedback. The support came in two forms, training using videotapes or role-playing.
… we actually did role plays of giving feedback. Two students, two real students, and then the students fed back to us, how they felt receiving the feedback (T6) The second was the provision of guidelines on how to assess the students and provide feedback, expected standards of performance and examples of feedback provided following the performance.

Characteristic of feedback
… and I guess some examples of strengths and weaknesses in that blank space [of Mini CEX form] might just help the tutor to think something a bit more specific… (T4) Time was one of several significant factors that may limit the process of delivering feedback and the amount of feedback given. Clinical teachers had other responsibilities, such as caring for the patients, therefore sometimes they found it difficult to have ample time to give and discuss feedback with students.
…, I think time is a big factor. So, I think most of the assessors are busy hospital doctors, yeah, and they got huge demands, not only the students… (T3)

Type of Feedback
Most comments from the students that fell into the type of feedback category were related to the usefulness of specific and detailed feedback. Most students felt that they needed to know exactly what they did well and what they did badly.
…I think it's very, very helpful when someone observes us while we actually do an interview or examination, so people can tell us, pinpoint exactly what was good and what was not so good in that… (S21) Few students experienced feedback that made them reflect on their current learning and performance. Students found that kind of feedback to be constructive by making them think about learning and aware of their deficiencies.
I think he really forced me to, like, ask some deeper questions about the case that I presented which I haven't considered (S8) According to the perceptions of the clinical teachers, good feedback is specific, immediate, provided in a one-on-one and private setting, using a pattern of positive feedback followed by areas of improvement and encourage reflection. Specific feedback was the characteristic with the highest number of responses, and the following quote is an example of response.
…from my experience, the least helpful for students is general feedback, I think, from my experience, most of them want specific feedback… they want to say, you forgot to do this, or you forgot to do that. I don't really feel that they appreciate if you say, look, overall you did really well. Generally, they just want to know what did I do wrong. (T2) There were also responses regarding the need of feedback that encouraged reflection, as exemplified below.
…I think it needs to be reflective, they need to be able to try and get some insights for themselves as to what the issues are and we need to facilitate that… they need to hopefully have the skills to work out strategies and take it to the next level.... (T5) The analysis captured a tendency towards more positive feedback. One respondent specifically argued that this particular tendency may not be caused by teachers' unwillingness to give negative feedback, but it was more due to their inability to provide it with a good approach.
I think the question is not so much that they are unwilling to give negative feedback, is that they may not understand the best way to do that… (T7)

Roles of Feedback Recipient
The analysis identified two aspects from the students' side, which were likely to affect the feedback delivery process. Students' prior experiences or expectation on feedback was a factor identified by the students.
I can see where he is coming from, that he wants to push us and go to more evidence based medicine but at the same time I just didn't feel like I firstly had the time to go read up on all of these journal articles to be really keep up to date when I haven't got my basic really… (S1) Students' attitude towards the feedback provider was the other factor identified by the students.
…I think, if there are advise that you agree about, like if you respect that persons advise on your presentation style, I think it can help you build your presentation style… (S23) The teachers perceived that students' approach to studying was still driven by assessment. If they were not assessed for summative purposes, then they would not have adequate motivation and moreover, they would not respond the feedback. Most students wanted to just pass the assessment, without paying enough attention to the learning process.
She came and did it [the Mini-CEX] and it wasn't much better, and my feeling was if she was preparing for an exam at the end of the year, that she would've done a lot more work, whereas all she did was just trying to get this hurdle [the Mini-CEX] out of the way, and found different assessors who would be more lenient on her. (T3) Responding to feedback: plans following the feedback provision The structured interviews managed to capture some responses from the students regarding their action plans. Some students were still confused about what they have to do next and only formulated a general learning plan.
…brush up the theory as usual because there so much more to learn, and yup just go down the path and we'll see (S20) The analysis also demonstrated that students still needed some guidance and confirmation as to how they can use the feedback. The guidance from tutors was necessary for students to thoroughly understand what the feedback meant and what they should do next.
I think he probably needs to just be a bit…you know…guide us a bit more, like what he meant by cognitive knowledge and how we can sort of go about with it… (S1) According to the clinical teachers, students need training on how to understand the meaning of feedback and how they can make use of it.
…I think they need to, not just respond, but they need to sort of understand why we're doing it at all. Like the discussion about negative and eJKI Vol. 7, No. 3, Desember 2019 positive, they need to understand that it's all about being constructive… (T5) Another way to help students interpret and go through the feedback was by providing a mentor or advisor to assist students. Students received feedback from various people in various settings in the context of different cases, and they may encounter some difficulties while trying to digest and follow up the feedback.
Would be helpful if they have a mentor or someone to go through the feedback from various people at the end of the semester (T1)

Questionnaire on Feedback
Two hundred and forty-six (response rate of 89.8%) of 12 th semester students attached to 3 metropolitan clinical schools completed the questionnaire. Most students were local students (n=166) who resided in Australia, female (n=126), and school-leaver (n=173). Figure 1 depicts the frequency of verbal feedback being reported by students during the Mini-CEX. There was one student who did not receive verbal feedback and five students said they received no feedback at all. …I think they need to, not just respond, but they need to sort of understand why we're doing it at all. Like the discussion about negative and positive, they need to understand that it's all about being constructive… (T5) Another way to help students interpret and go through the feedback was by providing a mentor or advisor to assist students. Students received feedback from various people in various settings in the context of different cases, and they may encounter some difficulties while trying to digest and follow up the feedback.

Would be helpful if they have a mentor or someone to go through the feedback from various people at the end of the semester (T1)
Questionnaire on feedback Two hundred and forty-six (response rate of 89.8%) of 12 th semester students attached to 3metropolitan clinical schools completed the questionnaire. Most students were local students (n=166) who resided in Australia, female (n=126), and school-leaver (n=173). Figure 1depicts the frequency of verbal feedback being reported by students during the Mini-CEX. There was one student who did not receive verbal feedback and five students said they received no feedback at all.

Figure 1. The Frequency of Verbal Feedback Received as Perceived by The Students
The participants were asked to identify the type of feedback, based on Holmboe et al's 7 hierarchical groups of feedback ( Figure 2). They were allowed to select more than one form of feedback. The findings demonstrated that as the level of feedback increased, the count decreased. Almost half of students (45.4%) perceived that most feedback in the Mini-CEX encourage them to improve their performance andthey also felt that feedback enable them to reflect on their learning. However, there were around 6% of students who did not feel those particular

Feedback in the form of suggestions
Feedback that gives student an opportunity to respond Feedback that asks student to self assess Feedback in which student and tutor agree on what to do next (action plan) Other The participants were asked to identify the type of feedback, based on Holmboe et al's 7 hierarchical groups of feedback ( Figure 2). They were allowed to select more than one form of feedback. The findings demonstrated that as the level of feedback increased, the count decreased.
Almost half of students (45.4%) perceived that most feedback in the Mini-CEX encourage them to improve their performance and they also felt that feedback enable them to reflect on their learning. However, there were around 6% of students who did not feel those particular benefits. Table 2 provided students' level of agreement on a number of feedback attributes. Most responses for the negative statements were either neutral or agree.

Discussion
Most feedback provided in the Mini-CEX has conformed to some of the characteristics of effective feedback proposed by authors such as, Van de Ridder et al 5 and Hesketh et al 13 although no feedback had all the characteristics. Some feedback comments were still general and this type of feedback may be difficult to interpret by students, thus, they may fail to appreciate the value of feedback and utilize it to advance their learning. Furthermore, there are limited instances that can be considered as descriptive feedback. Since descriptive feedback allows students to try to think through their performance and identify room for improvement, it increases the likelihood of the feedback being properly received and processed by students.
The findings indicate that feedback is still a one-way process. Medical teachers are able to provide specific feedback to students, but there is still a tendency to avoid involving students in the feedback delivery process, either by clarifying it with the students or giving descriptive feedback. Bing-You et al 20   argument supports the present finding that medical teachers are still lacking the ability to involve students in the feedback delivery process. They have not yet considered feedback delivery as a twoway communication process, in which the recipient has an essential role in receiving, processing and incorporating (or even rejecting) the feedback that will influence their learning development. The meta-review of variables that affect the process and outcome of feedback showed that factors on all four phases, from task performance and feedback reception, to observation and feedback provision, influence effective feedback. 21 Most of the feedback received by students in this study can be categorized as level one (lowest level) feedback, which is giving recommendations, according to Holmboe et al's 7 grouping. The present study demonstrates that as the level of feedback increases, the amount of feedback decreases. The results of the present study replicate the findings from Fernando et al 22 and Holmboe et al 7 Both studies demonstrate a lack of feedback that stimulates students' reflection on learning. Jackson and Wall 10 also acknowledge the difficulty for medical teachers in providing feedback that leads to formulation of a learning plan. Therefore, apart from being specific, eJKI Vol. 7, No. 3, Desember 2019 the process of delivering feedback needs to allow tutors and students to derive an action plan that can be applied in future similar situations. Feedback that is only in the form of suggestions is unlikely to be transferable for future learning, as supported by Hattie et al. 8 Students acknowledge that the tutors' level of experience, ability and attitudes in delivering feedback has an effect on whether they would receive or discard the feedback. The participants have also acknowledged some limitations on the teachers' side in providing feedback. The most prominent barriers were seen as limited time and lack of knowledge regarding students' level of expected performance. The importance of a teacher support system in providing feedback, either through teachers training or provision of guidelines, has been acknowledged by most participants. The emotional aspect of feedback process should also be addressed in the faculty development program. 23 On the other side is the feedback recipient. Some factors identified from the interview are students' prior experiences and knowledge, their expectations on feedback, and also their attitude, which, in accordance to the literature findings. However, there are several characteristics of the feedback recipient, which have not been identified by the interview participants, such as goal setting, self-efficacy and self-confidence. Teunissen et al 24 identified self-efficacy as one of the predictors of feedback -seeking behavior, while Young 25 explored the role of self-confidence in responding to feedback. This finding indicates that students may not fully understand the process of appropriately responding to feedback and the factors affecting it. It highlights the needs to equip students with the ability to process the feedback they have received in order to influence the learning positively.
The students provided mixed responses regarding the action plan following the feedback. Some students learn specific skills or knowledge they need to based upon the feedback they have received. Some are still confused since they do not know what to do next. They require more guidance on how to use the feedback to improve their learning, similar to the findings from Weaver. 19 After feedback has been provided, there are still some complex processes involved in processing and understanding the feedback. 26 Some teachers agreed that, overall, students need to be facilitated or trained in understanding the "what", "how" and "why" of feedback. Medical teachers are starting to provide a more student-centered feedback by acknowledging that students need to be assisted in critically responding to feedback. 27 When the feedback provided by the tutors encourages reflection, then it is likely that at the same time, the tutors have helped students going through the process of responding to feedback, because reflective feedback requires a two-way communication between the provider and recipient. Students cannot be passive recipients in the reflective feedback delivery process. 28 There has to be a learning environment that facilitates feedback provision and support feedback-seeking behaviour, such as feedback culture and also longitudinal, trusting relationships between students and tutors. 29,30 Students still prefer positive feedback to negative and they consider negative feedback as harmful for their self-confidence. Feedback that only contains praises upon students' performance is not included as positive feedback. 31 The statements of the questionnaire used in this study do not explore students' perceptions on the meaning of positive and negative feedback. This is one of the study's limitation that prevent us in understanding whether students are confusing the meaning of negative feedback with nasty or humiliating feedback. However, findings from the interviews might indicate that students expect feedback to also be targeted to their deficiencies but provided in a good manner.
The analysis identified a tendency of teachers to avoid negative feedback. While students need to receive positive feedback, they also need to look for negative feedback since it will help them to improve their performance. Reflective feedback is perhaps appropriate to counteract the difficulty of giving negative feedback, since students are encouraged to be actively involved in the process of identifying their own strengths and weaknesses. This process will likely lessen the discomfort that might occur both of the teacher and student. It is likely for students to consider negative feedback as positive feedback when they perceive it to be useful. 32 During the process of delivering feedback, tutors may also help nurturing students' self-confidence, so that they are more receptive upon negative feedback. Furthermore, since students' goal setting affects students' responses against negative or positive feedback, 33 tutors need to acknowledge the differences in students' goal setting or orientation and try to modify it if necessary.

Conclusions
The present study has enabled a deeper exploration into feedback especially that is being delivered in the Mini-CEX. In order to achieve the goal of delivering feedback in a two-way communication approach, tutors and students need to focus not only on the characteristics but also on the process of incorporating feedback to inform students' learning and its influencing factors. Individual characteristics and emotional responses among those aspects need to be taken into account when feedback is being delivered. Students should be considered as active recipients who are encouraged and facilitated to develop their own thinking ability, to learn self assess, to determine their positive and negative aspects, and to formulate an action plan relevant to the expectation of both students and tutors. Therefore, providing feedback is at the same time nurturing one's reflective learning ability.