Identifying the perceived challenges that aﬀect the use of portfolio as an assessment tool for family medicine residents in Jeddah program

Background: Portfolio can be deﬁned as "Purposeful collection of student work that exhibits to the student (and others) the student’s eﬀorts, progress, or achievement in a given area(s)." It encourages the student for self-learning and reﬂection, and it has a signiﬁcant educational impact in supervision and training. The purpose of the study is to increase the awareness about the factors that enhance or inhibit the utilization of portfolio as an assessment and educational tool from participants perspective to the scientiﬁc committee of family medicine program. Objective : To explore the challenges and obstacles in using the portfolio as perceived by residents of family medicine program in Jeddah. Methodology: Interviews were conducted using purposeful sampling technique till saturation was reached. All National Guard family medicine residents in Jeddah from R2-R4 (R2=9, R3=7, R4=5) were included. The total number of participants were 21. Interview time, date and location were arranged with each one of them through direct contact after he/she conﬁrmed their participation. Data collection was done through face-to-face, in-depth, semi-structured interview (in English) for each resident separately. All recorded interviews were transcribed. Predetermined themes were used to analyze the data. Results: Eleven interviews were conducted from November to December 2016. Analysis revealed eight themes: learning and practice were good when they discussed the topic/case with their supervisors and got feedback.


Introduction / Background
The portfolio is one of the effective assessment methods in medical education. It is a great addition to residents' assessment because it provides learner and teacher with the needed information about the learning process and student progression. Also, it encourages the student for self-learning, and it has a significant educational impact in supervision and training. 1 Portfolio can be define as "a collection of material, made by a professional, that records, and reflects on, key events and processes in that professional's career". 2 "Purposeful collection of student work that exhibits to the student (and/or others) the student's efforts, progress, or achievement in a given area(s)". 3 So, it is a tool to assess the residents continuously and represent his/her achievements, and also to assess their defects, needs, and progress. Selfdirected learning and reflection are the main characteristics of this method of assessment. Accordingly, the use of portfolio is based on the adult learning theory. In 1980, Knowles made four assumptions about the characteristics of adult learners (andragogy) that are different from the assumptions about child learners (pedagogy). In 1984, Knowles added the fifth assumption. These five assumptions include: self-concept, adult learner experience, readiness to learn, orientation to learning and motivation to learn. 4 The attitudes toward portfolio were assessed in a study which was done at the University of Dundee Medical School over four years (1999, 2000, 2002 and 2003). 5 They found that the attitude and acceptability of students improved over the years of the study but concerns about the load of fulfilling all the requirement in short time. But on the other hand, the portfolio was helpful in gaining knowledge and skills and supporting their lifelong learning and reflective clinical practice. 5 Another study in South Africa done to explore registrars' and supervisors' experience regarding the portfolio's educational impact, acceptability, and perceived usefulness for assessment of competence showed that portfolio had a substantial educational influence on the training and assessment. 6 There are some limitations with the use of portfolios that have been reported including time-consuming, distracting from other forms of learning. In addition to concerns about uniformity, acceptance, utilization and educational benefits of the portfolio. 7 Until now the utilization of portfolio in Saudi medical colleges is limited and under expectations from both faculty and students. 8 The family medicine module's portfolio has four main components: The learning contract, evidence of learning, reflections as per Gibbs and logbook. So, the perception, acceptability, and experience of the portfolio by both residents and their trainers are important factors in the implementation and better utilization of this method of assessment. Accordingly, involving all stakeholders in selecting the topic and competencies that can be assessed using this tool is very important to achieve the maximum benefits. So, the aim of the study is to identify the challenges which affect the use of portfolio by family medicine residents.

Rationale
Increase the awareness about the factors that enhance or inhibit the utilization of portfolio as an assessment 1. and educational tool from participants perspective to the scientific committee of family medicine program.

Aim
To identify the challenges which affect the use of portfolio by family medicine residents.

Objectives
To explore the challenges in using the portfolio as perceived and experienced by residents of family medicine program in Jeddah.

Methodology
This qualitative study was conducted in the family medicine program. Interviews were conducted using purposeful sampling technique till saturation was reached. Data collection was done through face-to-face, in-depth, semistructured interview (in English) for each resident separately. All recorded interviews were transcribed. Predetermined themes were used to analyze the data.

Ethical Considerations:
Permission from the program director of Family Medicine department was taken to start the study after approval from KAIMRC. All participants signed an informed consent form before each interview. All the information was confidential, and no name was written or asked during the interview. Confidentiality ensured by keeping the records of interviews in safe digital password protected vault. All participants' identity was kept confidential and was identified by a coded serial numbers. The researcher, supervisor, and co-supervisors are the ones who can access the data. No incentives were given to participate in the study.

Study Area / Setting
This study was conducted in the family medicine program center in Jeddah which was established in 1994. The program is four years, consisting of many rotations to cover most of the specialties. Residents' assessment depends on both formative and summative methods. Since two years National Guard Family Medicine program was separated and recently became independent. The total number of National Guard residents is 39 supervised by ten family medicine consultants in addition to other trainers participating in different centers.

Sample Size
All National Guard family medicine residents in Jeddah from R2-R4 (R2=9, R3=7, R4=5) were included. The total number of participants was 21. R1 residents excluded because they have just started their residency and they don't have any experience with the portfolio.

Study Design
The research question was answered through a phenomenological study which explored the perspective and experience of study subjects. 9 Phenomenological study has been chosen to allow the exploration of residents' perceptions and experiences of the factors that enhance or inhibit the use of portfolio as an assessment tool. So, this design examined residents lived experiences through the descriptions provided by them.

Sampling Technique
Silverman recommends using purposive sampling in research based on interviews. 10 In this sampling technique, investigator depends on his judgment when selecting members of the population to join in the study for the interview.

Data Collection Method
An invitation to all residents (R2-R4) was sent (with all the details) through email to participate in the study. Also, the idea of the research was explained to all of them through their chief resident. Each resident required to confirm his/her willingness and availability for participation through email. Interview time, date and location were arranged with each one of them through direct contact after he/she confirmed their participation.
One day before the interview the location, time, date, day and duration (30 min ) were confirmed by direct contact to all residents. Before starting the interview the place was prepared (privacy, away from noise and distraction, comfortable seats, suitable room temperature), the resident was welcomed and provided water, coffee/tea to be comfortable and ready for the interview.The interview instructions were explained to all of them and written consent was taken.
At the beginning of recording the date, day, time, the title of the research, participant code were mentioned, and verbal consent was taken for each resident. At the end of the interview, each resident received a certificate of thanks and appreciation.
Data collection started using a list of ten open-ended questions and probes which were pre-determined: Six of them based on the adult learning theory 4 ( Q.3-Q.8), three of them were added to explore more ideas about factors which can affect the utilization of portfolio (Q. 1,2,9 ) and finally, the last question (Q.10) was about the recommendations from residents' perspectives.
A pilot study was conducted before the intended study on two newly graduate assistant consultant family medicine to test the research process and questions. Both assistant consultants were not included in the main study.

Data Management and Analysis Plan
All recorded interviews were transcribed immediately after the interviews were done.Adult learning theory themes were used to analyze the data 4 in addition to two more important themes. Manual data analysis was done. Transcripts were reviewed two times and coded based on the predetermined themes (Template Analysis). 11 All transcripts were compared to the recorded material and reviewed to have a general understanding and comprehension of each participant (first review). Then categorizing (coding /labeling) relevant information according to the predetermined themes (each question was representing a theme) was done (second review). Finally, results were written by describing the categories and finding.

Validity
The results of the study were discussed again with a focus group of the participant to compare the finding to their answers. All of them agreed with the results and said they represent their answers.

Reliability
Silverman contends that reliability in qualitative research involves selection of what is recorded, the technical quality of recordings and the adequacy of transcripts. 10 Every effort was made to accurately record and transcribe the interviews using the best equipment as well as backup recorders.

Results
Eleven interviews were conducted. Analysis revealed eight themes: six major and two minor. Major themes are those representing the adult learning principles, and the minor themes are additional important factors that affect portfolio utilization. Major Motivation -As a person matures, he or she need the motivation to learn -The residents receive their motivation from internal (mainly) and external factors

Practicality
-Residents want to learn what they can apply and implement in real life

Self-directed learning
-Is an instructional strategy where the students, with guidance from the teacher, decide what and how they will learn -It can be done individually or with group learning, but the overall concept is that students take ownership and responsibility for their learning

Life experience (self-reflection)
-Educators help students in drawing out relevant past knowledge and experiences and relate the sum of learners' experiences to the current learning experiences -Adults like to be given the opportunity to use their existing knowledge and experiences gained throughout their life and apply it to their new learning experiences

Goal and relevancy oriented
-An adult needs clear goals and objectives to learn. They should have learning goals to be ready to learn -Adult learners want to know the relevance of what they are learning to what they want to achieve

Respect and collaboration
-When learners are considered by their instructors as colleagues, they become more productive -Respect can be demonstrated to your student by: -Taking an interest in the residents' work -Guide, support, help the resident in his / her training -Appreciate and respect his /her effort and experiences -Follow the resident progress

Minor Orientation and understanding
It a very important step at the beginning of the residency program for both residents, supervisors, and mentors for deep and comprehensive understanding of portfolio

Challenges
Factors which impede the utilization of portfolio

Life experience (self-reflection)
-Some residents (6/11) believed that portfolio is not helpful because they are doing that daily without the portfolio -Others (5/11) said that it helped them to certain extent "self-reflection is part of adult learning process, and as a resident, in my level of maturity, I'm doing selfreflection on a daily basis. So portfolio not that much helps me at this point. "

Goal and relevancy oriented
-Most of the residents (8/11) said writing objectives and discussing them with the supervisor will help in the learning process.
-Few (3/11) felt it's a waste of time and not helpful and if they need to write some goals, they will write them in their way in another place other than portfolio "It enhances me to achieve my goals." "It's in my mind, why I should write it again."

Respect and collaboration
-All the residents said they were respected and appreciated from their supervisors.
-Some residents (6/11 ) said it enhances the collaborative work and effort between them and their supervisors.Other said (5/11) said they didn't feel the difference while using portfolio.

Orientation and understanding
-Many residents (7/11 ) said that the orientation about portfolio wasn't enough and they didn't understand it very well.
-Also, most residents (9/11) mentioned that some supervisors and mentors still not fully oriented about how to utilize portfolio in teaching and assessment.
"Unfortunately, I did not have the full orientation about the portfolio. Still, there are things that I didn't understand about it." "The orientation at the beginning was vague; they give us the guidelines, curriculum, nobody explained the portfolio clearly at first, there were no clear examples how should we utilize and complete the content of the portfolio "

Challenges
-Most residents (10 /11) agreed that there were some challenges which affected the implementation and utilization of portfolio: -Restricted time was the primary factor for both residents and supervisors.
-Understanding of the utility and implementation of the portfolio by some of supervisors /mentors was not clear.
-Limited content of the portfolio which is too specific in types and number.
-The portfolio became the main method of assessment: -This started from this year 2016-2017 -Led to more stress and bias in assessment "It affects our time of studying just to finish writing and filling the forms." "I think we miss some point for the role of the supervisor. It was like just monitoring." "We are searching for someone to see the case and evaluate it with us. Maybe if all the medical staff were familiar with portfolio, it would be much easier."

Discussion
The portfolio was a good motivational tool for some residents mainly when they discussed it with their supervisors who helped in improving their learning and practice. Also, it enhances their self-directed learning because it allowed them to write their goals and plans which guided them in their training and act as a reference for them. Others felt that they are mature enough to motivate themselves and to write their goals if needed. They believed that discussing with their supervisors without portfolio is more beneficial because they were not restricted to forms or marks.
Some residents were satisfied with their supervisors because of their help, support, and guidance. They felt the portfolio enhanced the relationship between them and their supervisors who were oriented about portfolio and enthusiastic to teach. This collaborative work was reflected positively on their learning and practice. There were some factors which affected the collaborative work between both of them. The main one was the restricted time which was one of the primary challenges for both residents (time to write and fulfill the requirements and time for discussing with their supervisors /mentors) and supervisors who were busy in their work and with other residents (4 or more). Another factor was the understanding of portfolio and how to utilize it efficiently in assessment and education which was not clear for some supervisors and mentors who still not fully oriented about their role. The orientation wasn't enough and concentrated on the portfolio as an assessment tool, not as a life-long educational tool. This result was similar to the one mentioned in the Saudi Medical Journal. They found that the main problem in utilizing portfolio was the implementation and understanding. 8 The content and methodology were explained to all residents, but inspite of that, it wasn't adequate for most of them who perceived it as extra paperwork which consumed their time. The residents started to understand and accept it more during their rotations year by year through practice and from the further explanation from their mentors. So, they became more familiar with it and knew how to utilize it in the minimum time without affecting their learning to fulfill all the requirements and get the maximum benefit from it. This finding was similar to the result of the study, which was done at the University of Dundee Medical School over four years (1999, 2000, 2002 and 2003). They found that the attitude and acceptability of students improved over the years of the study but concerns about the load of fulfilling all the requirement in short time. 5 Also , the results were similar with the same study regarding portfolio as a helping tool in gaining knowledge and skills to a certain extent.
Regarding self-reflection, some residents didn't feel that portfolio has that much impact on their reflection because they are an adult learner and they were doing that daily to improve their practice regardless the portfolio which is mainly for assessment. This underutilization of portfolio in self-reflection was because of their limited understanding about the main idea behind portfolio which is an educational tool which encourages self-directed learning and reflection.
Most residents agreed that portfolio was a subjective method of assessment because it was based on specific forms (which didn't include other achievements and work) which was assessed and signed by a supervisor who even didn't see the case with them sometimes. Also, using portfolio didn't reflect the real competencies of the residents and led to more bias (from the side of resident, supervisor, and mentor) and stress. Some residents did more tasks and chose the best one of them to be given to their supervisor for evaluation. In addition to the final evaluation at the end of each month which was filled by their mentors who didn't supervise them during their training and rely on the portfolio documents, some discussion with residents and evaluation from a supervisor in the department where the training was done.
So, most of them found portfolio a helpful tool for learning which guided them, organizing their learning and help them to follow their progress and improve themselves specially when they discuss and got feedback from their supervisors /mentors. On the other hand, most of them found it a weak tool for assessment because of its subjectivity and limitations.Another objective method of assessment should be added like MCQs. So, instead of thinking how to overcome those challenges which affect the utilization and implementation of the portfolio as an assessment tool, we should think about these questions: what is the efficacy of portfolio as an assessment tool? Which competencies can portfolio cover as an assessment tool? Can portfolio be the only assessment tool at the end of each rotation? Do we need to use the portfolio as an assessment tool and what it will add to the assessment?

Conclusion
Most of the residents agreed that the limited time for them and their supervisors was the main challenge. Orientation and understanding about the utilization and implementation of the portfolio were the second important challenge. The benefits and impact of the portfolio on learning and practice were good when they discuss topic/case with their supervisors and get feedback (still need improvement, more time for discussion and feedback, better understanding).
The portfolio was not convincing as an assessment tool because it's subjectivity and limitations. All the residents emphasized on the important role of their supervisors /mentors to get the maximum benefits from the portfolio. The main problem was in the implementation and understanding.