Electronic portfolios for assessment in your postgraduate medical education program: essential questions to ask when selecting a platform for competency-based medical education (CBME)

Portfolios have re-emerged as a means of addressing the complex assessment challenges in postgraduate programs as we transition to competency-based medical education (CBME). However, implementing electronic portfolios (eportfolios) can be a daunting task. In this paper, we aim to provide guidance to program directors and administrators who are tasked with the implementation of an e-portfolio by reporting on the successful development of a low-cost e-portfolio system for our Developmental Pediatrics residency program. Twelve helpful tips have been developed: 1. Can the e-portfolio be customized to meet the needs of my program? What resources are required to customize the system? (tailor to fit); 2. Is it easy to learn, efficient to use, and easy to remember? (easy-peasey-lemon-squeezy); 3. Can it be customised by the learner? (make it their own, home sweet home); 4. Is the e-portfolio portable? Can the learner access the portfolio after completing training? (make it to go); 5. What are the privacy settings and controls? Are the privacy/sharing settings transparent and easy for everyone to understand? (private parts); 6. What level of mobile support does the system provide? (anywhere, anyplace, anytime); 7. How can mentor feedback be incorporated? (feedback friendly); 8. What are the financial (and hidden) costs? (count the costs); 9. How long do you need the platform to be sustainable? Can the platform give you the sustainability that you need? (check the expiry date); 10. Access to support (know who's got your back); 11. What kind of support is available? How fast can you get the support you need? (share and share alike); and 12. Can it be scaled for use with each new cohort of learners? (keep up with the times). In addition to the twelve tips, this paper includes insightful commentary from a program director and a recently graduated resident who used the portfolio throughout her training. As we continue to move toward competency-based learning, e-portfolios will provide a platform and stimulus for sustained excellence in medical education.


Introduction
A portfolio is a collection of different kinds of assessment models, from structured tools such as logbooks and encounter cards, to freeform artifacts such as a learning diaries and essays. Together, the varied components provide a "flexible, multifaceted means of collecting evidence of the achievement of competence over time" (Frank, Mann & Keely, 2006). Some of the key advantages of portfolios are their flexibility, learner-centeredness, facilitation of dialogue between learners and teachers, learner involvement, assessment of growth over time, and the platform to plan learning goals (Frank et al., 2006). These are essential elements needed to support assessment in CBME.
For medical educators who are new to e-portfolios, the literature provides useful starting points. Lewis and Baker (2007) outline a comprehensive how-to guide for e-portfolio creation for medical educators for use in their personal careers, and Clay, Petrusa, Harker and Andolsek (2007) suggest core content that can be shared between different subspecialty residency programs and readily adapted to an e-portfolio. Finally, Contact North (2012) presents ten guiding principles to reflect on when considering the application of technology in learning, which are useful in considering the implementation of e-portfolios from an institutional perspective. However, none of the above publications provide the perspective of the program director and learner in the process of e-portfolio development.

Background
The Developmental Pediatrics Residency Program at the University of Alberta is a two year subspecialty training program. While the portfolio was developed, six residents/clinical fellows were enrolled in training. Five learners participated in the use of the portfolio and contributed to its development through feedback and suggestions. The program maintains affiliations with several off site rotations through which learners rotate, though the majority of their time is at the local rehabilitation hospital site.
In 2012, we implemented a paper-based evaluation portfolio based on the seven CanMEDS competencies: 1) Medical Expert, 2) Communicator, 3) Collaborator, 4) Leader, 5) Health Advocate, 6) Scholar, and 7) Professional. CanMEDS is widely adopted educational framework that describes the capabilities physicians require by the end of training and serves as the basis for the educational and practice standards of the Royal College of Physicians and Surgeons of Canada (RCPSC) (http://canmeds.royalcollege.ca/en/framework). In developing our portfolio, objectives were selected from the Objectives of Training for our Specialty of Developmental Pediatrics mandated by the RCPSC, with a focus on those objectives that lent themselves for inclusion in a portfolio. These included objectives such as frameworks for evaluating screening tests and reflections on personal limitations, as these topics do not come up often during clinical experiences. We then selected specific assessment tools that were a good match with the objectives covered by the portfolio. These included: 1) patient-encounter and clinical procedural logs, 2) self-reflections, 3) personal learning plans, 4) research documentation, and 5) formative assessment. For several months, we tried out different tools. We wanted something that would allow us to create a portfolio that would be functional, easy to use, and appealing to our learners. We wanted to be able to make changes to the portfolio over time, and allow our residents to be creative and engaged. After using the paper-based system for a year, we decided to migrate to a web-based platform in order to make the portfolio more convenient for our learners to use and share with mentors. Due to budgetary constraints, we were faced with designing and implementing the eportfolio ourselves. After several months of trying out different software, we selected Google Sites, a structured web page creation tool available as part of an institutional suite of apps provided at our university. In addition to being free to use, the software provided us with a reasonable amount of creative control. This made it possible to develop the e-portfolio ourselves in a rapid and iterative fashion.
Of note, at our University, we use an academic institution version of the Google Apps suite, with appropriate agreements around privacy, security and data storage. This is an important distinction because the data security/privacy is different from the "free version". However, the recommendations we make regarding the required elements in an e-portfolio are independent from Google Sites and we are in no way endorsing Google Sites.
Program Director: As a program director of a smaller residency program, I needed a way to connect with my learners in a personable way that was both efficient and meaningful. I did not have extra time on my hands, nor did I have a significant budget to spend on the project. Being in my first ten years of practice, I knew what my graduates would need when entering practice of their own and I wanted them to be ready to produce a curriculum vitae and teaching sessions with ease and not struggle with locating documentation.
Learner: I entered subspecialty residency training after nearly five years of independent practice. By that point, I had encountered many of the new in practice challenges experienced by graduating physicians: hitting end of the year reporting for my division or for CME credits with insufficient records of all that I had done in the 12 months prior; wanting to give to a family "that handout I saw once in a specialty clinic" but not having it; hoping to quickly re-read a landmark paper at three in the morning to answer a clinical question, then not being able to find it. So I took to the idea of a dynamic, portable, and relevant portfolio like a med student to an algorithm.

Recommendations
Based on our collective experience and review of the existing literature, we make the following 12 recommendations around selecting an e-portfolio with a discussion of each point from the perspective of the program director and learner. The first seven points address acceptability, usability and functionality for both the program and the learner. The remaining five points outline the practical considerations at an institutional level.

Point 1: Tailor to fit
Utility, the ability of the software to do what is needed (Grudin 1992;Nielsen, 1994), is the first and most important consideration when selecting an e-portfolio platform. While there may be common needs across training programs, as suggested by Clay et al. (2007), there are also important differences between specialties, programs, and even sites. Thus, in order to deliver a high degree of utility, any e-portfolio platform is likely to require some customization to meet the needs of your program and learners. Customization usually comes with a cost, and the cost will depend on the specific software you select. Some systems require the purchase of additional modules or custom programming. In other cases, the software may allow you or your learners to make the necessary changes without additional assistance. Program Director: With regard to customization, it is important to determine who will implement and maintain the system. Our program is small, so custom development is beyond our reach. It was important to us that the software could be tailored by Program Directors and learners with as little external assistance as possible.
Learner: It was important to me that our portfolio was specifically adapted to our program. I appreciated that every task in our e-portfolio had a clear application to both my training and professional growth. Under each CanMEDS heading, we were guided regarding the specific measurables we would need to effectively finish our training and transition into practice. It was evident how the work I put into my portfolio related to the objectives of my program, or in some way contributed to building a strong base in my field. I think that anything else would have felt like "busy work."

Point 2: Easy-peasy-lemon-squeezy
The platform you select is more likely to be accepted and used by your learners if it is "easy to use." Since easy is a highly subjective concept, it can be helpful to adopt the more formal definition of usability, traditionally consisting of five attributes: 1) learnability, 2) efficiency, 3) memorability, 4) errors, 5) satisfaction (Nielsen 1994).
According to this definition, e-portfolio software is easy to learn and efficient to use once learnt. It is easy to remember, even after learners have not used it for a while. A usable e-portfolio platform is easy for learners to use without making many mistakes. When an error does happen, it is easy to recover from. A usable e-portfolio protects against serious or catastrophic errors. Finally, a usable e-portfolio is pleasant to use.
Program Director: I expected to meet resistance from our learners if the software was vexing to use. Although we did not undertake a formal usability study, our decisions were guided by many of the considerations outlined above. We tried to make the system easier to use by first reducing how much there was to learn. This meant selecting software that was potentially already familiar, and ideally simpler -even if it meant fewer features. We identified the most common tasks, such as completing a form, writing a narrative entry, or uploading a file, and worked on making these as frictionless as possible. We designed the navigation to provide learners with an overview of the entire portfolio, while making it easy to get to any section in one or two clicks. Finally, we put effort into making the portfolio aesthetically appealing, so that our learners would enjoy the time they spent using the software.

Learner:
The system chosen by my program was simply designed and intuitive to negotiate, which made it a reasonable tool to access and use every day. I tolerate difficult to navigate medical systems for the betterment of patient care, but I would not have had as much patience for a clumsy portfolio system. Simplicity and ease are critical from a learner's perspective.

Point 3: Make it their own, home sweet home
CBME is learner-centric with self-directed learning at the core. Learners need to be able to customize the eportfolio to their individual needs. Just as physicians approach a clinical problem from different perspectives, learners may have different needs and approaches to an e-portfolio.
Program Director: Some of our learners devised additional categories to describe participation levels during their clinical encounters, while others were content with the standard offerings. In additional to the purely functional aspects of the portfolio, personalization of the e-portfolio -to take ownership of it -is just as important. Sonnenberg L, von Hauff P, Lemieux L MedEdPublish https://doi.org/10.15694/mep.2017.000066 Page | 5 Learner: For me the most important thing was to be able to approach tasks in an order that made sense to me. The portfolio would not have been effective if it were a linear process. An ideal portfolio is an open, iterative platform, that allows learners to achieve tasks at a personal rate instead of the sometimes prescribed direction of training. This is in keeping with the move towards competency based education. As I "completed" areas of my e-portfolio, I could put a lot more time to the areas of greater challenge for me.

Point 4: Make it to go
A portfolio is representative of a learner's work and a summary of her or his experiences. Does the learner wish to take it with them when they leave the University walls? If the answer is yes, and in most cases it is, then portability is a key factor to consider. Since portfolios are starting to find a place in continuing medical education (CME) (Dornan, Carroll, & Parboosingh, 2002), the ability to have an established e-portfolio that learners can carry with them into practice is becoming a must.
Program Director: I remember some of the earlier e-portfolio versions we trialled. I was almost convinced to use one platform when I found out it had limited portability at the end of training. It was a deal-breaker for me and something that I had failed to ask about the portfolio right from the start. For the next platform, it was the first question I asked! Learner: I would question the purpose of a portfolio that was not portable. A strength of portfolios is to guide learners toward collecting critical artifacts -articles, concepts (in the form of essay tasks), templates (dictation reports with key phrases), and lessons (in the forms of reflections) -that can be carried into early practice. Within my first week of practice, I returned several times to my e-portfolio in search of a piece of my recorded history.

Point 5: Private parts
Good portfolio software should provide a range of privacy settings. Privacy of personal information is a universal concern, particularly in the field of healthcare. All patient information posted in a learner's portfolio should be anonymized. Learners' personal reflections and research projects require a judicious amount of protection. In order for a learner to truly be open in his or her reflections, there should be the opportunity to "lock down" a particular reflection so that it is not accessible to even the Program Director or mentor, for the sake of the learner. Learner: To make meaning of the charged moments in medical training -the successes, the setbacks, the errorsone has to have space to be vulnerable. For this to be achieved in a meaningful way, security for the learner's thoughts and reflections is critical.

Tailor to fit
Can the e-portfolio be customized to meet the needs of my program? What resources are required to customize the system?

Conclusion
Implementing e-portfolios continues to be a daunting task for many Program Directors. Our twelve points provide the medical educator with the solid foundation needed to embark on the journey of e-portfolio platform design and implementation. The twelve tips that have been outlined in this paper need to be combined with a solid content framework in order to ensure that the technology of e-portfolios will add value to the training program and not be a waste of time, energy and money. As we continue to move toward competency-based learning, e-portfolios will provide a platform and stimulus for sustained excellence in medical education.

Take Home Messages
Creating and implementing an e-portfolio can be overwhelming for program directors and administrators. Don't reinvent the wheel.
Determine what essentials are needed in your portfolio and what is non-negotiable. Work from there.
Choose a system that will work for your program today and in the relative future. No platform will last forever.

Notes On Contributors
Lyn K. Sonnenberg is Program Director for Developmental Pediatrics with particular interest academic technology as it relates to teaching and assessment in medicine.
Patrick von Hauff is a designer and academic technology specialist at the University of Alberta, FoMD. Patrick is interested in design, technology, and interprofessional learning.
Lisa Lemieux is a recent graduate in Developmental Pediatrics and is now an avid medical educator at the University of Alberta in the Department of Pediatrics.