Abstract
Background
Reflective writing is used throughout medical education to help students navigate their transformation into medical professionals. Assessment of reflective writing, however, is challenging; each available methodology of assessment has distinct advantages and disadvantages. We tested if combining two independent assessment mechanisms—a faculty-designed rubric and Academic Writing Analytics (AWA), an automated technique—could be used together to form a more robust form of evaluation.
Methods
We obtained reflective essays written by first year medical students as part of a clinical skills course. Faculty scored essays using a rubric designed to evaluate Integration, Depth, and Writing. The same essays were subjected to AWA analysis, which counted the number of reflective phrases indicative of Context, Challenge, or Change.
Results
Faculty scored the essays uniformly high, indicating that most students met the standard for reflection as described by the rubric. AWA identified over 1400 instances of reflective behavior within the essays, and there was significant variability in how often different types of reflective phrases were used by individual students.
Conclusions
While data from faculty assessment or AWA alone is sufficient to evaluate reflective essays, combining these methods offer a richer and more valuable understanding of the student’s reflection.
Similar content being viewed by others
Data Availability
The raw dataset is available through the Open Science Framework repository at https://mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2Fvhwuq%2Fdownload
References
Liaison Committee on Medical Education. LCME accreditation standards. 2008. http://www.lcme.org/functions2008jun.pdf.
Sandars J. The use of reflection in medical education: AMEE guide no. 44. Med Teach. 2009;31(8):685–95.
Cruess RL, Cruess SR, Boudreau JD, Snell L, Steinert Y. A schematic representation of the professional identity formation and socialization of medical students and residents: a guide for medical educators. Acad Med. 2015;90(6):718–25.
Weaver R, Peters K, Koch J, Wilson I. ‘Part of the team’: Professional identity and social exclusivity in medical students. Med Educ. 2011;45(12):1220–9.
Branch WT Jr, George M. Reflection-based learning for professional ethical formation. AMA J Ethics. 2017;19(4):349–56.
Wald HS, Anthony D, Hutchinson TA, Liben S, Smilovitch M, Donato AA. Professional identity formation in medical education for humanistic, resilient physicians: pedagogic strategies for bridging theory to practice. Acad Med. 2015;90(6):753–60.
Quirk M. Intuition and metacognition in medical education: keys to developing expertise: Springer publishing company; 2006.
Tsingos-Lucas C, Bosnic-Anticevich S, Schneider CR, Smith L. Using reflective writing as a predictor of academic success in different assessment formats. Am J Pharm Educ. 2017;81(1).
Di Stefano G, Gino F, Pisano G, Staats B. An appraisal of medical students’ reflection-in-learning. Med Educ. 2000;34(3):182–7.
Hoffman LA, Shew RL, Vu TR, Brokaw JJ, Frankel RM. Is reflective ability associated with professionalism lapses during medical school? Acad Med. 2016;91(6):853–7.
Mamede S, Schmidt HG, Penaforte JC. Effects of reflective practice on the accuracy of medical diagnoses. Med Educ. 2008;42(5):468–75.
Croke E. The use of structured reflective journal questions to promote fundamental development of clinical decision-making abilities of the first-semester nursing student. Contemp Nurse. 2004;17(1–2):125–36.
Sukhato K, Sumrithe S, Wongrathanandha C, Hathirat S, Leelapattana W, Dellow A. To be or not to be a facilitator of reflective learning for medical students? A case study of medical teachers’ perceptions of introducing a reflective writing exercise to an undergraduate curriculum. BMC Emerg Med. 2016;16(1):102.
Branch WT Jr, Frankel R. Not all stories of professional identity formation are equal: an analysis of formation narratives of highly humanistic physicians. Patient Educ Couns. 2016;99(8):1394–9.
Koole S, Dornan T, Aper L, Scherpbier A, Valcke M, Cohen-Schotanus J, et al. Factors confounding the assessment of reflection: a critical review. BMC Emerg Med. 2011;11(1):104.
Aitchison C, Catterall J, Ross P, Burgin S. ‘Tough love and tears’: learning doctoral writing in the sciences. High Educ Res Dev. 2012;31(4):435–47. https://doi.org/10.1080/07294360.2011.559195.
McCarthy J. Reflective writing, higher education and professional practice. J Educ Built Environ. 2011;6(1):29–43.
Grant A, Kinnersley P, Metcalf E, Pill R, Houston H. Students' views of reflective learning techniques: an efficacy study at a UK medical school. Med Educ. 2006;40(4):379–88.
Gibson A, Aitken A, Sandor A, Buckingham-Shum S, Tsingos-Lucas C, Knight S. Reflective writing analytics for actionable feedback. In: Proc. 7th Int. Conf. Learning Analytics & Knowledge; 2017. p. 13–7. https://doi.org/10.1145/3027385.3027436.
Buckingham Shum S, Sándor Á, Goldsmith R, Bass R, McWilliams M. Towards reflective writing analytics: rationale, methodology and preliminary results. J Learn Anal. 2017;4(1):58–84. https://doi.org/10.18608/jla.2017.41.5.
Lucas C, Gibson A, Buckingham SS. Utilization of a novel online reflective learning tool for immediate formative feedback to assist pharmacy students’ reflective writing skills. Am J Pharm Educ. 2018;83:6800. https://doi.org/10.5688/ajpe6800.
Knight S, Buckingham Shum S, Ryan P, Sandor A, Wang X. Designing academic writing analytics for civil law student self assessment. Int J Artif Intell Educ. 2018;28(1):1–28. https://doi.org/10.1007/s40593-016-0121-0.
Nevalainen MK, Mantyranta T, Pitkala KH. Facing uncertainty as a medical student—a qualitative study of their reflective learning diaries and writings on specific themes during the first clinical year. Patient Educ Couns. 2010;78(2):218–23.
Cunningham H, Taylor D, Desai UA, Quiah SC, Kaplan B, Fei L, Catallozzi M, Richards B, Balmer DF, Charon R. Looking back to move forward: first-year medical students’ meta-reflections on their narrative portfolio writings. Acad Med. 2018;93(6):888–94.
Gaufberg EH, Batalden M, Sands R, Bell SK. The hidden curriculum: what can we learn from third-year medical student narrative reflections? Acad Med. 2010;85(11):1709–16.
Kristiansson MH, Troein M, Brorsson A. We lived and breathed medicine-then life catches up: medical students’ reflections. BMC Emerg Med. 2014;14(1):66.
Miller-Kuhlmann R, O’Sullivan PS, Aronson L. Essential steps in developing best practices to assess reflective skill: a comparison of two rubrics. Med Teach. 2016;38(1):75–81.
Aronson L, Niehaus B, Lindow J, Robertson PA, O'sullivan PS. Development and pilot testing of a reflective learning guide for medical education. Med Teach. 2011;33(10):e515–21.
Aronson L, Niehaus B, Hill-Sakurai L, Lai C, O’Sullivan PS. A comparison of two methods of teaching reflective ability in year 3 medical students. Med Educ. 2012;46:807–14.
Sharpless J, Baldwin N, Cook R, Kofman A, Morley-Fletcher A, Slotkin R, et al. The becoming: students’ reflections on the process of professional identity formation in medical education. Acad Med. 2015;90(6):713–7.
Bernard AW, Malone M, Kman NE, Caterino JM, Khandelwal S. Medical student professionalism narratives: a thematic analysis and interdisciplinary comparative investigation. BMC Emerg Med. 2011;11(1):11.
Moon JA. Reflection in learning and professional development: theory and practice. London: Kogan Page; 1999.
Boud D, Keogh R, Walker D. Reflection: turning experience into learning. London: Kogan Page; 1985.
Schön DA. The reflective practitioner: how professionals think in action. New York: Basic Books; 1983.
Korthagen FAJ. Reflective teaching and preservice teacher education in the Netherlands. J Teach Educ. 1985;9(3):317–26.
Kolb DA. Experiential learning: experience as a source for learning and development. Upper Saddle River: Prentice Hall; 1984.
Wald HS, Borkan JM, Taylor JS, Anthony D, Reis SP. Fostering and evaluating reflective capacity in medical education: developing the REFLECT rubric for assessing reflective writing. Acad Med. 2012;87(1):41–50.
Shibani A, Knight S, Buckingham Shum S. Contextualizable learning analytics design: a generic model, and writing analytics evaluations. In: Proc. 9th Int. Conf. Learning Analytics & Knowledge; 2019. p. 210–9. https://doi.org/10.1145/3303772.3303785.
DasGupta S, Charon R. Personal illness narratives: using reflective writing to teach empathy. Acad Med. 2004;79(4):351–6.
Charon R, Hermann MN. A sense of story, or why teach reflective writing? Acad Med. 2012;87(1):5–7 (Jan).
Cayley JW, Schilling R, Suechting R. Changes in themes over time from medical student journaling. WMJ. 2007;106(8):486–9 (Dec).
Sobral DT. An appraisal of medical students’ reflection-in-learning. Med Educ. 2000;34(3):182–7.
Heeneman S, de Grave W. Tensions in mentoring medical students toward self-directed and reflective learning in a longitudinal portfolio-based mentoring system–an activity theory analysis. Med Teach. 2017;39(4):368–76.
Kalén S, Ponzer S, Seeberger A, Kiessling A, Silén C. Continuous mentoring of medical students provides space for reflection and awareness of their own development. Int J Med Educ. 2012;3:236–44.
Sargeant JM, Mann KV, Van der Vleuten CP, Metsemakers JF. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ. 2009;14(3):399–410.
Boehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, et al. An investigation of medical student reactions to feedback: a randomised controlled trial. Med Educ. 2006;40(8):746–9.
Eckes T. Rater types in writing performance assessments: a classification approach to rater variability. Lang Test. 2008;25(2):155–85.
Acknowledgements
The authors thank Dr. Catherine G. Takizawa, PhD, for her helpful comments and critical analysis of the manuscript.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Caitlin Hanlon. AWA analysis was performed by Andrew Gibson. The first draft of the manuscript was written by Caitlin Hanlon and Emily Frosch and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethics Approval
Johns Hopkins University Internal Review Board (IRB:00145005).
Consent to Participate
Not applicable.
Consent for Publication
All authors read and approved the final manuscript.
Code Availability
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic Supplementary Material
ESM 1
(PDF 169 kb)
Rights and permissions
About this article
Cite this article
Hanlon, C.D., Frosch, E.M., Shochet, R.B. et al. Recognizing Reflection: Computer-Assisted Analysis of First Year Medical Students’ Reflective Writing. Med.Sci.Educ. 31, 109–116 (2021). https://doi.org/10.1007/s40670-020-01132-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40670-020-01132-7