Skip to main content
Log in

Recognizing Reflection: Computer-Assisted Analysis of First Year Medical Students’ Reflective Writing

Medical Science Educator Aims and scope Submit manuscript

Abstract

Background

Reflective writing is used throughout medical education to help students navigate their transformation into medical professionals. Assessment of reflective writing, however, is challenging; each available methodology of assessment has distinct advantages and disadvantages. We tested if combining two independent assessment mechanisms—a faculty-designed rubric and Academic Writing Analytics (AWA), an automated technique—could be used together to form a more robust form of evaluation.

Methods

We obtained reflective essays written by first year medical students as part of a clinical skills course. Faculty scored essays using a rubric designed to evaluate Integration, Depth, and Writing. The same essays were subjected to AWA analysis, which counted the number of reflective phrases indicative of Context, Challenge, or Change.

Results

Faculty scored the essays uniformly high, indicating that most students met the standard for reflection as described by the rubric. AWA identified over 1400 instances of reflective behavior within the essays, and there was significant variability in how often different types of reflective phrases were used by individual students.

Conclusions

While data from faculty assessment or AWA alone is sufficient to evaluate reflective essays, combining these methods offer a richer and more valuable understanding of the student’s reflection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Data Availability

The raw dataset is available through the Open Science Framework repository at https://mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2Fvhwuq%2Fdownload

References

  1. Liaison Committee on Medical Education. LCME accreditation standards. 2008. http://www.lcme.org/functions2008jun.pdf.

  2. Sandars J. The use of reflection in medical education: AMEE guide no. 44. Med Teach. 2009;31(8):685–95.

    Article  Google Scholar 

  3. Cruess RL, Cruess SR, Boudreau JD, Snell L, Steinert Y. A schematic representation of the professional identity formation and socialization of medical students and residents: a guide for medical educators. Acad Med. 2015;90(6):718–25.

    Article  Google Scholar 

  4. Weaver R, Peters K, Koch J, Wilson I. ‘Part of the team’: Professional identity and social exclusivity in medical students. Med Educ. 2011;45(12):1220–9.

    Article  Google Scholar 

  5. Branch WT Jr, George M. Reflection-based learning for professional ethical formation. AMA J Ethics. 2017;19(4):349–56.

    Article  Google Scholar 

  6. Wald HS, Anthony D, Hutchinson TA, Liben S, Smilovitch M, Donato AA. Professional identity formation in medical education for humanistic, resilient physicians: pedagogic strategies for bridging theory to practice. Acad Med. 2015;90(6):753–60.

    Article  Google Scholar 

  7. Quirk M. Intuition and metacognition in medical education: keys to developing expertise: Springer publishing company; 2006.

  8. Tsingos-Lucas C, Bosnic-Anticevich S, Schneider CR, Smith L. Using reflective writing as a predictor of academic success in different assessment formats. Am J Pharm Educ. 2017;81(1).

  9. Di Stefano G, Gino F, Pisano G, Staats B. An appraisal of medical students’ reflection-in-learning. Med Educ. 2000;34(3):182–7.

    Article  Google Scholar 

  10. Hoffman LA, Shew RL, Vu TR, Brokaw JJ, Frankel RM. Is reflective ability associated with professionalism lapses during medical school? Acad Med. 2016;91(6):853–7.

    Article  Google Scholar 

  11. Mamede S, Schmidt HG, Penaforte JC. Effects of reflective practice on the accuracy of medical diagnoses. Med Educ. 2008;42(5):468–75.

    Article  Google Scholar 

  12. Croke E. The use of structured reflective journal questions to promote fundamental development of clinical decision-making abilities of the first-semester nursing student. Contemp Nurse. 2004;17(1–2):125–36.

    Article  Google Scholar 

  13. Sukhato K, Sumrithe S, Wongrathanandha C, Hathirat S, Leelapattana W, Dellow A. To be or not to be a facilitator of reflective learning for medical students? A case study of medical teachers’ perceptions of introducing a reflective writing exercise to an undergraduate curriculum. BMC Emerg Med. 2016;16(1):102.

    Google Scholar 

  14. Branch WT Jr, Frankel R. Not all stories of professional identity formation are equal: an analysis of formation narratives of highly humanistic physicians. Patient Educ Couns. 2016;99(8):1394–9.

    Article  Google Scholar 

  15. Koole S, Dornan T, Aper L, Scherpbier A, Valcke M, Cohen-Schotanus J, et al. Factors confounding the assessment of reflection: a critical review. BMC Emerg Med. 2011;11(1):104.

    Google Scholar 

  16. Aitchison C, Catterall J, Ross P, Burgin S. ‘Tough love and tears’: learning doctoral writing in the sciences. High Educ Res Dev. 2012;31(4):435–47. https://doi.org/10.1080/07294360.2011.559195.

    Article  Google Scholar 

  17. McCarthy J. Reflective writing, higher education and professional practice. J Educ Built Environ. 2011;6(1):29–43.

    Article  Google Scholar 

  18. Grant A, Kinnersley P, Metcalf E, Pill R, Houston H. Students' views of reflective learning techniques: an efficacy study at a UK medical school. Med Educ. 2006;40(4):379–88.

    Article  Google Scholar 

  19. Gibson A, Aitken A, Sandor A, Buckingham-Shum S, Tsingos-Lucas C, Knight S. Reflective writing analytics for actionable feedback. In: Proc. 7th Int. Conf. Learning Analytics & Knowledge; 2017. p. 13–7. https://doi.org/10.1145/3027385.3027436.

    Chapter  Google Scholar 

  20. Buckingham Shum S, Sándor Á, Goldsmith R, Bass R, McWilliams M. Towards reflective writing analytics: rationale, methodology and preliminary results. J Learn Anal. 2017;4(1):58–84. https://doi.org/10.18608/jla.2017.41.5.

    Article  Google Scholar 

  21. Lucas C, Gibson A, Buckingham SS. Utilization of a novel online reflective learning tool for immediate formative feedback to assist pharmacy students’ reflective writing skills. Am J Pharm Educ. 2018;83:6800. https://doi.org/10.5688/ajpe6800.

    Article  Google Scholar 

  22. Knight S, Buckingham Shum S, Ryan P, Sandor A, Wang X. Designing academic writing analytics for civil law student self assessment. Int J Artif Intell Educ. 2018;28(1):1–28. https://doi.org/10.1007/s40593-016-0121-0.

    Article  Google Scholar 

  23. Nevalainen MK, Mantyranta T, Pitkala KH. Facing uncertainty as a medical student—a qualitative study of their reflective learning diaries and writings on specific themes during the first clinical year. Patient Educ Couns. 2010;78(2):218–23.

    Article  Google Scholar 

  24. Cunningham H, Taylor D, Desai UA, Quiah SC, Kaplan B, Fei L, Catallozzi M, Richards B, Balmer DF, Charon R. Looking back to move forward: first-year medical students’ meta-reflections on their narrative portfolio writings. Acad Med. 2018;93(6):888–94.

    Article  Google Scholar 

  25. Gaufberg EH, Batalden M, Sands R, Bell SK. The hidden curriculum: what can we learn from third-year medical student narrative reflections? Acad Med. 2010;85(11):1709–16.

    Article  Google Scholar 

  26. Kristiansson MH, Troein M, Brorsson A. We lived and breathed medicine-then life catches up: medical students’ reflections. BMC Emerg Med. 2014;14(1):66.

    Google Scholar 

  27. Miller-Kuhlmann R, O’Sullivan PS, Aronson L. Essential steps in developing best practices to assess reflective skill: a comparison of two rubrics. Med Teach. 2016;38(1):75–81.

    Article  Google Scholar 

  28. Aronson L, Niehaus B, Lindow J, Robertson PA, O'sullivan PS. Development and pilot testing of a reflective learning guide for medical education. Med Teach. 2011;33(10):e515–21.

    Article  Google Scholar 

  29. Aronson L, Niehaus B, Hill-Sakurai L, Lai C, O’Sullivan PS. A comparison of two methods of teaching reflective ability in year 3 medical students. Med Educ. 2012;46:807–14.

    Article  Google Scholar 

  30. Sharpless J, Baldwin N, Cook R, Kofman A, Morley-Fletcher A, Slotkin R, et al. The becoming: students’ reflections on the process of professional identity formation in medical education. Acad Med. 2015;90(6):713–7.

    Article  Google Scholar 

  31. Bernard AW, Malone M, Kman NE, Caterino JM, Khandelwal S. Medical student professionalism narratives: a thematic analysis and interdisciplinary comparative investigation. BMC Emerg Med. 2011;11(1):11.

    Article  Google Scholar 

  32. Moon JA. Reflection in learning and professional development: theory and practice. London: Kogan Page; 1999.

    Google Scholar 

  33. Boud D, Keogh R, Walker D. Reflection: turning experience into learning. London: Kogan Page; 1985.

    Google Scholar 

  34. Schön DA. The reflective practitioner: how professionals think in action. New York: Basic Books; 1983.

    Google Scholar 

  35. Korthagen FAJ. Reflective teaching and preservice teacher education in the Netherlands. J Teach Educ. 1985;9(3):317–26.

    Google Scholar 

  36. Kolb DA. Experiential learning: experience as a source for learning and development. Upper Saddle River: Prentice Hall; 1984.

    Google Scholar 

  37. Wald HS, Borkan JM, Taylor JS, Anthony D, Reis SP. Fostering and evaluating reflective capacity in medical education: developing the REFLECT rubric for assessing reflective writing. Acad Med. 2012;87(1):41–50.

    Article  Google Scholar 

  38. Shibani A, Knight S, Buckingham Shum S. Contextualizable learning analytics design: a generic model, and writing analytics evaluations. In: Proc. 9th Int. Conf. Learning Analytics & Knowledge; 2019. p. 210–9. https://doi.org/10.1145/3303772.3303785.

    Chapter  Google Scholar 

  39. DasGupta S, Charon R. Personal illness narratives: using reflective writing to teach empathy. Acad Med. 2004;79(4):351–6.

    Article  Google Scholar 

  40. Charon R, Hermann MN. A sense of story, or why teach reflective writing? Acad Med. 2012;87(1):5–7 (Jan).

    Article  Google Scholar 

  41. Cayley JW, Schilling R, Suechting R. Changes in themes over time from medical student journaling. WMJ. 2007;106(8):486–9 (Dec).

    Google Scholar 

  42. Sobral DT. An appraisal of medical students’ reflection-in-learning. Med Educ. 2000;34(3):182–7.

    Article  Google Scholar 

  43. Heeneman S, de Grave W. Tensions in mentoring medical students toward self-directed and reflective learning in a longitudinal portfolio-based mentoring system–an activity theory analysis. Med Teach. 2017;39(4):368–76.

    Article  Google Scholar 

  44. Kalén S, Ponzer S, Seeberger A, Kiessling A, Silén C. Continuous mentoring of medical students provides space for reflection and awareness of their own development. Int J Med Educ. 2012;3:236–44.

    Article  Google Scholar 

  45. Sargeant JM, Mann KV, Van der Vleuten CP, Metsemakers JF. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ. 2009;14(3):399–410.

    Article  Google Scholar 

  46. Boehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, et al. An investigation of medical student reactions to feedback: a randomised controlled trial. Med Educ. 2006;40(8):746–9.

    Article  Google Scholar 

  47. Eckes T. Rater types in writing performance assessments: a classification approach to rater variability. Lang Test. 2008;25(2):155–85.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Dr. Catherine G. Takizawa, PhD, for her helpful comments and critical analysis of the manuscript.

Funding

Not applicable.

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Caitlin Hanlon. AWA analysis was performed by Andrew Gibson. The first draft of the manuscript was written by Caitlin Hanlon and Emily Frosch and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Caitlin D. Hanlon.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethics Approval

Johns Hopkins University Internal Review Board (IRB:00145005).

Consent to Participate

Not applicable.

Consent for Publication

All authors read and approved the final manuscript.

Code Availability

https://github.com/heta-io

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

ESM 1

(PDF 169 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hanlon, C.D., Frosch, E.M., Shochet, R.B. et al. Recognizing Reflection: Computer-Assisted Analysis of First Year Medical Students’ Reflective Writing. Med.Sci.Educ. 31, 109–116 (2021). https://doi.org/10.1007/s40670-020-01132-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40670-020-01132-7

Keywords

Navigation