Skip to main content
SpringerLink
Log in

Using natural language processing to determine factors associated with high-quality feedback

  • Original Article
  • Published:
Global Surgical Education - Journal of the Association for Surgical Education Aims and scope Submit manuscript

Abstract

Purpose

Feedback is a cornerstone of medical education. However, not all feedback that residents receive is high-quality. Natural language processing (NLP) can be used to efficiently examine the quality of large amounts of feedback. We used a validated NLP model to examine factors associated with the quality of feedback that general surgery trainees received on 24,531 workplace-based assessments of operative performance.

Methods

We analyzed transcribed, dictated feedback from the Society for Improving Medical Professional Learning’s (SIMPL) smartphone-based app. We first applied a validated NLP model to all SIMPL evaluations that had dictated feedback, which resulted in a predicted probability that an instance of feedback was “relevant”, “specific”, and/or “corrective.” Higher predicted probabilities signaled an increased likelihood that feedback was high quality. We then used linear mixed-effects models to examine variation in predictive probabilities across programs, attending surgeons, trainees, procedures, autonomy granted, operative performance level, case complexity, and a trainee’s level of clinical training.

Results

Linear mixed-effects modeling demonstrated that predicted probabilities, i.e., a proxy for quality, were lower as operative autonomy increased (“Passive Help” B = − 1.29, p < .001; “Supervision Only” B = − 5.53, p < 0.001). Similarly, trainees who demonstrated “Exceptional Performance” received lower quality feedback (B = − 12.50, p < 0.001). The specific procedure or trainee did not have a large effect on quality, nor did the complexity of the case or the PGY level of a trainee. The individual faculty member providing the feedback, however, had a demonstrable impact on quality with approximately 36% of the variation in quality attributable to attending surgeons.

Conclusions

We were able to identify actionable items affecting resident feedback quality using an NLP model. Attending surgeons are the most influential factor in whether feedback is high quality. Faculty should be directly engaged in efforts to improve the overall quality of feedback that residents receive.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Jackson JL, Kay C, Jackson WC, Frank M. The quality of written feedback by attendings of internal medicine residents. J Gen Intern Med. 2015;30(7):973–8. https://doi.org/10.1007/s11606-015-3237-2.

    Article  Google Scholar 

  2. Bing-You RG, Trowbridge RL. Why medical educators may be failing at feedback. JAMA. 2009;302(12):1330–1.

    Article  CAS  Google Scholar 

  3. Jug R, Jiang X “Sara”, Bean SM. Giving and receiving effective feedback: a review article and how-to guide. Arch Pathol Lab Med 2019;143(2):244–250. https://doi.org/10.5858/arpa.2018-0058-RA

  4. Hewson MG, Little ML. Giving feedback in medical education. J Gen Intern Med. 1998;13(2):111–6. https://doi.org/10.1046/j.1525-1497.1998.00027.x.

    Article  CAS  Google Scholar 

  5. Lefroy J, Watling C, Teunissen PW, Brand P. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ. 2015;4(6):284–99. https://doi.org/10.1007/s40037-015-0231-7.

    Article  Google Scholar 

  6. Jensen AR, Wright AS, Kim S, Horvath KD, Calhoun KE. Educational feedback in the operating room: a gap between resident and faculty perceptions. Am J Surg. 2012;204(2):248–55. https://doi.org/10.1016/J.AMJSURG.2011.08.019.

    Article  Google Scholar 

  7. Bello RJ, Sarmiento S, Meyer ML, et al. Understanding surgical resident and fellow perspectives on their operative performance feedback needs: a qualitative study. J Surg Educ. 2018;75(6):1498–503. https://doi.org/10.1016/J.JSURG.2018.04.002.

    Article  Google Scholar 

  8. Rose JS, Waibel BH, Schenarts PJ. Disparity between resident and faculty surgeons’ perceptions of preoperative preparation, intraoperative teaching, and postoperative feedback. J Surg Educ. 2011;68(6):459–64. https://doi.org/10.1016/J.JSURG.2011.04.003.

    Article  Google Scholar 

  9. Kornegay JG, Kraut A, Manthey D, et al. Feedback in medical education: a critical appraisal. AEM Educ Train. 2017;1(2):98–109. https://doi.org/10.1002/aet2.10024.

    Article  Google Scholar 

  10. McKendy KM, Watanabe Y, Lee L, et al. Perioperative feedback in surgical training: a systematic review. Am J Surg. 2017;214(1):117–26. https://doi.org/10.1016/J.AMJSURG.2016.12.014.

    Article  Google Scholar 

  11. Norcini JJ. Workplace assessment. In: Understanding medical education: evidence, theory and practice. Wiley, 2013; 2013. https://books.google.com/books?hl=en&lr=&id=EWsKAgAAQBAJ&oi=fnd&pg=PA279&dq=Norcini+Workplace+assessment&ots=5SNX3lR8VE&sig=GZgxsYdM7MDc_xsrIiKvbnj1fv8. Accessed 15 May 2022.

  12. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29(9-10):855-871. https://doi.org/10.1080/01421590701775453

  13. Solano QP, Hayward L, Chopra Z, et al. Natural language processing and assessment of resident feedback quality. J Surg Educ. 2021;78(6):e72–7. https://doi.org/10.1016/J.JSURG.2021.05.012.

    Article  Google Scholar 

  14. Ötles E, Kendrick DE, Solano QP, et al. Using natural language processing to automatically assess feedback quality: findings from 3 surgical residencies. Acad Med. 2021. https://doi.org/10.1097/ACM.0000000000004153.

    Article  Google Scholar 

  15. George BC, Bohnen JD, Schuller MC, Fryer JP. Using smartphones for trainee performance assessment: a SIMPL case study. Surgery. 2020;167(6):903–6. https://doi.org/10.1016/j.surg.2019.09.011.

    Article  Google Scholar 

  16. Bohnen JD, George BC, Williams RG, et al. The Feasibility of real-time intraoperative performance assessment with SIMPL (system for improving and measuring procedural learning): early experience from a multi-institutional trial. J Surg Educ. 2016;73(6):e118–30. https://doi.org/10.1016/j.jsurg.2016.08.010.

    Article  Google Scholar 

  17. Williams RG, George BC, Bohnen JD, et al. A proposed blueprint for operative performance training, assessment, and certification. Ann Surg. 2021. https://doi.org/10.1097/SLA.0000000000004467.

    Article  Google Scholar 

  18. Ahle SL, Eskender M, Schuller M, et al. The quality of operative performance narrative feedback: a retrospective data comparison between end of rotation evaluations and workplace-based assessments. Ann Surg. 2020. https://doi.org/10.1097/SLA.0000000000003907.

    Article  Google Scholar 

  19. DaRosa DA, Zwischenberger JB, Meyerson SL, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ. 2013;70(1):24–30. https://doi.org/10.1016/j.jsurg.2012.07.007.

    Article  Google Scholar 

  20. Kogan JR, Conforti LN, Bernabeo EC, Durning SJ, Hauer KE, Holmboe ES. Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Med Educ. 2012;46(2):201–15. https://doi.org/10.1111/j.1365-2923.2011.04137.x.

    Article  Google Scholar 

  21. Junod Perron N, Nendaz M, Louis-Simonet M, et al. Effectiveness of a training program in supervisors’ ability to provide feedback on residents’ communication skills. Adv Heal Sci Educ. 2013;18(5):901–15. https://doi.org/10.1007/s10459-012-9429-1.

    Article  Google Scholar 

  22. Minehart RD, Rudolph J, Pian-Smith MCM, Raemer DB. Improving faculty feedback to resident trainees during a simulated case: a randomized, controlled trial of an educational intervention. Anesthesiology. 2014;120(1):160–71. https://doi.org/10.1097/ALN.0000000000000058.

    Article  Google Scholar 

  23. Zendejas B, Toprak A, Harrington AW, Lillehei CW, Modi BP. Quality of dictated feedback associated with SIMPL operative assessments of pediatric surgical trainees. Am J Surg. 2021;221(2):303–8. https://doi.org/10.1016/j.amjsurg.2020.10.014.

    Article  Google Scholar 

  24. Baker K. Clinical teaching improves with resident evaluation and feedback. Anesthesiology. 2010;113(3):693–703. https://doi.org/10.1097/ALN.0b013e3181eaacf4.

    Article  Google Scholar 

  25. Springer MV, Sales AE, Islam N, et al. A step toward understanding the mechanism of action of audit and feedback: a qualitative study of implementation strategies. Implement Sci. 2021;16(1):35. https://doi.org/10.1186/s13012-021-01102-6.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Surgical Training and Research, Department of Surgery, Michigan Medicine, 2800 Plymouth Road, NCRC Building 10 Room A193, Ann Arbor, MI, 48109-2800, USA

    Kayla M. Marcotte, Angela E. Thelen, Rebecca Gates, Brian C. George & Andrew E. Krumm

  2. Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI, USA

    Kayla M. Marcotte & Andrew E. Krumm

  3. Department of Industrial and Operations Engineering, University of Michigan Medical School, Ann Arbor, MI, USA

    Erkin Ötleş

  4. Department of Surgery, Michigan Medicine, Ann Arbor, MI, USA

    Brian C. George

Authors
  1. Kayla M. Marcotte
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erkin Ötleş
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Angela E. Thelen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rebecca Gates
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Brian C. George
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andrew E. Krumm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kayla M. Marcotte.

Ethics declarations

Conflict of interest

The authors have no financial disclosures.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Marcotte, K.M., Ötleş, E., Thelen, A.E. et al. Using natural language processing to determine factors associated with high-quality feedback. Global Surg Educ 1, 58 (2022). https://doi.org/10.1007/s44186-022-00051-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s44186-022-00051-y

Keywords

Search

Navigation