Skip to main content

Advertisement

SpringerLink
Log in

Automatic scoring of virtual mastoidectomies using expert examples

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Automatic scoring of resident performance on a virtual mastoidectomy simulation system is needed to achieve consistent and efficient evaluations. By not requiring immediate expert intervention, the system provides a completely objective assessment of performance as well as a self-driven user assessment mechanism.

Methods

An iconic temporal bone with surgically important regions defined into a fully partitioned segmented dataset was created. Comparisons between expert-drilled bones and student-drilled bones were computed based on gradations with both Euclidean and Earth Mover’s Distance. Using the features derived from these comparisons, a decision tree was constructed. This decision tree was used to determine scores of resident surgical performance. The algorithm was applied on multiple expert comparison bones and the scores averaged to provide reliability metric.

Results

The reliability metrics for the multi-grade scoring system are better in some cases than previously reported binary classification metrics. The two scoring methods given provide a trade-off between accuracy and speed.

Conclusions

Comparison of virtually drilled bones with expert examples on a voxel level provides sufficient information to score them and provide several specific quality metrics. By merging scores from different expert examples, two related metrics were developed; one is slightly faster and less accurate, while a second is more accurate but takes more processing time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahmidi N, Hager GD, Ishii L, Fichtinger G, Gallia GL, Ishii M (2010) Surgical task and skill classification from eye tracking and tool motion in minimally invasive surgery. In: Jiang T, Navab N, Pluim JP, Viergever MA (eds) MICCAI. Springer, Berlin, pp 295–302

    Google Scholar 

  2. Bryan J, Stredney D, Wiet G, Sessanna D (2001) Virtual temporal bone dissection: a case study. In: IEEE visualization, pp 497–500

  3. Butler NN, Wiet GJ (2007) Reliability of the Welling scale (WS1) for rating temporal bone dissection performance. Laryngoscope 117(10): 1803–1808. doi:10.1097/MLG.0b013e31811edd7a

    Article  PubMed  Google Scholar 

  4. Cleveland WS, McGill R (1984) The many faces of a scatterplot. J Am Stat Assoc 79(388): 807–822

    Article  Google Scholar 

  5. Cotin S, Stylopoulos N, Ottensmeyer MP, Neumann PF, Rattner D, Dawson S (2002) Metrics for laparoscopic skills trainers: the weakest link! In: MICCAI, pp 35–43

  6. Cristancho SM, Hodgson AJ, Panton ONM, Meneghetti A, Warnock G, Qayumi K (2009) Intraoperative monitoring of laparoscopic skill development based on quantitative measures. Surg endosc 23(10): 90–2181. doi:10.1007/s00464-008-0246-9

    Article  Google Scholar 

  7. Hothorn T, Hornik K, Zeileis A (2006) Unbiased recursive partitioning: a conditional inference framework. J Comput Graph Stat 15(3): 651–674. doi:10.1198/106186006X133933

    Article  Google Scholar 

  8. Janoos F, Machiraju R, Sammet S, Knopp M, Mórocz I (2010) Unsupervised learning of brain states from fMRI data. In: Jiang T, Navab N, Pluim J, Viergever M (eds) MICCAI Lecture notes in computer science, vol 6362. Springer, Berlin, pp 201–208

    Google Scholar 

  9. Kerwin T, Shen HW, Stredney D (2009) Enhancing realism of wet surfaces in temporal bone surgical simulation. IEEE Trans Vis Comput Graph 15(5): 747–758. doi:10.1109/TVCG.2009.31

    Article  PubMed  Google Scholar 

  10. Kuroiwa S, Umeda Y, Tsuge S, Ren F (2006) Nonparametric speaker recognition method using earth mover’s distance. IEICE Trans Inf Syst. 1074–1081

  11. Laeeq K, Bhatti NI, Carey JP, Della Santina CC, Limb CJ, Niparko JK, Minor LB, Francis HW (2009) Pilot testing of an assessment tool for competency in mastoidectomy. Laryngoscope 119(12): 2402–2410. doi:10.1002/lary.20678

    Article  PubMed  Google Scholar 

  12. Mackel T, Rosen J, Pugh C (2006) Data mining of the E-pelvis simulator database: a quest for a generalized algorithm for objectively assessing medical skill. In: MMVR, vol 119, pp 355–60

  13. Megali G, Sinigaglia S, Tonet O, Dario P (2006) Modelling and evaluation of surgical performance using hidden Markov models. IEEE Trans Biomed Eng 53(10): 1911–1919. doi:10.1109/TBME.2006.881784

    Article  PubMed  Google Scholar 

  14. Murphy TE (2004) Towards objective surgical skill evaluation with hidden Markov Model-based motion recognition. Master’s thesis, Johns Hopkins University

  15. Pele O, Werman M (2009) Fast and robust earth mover’s distances. In: International conference on computer vision. Kyoto, Japan

  16. Porte MC, Xeroulis G, Reznick RK, Dubrowski A (2007) Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. Am J Surg 193(1): 105–110. doi:10.1016/j.amjsurg.2006.03.016

    Article  PubMed  Google Scholar 

  17. Rosen J, Brown JD, Chang L, Sinanan MN, Hannaford B (2006) Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Trans Biomed Eng 53(3): 399–413. doi:10.1109/TBME.2005.869771

    Article  PubMed  Google Scholar 

  18. Rosen J, Hannaford B, Richards CG, Sinanan MN (2001) Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans Biomed Eng 48(5): 579–591. doi:10.1109/10.918597

    Article  PubMed  CAS  Google Scholar 

  19. Sewell C, Morris D, Blevins NH, Dutta S, Agrawal S, Barbagli F, Salisbury K (2008) Providing metrics and performance feedback in a surgical simulator. Comput Aided Surg 13(2): 63–81. doi:10.1080/10929080801957712

    PubMed  Google Scholar 

  20. Shaffer DW, Gordon J, Bennett N (2004) Learning, testing, and the evaluation of learning environments in medicine: global performance assessment in medical education. Interact Learn Environ 12(3): 167–178. doi:10.1080/10494820512331383409

    Article  Google Scholar 

  21. Sun Y, Lei M (2009) Method for optical coherence tomography image classification using local features and earth mover’s distance. J Biomed Opt 14(5):054,037-6

    Google Scholar 

  22. Wan D, Wiet GJ, Welling DB, Kerwin T, Stredney D (2010) Creating a cross-institutional grading scale for temporal bone dissection. Laryngoscope 120(7): 1422–1427. doi:10.1002/lary.20957

    Article  PubMed  Google Scholar 

  23. Wiet GJ (2010) Virtual temporal bone dissection system: development and testing. Triological Society Thesis (Submitted)

  24. Yuxin P, Cuihua F, Xiaoou C (2006) Using earth movers distance for audio clip retrieval. In: Zhuang Y, Yang SQ, Rui Y, He Q (eds) Advances in multimedia information processing, lecture notes in computer science, vol 4261. Springer, Berlin, pp 405–413. doi:10.1007/11922162

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Ohio Supercomputer Center, Columbus, OH, USA

    Thomas Kerwin & Don Stredney

  2. Department of Otolaryngology and Biomedical Informatics, Nationwide Children’s Hospital, Columbus, OH, USA

    Gregory Wiet

  3. The Ohio State University Medical Center, Columbus, OH, USA

    Gregory Wiet

  4. Department of Computer Science and Engineering, Ohio State University, Columbus, OH, USA

    Thomas Kerwin & Han-Wei Shen

Authors
  1. Thomas Kerwin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregory Wiet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Don Stredney
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Han-Wei Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Kerwin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kerwin, T., Wiet, G., Stredney, D. et al. Automatic scoring of virtual mastoidectomies using expert examples. Int J CARS 7, 1–11 (2012). https://doi.org/10.1007/s11548-011-0566-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-011-0566-4

Keywords

Search

Navigation