Skip to main content

Advertisement

SpringerLink
Log in

Machine learning distinguishes neurosurgical skill levels in a virtual reality tumor resection task

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

This study outlines the first investigation of application of machine learning to distinguish “skilled” and “novice” psychomotor performance during a virtual reality (VR) brain tumor resection task. Tumor resection task participants included 23 neurosurgeons and senior neurosurgery residents as the “skilled” group and 92 junior neurosurgery residents and medical students as the “novice” group. The task involved removing a series of virtual brain tumors without causing injury to surrounding tissue. Originally, 150 features were extracted followed by statistical and forward feature selection. The selected features were provided to 4 classifiers, namely, K-Nearest Neighbors, Parzen Window, Support Vector Machine, and Fuzzy K-Nearest Neighbors. Sets of 5 to 30 selected features were provided to the classifiers. A working point of 15 premium features resulted in accuracy values as high as 90% using the Supprt Vector Machine. The obtained results highlight the potentials of machine learning, applied to VR simulation data, to help realign the traditional apprenticeship educational paradigm to a more objective model, based on proven performance standards.

Using several scenarios of virtual reality neurosurgical tumor resection together with machine learning classifiers to distinguish skill level

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Kockro RA, Serra L, Tseng-Tsai Y, Chan C, Yih-Yian S, Gim-Guan C et al (2000) Planning and simulation of neurosurgery in a virtual reality environment. Neurosurgery. 46(1):118–137

    CAS  PubMed  Google Scholar 

  2. Bernardo A, Preul MC, Zabramski JM, Spetzler RF (2003) A three-dimensional interactive virtual dissection model to simulate transpetrous surgical avenues. Neurosurgery. 52(3):499–505 discussion 504–505

    PubMed  Google Scholar 

  3. Radetzky A, Rudolph M (2001) Simulating tumour removal in neurosurgery. Int J Med Inform 64(2–3):461–472

    CAS  PubMed  Google Scholar 

  4. Lemole GM Jr, Banerjee PP, Luciano C, Neckrysh S, Charbel FT (2007) Virtual reality in neurosurgical education: part-task ventriculostomy simulation with dynamic visual and haptic feedback. Neurosurgery. 61(1):142–149

    PubMed  Google Scholar 

  5. Delorme S, Laroche D, DiRaddo R, Del Maestro RF (2012) NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training. Neurosurgery 71(suppl_1):ons32–ons42

    Google Scholar 

  6. Choudhury N, Gelinas-Phaneuf N, Delorme S, Del Maestro R (2013) Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg 80(5):e9–e19

    PubMed  Google Scholar 

  7. Gelinas-Phaneuf N, Del Maestro RF (2013) Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery 73(suppl_1):S30–S38

    Google Scholar 

  8. Gelinas-Phaneuf N, Choudhury N, Al-Habib AR, Cabral A, Nadeau E, Mora V et al (2014) Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J Comput Assist Radiol Surg 9(1):1–9

    PubMed  Google Scholar 

  9. Azarnoush H, Alzhrani G, Winkler-Schwartz A, Alotaibi F, Gelinas-Phaneuf N, Pazos V, Choudhury N, Fares J, DiRaddo R, del Maestro R (2015) Neurosurgical virtual reality simulation metrics to assess psychomotor skills during brain tumor resection. Int J Comput Assist Radiol Surg 10(5):603–618

    PubMed  Google Scholar 

  10. Cline BC, Badejo AO, Rivest II, Scanlon JR, Taylor WC, Gerling GJ (2008) Human performance metrics for a virtual reality simulator to train chest tube insertion. IEEE SIEDS :168–173

  11. Kazemi H, Rappel JK, Poston T, Hai Lim B, Burdet E, Leong TC (2010) Assessing suturing techniques using a virtual reality surgical simulator. Microsurgery. 30(6):479–486

    PubMed  Google Scholar 

  12. Trejos AL, Patel RV, Malthaner RA, Schlachta CM (2014) Development of force-based metrics for skills assessment in minimally invasive surgery. Surg Endosc 28(7):2106–2119

    PubMed  Google Scholar 

  13. Kovac ERA, Azhar A, Quirouet J, Delisle, Anidjar M (2012) Construct validity of the lapSim virtual reality laparoscopic simulator within a urology residency program. CUAJ 6(4):253

    PubMed  Google Scholar 

  14. Alotaibi FE, Al Zhrani G, Bajunaid K, Winkler-Schwartz A, Azarnoush H et al (2015) Assessing neurosurgical psychomotor performance: role of virtual reality simulators, current and future potential. SOJ Neurol 2(1):1–7

    Google Scholar 

  15. Alotaibi FE, AlZhrani GA, Mullah MA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A et al (2015) Assessing bimanual performance in brain tumor resection with NeuroTouch, a virtual reality simulator. Oper Neurosurg 11(1):89–98

    Google Scholar 

  16. Alotaibi FE, AlZhrani GA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, Del Maestro RF (2015) Neurosurgical assessment of metrics including judgment and dexterity using the virtual reality simulator NeuroTouch (NAJD Metrics). Surg Innov 22(6):636–642

    PubMed  Google Scholar 

  17. Jensen Ang WJ, Hopkins ME, Partridge R, Hennessey I, Brennan PM, Fouyas I, Hughes MA (2013) Validating the use of smartphone-based accelerometers for performance assessment in a simulated neurosurgical task. Oper Neurosurg 10(1):57–65

    Google Scholar 

  18. Azarnoush H, Siar S, Sawaya R, Zhrani GA, Winkler-Schwartz A, Alotaibi FE, Bugdadi A, Bajunaid K, Marwa I, Sabbagh AJ, del Maestro R (2017) The force pyramid: a spatial analysis of force application during virtual reality brain tumor resection. J Neurosurg 127(1):171–181

    PubMed  Google Scholar 

  19. Sawaya R, Bugdadi A, Azarnoush H, Winkler-Schwartz A, Alotaibi FE, Bajunaid K, AlZhrani GA, Alsideiri G, Sabbagh AJ, Del Maestro RF (2017) Virtual reality tumor resection: the force pyramid approach. Operative Neurosurgery. 14(6):686–696

    Google Scholar 

  20. Bugdadi A, Sawaya R, Olwi D, AlZahrani G, Azarnoush H, Sabbagh A et al (2018) Automaticity of force application during simulated brain tumor resection: testing the Fitts and Posner model. J Surg Educ 75(1):104–115

    PubMed  Google Scholar 

  21. Sawaya R, Alsidieri G, Bugdadi A, Winkler-Schwartz A, Azarnoush A, Bajunaid K, AJ JS, Del Maestro R (2018) Development of a performance model for virtual reality tumor resections. J Neurosurg 1(aop):1–9

    Google Scholar 

  22. Winkler-Schwartz A, Bajunaid K, Mullah MA, Marwa I, Alotaibi FE, Fares J et al (2016) Bimanual psychomotor performance in neurosurgical resident applicants assessed using NeuroTouch, a virtual reality simulator. J Surg Educ 73(6):942–953

    PubMed  Google Scholar 

  23. Holloway T, Lorsch Z, Chary M, Sobotka S, Moore MM, Costa AB, del Maestro R, Bederson J (2015) Operator experience determines performance in a simulated computer-based brain tumor resection task. Int J Comput Assist Radiol Surg 10(11):1853–1862

    PubMed  Google Scholar 

  24. Bajunaid K, Mullah MA, Winkler-Schwartz A, Alotaibi FE, Fares J, Baggiani M et al (2017) Impact of acute stress on psychomotor bimanual performance during a simulated tumor resection task. J Neurosurg 126(1):71–80

    PubMed  Google Scholar 

  25. Alzhrani G, Del Maestro RF (2014) A validation study of NeuroTouch in neurosurgical training. LAP LAMBERT Academic Publishing, Saarbrücken

    Google Scholar 

  26. Alzhrani G, Alotaibi F, Azarnoush H, Winkler-Schwartz A, Sabbagh A, Bajunaid K et al (2015) Proficiency performance benchmarks for removal of simulated brain tumors using a virtual reality simulator NeuroTouch. Journal of Surgical Education 72(4):685–696

    PubMed  Google Scholar 

  27. Samuel AL (1988) Some studies in machine learning using the game of checkers. In: Computer games I. Springer, New York, pp 366–400

    Google Scholar 

  28. Obermeyer Z, Emanuel EJ (2016) Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med 375(13):1216

    PubMed  PubMed Central  Google Scholar 

  29. Senders JT, Arnaout O, Karhade AV, Dasenbrock HH, Gormley WB, Broekman ML et al (2017) Natural and artificial intelligence in neurosurgery: a systematic review. Neurosurgery 83(2):181–192

    Google Scholar 

  30. Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S, Montazeri A (2015) Artificial neural networks in neurosurgery. J Neurol Neurosurg Psychiatry 86(3):251–256

    PubMed  Google Scholar 

  31. Watson RA (2014) Use of a machine learning algorithm to classify expertise: analysis of hand motion patterns during a simulated surgical task. Acad Med 89(8):1163–1167

    PubMed  Google Scholar 

  32. Rhienmora P, Haddawy P, Khanal P, Suebnukarn S, Dailey MN (2010) A virtual reality simulator for teaching and evaluating dental procedures. Methods Inf Med 49(04):396–405

    CAS  PubMed  Google Scholar 

  33. Kerwin T, Wiet G, Stredney D, Shen HW (2012) Automatic scoring of virtual mastoidectomies using expert examples. Int J Comput Assist Radiol Surg 7(1):1–11

    PubMed  Google Scholar 

  34. Ma X, Wijewickrema S, Zhou S, Zhou Y, Mhammedi Z, O'Leary S, et al. Adversarial generation of real-time feedback with neural networks for simulation-based training. arXiv preprint:1703.01460. 2017 Mar 4

  35. Ma X, Wijewickrema S, Zhou Y, Zhou S, O’Leary S, Bailey J (2017) Providing effective real-time feedback in simulation-based surgical training. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, pp 566–574

    Google Scholar 

  36. Wijewickrema S, Ma X, Piromchai P, Piromchai P, Briggs R, James BJ et al (2018) Providing automated real-time technical feedback for virtual reality based surgical training: is the simpler the better? In: International Conference on Artificial Intelligence in Education. Springer, pp 584–598, Cham

  37. Sewell C, Morris D, Blevins NH, Dutta S, Agrawal S, Federico Barbagli F et al (2008) Providing metrics and performance feedback in a surgical simulator. Comput Aided Surg 13(2):63–81

    PubMed  Google Scholar 

  38. Rashidi S, Fallah A, Towhidkhah F (2013) Authentication based on pole-zero models of signature velocity. J Medical Signals Sens 3(4):195–208

    Google Scholar 

  39. Rohrer B, Fasoli S, Krebs H, Hughes R, Volpe B, Frontera W, Stein J, Hogan N (2002 Sep 15) Movement smoothness changes during stroke recovery. J Neurosci 22(18):8297–8304

    CAS  PubMed  PubMed Central  Google Scholar 

  40. Cavallo F, Megali G, Sinigaglia S, Tonet O, Dario P (2006) A biomedical analysis of a surgeon’s gesture in a laparoscopic virtual scenario. Stud Health Technol Inf 119:79–84

    Google Scholar 

  41. Al T, Patel RV, Naish MD, Malthaner RA, Schlachta CM (2013) The application of force sensing to skills assessment in minimally invasive surgery. In: 2013 IEEE international conference on robotics and automation, pp 4370–4375

    Google Scholar 

  42. Jain A, Nandakumar K, Ross A (2005) Score normalization in multimodal biometric systems. Pattern Recogn 38(12):2270–2285

    Google Scholar 

  43. Deng K (1998) Omega: on-line memory-based general purpose system classifier. PhD diss. Carnegie Mellon University, Pittsburgh

    Google Scholar 

  44. Ladha L, Deepa T (2011) Feature selection methods and algorithms. IJCSE 3(5):1787–1797

    Google Scholar 

  45. Kumari B, Swarnkar T (2011) Filter versus wrapper feature subset selection in large dimensionality micro array: a review. Int J Comput 2(3):1048–1041

    Google Scholar 

  46. Kung SY (2014) Kernel Methods and Machine Learning. Cambridge University Press, Cambridge, p 34

    Google Scholar 

  47. Keller JM, Gray MR, Givens JA (1985) A fuzzy k-nearest neighbor algorithm. IEEE T SYST MAN CYB (4):580–585

  48. Lipton, Z.C., 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490

Download references

Acknowledgments

We thank all the neurosurgeons, residents, and medical students from the Montreal Neurological Institute and Hospital and other institutions who participated in this study. We would also like to thank Robert DiRaddo, Group Leader, Simulation, Life Sciences Division, National Research Council Canada at Boucherville and his team, including Denis Laroche, Valérie Pazos, Nusrat Choudhury, and Linda Pecora, for their support in the development of the scenarios used in these studies and all the members of the Simulation, Life Sciences Division, National Research Council Canada.

Funding

This work was supported by the Di Giovanni Foundation, the Montreal English School Board, the B-Strong Foundation, the Colannini Foundation, the Montreal Neurological Institute and Hospital, and the McGill Department of Orthopedics.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran

    Samaneh Siyar & Hamed Azarnoush

  2. Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, 3801 University, E2.89, Montreal, Quebec, H3A 2B4, Canada

    Samaneh Siyar, Hamed Azarnoush, Alexander Winkler-Schwartz, Vincent Bissonnette, Nirros Ponnudurai & Rolando F. Del Maestro

  3. Science and Research Branch, Islamic Azad University, Tehran, Iran

    Saeid Rashidi

Authors
  1. Samaneh Siyar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hamed Azarnoush
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saeid Rashidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexander Winkler-Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vincent Bissonnette
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nirros Ponnudurai
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rolando F. Del Maestro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hamed Azarnoush.

Ethics declarations

Disclaimer

Samaneh Siyar is a Visiting Scholar in the Neurosurgical Simulation Research and Training Centre. Dr. H. Azarnoush previously held the Postdoctoral Neuro-Oncology Fellowship from the Montreal Neurological Institute and Hospital and is a Visiting Professor in the Neurosurgical Simulation Research and Training Centre. Dr. Winkler-Schwartz holds a Robert Maudsley Fellowship from the Royal College of Physicians and Surgeons of Canada and Nirros Ponnudurai is supported by a Heffez Family Bursary. Dr. Del Maestro is the William Feindel Emeritus Professor in Neuro-Oncology at McGill University.

Conflict of interest

The authors declare that they have no conflict of interest.

Ethics approval

Research Ethics Board at McGill University approved this study and Informed consent was obtained for experimentation with human subjects.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Siyar, S., Azarnoush, H., Rashidi, S. et al. Machine learning distinguishes neurosurgical skill levels in a virtual reality tumor resection task. Med Biol Eng Comput 58, 1357–1367 (2020). https://doi.org/10.1007/s11517-020-02155-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-020-02155-3

Keywords

Search

Navigation