Skip to main content
SpringerLink
Log in

Skill-based human–robot cooperation in tele-operated path tracking

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This work proposes a shared-control tele-operation framework that adapts its cooperative properties to the estimated skill level of the operator. It is hypothesized that different aspects of an operator’s performance in executing a tele-operated path tracking task can be assessed through conventional machine learning methods using motion-based and task-related features. To identify performance measures that capture motor skills linked to the studied task, an experiment is conducted where users new to tele-operation, practice towards motor skill proficiency in 7 training sessions. A set of classifiers are then learned from the acquired data and selected features, which can generate a skill profile that comprises estimations of user’s various competences. Skill profiles are exploited to modify the behavior of the assistive robotic system accordingly with the objective of enhancing user experience by preventing unnecessary restriction for skilled users. A second experiment is implemented in which novice and expert users execute the path tracking on different pathways while being assisted by the robot according to their estimated skill profiles. Results validate the skill estimation method and hint at feasibility of shared-control customization in tele-operated path tracking.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Abbott, J. J., Marayong, P., & Okamura A. M. (2007). Haptic virtual fixtures for robot-assisted manipulation. Robotics Research. https://doi.org/10.1007/978-3-540-48113-3_5

  • Ahmidi, N., et al. (2017). A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Transactions on Biomedical Engineering, 9294(c), 1–16.

    Google Scholar 

  • Anderson, S. J., Karumanchi, S. B., & Iagnemma, K. (2012). Constraint-based planning and control for safe, semi-autonomous operation of vehicles. In 2012 IEEE intelligent vehicles symposium (pp. 383–88). IEEE.

  • Bauzano, E., Estebanez, B., & Muñoz, V. F. (2014). Collaborative human-robot system for HALS suture procedures. IEEE Systems Journal, 10, 1–10.

    Google Scholar 

  • Beretta, E., et al. (2015). Enhanced torque-based impedance control to assist brain targeting during open-skull neurosurgery: A feasibility study. International Journal of Medical Robotics and Computer Assisted Surgery, 12(3), 326–41. https://doi.org/10.1002/rcs.1690.

    Article  Google Scholar 

  • Berthet-Rayne, P., Power, M., King, H., & Yang Fellow, G.-Z. (2016). Hubot?: A three state human–robot collaborative framework for bimanual surgical tasks based on learned models. In International conference on robotics and automation (pp. 715–722).

  • Bowyer, S. A., Davies, B. L., & Baena, F. R. Y. (2014). Active constraints/virtual fixtures: A survey. IEEE Transactions on Robotics, 30(1), 138–57.

    Article  Google Scholar 

  • Bowyer, S. A., & Baena, F. R. Y. (2015). Dissipative control for physical human–robot interaction. IEEE Transactions on Robotics, 31(6), 1281–1293.

    Article  Google Scholar 

  • Bruyninckx, H. (2001). Open robot control software: The OROCOS project. In Proceedings 2001 ICRA. IEEE international conference on robotics and automation (Cat. No. 01CH37164) (pp. 2523–2528). IEEE.

  • Chmarra, M. K., et al. (2010). Objective classification of residents based on their psychomotor laparoscopic skills. Surgical Endoscopy, 24(5), 1031–1039.

    Article  Google Scholar 

  • Cotin, S., et al. (2002). Metrics for laparoscopic skills trainers: The weakest link!. Medical Image Computing and Computer-Assisted Intervention—MICCAI, 2488, 35–43.

    MATH  Google Scholar 

  • Dragan, A. D., & Srinivasa, S. S. (2013). A policy-blending formalism for shared control. The International Journal of Robotics Research, 32(7), 790–805.

    Article  Google Scholar 

  • Enayati, N., Costa, E. C. A., Ferrigno, G., & De Momi, E. (2016a). A dynamic non-energy-storing guidance constraint with motion redirection for robot-assisted surgery. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Daejeon, Korea: IEEE.

  • Enayati, N., De Momi, E., & Ferrigno, G. (2016b). Haptics in robot-assisted surgery: Challenges and benefits. IEEE Reviews in Biomedical Engineering, 9, 49–65.

  • Enayati, N., Ferrigno, G., & De Momi, E. (2017). Performance metrics for guidance active constraints in surgical robotics. The International Journal of Medical Robotics and Computer Assisted Surgery. https://doi.org/10.1002/rcs.1873.

  • Enayati, N., & Najafi, F. (2011). Design and manufacturing of a tele-operative rescue robot with a novel track arrangement. Industrial Robot: An International Journal, 38(5), 476–85.

    Article  Google Scholar 

  • Fard, M. J., Ameri, S., Chinnam, R. B., & Ellis, R. D. (2017). Soft boundary approach for unsupervised gesture segmentation in robotic-assisted surgery. IEEE Robotics and Automation Letters, 2(1), 171–178.

    Article  Google Scholar 

  • Finan, M. A., Clark, M. E., & Rocconi, R. P. (2010). A novel method for training residents in robotic hysterectomy. Journal of Robotic Surgery, 4(1), 33–39.

    Article  Google Scholar 

  • Franchi, A., et al. (2012). Shared control: Balancing autonomy and human assistance with a group of quadrotor UAVs. IEEE Robotics & Automation Magazine, 19(3), 57–68.

    Article  Google Scholar 

  • Gao, M., Oberlander, J., Schamm, T., & Zollner, J. M. (2014). Contextual task-aware shared autonomy for assistive mobile robot teleoperation. In 2014 IEEE/RSJ international conference on intelligent robots and systems (pp. 3311–3318). IEEE.

  • Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F. J., & Marín-Jiménez, M. J. (2014). Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6), 2280–2292.

    Article  Google Scholar 

  • Ghez, C., Krakauer, J. W., & Ghilardi, M.-F. (1999). Independent learning of internal models for kinematic and dynamic control of reaching. Nature Neuroscience, 2(11), 1026–1031.

    Article  Google Scholar 

  • Gomes, P. (2011). Surgical robotics: Reviewing the past, analysing the present, imagining the future. Robotics and Computer-Integrated Manufacturing, 27(2), 261–266.

    Article  Google Scholar 

  • Hajshirmohammadi, I., & Payandeh, S. (2007). Fuzzy set theory for performance evaluation in a surgical simulator. Presence: Teleoperators and Virtual Environments, 16(6), 603–622.

    Article  Google Scholar 

  • Van Hove, P. D., et al. (2010). Objective assessment of technical surgical skills. British Journal of Surgery, 97(7), 972–987.

    Article  Google Scholar 

  • Kim, S., et al. (2010). Lack of a discriminatory function for endoscopy skills on a computer-based simulator. Surgical Endoscopy, 24(12), 3008–3015. https://doi.org/10.1007/s00464-010-1077-z.

    Article  Google Scholar 

  • Kranzfelder, M., et al. (2013). Toward increased autonomy in the surgical OR: needs, requests, and expectations. Surgical Endoscopy and Other Interventional Techniques, 27(5), 1681–1688.

    Article  Google Scholar 

  • Lawrence, D. A. (1993). Stability and transparency in bilateral teleoperation. IEEE Transactions on Robotics and Automation, 9(5), 624–637.

    Article  Google Scholar 

  • Lin, H. C., Shafran, I., Yuh, D., & Hager, G. D. (2006). Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions. Computer Aided Surgery, 11(5), 220–230.

    Article  Google Scholar 

  • Liu, Y.-T., Mayer-Kress, G., & Newell, K. M. (2006). Qualitative and quantitative change in the dynamics of motor learning. Journal of Experimental Psychology: Human Perception and Performance, 32(2), 380–393.

    Google Scholar 

  • Marchal-Crespo, L., & Reinkensmeyer, D. J. (2008). Haptic guidance can enhance motor learning of a steering task. Journal of Motor Behavior, 40(6), 545–556.

    Article  Google Scholar 

  • Megali, G., Sinigaglia, S., Tonet, O., & Dario, P. (2006). Modelling and evaluation of surgical performance using hidden markov models. IEEE Transactions on Biomedical Engineering, 53(10), 1911–1919.

    Article  Google Scholar 

  • Milliken, L., & Hollinger, G. A. (2016). Modeling user expertise for choosing levels of shared autonomy. In Planning for human–robot interaction shared autonomy and collaborative robotics workshop, robotics: Science and systems conference.

  • Nichols, K. A. et al. (2015). Models of human-centered automation in a debridement task. In IEEE international conference on intelligent robots and systems 2015–December (pp. 5784–5789).

  • Padoy, N., Hager, G.D. (2011) Human-machine collaborative surgery using learned models. In Proceedings of the IEEE international conference on robotics and automation (pp. 5285–5292).

  • Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825–2830.

    MathSciNet  MATH  Google Scholar 

  • Penin, L. F., Matsumoto, K., & Wakabayashi, S. (2000). Force reflection for time-delayed teleoperation of space robots. In Proceedings 2000 ICRA. Millennium conference. IEEE International conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065) (pp. 3120–3125). IEEE.

  • Rafii-Tari, H., et al. (2013). Learning-based modeling of endovascular navigation for collaborative robotic catheterization. In International conference on medical image computing and computer-assisted intervention (pp. 369–377). Berlin, Heidelberg: Springer.

  • Reiley, C. E., & Hager, G. D. (2009). Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. In International conference on medical image computing and computer-assisted intervention (pp. 435–442). Berlin, Heidelberg: Springer.

  • Richards, C., et al. (2000). Skills evaluation in minimally invasive surgery using force/torque signatures. Surgical Endoscopy, 14(9), 791–798.

    Article  Google Scholar 

  • Rosen, J., et al. (2006). Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Transactions on Biomedical Engineering, 53(3), 399–413.

    Article  Google Scholar 

  • Sainburg, R. L., & Wang, J. (2002). Interlimb transfer of visuomotor rotations: Independence of direction and final position information. Experimental Brain Research, 145(4), 437–447.

    Article  Google Scholar 

  • Schmidt, R. A. (1991). Frequent augmented feedback can degrade learning: Evidence and interpretations. In: Tutorials in motor neuroscience (pp. 59–75). Dordrecht, Netherlands: Springer.

  • Schreuder, H. W. R., et al. (2012). Training and learning robotic surgery, time for a more structured approach: A systematic review. BJOG: An International Journal of Obstetrics and Gynaecology, 119(2), 137–149.

    Article  Google Scholar 

  • Shadmehr, R., Smith, M. A., & Krakauer, J. W. (2010). Error correction, sensory prediction, and adaptation in motor control. Annual Review of Neuroscience, 33(1), 89–108.

    Article  Google Scholar 

  • Shmuelof, L., Krakauer, J. W., & Mazzoni, P. (2012). How is a motor skill learned? Change and invariance at the levels of task success and trajectory control. Journal of Neurophysiology, 108(2), 578–594.

    Article  Google Scholar 

  • Sigrist, R., Rauter, G., Riener, R., & Wolf, P. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychonomic Bulletin & Review, 20(1), 21–53.

    Article  Google Scholar 

  • Takayama, L., Marder-Eppstein, E., Harris, H., & Beer, J. M. (2011). Assisted driving of a mobile remote presence system: System design and controlled user evaluation. In 2011 IEEE international conference on robotics and automation (pp. 1883–1889). IEEE.

  • Trejos, A. L., et al. (2013). The application of force sensing to skills assessment in minimally invasive surgery. In 2013 IEEE international conference on robotics and automation (pp. 4370–4375). IEEE.

Download references

Acknowledgements

This work has received funding from the European Union’s Horizon 2020 research and innovation program under Grant agreement no. H2020-ICT-26-2016-1-732515 (SMARTsurg project).

Author information

Authors and Affiliations

  1. Department of Electronics, Information and Bioengineering, Politecnico di Milano, 20133, Milan, Italy

    Nima Enayati, Giancarlo Ferrigno & Elena De Momi

Authors
  1. Nima Enayati
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giancarlo Ferrigno
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elena De Momi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nima Enayati.

Additional information

This is one of the several papers published in Autonomous Robots comprising the Special Issue on Learning for Human-Robot Collaboration.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Enayati, N., Ferrigno, G. & De Momi, E. Skill-based human–robot cooperation in tele-operated path tracking. Auton Robot 42, 997–1009 (2018). https://doi.org/10.1007/s10514-017-9675-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-017-9675-4

Keywords

Search

Navigation