Skip to main content
SpringerLink
Log in

Model-Based Reinforcement Learning Variable Impedance Control for Human-Robot Collaboration

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Industry 4.0 is taking human-robot collaboration at the center of the production environment. Collaborative robots enhance productivity and flexibility while reducing human’s fatigue and the risk of injuries, exploiting advanced control methodologies. However, there is a lack of real-time model-based controllers accounting for the complex human-robot interaction dynamics. With this aim, this paper proposes a Model-Based Reinforcement Learning (MBRL) variable impedance controller to assist human operators in collaborative tasks. More in details, an ensemble of Artificial Neural Networks (ANNs) is used to learn a human-robot interaction dynamic model, capturing uncertainties. Such a learned model is kept updated during collaborative tasks execution. In addition, the learned model is used by a Model Predictive Controller (MPC) with Cross-Entropy Method (CEM). The aim of the MPC+CEM is to online optimize the stiffness and damping impedance control parameters minimizing the human effort (i.e, minimizing the human-robot interaction forces). The proposed approach has been validated through an experimental procedure. A lifting task has been considered as the reference validation application (weight of the manipulated part: 10 kg unknown to the robot controller). A KUKA LBR iiwa 14 R820 has been used as a test platform. Qualitative performance (i.e, questionnaire on perceived collaboration) have been evaluated. Achieved results have been compared with previous developed offline model-free optimized controllers and with the robot manual guidance controller. The proposed MBRL variable impedance controller shows improved human-robot collaboration. The proposed controller is capable to actively assist the human in the target task, compensating for the unknown part weight. The human-robot interaction dynamic model has been trained with a few initial experiments (30 initial experiments). In addition, the possibility to keep the learning of the human-robot interaction dynamics active allows accounting for the adaptation of human motor system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Lasi, H., Fettke, P., Kemper, H.G., et al.: Industry 4.0. Business & Information Systems Engineering 6 (4), 239–242 (2014)

    Google Scholar 

  2. Longo, F., Nicoletti, L., Padovano, A.: Smart operators in industry 4.0: a human-centered approach to enhance operators’ capabilities and competencies within the new smart factory context. Comput. Indust. Eng. 113, 144–159 (2017)

    Google Scholar 

  3. Vysocky, A., Novak, P.: Human-robot collaboration in industry. MM Sci. J. 9(2), 903–906 (2016)

    Google Scholar 

  4. Schmidtler, J., Knott, V., Hölzel, C., et al.: Human centered assistance applications for the working environment of the future. Occupational Ergonomics 12(3), 83–95 (2015)

    Google Scholar 

  5. Munoz, L.: Ergonomics in the industry 4.0: exoskeletons. J. Ergonomics 8, e176 (2018)

    Google Scholar 

  6. Nahavandi, S.: Industry 5.0–a human-centric solution. Sustainability 11(16), 4371 (2019)

    Google Scholar 

  7. Hogan, N.: Impedance control: an approach to manipulation. In: 1984 American Control Conference, pp 304–313. IEEE (1984)

  8. Roveda, L.: A user-intention based adaptive manual guidance with force-tracking capabilities applied to walk-through programming for industrial robots. In: 2018 15th International Conference On Ubiquitous Robots (UR), pp 369–376. IEEE (2018)

  9. Roveda, L., Haghshenas, S., Prini, A., et al.: Fuzzy impedance control for enhancing capabilities of humans in onerous tasks execution. In: 2018 15th International Conference on Ubiquitous Robots (UR), pp 406–411. IEEE (2018)

  10. Khan, S.G., Herrmann, G., Al Grafi, M., et al.: Compliance control and human–robot interaction: part 1–survey. Int. J. Humanoid Robotics 11(03), 1430001 (2014)

    Google Scholar 

  11. Liang, P., Yang, C., Wang, N., et al.: Implementation and test of human-operated and human-like adaptive impedance controls on baxter robot. In: Conference Towards Autonomous Robotic Systems, pp 109–119. Springer (2014)

  12. Yang, C., Zeng, C., Fang, C., et al.: A dmps-based framework for robot learning and generalization of humanlike variable impedance skills. IEEE/ASME Transactions on Mechatronics 23(3), 1193–1203 (2018)

    Google Scholar 

  13. Grafakos, S., Dimeas, F., Aspragathos, N.: Variable admittance control in pHRI using EMG-based arm muscles co-activation. In: 2016 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2016 - Conference Proceedings, pp 1900–1905 (2016)

  14. Ranatunga, I., Lewis, F., Popa, D.O., Tousif, S.M.: Adaptive admittance control for human-robot interaction using model reference design and adaptive inverse filtering. IEEE Trans. Control Syst. Technol. 25(1), 278–285 (2016)

    MATH  Google Scholar 

  15. Wai, H., Hlaing, W., Myo, A., Sin, T.: Variable admittance controller for physical human robot interaction. In: IECON 2017 - 43rd Annual Conference of the IEEE Industrial Electronics Society, pp 2929–2934 (2017)

  16. Landi, C.T., Ferraguti, F., Sabattini, L., et al.: Admittance control parameter adaptation for physical human-robot interaction. In: Proceedings - IEEE International Conference on Robotics and Automation, pp 2911–2916 (2017)

  17. Secchi, C., Stramigioli, S., Fantuzzi, C.: Position drift compensation in port-hamiltonian based telemanipulation. In: International Conference on Intelligent Robots and Systems, IROS 2006, pp 4211–4216 (2006)

  18. Gomes, W.: Role adaptive admittance controller for human-robot co-manipulation. Robotics - HRC/New-L (2018)

  19. Gopinathan, S., Mohammadi, P., Steil, J.J.: Improved human-robot interaction : a manipulability based approach improved human-robot interaction. Robotics - HRC/Control-pHRI,Thesis (2018)

  20. Li, Z., Huang, B., Ye, Z., et al.: Physical human-robot interaction of a robotic exoskeleton by admittance control. IEEE Trans. Indust. Electron. 0046(c), 1–11 (2018)

    Google Scholar 

  21. Buchli, J., Theodorou, E., Stulp, F., et al.: Variable impedance control-a reinforcement learning approach. Robotics, Science and Systems. 1. Available form: http://www.cs.washington.edu/homes/etheodor/papers/buchli-RSS2010.pdf%5Cnhttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.7254&rep=rep1&type=pdf (2010)

  22. Dimeas, F., Aspragathos, N.: Reinforcement learning of variable admittance control for human-robot co-manipulation. IEEE Int. Conf. Intell. Robots Sys. 2015-Decem, 1011–1016 (2015)

    Google Scholar 

  23. Rozo, L., Calinon, S., Caldwell, D.G., et al.: Learning physical collaborative robot behaviors from human demonstrations. IEEE Transactions on Robotics. 32(3), 513–527 (2016)

    Google Scholar 

  24. Alqaudi, B., Modares, H., Ranatunga, I., et al.: Model reference adaptive impedance control for physical human-robot interaction. Control Theory Technol. 14(1), 68–82 (2016)

    MathSciNet  MATH  Google Scholar 

  25. Modares, H., Ranatunga, I., Lewis, F.L., et al.: Optimized assistive human – robot interaction using reinforcement learning. Trans. Cybern. 46(3), 1–13 (2016)

    Google Scholar 

  26. Li, Z., Liu, J., Huang, Z., et al.: Adaptive impedance control of human–robot cooperation using reinforcement learning. IEEE Trans. Indust. Electron. 64(10), 8013–8022 (2017)

    Google Scholar 

  27. Rahimi, H.N., Howard, I., Cui, L.: Neural impedance adaption for assistive human-robot interaction. Neurocomputing 290, 50–59 (2018)

    Google Scholar 

  28. Li, C., Zhang, Z., Xia, G., et al.: Efficient force control learning system for industrial robots based on variable impedance control. Sensors 18(8), 2018 (2539)

    Google Scholar 

  29. Argall, B.D., Chernova, S., Veloso, M., et al.: A survey of robot learning from demonstration. Robotics and Autonomous Systems 57(5), 469–483 (2009)

    Google Scholar 

  30. Lee, J.: A survey of robot learning from demonstrations for human-robot collaboration. arXiv:171008789 (2017)

  31. Deisenroth, M., Rasmussen, C.E.: Pilco: a model-based and data-efficient approach to policy search. In: Proceedings of the 28th International Conference on machine learning (ICML-11), pp 465–472 (2011)

  32. Chua, K., Calandra, R., McAllister, R., et al.: Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Robotics - HRC/ML Control,Thesis. Available from: arXiv:1805.12114 (2018)

  33. Admiraal, M.A., Kusters, M.J., Gielen, S.C.: Modeling kinematics and dynamics of human arm movements. Motor Control. 8(3), 312–338 (2004)

    Google Scholar 

  34. Rosen, J., Perry, J.C., Manning, N., et al.: The human arm kinematics and dynamics during daily activities-toward a 7 dof upper limb powered exoskeleton. In: 12th International Conference on Advanced Robotics, 2005. ICAR’05. Proceedings, pp 532–539. IEEE (2005)

  35. Rahman, M.M., Ikeura, R., Mizutani, K.: Investigation of the impedance characteristic of human arm for development of robots to cooperate with humans. JSME International Journal Series C Mechanical Systems, Machine Elements and Manufacturing 45(2), 510–518 (2002)

    Google Scholar 

  36. Mobasser, F., Hashtrudi-Zaad, K.: A method for online estimation of human arm dynamics. In: 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pp 2412–2416. IEEE (2006)

  37. Howard, M., Braun, D.J., Vijayakumar, S.: Transferring human impedance behavior to heterogeneous variable impedance actuators. IEEE Transactions on Robotics 29(4), 847–862 (2013)

    Google Scholar 

  38. Medina, J.R., Endo, S., Hirche, S.: Impedance-based gaussian processes for predicting human behavior during physical interaction. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp 3055–3061. IEEE (2016)

  39. Noohi, E., žefran, M., Patton, J.L.: A model for human–human collaborative object manipulation and its application to human–robot interaction. IEEE Trans. Robotics 32(4), 880–896 (2016)

    Google Scholar 

  40. Ghadirzadeh, A., Bütepage, J., Maki, A., et al.: A sensorimotor reinforcement learning framework for physical human-robot interaction. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 2682–2688. IEEE (2016)

  41. Hamaya, M., Matsubara, T., Noda, T., et al.: Learning assistive strategies for exoskeleton robots from user-robot physical interaction. Pattern Recogn. Lett. 99, 67–76 (2017)

    Google Scholar 

  42. Peternel, L., Tsagarakis, N., Caldwell, D., et al.: Robot adaptation to human physical fatigue in human–robot co-manipulation. Autonomous Robots 42(5), 1011–1021 (2018)

    Google Scholar 

  43. Bekey, G.A., Goldberg, K.Y.: Neural Networks in Robotics, vol. 202. Springer, Berlin (2012)

    Google Scholar 

  44. Wang, F.Y., Zhang, H., Liu, D.: Adaptive dynamic programming: an introduction. IEEE Comput Intell Magazine 4(2), 39–47 (2009)

    Google Scholar 

  45. Lewis, F.L., Vrabie, D.: Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circ Sys Magazine 9(3), 32–50 (2009)

    Google Scholar 

  46. Liu, D., Wei, Q., Wang, D., et al.: Adaptive Dynamic Programming with Applications in Optimal Control. Springer, Berlin (2017)

    MATH  Google Scholar 

  47. Roveda, L., Haghshenas, S., Caimmi, M., et al.: Assisting operators in heavy industrial tasks: on the design of an optimized cooperative impedance fuzzy-controller with embedded safety rules. Frontiers in Robotics and AI 6, 75 (2019)

    Google Scholar 

  48. Siciliano, B., Villani, L.: Robot Force Control, 1st edn. Kluwer Academic Publishers, Norwell (2000)

    MATH  Google Scholar 

  49. Roveda, L., Castaman, N., Ghidoni, S., et al.: Human-robot cooperative interaction control for the installation of heavy and bulky components. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp 339–344. IEEE (2018)

  50. Nikolaidis, S., Ramakrishnan, R., Gu, K., et al.: Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In: 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp 189–196. IEEE (2015)

  51. Gläscher, J., Daw, N., Dayan, P., et al.: States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66(4), 585–595 (2010)

    Google Scholar 

  52. Polydoros, A.S., Nalpantidis, L.: Survey of model-based reinforcement learning: applications on robotics. J. Intell. Robotic Sys. 86(2), 153–173 (2017)

    Google Scholar 

  53. Nagabandi, A., Kahn, G., Fearing, R.S., et al.: Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. Robotics - HRC/ML Control, Thesis. Available from: arXiv:1708.02596 (2017)

  54. Sutton, R.S., Barto, A.G.: Reinforcement Learning: an Introduction. MIT Press, Cambridge (2018)

    MATH  Google Scholar 

  55. Campeau-Lecours, A., Otis, M.J., Gosselin, C.: Modeling of physical human–robot interaction: admittance controllers applied to intelligent assist devices with large payload. Int. J. Adv. Robotic Sys. 13(5), 1729881416658167 (2016)

    Google Scholar 

  56. Zhou, Z.H., Wu, J., Tang, W.: Ensembling neural networks: many could be better. Artificial Intelligence 137(1-2), 239–263 (2002). Available from: https://doi.org/10.1016/j.artint.2010.10.001

    MathSciNet  MATH  Google Scholar 

  57. Srivastava, R.K., Masci, J., Gomez, F., et al.: Understanding locally competitive networks. Available from: arXiv:1410.1165 (2014)

  58. Brownlee, J.: Clever algorithms: nature-inspired programming recipes. Jason Brownlee (2011)

  59. Findeisen, R., Allgöwer, F., Biegler, L.T.: Assessment and Future Directions of Nonlinear Model Predictive Control. Springer, Berlin (2007)

    MATH  Google Scholar 

  60. Wahrburg, A., Bös, J., Listmann, K.D., et al.: Motor-current-based estimation of cartesian contact forces and torques for robotic manipulators and its application to force control. IEEE Trans. Automation Sci. Eng. 15(2), 879–886 (2017)

    Google Scholar 

  61. Technical University of Munich, Germany. (2019). https://github.com/ifl-camp/iiwa_stack. Online; Accessed 24 Jul 2019

  62. Paszke, A., Gross, S., Chintala, S., et al.: Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017)

  63. Keemink, A.Q., van der Kooij, H., Stienen, A.H.: Admittance control for physical human–robot interaction. The Int. J. Robotics Res. 37(11), 1421–1444 (2018)

    Google Scholar 

  64. Mitsantisuk, C., Katsura, S., Ohishi, K.: Kalman-filter-based sensor integration of variable power assist control based on human stiffness estimation. IEEE Trans. Indust. Electron. 56(10), 3897–3905 (2009)

    Google Scholar 

  65. Warrier, R.B., Devasia, S.: Kernel-based human-dynamics inversion for precision robot motion-primitives. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 6037–6042. IEEE (2018)

  66. Rahman, S.M., Ikeura, R.: Cognition-based control and optimization algorithms for optimizing human-robot interactions in power-assisted object manipulation. J Inf Sci Eng. 32(5), 1325–1344 (2016)

    MathSciNet  Google Scholar 

  67. Norouzzadeh, S., Lorenz, T., Hirche, S.: Towards safe physical human-robot interaction: an online optimal control scheme. In: 2012 IEEE RO-MAN: the 21st IEEE International Symposium on Robot and Human Interactive Communication, pp 503–508. IEEE (2012)

  68. Avanzini, G.B., Zanchettin, A.M., Rocco, P.: Constrained model predictive control for mobile robotic manipulators. Robotica 36(1), 19–38 (2018)

    Google Scholar 

  69. Sun, K., Mou, S., Qiu, J., et al.: Adaptive fuzzy control for nontriangular structural stochastic switched nonlinear systems with full state constraints. IEEE Trans. Fuzzy Sys. 27(8), 1587–1601 (2018)

    Google Scholar 

Download references

Acknowledgments

This project has received funding from the European Union’s Horizon 2020 research and innovation programme, via an Open Call issued and executed under Project EUROBENCH (grant agreement N∘ 779963).

Author information

Authors and Affiliations

  1. Institute of Intelligent Industrial Systems and Technologies for Advanced Manufacturing (STIIMA-CNR), Milano, 20133, Italy

    Loris Roveda, Paolo Franceschi, Arash Abdi, Lorenzo Molinari Tosatti & Nicola Pedrocchi

  2. Istituto Dalle Molle di studi sull’Intelligenza Artificiale (IDSIA), Scuola Universitaria Professionale della Svizzera Italiana (SUPSI), Università della Svizzera Italiana (USI) IDSIA-SUPSI, Manno, 6928, Switzerland

    Loris Roveda

  3. School of Industrial and Information Engineering Politecnico di Milano, Milano, 20156, Italy

    Jeyhoon Maskani & Francesco Braghin

Authors
  1. Loris Roveda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeyhoon Maskani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paolo Franceschi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arash Abdi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Francesco Braghin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lorenzo Molinari Tosatti
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nicola Pedrocchi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Loris Roveda.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roveda, L., Maskani, J., Franceschi, P. et al. Model-Based Reinforcement Learning Variable Impedance Control for Human-Robot Collaboration. J Intell Robot Syst 100, 417–433 (2020). https://doi.org/10.1007/s10846-020-01183-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-020-01183-3

Keywords

Search

Navigation