Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Versatile modular neural locomotion control with fast learning

Nature Machine Intelligence volume 4, pages 169–179 (2022)Cite this article

Subjects

A preprint version of the article is available at arXiv.

Abstract

Legged robots have significant potential to operate in unstructured environments. The design of locomotion control is, however, still challenging. Currently, controllers must be manually designed for specific robots and tasks, or automatically designed via machine learning methods that require long training times and yield large opaque controllers. Drawing inspiration from animal locomotion, we propose a simple yet versatile modular neural control structure with fast learning. The key advantages of our approach are that behaviour-specific control modules can be added incrementally to obtain increasingly complex emergent locomotion behaviours, and that neural connections can be quickly and automatically learned. In a series of experiments, we show how eight modules can be quickly learned and added to a base control module to obtain emergent adaptive behaviours allowing a hexapod robot to navigate in complex environments. We also show that modules can be added and removed during operation without affecting the functionality of the remaining controller. Finally, the controller is successfully demonstrated on a physical robot. Taken together, our study reveals a significant step towards fast automatic design of versatile neural locomotion control.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of the versatile modular neural locomotion controller.
Fig. 2: Results of learning the base and obstacle reflex controllers.
Fig. 3: Results of learning the body posture and directional locomotion controllers.
Fig. 4: Results for using the learned primitive control modules.
Fig. 5: The advanced behaviours and the results for using all eight control modules.
Fig. 6: Detailed overview of the versatile modular neural locomotion controller.

Similar content being viewed by others

Data availability

All data from running the experiments as well as the learned weight sets can be accessed at https://github.com/MathiasThor/CPG-RBFN-framework/tree/main/data50.

Code availability

The source code for running the controller in simulation can be accessed at https://github.com/MathiasThor/CPG-RBFN-framework50.

References

  1. Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V. & Hutter, M. Learning quadrupedal locomotion over challenging terrain. Sci. Robot. 5, eabc5986 (2020).

    Article  Google Scholar 

  2. Winkler, A. W. Optimization-Based Motion Planning for Legged Robots. PhD thesis, ETH Zurich (2018).

  3. Machado, J. A. T. & Silva, M. F. An overview of legged robots. In Proc. MME International Symposium on Mathematical Methods in Engineering (2006).

  4. Thor, M., Kulvicius, T. & Manoonpong, P. Generic neural locomotion control framework for legged robots. In IEEE Transactions on Neural Networks and Learning Systems Vol. 32, 4013–4025 (IEEE, 2021).

  5. Cully, A., Clune, J., Tarapore, D. & Mouret, J.-B. Robots that can adapt like animals. Nature 521, 503–507 (2015).

    Article  Google Scholar 

  6. Silva, M. F. & Machado, J. A. T. A literature review on the optimization of legged robots. J. Vib. Control 18, 1753–1767 (2012).

    Article  MathSciNet  Google Scholar 

  7. Hwangbo, J. et al. Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4, eaau5872 (2019).

    Article  Google Scholar 

  8. Clune, J., Stanley, K. O., Pennock, R. T. & Ofria, C. On the performance of indirect encoding across the continuum of regularity. IEEE Trans. Evol. Comput. 15, 346–367 (2011).

    Article  Google Scholar 

  9. Schilling, M., Konen, K., Ohl, F. W. & Korthals, T. Decentralized deep reinforcement learning for a distributed and adaptive locomotion controller of a hexapod robot. In Proc. IEEE Int. Conf. Intell. Robots Syst. 5335–5342 (IEEE, 2020).

  10. Yang, C., Yuan, K., Zhu, Q., Yu, W. & Li, Z. Multi-expert learning of adaptive legged locomotion. Sci. Robot. 5, eabb2174 (2020).

    Article  Google Scholar 

  11. Schilling, M., Konen, K. & Korthals, T. Modular deep reinforcement learning for emergent locomotion on a six-legged robot. In Proc. 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob) 946–953 (IEEE, 2020).

  12. Oliveira, M., Costa, L., Rocha, A., Santos, C. & Ferreira, M. Multiobjective optimization of a quadruped robot locomotion using a genetic algorithm. In Proc. Soft Computing in Industrial Applications 427–436 (Springer, 2011).

  13. Kullander, K. et al. Role of epha4 and ephrinb3 in local neuronal circuits that control walking. Science 299, 1889–1892 (2003).

    Article  Google Scholar 

  14. Biewener, A. A. Animal Locomotion 1st edn (Oxford Univ. Press, 2003).

  15. Büschges, A. Sensory control and organization of neural networks mediating coordination of multisegmental organs for locomotion. J. Neurophysiol. 93, 1127–1135 (2005).

    Article  Google Scholar 

  16. Azayev, T. & Zimmerman, K. Blind hexapod locomotion in complex terrain with gait adaptation using deep reinforcement learning and classification. J. Intell. Robot. Syst. 99, 659–671 (2020).

    Article  Google Scholar 

  17. Delcomyn, F. Walking robots and the central and peripheral control of locomotion in insects. Auton. Robot. 7, 259–270 (1999).

    Article  Google Scholar 

  18. Samek, W. & Müller, K.-R. Towards Explainable Artificial Intelligence 5–22 (Springer, 2019).

  19. Thor, M., Larsen, J. C. & Manoonpong, P. MORF—modular robot framework. In Proc. 2nd International Youth Conference of Bionic Engineering 21–23 (Frontiers, 2018).

  20. Yadav, R. N., Kalra, P. K. & John, J. On the use of multiplicative neuron in feedforward neural networks. Int. J. Simul. Model. 26, 331–336 (2006).

    Article  Google Scholar 

  21. Schmitt, M. On the complexity of computing and learning with multiplicative neural networks. Neural Comput. 14, 241–301 (2002).

    Article  Google Scholar 

  22. Koch, C. & Poggio, T. in Single Neuron Computation, Neural Networks: Foundations to Applications (eds McKenna, T. et al.) Ch. 12 (Academic Press, 1992).

  23. Hashlamon, I. & Erbatur, K. Joint sensor fault detection and recovery based on virtual sensor for walking legged robots. In Proc. IEEE 23rd International Symposium on Industrial Electronics (ISIE) 1210–1214 (IEEE, 2014).

  24. Perla, R., Mukhopadhyay, S. & Samanta, A. N. Sensor fault detection and isolation using artificial neural networks. In Proc. IEEE Region 10 Conference TENCON Vol. 4, 676–679 (IEEE, 2004).

  25. Christensen, A. L., O’Grady, R., Birattari, M. & Dorigo, M. Fault detection in autonomous robots based on fault injection and learning. Auton. Robot. 24, 49–67 (2008).

    Article  Google Scholar 

  26. Patle, B., Babu L, G., Pandey, A., Parhi, D. & Jagadeesh, A. A review: on path planning strategies for navigation of mobile robot. Def. Technol. 15, 582–606 (2019).

    Google Scholar 

  27. Goldschmidt, D., Manoonpong, P. & Dasgupta, S. A neurocomputational model of goal-directed navigation in insect-inspired artificial agents. Front. Neurorobot. 11, 20 (2017).

    Article  Google Scholar 

  28. Brooks, R. A robust layered control system for a mobile robot. IEEE Robot. Autom. Lett. 2, 14–23 (1986).

    Article  Google Scholar 

  29. Manoonpong, P., Geng, T., Kulvicius, T., Porr, B. & Wörgötter, F. Adaptive, fast walking in a biped robot under neuronal control and learning. PLoS Comput. Biol. 3, e134 (2007).

    Article  Google Scholar 

  30. Jakobi, N. Evolutionary robotics and the radical envelope-of-noise hypothesis. Adapt. Behav. 6, 325–368 (1997).

    Article  Google Scholar 

  31. Demin, V. & Nekhaev, D. Recurrent spiking neural network learning based on a competitive maximization of neuronal activity. Front. Neurorobot. 12, 79 (2018).

    Google Scholar 

  32. Pfeiffer, M. & Pfeil, T. Deep learning with spiking neurons: opportunities and challenges. Front. Neurorobot. 12, 774 (2018).

    Article  Google Scholar 

  33. Strohmer, B., Manoonpong, P. & Larsen, L. B. Flexible spiking CPGs for online manipulation during hexapod walking. Front. Neurorobot. 14, 41 (2020).

    Article  Google Scholar 

  34. Gutierrez-Galan, D., Dominguez-Morales, J. P., Perez-Peña, F., Jimenez-Fernandez, A. & Linares-Barranco, A. Neuropod: a real-time neuromorphic spiking CPG applied to robotics. Neurocomputing 381, 10–19 (2020).

    Article  Google Scholar 

  35. Espinal, A. et al. Design of spiking central pattern generators for multiple locomotion gaits in hexapod robots by christiansen grammar evolution. Front. Neurorobot. 10, 6 (2016).

    Article  Google Scholar 

  36. Donati, E., Indiveri, G. & Stefanini, C. A novel spiking CPG-based implementation system to control a lamprey robot. In Proc. IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob) 1364–1364 (IEEE, 2016).

  37. Polykretis, I., Tang, G. & Michmizos, K. P. An astrocyte-modulated neuromorphic central pattern generator for hexapod robot locomotion on intel’s loihi. In Proc. ICONS: International Conference on Neuromorphic Systems 1–9 (Association for Computing Machinery, 2020).

  38. Aoi, S., Manoonpong, P., Ambe, Y., Matsuno, F. & Wörgötter, F. Adaptive control strategies for interlimb coordination in legged robots: a review. Front. Neurorobot. 11, 39 (2017).

    Article  Google Scholar 

  39. Nachstedt, T., Tetzlaff, C. & Manoonpong, P. Fast dynamical coupling enhances frequency adaptation of oscillators for robotic locomotion control. Front. Neurorobot. 11, 14 (2017).

    Article  Google Scholar 

  40. Pasemann, F., Hild, M. & Zahedi, K. SO(2)-networks as neural oscillators. In Proc. Computational Methods in Neural Modeling 144–151 (Springer, 2003).

  41. Pasemann, F. & Stollenwerk, N. Attractor switching by neural control of chaotic neurodynamics. Netw. Comput. Neural Syst. 9, 549–561 (1998).

    Article  Google Scholar 

  42. Pasemann, F. Complex dynamics and the structure of small neural networks. Netw. Comput. Neural Syst. 13, 195–216 (2002).

    Article  Google Scholar 

  43. Steingrube, S., Timme, M., Wörgötter, F. & Manoonpong, P. Self-organized adaptation of a simple neural circuit enables complex robot behaviour. Nat. Phys. 6, 224–230 (2010).

    Article  Google Scholar 

  44. Thor, M. & Manoonpong, P. Error-based learning mechanism for fast online adaptation in robot motor control. IEEE Trans. Neural Netw. Learn. Syst. 31, 2042–2051 (2019).

    Article  Google Scholar 

  45. Manoonpong, P., Parlitz, U. & Wörgötter, F. Neural control and adaptive neural forward models for insect-like, energy-efficient, and adaptable locomotion of walking machines. Front. Neural Circuits 7, 12 (2013).

    Article  Google Scholar 

  46. Broomhead, D. & Lowe, D. Radial basis functions, multi-variable functional interpolation and adaptive networks. R. Signals Radar Estab. 4148, 801–849 (1988).

    Google Scholar 

  47. Stulp, F. & Sigaud, O. Policy improvement: between black-box optimization and episodic reinforcement learning. In Proc. Journées Francophones Planification, Décision, et Apprentissage Pour la Conduite de Systémes hal-00922133 (2013).

  48. Theodorou, E., Buchli, J. & Schaal, S. A generalized path integral control approach to reinforcement learning. J. Mach. Learn. Res. 11, 3137–3181 (2010).

    MathSciNet  MATH  Google Scholar 

  49. Chatterjee, S. et al. Learning and chaining of motor primitives for goal-directed locomotion of a snake-like robot with screw-drive units. Int. J. Adv. Robot. Syst. 12, 176 (2015).

    Google Scholar 

  50. Thor, M. Mathiasthor/cpg-rbfn-framework: first release. Zenodo https://doi.org/10.5281/zenodo.5524494 (2021).

Download references

Acknowledgements

We thank the members of the SDU Biorobotics group for their technical support and helpful discussions. We also thank A. L. Christensen for fruitful discussions and detailed feedback. This work was supported in part by the Horizon 2020 Framework Programme (FETPROACT-01-2016 FET Proactive: Emerging Themes and Communities) under grant 732266 (Plan4Act) (P.M., project workpackage principal investigator) and in part by a start-up grant on Bio-inspired Robotics from the Vidyasirimedhi Institute of Science and Technology (VISTEC) (P.M., project principal investigator).

Author information

Authors and Affiliations

  1. Embodied AI and Neurorobotics Laboratory, SDU Biorobotics, The Mærsk Mc-Kinney Møller Institute, The University of Southern Denmark, Odense, Denmark

    Mathias Thor & Poramate Manoonpong

  2. Bio-Inspired Robotics and Neural Engineering Laboratory, School of Information Science and Technology, Vidyasirimedhi Institute of Science and Technology, Rayong, Thailand

    Poramate Manoonpong

Authors
  1. Mathias Thor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Poramate Manoonpong
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.T. contributed to the modular neural control structure, implementation, experiments, data analysis and the manuscript. P.M. contributed to the embodied neural control structure, data analysis and the manuscript.

Corresponding author

Correspondence to Mathias Thor.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Luca Patané, Fernando Perez-Peña and Ren Qinyuan for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Sections 1–6, Figs. 1–9, Tables 1 and 2, and Algorithm 1.

Supplementary Video 1

Learning the base controller.

Supplementary Video 2

Learning the obstacle reflex controller.

Supplementary Video 3

Learning the body posture controller.

Supplementary Video 4

Learning the directional locomotion controller.

Supplementary Video 5

Deploying primitive controllers in simulation.

Supplementary Video 6

Deploying primitive controllers on a physical robot.

Supplementary Video 7

Deploying primitive and advanced controllers in simulation.

Supplementary Video 8

Controller generalization and limitations.

Supplementary Video 9

Disabling primitive control modules.

Supplementary Video 10

Disabling advanced control modules.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thor, M., Manoonpong, P. Versatile modular neural locomotion control with fast learning. Nat Mach Intell 4, 169–179 (2022). https://doi.org/10.1038/s42256-022-00444-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00444-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics