Abstract
Learning from natural language is a programming-free and user friendly teaching method that allows users without programming knowledge or demonstration capabilities to instruct robots, which has great value in industry and daily life. The manipulation skills of robots are often hierarchical skills composed of low-level primitive skills, so they can be conveniently represented by behavior trees (BTs). Based on this idea, we propose NL2BT, a framework for generating behavior trees from natural language and controlling robots to complete hierarchical tasks in real time. The framework consists of two language processing stages, an initial behavior tree library composed of primitive skill subtrees, and a BT-Generation algorithm. To validate the effectiveness of NL2BT, we use it to build a Chinese natural language system for instructing robots in performing 3C assembly tasks, which is a significant application of Industry 4.0. We also discuss the positive impact of real-time teaching, visual student models, and the synonymous skill module in the framework. In addition to the demonstrated application, NL2BT can be easily migrated to other languages and hierarchical task learning scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ahn, M., et al.: Do as i can, not as i say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022)
Cao, Y., Lee, C.: Robot behavior-tree-based task generation with large language models. arXiv preprint arXiv:2302.12927 (2023)
Chai, J.Y., Gao, Q., She, L., Yang, S., Saba-Sadiya, S., Xu, G.: Language to action: towards interactive task learning with physical agents. In: IJCAI, pp. 2–9 (2018)
Chen, Q., Zhuo, Z., Wang, W.: Bert for joint intent classification and slot filling. arXiv preprint arXiv:1902.10909 (2019)
Colledanchise, M., Natale, L.: On the implementation of behavior trees in robotics. IEEE Rob. Autom. Lett. 6(3), 5929–5936 (2021)
Colledanchise, M., Ögren, P.: How behavior trees modularize hybrid control systems and generalize sequential behavior compositions, the subsumption architecture, and decision trees. IEEE Trans. Rob. 33(2), 372–389 (2016)
Colledanchise, M., Ögren, P.: Behavior Trees in Robotics and AI: An Introduction. CRC Press, Boca Raton (2018)
De Rossi, G., et al.: Cognitive robotic architecture for semi-autonomous execution of manipulation tasks in a surgical environment. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7827–7833. IEEE (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Faconti, D.: Groot (2018). https://github.com/BehaviorTree/Groot
French, K., Wu, S., Pan, T., Zhou, Z., Jenkins, O.C.: Learning behavior trees from demonstration. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 7791–7797. IEEE (2019)
Hinkin, T.R.: A brief tutorial on the development of measures for use in survey questionnaires. Organ. Res. Methods 1(1), 104–121 (1998)
IFLYTEK: Iflytek open platform (2021). https://www.xfyun.cn
Jawahar, G., Sagot, B., Seddah, D.: What does BERT learn about the structure of language? In: ACL 2019–57th Annual Meeting of the Association for Computational Linguistics (2019)
Laird, J.E., et al.: Interactive task learning. IEEE Intell. Syst. 32(4), 6–21 (2017)
Li, T.J.J., Azaria, A., Myers, B.A.: SUGILITE: creating multimodal smartphone automation by demonstration. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 6038–6049 (2017)
Li, T.J.J., Mitchell, T., Myers, B.: Interactive task learning from GUI-grounded natural language instructions and demonstrations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 215–223 (2020)
Liang, P.: Learning executable semantic parsers for natural language understanding. Commun. ACM 59(9), 68–76 (2016)
Liu, C., et al.: Jointly learning grounded task structures from language instruction and visual demonstration. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1482–1492 (2016)
Lynch, C., et al.: Interactive language: talking to robots in real time. arXiv preprint arXiv:2210.06407 (2022)
Mees, O., Hermann, L., Rosete-Beas, E., Burgard, W.: Calvin: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Rob. Autom. Lett. 7(3), 7327–7334 (2022)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol. 26 (2013)
Petit, M., Demiris, Y.: Hierarchical action learning by instruction through interactive grounding of body parts and proto-actions. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3375–3382. IEEE (2016)
She, L., Yang, S., Cheng, Y., Jia, Y., Chai, J., Xi, N.: Back to the blocks world: learning new actions through situated human-robot dialogue. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 89–97 (2014)
Stepputtis, S., Campbell, J., Phielipp, M., Lee, S., Baral, C., Ben Amor, H.: Language-conditioned imitation learning for robot manipulation tasks. Adv. Neural. Inf. Process. Syst. 33, 13139–13150 (2020)
Vemprala, S., Bonatti, R., Bucker, A., Kapoor, A.: Chatgpt for robotics: design principles and model abilities. Microsoft Auton. Syst. Robot. Res 2, 20 (2023)
Wang, H., Hu, Y.: Synonyms (2017). https://github.com/chatopera/Synonyms
Welschehold, T., Abdo, N., Dornhege, C., Burgard, W.: Combined task and action learning from human demonstrations for mobile manipulation applications. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4317–4324. IEEE (2019)
Acknowledgments
This research was supported by the National Key Research & Development Program of China (No.2018AAA0102902). We would also like to thank the Institute for Artificial Intelligence, Tsinghua University, for providing equipment and data support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, K., Zhao, Y., Dai, S., Yang, M., He, Y., Zhang, N. (2024). Learning Hierarchical Robot Skills Represented by Behavior Trees from Natural Language. In: Sellami, M., Vidal, ME., van Dongen, B., Gaaloul, W., Panetto, H. (eds) Cooperative Information Systems. CoopIS 2023. Lecture Notes in Computer Science, vol 14353. Springer, Cham. https://doi.org/10.1007/978-3-031-46846-9_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-46846-9_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46845-2
Online ISBN: 978-3-031-46846-9
eBook Packages: Computer ScienceComputer Science (R0)