Abstract
This paper introduces a benchmark framework to evaluate the performance of reaching motion generation approaches that learn from demonstrated examples. The system implements ten different performance measures for typical generalization tasks in robotics using open source MATLAB software. Systematic comparisons are based on a default training data set of human motions, which specify the respective ground truth. In technical terms, an evaluated motion generation method needs to compute velocities, given a state provided by the simulation system. This however is agnostic to how this is done by the method or how the methods learns from the provided demonstrations. The framework focuses on robustness, which is tested statistically by sampling from a set of perturbation scenarios. These perturbations interfere with motion generation and challenge its generalization ability. The benchmark thus helps to identify the strengths and weaknesses of competing approaches, while allowing the user the opportunity to configure the weightings between different measures.
References
[1] B. Adams, C.L. Breazeal, R. Brooks, and B. Scassellati. Humanoid robots: a new kind of tool. IEEE Intell. Syst. Control lett., 15(4):25–31, 2000. Search in Google Scholar
[2] Alin Albu-Schäffer, Sami Haddadin, Ch Ott, Andreas Stemmer, ThomasWimböck, and Gerd Hirzinger. The dlr lightweight robot: design and control concepts for robots in human environments. Industrial Robot: An International Journal, 34(5):376–385, 2007. Search in Google Scholar
[3] E. Oztop, D. Franklin, T. Chaminade, and G. Cheng. Human– humanoid interaction: Is a humanoid robot perceived as a human? Humanoid Robotics, 2(04):537–559, 2005. 10.1142/S0219843605000582Search in Google Scholar
[4] T. Chaminade, D. Franklin, E. Oztop, and G. Cheng. Motor interference between humans and humanoid robots: Effect of biological and artificial motion. In Proc. of Int. Conf. on Development and Learning, pages 96–101. IEEE, 2005. Search in Google Scholar
[5] Aleš Ude, Christopher G. Atkeson, and Marcia Riley. Programming full-body movements for humanoid robots by observation. Robotics and Autonomous Systems, 47(2-3):93–108, 2004. 10.1016/j.robot.2004.03.004Search in Google Scholar
[6] Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. Robot Programming by Demonstration, chapter 59, pages 1371–1394. Springer, 2008. 10.1007/978-3-540-30301-5_60Search in Google Scholar
[7] Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483, 2009. 10.1016/j.robot.2008.10.024Search in Google Scholar
[8] D. Bailey and J. Barton. The NAS kernel benchmark program. National Aeronautics and Space Administration, Ames Research Center, 1985. Search in Google Scholar
[9] K. Gaj, E. Homsirikamol, and M. Rogawski. Fair and comprehensive methodology for comparing hardware performance of fourteen round two SHA-3 candidates using FPGAs. In Cryptographic Hardware and Embedded Systems (CHES), pages 264– 278. Springer, 2010. 10.1007/978-3-642-15031-9_18Search in Google Scholar
[10] E. Dolan and J. More. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2):201– 213, 2002. 10.1007/s101070100263Search in Google Scholar
[11] A. Paraschos, G. Neumann, and J. Peters. A probabilistic approach to robot trajectory generation. In Proc. of Int. Conf. on Humanoid Robots (Humanoids), pages 477–483, 2013. 10.1109/HUMANOIDS.2013.7030017Search in Google Scholar
[12] S. Calinon, T. Alizadeh, and D. G. Caldwell. On improving the extrapolation capability of task-parameterized movement models. In Proc. of Int. Conf on Intelligent Robots and Systems (IROS), pages 610–616. IEEE, 2013. 10.1109/IROS.2013.6696414Search in Google Scholar
[13] S.M. Khansari-Zadeh and A. Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. Transactions on Robotics, 27(5):943–957, 2011. 10.1109/TRO.2011.2159412Search in Google Scholar
[14] A. Lemme, Neumann K., F. R. Reinhart, and J. J. Steil. Neural learning of vector fields for encoding stable dynamical systems. Neurocomputing, 141(0):3–14, 2014. 10.1016/j.neucom.2014.02.012Search in Google Scholar
[15] K. Neumann, Lemme A., and J. J. Steil. Neural learning of stable dynamical systems based on data-driven Lyapunov candidates. In Proc. of Int. Conf. Intelligent Robots and Systems (IROS), pages 1216–1222. IEEE, 2013. 10.1109/IROS.2013.6696505Search in Google Scholar
[16] A. Ude, A. Gams, T. Asfour, and J. Morimoto. Task-specific generalization of discrete and periodic dynamic movement primitives. Transactions on Robotics, 26(5):800–815, 2010. 10.1109/TRO.2010.2065430Search in Google Scholar
[17] H. Hoffmann, P. Pastor, Dae-Hyung Park, and S. Schaal. Biologically-inspired dynamical systems for movement generation: Automatic real-time goal adaptation and obstacle avoidance. In Proc. of Int. Conf. on Robotics and Automation (ICRA), pages 2587–2592, 2009. 10.1109/ROBOT.2009.5152423Search in Google Scholar
[18] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In Proc. of Int. Conf. on Robotics and Automation (ICRA), pages 763–768. IEEE, 2009. 10.1109/ROBOT.2009.5152385Search in Google Scholar
[19] S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert. Learning movement primitives. In Robotics Research, pages 561–572. Springer, 2005. 10.1007/11008941_60Search in Google Scholar
[20] F. Lacquaniti, C. Terzuolo, and P. Viviani. The law relating kinematic and figural aspects of drawing movements. Acta Psychologica, 54:115–130, 1983. 10.1016/0001-6918(83)90027-6Search in Google Scholar
[21] P. Viviani and M. Cenzato. Segmentation and coupling in complex movements. Journal Experimental Psychology Humam Perception and Performence, 11(6):828–845, 1985. 10.1037/0096-1523.11.6.828Search in Google Scholar
[22] N. Hogan. An organizing principle for a class of voluntary movements. Journal of Neuroscience, 4(11):2745, 1984. 10.1523/JNEUROSCI.04-11-02745.1984Search in Google Scholar
[23] T. Flash and N. Hogan. The coordination of arm movements - an experimentally confirmed mathematical-model. Journal of Neuroscience, 5(7):1688–1703, 1985. 10.1523/JNEUROSCI.05-07-01688.1985Search in Google Scholar
[24] E. Todorov and M. I. Jordan. Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. J. Neurophysiol, 80:696–714, 1998. 10.1152/jn.1998.80.2.696Search in Google Scholar PubMed
[25] T. Flash, Y. Meirovitch, and A. Barliya. Models of human movement: trajectory planning and inverse kinematics studies. Robotics and Autonomous Systems, 61(4):330–339, 2013. Search in Google Scholar
[26] S.M. Khansari-Zadeh. Benchmark data. http://www.amarsiproject. eu/open-source, 2012. Search in Google Scholar
[Online; accessed 17-October- 2014]. Search in Google Scholar
[27] S.-M. Khansari-Zadeh and Aude Billard. BM: An iterative algorithm to learn stable non-linear dynamical systems with gaussian mixture models. In Proc. of Int. Conf. on Robotics and Automation (ICRA), pages 2381–2388, 2010. 10.1109/ROBOT.2010.5510001Search in Google Scholar
[28] S.M. Khansari-Zadeh and A. Billard. Imitation learning of globally stable non-linear point-to-point robot motions using nonlinear programming. In Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pages 2676–2683, 2010. 10.1109/IROS.2010.5651259Search in Google Scholar
[29] S. M. Khansari-Zadeh. A Dynamical System-based Approach to Modeling Stable Robot Control Policies via Imitation Learning. PhD thesis, EPFL, 2012. Search in Google Scholar
[30] S.M. Khansari-Zadeh and A. Billard. Learning control lyapunov function to ensure stability of dynamical system-based robot reaching motions robotics and autonomous systems. Robotics and Autonomous Systems, 62(6):752–765, 2014. 10.1016/j.robot.2014.03.001Search in Google Scholar
[31] J. Gómez, D. Alvarez, S. Garrido, and L. Moreno. Kinesthetic teaching via fast marching square. In Proc. of Int. Conf. on Intelligent Robots and Systems (IROS), pages 1305–1310. IEEE, 2012. 10.1109/IROS.2012.6385497Search in Google Scholar
[32] R. Shadmehr and F. Mussa-Ivaldi. Adaptive representation of dynamics during learning of a motor task. Neuroscience, 14(5):3208–3224, 1994. 10.1523/JNEUROSCI.14-05-03208.1994Search in Google Scholar
[33] F. Gandolfo, F.Mussa-Ivaldi, and E. Bizzi. Motor learning by field approximation. National Academy of Sciences, 93(9):3843– 3846, 1996. 10.1073/pnas.93.9.3843Search in Google Scholar PubMed PubMed Central
[34] M. Conditt, F. Gandolfo, and F. Mussa-Ivaldi. The motor system does not learn the dynamics of the arm by rote memorization of past experience. Neurophysiology, 78(1):554–560, 1997. 10.1152/jn.1997.78.1.554Search in Google Scholar PubMed
[35] A Karniel and F. Mussa-Ivaldi. Does the motor control system use multiple models and context switching to cope with a variable environment? Experimental Brain Research, 143(4):520– 524, 2002. 10.1007/s00221-002-1054-4Search in Google Scholar PubMed
[36] D. Sternad and S. Schaal. Segmentation of endpoint trajectories does not imply segmented control. Experimental Brain Research, 124(1):118–136, 1999. 10.1007/s002210050606Search in Google Scholar PubMed
[37] F. Pollick, U. Maoz, A. Handzel, P. Giblin, G. Sapiro, and T. Flash. Three-dimensional arm movements at constant equiaflne speed. Cortex, 45(3):325–339, 2009. 10.1016/j.cortex.2008.03.010Search in Google Scholar PubMed
[38] D. Bennequin, R. Fuchs, A. Berthoz, and T. Flash. Movement timing and invariance arise from several geometries. PLoS computational biology, 5(7):e1000426, 2009. 10.1371/journal.pcbi.1000426Search in Google Scholar PubMed PubMed Central
[39] K. Lashley. The problem of serial order in psychology. Cerebral mechanisms in behavior. New York: Wiley, 1951. Search in Google Scholar
[40] N. Bernstein. The Co-ordination and Regulation of Movements. Pergamon Press, Oxford, 1967. Search in Google Scholar
[41] W. Abend, E. Bizzi, and P. Morasso. Human arm trajectory formation. Exp Brain Res, 105:331–348, 1982. 10.1093/brain/105.2.331Search in Google Scholar PubMed
[42] T. Flash. Organizing principles underlying the formation of hand trajectories. Doctoral dissertation, Massachusetts Institute of Technology, Cambridge, MA, 1983. Search in Google Scholar
[43] C. M. Harris and D. M. Wolpert. Signal-dependent noise determines motor planning. Nature, 394:780–784, 1998. 10.1038/29528Search in Google Scholar PubMed
[44] Bizzi E. Mussa-Ivaladi F.A. Motor learning through the combination of primitives. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences, 355:1755–1759, 2000. 10.1098/rstb.2000.0733Search in Google Scholar PubMed PubMed Central
[45] E. Bizzi, M.C. Tresch, P. Saltiel, and A. d Avella. New perspectives on spinal motor systems. Nature Reviews Neuroscience, 1(2):101–108, 2000. 10.1038/35039000Search in Google Scholar PubMed
[46] T. Flash and B. Hochner. Motor primitives in vertebrates and invertebrates. Current Opinion in Neurobiology, 15(6):660 – 666, 2005. Motor sytems / Neurobiology of behaviour. 10.1016/j.conb.2005.10.011Search in Google Scholar PubMed
[47] P. Viviani and C. deSperati. The relationsheep between curvature and velocity in two dimensional smooth pursuit eye movement. The Journal of Neuroscience, 17:3932–3945, 1997. 10.1523/JNEUROSCI.17-10-03932.1997Search in Google Scholar
[48] D. Endres, Y. Meirovitch, T. Flash, and M. Giese. Segmenting sign language into motor primitives with Bayesian binning. Frontiers in computational neuroscience, 7, 2013. 10.3389/fncom.2013.00068Search in Google Scholar PubMed PubMed Central
[49] F. Polyakov, E. Stark, R. Drori, M. Abeles, and T. Flash. Parabolic movement primitives and cortical states: merging optimality with geometric invariance. Biological Cybernetics, 100(2):159– 184, 2009. Search in Google Scholar
[50] M.S. Khansari, A. Lemme, Y. Meirovitch, B. Schrauwen, M. A. Giese, A.J. Ijspeert, A. Billard, and J.J. Steil. Workshop on benchmarking of state-of-the-art algorithms in generating human-like robot reaching motions. In Humanoids. IEEE, 2013. Search in Google Scholar
[51] A. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2):328–373, 2013. Search in Google Scholar
© 2015 A. Lemme et al.
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.
