Skip to main content
SpringerLink
Log in

Bidirectional long short-term memory networks and sparse hierarchical modeling for scalable educational learning of dance choreographies

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Recently, several educational game platforms have been proposed in the literature for choreographic training. However, their main limitation is that they fail to provide a quantitative assessment framework of a performing choreography against a groundtruth one. In this paper, we address this issue by proposing a machine learning framework exploiting deep learning paradigms. In particular, we introduce a long short-term memory network with the main capability of analyzing 3D captured skeleton feature joints of a dancer into predefined choreographic postures. This pose identification procedure is capable of providing a detailed (fine) evaluation score of a performing dance. In addition, the paper proposes a choreographic summarization architecture based on sparse modeling representative selection (SMRS) in order to abstractly represent the performing choreography through a set of key choreographic primitives. We have modified the SMRS algorithm in a way to extract hierarchies of key representatives. Choreographic summarization provides an efficient tool for a coarse quantitative evaluation of a dance. Moreover, hierarchical representation scheme allows for a scalable assessment of a choreography. The serious game platform supports advanced visualization toolkits using Labanotation in order to deliver the performing sequence in a formal documentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Aristidou, A., Charalambous, P., Chrysanthou, Y.: Emotion analysis and classification: understanding the performers’ emotions using the lma entities. Comput. Graph. Forum 34, 262–276 (2015)

    Article  Google Scholar 

  2. Aristidou, A., Stavrakis, E., Charalambous, P., Chrysanthou, Y., Himona, S.: Folk dance evaluation using laban movement analysis. J. Comput. Cult. Heritage 8, 20 (2015)

    Article  Google Scholar 

  3. Aristidou, A., Stavrakis, E., Papaefthimiou, M., Papagiannakis, G., Chrysanthou, Y.: Style-based motion analysis for dance composition. Vis. Comput. 34(12), 1725–1737 (2018)

    Article  Google Scholar 

  4. Aristidou, A., Yin, K., Zeng, Q., Cohen-Or, D., Chen, B., Stavrakis, E., Chrysanthou, Y.: Emotion control of unstructured dance movements. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation (2017). https://doi.org/10.1145/3099564.3099566

  5. Ballas, A., Santad, T., Sookhanaphibarn, K., Choensawat, W.: Game-based system for learning labanotation using microsoft kinect. In: 2017 IEEE 6th Global Conference on Consumer Electronics (GCCE), pp. 1–3. IEEE (2017)

  6. Baraff, D.: Rigid body simulation. In: Proceedings of the SIGGRAPH Course Notes, vol. 19, pp. 1–68 (1992)

  7. Bernstein, R., Shafir, T., Tsachor, R., Studd, K., Schuster, A.: Laban movement analysis using kinect. Int. J. Comput. Electr. Autom. Control Inform. Eng. 9, 1394–1398 (2015)

    Google Scholar 

  8. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2011)

    Article  Google Scholar 

  9. Chowdhury, G.G.: Introduction to Modern Information Retrieval. Facet publishing, London (2010)

    Google Scholar 

  10. Dimitropoulos, K., Manitsaris, S., Tsalakanidou, F., Nikolopoulos, S., Denby, B., Kork, S., Crevier-Buchman, L., Pillot-Loiseau, C., Adda-Decker, M., Dupont, S., Tilmanne, J., Ott, M., Alivizatou, M., Yilmaz, E., Hadjileontiadis, L., Charisis, V., Deroo, O., Manitsaris, A., Kompatsiaris, I., Grammalidis, N.: Capturing the intangible: an introduction to the i-treasures project. In: Proceedings of the 9th International Conference on Computer Vision Theory and Applications (VISAPP), pp. 773–781 (2014)

  11. Doulamis, A.D., Doulamis, N.D., Kollias, S.D.: An adaptable neural-network model for recursive nonlinear traffic prediction and modeling of MPEG video sources. IEEE Trans. Neural Netw. 14(1), 150–166 (2003)

    Article  Google Scholar 

  12. Doulamis, A.D., Voulodimos, A., Doulamis, N.D., Soile, S., Lampropoulos, A.: Transforming intangible folkloric performing arts into tangible choreographic digital objects: the terpsichore approach. In: VISIGRAPP (5: VISAPP), pp. 451–460 (2017)

  13. Doulamis, N., Doulamis, A., Ioannidis, C., Klein, M., Ioannides, M.: Modelling of static and moving objects: digitizing tangible and intangible cultural heritage. In: Mixed Reality and Gamification for Cultural Heritage, pp. 567–589 (2017)

  14. Doulamis, N., Voulodimos, A.: FAST-MDL: fast adaptive supervised training of multi-layered deep learning models for consistent object tracking and classification. In: 2016 IEEE International Conference on Imaging Systems and Techniques (IST), pp. 318–323 (2016). https://doi.org/10.1109/IST.2016.7738244

  15. Elhamifar, E., Sapiro, G., Vidal, R.: See all by looking at a few: sparse modeling for finding representative objects. In: In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1600–16007 (2012)

  16. Griesbeck, C.: Introduction to labanotation (1996). https://user.uni-frankfurt.de/~griesbec/LABANE.HTML. Accessed 15 July 2019

  17. Hachimura, K., Nakamura, M.: Method of generating coded description of human body motion from motion-captured data. In: 10th IEEE International Workshop on Robot and Human Interactive Communication, 2001. Proceedings, pp. 122–127. IEEE (2001)

  18. Hisatomi, K., Katayama, M., Tomiyama, K., Iwadate, Y.: 3D archive system for traditional performing arts: application of 3D reconstruction method using graph-cuts. Int. J. Comput. Vis. 94, 78–88 (2011)

    Article  Google Scholar 

  19. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  20. Hsu, HmJ: The potential of kinect in education. Int. J. Inf. Educ. Technol. 1(5), 365 (2011)

    Google Scholar 

  21. Keselman, L., Woodfill, J.I., Grunnet-Jepsen, A., Bhowmik, A.: Intel realsense stereoscopic depth cameras (2017). arXiv preprint arXiv:1705.05548

  22. Kim, D., Kim, D.H., Kwak, K.C.: Classification of k-pop dance movements based on skeleton information obtained by a kinect sensor. Sens. MDPI 17, 1261 (2017)

    Article  Google Scholar 

  23. Kitsikidis, A., Dimitropoulos, K., Uğurca, D., Bayçay, C., Yilmaz, E., Tsalakanidou, F., Douka, S., Grammalidis, N.: A game-like application for dance learning using a natural human computer interface. In: International Conference on Universal Access in Human-Computer Interaction, pp. 472–482. Springer (2015)

  24. Kojima, K., Hachimura, K., Nakamura, M.: Labaneditor: graphical editor for dance notation. In: 11th IEEE International Workshop on Robot and Human Interactive Communication, 2002. Proceedings, pp. 59–64. IEEE (2002)

  25. Kurin, R.: Safeguarding intangible cultural heritage in the 2003 unesco convention: a critical appraisal. Mus. Int. 56(1–2), 66–77 (2004)

    Article  Google Scholar 

  26. Laggis, A., Doulamis, N., Protopapadakis, E., Georgopoulos, A.: A low-cost markerless tracking system for trajectory interpretation, pp. 413–418 (2017)

  27. Masurelle, A., Essid, S., Richard, G.: Multimodal classification of dance movements using body joint trajectories and step sounds. In: Proceedings of the International Workshop on Image Analysis for Multimedia Interactive Services, pp. 1–4 (2013)

  28. Pforsich, J.: Handbook for Laban Movement Analysis (1997)

  29. Protopapadakis, E., Grammatikopoulou, A., Doulamis, A., Grammalidis, N.: Folk dance pattern recognition over depth images acquired via kinect sensor. In: Proceedings of the International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences—ISPRS Archives, vol. 42, pp. 587–593 (2017)

  30. Rallis, I., Doulamis, N., Doulamis, A., Voulodimos, A., Vescoukis, V.: Spatio-temporal summarization of dance choreographies. Comput. Graph. 73, 88–101 (2018)

    Article  Google Scholar 

  31. Rallis, I., Georgoulas, I., Doulamis, N., Voulodimos, A., Terzopoulos, P.: Extraction of key postures from 3D human motion data for choreography summarization. In: Proceedings of the IEEE 9th International Conference on Virtual Worlds and Games for Serious Applications, (VS-Games), pp. 94–101 (2017)

  32. Rallis, I., Langis, A., Georgoulas, I., Voulodimos, A., Doulamis, N., Doulamis, A.: An embodied learning game using kinect and labanotation for analysis and visualization of dance kinesiology. In: 2018 10th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games), pp. 1–8. IEEE (2018)

  33. Raptis, M., Kirovski, D., Hoppe, H.: Real-time classification of dance gestures from skeleton animation. In: Proceedings of the ACM Symposium on Computer Animation—SIGGRAPH/Eurographics, pp. 147–156 (2011)

  34. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)

    Article  Google Scholar 

  35. Sheppard, R., Kamali, M., Rivas, R., Tamai, M., Yang, Z., Wu, W., Nahrstedt, K.: Advancing interactive collaborative mediums through tele-immersive dance (ted): a symbiotic creativity and design environment for art and computer science. In: Proceedings of the 2008 ACM International Conference on Multimedia, with co-located Symposium and Workshops, pp. 579–588 (2008)

  36. Stavrakis, E., Aristidou, A., Savva, M., Himona, S., Chrysanthou, Y.: Digitization of cypriot folk dances. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 7616, pp. 404–413. LNCS (2012)

  37. Vögele, A., Krüger, B., Klein, R.: Efficient unsupervised temporal segmentation of human motion. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 167–176. Eurographics Association (2014)

  38. Voulodimos, A., Doulamis, N., Doulamis, A., Protopapadakis, E.: Deep learning for computer vision: a brief review. Comput. Intell. Neurosci. 2018, 7068349 (2018). https://doi.org/10.1155/2018/7068349

    Article  Google Scholar 

  39. Voulodimos, A., Doulamis, N., Doulamis, A., Rallis, I.: Kinematics-based extraction of salient 3D human motion data for summarization of choreographic sequences. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3013–3018. IEEE (2018)

  40. Voulodimos, A., Rallis, I., Doulamis, N.: Physics-based keyframe selection for human motion summarization. Multimed. Tools Appl. (2018). https://doi.org/10.1007/s11042-018-6935-z

    Article  Google Scholar 

  41. Wang, J., Miao, Z., Guo, H., Zhou, Z., Wu, H.: Using automatic generation of labanotation to protect folk dance. J. Electron. Imaging 26(1), 011028 (2017)

    Article  Google Scholar 

  42. Zhang, Z.: Microsoft kinect sensor and its effect. IEEE Multimed. 19, 4–10 (2012)

    Article  Google Scholar 

  43. Zhou, F., De la Torre, F., Hodgins, J.K.: Hierarchical aligned cluster analysis for temporal clustering of human motion. IEEE Trans. Pattern Anal. Mach. Intell. 35(3), 582–596 (2013)

    Article  Google Scholar 

Download references

Funding

This work is funded by the European Union project TERPSICHORE Transforming Intangible Folkloric Performing Arts into Tangible Choreographic Digital Objects funded under the Grant Agreement 691218.

Author information

Authors and Affiliations

  1. National Technical University of Athens, 9, Herroon Polytechniou Str., 15773, Zografou, Athens, Greece

    Ioannis Rallis, Nikolaos Bakalos, Nikolaos Doulamis & Anastasios Doulamis

  2. Department of Informatics and Computer Engineering, University of West Attica, Agiou Spyridonos Str., 12243, Egaleo, Athens, Greece

    Athanasios Voulodimos

Authors
  1. Ioannis Rallis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikolaos Bakalos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nikolaos Doulamis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anastasios Doulamis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Athanasios Voulodimos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioannis Rallis.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

All procedures performed in studies involving human participants were in accordance with the ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rallis, I., Bakalos, N., Doulamis, N. et al. Bidirectional long short-term memory networks and sparse hierarchical modeling for scalable educational learning of dance choreographies. Vis Comput 37, 47–62 (2021). https://doi.org/10.1007/s00371-019-01741-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01741-3

Keywords

Search

Navigation