Abstract
Inspired by recent spatio-temporal Convolutional Neural Networks in computer vision field, we propose OLT-C3D (Online Long-Term Convolutional 3D), a new architecture based on a 3D Convolutional Neural Network (3D CNN) to address the complex task of early recognition of 2D handwritten gestures in real time. The input signal of the gesture is translated into an image sequence along time with the trajectory history. The image sequence is passed into our 3D CNN OLT-C3D which gives a prediction at each new frame. OLT-C3D is coupled with an integrated temporal reject system to postpone the decision in time if more information is needed. Moreover our system is end-to-end trainable, OLT-C3D and the temporal reject system are jointly trained to optimize the earliness of the decision. Our approach achieves superior performances on two complementary and freely available datasets: ILGDB and MTGSetB.
This study is funded by the ANR within the framework of the PIA EUR DIGISPORT project (ANR-18-EURE-0022).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bloom, V., Argyriou, V., Makris, D.: Linear latent low dimensional space for online early action recognition and prediction. Pattern Recogn. 72, 532–547 (2017). https://doi.org/10.1016/j.patcog.2017.07.003
Boulahia, S.Y., Anquetil, E., Multon, F., Kulpa, R.: Détection précoce d’actions squelettiques 3D dans un flot non segmenté à base de modèles curvilignes. In: Reconnaissance des Formes. Image, Apprentissage et Perception, RFIAP 2018, June 2018, Paris, France, pp. 1–8 (2018)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017). https://doi.org/10.1109/CVPR.2017.502
Chen, Z., Anquetil, E., Viard-Gaudin, C., Mouchère, H.: Early recognition of handwritten gestures based on multi-classifier reject option. In: 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 01, pp. 212–217 (2017). https://doi.org/10.1109/ICDAR.2017.43
Chen, Z., Anquetil, E., Mouchère, H., Viard-Gaudin, C.: Recognize multi-touch gestures by graph modeling and matching. In: 17th Biennial Conference of the International Graphonomics Society. Drawing, Handwriting Processing Analysis: New Advances and Challenges. International Graphonomics Society (IGS) and Université des Antilles (UA), Pointe-a-Pitre, Guadeloupe (June 2015)
Escalante, H.J., Morales, E.F., Sucar, L.E.: A naïve Bayes baseline for early gesture recognition. Pattern Recogn. Lett. 73, 91–99 (2016). https://doi.org/10.1016/j.patrec.2016.01.013
Geifman, Y., El-Yaniv, R.: SelectiveNet: a deep neural network with an integrated reject option. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, 09–15 June 2019, vol. 97, pp. 2151–2159. PMLR (2019)
Kawashima, M., Shimada, A., Nagahara, H., Taniguchi, R.: Adaptive template method for early recognition of gestures. In: 17th Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), pp. 1–6 (2011). https://doi.org/10.1109/FCV.2011.5739719
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2017)
Kurtenbach, G., Buxton, W.: Issues in combining marking and direct manipulation techniques. In: Proceedings of the 4th Annual ACM Symposium on User Interface Software and Technology, UIST 1991, pp. 137–144. Association for Computing Machinery, New York (1991). https://doi.org/10.1145/120782.120797
Liu, J., Shahroudy, A., Wang, G., Duan, L., Kot, A.C.: Skeleton-based online action prediction using scale selection network. IEEE Trans. Pattern Anal. Mach. Intell. 42(6), 1453–1467 (2020). https://doi.org/10.1109/TPAMI.2019.2898954
Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., Kautz, J.: Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016). https://doi.org/10.1109/CVPR.2016.456
Mori, A., Uchida, S., Kurazume, R., Taniguchi, R., Hasegawa, T., Sakoe, H.: Early recognition and prediction of gestures. In: 18th International Conference on Pattern Recognition, ICPR 2006, vol. 3, pp. 560–563 (2006). https://doi.org/10.1109/ICPR.2006.467
van den Oord, A., et al.: WaveNet: a generative model for raw audio. CoRR (2016)
Petit, E., Maldivi, C.: Unifying gestures and direct manipulation in touchscreen interfaces (December 2013)
Renau-Ferrer, N., Li, P., Delaye, A., Anquetil, E.: The ILGDB database of realistic pen-based gestural commands. In: Proceedings of the 21st International Conference on Pattern Recognition, ICPR 2012, pp. 3741–3744 (2012)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: IEEE International Conference on Computer Vision (ICCV), pp. 4489–4497 (2015). https://doi.org/10.1109/ICCV.2015.510
Uchida, S., Amamoto, K.: Early recognition of sequential patterns by classifier combination. In: 19th International Conference on Pattern Recognition, pp. 1–4 (2008). https://doi.org/10.1109/ICPR.2008.4761137
Weber, M., Liwicki, M., Stricker, D., Scholzel, C., Uchida, S.: LSTM-based early recognition of motion patterns. In: 2014 22nd International Conference on Pattern Recognition, pp. 3552–3557 (2014). https://doi.org/10.1109/ICPR.2014.611
Yamagata, M., Hayashi, H., Uchida, S.: Handwriting prediction considering inter-class bifurcation structures. In: 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 103–108 (2020). https://doi.org/10.1109/ICFHR2020.2020.00029
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Mocaër, W., Anquetil, E., Kulpa, R. (2021). Online Spatio-temporal 3D Convolutional Neural Network for Early Recognition of Handwritten Gestures. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12821. Springer, Cham. https://doi.org/10.1007/978-3-030-86549-8_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-86549-8_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86548-1
Online ISBN: 978-3-030-86549-8
eBook Packages: Computer ScienceComputer Science (R0)
