Skip to main content
SpringerLink
Log in

Relative Position Embedding Asymmetric Siamese Network for Offline Handwritten Mathematical Expression recognition

  • Conference paper
  • First Online:
Document Analysis and Recognition - ICDAR 2023 (ICDAR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14187))

Included in the following conference series:

  • 1180 Accesses

Abstract

Currently, Recurrent Neural Network(RNN)-based encoder-decoder models are widely used in handwritten mathematical expression recognition (HMER). Due to its recursive pattern, the problem of gradient disappearance or gradient explosion also exists for RNN, which makes them inefficient in processing long HME sequences. In order to solve above problems, this paper proposes a Transformer-based encoder-decoder model consisting of an asymmetric siamese network, relative position embedding Transformer (ASNRT). With the assistance of printed images, the asymmetric siamese network further narrows the difference betweeen feature maps of similar formula images and increases the encoding gap between dissimilar formula images. We insert coordinate attention into the encoder, additionally we replace RNN with Transformer as the decoder. Moreover, rotary position embedding is used, incorporating relative position information through absolute embedding ways. Given the symmetry of MEs, we adopt the bidirectional decoding strategy. Extensive experiments show that our model improves the ExpRate of state-of-the-art methods on CROHME 2014, CROHME 2016, and CROHME 2019 by 0.94\(\%\), 2.18\(\%\) and 2.12\(\%\), respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Álvaro, F., Sánchez, J.A., Benedí, J.M.: An integrated grammar-based approach for mathematical expression recognition. Pattern Recogn. 51, 135–147 (2016)

    Article  MATH  Google Scholar 

  2. Awal, A.M., Mouchère, H., Viard-Gaudin, C.: Towards handwritten mathematical expression recognition. In: 2009 10th International Conference on Document Analysis and Recognition, pp. 1046–1050. IEEE (2009)

    Google Scholar 

  3. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  4. Bian, X., Qin, B., Xin, X., Li, J., Su, X., Wang, Y.: Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 113–121 (2022)

    Google Scholar 

  5. Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758 (2021)

    Google Scholar 

  6. Cheng, T., Khan, J., Liu, H., Yun, D.: A symbol recognition system. In: Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR’93), pp. 918–921. IEEE (1993)

    Google Scholar 

  7. Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-markup generation with coarse-to-fine attention. In: International Conference on Machine Learning, pp. 980–989. PMLR (2017)

    Google Scholar 

  8. Grill, J.B., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural. Inf. Process. Syst. 33, 21271–21284 (2020)

    Google Scholar 

  9. Hou, Q., Zhou, D., Feng, J.: Coordinate attention for efficient mobile network design. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13713–13722 (2021)

    Google Scholar 

  10. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  11. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  12. Keshari, B., Watt, S.: Hybrid mathematical symbol recognition using support vector machines. In: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), vol. 2, pp. 859–863. IEEE (2007)

    Google Scholar 

  13. Le, A.D.: Recognizing handwritten mathematical expressions via paired dual loss attention network and printed mathematical expressions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 566–567 (2020)

    Google Scholar 

  14. Le, A.D., Indurkhya, B., Nakagawa, M.: Pattern generation strategies for improving recognition of handwritten mathematical expressions. Pattern Recogn. Lett. 128, 255–262 (2019)

    Article  Google Scholar 

  15. Li, Z., Jin, L., Lai, S., Zhu, Y.: Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 175–180. IEEE (2020)

    Google Scholar 

  16. Lin, Q., Wang, C., Bi, N., Suen, C.Y., Tan, J.: An encoder-decoder approach to offline handwritten mathematical expression recognition with residual attention. In: International Conference on Pattern Recognition and Artificial Intelligence, pp. 335–345. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-09037-0_28

  17. Luo, C., Zhu, Y., Jin, L., Wang, Y.: Learn to augment: joint data augmentation and network optimization for text recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13746–13755 (2020)

    Google Scholar 

  18. Mahdavi, M., Zanibbi, R., Mouchere, H., Viard-Gaudin, C., Garain, U.: Icdar 2019 crohme+ tfd: competition on recognition of handwritten mathematical expressions and typeset formula detection. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1533–1538. IEEE (2019)

    Google Scholar 

  19. Malon, C., Uchida, S., Suzuki, M.: Support vector machines for mathematical symbol recognition. In: Yeung, D.-Y., Kwok, J.T., Fred, A., Roli, F., de Ridder, D. (eds.) SSPR /SPR 2006. LNCS, vol. 4109, pp. 136–144. Springer, Heidelberg (2006). https://doi.org/10.1007/11815921_14

    Chapter  Google Scholar 

  20. Malon, C., Uchida, S., Suzuki, M.: Mathematical symbol recognition with support vector machines. Pattern Recogn. Lett. 29(9), 1326–1332 (2008)

    Article  Google Scholar 

  21. Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr 2014 competition on recognition of on-line handwritten mathematical expressions (crohme 2014). In: 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796. IEEE (2014)

    Google Scholar 

  22. Mouchère, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr 2016 crohme: competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 607–612. IEEE (2016)

    Google Scholar 

  23. Rhee, T.H., Kim, J.H.: Efficient search strategy in structural analysis for handwritten mathematical expression recognition. Pattern Recogn. 42(12), 3192–3201 (2009)

    Article  MATH  Google Scholar 

  24. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  25. Schaefer, S., McPhail, T., Warren, J.: Image deformation using moving least squares. In: ACM SIGGRAPH 2006 Papers, pp. 533–540 (2006)

    Google Scholar 

  26. Simistira, F., Papavassiliou, V., Katsouros, V., Carayannis, G.: Structural analysis of online handwritten mathematical symbols based on support vector machines. In: Document Recognition and Retrieval XX, vol. 8658, pp. 332–339. SPIE (2013)

    Google Scholar 

  27. Su, J., Lu, Y., Pan, S., Wen, B., Liu, Y.: Roformer: enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 (2021)

  28. Truong, T.N., Nguyen, C.T., Phan, K.M., Nakagawa, M.: Improvement of end-to-end offline handwritten mathematical expression recognition by weakly supervised learning. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 181–186. IEEE (2020)

    Google Scholar 

  29. Truong, T.-N., Ung, H.Q., Nguyen, H.T., Nguyen, C.T., Nakagawa, M.: Relation-based representation for handwritten mathematical expression recognition. In: Barney Smith, E.H., Pal, U. (eds.) ICDAR 2021. LNCS, vol. 12916, pp. 7–19. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86198-8_1

    Chapter  Google Scholar 

  30. Vaswani, A., et al.: Attention is all you need. Advances in neural information processing systems 30 (2017)

    Google Scholar 

  31. Wang, H., Shan, G.: Recognizing handwritten mathematical expressions as latex sequences using a multiscale robust neural network. arXiv preprint arXiv:2003.00817 (2020)

  32. Wang, J., Du, J., Zhang, J., Wang, Z.R.: Multi-modal attention network for handwritten mathematical expression recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1181–1186. IEEE (2019)

    Google Scholar 

  33. Winkler, H.J.: Hmm-based handwritten symbol recognition using on-line and off-line features. In: 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings. vol. 6, pp. 3438–3441. IEEE (1996)

    Google Scholar 

  34. Winkler, H.J., Lang, M.: Symbol segmentation and recognition for understanding handwritten mathematical expressions. In: Tagungs band 5th International Workshop on Frontiers in Handwriting Recognition IWFHR-5, pp. 407–412 (1997)

    Google Scholar 

  35. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

    Google Scholar 

  36. Wu, J.-W., Yin, F., Zhang, Y.-M., Zhang, X.-Y., Liu, C.-L.: Image-to-Markup generation via paired adversarial learning. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11051, pp. 18–34. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10925-7_2

    Chapter  Google Scholar 

  37. Wu, J.W., Yin, F., Zhang, Y.M., Zhang, X.Y., Liu, C.L.: Handwritten mathematical expression recognition via paired adversarial learning. Int. J. Comput. Vision 128(10), 2386–2401 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  38. Yan, Z., Zhang, X., Gao, L., Yuan, K., Tang, Z.: Convmath: a convolutional sequence network for mathematical expression recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4566–4572. IEEE (2021)

    Google Scholar 

  39. Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2245–2250. IEEE (2018)

    Google Scholar 

  40. Zhang, J., Du, J., Yang, Y., Song, Y.Z., Wei, S., Dai, L.: A tree-structured decoder for image-to-markup generation. In: International Conference on Machine Learning, pp. 11076–11085. PMLR (2020)

    Google Scholar 

  41. Zhang, J., et al.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)

    Article  Google Scholar 

  42. Zhao, W., Gao, L., Yan, Z., Peng, S., Du, L., Zhang, Z.: Handwritten mathematical expression recognition with bidirectionally trained transformer. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 570–584. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86331-9_37

    Chapter  Google Scholar 

Download references

Acknowledgments

This work was supported by Guangdong Province Key Laboratory of Computational Science at the Sun Yat-sen University (2020B1212060032), and the National Science Foundation of China (11971491, 12171490), and the Foundation of Guangdong-Hong Kong-Macao National Center for Applied Mathematics (2021B1515310002).

Author information

Authors and Affiliations

  1. School of Mathematics, Sun Yat-Sen University, Guangzhou, 510275, People’s Republic of China

    Chunyi Wang, Xiaqing Rao, Wei Hu, Ning Bi & Jun Tan

  2. Guangdong Province Key Laboratory of Computational Science, Sun Yat-Sen University, Guangzhou, 510275, People’s Republic of China

    Ning Bi & Jun Tan

  3. Meituan, Shanghai, China

    Hurunqi Luo

Authors
  1. Chunyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hurunqi Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaqing Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ning Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jun Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Tan .

Editor information

Editors and Affiliations

  1. TU Dortmund University, Dortmund, Germany

    Gernot A. Fink

  2. Adobe, College Park, MN, USA

    Rajiv Jain

  3. Osaka Metropolitan University, Osaka, Japan

    Koichi Kise

  4. Rochester Institute of Technology, Rochester, NY, USA

    Richard Zanibbi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, C., Luo, H., Rao, X., Hu, W., Bi, N., Tan, J. (2023). Relative Position Embedding Asymmetric Siamese Network for Offline Handwritten Mathematical Expression recognition. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14187. Springer, Cham. https://doi.org/10.1007/978-3-031-41676-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-41676-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-41675-0

  • Online ISBN: 978-3-031-41676-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation