Abstract
Due to its profound significance for the hearing-impaired community, there has been an abundance of recent research focused on Sign Language Translation (SLT), which is often decomposed into video-to-gloss recognition (S2G) and gloss-to-text (G2T) translation. Here, a gloss represents a sequence of transcribed spoken language words arranged in the order they are signed. In this paper, our emphasis lies in G2T, a crucial aspect of sign language translation. However, G2T encounters a scarcity of data, leading us to approach it as a low-resource neural machine translation (NMT) problem. Nevertheless, in contrast to traditional low-resource NMT, gloss-text pairs exhibit a greater lexical overlap but lower syntactic overlap. Hence, leveraging this characteristic, we utilize part-of-speech (POS) distributions obtained from numerous monolingual spoken language text to generate high-quality pseudo glosses. By simultaneously training numerous pseudo gloss-text sentence pairs alongside authentic data, we have significantly improved the translation performance of Chinese Sign Language (+3.10 BLEU), German Sign Language (+1.10 BLEU), and American Sign Language (+5.06 BLEU) in G2T translation, respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bungeroth, J., Ney, H.: Statistical sign language translation, p. 4
Camgoz, N.C., Hadfield, S., Koller, O., Ney, H., Bowden, R.: Neural sign language translation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, pp. 7784–7793. IEEE (2018). https://doi.org/10.1109/CVPR.2018.00812
Chen, Y., Wei, F., Sun, X., Wu, Z., Lin, S.: A simple multi-modality transfer learning baseline for sign language translation (2022)
Cihan Camgoz, N., Koller, O., Hadfield, S., Bowden, R.: Sign language transformers: joint end-to-end sign language recognition and translation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, pp. 10020–10030. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.01004
Cui, R., Hu, L., Zhang, C.: Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In: IEEE Conference on Computer Vision & Pattern Recognition (2017)
Duarte, A., et al.: How2Sign: a large-scale multimodal dataset for continuous American sign language (2021). https://doi.org/10.48550/arXiv.2008.08143
Gan, S., Yin, Y., Jiang, Z., Xie, L., Lu, S.: Skeleton-aware neural sign language translation. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4353–4361. ACM, Virtual Event China (2021). https://doi.org/10.1145/3474085.3475577
Hao, A., Min, Y., Chen, X.: Self-mutual distillation learning for continuous sign language recognition. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, pp. 11283–11292. IEEE (2021). https://doi.org/10.1109/ICCV48922.2021.01111
Kambhatla, N., Born, L., Sarkar, A.: CipherDAug: ciphertext based data augmentation for neural machine translation (2022)
Kan, J., Hu, K., Hagenbuchner, M., Tsoi, A.C., Bennamoun, M., Wang, Z.: Sign language translation with hierarchical spatio-temporal graph neural network. In: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2131–2140 (2022). https://doi.org/10.1109/WACV51458.2022.00219
Luqman, H., Mahmoud, S.A.: A machine translation system from Arabic sign language to Arabic. Univ. Access Inf. Soc. 19(4), 891–904 (2019). https://doi.org/10.1007/s10209-019-00695-6
Moryossef, A., Yin, K., Neubig, G., Goldberg, Y.: Data augmentation for sign language gloss translation, p. 11 (2021)
Othman, A., Jemni, M.: English-ASL gloss parallel corpus 2012: ASLG-PC12. In: 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon LREC (2012)
Şahin, G.G., Steedman, M.: Data augmentation via dependency tree morphing for low-resource languages (2019). https://doi.org/10.48550/arXiv.1903.09460
Ye, J., Jiao, W., Wang, X., Tu, Z.: Scaling back-translation with domain text generation for sign language gloss translation (2022)
Yin, K., Read, J.: Better sign language translation with STMC-transformer (2020)
Zhang, X., Duh, K.: Approaching sign language gloss translation as a low-resource machine translation task. In: Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pp. 60–70. Association for Machine Translation in the Americas, Virtual (2021)
Zhou, H., Zhou, W., Qi, W., Pu, J., Li, H.: Improving sign language translation with monolingual data by sign back-translation (2021)
Acknowledgement
This work was supported by the National Natural Science Foundation of China (No. 62006138), the Key Support Project of NSFC-Liaoning Joint Foundation (No. U1908216), and the Major Scientific Research Project of the State Language Commission in the 13th Five-Year Plan (No. WT135-38). We thank all anonymous reviewers for their valuable suggestions on this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, S., Zheng, Y., Lin, L., Chen, Y., Shi, X. (2023). A Novel POS-Guided Data Augmentation Method for Sign Language Gloss Translation. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14303. Springer, Cham. https://doi.org/10.1007/978-3-031-44696-2_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-44696-2_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44695-5
Online ISBN: 978-3-031-44696-2
eBook Packages: Computer ScienceComputer Science (R0)
