Abstract
This paper describes a novel approach for assessing the quality of machine-translated subtitles. Although machine translation (MT) is widely used for subtitling, in comparison to text translation, there is little research in this area. For our investigation, we are using the English to German machine translated subtitles from the SubCo corpus [11], a corpus consisting of human and machine-translated subtitles from English. In order to provide information about the quality of the machine-produced subtitles error annotation and evaluation is performed manually. Both the applied error annotation and evaluation schemes are covering the four dimensions content, language, format and semiotics allowing for a fine-grained detection of errors and weaknesses of the MT engine. Besides the human assessment of the subtitles, our approach comprises also the measurement of the inter-annotator agreement (IAA) of the human error annotation and evaluation, as well as the estimation of post-editing effort. The combination of these three steps represents a novel evaluation method that finds its use in both improving the subtitling quality assessment process and the machine translation systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
The subtitles were error annotated, evaluated and post-edited by novice translators, who were trained for several weeks to perform these tasks.
- 4.
This error annotation was performed by one professional translator.
- 5.
In order to increase the visualisation effect, we depicted only the first 15 subtitles. Depicting all subtitles would have made the visualisation impossible.
- 6.
IQR \(<1\) means that not all subtitles are depicted in Fig. 5, but only the ones with IQR \(<1\), leading to a different number of subtitles per error category.
- 7.
As in Fig. 4, we decided to show only the first 15 subtitles, increasing this way the visualisation effect. Depicting all subtitles would have made the visualisation impossible.
References
Abdallah, K.: Audiovisual translation in close-up: practical and theoretical approaches. In: Quality Problems in AVT Production Networks: Reconstructing An Actor-network In The Subtitling Industry, pp. 173–186. Peter Lang, Bern (2011)
Díaz-Cintas, J., Remael, A.: Subtitling: Concepts and Practices. Translation practices explained. Routledge, London (2020)
Del Pozo, A.: SUMAT Final Report. VICOMTECH (2014)
Etchegoyhen, T., et al.: Machine translation for subtitling: a large-scale evaluation. Proceedings of the Ninth International Conference On Language Resources and Evaluation (LREC), pp. 46–53 (2014,5)
Fleiss, B., Cho Paik, M.: Statistical Methods for Rates and Proportions. Wiley (1973)
Gupta, P., Sharma, M., Pitale, K., Kumar, K.: Problems with automating translation of movie/TV show subtitles. CoRR
Ivarsson, J., Carroll, M.: Subtitling.: TransEdit (1998)
Karakanta, A., Negri, M., Turchi, M.: Are Subtitling Corpora really Subtitle-like?. Conference: Sixth Italian Conference on Computational Linguistics (CLiC-It), At Bari, Italy (2019)
Kuo, A.: Professional realities of the subtitling industry: The subtitlers perspective. Audiovisual Translation in a Global Context: Mapping An Ever-Changing Landscape, pp. 163–191 (2015)
Lommel, A., Burchardt, A. & Uszkoreit, H.: Multidimensional quality metrics: a flexible system for assessing translation quality. In: Proceedings of Translating and the Computer 35 (ASLIB), November 2013
Martínez, J., Vela, M.: SubCo: a learner translation corpus of human and machine subtitles. In: Proceedings of the 10th International Conference On Language Resources and Evaluation, pp. 2246–2254 (2016)
Müller, M., Volk, M.: Statistical Machine Translation of Subtitles: From OpenSubtitles to TED. Language Processing And Knowledge In The Web SE - 14(8105), 132–138 (2013)
Nikolić, K. The Pros and Cons of Using Templates in Subtitling. Audiovisual Translation In A Global Context: Mapping An Ever-changing Landscape, pp. 192–202 (2015)
Robert, I., Remael, A.: Quality control in the subtitling industry: an exploratory survey study. Meta 61, 578–605 (2016)
Petukhova, V., et al.: Data Collection and Parallel Corpus Compilation for Machine Translation of Subtitles. LREC (2012)
Romero-Fresco, P.: Accessible filmmaking: Joining the dots between audiovisual translation, accessibility and filmmaking. J. Specialised Trans., 201–223 (2013,1)
Volk, M.: The Automatic Translation of Film Subtitles. A Machine Translation Success Story. In: Resourceful Language Technology: Festschrift In Honor Of Anna Saagvall Hein, pp. 202–214 (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Brendel, J., Vela, M. (2022). Quality Assessment of Subtitles – Challenges and Strategies. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds) Text, Speech, and Dialogue. TSD 2022. Lecture Notes in Computer Science(), vol 13502. Springer, Cham. https://doi.org/10.1007/978-3-031-16270-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-16270-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16269-5
Online ISBN: 978-3-031-16270-1
eBook Packages: Computer ScienceComputer Science (R0)