Dialect Text Normalization to Normative Standard Finnish

Niko Partanen, Mika Hämäläinen, Khalid Alnajjar


Abstract
We compare different LSTMs and transformer models in terms of their effectiveness in normalizing dialectal Finnish into the normative standard Finnish. As dialect is the common way of communication for people online in Finnish, such a normalization is a necessary step to improve the accuracy of the existing Finnish NLP tools that are tailored for normative Finnish text. We work on a corpus consisting of dialectal data of 23 distinct Finnish dialects. The best functioning BRNN approach lowers the initial word error rate of the corpus from 52.89 to 5.73.
Anthology ID:
D19-5519
Volume:
Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Wei Xu, Alan Ritter, Tim Baldwin, Afshin Rahimi
Venue:
WNUT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
141–146
Language:
URL:
https://aclanthology.org/D19-5519
DOI:
10.18653/v1/D19-5519
Bibkey:
Cite (ACL):
Niko Partanen, Mika Hämäläinen, and Khalid Alnajjar. 2019. Dialect Text Normalization to Normative Standard Finnish. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 141–146, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Dialect Text Normalization to Normative Standard Finnish (Partanen et al., WNUT 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-5519.pdf
Code
 mikahama/murre