Abstract
Normalization of SMS text, commonly known as texting language, is being pursued for more than a decade. A probabilistic approach based on the Trie data structure was proposed in literature which was found to be better performing than HMM based approaches proposed earlier in predicting the correct alternative for an out-of-lexicon word. However, success of the Trie-based approach depends largely on how correctly the underlying probabilities of word occurrences are estimated. In this work we propose a structural modification to the existing Trie-based model along with a novel training algorithm and probability generation scheme. We prove two theorems on statistical properties of the proposed Trie and use them to claim that is an unbiased and consistent estimator of the occurrence probabilities of the words. We further fuse our model into the paradigm of noisy channel based error correction and provide a heuristic to go beyond a Damerau-Levenshtein distance of one.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Generally, in more branched Tries, whenever \(w_1\) is a prefix of \(w_2\), Trie probability of \(w_2\) \(\le \) Trie probability of \(w_1\).
- 2.
- 3.
Value of the exponent characterising the distribution was set to 0.25.
- 4.
Not needed once deployed, model learns probability directly from the user.
References
Chatterjee, N.: A Trie based model for SMS text normalization. In: Arai, K., Bhatia, R., Kapoor, S. (eds.) CompCom 2019. AISC, vol. 997, pp. 846–859. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22871-2_60
Choudhury, M., Saraf, R., Jain, V., Mukherjee, A., Sarkar, S., Anupam, B.: Investigation and modeling of the structure of texting language. IJDAR 10, 157–174 (2007)
Crystal, D.: Language and the Internet, 2nd edn. Cambridge University Press (2006)
Damerau, F.J.: A technique for computer detection and correction of spelling errors. Commun. ACM 7(3), 171–176 (1964)
Desai, N., Narvekar, M.: Normalization of noisy text data. Procedia Comput. Sci. 45, 12 (2015)
Grinter, R.E., Eldridge, M.A.: Y do tngrs luv 2 txt msg? In: Proceedings of the Seventh European Conference on Computer Supported Cooperative Work (ECSCW 2001) (2001)
Jurafsky, D., Martin, J.H.: Speech and Language Processing, 2nd edn. Prentice-Hall Inc., USA (2009)
Kernighan, M.D., Church, K.W., Gale, W.A.: A spelling correction program based on a noisy channel model. In: Proceedings of the 13th Conference on Computational Linguistics - Volume 2 (1990)
Khan, O.A., Karim, A.: A rule-based model for normalization of SMS text. In: 2012 IEEE 24th International Conference on Tools with Artificial Intelligence, vol. 1, pp. 634–641 (2012)
Kobus, C., Yvon, F., Damnati, G.: Normalizing SMS: are two metaphors better than one ? In: COLING (2008)
Li, C., Liu, Y.: Normalization of text messages using character- and phone-based machine translation approaches. In: INTERSPEECH (2012)
van der Vaart, A.W.: Asymptotic Statistics. Cambridge University Press (2000)
Veliz, C.M.: Orphée De Clercq, and VÃ\({\copyright }\)ronique Hoste. Comparing MT approaches for text normalization. In: RANLP (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sikdar, A., Chatterjee, N. (2022). An Improved Bayesian TRIE Based Model for SMS Text Normalization. In: Arai, K. (eds) Intelligent Computing. SAI 2022. Lecture Notes in Networks and Systems, vol 507. Springer, Cham. https://doi.org/10.1007/978-3-031-10464-0_39
Download citation
DOI: https://doi.org/10.1007/978-3-031-10464-0_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-10463-3
Online ISBN: 978-3-031-10464-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)