Abstract
Tibetan is a monosyllabic language for which computerized language tools are largely lacking. We describe the development of a syllable stemmer for Tibetan. The stemmer is based on a set of rules that strive to identify the vowel, the core letter of the syllable, and then the other parts. We demonstrate the value of the stemmer with two applications: determining stem similarity of two syllables and word segmentation. Our stemmer is being made available as an open-source tool and word segmentation as a freely-available online tool.
It is worthy of remark that a tongue which in its nature was monosyllabic, when written in the characters of a polysyllabic language like the Sanskrit, had necessarily to undergo some modification.
Sarat Chandra Das, “Life of Sum-pa mkhan-po, also styled Ye-śes dpal-’byor, the author of Rehumig (Chronological Table)”, Journal of the Asiatic Society of Bengal (1889)
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Transliterated g.y to differentiate from core g with subscript y.
- 2.
The reason for omitting the coda s for the sake of normalization is that in cases where it is added to form the past tense, which results in a syllable that appears to have a stem with coda s, we treat this s as equivalent to the postscript s often added to form the past tense.
References
Almogi, O., Dankin, L., Dershowitz, N., Wolf, L.: A hackathon for classical Tibetan J. Data Min. Digital Humanit. (to appear)
Chen, X., Qiu, X., Zhu, C., Liu, P., Huang, X.: Long short-term memory neural networks for Chinese word segmentation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 1197–1206. Association for Computational Linguistics, September 2015. http://aclweb.org/anthology/D15-1141
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995). https://doi.org/10.1007/BF00994018
Hahn, M.: Lehrbuch der klassischen tibetischen Schriftsprache, Indica et Tibetica, 7th edn., vol. 10. Indica et Tibetica Verlag, Marburg (1996)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Huang, H., Da, F.: General structure based collation of Tibetan syllables. J. Comput. Inf. Syst. 6(5), 1693–1703 (2010)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, May 2015. http://arxiv.org/pdf/1412.6980v8.pdf
Klein, B., Dershowitz, N., Wolf, L., Almogi, O., Wangchuk, D.: Finding inexact quotations within a Tibetan Buddhist corpus. In: Digital Humanities (DH 2014), Lausanne, Switzerland, pp. 486–488, July 2014. http://nachum.org/papers/textalignment.pdf
Liu, H., Nuo, M., Wu, J.: Zipf’s law and statistical data on modern Tibetan. In: COLING (2014)
Wylie, T.V.: A standard system of Tibetan transcription. Harvard J. Asiatic Stud. 22, 261–267 (1959)
Zipf, G.K.: Human Behaviour and the Principle of Least Effort. Hafner Pub. Co., New York (1949)
Acknowledgements
We would like to express our deep gratitude to the other participants in the “Hackathon in the Arava” event (held in Kibbutz Lotan, Israel, February 2016; see [1]), who all contributed to the development of new digital tools for analyzing Tibetan texts: Kfir Bar, Marco Büchler, Daniel Hershcovich, Marc W. Küter, Daniel Labenski, Peter Naftaliev, Elad Shaked, Nadav Steiner, Lior Uzan, and Eric Werner. We thank Paul Hacket for crucially providing the necessary data.
This research was supported in part by a Grant (#I-145-101.3-2013) from the GIF, the German-Israeli Foundation for Scientific Research and Development, and by the Khyentse Center for Tibetan Buddhist Textual Scholarship, Universität Hamburg, thanks to a grant by the Khyentse Foundation. N.D.’s and L.W.’s research was supported in part by the Israeli Ministry of Science, Technology and Space (Israel-Taiwan grant #3-10341). N.D.’s research benefitted from a fellowship at the Paris Institute for Advanced Studies (France), with the financial support of the French state, managed by the French National Research Agency’s “Investissements d’avenir” program (ANR-11-LABX-0027-01 Labex RFIEA+).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Almogi, O. et al. (2018). Stemming and Segmentation for Classical Tibetan. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2016. Lecture Notes in Computer Science(), vol 9623. Springer, Cham. https://doi.org/10.1007/978-3-319-75477-2_20
Download citation
DOI: https://doi.org/10.1007/978-3-319-75477-2_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-75476-5
Online ISBN: 978-3-319-75477-2
eBook Packages: Computer ScienceComputer Science (R0)