Abstract
Recent studies indicated a trend of quantifying lexical semantic changes with distributional models. In this study, we investigated whether state-of-the-art language models can tell us the story of how a word developed its senses over time. Specifically, we exploited the Bert model to obtain sense representations and quantitatively track usage changes after performing sense classification for each occurrence of targets in a historical newspaper dataset(People’s Daily(1954–2003). Our experiment provided a positive answer to the research question, as the model has an overall precision score of 91.82% on classifying senses against human judgments. We also charted usage changes of targets, which demonstrates a possible way to (semi-)automatically observe the development of word meanings.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
CCL corpus: http://ccl.pku.edu.cn:8080/ccl_corpus/index.jsp, and the BCC corpus: http://bcc.blcu.edu.cn.
- 2.
We excluded those senses that have very few sentences found in both the CCL corpus and the BCC corpus.
References
Hamilton, W.L., Leskovec, J., Jurafsky, D.: Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, November 2016, pp. 2116–2121. Association for Computational Linguistics (2016)
Hamilton, W.L., Leskovec, J., Jurafsky, D.: Diachronic word embeddings reveal statistical laws of semantic change. In: Erk, K., mith, N.A. (eds.) Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, August 2016, pp. 1489–1501. Association for Computational Linguistics (2016)
Tahmasebi, N., Borin, L., Jatowt, A.: Survey of computational approaches to lexical semantic change detection, chap. 1. Language Science Press (2021)
Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., Tahmasebi, N.: SemEval-2020 task 1: unsupervised lexical semantic change detection. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, Barcelona (online), December 2020, pp. 1–23. International Committee for Computational Linguistics (2020)
Kulkarni, V., Al-Rfou, R., Perozzi, B., Skiena, S.: Statistically significant detection of linguistic change. In: Proceedings of the 24th International Conference on World Wide Web, WWW ’15, Republic and Canton of Geneva, CHE, pp. 625–635. International World Wide Web Conferences Steering Committee (2015)
F Dubossarsky, H., Hengchen, S., Tahmasebi, N., Schlechtweg, D.: Time-out: temporal referencing for robust modeling of lexical semantic change. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019, pp. 457–470. Association for Computational Linguistics (2019)
Chen, J., Peng, B., Huang, C.-R.: Tracing lexical semantic change with distributional semantics: change and stability. In: Su, Q., Xu, G., Yang, X. (eds.) Chinese Lexical Semantics: 23rd Workshop, CLSW 2022, Virtual Event, May 14–15, 2022, Revised Selected Papers, Part I, pp. 244–252. Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031-28953-8_19
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics (2019)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Hu, R., Li, S., Liang, S.: Diachronic sense modeling with deep contextualized word embeddings: an ecological view. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019, pp. 3899–3908. Association for Computational Linguistics (2019)
Giulianelli, M., Del Tredici, M., Fernández, R.: Analysing lexical semantic change with contextualised word representations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020, pp. 3960–3973. Association for Computational Linguistics (2020)
Chinese Academy of Social Science Department of Chinese Lexicography, Institute of Linguistics. Contemporary Chinese Dictionary (Xiandai Hanyu Cidian), 7th edn. Commercial Press, Peking (2019)
Sagi, E., Kaufmann, S., Clark, B.: Semantic density analysis: comparing word meaning across time and phonetic space. In: Proceedings of the EACL 2009 Workshop on GEMS: Geometrical Models of Natural Language Semantics, March 2009, pp. 104–111 (2009)
Sagi, E., Kaufmann, S., Clark, B.: Tracing semantic change with Latent Semantic Analysis, pp. 161–183. De Gruyter Mouton, Berlin, Boston (2012)
Tang, X., Qu, W., Chen, X.: Semantic change computation: a successive approach. World Wide Web 19(3), 375–415 (2015). https://doi.org/10.1007/s11280-014-0316-y
Tang, X.: A state-of-the-art of semantic change computation. Natural Lang. Eng. 24(5), 649–676 (2018)
Rodda, M.A., Senaldi, M.S., Lenci, A.: Panta rei: tracking semantic change with distributional semantics in Ancient Greek. Italian J. Comput. Linguist. 3, 11–24 (2017)
Kim, Y., Chiu, Y.I., Hanaki, K., Hegde, D., Petrov, S.: Temporal analysis of language through neural language models. In: Danescu-Niculescu-Mizil, C., Eisenstein, J., McKeown, K., Smith, N.A. (eds.) Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, Baltimore, MD, USA, June 2014, pp. 61–65. Association for Computational Linguistics (2014)
Eger, S., Mehler, A.: On the linearity of semantic change: investigating meaning variation via dynamic graph models. In: Erk, K., Smith, N.A. (eds.) Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Berlin, Germany, August 2016, pp. 52–58. Association for Computational Linguistics (2016)
Gonen, H., Jawahar, G., Seddah, D., Goldberg, Y.: Simple, interpretable and stable method for detecting words with usage change across corpora. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020, pp. 538–555. Association for Computational Linguistics (2020)
Rosenfeld, A., Erk, K.: Deep neural models of semantic shift. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, June 2018, pp. 474–484. Association for Computational Linguistics (2018)
Yao, Z., Sun, Y., Ding, W., Rao, N., Xiong, H.: Dynamic word embeddings for evolving semantic discovery. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, February 2018. ACM (2018)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 601–608 (2001)
Lau, J.H., Cook, P., McCarthy, D., Newman, D., Baldwin, T.: Word sense induction for novel sense detection. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 2012, pp. 591–601. Association for Computational Linguistics (2012)
Lau, J.H., Cook, P., McCarthy, D., Newman, D., Baldwin, T.: Learning word sense distributions, detecting unattested senses and identifying novel senses using topic models. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, Maryland, June 2014, pp. 259–270. Association for Computational Linguistics (2014)
Mitra, S., Mitra, R., Riedl, M., Biemann, C., Mukherjee, A., Goyal, P.: That’s sick dude!: Automatic identification of word sense change across different timescales. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, Maryland, June 2014, pp. 1020–1029. Association for Computational Linguistics (2014)
Wiedemann, G., Remus, S., Chawla, A., Biemann, C.: Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. arXiv:1909.10430 (2019)
Pilehvar, M.T., Camacho-Collados, J.: WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019, pp. 1267–1273. Association for Computational Linguistics (2019)
Gulordava, K., Baroni, M.: A distributional similarity approach to the detection of semantic change in the Google Books ngram corpus. In: Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, Edinburgh, UK, July 2011, pp. 67–71. Association for Computational Linguistics (2011)
Diao, Y.: Xinshiqi Dalu Hanyu de Fazhan yu Biange
. Hung Yeh Publishing, Taibei (1995)Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., Hu, G.: Revisiting pre-trained models for Chinese natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, Online, November 2020, pp. 657–668. Association for Computational Linguistics (2020)
. Hung Yeh Publishing, Taibei (1995)Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chen, J., Qiu, L., Peng, B., Huang, CR. (2024). Does Bert Know How ‘Virus’ Evolved: Tracking Usage Changes in Chinese Textual Data. In: Dong, M., Hong, JF., Lin, J., Jin, P. (eds) Chinese Lexical Semantics. CLSW 2023. Lecture Notes in Computer Science(), vol 14515. Springer, Singapore. https://doi.org/10.1007/978-981-97-0586-3_10
Download citation
DOI: https://doi.org/10.1007/978-981-97-0586-3_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-0585-6
Online ISBN: 978-981-97-0586-3
eBook Packages: Computer ScienceComputer Science (R0)