Abstract
In this research, we manually create high-quality datasets in the digital humanities domain for the evaluation of language models, specifically word embedding models. The first step comprises the creation of unigram and n-gram datasets for two fantasy novel book series for two task types each, analogy and doesn’t-match. This is followed by the training of models on the two book series with various popular word embedding model types such as word2vec, GloVe, fastText, or LexVec. Finally, we evaluate the suitability of word embedding models for such specific relation extraction tasks in a situation of comparably small corpus sizes. In the evaluations, we also investigate and analyze particular aspects such as the impact of corpus term frequencies and task difficulty on accuracy. The datasets, and the underlying system and word embedding models are available on github and can be easily extended with new datasets and tasks, be used to reproduce the presented results, or be transferred to other domains.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For Example: https://www.clarin-d.net/en/current-issues/lt4dh.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
-cbow 0 -size 200 -window 5 -negative 0 -hs 1 -sample 1e-3 -threads 12.
References
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: EMNLP, pp. 1532–1543 (2014)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016)
Salle, A., Idiart, M., Villavicencio, A.: Enhancing the LexVec distributed word representation model using positional contexts and external memory. CoRR abs/1606.01283 (2016)
Ghannay, S., Favre, B., Esteve, Y., Camelin, N.: Word embedding evaluation and combination. In: Calzolari, N., et al. (eds.) LREC 2016. ELRA (2016)
Wohlgenannt, G., Chernyak, E., Ilvovsky, D.: Extracting social networks from literary text with word embedding tools. In: Proceedings of Workshop LT4DH, Osaka, Japan, COLING 2016 (2016) 18–25
Laparra, E., Aldabe, I., Rigau, G.: From timelines to storylines: a preliminary proposal for evaluating narratives. In: Proceedings of the First Workshop on Computing News Storylines, pp. 50–55 (2015)
Flekova, L., Gurevych, I.: Personality profiling of fictional characters using sense-level links between lexical resources. In: EMNLP, pp. 1805–1816 (2015)
Bonato, A., D’Angelo, D.R., Elenberg, E.R., Gleich, D.F., Hou, Y.: Mining and modeling character networks. In: Bonato, A., Graham, F.C., Prałat, P. (eds.) WAW 2016. LNCS, vol. 10088, pp. 100–114. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49787-7_9
Hellrich, J., Hahn, U.: Bad company-neighborhoods in neural embedding spaces considered harmful. In: COLING, pp. 2785–2796 (2016)
Ustalov, D., Arefyev, N., Biemann, C., Panchenko, A.: Negative sampling improves hypernymy extraction based on projection learning. In: EACL 2017, p. 543 (2017)
Hamilton, W.L., Leskovec, J., Jurafsky, D.: Diachronic word embeddings reveal statistical laws of semantic change. arXiv preprint arXiv:1605.09096 (2016)
Kågeback, M., Johansson, F., Johansson, R., Dubhashi, D.: Neural context embeddings for automatic discovery of word senses. In: Proceedings of 1st Workshop on Vector Space Modeling for NLP, pp. 25–32 (2015)
Řehůřek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010, Valletta, Malta, ELRA, pp. 45–50 (2010)
Thoma, S., Rettinger, A., Both, F.: Towards holistic concept representations: embedding relational knowledge, visual attributes, and distributional word semantics. In: d’Amato, C., et al. (eds.) ISWC 2017. LNCS, vol. 10587, pp. 694–710. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68288-4_41
Acknowledgements
This work was supported by the Government of the Russian Federation (Grant 074-U01) through the ITMO Fellowship and Professorship Program. Furthermore, the article was prepared within the framework of the Basic Research Program at the National Research University Higher School of Economics (HSE) and supported within the framework of a subsidy by the Russian Academic Excellence Project ‘5-100’. It was supported by the RFBR grants 16-29-12982, 16-01-00583.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Wohlgenannt, G., Chernyak, E., Ilvovsky, D., Barinova, A., Mouromtsev, D. (2023). Relation Extraction Datasets in the Digital Humanities Domain and Their Evaluation with Word Embeddings. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2018. Lecture Notes in Computer Science, vol 13396. Springer, Cham. https://doi.org/10.1007/978-3-031-23793-5_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-23793-5_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23792-8
Online ISBN: 978-3-031-23793-5
eBook Packages: Computer ScienceComputer Science (R0)