Abstract
Here we present NYTK-NerKor, a gold standard Hungarian named entity annotated corpus containing 1 million tokens. This is the largest corpus ever in its kind. It contains balanced text selection from five genres: fiction, legal, news, web, and Wikipedia. A ca. 200,000 tokens subcorpus contains gold standard morphological annotation besides NE labels. We provide official train, development and test datasets in a proportion of 80%-10%-10%. All sets provide a balanced selection from all genres and sources, while the morphologically annotated subcorpus is also represented in all sets in a balanced way. The format of data files are CoNLL-U Plus, in which the NE annotation follows the CoNLL2002 labelling standard, while morphological information is encoded using the well-known Universal Dependencies POS tags and morphosyntactic features. The novelty of NYTK-NerKor as opposed to similar existing corpora is that it is: by an order of magnitude larger, freely available for any purposes, containing text material from different genres and sources, and following international standards in its format and tagset. The corpus is available under the license CC-BY-SA 4.0 from its GitHub repository: https://github.com/nytud/NYTK-NerKor.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
https://www.clarin.eu/resource-families/manually-annotated-corpora#Named20Entity20recognition
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
References
Barrault, L., et al.: Findings of the 2019 conference on machine translation (WMT19). In: Proceedings of the 4th Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 1–61. Association for Computational Linguistics (2019)
Eckart de Castilho, R., Mújdricza-Maydt, É., Yimam, S.M., Hartmann, S., Gurevych, I., Frank, A., Biemann, C.: A web-based tool for the integrated annotation of semantic and syntactic structures. In: Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), Osaka, Japan, pp. 76–84 (2016)
Csendes, D., Csirik, J., Gyimóthy, T., Kocsor, A.: The szeged treebank. In: Matoušek, V., Mautner, P., Pavelka, T. (eds.) TSD 2005. LNCS (LNAI), vol. 3658, pp. 123–131. Springer, Heidelberg (2005). https://doi.org/10.1007/11551874_16
Indig, B., Sass, B., Simon, E., Mittelholcz, I., Vadász, N., Makrai, M.: One format to rule them all - the emtsv pipeline for Hungarian. In: Proceedings of the 13th Linguistic Annotation Workshop, Florence, Italy, pp. 155–165. Association for Computational Linguistics (2019)
Nemeskey, D.M.: Natural Language Processing methods for Language Modeling. Ph.D. thesis, Eötvös Loránd University (2020)
Novák, A., Siklósi, B., Oravecz, Cs.: A new integrated open-source morphological analyzer for Hungarian. In: Proceedings of the 10th International Conference on Language Resources and Evaluation, LREC 2016. European Language Resources Association (ELRA) (2016)
Simon, E., Farkas, R., Halácsy, P., Sass, B., Szarvas, Gy., Varga, D.: A HunNER korpusz (The HunNER corpus). In: Alexin, Z., Csendes, D. (eds.) IV. Magyar Számítógépes Nyelvészeti Konferencia (4th Conference on Hungarian Computational Linguistics). Szeged (2006)
Simon, E., Lendvai, P., Németh, G., Olaszy, G., Vicsi, K.: A magyar nyelv a digitális korban - The Hungarian Language in the Digital Age. In: Rehm, G., Uszkoreit, H. (eds.) META-NET White Paper Series. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30379-1
Simon, E.: Approaches to Hungarian Named Entity Recognition. Ph.D. thesis, Ph.D. School in Cognitive Sciences, Budapest University of Technology and Economics (2013)
Simon, E., Nemeskey, D.M.: Automatically generated NE tagged corpora for English and Hungarian. In: Proceedings of the 4th Named Entity Workshop (NEWS) 2012, Jeju, Korea, pp. 38–46. Association for Computational Linguistics (2012)
Steinberger, R., et al.: An overview of the European Union’s highly multilingual parallel corpora file. Lang. Resour. Eval. 48, 679–707 (2014)
Steinberger, R., et al.: The JRC-Acquis: a multilingual aligned parallel corpus with 20+ languages. In: Proceedings of the 5th International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy (2006)
Szarvas, Gy., Farkas, R., Felföldi, L., Kocsor, A., Csirik, J.: A highly accurate Named Entity corpus for Hungarian. In: Electronic Proceedings of the 5th International Conference on Language Resources and Evaluation (2006)
Tjong Kim Sang, E.F.: Introduction to the CoNLL-2002 shared task: language-independent named entity recognition. In: Roth, D., van den Bosch, A. (eds.) Proceedings of CoNLL-2002, Taipei, Taiwan, pp. 155–158 (2002)
Tjong Kim Sang, E.F., De Meulder, F.: Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In: Daelemans, W., Osborne, M. (eds.) Proceedings of CoNLL-2003, Edmonton, Canada (2003)
Vadász, N.: KorKorpusz: kézzel annotált, többrétegű pilotkorpusz építése (The KorKor corpus: building of a manually annotated multi-layer pilot corpus). In: XVI. Magyar Számítógépes Nyelvészeti Konferencia (16th Conference on Hungarian Computational Linguistics), pp. 141–154. Szegedi Tudományegyetem, Szeged (2020)
Vadász, N., Simon, E.: Konverterek magyar morfológiai címkekészletek között (Converters between Hungarian morphological tagsets). In: Berend, G., Gosztolya, G., Vincze, V. (eds.) XV. Magyar Számítógépes Nyelvészeti Konferencia (15th Conference on Hungarian Computational Linguistics), pp. 99–111. Szegedi Tudományegyetem Informatikai Intézet, Szeged (2019)
Váradi, T.: The Hungarian national corpus. In: Proceedings of the 3rd International Conference on Language Resources and Evaluation, LREC-2002, Las Palmas de Gran Canaria, pp. 385–389. European Language Resources Association (2002)
Váradi, T., et al.: E-magyar – a digital language processing system. In: Calzolari, N., et al. (eds.) Proceedings of the 11th International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan. European Language Resources Association (ELRA) (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Simon, E., Vadász, N. (2021). Introducing NYTK-NerKor, A Gold Standard Hungarian Named Entity Annotated Corpus. In: Ekštein, K., Pártl, F., Konopík, M. (eds) Text, Speech, and Dialogue. TSD 2021. Lecture Notes in Computer Science(), vol 12848. Springer, Cham. https://doi.org/10.1007/978-3-030-83527-9_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-83527-9_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-83526-2
Online ISBN: 978-3-030-83527-9
eBook Packages: Computer ScienceComputer Science (R0)