Named entity recognition in Russian using multi-task LSTM-CRF

D. Mazitov   ; I. Alimova   ; E. Tutubalina

Zapiski Nauchnykh Seminarov POMI, Investigations on applied mathematics and informatics. Part I, Tome 499 (2021), pp. 222-235

Résumé

Named entity recognition (NER) is aimed at obtaining the important information from the unstructured data presented in the form of natural language texts. In this paper, we investigate the efficiency of modern multi-task NER approach on Russian corpora by employing several different NER datasets and a dataset of part-of-speech (POS) tags. We apply a state-of-the-art neural architecture based on bidirectional LSTMs and conditional random fields. Convolutional neural networks were utilized to learn character-level features. We carry out an extensive experimental evaluation over three standard datasets of news written in Russian. The proposed multi-task model achieve states-of-the-art results with an F1 score of 88.04% on Gareev's dataset and an F1 score of 99.49% on Person-1000 dataset.

Bibliographie

[1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., “Tensorflow: a system for large-scale machine learning”, OSDI, 16 (2016), 265–283

[2] C. Adak, B. B. Chaudhuri, M. Blumenstein, “Named entity recognition from unstructured handwritten document images”, 12th IAPR Workshop on Document Analysis Systems (DAS), 2016, 375–380

[3] L. T. Anh, M. Y. Arkhipov, M. S. Burtsev, Application of a hybrid bi-LSTM-CRF model to the task of Russian named entity recognition, Communications in Computer and Information Science book series – CCIS, 789, 2017

[4] L.T. Anh, M. Y. Arkhipov, M. S. Burtsev, Application of a hybrid bi-LSTM-CRF model to the task of Russian named entity recognition, 2017, arXiv: 1709.09686

[5] A. Y. Antonova, A. N. Soloviev, “Conditional random field models for the processing of Russian”, Communications of the ACM, 56:6 (2013) | Zbl

[6] M. Y. Arkhipov, M. S. Burtsev, L. T. Anh, “Application of a hybrid bi-LSTM-CRF model to the task of Russian named entity recognition”, Conference on Artificial Intelligence and Natural Language, Springer, Cham, 2017 | Zbl

[7] M. M. Brykina, A. V. Faynveyts, S. Yu. Toldova, “Dictionary-based ambiguity resolution in Russian named entities recognition”, International Workshop on Computational Linguistics and its Applications, v. 1, ed. A. Narin'yani, 2013 | Zbl

[8] R. Chalapathy, E. Z. Borzeshi, M. Piccardi, Bidirectional LSTM-CRF for clinical concept extraction, 2016, arXiv: 1611.08373

[9] J.P.C. Chiu, E. Nichols, “Named entity recognition with bidirectional LSTM-cnns”, Transactions of the Association for Computational Linguistics, 4 (2016), 357–370 | DOI

[10] L.G. Craidlin, “Program of allocation of Russian individualized nominal groups taglite”, Computational linguistics and intellectual technologies Dialog, 2005

[11] D. Kingma, J. Ba, “Adam: A method for stochastic optimization”, 3rd International Conference for Learning Representations (San Diego, 2014)

[12] C. Dong, J. Zhang, C. Zong, M. Hattori, H. Di, “Character-based LSTM-CRF with radical-level features for chinese named entity recognition”, Natural Language Understanding and Intelligent Applications, Springer, 2016, 239–250 | DOI

[13] R. Gareev, M. Tkachenko, V. Solovyev, A. Simanovsky, V. Ivanov, “Introducing baselines for Russian named entity recognition”, Computational Linguistics and Intelligent Text Processing, 2013

[14] A. Graves, S. Fernández, J. Schmidhuber, “Bidirectional LSTM networks for improved phoneme classification and recognition”, Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005, 2005

[15] K. Greff, R. K. Srivastava, J. Koutnik, B. R. Steunebrink, J. Schmidhuber, “LSTM: A search space odyssey”, IEEE Trans Neural Netw Learn Syst., 2016 | DOI | Zbl

[16] Z. Huang, W. Xu, K. Yu, Bidirectional LSTM-CRF models for sequence tagging, 2015, arXiv: 1508.01991

[17] Kaggle, Predict Russian universal dependencies POS tags, 2017

[18] G. Konoplich, E. Putin, A. Filchenkov, R. Rybka, “Named entity recognition in Russian with word representation learned by a bidirectional language model”, AINL, 2018

[19] G. Konoplich, E. Putin, A. Filchenkov, R. Rybka, “Named entity recognition in Russian with word representation learned by a bidirectional language model”, Conference on Artificial Intelligence and Natural Language, Springer, 2018, 48–58

[20] J. Lafferty, A. McCallum, F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data”, Proc. 18th International Conference on Machine Learning, 2001

[21] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer, “Neural architectures for named entity recognition”, Proc. 2016 NAACL, 2016, 260–270

[22] X. Ma, E. Hovy, End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF, 2016, arXiv: 1603.01354

[23] V. Malykh, A. Ozerin, “Reproducing Russian ner baseline quality without additional data”, CDUD@ CLA, 2016, 54–59

[24] S. Misawa, M. Taniguchi, Y. Miura, T. Ohkuma, “Character-based bidirectional LSTM-CRF with words and characters for japanese named entity recognition”, Proc. 1st Workshop on Subword and Character Level Models in NLP, 2017, 97–102 | DOI

[25] V. Mozharova, N. Loukachevitch, “Two-stage approach in Russian named entity recognition”, Proc. 2016 International FRUCT Conference on Intelligence, Social Media and Web, ISMW FRUCT, IEEE, 2016, 1–6

[26] A. V. Podobryaev, “Searching for person memories in news texts with the use of a model of conditional random fields”, RCDL 2013

[27] B. Popov, A. Kiryakov, D. Ognyanoff, D. Manov, A. Kirilov, “Kim — a semantic platform for information extraction and retrieval”, Journal of Natural Language Engineering, 10 (2004) | DOI | Zbl

[28] R. M. Zavala, P. Martinez, I. Segura-Bedmar, “A hybrid bi-LSTM-CRF model for knowledge recognition from ehealth documents”, TASS 2018: Workshop on Semantic Analysis at SEPLN, 2018, 65–70

[29] R. Ivanitskiy, A. Shipilo, L. Kovriguina, “Russian named entities recognition and classification using distributed word and phrase representations”, SIMBig, 2016

[30] A. V. Rubaylo, M. Y. Kosenko, Software utilities for natural language information retrievial, Almanac of modern science and education, 12, 2016

[31] E. Sheng, S. Miller, J.S. Ambite, P. Natarajan, “A neural named entity recognition approach to biological entity identification”, Proc. BioCreative VI Workshop, 2017, 24–27

[32] A. S. Starostin, V. V. Bocharov, S. V. Alexeeva, A. Bodrova, A. S. Chuchunkov, S. S. Dzhumaev, M. A. Nikolaeva, “Evaluation of named entity recognition and fact extraction systems for Russian”, Annual International Conference Dialogue, 2016

[33] A.A. Sysoev, I.A. Andrianov, “Named entity recognition in Russian: the power of wiki-based approach”, Proc. International Conference Dialogue, 2016, 746–755

[34] I. V. Trofimov, “Person name recognition in news articles based on the persons-1000/1111-f collections”, 16th All-Russian Scientific Conference Digital Libraries: Advanced Methods and Technologies, Digital Collections, RCDL, 2014, 217–221

[35] E. Tutubalina, S. Nikolenko, “Combination of deep recurrent neural networks and conditional random fields for extracting adverse drug reactions from user reviews”, Journal of Healthcare Engineering, 2017 (2017), 9451342 | DOI

[36] N. A. Vlasova, E. A. Suleymanova, I. V. Trofimov, “Report on Russian corpus for personal name retrieval”, Proceedings of computational and cognitive linguistics TEL, 2014

[37] Q. Wei, T. Chen, R. Xu, Y. He, L. Gui, Disease named entity recognition by combining conditional random fields and bidirectional recurrent neural networks, Database, 2016, 2016 | DOI