Deep learning for natural language processing: a survey

E. Arkhangelskaya ; S. Nikolenko

Zapiski Nauchnykh Seminarov POMI, Investigations on applied mathematics and informatics. Part I, Tome 499 (2021), pp. 137-205

Résumé

Over the last decade, deep learning has revolutionized machine learning. Neural network architectures have become the method of choice for many different applications; in this paper, we survey the applications of deep learning to natural language processing (NLP) problems. We begin by briefly reviewing the basic notions and major architectures of deep learning, including some recent advances that are especially important for NLP. Then we survey distributed representations of words, showing both how word embeddings can be extended to sentences and paragraphs and how words can be broken down further in character-level models. Finally, the main part of the survey deals with various deep architectures that have either arisen specifically for NLP tasks or have become a method of choice for them; the tasks include sentiment analysis, dependency parsing, machine translation, dialog and conversational models, question answering, and other applications. Disclaimer: this survey was written in 2016 and reflects the state of the art at the time. Although the field of deep learning moves very quickly, and all directions outlined here have already found many new developments, we hope that this survey can still be useful as an overview of already classical works in the field and a systematic introduction to deep learning for natural language processing.