Word-embedding based text vectorization using clustering

V. I. Yuferev ; N. A. Razin

Modelirovanie i analiz informacionnyh sistem, Tome 28 (2021) no. 3, pp. 292-311.

Résumé

It is known that in the tasks of natural language processing, the representation of texts by vectors of fixed length using word-embedding models makes sense in cases where the vectorized texts are short. The longer the texts being compared, the worse the approach works. This situation is due to the fact that when using word-embedding models, information is lost when converting the vector representations of the words that make up the text into a vector representation of the entire text, which usually has the same dimension as the vector of a single word. This paper proposes an alternative way for using pre-trained word-embedding models for text vectorization. The essence of the proposed method consists in combining semantically similar elements of the dictionary of the existing text corpus by clustering their (dictionary elements) embeddings, as a result of which a new dictionary is formed with a size smaller than the original one, each element of which corresponds to one cluster. The original corpus of texts is reformulated in terms of this new dictionary, after which vectorization is performed on the reformulated texts using one of the dictionary approaches (TF-IDF was used in the work). The resulting vector representation of the text can be additionally enriched using the vectors of words of the original dictionary obtained by decreasing the dimension of their embeddings for each cluster. A series of experiments to determine the optimal parameters of the method is described in the paper, the proposed approach is compared with other methods of text vectorization for the text ranking problem — averaging word embeddings with TF-IDF weighting and without weighting, as well as vectorization based on TF-IDF coefficients.

Détail

Keywords: word embedding, TF-IDF, averaging, clustering, text similarity, text ranking.
Mots-clés : Fasttext, distance

Bibliographie

[1] P. Sitikhu, K. Pahi, P. Thapa, S. Shakya, “A comparison of semantic similarity methods for maximum human interpretability”, 2019 Artificial Intelligence for Transforming Business and Society (AITB), v. 1, 2019, 1-4 | DOI

[2] C. D. Manning, P. Raghavan, H. Schütze, Introduction to information retrieval, Cambridge University Press, USA, 2008 | Zbl

[3] C. De Boom, S. Van Canneyt, T. Demeester, B. Dhoedt, “Representation learning for very short texts using weighted word embedding aggregation”, Pattern Recogn. Lett., 80C (2016), 150–156 | DOI

[4] G. Kim, K. Cho, Length-adaptive transformer: train once with length drop, use anytime with search, 2020, arXiv: 2010.07003

[5] O. Zafrir, G. Boudoukh, P. Izsak, M. Wasserblat, Q8bert: quantized 8bit bert, 2019, arXiv: 1910.06188

[6] H. Gong, Y. Shen, D. Yu, J. Chen, D. Yu, “Recurrent chunking mechanisms for long-text machine reading comprehension”, Proceedings of the 58th annual meeting of the association for computational linguistics, Association for Computational Linguistics, 2020, 6751–6761 https://www.aclweb.org/anthology/2020.acl-main.603 | DOI

[7] Y. Qi, D. Sachan, M. Felix, S. Padmanabhan, G. Neubig, When and why are pre-trained word embeddings useful for neural machine translation?, Proceedings of the 2018 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, v. 2, short papers, Association for Computational Linguistics, New Orleans, Louisiana, 2018, 529–535 https://www.aclweb.org/anthology/N18-2084 | DOI

[8] D. Shen, G. Wang, W. Wang, M. R. Min, Q. Su, Y. Zhang, C. Li, R. Henao, L. Carin, “Baseline needs more love: on simple word-embedding-based models and associated pooling mechanisms”, Proceedings of the 56th annual meeting of the Association for Computational Linguistics, v. 1, long papers, Association for Computational Linguistics, Melbourne, Australia, 2018, 440–450 https://www.aclweb.org/anthology/P18-1041 | DOI

[9] A. Rücklé, S. Eger, M. Peyrard, I. Gurevych, Concatenated p-mean word embeddings as universal cross-lingual sentence representations, 2018, arXiv: 1803.01400

[10] P. Turney, P. Pantel, “From frequency to meaning: vector space models of semantics”, Journal of Artificial Intelligence Research, 37 (2010), 141–188 | DOI | MR | Zbl

[11] A. L. O. Shahmirzadi, K. Younge, “Text similarity in vector space models: a comparative study”, 2019 18th IEEE international conference on machine learning and applications, ICMLA, 2019, 659–666 | DOI

[12] V. Gupta, A. Kumar, P. Nokhiz, H. Gupta, P. Talukdar, “Improving document classification with multi-sense embeddings”, 24th European conference on artificial intelligence, ECAI 2020, 2020, 2030–2037 | DOI

[13] V. Mekala, Dheeraj, Gupta, B. Paranjape, H. Karnick, “SCDV: sparse composite document vectors using soft clustering over distributional representations”, Proceedings of the 2017 conference on empirical methods in natural language processing, Association for Computational Linguistics, Copenhagen, Denmark, 2017, 659–669 https://www.aclweb.org/anthology/D17-1069 | DOI

[14] V. Gupta, H. Karnick, A. Bansal, P. Jhala, “Product classification in e-commerce using distributional semantics”, Proceedings of COLING 2016, the 26th International conference on Computational Linguistics, technical papers, The COLING 2016 Organizing Committee, Osaka, Japan, 2016, 536–546 https://www.aclweb.org/anthology/C16-1052