Convergence of the partition function in the static word embedding model

K. Mynbaev ; Zh. Assylbekov

Eurasian mathematical journal, Tome 13 (2022) no. 4, pp. 70-81.

Résumé

We develop an asymptotic theory for the partition function of the word embeddings model WORD2VEC. The proof involves a study of properties of matrices, their determinants and distributions of random normal vectors when their dimension tends to infinity. The conditions imposed are mild enough to cover practically important situations. The implication is that for any word $i$ from a vocabulary $\mathcal{W}$, the context vector $\mathbf{c}_i$ is a reflection of the word vector $\mathbf{w}_i$ in approximately half of the dimensions. This allows us to halve the number of trainable parameters in static word embedding models.

Bibliographie

[1] C. Allen, I. Balazevic, T. M. Hospedales, “What the vec? towards probabilistically grounded embeddings”, Advances in Neural Information Processing Systems 32, Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 (Vancouver, BC, Canada, December 8-14, 2019), eds. Wallach H. M., Larochelle H., Beygelzimer A., d'Alché-Buc F., Fox E.B., Garnett R., 7465–7475

[2] C. Allen, T. M. Hospedales, “Analogies explained: Towards understanding word embeddings”, Proceedings of the 36th International Conference on Machine Learning, ICML 2019 (Long Beach, California, USA, 9-15 June 2019), Proceedings of Machine Learning Research, 97, eds. Chaudhuri K, Salakhutdinov R., 2019, 223–231

[3] S. Arora, Y. Li, Y. Liang, T. Ma, A. Risteski, “A latent variable model approach to pmi-based word embeddings”, Trans. Assoc. Comput. Linguistics, 4 (2016), 385–399 | DOI

[4] Zh. Assylbekov, R. Takhanov, “Context vectors are reflections of word vectors in half the dimensions”, J. Artif. Intell. Res., 66 (2019), 225–242 | DOI | MR

[5] J. Devlin, M. W. Chang, K. Lee, K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding”, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACLHLT 2019 (Minneapolis, MN, USA, June 2–7, 2019), v. 1, eds. Burstein J., Doran C., Solorio T., 4171–4186 | MR

[6] K. Ethayarajh, D. Duvenaud, G. Hirst, “Towards understanding linear word analogies”, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019 (Florence, Italy, July 28-August 2, 2019), v. 1, eds. Korhonen A., Traum D.R., Màrquez L., 2019, 3253–3262

[7] L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, E. Ruppin, “Placing search in context: the concept revisited”, Proceedings of the Tenth International World Wide Web Conference, WWW 10 (Hong Kong, China, May 2001), eds. Shen V. Y., Saito N., Lyu M.R., Zurko M. E., 406–414, 1–5

[8] A. Gittens, D. Achlioptas, M.-W. Mahoney, “Skip-gram zipf + uniform = vector additivity”, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017 (Vancouver, Canada, July 30–August 4, 2017), v. 1, eds. Barzilay R., Kan M. Y., 2017, 69–76 | DOI

[9] I. Gohberg, M. Kre–n, Introduction to the theory of linear nonselfadjoint operators, Translations of Mathematical Monographs, 18, 1968 | MR

[10] T. B. Hashimoto, D. Alvarez-Melis, T. S. Jaakkola, “Word embeddings as metric recovery in semantic spaces”, Trans. Assoc. Comput. Linguistics, 4 (2016), 273–286 | DOI

[11] A. Jacot, C. Hongler, F. Gabriel, “Neural tangent kernel: Convergence and generalization in neural networks”, Advances in Neural Information Processing Systems 31, Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018 (Montreal, Canada, December 3-8, 2018), eds. Bengio S., Wallach H.M., Larochelle H., Grauman K., Cesa-Bianchi N., Garnett R., 8580–8589

[12] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, J. Sohl-Dickstein, “Deep neural networks as gaussian processes”, Conference Track Proceedings, 6th International Conference on Learning Representations, ICLR 2018 (Vancouver, BC, Canada, April 30 May 3, 2018)

[13] O. Levy, Y. Goldberg, “Neural word embedding as implicit matrix factorization”, Advances in Neural Information Processing Systems 27, Annual Conference on Neural Information Processing Systems 2014 (Montreal, Quebec, Canada, December 8-13, 2014), eds. Ghahramani Z., Welling M., Cortes C., Lawrence N.D., Weinberger K.Q., 2177–2185

[14] T. Mikolov, K. Chen, G. Corrado, J. Dean, “Efficient estimation of word representations in vector space”, Workshop Track Proceedings, 1st International Conference on Learning Representations, ICLR 2013 (Scottsdale, Arizona, USA, May 2-4, 2013), eds. Bengio Y., LeCun Y.

[15] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, “Distributed representations of words and phrases and their compositionality”, Advances in Neural Information Processing Systems 26, 27th Annual Conference on Neural Information Processing Systems, Proceedings of a meeting (Lake Tahoe, Nevada, United States, December 5-8, 2013), eds. Burges C. J. C., Bottou L., Ghahramani Z., Weinberger K. Q., 2013, 3111–3119

[16] J. Pennington, R. Socher, C. D. Manning, “Glove: Global vectors for word representation”, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, A meeting of SIGDAT, a Special Interest Group of the ACL (Doha, Qatar, October 25-29, 2014), eds. Moschitti A., Pang B., Daelemans W., 1532–1543

[17] M. W. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, “Deep contextualized word representations”, Proceedings A shortened title of the paper 13 of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018 (New Orleans, Louisiana, USA, June 1–6, 2018), v. 1, eds. Walker M. A., Ji H., Stent A., 2018, 2227–2237

[18] A. C. Rencher, G. B. Schaalje, Linear models in statistics, 2nd ed, Wiley, New York, 2008 | MR | Zbl

[19] R. Tian, N. Okazaki, K. Inui, “The mechanism of additive composition”, Mach. Learn, 106:7 (2017), 1083–1130 | DOI | MR | Zbl

[20] Z. Zobnin, E. Elistratova, “Learning word embeddings without context vectors”, Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019 (Florence, Italy, August 2019), v. 2, eds. Augenstein I., Gella S., Ruder S., Kann K., Can B., Welbl J., Conneau A., Ren X., Rei M., 244–249 | MR