Topic models with sentiment priors based on distributed representations

E. Tutubalina   ; S. I. Nikolenko

Zapiski Nauchnykh Seminarov POMI, Investigations on applied mathematics and informatics. Part I, Tome 499 (2021), pp. 284-301

Résumé

In recent works, topic models for aspect-based opinion mining have been extended to automatically train sentiment priors for topic-word distributions, leading to automated discovery of sentiment words and improved sentiment classification. In this work, we propose an approach where sentiment priors are trained in the space of word embeddings; this allows us to both discover more aspect-related sentiment words and further improve classification. We also present an experimental study that validates our results.

Bibliographie

[1] R. Al-Rfou, B. Perozzi, S. Skiena, “Polyglot: Distributed word representations for multilingual nlp”, Proc. 17th Conference on Computational Natural Language Learning (Sofia, Bulgaria, August 2013), Association for Computational Linguistics, 183–192

[2] N. Arefyev, A. Panchenko, A. Lukanin, O. Lesota, P. Romanov, “Evaluating three corpus-based semantic similarity systems for russian”, Proceedings of International Conference on Computational Linguistics, Dialogue 2015

[3] J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley, Y. Bengio, “Theano: a CPU and GPU math expression compiler”, Proc. Python for scientific computing conference, SciPy (Austin, TX, 2010), v. 4, 3

[4] D. M. Blei, J. D. Lafferty, “Dynamic topic models”, Proc. $23^{\text{rd}}$ International Conference on Machine Learning (New York, NY, USA), ACM, 2006, 113–120 | DOI

[5] D. M. Blei, J. D. McAuliffe, “Supervised topic models”, Advances in Neural Information Processing Systems, 22 (2007)

[6] D. M. Blei, A. Y. Ng, N. I. Jordan, “Latent Dirichlet allocation”, Journal of Machine Learning Research, 3:4–5 (2003), 993–1022 | Zbl

[7] Z. Cao, S. Li, Y. Liu, W. Li, H. Ji, “A novel neural topic model and its supervised extension”, Proc. 29th AAAI Conference on Artificial Intelligence (January 25–30, 2015, Austin, Texas, USA), 2015, 2210–2216

[8] J. Chang, D. M. Blei, “Hierarchical relational models for document networks”, Annals of Applied Statistics, 4:1 (2010), 124–150 | DOI | MR | Zbl

[9] Q. Diao, M. Qiu, C.-Y. Wu, A. J. Smola, J. Jiang, C. Wang, “Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars)”, Proc. 20th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2014, 193–202 | DOI

[10] Y. Goldberg, A primer on neural network models for natural language processing, 2015, arXiv: 1510.00726

[11] T. Griffiths, M. Steyvers, “Finding scientific topics”, Proceedings of the National Academy of Sciences, 101, Suppl. 1 (2004), 5228–5335 | DOI

[12] L. Hong, A. Ahmed, S. Gurumurthy, A. J. Smola, K. Tsioutsiouliklis, “Discovering geographical topics in the twitter stream”, Proc. 21st international conference on World Wide Web, ACM, 2012, 769–778 | DOI

[13] N. Kalchbrenner, E. Grefenstette, P. Blunsom, “A convolutional neural network for modelling sentences”, Proc. 52nd Annual Meeting of the Association for Computational Linguistics (Baltimore, Maryland, June 2014), v. 1, Long Papers, Association for Computational Linguistics, 655–665

[14] S. Kim, J. Zhang, Z. Chen, A. H. Oh, S. Liu, “A hierarchical aspect-sentiment model for online reviews”, Proc. Twenty-Seventh AAAI Conference on Artificial Intelligence (July 14–18, 2013, Bellevue, Washington, USA), 2013

[15] F. Li, S. Wang, S. Liu, M. Zhang, “Suit: A supervised user-item based topic model for sentiment analysis”, Proc. 28th AAAI Conference on Artificial Intelligence, 2014

[16] S. Z. Li, “Markov random field modeling in image analysis”, Advances in Pattern Recognition, Springer, Berlin–Heidelberg, 2009

[17] C. Lin, Y. He, R. Everson, S. Ruger, “Weakly supervised joint sentiment-topic detection from text”, IEEE Transactions on Knowledge and Data Engineering, 24:6 (2012), 1134–1145 | DOI

[18] B. Liu, Sentiment analysis: Mining opinions, sentiments, and emotions, Cambridge University Press, 2015

[19] B. Lu, M. Ott, C. Cardie, B. K. Tsou, “Multi-aspect sentiment analysis with topic models”, Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference, 2011, 81–88

[20] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, 2013, arXiv: 1301.3781

[21] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, 2013, arXiv: 1310.4546

[22] T. Minka, Estimating a dirichlet distribution, 2000

[23] S. Moghaddam, M. Ester, “On the design of LDA models for aspect-based opinion mining”, Proc. 21st ACM international conference on Information and knowledge management, ACM, 2012, 803–812

[24] A. Panchenko, N. V. Loukachevitch, D. Ustalov, D. Paperno, C. M. Meyer, N. Konstantinova, “Russe: The first workshop on Russian semantic similarity”, Proc. International Conference on Computational Linguistics and Intellectual Technologies, Dialogue (May, 2015), 89–105

[25] J. Pennington, R. Socher, C. Manning, “GloVe: Global vectors for word representation”, Proc. 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP (Doha, Qatar, October 2014), Association for Computational Linguistics, 2014, 1532–1543 | DOI

[26] D. Quercia, H. Askham, J. Crowcroft, “TweetLDA: supervised topic classification and link prediction in twitter”, WebSci, eds. Noshir S. Contractor, Brian Uzzi, N. W. Macy, Wolfgang Nejdl, ACM, 2012, 247–250 | DOI

[27] R. Řehůřek, P. Sojka, “Software Framework for Topic Modelling with Large Corpora”, Proc. LREC 2010 Workshop on New Challenges for NLP Frameworks (Valletta, Malta,), ELRA, May 2010, 45–50 (English) http://is.muni.cz/publication/884893/en

[28] D. O. Séaghdha, S. Teufel, “Unsupervised learning of rhetorical structure with un-topic models”, COLING 2014, 2–13

[29] I. Titov, R. McDonald, “Modeling online reviews with multi-grain topic models”, Proc. 17th International conference on World Wide Web, ACM, 2008, 111–120 | DOI

[30] E. Tutubalina, S. I. Nikolenko, “Inferring sentiment-based priors in topic models”, Proc. 14th Mexican International Conference on Artificial Intelligence, LNCS, 9414, Springer, 2015, 92–104

[31] C. Wang, D. M. Blei, D. Heckerman, “Continuous time dynamic topic models”, Proc. $24^{\text{th}}$ Conference on Uncertainty in Artificial Intelligence, 2008

[32] H. Wang, Y. Lu, C. Zhai, “Latent aspect rating analysis without aspect keyword supervision”, Proc. 17th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2011, 618–626 | DOI

[33] X. Wang, Y. Liu, C. Sun, B. Wang, X. Wang, “Predicting polarities of tweets by composing word embeddings with long short-term memory”, Proc. 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (July 2015, Beijing, China), v. 1, Long Papers, Association for Computational Linguistics, 1343–1353

[34] X. Wang, A. McCallum, “Topics over time: a non-Markov continuous-time model of topical trends”, Proc. $12^{\text{th}}$ ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA), ACM, 2006, 424–433 | DOI

[35] T. Wilson, J. Wiebe, P. Hoffmann, “Recognizing contextual polarity in phrase-level sentiment analysis”, Proc. conference on human language technology and empirical methods in natural language processing, Association for Computational Linguistics, 2005, 347–354

[36] Y. Wu, W. Wu, Z. Li, M. Zhou, Topic augmented neural network for short text conversation, 2016, arXiv: 1605.00090

[37] M. Yang, T. Cui, W. Tu, Ordering-sensitive and semantic-aware topic modeling, 2015, arXiv: 1502.0363

[38] Z. Yang, A. Kotov, A. Mohan, S. Lu, “Parametric and non-parametric user-aware sentiment topic models”, Proc. 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2015, 413–422

[39] J. Yohan, A. H. Oh, “Aspect and sentiment unification model for online review analysis”, Proc. 4th ACM International Conference on Web Search and Data Mining, WSDM '11 (New York, NY, USA), ACM, 2011, 815–824