Explicative deep learning with probabilistic formal concepts in a natural language processing task

E. E. Vityaev   ; V. V. Martinovich

The Bulletin of Irkutsk State University. Series Mathematics, Tome 22 (2017), pp. 31-49

Résumé

Despite the high effectiveness of the Deep Learning methods, they remain a "thing in themselves", a "black box" decisions of which can not be trusted. This is critical for such areas as medicine, financial investments, military applications and others, where the price of the error is too high. In this regard, the European Union is going to demand in 2018 from companies that they give users an explanation of the solutions obtained by automatic systems. In this paper, we offer an alternative, logical-probabilistic method of Deep Learning that can explain its decisions. This is a method of hierarchical clustering, based on the original logical-probabilistic generalization of formal concepts [13]. For comparison with deep learning based on neural networks, the work [12] was chosen, in which the task of processing natural language on a set of data UMLS is solved. To apply the logical-probabilistic generalization of formal concepts, a classification algorithm based on the energy of the contradictions Energy Learning [10] is defined. Logical-probabilistic formal concepts are defined through fixed points, as well as the formal concepts themselves, only certain rules of probability are used as rules. The energy of contradictions allows us to resolve the contradictions arising at fixed points that form probabilistic formal concepts. It is shown that this clustering algorithm is not inferior in accuracy to the method of Deep Learning [12]; nevertheless, the solutions obtained by it are explained by the set of probabilistic fixed-point rules.

Détail

Keywords: probability, formal concepts, natural language processing, Deep Learning, Data Mining, semantic energy.

Bibliographie

[1] Vityaev E. E., Knowledge discovery. Computational cognition. Cognitive process models, Novosibirsk State University Press, Novosibirsk, 2006, 293 pp. (in Russian)

[2] Vorontsov K. V., “Combinatorial approach to learning algorithms estimation”, Matematicheskie voprosi kibernetiki, 13, ed. Lupanov O. B., FizMatLit, M., 2004, 5–36 (in Russian) | MR

[3] A. Bordes, J. Weston, R. Collobert, Y. Bengio, “Learning structured embeddings of knowledge bases”, Proceedings of the 25th Conference on Artificial Intelligence, AAAI-11 (USA, San Francisco, 2011)

[4] A. Bordes, X. Glorot, J. Weston, Y. Bengio, “Joint learning of words and meaning representations for open-text semantic parsing”, Proc. of the 15th Intern. Conf. on Artif. Intel. and Stat., JMLR, 22, 2012, 127–135

[5] A. Bordes et al., “A Semantic Matching Energy Function for Learning with Multi-relational Data”, Machine Learning, 2013, Special Issue on Learning Semantics

[6] Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin, “A Neural Probabilistic Language Model”, Machine Learning Research, 2003, 1137–1155 | Zbl

[7] B. Ganter, Formal Concept Analysis: Methods, and Applications in Computer Science, Springer, TU Dresden, 2003 | MR

[8] B. Ganter, S. Obiedkov, Implications in Triadic Formal Contexts, Springer, TU Dresden, 2004 | MR | Zbl

[9] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016 | MR | Zbl

[10] Y. LeCun et al., “A Tutorial on Energy-Based Learning”, Predicting Structured Outputs, eds. Bakir et al., MIT Press, 2006

[11] J. M. Lobo, A. Jimenez-Valverde, R. Real, “AUC: a misleading measure of the performance of predictive distribution models”, Global Ecology and Biogeography, 17 (2008), 145–151 | DOI

[12] A. T. McCray, “An upper level ontology for the biomedical domain”, Comparative and Functional Genomics, 4 (2003), 80–88 | DOI

[13] E. E. Vityaev, V. V. Martynovich, “Probabilistic Formal Concepts with Negation”, Perspectives of System Informatics, LNCS, 8974, eds. A. Voronkov, I. Virbitskaite, 2015, 385–399 | MR | Zbl