Extraction of fuzzy logic rules from data by means of artificial neural networks

Holeňa, Martin

Kybernetika, Tome 41 (2005) no. 3, pp. 297-314

Abstract (VO)

The extraction of logical rules from data has been, for nearly fifteen years, a key application of artificial neural networks in data mining. Although Boolean rules have been extracted in the majority of cases, also methods for the extraction of fuzzy logic rules have been studied increasingly often. In the paper, those methods are discussed within a five-dimensional classification scheme for neural-networks based rule extraction, and it is pointed out that all of them share the feature of being based on some specialized neural network, constructed directly for the rule extraction task. As an important representative, a method for the extraction of rules in a general fuzzy disjunctive normal form is described in detail and illustrated on real-world applications. Finally, the paper proposes an algorithm demonstrating a principal possibility to extract fuzzy logic rules from multilayer perceptrons with continuous activation functions, i. e., from the kind of neural networks most universally used in applications. However, complexity analysis of the individual steps of that algorithm reveals that it involves computations with doubly-exponential complexity, due to which it can not without simplifications serve as a practically applicable alternative to methods based on specialized neural networks.

Détail

Classification : 03B52, 03E72, 62-07, 62M45, 68T30, 68T37
Keywords: knowledge extraction from data; artificial neural networks; fuzzy logic; Lukasiewicz logic; disjunctive normal form

Bibliographie

[1] Adamo J. M.: Data Mining for Association Rules and Sequential Patterns: Sequential and Parallel Algorithms. Springer–Verlag, Berlin 2001 | Zbl

[2] Aguzzoli S., Mundici D.: Weierstrass approximations by Lukasiewicz formulas with one quantified variable. In: 31st IEEE Internat. Symposium on Multiple-Valued Logic, 2001

[3] Alexander J. A., Mozer M. C.: Template-based procedures for neural network interpretation. Neural Networks 12 (1999), 479–498 | DOI

[4] Amato P., Nola, A. Di, Gerla B.: Neural networks and rational Lukasiewicz logic. J. Multiple-Valued Logic and Soft Computing (accepted for publication)

[5] Amato P., Porto M.: An algorithm for the automatic generation of logical formula representing a control law. Neural Network World 10 (2000), 777–786

[6] Andrews R., Diederich, J., Tickle A. B.: Survey and critique of techniques for extracting rules from trained artificical neural networks. Knowledge-based Systems 8 (1995), 378–389 | DOI

[7] Bern M., Chew L. P., Eppstein, D., Ruppert J.: Dihedral bounds for mesh generation in high dimensions. In: Proc. Sixth ACM-SIAM Symposium on Discrete Algorithms, ACM, San Francisco 1995, pp. 189–196 | MR | Zbl

[8] Berthold M., Hand D., editors: Intelligent Data Analysis. An Introduction. Springer–Verlag, Berlin 1999 | DOI | MR

[9] Chen J., Liu J.: Using mixture principal component analysis networks to extract fuzzy rules from data. Indust. Engrg. Chemistry Research 39 (2000), 2355–2367 | DOI

[10] Cignoli L. O., D’Ottaviano I. M. L., Mundici D.: Algebraic Foundations of Many-valued Reasoning. Kluwer Academic Publishers, Dordrecht 2000 | MR | Zbl

[11] Garcez A. S. d’Avila, Broda, K., Gabbay D. M.: Symbolic knowledge extraction from artificial neural networks: A sound approach. Artificial Intelligence 125 (2001), 155–207 | DOI | MR

[12] Daňková M., Perfilieva I.: Logical approximation II. Soft Computing 7 (2003), 228–233 | DOI | Zbl

[13] Raedt L. De: Interactive Theory Revision: An Inductive Logic Programming Approach. Academic Press, London 1992

[14] Duch W., Adamczak, R., Grabczewski K.: Extraction of logical rules from neural networks. Neural Processing Lett. 7 (1998), 211–219 | DOI

[15] Duch W., Adamczak, R., Grabczewski K.: A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Trans. Neural Networks 11 (2000), 277–306

[16] Dzeroski S., Lavrac N.: Relational Data Mining. Springer–Verlag, Berlin 2001 | Zbl

[17] Edelsbrunner H.: Algorithms in Combinatorial Geometry. Springer–Verlag, Heidelberg 1987 | MR | Zbl

[18] Esteva F., Godo, L., Montagna F.: The L$\Pi $ and L$\Pi \frac{1}{2}$ logics: Two complete fuzzy systems joining Lukasiewicz and product logic. Arch. Math. Logic 40 (2001), 39–67 | DOI | MR

[19] Faber J., Novák M., Svoboda, P., Tatarinov V.: Electrical brain wave analysis during hypnagogium. Neural Network World 13 (2003), 41–54

[20] Finn G. D.: Learning fuzzy rules from data. Neural Computing Appl. 8 (1999), 9–24 | DOI

[21] Freitas A. A.: Data Mining and Knowledge Discovery with Evolutionary Algorithms. Springer–Verlag, Berlin 2002 | Zbl

[22] Gehrke M., Walker C. L., Walker E. A.: Normal forms and truth tables for fuzzy logics. Fuzzy Sets and Systems 138 (2003), 25–51 | MR | Zbl

[23] Hájek P.: Metamathematics of Fuzzy Logic. Kluwer Academic Publishers, Dordrecht 1998 | MR | Zbl

[24] Hájek P., Havránek T.: Mechanizing Hypothesis Formation. Springer–Verlag, Berlin 1978 | MR | Zbl

[25] Healy M. J., Caudell T. P.: Acquiring rule sets as a product of learning in a logical neural architecture. IEEE Trans. Neural Networks 8 (1997), 461–474 | DOI

[26] Holeňa M.: Extraction of logical rules from data by means of piecewise-linear neural networks. In: Proc. 5th Internat. Conference on Discovery Science, Springer–Verlag, Berlin 2002, pp. 192–205 | Zbl

[27] Holeňa M., Baerns M.: Artificial neural networks in catalyst development. In: Experimental Design for Combinatorial and High Throughput Materials Development (J. N. Cawse, ed.), Wiley, Hoboken 2003, pp. 163–202

[28] Holeňa M., Baerns M.: Feedforward neural networks in catalysis. A tool for the approximation of the dependency of yield on catalyst composition, and for knowledge extraction. Catalysis Today 81 (2003), 485–494

[29] Horčík R., Cintula P.: Product Lukasiewicz logic. Arch. Math. Logic 43 (2004), 477–503 | DOI | MR | Zbl

[30] Ishikawa M.: Rule extraction by successive regularization. Neural Networks 13 (2000), 1171–1183 | DOI

[31] Lu H., Setiono, R., Liu H.: Effective data mining using neural networks. IEEE Trans. Knowledge and Data Engrg. 8 (1996), 957–961 | DOI

[32] Maire F.: Rule-extraction by backpropagation of polyhedra. Neural Networks 12 (1999), 717–725 | DOI

[33] McNaughton R.: A theorem about infinite-valued sentential logic. J. Symbolic Logic 16 (1951), 1–13 | DOI | MR | Zbl

[34] Mitra S., De R. K., Pal S. K.: Knowledge-based fuzzy MLP for classification and rule generation. IEEE Trans. Neural Networks 8 (1997), 1338–1350 | DOI

[35] Mitra S., Hayashi Y.: Neuro-fuzzy rule generation: Survey in soft computing framework. IEEE Trans. Neural Networks 11 (2000), 748–768 | DOI

[36] Muggleton S.: Inductive Logic Programming. Academic Press, London 1992 | Zbl

[37] Mundici D.: A constructive proof of McNaughton’s theorem in infinite-valued logic. J. Symbolic Logic 59 (1994), 596–602 | DOI | MR | Zbl

[38] Narazaki H., Watanabe, T., Yamamoto M.: Reorganizing knowledge in neural networks: An exploratory mechanism for neural networks in data classification problems. IEEE Trans. Systems Man Cybernet. 26 (1996), 107–117 | DOI

[39] Nauck D., Nauck, U., Kruse R.: Generating classification rules with the neuro-fuzzy system NEFCLASS. In: Proc. Biennial Conference of the North American Fuzzy Information Processing Society NAFIPS’96, 1996, pp. 466–470

[40] Novák V., Perfilieva I.: Some consequences of herbrand and McNaughton theorems in fuzzy logic. In: Discovering World with Fuzzy Logic: Perspectives and Approaches to Formalization of Human-Consistent Logical Systems (V. Novák and I. Perfilieva, eds.), Springer–Verlag, Heidelberg 1999, pp. 271–295 | MR

[41] Novák V., Perfilieva, I., Močkoř J.: Mathematical Principles of Fuzzy Logic. Kluwer Academic Publishers, Dordrecht 1999 | MR | Zbl

[42] Perfilieva I.: Neural nets and normal forms from fuzzy logic point of view. Technical Report, Institute for Research and Applications of Fuzzy Modelling, 2001

[43] Perfilieva I.: Normal forms for fuzzy logic functions and their approximation ability. Fuzzy Sets and Systems 124 (2001), 371–384 | MR | Zbl

[44] Perfilieva I.: Logical approximaion. Soft Computing 7 (2003), 73–78 | DOI

[45] Perfilieva I.: Normal forms in BL-algerbra and their contribution fo universal approximation of functions. Fuzzy Sets and Systems 143 (2004), 111–127 | DOI | MR

[46] Perfilieva I.: Normal forms in BL and L$\Pi $ algebras of functions. Soft Computing 8 (2004), 291–298 | DOI | Zbl

[47] Perfilieva I., Kreinovich V.: A new universal approximation result for fuzzy systems, which reflects CNF-DNF duality. Internat. J. Intelligent Systems 17 (2002), 1121–1130 | DOI | Zbl

[48] Polkowski L.: Rough Sets. Mathematical Foundations. Physica–Verlag, Heidelberg 2002 | MR | Zbl

[49] Quinlan J.: C4. 5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Francisco 1992 | Zbl

[50] Setiono R.: Extracting rules from neural networks by pruning and hidden unit splitting. Neural Computation 9 (1997), 05–225 | DOI | Zbl

[51] Tickle A. B., Andrews R., Golea, M., Diederich J.: The truth will come to light: Directions and challenges in extracting rules from trained artificial neural networks. IEEE Trans. Neural Networks 9 (1998), 1057–1068 | DOI

[52] Towell G. G., Shavlik J. W.: Extracting refined rules from knowledge-based neural networks. Mach. Learning 13 (1993), 71–101 | DOI

[53] Triantaphyllou E., (eds.) G. Felici: Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques. Kluwer Academic Publishers, Dordrecht 2003 | Zbl

[54] Tsukimoto H.: Extracting rules from trained neural networks. IEEE Trans. Neural Networks 11 (2000), 333–389 | DOI

[55] Wong M. L., Leung K. S.: Data Mining Using Grammar Based Genetic Programming and Applications. Kluwer Academic Publishers, Dordrecht 2000 | Zbl

[56] Zhang C., Zhang, S., Heymer B. E.: Association Rule Mining: Models and Algoritms. Springer–Verlag, Berlin 2002