On the predictive power of meta-features in OpenML

Bilalli, B. ; Abelló, A. ; Aluja-Banet, T.

International Journal of Applied Mathematics and Computer Science, Tome 27 (2017) no. 4, pp. 697-712.

Résumé

The demand for performing data analysis is steadily rising. As a consequence, people of different profiles (i.e., nonexperienced users) have started to analyze their data. However, this is challenging for them. A key step that poses difficulties and determines the success of the analysis is data mining (model/algorithm selection problem). Meta-learning is a technique used for assisting non-expert users in this step. The effectiveness of meta-learning is, however, largely dependent on the description/characterization of datasets (i.e., meta-features used for meta-learning). There is a need for improving the effectiveness of meta-learning by identifying and designing more predictive meta-features. In this work, we use a method from exploratory factor analysis to study the predictive power of different meta-features collected in OpenML, which is a collaborative machine learning platform that is designed to store and organize meta-data about datasets, data mining algorithms, models and their evaluations. We first use the method to extract latent features, which are abstract concepts that group together meta-features with common characteristics. Then, we study and visualize the relationship of the latent features with three different performance measures of four classification algorithms on hundreds of datasets available in OpenML, and we select the latent features with the highest predictive power. Finally, we use the selected latent features to perform meta-learning and we show that our method improves the meta-learning process. Furthermore, we design an easy to use application for retrieving different meta-data from OpenML as the biggest source of data in this domain.

Détail

Keywords: feature extraction, feature selection, meta learning
Mots-clés : ekstrakcja danych, selekcja danych, uczenie maszynowe

Bibliographie

[1] Bensusan, H. and Giraud-Carrier, C. (2000a). Casa batló is in passeig de grácia or how landmark performances can describe tasks, Proceedings of the ECML-00 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, Barcelona, Spain, pp. 29–46.

[2] Bensusan, H. and Giraud-Carrier, C.G. (2000b). Discovering task neighbourhoods through landmark learning performances, Proceedings of the European Conference on Principles of Data Mining and Knowledge Discovery, Lyon, France, pp. 325–330.

[3] Bensusan, H., Giraud-Carrier, C.G. and Kennedy, C.J. (2000). A higher-order approach to meta-learning, Proceedings of the International Conference on Inductive Logic Programming, London, UK.

[4] Bensusan, H. and Kalousis, A. (2001). Estimating the predictive accuracy of a classifier, in M. Bramer (Ed.), Principles of Data Mining. Undergraduate Topics in Computer Science, Springer, London, pp. 25–36.

[5] Bilalli, B., Abelló, A., Aluja-Banet, T. and Wrembel, R. (2016). Towards intelligent data analysis: The metadata challenge, Proceedings of the International Conference on Internet of Things and Big Data, Rome, Italy, pp. 331–338.

[6] Bilalli, B., Abelló, A., Aluja-Banet, T. and Wrembel, R. (2017). Intelligent assistance for data pre-processing, Computer Standards Interfaces, DOI: 10.1016/j.csi.2017.05.004, (in press).

[7] Brazdil, P., Gama, J.A. and Henery, B. (1994). Characterizing the applicability of classification algorithms using meta-level learning, Proceedings of the European Conference on Machine Learning, Catania, Italy, pp. 83–102.

[8] Brazdil, P., Giraud-Carrier, C., Soares, C. and Vilalta, R. (2008). Metalearning: Applications to Data Mining, 1st Edn., Springer Publishing Company, Berlin.

[9] Castiello, C., Castellano, G. and Fanelli, A.M. (2005). Meta-data: Characterization of input features for meta-learning, Proceedings of the International Conference on Modeling Decisions for Artificial Intelligence, Tsukuba, Japan, pp. 457–468.

[10] Fayyad, U.M., Piatetsky-Shapiro, G. and Smyth, P. (1996). From data mining to knowledge discovery in databases, AI Magazine 17(3): 1–34.

[11] Fürnkranz, J. and Petrak, J. (2001). An evaluation of landmarking variants, Proceedings of the ECML/PKDD Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning, Freiburg, Germany, pp. 57–68.

[12] Giraud-Carrier, C. (2005). The data mining advisor: Meta-learning at the service of practitioners, Proceedings of the International Conference on Machine Learning and Applications, Los Angeles, CA, USA, pp. 113–119.

[13] Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components, Journal of Educational Psychology 24(6): 417–441.

[14] Kaiser, H.F. (1958). The varimax criterion for analytic rotation in factor analysis, Psychometrika 23(3): 187–200.

[15] Kalousis, A. and Hilario, M. (2001). Feature selection for meta-learning, Proceedings of the International Conference on Knowledge Discovery and Data Mining, Hong Kong, China, pp. 222–233.

[16] Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection, Proceedings of the International Joint Conference on Artificial Intelligence, Montréal, Québec, Canada, pp. 1137–1143.

[17] Lemke, C., Budka, M. and Gabrys, B. (2015). Metalearning: A survey of trends and technologies, Artificial Intelligence Review 44(1): 117–130.

[18] Michie, D., Spiegelhalter, D.J., Taylor, C.C. and Campbell, J. (Eds.) (1994). Machine Learning, Neural and Statistical Classification, Ellis Horwood, Chichester.

[19] Morchid, M., Dufour, R., Bousquet, P., Linarès, G. and Torres-Moreno, J. (2014). Feature selection using principal component analysis for massive retweet detection, Pattern Recognition Letters 49: 33–39.

[20] Peng, Y., Flach, P.A., Soares, C. and Brazdil, P. (2002). Improved dataset characterisation for meta-learning, Proceedings of the 5th International Conference on Discovery Science, Lübeck, Germany, pp. 141–152.

[21] Pfahringer, B., Bensusan, H. and Giraud-Carrier, C.G. (2000). Meta-learning by landmarking various learning algorithms, Proceedings of the 17th International Conference on Machine Learning, San Francisco, CA, USA, pp. 743–750.

[22] Reif, M., Shafait, F. and Dengel, A. (2012). Meta2-features: Providing meta-learners more information, 35th German Conference on Artificial Intelligence, Staarbrücken, Germany.

[23] Reif, M., Shafait, F., Goldstein, M., Breuel, T. and Dengel, A. (2014). Automatic classifier selection for non-experts, Pattern Analysis and Applications 17(1): 83–96.

[24] Rendell, L., Seshu, R. and Tcheng, D. (1987). Layered concept-learning and dynamically-variable bias management, Proceedings of the International Joint Conference on Artificial Intelligence, Milan, Italy, pp. 308–314.

[25] Serban, F., Vanschoren, J., Kietz, J. and Bernstein, A. (2013). A survey of intelligent assistants for data analysis, ACM Computing Surveys 45(3): 31:1–31:35.

[26] Sohn, S.Y. (1999). Meta analysis of classification algorithms for pattern recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 21(11): 1137–1144.

[27] Todorovski, L., Brazdil, P. and Soares, C. (2000). Report on the experiments with feature selection in meta-level learning, Proceedings of the PKDD Workshop on Data Mining, Lyon, France, pp. 27–39.

[28] Todorovski, L. and Dzeroski, S. (1999). Experiments in meta-level learning with ILP, Proceedings of the European Conference on Principles of Data Mining and Knowledge Discovery, Prague, Czech Republic, pp. 98–106.

[29] Vanschoren, J., van Rijn, J.N., Bischl, B. and Torgo, L. (2014). OpenML: Networked science in machine learning, ACM SIGKDD Explorations Newsletter 15(2): 49–60.