Highly robust training of regularizedradial basis function networks

Kalina, Jan ; Vidnerová, Petra ; Janáček, Patrik

Kybernetika, Tome 60 (2024) no. 1, pp. 38-59

Abstract (VO)

Radial basis function (RBF) networks represent established tools for nonlinear regression modeling with numerous applications in various fields. Because their standard training is vulnerable with respect to the presence of outliers in the data, several robust methods for RBF network training have been proposed recently. This paper is interested in robust regularized RBF networks. A robust inter-quantile version of RBF networks based on trimmed least squares is proposed here. Then, a systematic comparison of robust regularized RBF networks follows, which is evaluated over a set of 405 networks trained using various combinations of robustness and regularization types. The experiments proceed with a particular focus on the effect of variable selection, which is performed by means of a backward procedure, on the optimal number of RBF units. The regularized inter-quantile RBF networks based on trimmed least squares turn out to outperform the competing approaches in the experiments if a highly robust prediction error measure is considered.

Détail

DOI : 10.14736/kyb-2024-1-0038

Classification : 62J02, 68T37, 68W25
Keywords: regression neural networks; robust training; effective regularization; quantile regression; robustness

Bibliographie

[1] Blokdyk, G.: Artificial Neural Network: A Complete Guide. Createspace Independent Publishing Platform, Scotts Valey 2017.

[2] Borş, A. G., Pitas, I.: Robust RBF networks. In: Radial basis function networks 1. (R. J. Howlett, L. C. Jain, and J. Kacprzyk, eds.). Recent developments in theory and applications, Physica Verlag Rudolf Liebing KG, Vienna 2001, pp. 123-133.

[3] Boudt, K., Rousseeuw, P. J., Vanduffel, S., Verdonck, T.: The minimum regularized covariance determinant estimator. Statist. Computing 30 (2020), 113-128. | DOI | MR

[4] Chatterjee, S., Hadi, A. S.: Regression Analysis by Example. (Fifth edition.). Wiley, Hoboken 2012.

[5] Cheng, Y. B., Chen, X. H., Li, H. L., Cheng, Z. Y., al., R. Jiang et: Analysis and comparison of Bayesian methods for measurement uncertainty evaluation. Math. Problems Engrg. (2018), 7509046. | DOI | MR

[6] Chollet, F.: Keras.

[7] Dodge, Y., J, Jurečková: Adaptive Regression. Springer, New York 2000. | MR

[8] Dong, J., Zhao, Y., Liu, C., Han, Z. F., Leung, C. S.: Orthogonal least squares based center selection for fault-tolerant RBF networks. Neurocomputing 339 (2019), 217-231. | DOI

[9] Dua, D., Graff, C.: UCI Machine Learning Repository.

[10] Egrioglu, E., Bas, E., Karahasan, O.: Winsorized dendritic neuron model artificial neural network and a robust training algorithm with Tukey's biweight loss function based on particle swarm optimization. Granular Comput. 8 (2023), 491-501. | DOI

[11] Fath, A. H., Madanifar, F., Abbasi, M.: Implementation of multilayer perceptron (MLP) and radial basis function (RBF) neural networks to predict solution gas-oil ratio of crude oil systems. Petroleum 6 (2020), 80-91. | DOI

[12] Hallin, M., Paindaveine, D., Šiman, M.: Multivariate quantiles and multiple-output regression quantiles: From $L_1$ optimization to halfspace depth. Ann. Statist. 38 (2010), 635-669. | DOI | MR | Zbl

[13] Han, I., Qian, X., Huang, H., Huang, T.: Efficient design of multicolumn RBF networks. Neurocomputing 450 (2021), 253-263. | DOI

[14] Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. (Second edition.). Springer, New York 2009. | MR

[15] Jurečková, J., Picek, J., Schindler, M.: Robust Statistical Methods with R. (Second edition.). Chapman and Hall/CRC, Boca Raton 2019. | MR

[16] Kalina, J., Tichavský, J.: On robust estimation of error variance in (highly) robust regression. Measurement Sci. Rev. 20 (2020), 6-14. | DOI

[17] Kalina, J., Neoral, A., Vidnerová, P.: Effective automatic method selection for nonlinear regression modeling. Int. J. Neural Syst. 31 (2021), 2150020. | DOI

[18] Kalina, J., Tumpach, J., Holeňa, M.: On combining robustness and regularization in training multilayer perceptrons over small data. In: 2000 International Joint Conference on Neural Networks (IJCNN), IEEE 2022. | DOI

[19] Karar, M. E.: Robust RBF neural network-based backstepping controller for implantable cardiac pacemakers. Int. J. Adaptive Control Signal Process. 32 (2018), 1040-1051. | DOI | MR

[20] Khan, I. A., Hussain, T., Ullah, A., Rho, S., Lee, M., M., Baik, S. W.: Towards efficient electricity forecasting in residential and commercial buildings: A novel hybrid CNN with a LSTM-AE based framework. Sensors 20 (2020), 1399. | DOI

[21] Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: NIPS'17: Proc. 31st International Conference on Neural Information Processing Systems, Curran Associates, New York 2017, pp. 972-981.

[22] Knefati, M. A., Chauvet, P. E., N'Guyen, S., Daya, B.: Reference curves estimation using conditional quantile and radial basis function network with mass constraint. Neural Process. Lett. 43 (2016), 17-30. | DOI

[23] Koenker, R.: Quantile regression: 40 years on. Annual Rev. Econom. 9 (2917), 155-176. | DOI | MR

[24] Kordos, M., Rusiecki, A.: Reducing noise impact on MLP training - Techniques and algorithms to provide noise-robustness in MLP network training. Soft Comput. 20 (2016), 46-65. | DOI

[25] Lee, C. C., Chung, P. C., Tsai, J. R., Chang, C. I.: Robust radial basis function neural networks. IEEE Trans. Systems Man Cybernet. B 29 (1999), 674-685. | DOI

[26] Li, X., Sun, Y.: Application of RBF neural network optimal segmentation algorithm in credit rating. Neural Computing Appl. 33 (2021), 8227-8235. | DOI

[27] Liu, Z., Leung, C. S., So, H. C.: Formal convergence analysis on deterministic $\ell_1$-regularization based mini-batch learning for RBF networks. Neurocomputing 532 (2023), 77-93. | DOI

[28] Standards, National Institute of, (NIST), Technology: Nonlinear Regression Datasets.

[29] Paul, C., Vishwakarma, G. K.: Back propagation neural networks and multiple regressions in the case of heteroscedasticity. Commun. Statist. Simul. Comput. 46 (2017), 6772-6789. | DOI | MR

[30] Petneházi, G.: Quantile convolutional neural networks for value at risk forecasting. Machine Learning Appl. 6 (2021), 100096. | DOI

[31] Poggio, T., Smale, S.: The mathematics of learning: Dealing with data. Notices Amer. Math. Soc. 50 (2003), 537-544. | DOI | MR

[32] Procházka, B.: Regression quantiles and trimmed least squares estimator in the nonlinear regression model. Comput. Statist. Data Anal. 6 (1988), 385-391. | DOI | MR

[33] Que, Q., Belkin, M.: Back to the future: Radial basis function network revisited. IEEE Trans. Pattern Anal. Machine Intell. 42 (2020), 1856-1867. | DOI

[34] Romano, Y., Patterson, E., Candès, E. J.: Conformalized quantile regression. ArXiv 2019. | arXiv

[35] Rusiecki, A.: Trimmed categorical cross-entropy for deep learning with label noise. Electron. Lett. 55 (2019), 319-320. | DOI

[36] Rusiecki, A.: Robust learning algorithm based on LTA estimator. Neurocomputing 120 (2013), 624-632. | DOI

[37] Saleh, A. K. M. E., Picek, J., Kalina, J.: R-estimation of the parameters of a multiple regression model with measurement errors. Metrika 75 (2012), 311-328. | DOI | MR

[38] Seghouane, A. K., Shokouhi, N.: Adaptive learning for robust radial basis function networks. IEEE Trans. Cybernet. 51 (2021), 2847-2856. | DOI

[39] Šíma, J., Vidnerová, P., Mrázek, V.: Energy complexity model for convolutional neural networks. Lecture Notes Computer Sci. 14263 (2023), 186-198. | DOI

[40] Su, M. J., Deng, W.: A fast robust learning algorithm for RBF network against outliers. Lecture Notes Computer Sci. 4113 (2006), 280-285. | DOI

[41] Sze, V., Chen, Y. B., Yang, T. J., Emer, J. S.: Efficient Processing of Deep Neural Networks. Morgan and Claypool, San Rafael 2020.

[42] Tukey, J.: Comparing individual means in the analysis of variance. Biometrics 5 (1949), 99-114. | DOI | MR

[43] Ullah, I., Youn, H. Y., Han, Y. H.: An efficient data aggregation and outlier detection scheme based on radial basis function neural network for WSN. J. Ambient Intell. Humanized Comput., in press (2022).

[44] Víšek, J.Á.: Consistency of the least weighted squares under heteroscedasticity. Kybernetika 47 (2011), 179-206. | MR | Zbl

[45] Werner, T.: Quantitative robustness of instance ranking problems. ArXiv 2021. | arXiv | MR

[46] Wilcox, R. R.: Introduction to Robust Estimation and Hypothesis Testing. Fourth edition. Academic Pres, London 2017. | MR

[47] Yang, C., Oh, S. K., Pedrycz, W., Fu, Z., Yang, B.: Design of reinforced fuzzy radial basis function neural network classifier driven with the aid of iterative learning techniques and support vector-based clustering. IEEE Trans. Fuzzy Systems 29 (2021), 2506-2520. | DOI

[48] Yerpude, P., Gudur, V.: Predictive modelling of crime dataset using data mining. Int. J. Data Mining Knowledge Management Process 7 (2017), 43-58. | DOI

[49] Zhang, D., Zang, G., Li, J., Ma, K., Liu, H.: Prediction of soybean price in China using QR-RBF neural network model. Computers Electron. Agriculture 154 (2018), 10-17. | DOI

[50] Zuo, Y.: New algorithms for computing the least trimmed squares estimator. ArXiv 2022. | arXiv