Applicability of genetic algorithms for determining the weighting coefficients of an artificial neural network with one hidden layer

A. D. Smorodinov   ; T. V. Gavrilenko   ; V. A. Galkin

Vestnik KRAUNC. Fiziko-matematičeskie nauki, Tome 43 (2023) no. 2, pp. 69-86

Résumé

In the training of an artificial neural network, one of the central problems is the initial initialization and adjustment of weighting coefficients associated with pseudo-random initialization of weighting coefficients. The article describes a basic genetic algorithm, as well as a method for determining weight coefficients using this algorithm. A combined method for determining weighting coefficients is also presented, which provides for initial initialization using a genetic algorithm at the first stage and the use of stochastic gradient descent at the second stage of training, the proposed methods are tested on a number of artificial neural networks of direct propagation for various tasks of binary classification of real and synthetic data, as well as for unambiguous multiclass classification of handwritten digits on images from the database MNIST data. Artificial neural networks are constructed on the basis of the Kolmogorov-Arnold theorem. This article presents a comparative analysis of two methods for determining weight coefficients – using a genetic algorithm and gradient descent. Based on the results of the comparative analysis, it is concluded that a genetic algorithm can be used to determine the weighting coefficients both as an algorithm for the initial initialization of an artificial neural network and as an algorithm for adjusting the weighting coefficients.

Détail

Keywords: artificial neural networks, genetic algorithm, Kolmogorov-Arnold theorem, neural network training.

Bibliographie

[1] Galkin V. A., Gavrilenko T. V., Smorodinov A. D., “Nekotorye aspekty approksimatsii i interpolyatsii funktsii iskusstvennymi neironnymi setyami”, Vestnik KRAUNTs. Fiz.-mat. nauki, 38:1 (2022), 54-73 DOI: 10.26117/2079-6641-2022-38-1-54-73 | DOI | MR | Zbl

[2] Holland J. H., Adaptation in Natural and Artificial Systems: An Introductory Analysis With Applications to Biology, Control, and Artificial Intelligence, The MIT Press, Cambridge, 1992 | MR

[3] Goldberg D., Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley, Massachusetts, 1989 | Zbl

[4] Sholle F., Glubokoe obuchenie na Python, Piter, SPb, 2018, 400 pp.

[5] Zakharov I. V. , Shushakov A. O., Zykova S. S., “The Choice of Structures of Heterogeneous Information-Computer Systems Based on the Apparatus of Genetic Algorithms”, Intellektualnye tekhnologii na transporte, 2022, no. 3(31), 46-51 DOI: 10.24412/2413-2527-2022-331-46-51

[6] Bedin D. A., Ivanov A. G., “Ispolzovanie geneticheskogo algoritma dlya opredeleniya parametrov mnogogipoteznogo algoritma vosstanovleniya traektorii vozdushnogo sudna”, XXVI Sankt-Peterburgskaya mezhdunarodnaya konferentsiya po integrirovannym navigatsionnym sistemam, Sbornik materialov, Sankt-Peterburg, 27–29 maya 2019 goda, “Kontsern "Tsentralnyi nauchno-issledovatelskii institut "Elektropribor”, Sankt-Peterburg, 2019, 87-90

[7] Alkhussain A. Kh., “Simmetrichnyi algoritm shifrovaniya s pomoschyu geneticheskogo algoritma i generatorov psevdosluchainykh chisel”, Estestvennye i tekhnicheskie nauki, 2015, no. 7(85), 75-81

[8] Stefanyuk V. L., Alkhussain A. Kh., “Kriptografiya s simmetrichnym klyuchom s ispolzovaniem geneticheskogo algoritma”, KII-2014, Chetyrnadtsataya natsionalnaya konferentsiya po iskusstvennomu intellektu s mezhdunarodnym uchastiem, v. 1, RITs «Shkola», Kazan, 2014, 267-275

[9] Trokoz D. A., “Algoritm mashinnogo obucheniya shirokikh neironnykh setei s ispolzovaniem algebry giperrazmernykh dvoichnykh vektorov i geneticheskikh algoritmov”, Yuzhno-Sibirskii nauchnyi vestnik, 2020, no. 6(34), 148-154

[10] Nair V., Hinton G. E., “Rectified Linear Units Improve Restricted Boltzmann Machines”, 27th International Conference on International Conference on Machine Learning., Omnipress, USA, 2010, 807–814

[11] Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng., “Rectifier nonlinearities improve neural network acoustic models”, Proc. ICML, 30:1 (2013)

[12] Tieleman, Tijmen and Hinton, Geoffrey, Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural Networks for Machine Learning, 2012

[13] Kolmogorov A. N., “O predstavlenii nepreryvnykh funktsii neskolkikh peremennykh v vide superpozitsii nepreryvnykh funktsii odnogo peremennogo i slozheniya”, Dokl. AN SSSR, 114:5 (1957), 953–956 | Zbl

[14] Klambauer G., Unterthiner T., Mayr A., Hochreiter S., “Self-Normalizing Neural Networks”, Advances in Neural Information Processing Systems, 30:2017 (2017), arXiv: 170602515k

[15] Clevert, Djork-Arné; Unterthiner, Thomas Hochreiter, Fast and Accurate Deep Network Learning by Exponential Linear Units, 2015, arXiv: 1511.07289