Noise influence on recurrent neural network with nonlinear neurons

V. M. Moskvitin ; N. I. Semenova

Izvestiya VUZ. Applied Nonlinear Dynamics, Tome 31 (2023) no. 4, pp. 484-500.

Résumé

The purpose of this study is to establish the features of noise propagation and accumulation in a recurrent neural network using a simplified echo network as an example. In this work, we studied the influence of activation function of artificial neurons and the connection matrices between them. Methods. We have considered white Gaussian noise sources. We used additive, multiplicative and mixed noise depending on how the noise is introduced into artificial neurons. The noise impact was estimated using the dispersion (variance) of the output signal. Results. It is shown that the activation function plays a significant role in noise accumulation. Two nonlinear activation functions have been considered: the hyperbolic tangent and the sigmoid function with range form 0 to 1. It is shown that some types of noise are suppressed in the case of the second function. As a result of considering the influence of coupling matrices, it was found that diagonal coupling matrices with a large blurring coefficient lead to less noise accumulation in the echo network reservoir with an increase in the reservoir memory influence. Conclusion. It is shown that activation functions of the form of sigmoid with range from 0 to 1 are suitable for suppressing multiplicative and mixed noise. The accumulation of noise in the reservoir was considered for three types of coupling matrices inside the reservoir: a uniform matrix, a band matrix with a small blurring coefficient, and a band matrix with a large blurring coefficient. It has been found that the band matrix echo networks with a high blurring coefficient accumulates the least noise. This holds for both additive and multiplicative noise.

Détail

Keywords: neural networks, recurrent neural networks, echo-state networks, noise influence, white noise, nonlinear activation function.

Bibliographie

[1] LeCun Y., Bengio Y., Hinton G., “Deep learning”, Nature, 521:7553 (2015), 436–444 | DOI | MR

[2] Krizhevsky A., Sutskever I., Hinton G. E., “ImageNet classification with deep convolutional neural networks”, Commun. ACM, 60:6 (2017), 84–90 | DOI

[3] Maturana D., Scherer S., “VoxNet: A 3D Convolutional Neural Network for real-time object recognition”, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS (28 September 2015 - 02 October 2015, Hamburg, Germany), IEEE, New York, 2015, 922–928 | DOI

[4] Graves A., Mohamed A.-R., Hinton G., “Speech recognition with deep recurrent neural networks”, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (26-31 May 2013, Vancouver, BC, Canada), IEEE, New York, 2013, 6645–6649 | DOI

[5] Kar S., Moura J. M. F., “Distributed consensus algorithms in sensor networks with imperfect communication: Link failures and channel noise”, IEEE Transactions on Signal Processing, 57:1 (2009), 355–369 | DOI | MR | Zbl

[6] Mandic D. P., Chambers J. A., Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability, Wiley, New York, 2001, 304 pp. | DOI

[7] Bailador G., Roggen D., Tröster G., Triviño G., “Real time gesture recognition using continuous time recurrent neural networks”, 2nd International ICST Conference on Body Area Networks (11th–13th Jun 2007, Florence, Italy), ICST, 2007, 8 | DOI

[8] Hasler J., Marr H., “Finding a roadmap to achieve large neuromorphic hardware systems”, Frontiers in Neuroscience, 7 (2013), 118 | DOI

[9] Gupta S., Agrawal A., Gopalakrishnan K., Narayanan P., “Deep learning with limited numerical precision”, Proceedings of the 32nd International Conference on International Conference on Machine Learning (6-11 July 2015, Lille, France), v. 37, JMLR, 2015, 1737–1746.

[10] Karniadakis G. E., Kevrekidis I. G., Lu L., Perdikaris P., Wang S., Yang L., “Physics-informed machine learning”, Nature Reviews Physics, 3:6 (2021), 422–440 | DOI

[11] Brunner D., Soriano M. C., Mirasso C. R., Fischer I., “Parallel photonic information processing at gigabyte per second data rates using transient states”, Nature Communications, 4:1 (2013), 1364 | DOI

[12] Tuma T., Pantazi A., {Le Gallo} M., Sebastian A., Eleftheriou E., “Stochastic phase-change neurons”, Nature Nanotechnology, 11:8 (2016), 693–699 | DOI

[13] Torrejon J., Riou M., Araujo F. A., Tsunegi S., Khalsa G., Querlioz D., Bortolotti P., Cros V., Yakushiji K., Fukushima A., Kubota H., Yuasa S., Stiles M. D., Grollier J., “Neuromorphic computing with nanoscale spintronic oscillators”, Nature, 547:7664 (2017), 428–431 | DOI

[14] Psaltis D., Brady D., Gu X.-G., Lin S., “Holography in artificial neural networks”, Nature, 343:6256 (1990), 325–330 | DOI

[15] Bueno J., Maktoobi S., Froehly L., Fischer I., Jacquot M., Larger L., Brunner D., “Reinforcement learning in a large-scale photonic recurrent neural network”, Optica, 5:6 (2018), 756–760 | DOI | MR

[16] Lin X., Rivenson Y., Yardimci N. T., Veli M., Luo Y., Jarrahi M., Ozcan A., “All-optical machine learning using diffractive deep neural networks”, Science, 361:6406 (2018), 1004–1008 | DOI | MR | Zbl

[17] Shen Y., Harris N. C., Skirlo S., Prabhu M., {Baehr-Jones} T., Hochberg M., Sun X., Zhao S., Larochelle H., Englund D., Soljačić M., “Deep learning with coherent nanophotonic circuits”, Nature Photonics, 11:93 (2017), 441–446 | DOI

[18] Tait A. N., {de Lima} T. F., Zhou E., Wu A. X., Nahmias M. A., Shastri B. J., Prucnal P. R., “Neuromorphic photonic networks using silicon photonic weight banks”, Scientific Reports, 7:1 (2017), 7430 | DOI

[19] Moughames J., Porte X., Thiel M., Ulliac G., Larger L., Jacquot M., Kadic M., Brunner D., “Three-dimensional waveguide interconnects for scalable integration of photonic neural networks”, Optica, 7:6 (2020), 640–646 | DOI

[20] Dinc N. U., Psaltis D., Brunner D., “Optical neural networks The 3D connection”, Photoniques, 2020, no. 104, 34–38 | DOI

[21] Moughames J., Porte X., Larger L., Jacquot M., Kadic M., Brunner D., “3D printed multimode-splitters for photonic interconnects”, Opt. Mater. Express, 10:11 (2020), 2952–2961 | DOI

[22] Semenova N., Porte X., Andreoli L., Jacquot M., Larger L., Brunner D., “Fundamental aspects of noise in analog-hardware neural networks”, Chaos: An Interdisciplinary Journal of Nonlinear Science, 29:10 (2019), 103128 | DOI | MR | Zbl

[23] Semenova N., Larger L., Brunner D., “Understanding and mitigating noise in trained deep neural networks”, Neural Networks, 146 (2022), 151–160 | DOI

[24] Semenova N., Brunner D., “Noise-mitigation strategies in physical feedforward neural networks”, Chaos: An Interdisciplinary Journal of Nonlinear Science, 32:6 (2022), 061106 | DOI | MR

[25] Jaeger H., Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the “echo state network” approach GMD-Report 159, German National Research Center for Information Technology, Bonn, 2002, 48 pp.

[26] Prokhorov D., “Echo state networks: appeal and challenges”, Proceedings 2005 IEEE International Joint Conference on Neural Networks (31 July 2005 - 04 August 2005, Montreal, QC, Canada), v. 3, IEEE, New York, 2005, 1463–1466 | DOI

[27] Cerina L., Santambrogio M. D., Franco G., Gallicchio C., Micheli A., “EchoBay: Design and optimization of echo state networks under memory and time constraints”, ACM Transactions on Architecture and Code Optimization, 17:3 (2020), 1–24 | DOI

[28] Lukoševičius M., Jaeger H., “Reservoir computing approaches to recurrent neural network training”, Computer Science Review, 3:3 (2009), 127–149 | DOI | MR | Zbl