Noise Impact on a Recurrent Neural Network with

V. M. Moskvitin ; N. I. Semenova

Russian journal of nonlinear dynamics, Tome 19 (2023) no. 2, pp. 281-293.

Résumé

In recent years, more and more researchers in the field of artificial neural networks have been interested in creating hardware implementations where neurons and the connection between them are realized physically. Such networks solve the problem of scaling and increase the speed of obtaining and processing information, but they can be affected by internal noise. In this paper we analyze an echo state neural network (ESN) in the presence of uncorrelated additive and multiplicative white Gaussian noise. Here we consider the case where artificial neurons have a linear activation function with different slope coefficients. We consider the influence of the input signal, memory and connection matrices on the accumulation of noise. We have found that the general view of variance and the signal-to-noise ratio of the ESN output signal is similar to only one neuron. The noise is less accumulated in ESN with a diagonal reservoir connection matrix with a large “blurring” coefficient. This is especially true of uncorrelated multiplicative noise.

Détail

Keywords: artificial neural networks, recurrent neural network, echo state network, statistic, white gaussian noise.
Mots-clés : noise, dispersion

Bibliographie

[1] LeCun, Y., Bengio, Y., and Hinton, G., “Deep Learning”, Nature, 521:7553 (2015), 436–444 | DOI | MR

[2] Maturana, D. and Scherer, S., “VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition”, IEEE/RSJ Internat. Conf. on Intelligent Robots and Systems (IROS, Hamburg, Germany, 2015), 922–928

[3] Krizhevsky, A., Sutskever, I., and Hinton, G. E., “ImageNet Classification with Deep Convolutional Neural Networks”, Commun. ACM, 60:6 (2017), 84–90 | DOI

[4] Graves, A., Mohamed, A., and Hinton, G., “Speech Recognition with Deep Recurrent Neural Networks”, IEEE Internat. Conf. on Acoustics, Speech and Signal Processing (Vancouver, British Columbia, Canada, May 2013), 6645–6649

[5] Kar, S. and Moura, J. M. F., “Distributed Consensus Algorithms in Sensor Networks with Imperfect Communication: Link Failures and Channel Noise”, IEEE Trans. Signal Process., 57:1 (2009), 355–369 | DOI | MR | Zbl

[6] Brunner, D., Soriano, M. C., Mirasso, C. R., and Fischer, I., “Parallel Photonic Information Processing at Gigabyte per Second Data Rates Using Transient States”, Nat. Commun., 4:1 (2013), 1364, 6 pp. | DOI

[7] Tuma, T., Pantazi, A., Le Gallo, M., Sebastian, A., and Eleftheriou, E., “Stochastic Phase-Change Neurons”, Nat. Nanotechnol., 11:8 (2016), 693–699 | DOI

[8] Torrejon, J., Riou, M., Araujo, F. A., Tsunegi, S., Khalsa, G., Querlioz, D., Bortolotti, P., Cros, V., Yakushiji, K., Fukushima, A., Kubota, H., Yuasa, S., Stiles, M. D., and Grollier, J., “Neuromorphic Computing with Nanoscale Spintronic Oscillators”, Nature, 547:7664 (2017), 428–431 | DOI

[9] Psaltis, D., Brady, D., Gu, X.-G., and Lin, S., “Holography in Artificial Neural Networks”, Nature, 343:6256 (1990), 325–330 | DOI

[10] Bueno, J., Maktoobi, S., Froehly, L., Fischer, I., Jacquot, M., Larger, L., and Brunner, D., “Reinforcement Learning in a Large Scale Photonic Recurrent Neural Network”, Optica, 5:6 (2018), 756–760 | DOI | MR

[11] Lin, X., Rivenson, Y., Yardimci, N. T., Veli, M., Luo, Y., Jarrahi, M., and Ozcan, A., “All-Optical Machine Learning Using Diffractive Deep Neural Networks”, Science, 361:6406 (2018), 1004–1008 | DOI | MR | Zbl

[12] Shen, Y., Harris, N. C., Skirlo, S., Prabhu, M., Baehr-Jones, T., Hochberg, M., Sun, X., Zhao, S., Larochelle, H., Englund, D., and Soljacic, M., “Deep Learning with Coherent Nanophotonic Circuits”, Nature Photon., 11:7 (2017), 441–446 | DOI

[13] Tait, A. N., De Lima, T. F., Zhou, E., Wu, A. X., Nahmias, M. A., Shastri, B. J., and Prucnal, P. R., “Neuromorphic Photonic Networks Using Silicon Photonic Weight Banks”, Sci. Rep., 7:1 (2017), 7430, 10 pp. | DOI | MR

[14] Moughames, J., Porte, X., Thiel, M., Ulliac, G., Larger, L., Jacquot, M., Kadic, M., and Brunner, D., “Three-Dimensional Waveguide Interconnects for Scalable Integration of Photonic Neural Networks”, Optica, 7:6 (2020), 640–646 | DOI

[15] Dinc, N. U., Psaltis, D., and Brunner, D., “Optical Neural Networks: The 3D Connection”, Photoniques, 2020, no. 104, 34–38 | DOI

[16] Moughames, J., Porte, X., Larger, L., Jacquot, M., Kadic, M., and Brunner, D., “3D Printed Multimode-Splitters for Photonic Interconnects”, Opt. Mater. Express, 10:11 (2020), 2952–2961 | DOI

[17] Mourgias-Alexandris, G., Moralis-Pegios, M., Tsakyridis, A., Simos, S., Dabos, G., Totovic, A., Passalis, N., Kirtas, M., Rutirawut, T., Gardes, F. Y., Tefas, A., and Pleros, N., “Noise-Resilient and High-Speed Deep Learning with Coherent Silicon Photonics”, Nat. Commun., 13:1 (2022), 5572, 7 pp. | DOI

[18] Wang, T., Ma, S.-Y., Wright, L. G., Onodera, T., Richard, B. C., and McMahon, P. L., “An Optical Neural Network Using Less Than $1$ Photon per Multiplication”, Nat. Commun., 13:1 (2022), 123, 7 pp. | DOI

[19] Semenova, N., Porte, X., Andreoli, L., Jacquot, M., Larger, L., and Brunner, D., “Fundamental Aspects of Noise in Analog-Hardware Neural Networks”, Chaos, 29:10 (2019), 103128, 11 pp. | DOI | MR | Zbl

[20] Semenova, N., Larger, L., and Brunner, D., “Understanding and Mitigating Noise in Trained Deep Neural Networks”, Neural Netw., 146 (2022), 151–160 \itemsep=2pt | DOI

[21] Semenova, N. and Brunner, D., “Noise-Mitigation Strategies in Physical Feedforward Neural Networks”, Chaos, 32:6 (2022), 061106, 10 pp. | DOI | MR

[22] Johnson, D. H., “Signal-to-Noise Ratio”, Scholarpedia, 1:12 (2006), 2088 | DOI

[23] Lukoševičius, M. and Jaeger, H., “Reservoir Computing Approaches to Recurrent Neural Network Training”, Comput. Sci. Rev., 3:3 (2009), 127–149 | DOI | MR | Zbl