Backpropagation generalized delta rule for the selective attention Sigma-if artificial neural network

Huk, M.

International Journal of Applied Mathematics and Computer Science, Tome 22 (2012) no. 2, pp. 449-459.

Résumé

In this paper the Sigma-if artificial neural network model is considered, which is a generalization of an MLP network with sigmoidal neurons. It was found to be a potentially universal tool for automatic creation of distributed classification and selective attention systems. To overcome the high nonlinearity of the aggregation function of Sigma-if neurons, the training process of the Sigma-if network combines an error backpropagation algorithm with the self-consistency paradigm widely used in physics. But for the same reason, the classical backpropagation delta rule for the MLP network cannot be used. The general equation for the backpropagation generalized delta rule for the Sigma-if neural network is derived and a selection of experimental results that confirm its usefulness are presented.

Détail

Keywords: artificial neural networks, selective attention, self-consistency, error back-propagation, delta rule
Mots-clés : sztuczna sieć neuronowa, wsteczna propagacja błędu

Bibliographie

[1] Broadbent, D. (1982). Task combination and selective intake of information, Acta Psychologica 50(3): 253-290.

[2] Desimone, R. and Duncan, J. (1995). Neural mechanisms of selective visual-attention, Annual Review of Neuroscience 18(1): 193-222.

[3] Duch, W. and Jankowski, N. (1999). Survey of neural transfer functions, Neural Computing Surveys 2(1): 163-212.

[4] Durbin, R. and Rumelhart, D. (1990). Product units: A computationally powerful and biologically plausible extension to backpropagation networks, Neural Computation 1(1): 133-142.

[5] Feldman, J. and Ballard, D. (1982). Connectionist models and their properties, Cognitive Science 6(3): 205-254.

[6] Ferguene, F. and Toumi, F. F. (2009). Dynamic external force feedback loop control of a robot manipulator using a neural compensator-Application to the trajectory following in an unknown environment, International Journal of Applied Mathematics and Computer Science 19(1): 113-126, DOI: 10.2478/v10006-009-0011-9.

[7] Fonseca, L., Jimenez, J., Leburton, J. and Martin, R. (1998). Self-consistent calculation of the electronic structure and electron-electron interaction in self-assembled InAs-GaAs quantum dot structures, Physical Review B 57(7): 4017-4026.

[8] Gupta, M. (2008). Correlative type higher-order neural units with applications, IEEE International Conference on Automation and Logistics, ICAL 2008, Qingdao, China, pp. 715-718.

[9] Hager, G. and Toyama, K. (1999). Incremental focus of attention for robust visual tracking, International Journal of Computer Vision 35(1): 45-63.

[10] Houghton, G. and Tipper, S. (1996). Inhibitory mechanisms of neural and cognitive control: Applications to selective attention and sequential action, Brain and Cognition 30(1): 20-43.

[11] Huk, M. (2004). The sigma-if neural network as a method of dynamic selection of decision subspaces for medical reasoning systems, Journal of Medical Informatics Technologies 7(1): 65-73.

[12] Huk, M. (2006). Sigma-if neural network as a use of selective attention technique in classification and knowledge discovery problems solving, Annales UMCS Informatica AI 5(2): 121-131.

[13] Huk, M. (2009). Learning distributed selective attention strategies with the Sigma-if neural network, in M. Akbar and D. Hussain (Eds.), Advances in Computer Science and IT, In-Tech, Vukovar, pp. 209-232.

[14] Indiveri, G. (2008). Neuromorphic VLSI models of selective attention: From single chip vision sensors to multi-chip systems, Sensors 8(9): 5352-5375.

[15] Korbicz, J., Obuchowicz, A. and Uciński, D. (1994). Unidirectional networks, in L. Bolc (Ed.), Artificial Neural Networks: Foundations and Applications, Akademicka Oficyna Wydawnicza PLJ,Warsaw, pp. 35-58.

[16] Körding, K. and König, P. (2001). Neurons with two sites of synaptic integration learn invariant representations, Neural Computation 13(12): 2823-2849.

[17] Mel, B. (1990). The sigma-pi column: A model of associative learning in cerebral cortex, Technical report, CNS Memo 6, Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA.

[18] Mel, B. (1992). The clusteron: Toward a simple abstraction for a complex neuron, in J. Moody, S. Hanson and R. Lippmann (Eds.), Advances in Neural Information Processing Systems, Vol. 4, Morgan Kaufmann, San Mateo, CA, pp. 35-42.

[19] Neville, R. and Eldridge, S. (2002). Transformations of sigma-pi nets: Obtaining reflected functions by reflecting weight matrices, Neural Networks 15(3): 375-393.

[20] Niebur, E., Hsiao, S. and Johnson, K. (2002). Synchrony: A neuronal mechanism for attentional selection?, Current Opinion in Neurobiology 12(2): 190-194.

[21] Noh, T., Song, P. and Sievers, A. (1991). Self-consistency conditions for the effective-medium approximation in composite materials, Physical Review B 44(11): 5459-5464.

[22] Noton, D. and Stark, L. (1971). Scanpaths in saccadic eye movements while viewing and recognizing patterns, Vision Research 11(9): 929-942.

[23] Olshausen, B., Anderson, C. and Van Essen, D. (1993). A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information, The Journal of Neuroscience 13(11): 4700-4719.

[24] Pedro, J. O. and Dahunsi, O. A. (2011). Neural network based feedback linearization control of a servo-hydraulic vehicle suspension system, International Journal of Applied Mathematics and Computer Science 21(1): 137-147, DOI: 10.2478/v10006-011-0010-5.

[25] Raczkowski, D., Canning, A. and Wang, L. (2001). Thomas-fermi charge mixing for obtaining self-consistency in density functional calculations, Physical Review B 64(12): 121101-121105.

[26] Rumelhart, D., Hinton, G. and McClelland, J. (1986). A general framework for parallel distributed processing, in D. Rumelhart and J. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, Vol. 1, The MIT Press, Cambridge, MA, pp. 45-76.

[27] Stark, L., Privitera, C. and Azzariti, M. (2000). Locating regions-of-interest for the mars rover expedition, International Journal of Remote Sensing 21(17): 3327-3347.

[28] Treisman, A. (1960). Contextual cues in selective listening, Quarterly Journal of Experimental Psychology 12(4): 242-248.

[29] Tsotsos, J., Culhane, S. and Cutzu, F. (2001). From foundational principles to a hierarchical selection circuit for attention, in J. Braun, C. Koch and J. Davis (Eds.), Visual Attention and Cortical Circuits, MIT Press, Cambridge, MA, pp. 285-306.

[30] Vanrullen, R. and Koch, C. (2003). Visual selective behavior can be triggered by a feed-forward process, Journal of Cognitive Neuroscience 15(2): 209-217.

[31] Weber, C. and Wermter, S. (2007). A self-organizing map of sigma-pi units, Neurocomputing 70(13-15): 2552-2560.