Layer-wise knowledge distillation for simplified bipolar morphological neural networks
Informacionnye tehnologii i vyčislitelnye sistemy, no. 3 (2023), pp. 46-54
Voir la notice de l'article provenant de la source Math-Net.Ru
Various neuron approximations can be used to reduce the computational complexity of neural networks. One such approximation based on summation and maximum operations is a bipolar morphological neuron. This paper presents an improved structure of the bipolar morphological neuron that enhances its computational efficiency and a new approach to training based on continuous approximations of the maximum and knowledge distillation. Experiments were conducted on the MNIST dataset using a LeNet-like neural network architecture and on the CIFAR10 dataset using a ResNet-22 model architecture. The proposed training method achieves 99.45% classification accuracy on the LeNet-like model, with the same accuracy of the classical network, and 86.69% accuracy on the ResNet-22 model, compared to 86.43% accuracy of the classical model. The results show that the proposed method with logsum-exp (LSE) approximation of the maximum and layer-by-layer knowledge distillation, allows for a simplified bipolar morphological network that is not inferior to classical networks.
Keywords:
bipolar morphological networks, approximations, artificial neural networks, computational efficiency.
@article{ITVS_2023_3_a4,
author = {M. V. Zingerenko and E. E. Limonova},
title = {Layer-wise knowledge distillation for simplified bipolar morphological neural networks},
journal = {Informacionnye tehnologii i vy\v{c}islitelnye sistemy},
pages = {46--54},
publisher = {mathdoc},
number = {3},
year = {2023},
language = {ru},
url = {http://geodesic.mathdoc.fr/item/ITVS_2023_3_a4/}
}
TY - JOUR AU - M. V. Zingerenko AU - E. E. Limonova TI - Layer-wise knowledge distillation for simplified bipolar morphological neural networks JO - Informacionnye tehnologii i vyčislitelnye sistemy PY - 2023 SP - 46 EP - 54 IS - 3 PB - mathdoc UR - http://geodesic.mathdoc.fr/item/ITVS_2023_3_a4/ LA - ru ID - ITVS_2023_3_a4 ER -
%0 Journal Article %A M. V. Zingerenko %A E. E. Limonova %T Layer-wise knowledge distillation for simplified bipolar morphological neural networks %J Informacionnye tehnologii i vyčislitelnye sistemy %D 2023 %P 46-54 %N 3 %I mathdoc %U http://geodesic.mathdoc.fr/item/ITVS_2023_3_a4/ %G ru %F ITVS_2023_3_a4
M. V. Zingerenko; E. E. Limonova. Layer-wise knowledge distillation for simplified bipolar morphological neural networks. Informacionnye tehnologii i vyčislitelnye sistemy, no. 3 (2023), pp. 46-54. http://geodesic.mathdoc.fr/item/ITVS_2023_3_a4/