Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition
ESAIM. Proceedings, Tome 73 (2023), pp. 218-237
Cet article a éte moissonné depuis la source EDP Sciences
We design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a direct minimization problem, the solution of which can be computed via the Kronecker product singular value decomposition technique. Experimental results on the three standard deep auto-encoder benchmarks showed that they provide more accurate approximations to the FIM. Furthermore, they outperform KFAC and state-of-the-art first-order methods in terms of optimization speed.
Affiliations des auteurs :
Abdoulaye Koroko 1, 2 ; Ani Anciaux-Sedrakian 1 ; Ibtihel Ben Gharbia 1 ; Valérie Garès 3 ; Mounir Haddou 3 ; Quang Huy Tran 1
@article{EP_2023_73_a11,
author = {Abdoulaye Koroko and Ani Anciaux-Sedrakian and Ibtihel Ben Gharbia and Val\'erie Gar\`es and Mounir Haddou and Quang Huy Tran},
title = {Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition},
journal = {ESAIM. Proceedings},
pages = {218--237},
year = {2023},
volume = {73},
doi = {10.1051/proc/202373218},
language = {en},
url = {http://geodesic.mathdoc.fr/articles/10.1051/proc/202373218/}
}
TY - JOUR AU - Abdoulaye Koroko AU - Ani Anciaux-Sedrakian AU - Ibtihel Ben Gharbia AU - Valérie Garès AU - Mounir Haddou AU - Quang Huy Tran TI - Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition JO - ESAIM. Proceedings PY - 2023 SP - 218 EP - 237 VL - 73 UR - http://geodesic.mathdoc.fr/articles/10.1051/proc/202373218/ DO - 10.1051/proc/202373218 LA - en ID - EP_2023_73_a11 ER -
%0 Journal Article %A Abdoulaye Koroko %A Ani Anciaux-Sedrakian %A Ibtihel Ben Gharbia %A Valérie Garès %A Mounir Haddou %A Quang Huy Tran %T Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition %J ESAIM. Proceedings %D 2023 %P 218-237 %V 73 %U http://geodesic.mathdoc.fr/articles/10.1051/proc/202373218/ %R 10.1051/proc/202373218 %G en %F EP_2023_73_a11
Abdoulaye Koroko; Ani Anciaux-Sedrakian; Ibtihel Ben Gharbia; Valérie Garès; Mounir Haddou; Quang Huy Tran. Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition. ESAIM. Proceedings, Tome 73 (2023), pp. 218-237. doi: 10.1051/proc/202373218
Cité par Sources :