Geodesic
Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition
Abdoulaye Koroko1, 2 ; Ani Anciaux-Sedrakian1 ; Ibtihel Ben Gharbia1 ; Valérie Garès3 ; Mounir Haddou3 ; Quang Huy Tran1
1 IFP Energies nouvelles, 1 et 4 avenue de Bois Préau, 92852 Rueil-Malmaison Cedex, France
2 CentraleSupélec – Université Paris-Saclay, 3 Rue Joliot Curie, 91190 Gif-sur-Yvette, France
3 Univ Rennes, INSA, CNRS, IRMAR - UMR 6625, F-35000 Rennes, France
ESAIM. Proceedings, Tome 73 (2023), pp. 218-237.

Voir la notice de l'article provenant de la source EDP Sciences

We design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a direct minimization problem, the solution of which can be computed via the Kronecker product singular value decomposition technique. Experimental results on the three standard deep auto-encoder benchmarks showed that they provide more accurate approximations to the FIM. Furthermore, they outperform KFAC and state-of-the-art first-order methods in terms of optimization speed.
DOI : 10.1051/proc/202373218

Abdoulaye Koroko 1, 2 ; Ani Anciaux-Sedrakian 1 ; Ibtihel Ben Gharbia 1 ; Valérie Garès 3 ; Mounir Haddou 3 ; Quang Huy Tran 1

1 IFP Energies nouvelles, 1 et 4 avenue de Bois Préau, 92852 Rueil-Malmaison Cedex, France
2 CentraleSupélec – Université Paris-Saclay, 3 Rue Joliot Curie, 91190 Gif-sur-Yvette, France
3 Univ Rennes, INSA, CNRS, IRMAR - UMR 6625, F-35000 Rennes, France 
@article{EP_2023_73_a11,
     author = {Abdoulaye Koroko and Ani Anciaux-Sedrakian and Ibtihel Ben Gharbia and Val\'erie Gar\`es and Mounir Haddou and Quang Huy Tran},
     title = {Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition},
     journal = {ESAIM. Proceedings},
     pages = {218--237},
     publisher = {mathdoc},
     volume = {73},
     year = {2023},
     doi = {10.1051/proc/202373218},
     language = {en},
     url = {http://geodesic.mathdoc.fr/articles/10.1051/proc/202373218/}
}
TY  - JOUR
AU  - Abdoulaye Koroko
AU  - Ani Anciaux-Sedrakian
AU  - Ibtihel Ben Gharbia
AU  - Valérie Garès
AU  - Mounir Haddou
AU  - Quang Huy Tran
TI  - Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition
JO  - ESAIM. Proceedings
PY  - 2023
SP  - 218
EP  - 237
VL  - 73
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/articles/10.1051/proc/202373218/
DO  - 10.1051/proc/202373218
LA  - en
ID  - EP_2023_73_a11
ER  - 
%0 Journal Article
%A Abdoulaye Koroko
%A Ani Anciaux-Sedrakian
%A Ibtihel Ben Gharbia
%A Valérie Garès
%A Mounir Haddou
%A Quang Huy Tran
%T Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition
%J ESAIM. Proceedings
%D 2023
%P 218-237
%V 73
%I mathdoc
%U http://geodesic.mathdoc.fr/articles/10.1051/proc/202373218/
%R 10.1051/proc/202373218
%G en
%F EP_2023_73_a11
Abdoulaye Koroko; Ani Anciaux-Sedrakian; Ibtihel Ben Gharbia; Valérie Garès; Mounir Haddou; Quang Huy Tran. Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition. ESAIM. Proceedings, Tome 73 (2023), pp. 218-237. doi : 10.1051/proc/202373218. http://geodesic.mathdoc.fr/articles/10.1051/proc/202373218/

Cité par Sources :