Geodesic
Backtracking Gradient Descent Method and some Applications in Large Scale Optimisation Part 1: Theory
Minimax theory and its applications, Tome 7 (2022) no. 1
Cet article a éte moissonné depuis la source Minimax Theory and its Applications website

Voir la notice de l'article

Deep Neural Networks (DNN) are essential in many realistic applications, including Data Science. At the core of DNN is numerical optimisation, in particular gradient descent methods (GD). The purpose of this paper is twofold. First, we prove some new results on the backtracking variant of GD under very general situations. Second, we present a comprehensive comparison of our new results to the previously known results in the literature, providing pros and cons of these methods. To illustrate the efficiency of Backtracking line search, we will present some experimental results (on validation accuracy, training time and so on) on CIFAR10, based on implemetations developed in another paper by the authors. Source codes for the experiments are available on GitHub.
Mots-clés : Backtracking, deep learning, global convergence, gradient descent, line search method, optimisation, random dynamical systems 
@article{MTA_2022_7_1_a2,
     author = {Tuyen Trung Truong,Hang-Tuan Nguyen},
     title = {Backtracking {Gradient} {Descent} {Method} and some {Applications} in {Large} {Scale} {Optimisation} {Part} 1: {Theory}},
     journal = {Minimax theory and its applications},
     year = {2022},
     volume = {7},
     number = {1},
     url = {http://geodesic.mathdoc.fr/item/MTA_2022_7_1_a2/}
}
TY  - JOUR
AU  - Tuyen Trung Truong,Hang-Tuan Nguyen
TI  - Backtracking Gradient Descent Method and some Applications in Large Scale Optimisation Part 1: Theory
JO  - Minimax theory and its applications
PY  - 2022
VL  - 7
IS  - 1
UR  - http://geodesic.mathdoc.fr/item/MTA_2022_7_1_a2/
ID  - MTA_2022_7_1_a2
ER  - 
%0 Journal Article
%A Tuyen Trung Truong,Hang-Tuan Nguyen
%T Backtracking Gradient Descent Method and some Applications in Large Scale Optimisation Part 1: Theory
%J Minimax theory and its applications
%D 2022
%V 7
%N 1
%U http://geodesic.mathdoc.fr/item/MTA_2022_7_1_a2/
%F MTA_2022_7_1_a2
Tuyen Trung Truong,Hang-Tuan Nguyen. Backtracking Gradient Descent Method and some Applications in Large Scale Optimisation Part 1: Theory. Minimax theory and its applications, Tome 7 (2022) no. 1. http://geodesic.mathdoc.fr/item/MTA_2022_7_1_a2/