Exponential expressivity of ${\rm ReLU}^k$ neural networks on Gevrey classes with point singularities

Opschoor, Joost A. A. ; Schwab, Christoph

Applications of Mathematics, Tome 69 (2024) no. 5, pp. 695-724

Abstract (VO)

We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains ${\rm D} \subset \mathbb R^d$, $d=2,3$. We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in ${\rm D}$, comprising the countably-normed spaces of I. M. Babuška and B. Q. Guo. \endgraf As intermediate result, we prove that continuous, piecewise polynomial high order (``$p$-version'') finite elements with elementwise polynomial degree $p\in \mathbb{N} $ on arbitrary, regular, simplicial partitions of polyhedral domains ${\rm D} \subset \mathbb R^d$, $d\geq 2$, can be \emph {exactly emulated} by neural networks combining ReLU and ReLU$^2$ activations. \endgraf On shape-regular, simplicial partitions of polytopal domains ${\rm D}$, both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the $hp$ finite element space of I. M. Babuška and B. Q. Guo.

Détail

DOI : 10.21136/AM.2024.0052-24

Classification : 41A25, 65N30
Keywords: neural network; $hp$-finite element method; singularities; Gevrey regularity; exponential convergence