Collocation approximation by deep neural ReLU networks for parametric and stochastic PDEs with lognormal inputs

Dinh Dũng

Sbornik. Mathematics, Tome 214 (2023) no. 4, pp. 479-515

Résumé

We find the convergence rates of the collocation approximation by deep ReLU neural networks of solutions to elliptic PDEs with lognormal inputs, parametrized by $\boldsymbol{y}$ in the noncompact set ${\mathbb R}^\infty$. The approximation error is measured in the norm of the Bochner space $L_2({\mathbb R}^\infty, V, \gamma)$, where $\gamma$ is the infinite tensor-product standard Gaussian probability measure on ${\mathbb R}^\infty$ and $V$ is the energy space. We also obtain similar dimension-independent results in the case when the lognormal inputs are parametrized by ${\mathbb R}^M$ of very large dimension $M$, and the approximation error is measured in the $\sqrt{g_M}$-weighted uniform norm of the Bochner space $L_\infty^{\sqrt{g}}({\mathbb R}^M, V)$, where $g_M$ is the density function of the standard Gaussian probability measure on ${\mathbb R}^M$. Bibliography: 62 titles.

Détail

Keywords: high-dimensional approximation, collocation approximation, deep ReLU neural networks, parametric elliptic PDEs, lognormal inputs.

Bibliographie

[1] M. Ali and A. Nouy, “Approximation of smoothness classes by deep rectifier networks”, SIAM J. Numer. Anal., 59:6 (2021), 3032–3051 | DOI | MR | Zbl

[2] R. Arora, A. Basu, P. Mianjy and A. Mukherjee, Understanding deep neural networks with rectified linear units, Electronic colloquium on computational complexity, report No. 98, 2017, 21 pp. https://eccc.weizmann.ac.il/report/2017/098/

[3] M. Bachmayr, A. Cohen, Dinh Dũng and Ch. Schwab, “Fully discrete approximation of parametric and stochastic elliptic PDEs”, SIAM J. Numer. Anal., 55:5 (2017), 2151–2186 | DOI | MR | Zbl

[4] M. Bachmayr, A. Cohen, R. DeVore and G. Migliorati, “Sparse polynomial approximation of parametric elliptic PDEs. Part II: Lognormal coefficients”, ESAIM Math. Model. Numer. Anal., 51:1 (2017), 341–363 | DOI | MR | Zbl

[5] M. Bachmayr, A. Cohen and G. Migliorati, “Sparse polynomial approximation of parametric elliptic PDEs. Part I: Affine coefficients”, ESAIM Math. Model. Numer. Anal., 51:1 (2017), 321–339 | DOI | MR | Zbl

[6] A. R. Barron, “Complexity regularization with application to artificial neural networks”, Nonparametric functional estimation and related topics (Spetses 1990), NATO Adv. Sci. Inst. Ser. C: Math. Phys. Sci., 335, Kluwer Acad. Publ., Dordrecht, 1991, 561–576 | DOI | MR | Zbl

[7] A. Chkifa, A. Cohen, R. DeVore and Ch. Schwab, “Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs”, ESAIM Math. Model. Numer. Anal., 47:1 (2013), 253–280 | DOI | MR | Zbl

[8] A. Chkifa, A. Cohen and Ch. Schwab, “High-dimensional adaptive sparse polynomial interpolation and applications to parametric PDEs”, Found. Comput. Math., 14:4 (2014), 601–633 | DOI | MR | Zbl

[9] A. Chkifa, A. Cohen and Ch. Schwab, “Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs”, J. Math. Pures Appl. (9), 103:2 (2015), 400–428 | DOI | MR | Zbl

[10] A. Cohen and R. DeVore, “Approximation of high-dimensional parametric PDEs”, Acta Numer., 24 (2015), 1–159 | DOI | MR | Zbl

[11] A. Cohen, R. DeVore and Ch. Schwab, “Convergence rates of best $N$-term Galerkin approximations for a class of elliptic sPDEs”, Found. Comput. Math., 10:6 (2010), 615–646 | DOI | MR | Zbl

[12] A. Cohen, R. DeVore and Ch. Schwab, “Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE's”, Anal. Appl. (Singap.), 9:1 (2011), 11–47 | DOI | MR | Zbl

[13] G. Cybenko, “Approximation by superpositions of a sigmoidal function”, Math. Control Signals Systems, 2:4 (1989), 303–314 | DOI | MR | Zbl

[14] Dinh Dũng, “Linear collective collocation approximation for parametric and stochastic elliptic PDEs”, Mat. Sb., 210:4 (2019), 103–127 ; English transl. in Sb. Math., 210:4 (2019), 565–588 | DOI | MR | Zbl | DOI

[15] Dinh Dũng, “Sparse-grid polynomial interpolation approximation and integration for parametric and stochastic elliptic PDEs with lognormal inputs”, ESAIM Math. Model. Numer. Anal., 55:3 (2021), 1163–1198 | DOI | MR | Zbl

[16] Dinh Dũng and Van Kien Nguyen, “Deep ReLU neural networks in high-dimensional approximation”, Neural Netw., 142 (2021), 619–635 | DOI

[17] Dinh Dũng, Van Kien Nguyen and Duong Thanh Pham, Deep ReLU neural network approximation of parametric and stochastic elliptic PDEs with lognormal inputs, arXiv: 2111.05854v1

[18] Dinh Dũng, Van Kien Nguyen, Ch. Schwab and J. Zech, Analyticity and sparsity in uncertainty quantification for PDEs with Gaussian random field inputs, arXiv: 2201.01912

[19] Dinh Dũng, Van Kien Nguyen and Mai Xuan Thao, “Computation complexity of deep ReLU neural networks in high-dimensional approximation”, J. Comp. Sci. Cybern., 37:3 (2021), 292–320 | DOI

[20] I. Daubechies, R. DeVore, S. Foucart, B. Hanin and G. Petrova, “Nonlinear approximation and (deep) ReLU networks”, Constr. Approx., 55:1 (2022), 127–172 | DOI | MR | Zbl

[21] R. DeVore, B. Hanin and G. Petrova, “Neural network approximation”, Acta Numer., 30 (2021), 327–444 | DOI | MR

[22] Weinan E and Qingcan Wang, “Exponential convergence of the deep neural network approximation for analytic functions”, Sci. China Math., 61:10 (2018), 1733–1740 | DOI | MR | Zbl

[23] D. Elbrächter, P. Grohs, A. Jentzen and Ch. Schwab, DNN expression rate analysis of high-dimensional PDEs: application to option pricing, SAM res. rep. 2018-33, Seminar for Applied Mathematics, ETH Zürich, Zürich, 2018, 50 pp. https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2018/2018-33.pdf

[24] O. G. Ernst, B. Sprungk and L. Tamellini, “Convergence of sparse collocation for functions of countably many Gaussian random variables (with application to elliptic PDEs)”, SIAM J. Numer. Anal., 56:2 (2018), 877–905 | DOI | MR | Zbl

[25] K.-I. Funahashi, “Approximate realization of identity mappings by three-layer neural networks”, Electron. Comm. Japan Part III Fund. Electron. Sci., 73:11 (1990), 61–68 | DOI | MR

[26] M. Geist, P. Petersen, M. Raslan, R. Schneider and G. Kutyniok, “Numerical solution of the parametric diffusion equation by deep neural networks”, J. Sci. Comput., 88:1 (2021), 22, 37 pp. | DOI | MR | Zbl

[27] L. Gonon and Ch. Schwab, Deep ReLU network expression rates for option prices in high-dimensional, exponential Lévy models, SAM res. rep. 2020-52 (rev. 1), Seminar for Applied Mathematics, ETH Zürich, Zürich, 2021, 35 pp. https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2020/2020-52_rev1.pdf

[28] L. Gonon and Ch. Schwab, Deep ReLU neural network approximation for stochastic differential equations with jumps, SAM res. rep. 2021-08, Seminar for Applied Mathematics, ETH Zürich, Zürich, 2021, 35 pp. https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2021/2021-08.pdf

[29] R. Gribonval, Kutyniok, M. Nielsen and F. Voigtländer, “Approximation spaces of deep neural networks”, Constr. Approx., 55:1 (2022), 259–367 | DOI | MR | Zbl

[30] P. Grohs and L. Herrmann, “Deep neural network approximation for high-dimensional elliptic PDEs with boundary conditions”, IMA J. Numer. Anal., 42:3 (2022), 2055–2082 | DOI | MR | Zbl

[31] D. Elbrachter, D. Perekrestenko, P. Grohs and H. Bölcskei, “Deep neural network approximation theory”, IEEE Trans. Inform. Theory, 67:5 (2021), 2581–2623 | DOI | MR | Zbl

[32] I. Gühring, G. Kutyniok and P. Petersen, “Error bounds for approximations with deep ReLU neural networks in $W^{s,p}$ norms”, Anal. Appl. (Singap.), 18:5 (2020), 803–859 | DOI | MR | Zbl

[33] L. Herrmann, J. A. A. Opschoor and Ch. Schwab, Constructive deep ReLU neural network approximation, SAM res. rep. 2021-04, Seminar for Applied Mathematics, ETH Zürich, Zürich, 2021, 32 pp. https://www.sam.math.ethz.ch/sam_reports/reports_fi-nal/reports2021/2021-04.pdf

[34] L. Herrmann, Ch. Schwab and J. Zech, “Deep neural network expression of posterior expectations in Bayesian PDE inversion”, Inverse Problems, 36:12 (2020), 125011, 32 pp. | DOI | MR | Zbl

[35] E. Hewitt and K. Stromberg, Real and abstract analysis. A modern treatment of the theory of functions of a real variable, Springer-Verlag, New York, 1965, vii+476 pp. | DOI | MR | Zbl

[36] Viet Ha Hoang and Ch. Schwab, “$N$-term Wiener chaos approximation rates for elliptic PDEs with lognormal Gaussian random inputs”, Math. Models Methods Appl. Sci., 24:4 (2014), 797–826 | DOI | MR | Zbl

[37] K. Hornik, M. Stinchcombe and H. White, “Multilayer feedforward networks are universal approximators”, Neural Netw., 2:5 (1989), 359–366 | DOI | Zbl

[38] G. Kutyniok, P. Petersen, M. Raslan and R. Schneider, “A theoretical analysis of deep neural networks and parametric PDEs”, Constr. Approx., 55:1 (2022), 73–125 | DOI | MR | Zbl

[39] Jianfeng Lu, Zuowei Shen, Haizhao Yang and Shijun Zhang, “Deep network approximation for smooth functions”, SIAM J. Math. Anal., 53:5 (2021), 5465–5506 | DOI | MR | Zbl

[40] D. M. Matjila, “Bounds for Lebesgue functions for Freud weights”, J. Approx. Theory, 79:3 (1994), 385–406 | DOI | MR | Zbl

[41] D. M. Matjila, “Convergence of Lagrange interpolation for Freud weights in weighted $L_p(\mathbb R)$, $0 P \le 1$”, Nonlinear numerical methods and rational approximation. II (Wilrijk 1993), Math. Appl., 296, Kluwer Acad. Publ., Dordrecht, 1994, 25–35 | DOI | MR | Zbl

[42] H. N. Mhaskar, “Neural networks for optimal approximation of smooth and analytic functions”, Neural Comput., 8 (1996), 164–177 | DOI

[43] H. Montanelli and Qiang Du, “New error bounds for deep ReLU networks using sparse grids”, SIAM J. Math. Data Sci., 1:1 (2019), 78–92 | DOI | MR | Zbl

[44] G. Montúfar, R. Pascanu, Kyunghyun Cho and Yoshua Bengio, “On the number of linear regions of deep neural networks”, NIPS 2014, Adv. Neural Inf. Process. Syst., 27, MIT Press, Cambridge, MA, 2014, 2924–2932 http://proceedings.neurips.cc/paper/2014

[45] J. A. A. Opschoor, Ch. Schwab and J. Zech, Deep learning in high dimension: ReLU network expression rates for Bayesian PDE inversion, SAM res. rep. 2020-47, Seminar for Applied Mathematics, ETH Zürich, Zürich, 2020, 50 pp. https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2020/2020-47.pdf

[46] J. A. A. Opschoor, Ch. Schwab and J. Zech, “Exponential ReLU DNN expression of holomorphic maps in high dimension”, Constr. Approx., 55:1 (2022), 537–582 | DOI | MR | Zbl

[47] P. C. Petersen, Neural network theory, 2022, 60 pp. http://pc-petersen.eu/Neural_Network_Theory.pdf

[48] P. Petersen and F. Voigtlaender, “Optimal approximation of piecewise smooth functions using deep ReLU neural networks”, Neural Netw., 108 (2018), 296–330 | DOI | Zbl

[49] Ch. Schwab and J. Zech, “Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ”, Anal. Appl. (Singap.), 17:1 (2019), 19–55 | DOI | MR | Zbl

[50] Ch. Schwab and J. Zech, Deep learning in high dimension: neural network approximation of analytic functions in $L^2(\mathbb R^d, \gamma_d)$, arXiv: 2111.07080

[51] Zuowei Shen, Haizhao Yang and Shijun Zhang, “Deep network approximation characterized by number of neurons”, Commun. Comput. Phys., 28:5 (2020), 1768–1811 | DOI | MR | Zbl

[52] J. Sirignano and K. Spiliopoulos, “DGM: a deep learning algorithm for solving partial differential equations”, J. Comput. Phys., 375 (2018), 1339–1364 | DOI | MR | Zbl

[53] T. Suzuki, Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality, ICLR 2019: International conference on learning representations (New Orleans, LA 2019) https://openreview.net/pdf?id=H1ebTsActm

[54] J. Szabados, “Weighted Lagrange and Hermité-Fejér interpolation on the real line”, J. Inequal. Appl., 1:2 (1997), 99–123 | MR | Zbl

[55] G. Szegö, Orthogonal polynomials, Amer. Math. Soc. Colloq. Publ., 23, Amer. Math. Soc., New York, 1939, ix+401 pp. | MR | Zbl

[56] M. Telgarsky, Representation benefits of deep feedforward networks, arXiv: 1509.08101

[57] M. Telgarsky, “Benefits of depth in neural nets”, 29th annual conference on learning theory (Columbia Univ., New York, NY 2016), Proceedings of Machine Learning Research (PMLR), 49, 2016, 1517–1539 https://proceedings.mlr.press/v49/telgarsky16.html

[58] R. K. Tripathy and I. Bilionis, “Deep UQ: learning deep neural network surrogate models for high dimensional uncertainty quantification”, J. Comput. Phys., 375 (2018), 565–588 | DOI | MR | Zbl

[59] D. Yarotsky, “Error bounds for approximations with deep ReLU networks”, Neural Netw., 94 (2017), 103–114 | DOI | Zbl

[60] D. Yarotsky, “Optimal approximation of continuous functions by very deep ReLU networks”, 31st annual conference on learning theory, Proceedings of Machine Learning Research (PMLR), 75, 2018, 639–649 https://proceedings.mlr.press/v75/yarotsky18a.html

[61] J. Zech, D. Dũng and Ch. Schwab, “Multilevel approximation of parametric and stochastic PDES”, Math. Models Methods Appl. Sci., 29:9 (2019), 1753–1817 | DOI | MR | Zbl

[62] J. Zech and Ch. Schwab, “Convergence rates of high dimensional Smolyak quadrature”, ESAIM Math. Model. Numer. Anal., 54:4 (2020), 1259–1307 | DOI | MR | Zbl