Geodesic
On Universal Estimators in Learning Theory
Informatics and Automation, Function spaces, approximation theory, and nonlinear analysis, Tome 255 (2006), pp. 256-272.

Voir la notice de l'article provenant de la source Math-Net.Ru

This paper addresses the problem of constructing and analyzing estimators for the regression problem in supervised learning. Recently, there has been great interest in studying universal estimators. The term “universal” means that, on the one hand, the estimator does not depend on the a priori assumption that the regression function $f_\rho$ belongs to some class $F$ from a collection of classes $\mathcal F$ and, on the other hand, the estimation error for $f_\rho$ is close to the optimal error for the class $F$. This paper is an illustration of how the general technique of constructing universal estimators, developed in the author's previous paper, can be applied in concrete situations. The setting of the problem studied in the paper has been motivated by a recent paper by Smale and Zhou. The starting point for us is a kernel $K(x,u)$ defined on $X\times \Omega$. On the base of this kernel, we build an estimator that is universal for classes defined in terms of nonlinear approximations with regard to the system $\{K(\cdot ,u)\}_{u\in \Omega }$. To construct an easily implementable estimator, we apply the relaxed greedy algorithm. 
@article{TRSPY_2006_255_a19,
     author = {V. N. Temlyakov},
     title = {On {Universal} {Estimators} in {Learning} {Theory}},
     journal = {Informatics and Automation},
     pages = {256--272},
     publisher = {mathdoc},
     volume = {255},
     year = {2006},
     language = {ru},
     url = {http://geodesic.mathdoc.fr/item/TRSPY_2006_255_a19/}
}
TY  - JOUR
AU  - V. N. Temlyakov
TI  - On Universal Estimators in Learning Theory
JO  - Informatics and Automation
PY  - 2006
SP  - 256
EP  - 272
VL  - 255
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/TRSPY_2006_255_a19/
LA  - ru
ID  - TRSPY_2006_255_a19
ER  - 
%0 Journal Article
%A V. N. Temlyakov
%T On Universal Estimators in Learning Theory
%J Informatics and Automation
%D 2006
%P 256-272
%V 255
%I mathdoc
%U http://geodesic.mathdoc.fr/item/TRSPY_2006_255_a19/
%G ru
%F TRSPY_2006_255_a19
V. N. Temlyakov. On Universal Estimators in Learning Theory. Informatics and Automation, Function spaces, approximation theory, and nonlinear analysis, Tome 255 (2006), pp. 256-272. http://geodesic.mathdoc.fr/item/TRSPY_2006_255_a19/

[1] Barron A.R., “Universal approximation bounds for superposition of $n$ sigmoidal functions”, IEEE Trans. Inform. Theory, 39:3 (1993), 930–945 | DOI | MR | Zbl

[2] Binev P., Cohen A., Dahmen W., DeVore R., Temlyakov V., “Universal algorithms for learning theory. Pt. I: Piecewise constant functions”, J. Mach. Learn. Res., 6 (2005), 1297–1321 | MR

[3] Carl B., “Entropy numbers, $s$-numbers, and eigenvalue problems”, J. Funct. Anal., 41 (1981), 290–306 | DOI | MR | Zbl

[4] Barron A., Cohen A., Dahmen W., DeVore R., Approximation and learning by greedy algorithms, Manuscript, 2005, 27 pp.; http://www.ann.jussieu.fr/c̃ohen/greedy.pdf.gz

[5] Cucker F., Smale S., “On the mathematical foundations of learning”, Bull. Amer. Math. Soc., 39 (2002), 1–49 | DOI | MR | Zbl

[6] DeVore R., Kerkyacharian G., Picard D., Temlyakov V., On mathematical methods of learning, Industr. Math. Inst. Res. Repts. N 10, Univ. South Carolina, Columbia, 2004, 24 pp. | MR

[7] DeVore R., Kerkyacharian G., Picard D., Temlyakov V., Mathematical methods for supervised learning, Industr. Math. Inst. Res. Repts. N 22, Univ. South Carolina, Columbia, 2004, 51 pp.

[8] Györfi L., Kohler M., Krzyzak A., Walk H., A distribution-free theory of nonparametric regression, Springer, Berlin, 2002 | MR

[9] DeVore R.A., Temlyakov V.N., “Some remarks on greedy algorithms”, Adv. Comput. Math., 5 (1996), 173–187 | DOI | MR | Zbl

[10] Huber P.J., “Projection pursuit”, Ann. Statist., 13 (1985), 435–525 | DOI | MR | Zbl

[11] Jones L., “On a conjecture of Huber concerning the convergence of projection pursuit regression”, Ann. Statist., 15 (1987), 880–882 | DOI | MR | Zbl

[12] Kerkyacharian G., Picard D., Thresholding in learning theory, math.ST/0510271 | MR

[13] Konyagin S.V., Temlyakov V.N., The entropy in the learning theory. Error estimates, Industr. Math. Inst. Res. Repts. N 09, Univ. South Carolina, Columbia, 2004, 25 pp. | MR

[14] Lee W.S., Bartlett P.L., Williamson R.C., “Efficient agnostic learning of neural networks with bounded fan-in”, IEEE Trans. Inform. Theory, 42:6 (1996), 2118–2132 | DOI | MR | Zbl

[15] Lee W.S., Bartlett P.L., Williamson R.C., “The importance of convexity in learning with squared loss”, IEEE Trans. Inform. Theory, 44:5 (1998), 1974–1980 | DOI | MR | Zbl

[16] Schmidt E., “Zur Theorie der linearen und nichtlinearen Integralgleichungen. I”, Math. Ann., 63 (1906–1907), 433–476 | DOI | MR

[17] Smale S., Zhou D.-X., Learning theory estimates via integral operators and their approximations, Manuscript, 2005, 20 pp.; http://www.tti-c.org/smale_ papers/sampIII5412.pdf

[18] Temlyakov V.N., Optimal estimators in learning theory, Industr. Math. Inst. Res. Repts. N 23, Univ. South Carolina, Columbia, 2004, 29 pp.

[19] Temlyakov V.N., Approximation in learning theory, Industr. Math. Inst. Res. Repts. N 05, Univ. South Carolina, Columbia, 2005, 42 pp.

[20] Temlyakov V.N., “Nonlinear methods of approximation”, Found. Comput. Math., 3 (2003), 33–107 | DOI | MR | Zbl

[21] Temlyakov V.N., “Greedy algorithms in Banach spaces”, Adv. Comput. Math., 14 (2001), 277–292 | DOI | MR | Zbl