Geodesic
Scaling of model approximation errors and expected entropy distances
Kybernetika, Tome 50 (2014) no. 2, pp. 234-245
Cet article a éte moissonné depuis la source Czech Digital Mathematics Library

Voir la notice de l'article

We compute the expected value of the Kullback-Leibler divergence of various fundamental statistical models with respect to Dirichlet priors. For the uniform prior, the expected divergence of any model containing the uniform distribution is bounded by a constant $1-\gamma$. For the models that we consider this bound is approached as the cardinality of the sample space tends to infinity, if the model dimension remains relatively small. For Dirichlet priors with reasonable concentration parameters the expected values of the divergence behave in a similar way. These results serve as a reference to rank the approximation capabilities of other statistical models.
We compute the expected value of the Kullback-Leibler divergence of various fundamental statistical models with respect to Dirichlet priors. For the uniform prior, the expected divergence of any model containing the uniform distribution is bounded by a constant $1-\gamma$. For the models that we consider this bound is approached as the cardinality of the sample space tends to infinity, if the model dimension remains relatively small. For Dirichlet priors with reasonable concentration parameters the expected values of the divergence behave in a similar way. These results serve as a reference to rank the approximation capabilities of other statistical models.
DOI : 10.14736/kyb-2014-2-0234
Classification : 62B10, 62F15, 62F25, 68T30
Keywords: exponential families; KL divergence; MLE; Dirichlet prior 
@article{10_14736_kyb_2014_2_0234,
     author = {Mont\'ufar, Guido F. and Rauh, Johannes},
     title = {Scaling of model approximation errors and expected entropy distances},
     journal = {Kybernetika},
     pages = {234--245},
     year = {2014},
     volume = {50},
     number = {2},
     doi = {10.14736/kyb-2014-2-0234},
     mrnumber = {3216992},
     zbl = {06325222},
     language = {en},
     url = {http://geodesic.mathdoc.fr/articles/10.14736/kyb-2014-2-0234/}
}
TY  - JOUR
AU  - Montúfar, Guido F.
AU  - Rauh, Johannes
TI  - Scaling of model approximation errors and expected entropy distances
JO  - Kybernetika
PY  - 2014
SP  - 234
EP  - 245
VL  - 50
IS  - 2
UR  - http://geodesic.mathdoc.fr/articles/10.14736/kyb-2014-2-0234/
DO  - 10.14736/kyb-2014-2-0234
LA  - en
ID  - 10_14736_kyb_2014_2_0234
ER  - 
%0 Journal Article
%A Montúfar, Guido F.
%A Rauh, Johannes
%T Scaling of model approximation errors and expected entropy distances
%J Kybernetika
%D 2014
%P 234-245
%V 50
%N 2
%U http://geodesic.mathdoc.fr/articles/10.14736/kyb-2014-2-0234/
%R 10.14736/kyb-2014-2-0234
%G en
%F 10_14736_kyb_2014_2_0234
Montúfar, Guido F.; Rauh, Johannes. Scaling of model approximation errors and expected entropy distances. Kybernetika, Tome 50 (2014) no. 2, pp. 234-245. doi: 10.14736/kyb-2014-2-0234

[1] Ay, N.: An information-geometric approach to a theory of pragmatic structuring. Ann. Probab. 30 (2002), 416-436. | DOI | MR | Zbl

[2] Drton, M., Sturmfels, B., Sullivant, S.: Lectures on Algebraic Statistics. Birkhäuser, Basel 2009. | MR | Zbl

[3] Frigyik, B. A., Kapila, A., Gupta, M. R.: Introduction to the Dirichlet Distribution and Related Processes. Technical Report, Department of Electrical Engineering University of Washington, 2010.

[4] Matúš, F., Ay, N.: On maximization of the information divergence from an exponential family. In: Proc. WUPES'03, University of Economics, Prague 2003, pp. 199-204.

[5] Matúš, F., Rauh, J.: Maximization of the information divergence from an exponential family and criticality. In: Proc. ISIT, St. Petersburg 2011, pp. 903-907.

[6] Montúfar, G., Rauh, J., Ay, N.: Expressive power and approximation errors of restricted Boltzmann machines. In: Advances in NIPS 24, MIT Press, Cambridge 2011, pp. 415-423.

[7] Nemenman, I., Shafee, F., Bialek, W.: Entropy and inference, revisited. In: Advances in NIPS 14, MIT Press, Cambridge 2001, pp. 471-478.

[8] Rauh, J.: Finding the Maximizers of the Information Divergence from an Exponential Family. Ph.D. Thesis, Universität Leipzig 2011. | MR

[9] Rauh, J.: Optimally approximating exponential families. Kybernetika 49 (2013), 199-215. | MR | Zbl

[10] Wolpert, D., Wolf, D.: Estimating functions of probability distributions from a finite set of samples. Phys, Rev. E 52 (1995), 6841-6854. | DOI | MR

Cité par Sources :