Geodesic
Optimality conditions for maximizers of the information divergence from an exponential family
Kybernetika, Tome 43 (2007) no. 5, pp. 731-746 Cet article a éte moissonné depuis la source Czech Digital Mathematics Library

Voir la notice de l'article

The information divergence of a probability measure $P$ from an exponential family $\mathcal{E}$ over a finite set is defined as infimum of the divergences of $P$ from $Q$ subject to $Q\in \mathcal{E}$. All directional derivatives of the divergence from $\mathcal{E}$ are explicitly found. To this end, behaviour of the conjugate of a log-Laplace transform on the boundary of its domain is analysed. The first order conditions for $P$ to be a maximizer of the divergence from $\mathcal{E}$ are presented, including new ones when $P$ is not projectable to $\mathcal{E}$.
The information divergence of a probability measure $P$ from an exponential family $\mathcal{E}$ over a finite set is defined as infimum of the divergences of $P$ from $Q$ subject to $Q\in \mathcal{E}$. All directional derivatives of the divergence from $\mathcal{E}$ are explicitly found. To this end, behaviour of the conjugate of a log-Laplace transform on the boundary of its domain is analysed. The first order conditions for $P$ to be a maximizer of the divergence from $\mathcal{E}$ are presented, including new ones when $P$ is not projectable to $\mathcal{E}$.
Classification : 52A20, 60A10, 62B10, 90C90, 94A17
Keywords: Kullback–Leibler divergence; relative entropy; exponential family; information projection; log-Laplace transform; cumulant generating function; directional derivatives; first order optimality conditions; convex functions; polytopes 
@article{KYB_2007_43_5_a9,
     author = {Mat\'u\v{s}, Franti\v{s}ek},
     title = {Optimality conditions for maximizers of the information divergence from an exponential family},
     journal = {Kybernetika},
     pages = {731--746},
     year = {2007},
     volume = {43},
     number = {5},
     mrnumber = {2376334},
     zbl = {1149.94007},
     language = {en},
     url = {http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a9/}
}
TY  - JOUR
AU  - Matúš, František
TI  - Optimality conditions for maximizers of the information divergence from an exponential family
JO  - Kybernetika
PY  - 2007
SP  - 731
EP  - 746
VL  - 43
IS  - 5
UR  - http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a9/
LA  - en
ID  - KYB_2007_43_5_a9
ER  - 
%0 Journal Article
%A Matúš, František
%T Optimality conditions for maximizers of the information divergence from an exponential family
%J Kybernetika
%D 2007
%P 731-746
%V 43
%N 5
%U http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a9/
%G en
%F KYB_2007_43_5_a9
Matúš, František. Optimality conditions for maximizers of the information divergence from an exponential family. Kybernetika, Tome 43 (2007) no. 5, pp. 731-746. http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a9/

[1] Ay N.: An information-geometric approach to a theory of pragmatic structuring. Ann. Probab. 30 (2002), 416–436 | MR | Zbl

[2] Ay N.: Locality of Global Stochastic Interaction in Directed Acyclic Networks. Neural Computation 14 (2002), 2959–2980 | Zbl

[3] Ay N., Knauf A.: Maximizing multi-information. Kybernetika 45 (2006), 517–538 | MR

[4] Ay N., Wennekers T.: Dynamical properties of strongly interacting Markov chains. Neural Networks 16 (2003), 1483–1497

[5] Barndorff-Nielsen O.: Information and Exponential Families in Statistical Theory. Wiley, New York 1978 | MR | Zbl

[6] Brown L. D.: Fundamentals of Statistical Exponential Families. (Lecture Notes – Monograph Series 9.) Institute of Mathematical Statistics, Hayward, CA 1986 | MR | Zbl

[8] Csiszár I., Matúš F.: Information projections revisited. IEEE Trans. Inform. Theory 49 (2003), 1474–1490 | MR | Zbl

[9] Csiszár I., Matúš F.: Closures of exponential families. Ann. Probab. 33 (2005), 582–600 | MR | Zbl

[10] Csiszár I., Matúš F.: Generalized maximum likelihood estimates for exponential families. To appear in Probab. Theory Related Fields (2008) | MR | Zbl

[11] Pietra S. Della, Pietra, V. Della, Lafferty J.: Inducing features of random fields. IEEE Trans. Pattern Anal. Mach. Intell. 19 (1997), 380–393

[12] Letac G.: Lectures on Natural Exponential Families and their Variance Functions. (Monografias de Matemática 50.) Instituto de Matemática Pura e Aplicada, Rio de Janeiro 1992 | MR | Zbl

[13] Matúš F.: Maximization of information divergences from binary i. i.d. sequences. In: Proc. IPMU 2004, Perugia 2004, Vol. 2, pp. 1303–1306

[14] Matúš F., Ay N.: On maximization of the information divergence from an exponential family. In: Proc. WUPES’03 (J. Vejnarová, ed.), University of Economics, Prague 2003, pp. 199–204

[15] Rockafellar R. T.: Convex Analysis. Princeton University Press, Priceton, N.J. 1970 | MR

[16] Wennekers T., Ay N.: Finite state automata resulting from temporal information maximization. Theory in Biosciences 122 (2003), 5–18 | Zbl