Geodesic
On generalized entropies, Bayesian decisions and statistical diversity
Kybernetika, Tome 43 (2007) no. 5, pp. 675-696 Cet article a éte moissonné depuis la source Czech Digital Mathematics Library

Voir la notice de l'article

The paper summarizes and extends the theory of generalized $\phi $-entropies $H_{\phi }(X)$ of random variables $X$ obtained as $\phi $-informations $I_{\phi }(X;Y)$ about $X$ maximized over random variables $Y$. Among the new results is the proof of the fact that these entropies need not be concave functions of distributions $p_{X}$. An extended class of power entropies $H_{\alpha }(X)$ is introduced, parametrized by $\alpha \in {\mathbb{R}}$, where $H_{\alpha }(X)$ are concave in $p_{X}$ for $\alpha \ge 0$ and convex for $\alpha 0$. It is proved that all power entropies with $\alpha \le 2$ are maximal $\phi $-informations $I_{\phi }(X;X)$ for appropriate $\phi $ depending on $\alpha $. Prominent members of this subclass of power entropies are the Shannon entropy $H_{1}(X)$ and the quadratic entropy $H_{2}(X)$. The paper investigates also the tightness of practically important previously established relations between these two entropies and errors $e(X)$ of Bayesian decisions about possible realizations of $X$. The quadratic entropy is shown to provide estimates which are in average more than 100 % tighter those based on the Shannon entropy, and this tightness is shown to increase even further when $\alpha $ increases beyond $\alpha =2$. Finally, the paper studies various measures of statistical diversity and introduces a general measure of anisotony between them. This measure is numerically evaluated for the entropic measures of diversity $H_1(X)$ and $H_2(X)$.
The paper summarizes and extends the theory of generalized $\phi $-entropies $H_{\phi }(X)$ of random variables $X$ obtained as $\phi $-informations $I_{\phi }(X;Y)$ about $X$ maximized over random variables $Y$. Among the new results is the proof of the fact that these entropies need not be concave functions of distributions $p_{X}$. An extended class of power entropies $H_{\alpha }(X)$ is introduced, parametrized by $\alpha \in {\mathbb{R}}$, where $H_{\alpha }(X)$ are concave in $p_{X}$ for $\alpha \ge 0$ and convex for $\alpha 0$. It is proved that all power entropies with $\alpha \le 2$ are maximal $\phi $-informations $I_{\phi }(X;X)$ for appropriate $\phi $ depending on $\alpha $. Prominent members of this subclass of power entropies are the Shannon entropy $H_{1}(X)$ and the quadratic entropy $H_{2}(X)$. The paper investigates also the tightness of practically important previously established relations between these two entropies and errors $e(X)$ of Bayesian decisions about possible realizations of $X$. The quadratic entropy is shown to provide estimates which are in average more than 100 % tighter those based on the Shannon entropy, and this tightness is shown to increase even further when $\alpha $ increases beyond $\alpha =2$. Finally, the paper studies various measures of statistical diversity and introduces a general measure of anisotony between them. This measure is numerically evaluated for the entropic measures of diversity $H_1(X)$ and $H_2(X)$.
Classification : 62C10, 94A17
Keywords: $\phi $-divergences; $\phi $-informations; power divergences; power entropies; Shannon entropy; quadratic entropy; Bayes error; Simpson diversity; Emlen diversity 
@article{KYB_2007_43_5_a6,
     author = {Vajda, Igor and Zv\'arov\'a, Jana},
     title = {On generalized entropies, {Bayesian} decisions and statistical diversity},
     journal = {Kybernetika},
     pages = {675--696},
     year = {2007},
     volume = {43},
     number = {5},
     mrnumber = {2376331},
     zbl = {1143.94006},
     language = {en},
     url = {http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a6/}
}
TY  - JOUR
AU  - Vajda, Igor
AU  - Zvárová, Jana
TI  - On generalized entropies, Bayesian decisions and statistical diversity
JO  - Kybernetika
PY  - 2007
SP  - 675
EP  - 696
VL  - 43
IS  - 5
UR  - http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a6/
LA  - en
ID  - KYB_2007_43_5_a6
ER  - 
%0 Journal Article
%A Vajda, Igor
%A Zvárová, Jana
%T On generalized entropies, Bayesian decisions and statistical diversity
%J Kybernetika
%D 2007
%P 675-696
%V 43
%N 5
%U http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a6/
%G en
%F KYB_2007_43_5_a6
Vajda, Igor; Zvárová, Jana. On generalized entropies, Bayesian decisions and statistical diversity. Kybernetika, Tome 43 (2007) no. 5, pp. 675-696. http://geodesic.mathdoc.fr/item/KYB_2007_43_5_a6/

[1] Cover T., Thomas J.: Elements of Information Theory. Wiley, New York 1991 | MR | Zbl

[3] Cressie N., Read T. R. C.: Multinomial goodness-of-fit tests. J. Roy. Statist. Soc. Ser. B 46 (1984), 440–464 | MR | Zbl

[4] Csiszár I.: Informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizität von Markoffschen Ketten. Publ. Math. Inst. Hungar. Acad. Sci. Ser. A 8 (1963), 85–108 | MR | Zbl

[5] Csiszár I.: Information-type measures of difference of probability distributions and indirect observations. Studia Sci. Math. Hungar. 2 (1967), 299–318 | MR

[6] Csiszár I.: A class of measures of informativity of observation channels. Period. Math. Hungar. 2 (1972), 191–213 | MR | Zbl

[7] Dalton H.: The Inequality of Incomes. Ruthledge & Keagan Paul, London 1925

[8] Devijver P., Kittler J.: Pattern Recognition: A Statistical Approach. Prentice Hall, Englewood Cliffs, NJ 1982 | MR | Zbl

[9] Devroy L., Györfi, L., Lugosi G.: A Probabilistic Theory of Pattern Recognition. Springer, Berlin 1996 | MR

[10] Emlen J. M.: Ecology: An Evolutionary Approach. Adison-Wesley, Reading 1973

[11] Gini C.: Variabilitá e Mutabilitá. Studi Economico-Giuridici della R. Univ. di Cagliari. 3 (1912), Part 2, p. 80

[12] Harremöes P., Topsøe F.: Inequalities between etropy and index of concidence. IEEE Trans. Inform. Theory 47 (2001), 2944–2960

[13] Havrda J., Charvát F.: Concept of structural $a$-entropy. Kybernetika 3 (1967), 30–35 | MR | Zbl

[14] Höffding W.: Masstabinvariante Korrelationstheorie. Teubner, Leipzig 1940

[15] Höffding W.: Stochastische Abhängigkeit und funktionaler Zusammenhang. Skand. Aktuar. Tidskr. 25 (1942), 200–207 | Zbl

[16] Kovalevskij V. A.: The problem of character recognition from the point of view of mathematical statistics. Character Readers and Pattern Recognition, 3–30. Spartan Books, New York 1967

[17] Liese F., Vajda I.: Convex Statistical Distances. Teubner, Leipzig 1987 | MR | Zbl

[18] Liese F., Vajda I.: On divergences and informations in statistics and information theory. IEEE Trans. Inform. Theory 52 (2006), 4394–4412 | MR

[19] Marshall A. W., Olkin I.: Inequalities: Theory of Majorization and its Applications. Academic Press, New York 1979 | MR | Zbl

[20] Morales D., Pardo, L., Vajda I.: Uncertainty of discrete stochastic systems. IEEE Trans. Systems, Man Cybernet. Part A 26 (1996), 681–697

[21] Pearson K.: On the theory of contingency and its relation to association and normal correlation. Drapers Company Research Memoirs, Biometric Ser. 1, London 1904

[22] Perez A.: Information-theoretic risk estimates in statistical decision. Kybernetika 3 (1967), 1–21 | MR | Zbl

[23] Rényi A.: On measures of dependence. Acta Math. Acad. Sci. Hungar. 10 (1959), 441–451 | MR | Zbl

[24] Rényi A.: On measures of entropy and information. In: Proc. Fourth Berkeley Symposium on Probab. Statist., Volume 1, Univ. Calif. Press, Berkeley 1961, pp. 547–561 | MR

[25] Sen A.: On Economic Inequality. Oxford Univ. Press, London 1973

[26] Simpson E. H.: Measurement of diversity. Nature 163 (1949), 688 | Zbl

[27] Tschuprow A.: Grundbegriffe und Grundprobleme der Korrelationstheorie. Berlin 1925

[28] Vajda I.: Bounds on the minimal error probability and checking a finite or countable number of hypotheses. Information Transmission Problems 4 (1968), 9–17 | MR

[29] Vajda I.: Theory of Statistical Inference and Information. Kluwer, Boston 1989 | Zbl

[30] Vajda I., Vašek K.: Majorization concave entropies and comparison of experiments. Problems Control Inform. Theory 14 (1985), 105–115 | MR | Zbl

[31] Vajda I., Zvárová J.: On relations between informations, entropies and Bayesian decisions. In: Prague Stochastics 2006 (M. Hušková and M. Janžura, eds.), Matfyzpress, Prague 2006, pp. 709–718

[32] Zvárová J.: On measures of statistical dependence. Čas. pěst. matemat. 99 (1974), 15–29 | MR | Zbl

[33] Zvárová J.: On medical informatics structure. Internat. J. Medical Informatics 44 (1997), 75–81

[34] Zvárová J.: Information Measures of Stochastic Dependence and Diversity: Theory and Medical Informatics Applications. Doctor of Sciences Dissertation, Academy of Sciences of the Czech Republic, Institute of Informatics, Prague 1998

[35] Zvárová J., Mazura I.: Stochastic Genetics (in Czech). Charles University, Karolinum, Prague 2001

[36] Zvárová J., Vajda I.: On genetic information, diversity and distance. Methods of Inform. in Medicine 2 (2006), 173–179

[37] Zvárová J., Vajda I.: On isotony and anisotony between Gini and Shannon measures of diversity. Preprint, Prague 2006