Limit distributions of a statistical estimate for the entropy

A. M. Zubkov

Teoriâ veroâtnostej i ee primeneniâ, Tome 18 (1973) no. 3, pp. 643-650

Résumé

Let $\widehat p_j$ be the relative frequency with which the event $E_j$ occurs in $N$ mutually independent identically distributed trials ($j=1,2,\dots,s$; $E_i\cap E_j=\varnothing$, $i\ne j$; $\mathbf P\{\bigcup\limits_{j=1}^sE_j\}=1$). We find conditions for the estimate $$ H=-\sum_{j=1}^s\widehat p_j\ln\widehat p_j $$ of the entropy $H=-\sum_{j=1}^s\mathbf P\{E_j\}\ln\mathbf P\{E_j\}$ to be asymptotically normal (as $N,s\to\infty$) or asymptotically $\chi^2$ (as $N\to\infty$, $s$ fixed).