Nonsymmetrical distance between probability distributions, entropy and the theorem of Pythagoras

N. N. Chentsov

Matematičeskie zametki, Tome 4 (1968) no. 3, pp. 323-332.

Résumé

The Kullback–Leibler information $I[Q\mid P]$ for discrimination in favor of the probability distribution $Q$ against $P$ is considered as a nonsymmetrical analog of one half of the square of the distance between the “points” $Q$ and $P$. For the $n$-dimensional “planes” we take the exponential families. We shall prove a nonsymmetrical analogue of the theorem of Pythagoras in the formulation: “The squared length of an oblique line equals the sum of the squared lengths of the perpendicular and the projection of the oblique line,” and also an analog of the cosine theorem and the like.