Rayleigh triangles and non-matrix interpolation of matrix beta integrals

Yu. A. Neretin

Sbornik. Mathematics, Tome 194 (2003) no. 4, pp. 515-540

Résumé

A Rayleigh triangle of size $n$ is a set of $n(n+1)/2$ real numbers $\lambda_{kl}$, where $1\leqslant l\leqslant k\leqslant n$, which are decreasing as $k$ increases for fixed $k$ and are increasing as $k$ increases for fixed $k-l$. We construct a family of beta integrals over the space of Rayleigh triangles which interpolate matrix integrals of the types of Siegel, Hua Loo Keng, and Gindikin with respect to the dimension of the ground field ($\mathbb R$, $\mathbb C$, or the quaternions $\mathbb H$). We also interpolate the Hua–Pickrell measures on the inverse limits of the symmetric spaces $\operatorname U(n)$, $\operatorname U(n)/\operatorname O(n)$, $\operatorname U(2n)/\operatorname{Sp}(n)$. Our family of integrals also includes the Selberg integral.