On the theory of controlled Markov processes

A. Barbarošie

Teoriâ veroâtnostej i ee primeneniâ, Tome 22 (1977) no. 1, pp. 55-71

Résumé

Let $\Xi^d=(\xi_t,\mathscr F_t,\mathbf P_x^d)_{t\in\mathscr N}$ be a family of Markov processes on $(\Omega,\mathscr F)$ with values in$(X,\mathscr X)$, $d\in D$. Any sequence $$ \delta=\{d_0(x_0),d_1(x_0,x_1),\dots,d_k(x_0,\dots,x_k),\dots\}, $$ where $d_k:(X,\mathscr X)^{k+1}\to(D,\mathscr D)$, $\mathscr D$ is a $\sigma$-algebra in $D$, is called a control policy. For each control policy $\delta$, a controlled Markov process $\Xi^{\delta}=(\xi_t,\mathscr F_t,\mathbf P_x^{\delta})_{t\in\mathscr N}$ is constructed. Let $\overline{\mathfrak M}$ be the set of stopping times with respect to $\{\mathscr F_t,t\in\mathscr N\bigcup\{+\infty\}\}$, $\Delta$ be the set of control policies, \begin{gather*} \overline{\Sigma}=\overline{\mathfrak M}\times\Delta;\ \Sigma=\{[\tau,\delta]\in\overline{\Sigma}:\mathbf P_x^{\delta}\{\tau\infty\}=1\},\\ \Sigma_n=\{[\tau,\delta]\in\Sigma:\mathbf P_x^{\delta}\{\tau\le n\}=1\}. \end{gather*} Let $g(x)$ be a real $\overline{\mathscr X}$-measurable function, $g^-(x)\le k\infty$, and \begin{gather*} \overline s(x)=\sup_{[\tau,\delta]\in\overline{\Sigma}}\mathbf M_x^{\delta}g(\xi_{\tau}),\qquad g(\xi_{\infty})=\varlimsup g(\xi_n);\\ s(x)=\sup_{[\tau,\delta]\in\Sigma}\mathbf M_x^{\delta}g(\xi_{\tau}),\\ s_n(x)=\sup_{[\tau,\delta]\in\Sigma_n}\mathbf M_x^{\delta}g(\xi_{\tau}). \end{gather*} We show that the gain functions $\xi(x)$ and $s(x)$ are equal and $s(x)$ is the least excessive majorant of $g(x)$. For each $\varepsilon>0$ and a probability measure $\mu$ on $(X,\mathscr X)$, $(\mu,\varepsilon,s)$- and $(\mu,\varepsilon,s_n)$-optimal strategies $[\tau,\delta]$ are constructed. We also show that $s_n(x)\to s(x)$ as $n\to\infty$.