First passage risk probability optimality for continuous time Markov decision processes

Huo, Haifeng ; Wen, Xian

Kybernetika, Tome 55 (2019) no. 1, pp. 114-133

Abstract (VO)

In this paper, we study continuous time Markov decision processes (CTMDPs) with a denumerable state space, a Borel action space, unbounded transition rates and nonnegative reward function. The optimality criterion to be considered is the first passage risk probability criterion. To ensure the non-explosion of the state processes, we first introduce a so-called drift condition, which is weaker than the well known regular condition for semi-Markov decision processes (SMDPs). Furthermore, under some suitable conditions, by value iteration recursive approximation technique, we establish the optimality equation, obtain the uniqueness of the value function and the existence of optimal policies. Finally, two examples are used to illustrate our results.

Détail

DOI : 10.14736/kyb-2019-1-0114

Classification : 60E20, 90C40
Keywords: continuous time Markov decision processes; first passage time; risk probability criterion; optimal policy

Bibliographie

[1] Bertsekas, D., S.Shreve: Stochastic Optimal Control: The Discrete-Time Case. Academic Press Inc 1996 | MR

[2] Bauerle, N., Rieder, U.: Markov Decision Processes with Applications to Finance. Springer, Heidelberg 2011 | MR

[3] Feinberg, E.: Continuous time discounted jump Markov decision processes: a discrete-event approach. Math. Operat. Res. 29 (2004), 492-524. | DOI | MR

[4] Guo, X. P., Hernández-Lerma, O.: Continuous-Time Markov Decision Process: Theorey and Applications. Springer-Verlag, Berlin 2009. | MR

[5] Guo, X. P., Hernández-Del-Valle, A., Hernández-Lerma, O.: First passage problems for nonstationary discrete-time stochastic control systems. Europ. J. Control 18 (2012), 528-538. | DOI | MR

[6] Guo, X. P., Song, X. Y., Zhang, Y.: First passage optimality for continuous time Markov decision processes with varying discount factors and history-dependent policies. IEEE Trans. Automat. Control 59 (2014), 163-174. | DOI | MR

[7] Guo, X. P., Huang, X. X., Huang, Y. H.: Finite-horizon optimality for continuous-time Markov decision processs with unbounded transition rates. Adv. Appl. Prob. 47 (2015), 1064-1087. | DOI | MR

[8] Hernández-Lerma, O., Lasserre, J. B.: Discrete-Time Markov Control Process: Basic Optimality Criteria. Springer-Verlag, New York 1996. | DOI | MR

[9] Hernández-Lerma, O., Lasserre, J. B.: Further Topics on Discrete-Time Markov Control Process. Springer-Verlag, New York 1999. | DOI | MR

[10] Huang, Y. H., Guo, X. P.: Optimal risk probability for first passage models in Semi-Markov processes. J. Math. Anal. Appl. 359 (2009), 404-420. | DOI | MR

[11] Huang, Y. H., Guo, X. P.: First passage models for denumberable Semi-Markov processes with nonnegative discounted cost. Acta. Math. Appl. Sinica 27 (2011), 177-190. | DOI | MR

[12] Huang, Y. H., Wei, Q. D., Guo, X. P.: Constrained Markov decision processes with first passage criteria. Ann. Oper. Res. 206 (2013), 197-219. | DOI | MR

[13] Huang, Y. H., Guo, X. P., Li, Z. F.: Minimum risk probability for finite horizon semi-Markov decision process. J. Math. Anal. Appl. 402 (2013), 378-391. | DOI | MR

[14] Huang, X. X., Zou, X. L., Guo, X. P.: A minimization problem of the risk probability in first passage semi-Markov decision processes with loss rates. Sci. China Math. 58 (2015), 1923-1938. | DOI | MR

[15] Huang, X. X., Huang, Y. H.: Mean-variance optimality for semi-Markov decision processes under first passage. Kybernetika 53 (2017), 59-81. | DOI | MR

[16] Huo, H. F., Zou, X. L., Guo, X. P.: The risk probability criterion for discounted continuous-time Markov decision processes. Discrete Event Dynamic system: Theory Appl. 27 (2017), 675-699. | DOI | MR

[17] Janssen, J., Manca, R.: Semi-Markov Risk Models For Finance, Insurance, and Reliability. Springer, New York 2006. | MR

[18] Lin, Y. L., Tomkins, R. J., Wang, C. L.: Optimal models for the first arrival time distribution function in continuous time with a special case. Acta. Math. Appl. Sinica 10 (1994), 194-212. | DOI | MR

[19] Liu, J. Y., Liu, K.: Markov decision programming - the first passage model with denumerable state space. Systems Sci. Math. Sci. 5 (1992), 340-351. | MR

[20] Liu, J. Y., Huang, S. M.: Markov decision processes with distribution function criterion of first-passage time. Appl. Math. Optim. 43 (2001), 187-201. | DOI | MR

[21] Ohtsubo, Y.: Optimal threshold probability in undiscounted Markov decision processes with a target set. Appl. Math. Anal. Comp. 149 (2004), 519-532. | DOI | MR

[22] Puterman, M. L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. | MR

[23] Piunovskiy, A., Zhang, Y.: Discounted continuous-time Markov decision processes with unbounded rates: the convex analytic approach. SIAM J. Control Optim. 49 (2011), 2032-2061. | DOI | MR

[24] Schäl, M.: Control of ruin probabilities by discrete-time investments. Math. Meth. Oper. Res. 70 (2005), 141-158. | DOI | MR

[25] Wu, C. B., Lin, Y. L.: Minimizing risk models in Markov decision processes with policies depending on target values. J. Math. Anal. Appl. 231 (1999), 47-57. | DOI | MR

[26] Wu, X., Guo, X. P.: First passage optimality and variance minimization of Markov decision processes with varying discount factors. J. Appl. Prob. 52 (2015), 441-456. | DOI | MR

[27] Yu, S. X., Lin, Y. L., Yan, P. F.: Optimization models for the first arrival target distribution function in discrete time. J. Math. Anal. Appl. 225 (1998), 193-223. | DOI | MR

[28] Zou, X. L., Guo, X. P.: Another set of verifiable conditions for average Markov decision processes with Borel spaces. Kybernetika 51 (2015), 276-292. | DOI | MR