Markov decision processes with time-varying discount factors and random horizon

Ilhuicatzi-Roldán, Rocio ; Cruz-Suárez, Hugo ; Chávez-Rodríguez, Selene

Kybernetika, Tome 53 (2017) no. 1, pp. 82-98

Abstract (VO)

This paper is related to Markov Decision Processes. The optimal control problem is to minimize the expected total discounted cost, with a non-constant discount factor. The discount factor is time-varying and it could depend on the state and the action. Furthermore, it is considered that the horizon of the optimization problem is given by a discrete random variable, that is, a random horizon is assumed. Under general conditions on Markov control model, using the dynamic programming approach, an optimality equation for both cases is obtained, namely, finite support and infinite support of the random horizon. The obtained results are illustrated by two examples, one of them related to optimal replacement.

Détail

DOI : 10.14736/kyb-2017-1-0082

Classification : 90C39, 90C40, 93E20
Keywords: Markov decision process; dynamic programming; varying discount factor; random horizon

Bibliographie

[1] Carmon, Y., Shwartz, A.: Markov decision processes with exponentially representable discounting. Oper. Res. Lett. 37 (2009), 51-55. | DOI | MR | Zbl

[2] Chen, X., Yang, X.: Optimal consumption and investment problem with random horizon in a BMAP model. Insurance Math. Econom. 61 (2015), 197-205. | DOI | MR | Zbl

[3] Cruz-Suárez, H., Ilhuicatzi-Roldán, R., Montes-de-Oca, R.: Markov decision processes on Borel spaces with total cost and random horizon. J. Optim. Theory Appl. 162 (2014), 329-346. | DOI | MR | Zbl

[4] Vecchia, E. Della, Marco, S. Di, Vidal, F.: Dynamic programming for variable discounted Markov decision problems. In: Jornadas Argentinas de Informática e Investigación O\-pe\-ra\-ti\-va (43JAIIO) XII Simposio Argentino de Investigación Operativa (SIO), Buenos Aires 2014, pp. 50-62.

[5] Feinberg, E., Shwartz, A.: Constrained dynamic programming with two discount factors: applications and an algorithm. IEEE Trans. Automat. Control 44 (1999), 628-631. | DOI | MR | Zbl

[6] Feinberg, E., Shwartz, A.: Markov decision models with weighted discounted criteria. Math. Oper. Res. 19 (1994), 152-168. | DOI | MR | Zbl

[7] García, Y. H., González-Hernández, J.: Discrete-time Markov control process with recursive discounted rates. Kybernetika 52 (2016), 403-426. | DOI | MR

[8] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A.: Adaptive policies for stochastic systems under a randomized discounted criterion. Bol. Soc. Mat. Mex. 14 (2008), 149-163. | MR

[9] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A.: Approximation, estimation and control of stochastic systems under a randomized discounted cost criterion. Kybernetika 45 (2009), 737-754. | MR | Zbl

[10] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A., Gabriel-Arguelles, J. A.: Constrained Markov control processes with randomized discounted cost criteria: occupation measures and external points. Risk and Decision Analysis 4 (2013), 163-176.

[11] González-Hernández, J., López-Martínez, R. R., Minjarez-Sosa, J. A., Gabriel-Arguelles, J. A.: Constrained Markov control processes with randomized discounted rate: infinite linear programming approach. Optimal Control Appl. Methods 35 (2014), 575-591. | DOI | MR

[12] González-Hernández, J., López-Martínez, R. R., Pérez-Hernández, J. R.: Markov control processes with randomized discounted cost. Math. Methods Oper. Res. 65 (2007), 27-44. | DOI | MR | Zbl

[13] Guo, X., Hernández-del-Valle, A., Hernández-Lerma, O.: First passage problems for nonstationary discrete-time stochastic control systems. Eur. J. Control 18 (2012), 528-538. | DOI | MR | Zbl

[14] Hernández-Lerma, O., Laserre, J. B.: Discrete-time Markov Control Processes: Basic Optimality Criteria. Springer-Verlag, New York 1996. | DOI | MR

[15] Hinderer, K.: Foundations of non-stationary dynamic programming with discrete time parameter. In: Lectures Notes Operations Research (M. Bechmann and H. Künzi, eds.), Springer-Verlag 33, Zürich 1970. | DOI | MR | Zbl

[16] Ilhuicatzi-Roldán, R., Cruz-Suárez, H.: Optimal replacement in a system of $n$-machines with random horizon. Proyecciones 31 (2012), 219-233. | DOI | MR | Zbl

[17] Minjares-Sosa, J. A.: Markov Control Models with unknown random state-action-dependent discounted factors. TOP 23 (2015), 743-772. | DOI | MR

[18] Puterman, M. L.: Markov Decision Process: Discrete Stochastic Dynamic Programming. John Wiley and Sons, New York 1994. | MR

[19] Sch{ä}l, M.: Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal. Probab. Theory Related Fields 32 (1975), 179-196. | DOI | MR | Zbl

[20] Wei, Q., Guo, X.: Markov decision processes with state-dependent discounted factors and unbounded rewards/costs. Oper. Res. Lett. 39 (2011), 369-374. | DOI | MR

[21] Wei, Q., Guo, X.: Semi-Markov decision processes with variance minimization criterion. 4OR, 13 (2015), 59-79. | DOI | MR | Zbl

[22] Wu, X., Guo, X.: First passage optimality and variance minimisation of Markov decision processes with varying discounted factors. J. Appl. Probab. 52 (2015), 441-456. | DOI | MR

[23] Wu, X., Zou, X., Guo, X.: First passage Markov decision processes with constraints and varying discount factors. Front. Math. China 10 (2015), 1005-1023. | DOI | MR | Zbl

[24] Wu, X., Zhang, J.: An application to the finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors. In: Proc. 11th World Congress on Intelligent Control and Automation 2015, pp. 1745-1748. | DOI | MR

[25] Wu, X., Zhang, J.: Finite approximation of the first passage models for discrete-time Markov decision processes with varying discounted factors. Discrete Event Dyn. Syst. 26 (2016), 669-683. | DOI | MR

[26] Ye, L., Guo, X.: Continuous-time Markov decision processes with state-dependent discount factors. Acta Appl. Math. 121 (2012), 5-27. | DOI | MR | Zbl

[27] Zhang, Y.: Convex analytic approach to constrained discounted Markov decision processes with non-constant discount factors. TOP 21 (2013), 378-408. | DOI | MR | Zbl