Risk-sensitive Markov stopping games with an absorbing state

López-Rivero, Jaicer ; Cavazos-Cadena, Rolando ; Cruz-Suárez, Hugo

Kybernetika, Tome 58 (2022) no. 1, pp. 101-122

Abstract (VO)

This work is concerned with discrete-time Markov stopping games with two players. At each decision time player II can stop the game paying a terminal reward to player I, or can let the system to continue its evolution. In this latter case player I applies an action affecting the transitions and entitling him to receive a running reward from player II. It is supposed that player I has a no-null and constant risk-sensitivity coefficient, and that player II tries to minimize the utility of player I. The performance of a pair of decision strategies is measured by the risk-sensitive (expected) total reward of player I and, besides mild continuity-compactness conditions, the main structural assumption on the model is the existence of an absorbing state which is accessible from any starting point. In this context, it is shown that the value function of the game is characterized by an equilibrium equation, and the existence of a Nash equilibrium is established.

Détail

DOI : 10.14736/kyb-2022-1-0101

Classification : 60J05, 93C55, 93E20
Keywords: monotone operator; fixed point; equilibrium equation; hitting time; bounded rewards; certainty equivalent

Bibliographie

[1] Alanís-Durán, A., Cavazos-Cadena, R.: An optimality system for finite average Markov decision chains under risk-aversion. Kybernetika 48 (2012), 83-104. | MR

[2] Altman, E., Shwartz, A.: Constrained Markov games: Nash equilibria. In: Annals of Dynamic Games (V. Gaitsgory, J. Filar, and K. Mizukami, eds.), Birkhauser, Boston 2000, pp. 213-221. | MR

[3] Atar, R., Budhiraja, A.: A stochastic differential game for the inhomogeneous Laplace equation. Ann. Probab. 38 (2010), 2, 498-531. | DOI | MR

[4] Balaji, S., Meyn, S. P.: Multiplicative ergodicity and large deviations for an irreducible Markov chain. Stoch. Proc. Appl. 90 (2000), 1, 123-144. | DOI | MR

[5] Bäuerle, N., Rieder, U.: Markov Decision Processes with Applications to Finance. Springer, New York 2011. | MR | Zbl

[6] Bäuerle, N., Rieder, U.: More risk-sensitive Markov decision processes. Math. Oper. Res. 39 (2014), 1, 105-120. | DOI | MR

[7] Bäuerle, N., Rieder, U.: Zero-sum risk-sensitive stochastic games. Stoch. Proc. Appl. 127 (2017), 2, 622-642. | DOI | MR

[8] Bielecki, T. R., Hernández-Hernández, D., Pliska, S. R.: Risk sensitive control of finite state Markov chains in discrete time, with applications to portfolio management. Mathematical Methods of OR 50 (1999), 167-188. | DOI | MR | Zbl

[9] Borkar, V. S., Meyn, S. F.: Risk-sensitive optimal control for Markov decision process with monotone cost. Math. Oper. Res. 27 (2002), 1, 192-209. | DOI | MR

[10] Cavazos-Cadena, R., Hernández-Hernández, D.: A system of Poisson equations for a non-constant {Varadhan} functional on a finite state space. Appl. Math. Optim. 53 (2006), 101-119. | DOI | MR

[11] Cavazos-Cadena, R., Hernández-Hernández, D.: Nash equilibria in a class of Markov stopping games. Kybernetika 48 (2012), 5, 1027-1044. | MR

[12] Cavazos-Cadena, R., Rodríguez-Gutiérrez, L., Sánchez-Guillermo, D. M.: Markov stopping games with an absorbing state and total reward criterion. Kybernetika 57 (2021), 474-492. | DOI | MR

[13] Denardo, E. V., Rothblum, U. G.: A turnpike theorem for a risk-sensitive Markov decision process with stopping. SIAM J. Control Optim. 45 (2006), 2, 414-431. | DOI | MR

[14] Masi, G. B. Di, Stettner, L.: Risk-sensitive control of discrete time Markov processes with infinite horizon. SIAM J. Control Optim. 38 (1999), 1, 61-78. | DOI | MR

[15] Masi, G. B. Di, Stettner, L.: Infinite horizon risk sensitive control of discrete time Markov processes with small risk. Syst. Control Lett. 40 (2000), 15-20. | DOI | MR | Zbl

[16] Masi, G. B. Di, Stettner, L.: Infinite horizon risk sensitive control of discrete time Markov processes under minorization property. SIAM J. Control Optim. 46 (2007), 1, 231-252. | DOI | MR

[17] A.Filar, J., Vrieze, O. J.: Competitive Markov Decision Processes. Springer, New York 1996. | MR

[18] Hernández-Lerma, O.: Adaptive Markov Control Processes. Springer, New York 1989. | MR | Zbl

[19] Howard, R. A., Matheson, J. E.: Risk-sensitive Markov decision processes. Manage. Sci. 18 (1972), 7, 349-463. | DOI | MR

[20] Jaśkiewicz, A.: Average optimality for risk sensitive control with general state space. Ann. Appl. Probab. 17 (2007), 2, 654-675. | DOI | MR

[21] Kolokoltsov, V. N., Malafeyev, O. A.: Understanding Game Theory. World Scientific, Singapore 2010. | MR | Zbl

[22] Kontoyiannis, I., Meyn, S. P.: Spectral theory and limit theorems for geometrically ergodic Markov processes. Ann. Appl. Probab. 13 (2003), 1, 304-362. | DOI | MR

[23] Martínez-Cortés, V. M.: Bi-personal stochastic transient Markov games with stopping times and total reward criterion. Kybernetika 57 (2021), 1, 1-14. | DOI | MR

[24] Peskir, G.: On the American option problem. Math. Finance 15 (2007), 169-181. | DOI | MR | Zbl

[25] Peskir, G., Shiryaev, A.: Optimal Stopping and Free-Boundary Problems. Birkhauser, Boston 2006. | MR | Zbl

[26] Pitera, M., Stettner, L.: Long run risk sensitive portfolio with general factors. Math. Meth. Oper. Res. 82 (2016), 2, 265-293. | DOI | MR

[27] Puterman, M. L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York 1994. | MR | Zbl

[28] Shapley, L. S.: Stochastic games. Proc. National Academy Sci. 39 (1953), 10, 1095-1100. | MR | Zbl

[29] Shiryaev, A.: Optimal Stopping Rules. Springer, New York 2008. | MR | Zbl

[30] Sladký, K.: Growth rates and average optimality in risk-sensitive Markov decision chains. Kybernetika 44 (2008), 2, 205-226. | MR

[31] Sladký, K.: Risk-sensitive average optimality in Markov decision processes. Kybernetika 54 (2018), 6, 1218-1230. | DOI | MR

[32] Stettner, L.: Risk sensitive portfolio optimization. Math. Meth. Oper. Res. 50 (1999), 3, 463-474. | DOI | MR

[33] Zachrisson, L. E.: Markov Games. Princeton University Press 12, Princeton 1964. | MR