An unbounded Berge's minimum theorem with applications to discounted Markov decision processes

Montes-de-Oca, Raúl   ; Lemus-Rodríguez, Enrique

Kybernetika, Tome 48 (2012) no. 2, pp. 268-286

Abstract (VO)

This paper deals with a certain class of unbounded optimization problems. The optimization problems taken into account depend on a parameter. Firstly, there are established conditions which permit to guarantee the continuity with respect to the parameter of the minimum of the optimization problems under consideration, and the upper semicontinuity of the multifunction which applies each parameter into its set of minimizers. Besides, with the additional condition of uniqueness of the minimizer, its continuity is given. Some examples of nonconvex optimization problems that satisfy the conditions of the article are supplied. Secondly, the theory developed is applied to discounted Markov decision processes with unbounded cost functions and with possibly noncompact actions sets in order to obtain continuous optimal policies. This part of the paper is illustrated with two examples of the controlled Lindley's random walk. One of these examples has nonconstant action sets.

Détail

Classification : 90A16, 90C40, 93E20
Keywords: Berge's minimum theorem; moment function; discounted Markov decision process; uniqueness of the optimal policy; continuous optimal policy

Bibliographie

[1] C. D. Aliprantis, K. C. Border: Infinite Dimensional Analysis. Third Edition. Springer-Verlag, Berlin 2006. | MR | Zbl

[2] R. B. Ash: Real Variables with Basic Metric Space Topology. IEEE Press, New York 1993. | MR | Zbl

[3] J. P. Aubin, I. Ekeland: Applied Nonlinear Analysis. John Wiley, New York 1984. | MR | Zbl

[4] L. M. Ausubel, R. J. Deneckere: A generalized theorem of the maximum. Econom. Theory 3 (1993), 99-107. | DOI | MR | Zbl

[5] C. Berge: Topological Spaces. Oliver and Boyd, Edinburgh and London 1963 (reprinted by Dover Publications, Inc., Mineola, New York 1997). | MR | Zbl

[6] D. Cruz-Suárez, R. Montes-de-Oca, F. Salem-Silva: Conditions for the uniqueness of optimal policies of discounted Markov decision processes. Math. Methods Oper. Res. 60 (2004), 415-436. | DOI | MR | Zbl

[7] J. Dugundji: Topology. Allyn and Bacon, Inc., Boston 1966. | MR | Zbl

[8] P. K. Dutta, M. K.Majumdar, R. K. Sundaram: Parametric continuity in dynamic programming problems. J. Econom. Dynam. Control 18 (1994), 1069-1092. | DOI | MR | Zbl

[9] P. K. Dutta, T. Mitra: Maximum theorems for convex structures with an application to the theory of optimal intertemporal allocations. J. Math. Econom. 18 (1989), 77-86. | DOI | MR

[10] O. Hernández-Lerma, J. B. Lasserre: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer-Verlag, New York 1996. | MR | Zbl

[11] O. Hernández-Lerma, W. J. Runggaldier: Monotone approximations for convex stochastic control problems. J. Math. Systems Estim. Control 4 (1994), 99-140. | MR | Zbl

[12] K. Hinderer: Lipschitz continuity of value functions in Markovian decision Processes. Math. Methods Oper. Res. 60 (2005), 3-22. | MR | Zbl

[13] K. Hinderer, M. Stieglitz: Increasing and Lipschitz continuous minimizers in one-dimensional linear-convex systems without constraints: the continuous and the discrete case. Math. Methods Oper. Res. 44 (1996), 189-204. | DOI | MR | Zbl

[14] A. Horsley, A. J. Wrobel, T. Van Zandt: Berge's maximum theorem with two topologies on the action set. Econom. Lett. 61 (1998), 285-291. | DOI | MR | Zbl

[15] J. S. Jordan: The continuity of optimal dynamic decision rules. Econometrica 45 (1977), 1365-1376. | DOI | MR | Zbl

[16] T. Kamihigashi: Stochastic optimal growth with bounded or unbounded utility and with bounded or unbounded shocks. J. Math. Econom. 43 (2007), 477-500. | DOI | MR | Zbl

[17] T. Kamihigashi, S. Roy: A nonsmooth, nonconvex model of optimal growth. J. Econom. Theory 132 (2007), 435-460. | DOI | MR | Zbl

[18] R. B. King: Beyond Quartic Equation. Birkhauser, Boston 1996. | MR

[19] M. Kitayev: Semi-Markov and jump Markov control models: average cost criterion. Theory Probab. Appl. 30 (1985), 272-288. | MR

[20] D. V. Lindley: The theory of queues with a single server. Proc. Cambridge Philos. Soc. 48 (1952), 277-289. | MR | Zbl

[21] M. Majumdar, R. Radner: Stationary optimal policies with discounting in a stochastic activity analysis model. Econometrica 51 (1983), 1821-1837. | DOI | MR

[22] S. P. Meyn: Ergodic Theorems for discrete time stochastic systems using a stochastic Lyapunov functions. SIAM J. Control Optim. 27 (1989), 1409-1439. | DOI | MR

[23] E. A. Ok: Real Analysis with Economic Applications. Princeton University Press, Princeton 2007. | MR | Zbl

[24] A. L. Peressini, F. E. Sullivan, J. J. Uhl: The Mathematics of Nonlinear Programming. Springer-Verlag, New York 1988. | MR | Zbl

[25] M. L. Puterman: Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley, New York 1994. | MR | Zbl

[26] U. Rieder: Measurable selection theorems for optimization problems. Manuscripta Math. 24 (1978), 115-131. | DOI | MR | Zbl

[27] H. L. Royden: Real Analysis. Third Edition. Macmillan, New York 1988. | MR | Zbl

[28] R. H. Stockbridge: Time-average control of martingale problems: a linear programming formulation. Ann. Probab. 18 (1990), 291-314. | MR | Zbl

[29] R. Sundaram: A First Course in Optimization Theory. Cambridge University Press, Cambridge 1996. | MR | Zbl

[30] G. Tian, J. Zhou: The maximum theorem and the existence of Nash equilibrium of (generalized) games without lower semicontinuities. J. Math. Anal. Appl. 166 (1992), 351-364. | DOI | MR | Zbl

[31] G. Tian, J. Zhou: Transfer continuities, generalizations of the Weierstrass and maximum theorem: a full characterization. J. Math. Econom. 24 (1995), 281-303. | DOI | MR

[32] M. Walker: A generalization of the maximum theorem. Internat. Econom. Rev. 20 (1979), 267-272. | DOI | MR | Zbl

[33] A. Yushkevich: Blackwell optimality in Borelian continuous-in-action Markov decision processes. SIAM J. Control Optim. 35 (1997), 2157-2182. | DOI | MR | Zbl