Discrete-time Markov control processes with recursive discount rates

García, Yofre H.   ; González-Hernández, Juan

Kybernetika, Tome 52 (2016) no. 3, pp. 403-426

Abstract (VO)

This work analyzes a discrete-time Markov Control Model (MCM) on Borel spaces when the performance index is the expected total discounted cost. This criterion admits unbounded costs. It is assumed that the discount rate in any period is obtained by using recursive functions and a known initial discount rate. The classic dynamic programming method for finite-horizon case is verified. Under slight conditions, the existence of deterministic non-stationary optimal policies for infinite-horizon case is proven. Also, to find deterministic non-stationary $\epsilon-$optimal policies, the value-iteration method is used. To illustrate an example of recursive functions that generate discount rates, we consider the expected values of stochastic processes, which are solutions of certain class of Stochastic Differential Equations (SDE) between consecutive periods, when the initial condition is the previous discount rate. Finally, the consumption-investment problem and the discount linear-quadratic problem are presented as examples; in both cases, the discount rates are obtained using a SDE, similar to the Vasicek short-rate model.

Détail

DOI : 10.14736/kyb-2016-3-0403

Classification : 49L20, 93E20
Keywords: dynamic programming method; optimal stochastic control

Bibliographie

[1] Arnold, L.: Stochastic Differential Equations: Theory and Applications. John Wiley and Sons, New York 1973. | DOI | MR | Zbl

[2] Ash, R., Doléans-Dade, C.: Probability and Measure Theory. Academic Press, San Diego 2000. | MR | Zbl

[3] Bellman, R.: Dynamic Programming. Princeton Univ. Press, New Jersey 1957. | MR | Zbl

[4] Bertsekas, D., Shreve, S.: Stochastic Optimal Control: The Discrete Time Case. Athena Scientific, Massachusetts 1996. | MR | Zbl

[5] Brigo, D., Mercurio, F.: Interest Rate Models Theory and Practice. Springer-Verlag, New York 2001. | DOI | MR | Zbl

[6] Black, F., Karasinski, P.: Bond and option pricing when short rates are lognormal. Financ. Anal. J. 47 (1991), 4, 52-59. | DOI

[7] Carmon, Y., Shwartz, A.: Markov decision processes with exponentially representable discounting. Oper. Res. Lett. 37 (2009), 51-55. | DOI | MR | Zbl

[8] Vecchia, E. Della, Marco, S. Di, Vidal, F.: Dynamic programming for variable discounted Markov decision problems. In: Jornadas Argentinas de Informática e Investigación Operativa (43JAIIO) - XII Simposio Argentino de Investigación Operativa (SIO), Buenos Aires, 2014, pp. 50-62.

[9] Cox, J., Ingersoll, J., Ross, S.: A theory of the term structure of interest rates. Econometrica 53 (1985), 385-407. | DOI | MR | Zbl

[10] Dothan, U.: On the term structure interest rates. J. Financ. Econ. 6 (1978), 59-69. | DOI

[11] Feinberg, E., Shwartz, A.: Markov decision models with weighted discounted criteria. J. Finan. Econ. 19 (1994), 152-168. | DOI | MR | Zbl

[12] González-Hernández, J., López-Martínez, R., Pérez-Hernández, J.: Markov control processes with randomized discounted cost. Math. Method Oper. Res. 65 (2006), 1, 27-44. | DOI | MR | Zbl

[13] González-Hernández, J., López-Martínez, R., Minjarez-Sosa, A.: Adaptive policies for stochastic systems under a randomized discounted cost criterion. Bol. Soc. Mat. Mex. 14 (2008), 3, 149-163. | MR | Zbl

[14] González-Hernández, J., López-Martínez, R., Minjarez-Sosa, A.: Approximation, estimation and control of stochastic systems under a randomized discounted cost criterion. Kybernetika 45 (2008), 5, 737-754. | MR | Zbl

[15] Guo, X., Hernández-Del-Valle, A., Hernández-Lerma, O.: First passage problems for a non-stationary discrette-time stochastic control systems. Eur. J. Control 15 (2012), 7, 528-538. | DOI | MR

[16] Hernández-Lerma, O., Lasserre, J. B.: Discrete-Time Markov Control Processes. Basic Optimality Criteria. Springer-Verlag, New York 1996. | DOI | MR | Zbl

[17] Minjarez-Sosa, J.: Markov control models with unknown random sate-action-dependent discount factors. TOP 23 (2015), 3, 743-772. | DOI | MR

[18] Hinderer, K.: Foundations of non-stationary dynamical programming with discrete time parameter. In: Lecture Notes Operations Research (M. Bechmann and H. Künzi, eds.), Springer-Verlag 33, Zürich 1970. | DOI | MR

[19] Ho, T., Lee, S.: Term structure movements and pricing interest rate claims. J. Financ. 41 (1986), 1011-1029. | DOI

[20] Hull, J.: Options, Futures and other Derivatives. Sixth edition. Prentice Hall, New Jersey 2006.

[21] Hull, J., White, A.: Pricing interest rate derivative securities. Rev. Financ. Stud. 3 (1990), 573-592. | DOI

[22] Mercurio, F., Moraleda, J.: A family of humped volatility models. Eur. J. Finance 7 (2001), 93-116. | DOI

[23] Rendleman, R., Bartter, B.: The pricing of options on debt securities. J. Financ. Quant. Anal. 15 (1980), 11-24. | DOI

[24] Rieder, U.: Measurable selection theorems for optimization problems. Manuscripta Math. 24 (1978), 115-131. | DOI | MR | Zbl

[25] Schäl, M.: Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal. Z. Wahrscheinlichkeit 32 (1975), 179-196. | DOI | MR | Zbl

[26] Vasicek, O.: An equilibrium characterization of the term structure. J. Financ. Econ. 5 (1977), 177-188. | DOI

[27] Wei, Q., X, X. Guo: Markov decision processes with state-dependent discount factors and unbounded rewards costs. Oper. Res. Lett. 39 (2011), 369-374. | DOI | MR | Zbl

[28] Ye, L., Guo, X.: Continuous-time Markov decision processes with state-dependent discount factors. Acta Appl. Math. 121 (2012), 1, 5-27. | DOI | MR | Zbl

[29] Zhang, Y.: Convex analytic approach to constrained discounted Markov decision processes with non-constant discount factors. TOP 21 (2013), 2, 378-408. | DOI | MR | Zbl