Abstract
We deal with countable state Markov decision processes with finite action sets and (possibly) unbounded costs. Assuming the existence of an expected average cost optimal stationary policyf, with expected average costg, when canf andg be found using undiscounted value iteration? We give assumptions guaranteeing the convergence of a quantity related tong−Ν n (i), whereΝ n (i) is the minimum expectedn-stage cost when the process starts in statei. The theory is applied to a queueing system with variable service rates and to a queueing system with variable arrival parameter.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
D.P. Bertsekas,Dynamic Programming (Prentice-Hall, Englewood Cliffs, NJ, 1987).
A. Federgruen and P.J. Schweitzer, A survey of asymptotic value-iteration for undiscounted Markovian decision processes, in:Recent Developments in Markov Decision Processes, R. Hartley, L.C. Thomas and D.J. White (eds.) (Academic Press, New York, 1980).
O. Hernandez-Lerma,Adaptive Markov Control Processes (Springer, New York, 1989).
A. Hordijk, P.J. Schweitzer and H. Tijms, The asymptotic behavior of the minimal total expected cost for the denumerable state Markov decision model, J. Appl. Prob. 12 (1975) 298–305.
V.G. Kulkarni and R.L. Karandikar, Convergence of moments of Markov and semi-Markov processes, University of North Carolina, Operations Research and Systems Analysis Technical Report 86-20 (1986).
A.G. Pakes, Some conditions for ergodicity and recurrence of Markov chains, Oper. Res. 17 (1969) 1058–1061.
S.M. Ross,Introduction to Stochastic Dynamic Programming (Academic Press, New York, 1983).
P.J. Schweitzer and A. Federgruen, The asymptotic behavior of undiscounted value iteration in Markov decision problems, Math. Oper. Res. 2 (1977) 360–381.
L.I. Sennott, A new condition for the existence of optimal stationary policies in average cost Markov decision processes, Oper. Res. Lett. 5 (1986) 17–23.
L.I. Sennott, A new condition for the existence of optimum stationary policies in average cost Markov decision processes — Unbounded cost case,Proc. 25th IEEE Conf. on Decision Control, Athens, Greece (1986) pp. 1719–1721.
L.I. Sennott, Average cost optimal stationary policies in infinite state Markov decision processes with unbounded costs, Oper. Res. 37 (1989) 626–633.
L.I. Sennott, Average cost semi-Markov decision processes and the control of queueing systems, Prob. Eng. Inform. Sci. 3 (1989) 247–272.
R.L. Tweedie, Hitting times of Markov chains, with application to state-dependent queues, Bull. Austral. Math. Soc. 17 (1977) 97–107.
D.J. White, Dynamic programming, Markov chains, and the method of successive approximations, J. Math. Anal. Appl. 6 (1963) 373–376.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Sennott, L.I. Value iteration in countable state average cost Markov decision processes with unbounded costs. Ann Oper Res 28, 261–271 (1991). https://doi.org/10.1007/BF02055585
Issue Date:
DOI: https://doi.org/10.1007/BF02055585