Abstract
In this chapter we indicate that when it is scaled properly, an absorbing Markov decision process (MDP) model with an expected total undiscounted cost can be approximated by two of its deterministic and continuous analogues, namely, the standard fluid model and the refined fluid model, at least when the scaling parameter \(n = 1,2,\ldots \) grows large. We obtain the level of accuracy of such fluid approximations by showing that the absolute difference between the objective function of the scaled MDP model and the one of the (standard and refined) fluid model goes to zero as fast as \(\frac{1} {n}.\) Under some extra conditions, we obtain that given a particular type of policy solving the refined fluid model, it can be translated into a policy, which is nearly optimal for the scaled MDP model.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Altman, E.: Constrained Markov Decision Processes. Chapman and Hall/CRC, Boca Raton (1999)
Avrachenkov, K., Ayesta, U., Piunovskiy, A.: Convergence of trajectories and optimal buffer sizing for AIMD congestion control. Perform. Evaluation. 67, 501–527 (2010)
Avrachenkov, K., Piunovskiy, A., Zhang, Y.: Asymptotic fluid optimality and efficiency of tracking policy for bandwidth-sharing networks. J. Appl. Probab. bf 48, 90–113 (2011)
Bäuerle, N.: Optimal control of queueing networks: an approach via fluid models. Adv. Appl. Prob. 34, 313–328 (2002)
Bertsekas, D., Shreve, S.: Stochastic Optimal Control. Academic Press, NY (1978)
Jacko, P., Sansó, B.: Optimal anticipative congestion control of flows with time-varying input stream. Perform. Evaluation. 69, 86–101 (2012)
Hernández-Lerma, O., Lasserre, J.: Discrete-time Markov Control Processes. Springer-Verlag, NY (1996)
Dai, J.: On positive Harris recurrence of multiclass queueing networks: a unified approach via fluid limit models. Ann. Appl. Prob. 5, 49–77 (1995)
Foss, S., Kovalevskii, A.: A stability criterion via fluid limits and its application to a Polling system. Queueing. Syst. 32, 131–168 (1999)
Mandelbaum, A., Pats, G.: State-dependent queues: approximations and applications. In Kelly, F., Williams, R. (eds.) Stochastic Networks, pp. 239–282. Springer, NY (1995)
Piunovskiy, A., Clancy, D.: An explicit optimal intervention policy for a deterministic epidemic model. Optim. Contr. Appl. Met. 29, 413–428 (2008)
Piunovskiy, A.: Random walk, birth-and-death process and their fluid approximations: absorbing case. Math. Meth. Oper. Res. 70, 285–312 (2009)
Piunovskiy, A., Zhang, Y.: Accuracy of fluid approximations to controlled Birth-and-Death processes: absorbing case. Math. Meth. Oper. Res. 73, 159–187 (2011)
Piunovskiy, A., Zhang, Y.: On the fluid approximations of a class of general inventory level-dependent EOQ and EPQ models. Adv. Oper. Res. (2011) doi: 10.1155/2011/301205
Puterman, M.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, NY (1994)
Zhang, Y., Piunovskiy, A., Ayesta, U., Avrachenkov, K.: Convergence of trajectories and optimal buffer sizing for MIMD congestion control. Com. Com. 33, 149–159 (2010)
Acknowledgements
Mr. Mantas Vykertas kindly helped us improve the English presentation of this chapter. We thank the referee for valuable comments, too.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Piunovskiy, A., Zhang, Y. (2012). Fluid Approximations to Markov Decision Processes with Local Transitions. In: Hernández-Hernández, D., Minjárez-Sosa, J. (eds) Optimization, Control, and Applications of Stochastic Systems. Systems & Control: Foundations & Applications. Birkhäuser, Boston. https://doi.org/10.1007/978-0-8176-8337-5_13
Download citation
DOI: https://doi.org/10.1007/978-0-8176-8337-5_13
Published:
Publisher Name: Birkhäuser, Boston
Print ISBN: 978-0-8176-8336-8
Online ISBN: 978-0-8176-8337-5
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)