Abstract
Police Iteration (PI) is a widely used traditional method for solving Markov Decision Processes (MDPs). In this paper, the cosine policy iteration (CPI) method for solving complex problems formulated as infinite-horizon MDPs is proposed. CPI combines the advantages of two methods: i) Cosine Simplex Method (CSM) which is based on the Karush, Kuhn, and Tucker (KKT) optimality conditions and finds rapidly an initial policy close to the optimal solution and ii) PI which is able to achieve the global optimum. In order to apply CSM to this kind of problems, a well- known LP formulation is applied and particular features are derived in this paper. Obtained results show that the application of CPI solves MDPs in a lower number of iterations that the traditional PI.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Putterman, L.: Markov Decision Processes Discrete Stochastic Dynamic Programming. Wiley-Interscience, Hoboken (2005)
Dai, P., Goldsmith, J.: Topological Value Iteration Algorithm for Markov Decision Processes. International Join Conference on Artificial Intelligence (2007)
Bernstein, D.S., Amato, C., Hanse, E.A., Zilberstein, S.: Policy Iteration for Decentralized Control of Markov Decision Processes. Journal of Artificial Intelligence Research, 89–132 (2009)
Kim, K.-E., Dean, T.L., Meuleau, N.: Approximate solutions to Factored Markov Decision Processes via Greedy Search in the Space of Finite State Controllers. American Association for Artificial Intelligence (2000)
Howard, R.: Dynamic Programming and Markov processes. Wiley, New York (1960)
Santos, M.S., Rust, J.: Convergence properties of policy iteration. SIAM Journal on Control and Optimization 42(6), 2094–2115 (2003)
Dantzig, G.B., Thapa, M.N.: Linear Programming. 1: Introduction. Springer Series in Operations Research and Financial Engineering, 1st edn. (1997)
Chvátal, V.: Linear Programming. W. H. Freeman and Company, New York (1983)
Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (2003) Republished 2003: Dover; ISBN 0486428095
Bertsekas, D.P.: Network Optimization continuous and discrete models. Athena Scientific (1998)
Bello, D., Riano, G.: Linear Programming solvers for Markov Decision Processes. Systems and Information Engineering Design Symposium, IEEE 28(28), 90–95 (2006)
Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, Heidelberg (1999)
Trigos, F., Frausto-Solis, J., Rivera-Lopez, R.: A Simplex Cosine Method for Solving the Klee-Minty Cube. In: Advances in Simulation System Theory and Systems Engineering, vol. 70X, pp. 27–32. WSEAS Press (2002) ISBN 960852
Trigos, F., Frausto-Solis, J.: Experimental Evaluation of the Theta Heuristic for Starting the Cosine Simplex Method. In: International Conference in Computational Science and its Applications, ICCSA, Proceedings Part IV (2005)
Corley, H.W., Rosenberger, J., Wei-Chang, Y., Sung, T.K.: The Cosine Simplex Algorithm. The International Journal of Advanced Manufacturing Technology 27(9-10), 1047–1050 (2006)
Wei-Chang, Y., Corley, H.W.: A simple direct cosine simplex algorithm. Applied Mathematics and Computation 214(1), 178–186 (2009)
Netlib Repository at UTK and ORNL, http://www.netlib.org/lp/data/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Frausto-Solis, J., Santiago, E., Mora-Vargas, J. (2009). Cosine Policy Iteration for Solving Infinite-Horizon Markov Decision Processes. In: Aguirre, A.H., Borja, R.M., Garciá, C.A.R. (eds) MICAI 2009: Advances in Artificial Intelligence. MICAI 2009. Lecture Notes in Computer Science(), vol 5845. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-05258-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-05258-3_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-05257-6
Online ISBN: 978-3-642-05258-3
eBook Packages: Computer ScienceComputer Science (R0)