Skip to main content
SpringerLink
Log in

Optimal cost and policy for a Markovian replacement problem

  • Contributed Papers
  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We consider the computation of the optimal cost and policy associated with a two-dimensional Markov replacement problem with partial observations, for two special cases of observation quality. Relying on structural results available for the optimal policy associated with these two particular models, we show that, in both cases, the infinitehorizon, optimal discounted cost function is piecewise linear, and provide formulas for computing the cost and the policy. Several examples illustrate the usefulness of the results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Åström, K. J.,Optimal Control of Markov Processes with Incomplete State Information, Journal of Mathematical Analysis and Applications, Vol. 10, pp. 174–205, 1965.

    Google Scholar 

  2. White, C. C.,Bounds on the Optimal Cost for a Replacement Problem with Partial Observations, Naval Research Logistics Quarterly, Vol. 26, pp. 415–422, 1979.

    Google Scholar 

  3. Bertsekas, D. P.,Dynamic Programming, Prentice-Hall, Englewood Cliffs, New Jersey, 1987.

    Google Scholar 

  4. Albright, S. C.,Structural Results for Partially Observable Markov Decision Processes, Operations Research, Vol. 27, pp. 1041–1053, 1979.

    Google Scholar 

  5. Ross, S. M.,Quality Control under Markovian Deterioration, Management Science, Vol. 17, pp. 587–596, 1971.

    Google Scholar 

  6. Sawaki, K.,Transformation of Partially Observable Markov Decision Processes into Piecewise Linear Ones, Journal of Mathematical Analysis and Applications, Vol. 91, pp. 112–118, 1983.

    Google Scholar 

  7. Sondik, E. J.,The Optimal Control of Partially Observable Markov Processes, PhD Thesis, Department of Electrical Engineering Systems, Stanford University, 1971.

  8. Sondik, E. J.,The Optimal Control of Partially Observable Markov Decision Processes over the Infinite Horizon: Discounted Costs, Operations Research, Vol. 26, pp. 282–304, 1978.

    Google Scholar 

  9. Wang, R. C.,Computing Optimal Control Policies—Two Actions, Journal of Applied Probability, Vol. 13, pp. 826–832, 1976.

    Google Scholar 

  10. Wang, R. C.,Optimal Replacement Policy with Unobservable States, Journal of Applied Probability, Vol. 14, pp. 340–348, 1977.

    Google Scholar 

  11. White, C. C.,A Markov Quality Control Process Subject to Partial Observation, Management Science, Vol. 23, pp. 843–852, 1977.

    Google Scholar 

  12. White, C. C.,Optimal Inspection and Repair of a Production Process Subject to Deterioration, Journal of the Operational Research Society, Vol. 29, pp. 235–243, 1978.

    Google Scholar 

  13. Lovejoy, W. S.,Computationally Feasible Bounds for Partially Observed Markov Decision Processes, Research Paper No. 1024, Graduate School of Business, Stanford University, 1988.

  14. White, C. C., andScherer, W. T.,Solution Procedures for Partially Observed Markov Decision Processes, Operations Research, Vol. 37, pp. 791–797, 1989.

    Google Scholar 

  15. Hernandez-Lerma, O., andMarcus, S. I.,Adaptive Control of Discrete Discounted Markov Decision Chains, Journal of Optimization Theory and Applications, Vol. 46, pp. 227–235, 1985.

    Google Scholar 

  16. Hernandez-Lerma, O., andMarcus, S. I.,Adaptive Control of Markov Processes with Incomplete State Information and Unknown Parameters, Journal of Optimization Theory and Applications, Vol. 52, pp. 227–241, 1987.

    Google Scholar 

  17. Monahan, G. E.,Optimal Stopping in a Partially Observable Binary-Valued Markov Chain with Costly Perfect Information, Journal of Applied Probability, Vol. 19, pp. 72–81, 1982.

    Google Scholar 

  18. Kumar, P. R., andSeidman, T. I.,On the Optimal Solution of the One-Armed Bandit Adaptive Control Problem, IEEE Transactions on Automatic Control, Vol. 26, pp. 1176–1184, 1981.

    Google Scholar 

  19. Thomas, L. C., Jacobs, P. A., andGaver, D. P.,Optimal Inspection Policies for Standby Systems, Communications in Statistics—Stochastic Models, Vol. 3, pp. 259–273, 1987.

    Google Scholar 

  20. Pollock, S. M.,Minimum-Cost Checking Using Imperfect Information, Management Science, Vol. 13, pp. 454–465, 1967.

    Google Scholar 

  21. Hughes, J. S.,Optimal Internal Audit Timing, Accounting Review, Vol. 52, pp. 56–68, 1977.

    Google Scholar 

  22. Hughes, J. S.,A Note on Quality Control under Markovian Deterioration, Operations Research, Vol. 28, pp. 421–424, 1980.

    Google Scholar 

  23. Bertsekas, D. P., andShreve, S. E.,Stochastic Optimal Control: The Discrete Time Case, Academic Press, New York, New York, 1978.

    Google Scholar 

  24. Fernandez-Gaucherand, E., Arapostathis, A., andMarcus, S. I.,On The Adaptive Control of a Partially Observable Markov Decision Process, Proceedings of the 27th Conference on Decision and Control, Austin, Texas, pp. 1204–1210, 1988.

  25. Monahan, G. E.,A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms, Management Science, Vol. 28, pp. 1–16, 1982.

    Google Scholar 

  26. White, C. C., andWhite, D. J.,Markov Decision Processes, European Journal of Operational Research, Vol. 39, pp. 1–16, 1989.

    Google Scholar 

  27. Sawaki, K., andIchikawa, A.,Optimal Control for Partially Observable Markov Decision Processes over an Infinite Horizon, Journal of the Operations Research Society of Japan, Vol. 21, pp. 1–15, 1978.

    Google Scholar 

  28. Martin, J. J.,Bayesian Decision Problems and Markov Chains, John Wiley and Sons, New York, New York, 1967.

    Google Scholar 

  29. Sernik, E. L., andMarcus, S. I.,Comments on the Sensitivity of the Optimal Cost and the Optimal Policy for a Discrete Markov Decision Process, Proceedings of the 27th Annual Allerton Conference on Communication, Control, and Computing, Monticello, Illinois, pp. 935–944, 1989.

  30. Sernik, E. L., andMarcus, S. I.,On the Computation of the Optimal Cost Function for Discrete Time Markov Models with Partial Observations, Annals of Operations Research, Vol. 29, pp. 471–512, 1991.

    Google Scholar 

  31. Federgruen, A., andSchweitzer, P. J.,Discounted and Undiscounted Value Iteration in Markov Decision Problems: A Survey, Dynamic Programming and Its Applications, Edited by M. Puterman, Academic Press, New York, New York, pp. 23–52, 1979.

    Google Scholar 

  32. Fernandez-Gaucherand, E., Arapostathis, A., andMarcus, S. I.,On Partially Observable Markov Decision Processes with an Average Cost Criterion, Proceedings of the 28th Conference on Decision and Control, Tampa, Florida, pp. 1267–1272, 1989.

  33. Andriyanov, V. A., Kogan, I. A., andUmnov, G. A.,Optimal Control of a Partially Observable Discrete Markov Process, Automation and Remote Control, Vol. 4, pp. 555–561, 1980.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Texas, Austin, Texas

    E. L. Sernik (Graduate Student) & S. I. Marcus (Professor)

Authors
  1. E. L. Sernik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. I. Marcus
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Communicated by Y. C. Ho

This research was supported by the Air Force Office of Scientific Research Grant AFOSR-86-0029, by the National Science Foundation Grant ECS-86-17860, by the Advanced Technology Program of the State of Texas, and by the Air Force Office of Scientific Research (AFSC) Contract F49620-89-C-0044.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sernik, E.L., Marcus, S.I. Optimal cost and policy for a Markovian replacement problem. J Optim Theory Appl 71, 105–126 (1991). https://doi.org/10.1007/BF00940042

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00940042

Key Words

Search

Navigation