The Discounted Cost Model

Abstract

In this chapter, let $\alpha >0$ be a fixed discount factor. We shall consider the $\alpha$-discounted CTMDP problems (1.15) and (1.16). Without loss of generality we can restrict to (relaxed) $\pi$-strategies because (Markov) $\pi$-strategies are sufficient for $\alpha$-discounted problems; see Remark 1.3.1. (In fact, in this chapter, the class of natural Markov strategies is also sufficient, according to (2.92). However, we choose to stay with the more general class of $\pi$-strategies. The same is true for some of the subsequent chapters.) Although most $\pi$-strategies are not realizable, under appropriate conditions, the solution to the unconstrained problem is given by a realizable deterministic stationary strategy (see Theorem 3.1.2). Moreover, as was explained in Sect. 1.3.5, for every $\pi$-strategy there exists an equivalent standard $\xi$-strategy which is realizable.