Continuous Time Markov Chains

Abstract

In Chap. 1 we considered Markov chains X _n with a discrete time index n = 0, 1, 2, … In this chapter we will extend the notion to a continuous time parameter t ≥ 0, a setting that is more convenient for some applications. In discrete time we formulated the Markov property as: for any possible values of j, i, i _n−1, … i ₀

$$\displaystyle{P(X_{n+1} = j\vert X_{n} = i,X_{n-1} = i_{n-1},\ldots,X_{0} = i_{0}) = P(X_{n+1} = j\vert X_{n} = i)}$$

In continuous time, it is technically difficult to define the conditional probability given all of the X _r for r ≤ s, so we instead say that X _t , t ≥ 0 is a Markov chain if for any 0 ≤ s ₀ < s ₁⋯ < s _n  < s and possible states i ₀, …, i _n , i, j we have

$$\displaystyle{P(X_{t+s} = j\vert X_{s} = i,X_{s_{n}} = i_{n},\ldots,X_{s_{0}} = i_{0}) = P(X_{t} = j\vert X_{0} = i)}$$

In words, given the present state, the rest of the past is irrelevant for predicting the future. Note that built into the definition is the fact that the probability of going from i at time s to j at time s + t only depends on t the difference in the times.