The Cover Time of a (Multiple) Markov Chain with Rational Transition Probabilities is Rational

The cover time of a Markov chain on a finite state space is the expected time until all states are visited. We show that if the cover time of a discrete-time Markov chain with rational transitions probabilities is bounded, then it is a rational number. The result is proved by relating the cover time of the original chain to the hitting time of a set in another higher dimensional chain. We also extend this result to the setting where $k\geq 1 $ independent copies of a Markov chain are run simultaneously on the same state space and the cover time is the expected time until each state has been visited by at least one copy of the chain.


Introduction and Results
Let (X t ) t≥0 be a discrete-time Markov chain with transition matrix P on a state space Ω, see [1,6] for background.We say a chain is rational if all its transition probabilities are rational numbers, i.e.P(x, y) ∈ Q for all x, y ∈ Ω.The stopping time τ cov is the first time all states are visited, that is be the cover time from x, that is, the expected time for the chain to visit all states when started from x ∈ Ω.
Along with mixing and hitting times, the cover time is one of the most natural and well studied stopping times for a Markov chain and has found applications in the analysis of algorithms, see for example [2], [1,Ch. 6.8] and [6,Ch. 11].It is clear that the stopping time τ cov is a natural number, however it is not so clear whether the cover time E x [ τ cov ] is rational, even if the transition probabilities are rational.Our main result shows that, under some natural assumptions, the cover time of a rational Markov chain is rational.Theorem 1.Let (X t ) t≥0 be a discrete-time rational Markov chain on a finite state space Ω.Then, for any The assumption that Ω is finite is necessary to ensure the cover time is bounded.Recall that a Markov chain is irreducible if for every x, y ∈ Ω there exists some t ≥ 0 such that Figure 1: Example of a non-irreducible Markov chain on seven states where the cover time from x is finite and from any other vertex the cover time is unbounded/undefined.
P t (x, y) > 0, where P t (x, y) denotes the probability a chain started at x is at state y after t ≥ 1 steps.Theorem 1 does not require irreducibility, just that the cover time from the given start vertex is bounded.An example of a non-irreducible Markov chain to which we can apply Theorem 1 is given in Figure 1.In this example the cover time from x is bounded however, the cover time from any other vertex is unbounded/undefined, as if a walk starts from any other vertex, then x (and possibly also the vertex immediately right of x) cannot be reached.For a concrete example of why rational transition probabilities are necessary in Theorem 1, if one fixes any real number r ≥ 1 then the two state chain with transition matrix given by has cover time r.It is well known, see for example [6, Lemma 1.13], that the cover time of finite irreducible Markov chain from any start vertex is bounded.This fact, and restricting the example given by (1) to r ∈ Q, implies the following corollary to Theorem 1.
Corollary 2. The set of cover times attainable by finite discrete-time irreducible rational Markov chains is We now introduce multiple Markov chains, which have been studied for their applications to parallelising algorithms driven by random walks, see [2] and subsequent papers citing it.

For any
is an independent copy of the chain P run simultaneously on the same state space Ω.
The k-multiple of P is itself a Markov chain (with transition matrix K) on Ω k with transition probabilities As before, we denote the conditional expectation where ∈ Ω is the start state of the i th walk for each 1 ≤ i ≤ k.We let the stopping time τ i } = Ω} be the first time every state in Ω (not Ω k ) has been visited by some walk X (i) t .We then let E x τ (k) cov denote the k-walk stopping time from x ∈ Ω k .Note that this is not simply the cover time of the chain K.The multiple walk cover time can have subtle dependences on k and the host underlying Markov chain, see [2].
We show that Theorem 1 also holds in the more general setting of k-multiple Markov chains.Theorem 3. Let k ≥ 1 and (X t ) t≥0 be the k-multiple of a discrete-time rational Markov chain on a finite state space Ω.Then for any Theorem 1 is the special case k = 1 of Theorem 3, thus it suffices to prove Theorem 3.

Proofs
In this section we shall prove Theorem 3. The first part of the proof (covered in Section 2.1) is to show the expected time to first visit any set of states (hitting time) in a rational Markov chain is rational.Then, in Section 2.2, we show for any k ≥ 1 and P, the multiple walk with transition matrix P can be coupled with a higher dimensional Markov chain Q on a state space V where |V | ≤ |Ω| k • 2 |Ω| .The coupling shows that the first time all states in Ω have been visited by at least one of the k walks has the same distribution as the first visit time a specific set C ⊂ V is visited in Q.

Rationality of Hitting Times
For S ⊆ Ω, a subset of the state space of a Markov chain P, let the stopping time Observe that if P is irreducible then B(S) = Ω for any S ⊆ by [6, Lemma 1.13].Before proving Proposition 4 we give some definitions and prove an elementary lemma.
For a field F and integers n, m ≥ 1 let F n and F m×n denote the set of n-dimensional vectors and m×n-dimension matrices respectively.Let I n denote the n×n identity matrix.Lemma 5. Let A ∈ Q n×n be non-singular and b ∈ Q n .Then there exists a unique vector Proof.Since A is non-singular there exists a unique solution x ∈ R n to the linear system given by Ax = b.Also, again since A is non-singular, we can compute A −1 by Gaussian elimination.Since all entries of A are rational, all multiplications preformed during the Gaussian elimination will be rational.Thus, as there are only finitely many row additions and multiplications, We now use this lemma to prove Proposition 4.
This can be expressed as Ah = b where b ∈ {0, 1} b and given by M(i, j) = P(i, j) if i, j / ∈ S and 0 otherwise.We shall show that (i) all rows i satisfy |A(i, ≥ j =i |A(i, j)|, and (ii) for each row r 0 , there exists a finite sequence of rows r 0 , r 1 . . ., r t such that A(r i−1 , r i ) = 0 for all 1 ≤ i ≤ t and |A(r t , r t )| > j =rt |A(r t , j)|.
Observe that Condition (i) holds since M a sub-matrix of P.
For Condition (ii), note that for every row s ∈ S we have j M(s, j) = 0. Thus |A(s, s)| > j =s |A(s, j)| for any row s ∈ S. The fact that each row r 0 corresponds to a state in B(S) implies that, for any row r 0 , there exists some r t ∈ S and a sequence of states/rows r 0 , r 1 . . ., r t such that A(r i−1 , r i ) = −P(r i−1 , r i ) = 0, thus Condition (ii) is satisfied.
Since A satisfies (i) and (ii) it is weakly chained diagonally dominant, thus by [3, Lemma
Figure 2 shows an example of the auxiliary chain Q of a single Markov chain P on three states, that is the case k = 1.Staying within the confines of k = 1 case for simplicity, one may think of Q as inducing a directed graph consisting of many 'layers', where each layer is a copy of P restricted to a subset of Ω.These layers are linked by directed edges which are crossed when a new state not in the current layer is first visited.Thus, since a sequence x 0 , x 1 , . . . in the first component of V evolves according to P by (2), each layer encodes which states of chain have been visited so far by a trajectory in P.
Similar constructions to Q(P, 1) were used by the author and co-authors in the study of the Choice and ε-TB random walks, which are walks where a controller can influence which vertices are visited.In particular they were used to show that there exist optimal strategies (1, {1}) for covering a graph by these walks which are time invariant in a certain sense [4] and to show the computational problem of finding optimal strategies to cover a graph by these walks is in PSPACE [5].
The next result equates the cover time by k ≥ 1 multiple Markov chain with transition matrix P to the hitting time of a specific set in the auxiliary chain Q(P, k).For clarity we use the notation E P • [ • ] to highlight the chain, in this case P, in which the expectation is taken.
Lemma 6.Let P be a Markov chain on Ω, and let k ≥ 1 be an integer.Let Q := Q(P, k) be the associated k-walk auxiliary chain with state space V := V (Ω, k), and set C = {(u, Ω) : u ∈ Ω k } ⊂ W.Then, for any x = (x (1) , . . ., x (k) ) ∈ Ω k and real number a, we have We must introduce some notation before proving Lemma 6.For real valued random variables X, Y we say that Y stochastically dominates X if P [ Y ≥ a ] ≥ P [ X ≥ a ] for all real a, and we denote this by X Y .Thus, if X Y and Y X, then X and Y are equidistributed.
Proof of Lemma 6.We first show how any trajectory (X t ) t≥0 of a k-multiple of the Markov chain P can be coupled with a trajectory (Y t ) t≥0 of the auxiliary Markov chain Q(P, k) given by (2).To begin, given any start vector X 0 = x 0 ∈ Ω k , where x 0 = (x be the first time S is visited.If S = {s} is a singleton set we abuse notation slightly by taking τ s to mean τ {s} .For x ∈ Ω, let E x [ τ S ] be the expected hitting time of S ⊆ Ω for a chain started from x.The next result is the hitting time analogue of Theorem 1. Proposition 4. Let P be a discrete-time rational Markov chain on a finite state space Ω.For a non-empty set S ⊆ Ω let B(S) = {x ∈ Ω : E x [ τ S ] < ∞}.Then for any S ⊆ Ω and x ∈ B(S) we have E x [τ S ] ∈ Q.

Proof of Proposition 4 .
Observe that B(S) = ∅ since S ⊆ B(S) and E s [ τ S ] = 0 for all s ∈ S. Let b := |B(S)|.Now, each entry of the vector h := (E x [τ S ]) x∈B(S) is bounded and h is a solution to the following set of linear equations

Figure 2 :
Figure 2: This figure shows an example of a Markov chain P on three states (bottom right) and its associated auxiliary chain Q(P, 1), where the set C from Lemma 6 is shown in the red shaded ellipse.