Skolem and Positivity Completeness of Ergodic Markov Chains

We consider the following decision problems: given a ﬁnite, rational Markov Chain, source and target states, and a rational threshold, does there exist an n such that the probability of reaching the target from the source at the n th step is equal to the threshold (resp. crosses the threshold)? These problems are known to be equivalent to the Skolem (resp. Positivity) problems for Linear Recurrence Sequences (LRS). These are number-theoretic problems whose decidability has been open for decades. We present a short, self-contained, and elementary reduction from LRS to Markov Chains that improves the state of the art as follows: (a) We reduce to ergodic Markov Chains, a class widely used in Model Checking. (b) We reduce LRS to Markov Chains of signiﬁcantly lower order than before. We thus get sharper hardness results for a more ubiquitous class of Markov Chains. Immediate applications include problems in modeling biological systems, and regular automata-based counting problems.


Introduction
Markov Chains are a natural mathematical framework to describe probabilistic systems, such as those arising in computational biology.There is an extensive body of work on model checking Markov Chains: see [3] for a comprehensive set of references.Most of the focus has been on the verification of linear-and branching-time properties of Markov Chains through solving systems of linear equations, or linear programs.An alternative approach [1,4,7,8] is to consider specifications on the state distribution at each time step, e.g., whether the probability of being in a given state at the n th step is at least 1/4.Decidability in this setting is a lot more inaccessible: [1,4] only present incomplete or approximate verification procedures, while [7,8] owe their model-checking procedures to additional mathematical assumptions.The inherent difficulty of precisely solving decision problems in this fundamental setting is established in [2]: it is formally shown that verifying such specifications is tantamount to solving the Skolem/Positivity Problem for Linear Recurrence Sequences (LRS).The reduction therein is from LRS of order k to periodic Markov Chains of order 2k+4.However, it is the ergodic Markov Chains (irreducible and aperiodic) that are widely assumed in practice, and it may well be that the hardness is somehow mitigated by the additional spectral structure.On the automata-theoretic front, [6] reduces LRS to counting words of length n in a regular language.The nature of the reduction means that the resulting automaton is necessarily periodic.Aperiodicity in this context relates to LTL-definability, and the logical restriction could lead to a combinatorial breakthrough.
In this paper, we show that the such breakthroughs that circumvent the original reduction are unlikely without significant progress in the underlying number-theory itself.Our novelty is a reduction from order k LRS to ergodic Markov Chains of order k + 1.An interesting feature of our reduction is that it shows that hard instances exist for every stationary distribution.In doing so, the translation of number-theoretic hardness for LRS (cf.[11]) to Markov Chains also becomes much sharper.

Markov Chain Preliminaries
Notation: Distributions are assumed to be column vectors.We use 1 to denote the column vector whose entries are all 1, and I to denote the identity matrix.We use 0 to denote the zero column vector, and O to denote the zero matrix.Superscript T denotes transposition.We use e i to denote the elementary column vector, i.e. the vector whose i th entry is 1 and all other entries are 0, e.g. e 1 = 1 0 . . .0 T .We use m (n) ij as shorthand to denote the entry in the i th row and j th column of M n , i.e. m T Me j denotes the probability of moving from state j to state i.We have Definition 2 (Irreducible Markov Chain).A Markov Chain is called irreducible if every state has a path to every other state.
M is said to be aperiodic iff all its states are aperiodic.

Definition 4 (Stationary distribution).
A distribution s is said to be a stationary distribution of a Markov Chain M, if Ms = s.
Theorem 1 (Fundamental Theorem of (Ergodic) Markov Chains).A Markov chain M is called ergodic if it is irreducible and aperiodic.An ergodic Markov Chain has a unique stationary distribution.
The following technical lemma will help us construct an Ergodic Markov Chain in our reduction.

Overview of Problems
Problem 1 (Ergodic Markov Chain Reachability).Given an ergodic Markov Chain M ∈ Q k×k and r ∈ Q, the Ergodic Markov Chain Reachability problem asks whether there exists an n ∈ N such that the probability of returning to state 1 at the n th step is exactly r, i.e. e 1 T M n e 1 = r.
Problem 2 (Threshold Ergodic Markov Chain Reachability).Given an ergodic Markov Chain M ∈ Q k×k and r ∈ Q, the Threshold Ergodic Markov Chain Reachability problem asks whether for all n ∈ N, the probability of returning to state 1 at the n th step is at least r, i.e. e 1 T M n e 1 ≥ r.
We relate these problems to long-standing open problems on Linear Recurrence Sequences.The Skolem Problem is known to be decidable for LRS of order up to 4, see [9,12].Very recently, there have been conditional decidability results for LRS of order 5 [5].The Positivity Problem is decidable up to order 5, decidability at order 6 would entail significant number-theoretic breakthroughs [11].If we restrict ourselves to the class of simple LRS (no repeated characteristic roots), then Positivity is decidable up to order 9, see [10].
We are now ready to state our main reduction, which is agnostic to the spectral nature of the LRS, and minimal with respect to the order.It is well known that for any matrix M, m (n) ij ∞ n=1 , i.e. the entries in the i th row and j th column in the powers of M form an LRS.This is most easily seen through the Cayley-Hamilton Theorem: any matrix M satisfies its characteristic polynomial equation, i.e. if p(λ) = det(M − λI), then p(M) = O.
Thus, we trivially have that Problems 3 and 4 respectively reduce to Problems 1 and 2.
One can define the "off-diagonal" variants of Problems 1 and 2, i.e. queries on the probability of reaching state 1 from state 2. The above equivalences hold for the off-diagonal variants with an almost identical proof.We discuss the difference after presenting the reduction.

The Reduction
The key idea is to use Lemma 2 to construct an ergodic Markov Chain via the decomposition M = S + D. Given an LRS u n ∞ n=0 of order k over Q, we will choose S, D ∈ Q (k+1)×(k+1) , r = s 11 = s 1 , a rational η and a large rational ρ in such a way that for all n ≥ 1, d 11 ≥ r = s 1 ), which is precisely the reduction we want.We assume without loss of generality that none of the initial terms of the LRS are 0, and that u 0 > 0.
To begin with, we choose some arbitrary probability distribution such that all the entries of s are strictly positive.S denotes the square matrix, each of whose columns is s.
Let A ∈ Q k×k be the companion matrix of the given LRS, i.e.
Observe that the top left entry in both F and We note, by a simple induction, that for n ≥ 1 The choice of ρ is large enough to ensure that: • The entries of S + D are non-negative.
• The spectral radius of D is less than 1.
By Lemma 2, this makes the stochastic M = S + D an ergodic Markov Chain with stationary distribution s, since, indeed, DS = SD = O.
We now observe that for n ≥ 1, D n = 1 ρ n C n (I − S).We see this inductively: To complete the proof, we now compute the top-left entry of D n , n ≥ 1:

Lemma 2 .
Let s be a distribution with all entries strictly positive, and let S = s s . . .s .A stochastic matrix M is an ergodic Markov Chain with stationary distribution s if and only if Ms = s and there exists D such that• M = S + D • DS = SD = O • D has spectral radius less than 1, i.e. lim n→∞ D n = OIn particular, we observe that the first two properties of D imply thatM n = S + D n for n ≥ 1.Proof.Only If : Let M be an ergodic Markov Chain with stationary distribution s.Then, by definition, Ms = s, MS = S, and lim n→∞ M n = S.We note that both M and S are stochastic matrices, and thus 1 T (M − S) = 0 T .Denote M − S by D. From the previous observation, it is clear that SD = O, since all the rows of S are scaled multiples of 1 T .We also have that S = MS = S 2 + DS = S + DS, which means that DS = O.We use the fact that DS = SD = O and that S n = S to observeM n = (S + D) n = S + D nbecause all other terms in the binomial expansion are nullified.Now, since lim n→∞ M n = S, it forces lim n→∞ D n = O.If : The above argument is reversible.If instead we are given the three properties of D to begin with, we can conclude that MS = S and lim n→∞ M n = S, which is precisely the definition of M being an ergodic Markov Chain with stationary distribution s.

Definition 5 (
Linear Recurrence Sequence a.k.a.LRS).An LRS of order k over Q is an infinite sequence u n ∞ n=0 satisfying a recurrence relation u n+k = k−1 i=0 a i u n+i for all n ∈ N. The recurrence relation is given by the tuple (a 0 , . . ., a k−1 ) ∈ Q k with a 0 = 0.The sequence is uniquely determined by the starting values (u 0 , . . ., u k−1 ) ∈ Q k .Problem 3 (Skolem Problem for LRS).Given an LRS u n ∞ n=0 (via the recurrence relation and starting values), the Skolem problem asks whether there exists an n ∈ N such that u n = 0. Problem 4 (Positivity Problem for LRS).Given an LRS u n ∞ n=0 (via the recurrence relation and starting values), the Positivity problem asks whether for all n ∈ N, u n ≥ 0.

Theorem 3 (
Main Result).Problem 3 reduces to Problem 1, while Problem 4 reduces to Problem 2.Moreover, applying the reduction to an LRS of order k results in an ergodic Markov Chain of order k + 1.

e 1 T D n e 1 = 1 ρ n e 1 T 1 = 1 ρ n e 1 Tn e 1 TLet r = s 1 .
C n (I − S)e A n u = ηu n ρ n which is precisely a scaled version of our LRS.We have thus established that m The original LRS is a YES instance of the Skolem Problem (resp.the Positivity Problem) if and only if there exists an n such that m (n) 11 = r (resp.for all n, m (n) 11 ≥ r).These are precisely the YES instances of the reachability problems we defined.