Qualitative reachability for open interval Markov chains

Interval Markov chains extend classical Markov chains with the possibility to describe transition probabilities using intervals, rather than exact values. While the standard formulation of interval Markov chains features closed intervals, previous work has considered open interval Markov chains, in which the intervals can also be open or half-open. In this article we focus on qualitative reachability problems for open interval Markov chains, which consider whether the optimal (maximum or minimum) probability with which a certain set of states can be reached is equal to 0 or 1. We present polynomial-time algorithms for these problems for both of the standard semantics of interval Markov chains. Our methods do not rely on the closure of open intervals, in contrast to previous approaches for open interval Markov chains, and can address situations in which probability 0 or 1 can be attained not exactly but arbitrarily closely.


Introduction
The development of modern computer systems can benefit substantially from a verification phase, in which a formal model of the system is exhaustively verified in order to identify undesirable errors or inefficiencies. In this paper we consider the verification of probabilistic systems, in which state-to-state transitions are accompanied by probabilities that specify the relative likelihood with which the transitions occur, using model-checking techniques; see [2,11,1] for general overviews of this field. One drawback of classical formalisms for probabilistic systems is that they typically require the specification of exact probability values for transitions: in practice, it is likely that such precise information concerning the probability of system behaviour is not available. A solution to this problem is to associate intervals of probabilities with transitions, rather than exact probability values, leading to interval Markov chains (IMCs) or interval Markov decision processes. IMCs have been studied in [14,15], and considered in the qualitative and quantitative model-checking context in [18,5,6]. Qualitative model checking concerns whether a property is satisfied by the system model with probability (equal to or strictly greater than) 0 or (equal to or strictly less than) 1, whereas quantitative model checking considers whether a property is satisfied with probability above or below some threshold in the interval [0, 1], and generally involves the computation of the probability of property satisfaction, which is then compared to the aforementioned threshold.  In [18,5,6], the intervals associated with transitions are closed. This limitation was adressed in [4], which considered the possibility of utilising open (and half-open) intervals, in addition to closed intervals. Example of such open IMCs are shown in Figure 1 and Figure 2. In [4], it was shown that the probability of the satisfaction of a property in an open IMC can be approximated arbitrarily closely by a standard, closed IMC obtained by changing all (half-)open intervals featured in the model to closed intervals with the same endpoints. However, although the issue of the determining the existence of exact solutions is mentioned in [4], closing the intervals can involve the loss of information concerning exact solutions. Take, for example, the open IMC in Figure 1: changing the intervals from (0, 1) to [0, 1] on both of the transitions means that the minimum probability of reaching the state s 1 after starting in state s 0 becomes 0, whereas the probability of reaching s 1 from s 0 is strictly greater than 0 for all ways of assigning probabilities to the transitions in the original IMC.
In this paper we propose verification methods for qualitative reachability properties of open IMCs. We consider both of the standard semantics for IMCs. The uncertain Markov chain (UMC) semantics associated with an IMC comprises an infinite number of standard Markov chains, each corresponding to a certain choice of probability for each transition. In contrast, the interval Markov decision process (IMDP) semantics associates a single Markov decision process (MDP) with the IMC, where from each state there is available an uncountable number of distributions, each corresponding to one assignment of probabilities belonging to the intervals of the transitions leaving that state. The key difference between the two semantics can be summarised by considering the behaviour from a particular state of the IMC: in the UMC semantics, the same probability distribution over outgoing transitions must always be used from the state, whereas in the IMDP semantics the outgoing probability distribution may change for each visit to the state. We show that we can obtain exact (not approximate) solutions for both semantics in polynomial time in the size of the open IMC.
For the UMC semantics, and for three of the four classes of qualitative reachability problem in the IMDP semantics, the algorithms presented are inspired by methods for finite MDPs. In the case of the IMDP semantics, these algorithms rely on the fact that retaining the memory of previous choices along the behaviour of an IMC is not necessary. A direct method for the construction of a finite MDP that represents an IMC and which can be used for the verification of qualitative properties is the following: the set of states of the finite MDP equals that of the IMC and, for each state s and each set X of states, there exists a single distribution from s in the finite MDP that assigns positive probability to each state in X if and only if there exists at least one probability assignment for transitions in the IMC that assigns positive probability to each transition from s with target state in X. Intuitively, a distribution associated with s and X in the finite MDP can be regarded as the representative distribution of all probability assignments of the IMC that assign positive probability to the transitions from s to states in X. However, such a finite MDP construction does not yield polynomial-time algorithms in the size of the open IMC, because the presence of transitions having zero as their left endpoint can result in an exponential number of distributions in the number of IMC transitions. In our methods, apart from considering issues concerning the difference between closed and open intervals and the subsequent implications for qualitative reachability problems, we avoid such an exponential blow up. In particular, we show how the predecessor operations used by some qualitative reachability algorithms for MDPs can be applied directly on the open IMC.
The fourth class of reachability problem in the IMDP semantics concerns determining whether the probability of reaching a certain set of states from the current state is equal to 1 for all schedulers, where a scheduler chooses an outgoing probability distribution from a state on the basis of the choices made so far. For this class of problem, retaining memory of previous choices can be important for showing that the problem is not satisfied, i.e., that there exists a scheduler such that the reachability probability is strictly less than 1.
As an example, we can take the open IMC in Figure 1. Consider the memoryful scheduler that assigns probability 1 2 i to the i-th attempt to take a transition from s 0 to s 1 , meaning that the overall probability of reaching s 1 when starting in s 0 under this scheduler is 1 2 + 1 2 ( 1 4 + 3 4 ( 1 8 +· · · )) < 1. Instead a memoryless scheduler will reach s 1 with probability 1: for any λ ∈ (0, 1) representing the (constant) probability of taking the transition from s 0 to s 1 , the overall probability of reaching s 1 is lim k→∞ 1 − (1 − λ) k = 1. Hence our results for this class of reachability problem take the inadequacy of memoryless schedulers into account; indeed, while the algorithms presented for all other classes of problems (and all problems for the UMC semantics) proceed in a manner similar to that introduced in the literature for finite MDPs, for this class we present an ad hoc algorithm, based on an adaptation of the classical notion of end components [9].
After introducing open IMCs in Section 2, the algorithms for the UMC semantics and the IMDP semantics are presented in Section 3 and Section 4, respectively. The proofs of the results can be found in the appendix.
Related work. Model checking of qualitative properties of Markov chains (see, for example, [20,7]) relies on the fact that transition probability values are fixed throughout the behaviour of the system, and does not require that exact probability values are taken into account during analysis. The majority of work on model checking for IMCs considers the more general quantitative problems: [18,5] present algorithms utilising a finite MDP construction based on encoding within distributions available from a state the extremal probabilities allowed from that state (known as the state's basic feasible solutions). Such a construction results in an exponential blow up, which is also not avoided in [5] for qualitative proper-ties (when transitions can have 0 as their left endpoint). [6,17] improve on these results to present polynomial-time algorithms for reachability problems based on linear or convex programming. The paper [12] includes polynomial-time methods for computing (maximal) end components, and for computing a single step of value iteration, for interval MDPs. We note that IMCs are a special case of constraint Markov chains [3], and that the UMC semantics of IMCs corresponds to a special case of parametric Markov chains [8,16]. As far as we are aware, only [4] considers open IMCs.

Open Interval Markov Chains
Preliminaries. A (probability) distribution over a finite set Q is a function µ : Q → [0, 1] such that q∈Q µ(q) = 1. Let Dist(Q) be the set of distributions over Q. We use support(µ) = {q ∈ Q | µ(q) > 0} to denote the support set of µ, i.e., the set of elements assigned positive probability by µ, and use {q → 1} to denote the distribution that assigns probability 1 to the single element q. Given a binary function f : Q × Q → [0, 1] and element q ∈ Q, we denote by f (q, ·) : Q → [0, 1] the unary function such that f (q, ·)(q ) = f (q, q ) for each q ∈ Q. We let I denote the set of (open, half-open or closed) intervals that are subsets of [0, 1] and that have rational-numbered endpoints. Given an interval I ∈ I, we let left(I) (respectively, right(I)) be the left (respectively, right) endpoint of I. I [·,·) ; I (·,·) ). Hence we have I = I [·,·] ∪ I (·,·] ∪ I [·,·) ∪ I (·,·) . Furthermore, we let I +,· (respectively, I [0,· ; I (0,· ) be the set of intervals in I such that the left endpoint is positive (respectively, left-closed intervals with the left endpoint equal to 0; left-open intervals with the left endpoint equal to 0). Finally, let I 0,· = I [0,· ∪ I (0,· be the set of intervals in I with left endpoint equal to zero. A discrete-time Markov chain (DTMC) D is a pair (S, P) where S is a set of states, and P : S × S → [0, 1] is a transition probability matrix, such that, for each state s ∈ S, we have s ∈S P(s, s ) = 1. Note that P(s, ·) is a distribution, for each state s ∈ S. A path of DTMC D is a sequence s 0 s 1 · · · such that P(s i , s i+1 ) > 0 for all i ≥ 0. Given a path ρ = s 0 s 1 · · · and i ≥ 0, we let ρ(i) = s i be the (i + 1)-th state along ρ. The set of paths of D starting in state s ∈ S is denoted by Paths D (s). In the standard manner (see, for example, [2,11]), given a state s ∈ S, we can define a probability measure Pr D s over Paths D (s). A Markov decision process (MDP) M is a pair (S, ∆) where S is a finite set of states and ∆ : S → 2 Dist(S) is a transition function such that ∆(s) = ∅ for all s ∈ S. We say that an MDP is finite if ∆(s) is finite for all s ∈ S.
A(n infinite) path of an MDP M is a sequence s 0 µ 0 s 1 µ 1 · · · such that µ i ∈ ∆(s i ) and µ i (s i+1 ) > 0 for all i ≥ 0. Given a path ρ = s 0 µ 0 s 1 µ 1 · · · and i ≥ 0, we let ρ(i) = s i be the (i + 1)-th state along ρ. A finite path is a sequence r = s 0 µ 0 s 1 µ 1 · · · µ µn−1 s n such that µ i ∈ ∆(s i ) and µ i (s i+1 ) > 0 for each 0 ≤ i < n. Let last(r) = s n denote the final state of r. Let Paths M * be the set of finite paths of the MDP M. Let Paths M (s) and Paths M * (s) be the sets of infinite paths and finite paths, respectively, of M starting in state s ∈ S.
A scheduler is a mapping σ : Paths M * → Dist( s∈S ∆(s)) such that σ(r) ∈ Dist(∆(last(r))) for each r ∈ Paths M * . Let Σ M be the set of schedulers of the MDP M. Given a state s ∈ S and a scheduler σ, we can define a countably infinite-state DTMC D σ s that corresponds to the behaviour of the scheduler σ from state s, which in turn can be used to define a probability measure Pr σ s over Paths M (s) in the standard manner (see [2,11]). A scheduler σ ∈ Σ M is memoryless if, for finite paths r, r ∈ Paths M * such that last(r) = last(r ), we have σ(r) = σ(r ). Let Σ M m be the set of memoryless schedulers of M. Note that, for a memoryless scheduler σ ∈ Σ M m , we can construct a finite DTMC D σ = (S,P) withP(s, s ) = µ∈∆(s) σ(s)(µ) · µ(s ): we call this DTMC the folded DTMC of σ. It can be shown that Pr σ s and PrD σ s assign the same probabilities to measurable sets of paths, because the state s of the DTMC D σ s is probabilistic bisimulation equivalent to the state s of the folded DTMCD σ (for a definition of probabilistic bisimulation and more information on this point, see [2,Section 10.4.2]).

Interval Markov Chains: syntax. An (open) interval Markov chain (IMC)
O is a pair (S, δ), where S is a finite set of states, and δ : S × S → I is a interval-based transition function.
In the following, we refer to edges as those state pairs for which the transition function does not assign the probability 0 point interval [0, 0]. Formally, let the set of edges E of O be defined as {(s, s ) ∈ S × S | δ(s, s ) = [0, 0]}. We use edges to define the notion of path for IMCs: a path of an IMC O is a sequence s 0 s 1 · · · such that (s i , s i+1 ) ∈ E for all i ≥ 0. Given a path ρ = s 0 s 1 · · · and i ≥ 0, we let ρ(i) = s i be the (i + 1)-th state along ρ. We use Paths O to denote the set of paths of O, Paths O * to denote the set of finite paths of O, and Paths O (s) and Paths O * (s) to denote the sets of paths and finite paths starting in state s ∈ S. Given a state s ∈ S, we say that a distribution a ∈ Dist(S) is an assignment for s if a(s ) ∈ δ(s, s ) for each state s ∈ S. We say that the IMC O is well formed if there exists at least one assignment for each state. Note that an assignment for state s ∈ S exists if and only if the following condi- We henceforth consider IMCs that are well formed. We define the size of an IMC O = (S, δ) as the size of the representation of δ, which is the sum over all states s, s ∈ S of the binary representation of the endpoints of δ(s, s ), where rational numbers are encoded as the quotient of integers written in binary. Reachability. Let O = (S, δ) be an IMC and let T ⊆ S be a set of states. We define Reach(T ) ⊆ Paths O to be the set of paths of O that reach at least one state in T . Formally, In the following we assume without loss of generality that states in T are absorbing in all the IMCs that we consider, i.e., δ(s, s) = [1, 1] for all states s ∈ T .
Given X ⊆ S, and given s and as defined above, Valid edge sets. We are interested in identifying the sets of edges from state s ∈ S that result from assignments. Such a set is characterised by two syntactic conditions: the first condition requires that the sum of the upper bounds of the set's edges' intervals is at least 1, whereas the second condition specifies the edges from state s that are not included in the set can be assigned probability 0. Formally, we say that a non-empty subset B ⊆ E(s) of edges from s is large if either (a) e∈B right(δ(e)) > 1 or (b) e∈B right(δ(e)) = 1 and Then we say that B ⊆ E(s) is valid if it is large and realisable. The following lemma specifies that a valid edge set for state s characterises exactly the support sets of some assignments for s.

Qualitative Reachability: UMC semantics
Qualitative reachability problems can be classified into four categories, depending on whether the probability of reaching the target set T is 0 or 1 for some or for all ways of assigning probabilities to intervals. For the UMC semantics, we consider the computation of the following sets:  Hence the set S 0,U ∀ is equal to the complement of the set of states from which there exists a path reaching T in the graph of the IMC (that is, the graph (S, E)). Given that the latter set of states can be computed in polynomial time, we conclude that S 0,U ∀ can be computed in polynomial time.
Computation of S 0,U ∃ . We show that S 0,U ∃ can be obtained by computing the set of states from which there exists a scheduler for which T is reached with probability 0 in the qualitative MDP abstraction [O] w = (S, ∆ w ) of O with respect to some (arbitrary) witness asignment function w.
First we establish that the set of states of [O] w for which there exists a scheduler such that T is reached with probability 0 (respectively, probability 1) is equal to the set of states of O for which there exists a DTMC in [O] U such that T is reached with probability 0 (respectively, probability 1). In particular, Lemma 3 allows us to reduce the problem of computing S 0,U ∃ to that of computing the set {s ∈ S | ∃σ ∈ Σ [O]w . Pr σ s (Reach(T )) = 0} on [O] w . As in the case of standard finite MDP techniques (see [11]), we proceed by computing the complement of this set, i.e., we compute the set {s ∈ S | ∀σ ∈ Σ [O]w . Pr σ s (Reach(T )) > 0}. For a set X ⊆ S, let CPre(X) = {s ∈ S | ∃µ ∈ ∆ w (s) . support(µ) ⊆ X} be the set of states for which there exists a distribution such that all states assigned positive probability by the distribution are in X. Furthermore, we let CPre(X) = {s ∈ S | ∀µ ∈ ∆ w (s) . support(µ) ∩ X = ∅} be the dual of the CPre operator (i.e., CPre(X) = S \ CPre(S \ X)), that is the set of states from which it is inevitable to make a transition to X with positive probability. The standard algorithm for computing the set of states of a finite MDP for which all schedulers are such that a set T of target states is reached with probability strictly greater than 0 operates in the following way: starting from X 0 = T , we let X i+1 = X i ∪ CPre(X i ) for progressively larger values of i ≥ 0, until we reach a fixpoint (that is, until we obtain X i * +1 = X i * for some i * ). However, a direct application of this algorithm to [O]   The intuition underlying Lemma 4 is that conditions (1) and (2) encode realisibility and largeness, i.e., validity, of edge set E(s, X). From Lemma 1, their satisfaction means that there exists a distribution in ∆ w (s) with support set equal to the set of target states of edges in E(s, X). We consider the largest edge set with target states in X, i.e., E(s, X), because taking smaller edge sets with targets in X would make the conditions (1) and (2)  Computation of S 1,U ∃ . We proceed in a manner analogous to that for the case of S 0,U ∃ . First we note that, by Lemma 3, we have that S 1,U ∃ is equal to the set of states of [O] w such that there exists a scheduler for which T is reached with probability 1. Hence, our aim is to compute the set {s ∈ S | ∃σ ∈ Σ [O]w . Pr σ s (Reach(T )) = 1} on [O] w . We recall the standard algorithm for the computation of this set on finite MDPs [9,10]. Given state sets X, Y ⊆ S, we let be the set of states for which there exists a distribution such that all states assigned positive probability by the distribution are in Y and there exists a state assigned positive probability by the distribution that is in X. The standard algorithm proceeds by setting Y 0 = S and X 0 0 = T . Then the sequence X 0 0 , X 0 1 , · · · is computed by letting X 0 i0+1 = X 0 i0 ∪ APre(Y 0 , X 0 i0 ) for progressively larger indices i 0 ≥ 0 until a fixpoint is obtained, that is, until we obtain X 0 and X 2 0 = T , and repeat the process. We terminate the algorithm when a fixpoint is reached in the sequence Y 0 , Y 1 , · · · . 1 The algorithm requires at most |S| 2 calls to APre. In an analogous manner to CPre in the case of S 0,U ∃ , we show that APre can characterised by efficiently checkable conditions on O. The intution underlying Lemma 5 is similar to that of Lemma 4.
S 1,U ∀ . We recall the standard algorithm for determining the set of states for which all schedulers reach a target set with probability 1 on a finite MDP (see [11]): from the set of states of the MDP, we first remove states from which the target state can be reached with probability 0 (for some scheduler), then successively remove states for which it is possible to reach a previously removed state with positive probability. For each of the remaining states, there exists a scheduler that can reach the target set with probability 1.
We propose an algorithm for IMCs that is inspired by this standard algorithm for finite MDPs. Our aim is to compute the complement of S 1,U ∀ , i.e., the state set S \ S 1,U

Qualitative Reachability: IMDP semantics
We now focus on the IMDP semantics, and consider the computation of the following sets: This section will be dedicated to showing the following result. We note that the cases for S 0,I ∀ , S 0,I ∃ and S 1,I ∃ proceed in a manner similar to the UMC case (using either graph reachability or reasoning based on the qualitative MDP abstraction); instead the case for S 1,I ∀ requires substantially different techniques. Therefore, to obtain S 0,I ∀ , we proceed by computing the state set S \ S 0,I ∀ = {s ∈ S | ∃σ ∈ Σ [O]I . Pr σ s (Reach(T )) > 0}, which reduces to reachability on the graph of the IMC according to Lemma 7, and then taking the complement. Given that we have shown in Section 3 that the set of states of the qualitative MDP abstraction [O] w for which there exists some scheduler such that T is reached with probability 0 (respectively, probability 1) can be computed in polynomial time in the size of O, we obtain polynomial-time algorithms for computing S 0,I ∃ (respectively, S 1,I ∃ ).
Computation of S 1,I ∀ . This case is notably different from the other three cases for the IMDP semantics, because schedulers that are not memoryless may influence whether a state is included in S 1,I ∀ . In particular, we recall the example of the IMC of Figure 1: as explained in Section 1, we have s 0 ∈ S 1,I ∀ . In contrast, we have s 0 ∈ S 1,U ∀ , and s 0 would be in S 1,I ∀ if we restricted the IMDP semantics to memoryless (actually finite-memory, in this case) schedulers. For this reason, a qualitative MDP abstraction is not useful for computing S 1,I ∀ , because it is based on the use of witness assignment functions that assign constant probabilities to sets of edges available from states: on repeated visits to a state, the (finite) set of available distributions remains the same in a qualitative MDP abstraction. Therefore we require alternative analysis methods that are not based on the qualitative MDP abstraction. Our approach is based on the notion of end components, which is a standard concept in the field of MDP verification [9]. In this section we introduce an alternative notion of end components, defined solely in terms of states of the IMC, which characterises situations in which the IMC can confine its behaviour to certain state sets with positive probability in the IMDP semantics (for example, the IMC of Figure 1 can confine itself to state s 0 with positive probability in the IMDP semantics).
An IMC-level end component (ILEC) is a set C ⊆ S of states that is strongly connected and such that the total probability assigned to edges that have a source state in C but a target state outside of C can be made to be arbitrarily small (note that such edges must have an interval with a left endpoint of 0). Formally, C ⊆ S is an ILEC if, for each state s ∈ C, we have (1) E +,· (s, S \ C) = ∅, (2) e∈E(s,C) right(δ(e)) ≥ 1, and (3) the graph (C, E ∩ (C × C)) is strongly connected.
Let I be the set of ILECs of O. We say that an ILEC C ∈ I is maximal if there does not exist any C ∈ I such that C ⊂ C . For a path ρ ∈ Paths [O]I (s), let infst(ρ) ⊆ S be the states that appear infinitely often along ρ, i.e., for ρ = s 0 µ 0 s 1 µ 1 · · · , we have infst(ρ) = {s ∈ S | ∀i ∈ N . ∃j > i . s j = s}. We present a result for ILECs that is analogous to the fundamental theorem of end components of [9]: the result specifies that, with probability 1, a scheduler of the IMDP semantics of O must confine itself to an ILEC. We now show that there exists a scheduler that, from a state within an ILEC, can confine the IMC to the ILEC with positive probability. This result is the ILEC analogue of a standard result for end components of finite MDPs that specifies that there exists a scheduler that, from a state of an end component, can confine the MDP to the end component with probability 1 (see [9,2]). In the case of IMCs and ILECs, it is not possible to obtain an analogous result for probability 1; in the example of Figure 1, the singleton set {s 0 } is an ILEC, but it is not possible to find a scheduler that remains in s 0 with probability 1, because with each transition the IMC goes to s 1 with positive probability. For our purposes, it is sufficient to have a result stating that, from an ILEC, the IMC can be confined to the ILEC with positive probability. The key point of the proof of Lemma 10 is the definition of a scheduler that assigns progressively decreasing probability to all edges in E 0,· that leave ILEC C, in such a way as to guarantee that the IMC is confined in C with positive probability. This is possible because condition (2) of the definition of ILECs specifies that there is no lower bound on the probability that must be assigned to edges that leave C. Furthermore, the scheduler is defined so that the remaining probability at each step that is assigned between all edges that stay in C is always no lower than some fixed lower bound; this characteristic of the scheduler, combined with the fact that we remain in C with positive probability and the fact that C is strongly connected, means that we visit all states of C with positive probability under the defined scheduler.
Let U ¬T = {C ∈ I | C ∩ T = ∅} be the union of states of ILECs that do not contain states in T . Using Lemma 9 and Lemma 10 in a standard way, we can show that the existence of a scheduler of [O] I that reaches T with probability strictly less than 1 is equivalent to the existence of a path in the graph of O that reaches U ¬T .  [9,2]). First we compute all strongly connected components (C 1 , E ∩ (C 1 × C 1 )), · · · , (C m , E ∩ (C m × C m )) of the graph (S \ T, E ∩ ((S \ T ) × (S \ T ))) of O. Then, for each 1 ≤ i ≤ m, we remove from C i all states for which conditions (1) or (2) in the definition of ILECs do not hold with respect to C i (these conditions can be checked in polynomial time for each state), to obtain the state set C i . Next, we compute the strongly connected components of the graph (C i , E ∩ (C i × C i )), and for each of these, repeat the procedure described above. We terminate the algorithm when it is not possible to remove a state (via a faliure to satisfy a least one of the conditions (1) and (2) in the definition of ILECs) from any generated strongly connected component. The generated state sets of the strongly connected components obtained will be be the maximal ILECs that do not contain any state in T , and their union is U ¬T . Hence the overall algorithm for computing S 1,I ∀ is in polynomial time in the size of O.

Conclusion
We have presented algorithms for qualitative reachability properties for open IMCs. In the context of qualitative properties of system models with fixed probabilities on their transitions, probability can be regarded as imposing a fairness constraint, i.e., paths for which a state is visited infinitely often and one of its successors is visited only finitely often have probability 0. In open IMCs, the possibility to make the probability of a transition converge to 0 in the IMDP semantics captures a different phenomenon, which is key for problems concerning the minimum reachability probability being compared to 1. We conjecture that finite-memory strategies are no more powerful than memoryless strategies for this class of problem. For the three other classes of qualitative reachability problems, we have shown that the UMC and IMDP semantics coincide. We note that the algorithms presented in this paper require some numerical computation (a sum and a comparison of the result with 1 in the CPre, APre and ILEC computations), but these operations are simpler than the polynomial-time solutions for quantitative properties of (closed) IMCs in [6,17]. Similarly, the CPre and APre operators are simpler than the polynomial-time step of value iteration used in the context of quantitative verification of [12]. For the IMDP semantics, our methods give directly a P-complete algorithm for the qualitative fragment of the temporal logic Pctl [13]. Future work could consider quantitative properties and ω-regular properties, and applying the results to develop qualitative reachability methods for interval Markov decision processes or for higher-level formalisms such as clock-dependent probabilistic timed automata [19]. (⇐) Let a be an assignment for s such that {(s, s ) | s ∈ support(a)} = B. We first show that B is large. By the definition of assignments, we have a ∈ δ(s, s ), and hence a(s ) ≤ right(δ(s, s )) for each state s ∈ S. From this fact, and given that a is a distribution (i.e., sums to 1), we have: In the case of (s,s )∈B right(δ(s, s )) > 1, we have shown that B is large. In the case of (s,s )∈B right(δ(s, s )) = 1, then a(s ) = right(δ(s, s )) (i.e., a(s ) must be equal to the right endpoint of δ(s, s )) for each (s, s ) ∈ B. Hence B ⊆ E ·,·] , and as a consequence B is large. The fact that {(s, s ) | s ∈ support(a)} = B implies that a(s ) = 0 for all (s, s ) ∈ E(s) \ B. Hence E(s) \ B ⊆ E [0,· , and therefore B is realisable. Because B is large and realisable, B is valid.

Proofs from Section 3
Proof of Lemma 2. (⇒) Let D = (S, P) ∈ [O] U be a DTMC such that Pr D s (Reach(T )) > 0. Because Pr D s (Reach(T )) > 0, there exists at least one finite path s 0 s 1 · · · s n ∈ Paths D * (s) such that s n ∈ T . For each i < n, we have that P(s i , s i+1 ) > 0. From this fact, and by the definition of assignment, we must have that δ(s i , s i+1 ) = [0, 0]. This means that (s i , s i+1 ) ∈ E. Repeating this reasoning for each i < n, we have that s 0 s 1 · · · s n ∈ Paths O * (s). Recalling that s n ∈ T , this direction of the proof is completed.
(⇐) Let s 0 s 1 · · · s n ∈ Paths O * (s) be a finite path of O such that s n ∈ T . By the definition of finite paths of O, we have δ(s i , s i+1 ) = [0, 0] for each i < n. This implies that, for each i < n, there exists an assignment to state s i that sets positive probability to s i+1 . In turn, this means that P(s i , s i+1 ) > 0 for each i < n, for some DTMC D = (S, P) ∈ [O] U . Given that s n ∈ T , hence we have Pr D s (Reach(T )) > 0.
Proof of Lemma 3. Before we present the proof of Lemma 3, we recall (and specialise to reachability) a technical result from [5] that specifies that the same qualitative properties are satisfied on (finite) DTMCs that have the same graph. Two finite DTMCs D 1 = (S 1 , P 1 ) and D 2 = (S 1 , P 2 ) are graph equivalent if (1)  [9], we can assume that σ is (a) pure (that is, for all finite paths r ∈ Paths [O]w * , we have that σ(r) = {µ → 1} for some µ ∈ ∆ w (last(r))) and (b) memoryless. Then we consider the DTMC D = (S, P), where, for all states s ∈ S, we define P(s , ·) = σ(s ). Given that σ(s ) is an assignment (because σ is pure, i.e., assigns probability 1 to a distribution w(B), for some B ∈ Valid (s ), and because w is a witness assignment funcion, i.e., w(B) is an assignment), we have that D ∈ [O] U . The folded DTMCD σ of σ and D are identical. Hence, given that Pr σ s (Reach(T )) λ, we have Pr D s (Reach(T )) λ.