On Occupation Time for On-Off Processes with Multiple Off-States

The need to model a Markov renewal on-off process with multiple off-states arise in many applications such as economics, physics, and engineering. Characterization of the occupation time of one specific off-state marginally or two off-states jointly is crucial to understanding such processes. We derive the exact marginal and joint distributions of the off-state occupation times. The theoretical results are confirmed numerically in a simulation study. A special case when all holding times have Levy distribution is considered for the possibility of simplification of the formulas.


Introduction
Markov renewal on-off processes, also known as alternating renewal processes or telegraph processes, arise in applications in a variety of fields. Classic alternating renewal processes were first studied by Cane [2], Page [12] and Newman [11] with appxlications to animal ethology, maintenance of electronic equipment, and communication engineering, respectively.
Recently, the telegraph process and their variations are employed in various fields of applications such as pricing in mathematical finance and insurance [e.g., 9,6,7,8,16], modelling the propagation of a damped wave in physics [e.g., 4,10], and inventory and storage models in engineering [18]. In certain applications, the off-state of an on-off process has multiple types, which raises new questions about its properties such as the distribution of the occupation time in specific off-state type marginally or multiple off-state types jointly. Our interest in on-off processes with multiple off-states is motivated by our recent work in animal movement modeling, where a predator can have different non-moving states such as resting or handling a kill [15,14]. The occupation time in the handling state is important for ecologists to understand the behavior of predators.
To fix ideas, consider a server that has two different types of failures, each requiring a different time and cost to repair. One basic question is: if the server is on at time 0, what is the distribution of the time one spends fixing the type one failures by time t > 0? We model the server state process with the following Markov renewal process. The process starts in state 0 (the on-state) and spend there a random holding time according to a given absolutely continuous distribution. When the first holding time is over, we flip an asymmetric coin to decide which type of failure (off-state 1 or off-state 2) comes next. The holding times in the off-states are also absolutely continuous, generally with different distributions. Once the second holding time is over the process returns to the on-state (state 0) and then we repeat the construction process.
The occupation time of a specific type of the off-state is our focus. For the on-state, the distribution of the occupation time is known from the results on the telegraph process [13,5,17,19]. Indeed, if we collapse the two off-states into one, then the resulting process is a regular on-off process (or alternating renewal process) whose off-state holding time distribution is a mixture of the two original off-state holding time distributions. If the different types of repair require different resources, however, we want to know the distribution of the occupation time in a particular type of off-state. For that task, the results on classical telegraph processes are not directly applicable. Moreover, assume that the cost for being in an off-state is proportional to its occupation time and different for different off-states. If one wants to know the distribution of the total cost for repairs of all kinds by a fixed time, then we need the joint distribution of the two types of off-state occupation times. 3 Our problem is related to but different from some recent works. An extension of the telegraph process to a process with three states is studied in Bshouty et al. [1], where within a renewal cycle all three states are visited in a given deterministic order. In our case, however, we have an on-off process with two off-states; only two states (the on-state and an off-state) are visited in each renewal cycle, but the off-states is chosen randomly. Another related work is Crimaldi et al. [3], where there are two states, but at each random epoch, the new state is determined by the outcome of a random trial, and, as a result, the process can stay in the same state.
The key idea to study the occupation time of a specific off-state is to exploit a certain periodicity of our Markov renewal process. In fact, when analyzing any Markov renewal process, it is always convenient to do the conditioning on returning to a certain state. For the on-off process with two off-states, state 0 has an additional nice property. The numbers of steps between two consecutive visits of state 0 is not random, and it is always equal to 2.
We derive the marginal distribution of the occupation time of a specific off-state first, and then we modify our derivation to obtain the joint distribution of the two off-state occupation times. Of course, one can get the marginal distribution by integrating the joint one. Our approach, however, is easier to follow.

Algorithmic Construction of On-Off Process with Two Off-States
Suppose that we are given the following collection of independent sequences of non-negative random variables: 4 (1) {U k } k≥1 are independent identically distributed (iid) random variables with cumulative distribution function (cdf) F U and probability density function (pdf) f U (these random variables will be used as the holding times when the server is up, state 0); (2) {S k } k≥1 are iid random variables with cdf F S and pdf f S (holding times when the server is down for short repairs, state 1); (3) {L k } k≥1 are iid random variables with cdf F L and pdf f L (holding times when the server is down for long repairs, state 2); (4) {ξ k } k≥1 are iid random variables with Pr(ξ k = 1) = p 1 > 0 and Pr(ξ k = 0) = p 2 = 1 − p 1 . Now, we present our construction of the on-off process with two off-states, X(t).
Then the occupation times in state 0, state 1, and state 2 are, respectively, The corresponding defective marginal densities of S(t) and L(t) are denoted as where t ≥ 0, 0 < s < t, j = 0, 1, 2. The defective joint two-dimensional density of S(t) and where j = 0, 1, 2. The densities are defective in the following sense: since both occupation times S(t) and L(t) have an atom at 0, all three sums are less than 1.

Marginal Distribution of Occupation Time of an Off-State
To derive the distribution of occupation time we will need some auxiliary random variables.
Let N(t) be the number of cycles (or returns to the on-state) by time t. Formally, for n ≥ 0 These random variables are associated with the off-state holding time of the regular on-off process when two off-states are combined.
Our formulas include convolutions of different distributions. We will use the following notation. If we are given a cdf G(·) and its pdf g(·), then G (n) (·) denotes n-fold convolution of G(·), and g (n) (·) denotes the n-fold convolution of g(·). If we are given two cdfs G(·) and H(·) with pdfs g(·) and h(·), then G * H(·) denotes the convolution of cdfs G(·) and H(·), and g * h(·) denotes the convolution of pdfs g(·) and h(·).
We also use the following conventions. Any summation over the empty set is 0. Zero-fold convolution G (0) (·) is the cdf of a random variable that is equal to 0 with probability 1.
Pr(S(t) = 0, X(t) = 1) = 0, Proof. First, note that the distribution of S(t) has an atom. Indeed, if U 1 > t or all the failures that occur before t are of the second type, then S(t) = 0. More specifically, by conditioning on the number of returns to the on-state, N(t), we obtain that Next, again by conditioning on N(t), we get that for 0 < s < t Pr (S(t) ∈ ds, X(t) = 0, N(t) = n) .
8 The next step is to fix the number of switches to failures of type 1. Since the total numbers of switches is n and there is at least one failure of type 1, we have Pr S(t) ∈ ds, X(t) = 0, N(t) = n, n j=1 ξ j = k . Because and it really does not matter during which cycles the switches to failures of type 1 occur, we find that Next, observe that in this case S(t) = k j=1 S j . Using independence between {ξ k } k≥1 and the holding time sequences we have Finally, independence of holding time sequences gives us that Now let us consider the case when X(t) = 1 (at time t the server is down for a short repair). One difference is that there is no atom in this case. As before, for 0 < s < t we have Pr (S(t) ∈ ds, X(t) Note that event can be rewritten as The other formulas can be derived in a similar way.
Theorem 1 gives us the distribution of occupation time in state 1. Since both off-states enter our story in a completely symmetric way, the formulas for the occupation time in state 2 can be obtained by interchanging state 1 and 2 in Theorem 1.

Joint Distribution of Off-State Occupation Times
As mentioned in the introduction, if we are interested in the distribution of the total cost, then we need the joint distribution of off-state occupation times. More specifically, assume that the total cost C(t) is a linear function of occupation times, that is, Then the distribution of C(t) is fully determined by the joint distribution of S(t) and L(t).
Note that we do not need the joint distribution of all three occupation times, because U(t) = t − S(t) − L(t).
Since both S(t) and L(t) have atoms at 0, for every value of X(t) we have four cases: (1) both occupation times are 0; (2) S(t) = 0, L(t) > 0; (3) S(t) > 0, L(t) = 0; and (4) both S(t) and L(t) are strictly greater than 0. In total, we have 12 formulas. Some of them are trivial. For instance, event {S(t) = 0, X(t) = 1} has probability 0, therefore, the corresponding defective one-dimensional density of L(t) is also 0. Moreover, since there is a certain symmetry between S(t) and L(t) some formulas can be found by interchanging state 1 and 2. That is why in the following theorem below we have only 5 formulas.
Proof. We will only derive the formula for p SL1 (u, v, t). The remaining formulas can be obtained by similar modifications of the proof of Theorem 1.
As before, we start with partitioning with respect to events {N(t) = n}. More specifically, for u, v > 0 and 0 < u + v < t we have Note that now the upper limit of the inner summation is n − 1, because n j=1 ξ j = n and ξ n+1 = 1 implies that L(t) = 0.
Next, observe that event can be rewritten as Finally, taking into account that in the case when {X(t) = 1} occupation time S(t) = t − n+1 j=1 U j − n j=k+1 L j and occupation time L(t) = n j=k+1 L j , we get that One can verify now that, for instance, Using symmetry between S(t) and L(t), we also can get that

Special Case: Lévy Distribution
In this section we consider a special case when all the holding times have the Lévy distribution (with location parameter 0 and possibly different scale parameters). The Lévy distribution with scale parameter c 2 has pdf and cdf 14 The Lévy distribution is heavy-tailed with infinite expectation. The median is given by 0.5c 2 (erfc −1 (0.5)) −2 ≈ 2.198112c 2 , where erfc is the complementary error function: [ Figure 1 about here.] [ Figure 2 about here.] The parametrization that we use is a bit unusual, but it allows us to shorten our notation for convolutions. More specifically, as a member of the family of stable distributions with mobility parameter 1/2, the Lévy distribution is closed under the operation of convolution in the following way: Let c 2 U , c 2 S , and c 2 L be the scale parameters for states 0, 1 and 2, respectively. Then the formulas from Theorem 1 and Theorem 2 do not involve any convolutions. For instance, in this case we have that The expectations and variances of the off times is given by E(S(t)) = Note that the discrete component of the occupation times is not used for these calculations, because the atoms are at 0. The covariance of S(t) and L(t) is given by .
As an example we consider a server with median holding time 7 days, 0.5 hour, and 4 hours for the on-state, the short off-state, and the long off-state, respectively. Using one day as the time unit, the model parameters are: c U ≈ 1.785, c S ≈ 0.097, and c L ≈ 0.275. Assume also that the total repair cost is given by that is, long repairs are twice costlier than the short ones.  We also ran a simulation study to numerically confirm the correctness of Theorem 1 and Theorem 2. A total 1, 000, 000 realizations of the above on-off process were generated and the empirical results are consistent with the theoretical densities (not shown). Table 1 summarizes the expectation and variance of the two types of off-state occupation times and the associated repair cost for t ∈ {30, 60} days and p 1 ∈ {0.70, 0.75, . . . , 0.95, 0.99}.
As p 1 increases, the expectation and the variance goes up for the less serious breakdown time but goes down fro the more serious breakdown time. The total repair cost has lower expectation and variance for smaller values of p 1 . The correlation of the two off-state occupation times is negative, with a magnitude decreasing as p 1 increases. When t is doubled, the expectation of the occupation times and the total repair cost is slightly more than doubled, which is because the process always starts from the on-state with a random holding time.

Concluding Remarks
The obtained results can be generalized to the case of more than two off-states. The only difference is that instead of binomial probabilities one has to use multinomial ones.
The marginal distribution of an off-state occupation time could be obtained by collapsing the on-state with the other off-state and using results on the telegraph process. Nonetheless, this approach would lead to more complicated formulas because the holding time of the new collapsed state is distributed as an infinite mixture of convolutions. And of course, this technique cannot be employed if we need the joint distribution of the two off-state occupation times.
The process C(t) also can be viewed as a telegraph process governed by the on-off process with multiple off-states. That is, C(t) is the time t position of a particle that moves with speed C j whenever X(t) = j (j = 0, 1, 2). The formulas for the joint distribution of X(t) and occupation times can be useful for parameter estimation when process C(t) is discretely observed. In particular, if X(t) is also Markov (that is, all the holding times are exponential) then the efficient likelihood estimation is possible with help of tools for hidden Markov model [14].