Markov models for duration-dependent transitions: selecting the states using duration values or duration intervals?

Abstract

In a Markov model the transition probabilities between states do not depend on the time spent in the current state. The present paper explores two ways of selecting the states of a discrete-time Markov model for a system partitioned into categories where the duration of stay in a category affects the probability of transition to another category. For a set of panel data, we compare the likelihood fits of the Markov models with states based on duration intervals and with states defined by duration values. For hierarchical systems, we show that the model with states based on duration values has a better maximum likelihood fit than the baseline Markov model where the states are the categories. We also prove that this is not the case for the duration-interval model, under conditions on the data that seem realistic in practice. Furthermore, we use the Akaike and Bayesian information criteria to compare these alternative Markov models. The theoretical findings are illustrated by an analysis of a real-world personnel data set.

Appendix: Proofs of theorems and lemmas

Let us recall the definition of the following functions:

$$\begin{aligned} \varphi (t)&= {\left\{ \begin{array}{ll} t\ln {t} &{} \text {if } t>0 \\ 0 &{} \text {if } t=0, \end{array}\right. } \end{aligned}$$

(13)

$$\begin{aligned} \phi (t_1,t_2)&=\varphi (t_1)+\varphi (t_2)-\varphi (t_1+t_2)\text {.} \end{aligned}$$

(12)

and

$$\begin{aligned} \psi (x,y) = \phi (x,y)+\phi (x+y,n_{AB}-y)+\phi (n_{AA}-x-y,y)-\phi (n_{AA},n_{AB})\text {.} \end{aligned}$$

(24)

Lemma 1

$\ln {\hat{L}_{\mathcal {S}}} = \phi (n_{11},n_{12})+\phi (n_{11}+n_{12},n_{13})+\phi (n_{22},n_{23})$

Proof

Using (4), (5) and the fact that $n_2-n_{23}=n_{22}$, we obtain

$$\begin{aligned} \ln {\hat{L}_{\mathcal {S}}}&= \varphi (n_{11})+\varphi (n_{12})+\varphi (n_{13})+\varphi (n_{22})+\varphi (n_{23})-\varphi (n_{1})-\varphi (n_{2}) \\&\begin{aligned}&=\varphi (n_{11})+\varphi (n_{12})-\varphi (n_{11}+n_{12}) \\&\quad + \varphi (n_{11}+n_{12}) +\varphi (n_{13})-\varphi (n_{1}) \\&\quad +\varphi (n_{22})+\varphi (n_{23})-\varphi (n_{2}) \end{aligned} \\&= \phi (n_{11},n_{12})+\phi (n_{11}+n_{12},n_{13})+\phi (n_{22},n_{23}) \text {.} \end{aligned}$$

$\square$

Lemma 2

$\ln {\hat{L}_{\tilde{\mathcal {S}}}} = \phi (n_{11}+n_{12},n_{13})+\phi (n_{22},n_{23})$

Proof

It follows from (10) that

$$\begin{aligned} \hat{L}_{\tilde{\mathcal {S}}}=\frac{ {(n_{11}+n_{12})}^{n_{11}+n_{12}} \, {n_{13}}^{n_{13}} \, {n_{22}}^{n_{22}} \, {n_{23}}^{n_{23}} }{ {n_{1}}^{n_{1}} \, {n_{2}}^{n_{2}}\text {,} } \end{aligned}$$

using $n_{1}=n_{11}+n_{12}+n_{13}$ and $n_{2}=n_{22}+n_{23}$. Hence,

$$\begin{aligned} \ln {\hat{L}_{\tilde{\mathcal {S}}}}&= \varphi (n_{11}+n_{12})+\varphi (n_{13})+\varphi (n_{22})+\varphi (n_{23})-\varphi (n_{1})-\varphi (n_{2}) \\&= \bigl (\varphi (n_{11}+n_{12})+\varphi (n_{13})-\varphi (n_{1}) \bigr ) + \bigl (\varphi (n_{22})+\varphi (n_{23})-\varphi (n_{2}) \bigr ) \\&= \phi (n_{11}+n_{12},n_{13})+\phi (n_{22},n_{23}) \text {.} \end{aligned}$$

$\square$

Lemma 3

The function $\phi$, defined by (12), is homogeneous of the first degree, i.e.

$$\begin{aligned} \phi (tx,ty)=t\,\phi (x,y)\quad \text {for all } t,x,y\ge 0 \end{aligned}$$

Proof

By (13), we have that $\varphi (tu)=t\,\varphi (u)+u\,\varphi (t)$ for all $t,u\ge 0$. The result then follows from (12). $\square$

Lemma 4

The function $\phi$, defined by (12), is convex.

Proof

Let $u,v>0$. We prove that the Hessian matrix of $\phi$ at (u, v), denoted H, is positive semi-definite. Using standard calculus,

$$\begin{aligned} H= \begin{bmatrix} \frac{v}{u(u+v)} &{} -\frac{1}{u+v} \\ -\frac{1}{u+v} &{} \frac{u}{v(u+v)} \end{bmatrix}= \frac{1}{uv(u+v)} \begin{bmatrix} v^2 &{} -uv \\ -uv &{} u^2 \end{bmatrix}\text {.} \end{aligned}$$

(23)

The eigenvalues of H are 0 and $\frac{u^2+v^2}{uv(u+v)}$. They are both non-negative, hence H is positive semi-definite. $\square$

Lemma 5

For the function $\phi$, defined by (12), holds

$$\begin{aligned} \phi (a,b)+\phi (c,d)\ge \phi (a+c,b+d)\quad \text {for all } a,b,c,d\ge 0 \end{aligned}$$

with equality if and only if $ad=bc$.

Proof

The inequality is an immediate consequence of the convexity and homogeneity properties of the function $\phi$ (Lemmas 3, 4):

$$\begin{aligned} \phi (a,b)+\phi (c,d)\ge 2\,\phi (\tfrac{a+c}{2},\tfrac{b+d}{2})=\phi (a+c,b+d)\text {.} \end{aligned}$$

Suppose $ad=bc$. If $ad=0=bc$, then equality holds surely because $\phi (u,v)=0$ if $u=0$ or $v=0$. If $ad=bc\ne 0$, then $c=ta$ and $d=tb$ for some number $t\ne 0$. Hence, by the homogeneity of $\phi$,

$$\begin{aligned} \phi (a,b)+\phi (c,d)&= \phi (a,b)+\phi (ta,tb) \\&= (1+t)\phi (a,b) \\&= \phi ((1+t)a,(1+t)b) =\phi (a+c,b+d)\text {.} \end{aligned}$$

Now suppose equality holds. Then, by homogeneity of $\phi$,

$$\begin{aligned} \frac{\phi (a,b)+\phi (c,d)}{2}=\phi (\tfrac{a+c}{2},\tfrac{b+d}{2})\text {.} \end{aligned}$$

Consequently, the function $f(t)=\phi (u,v)$ with $u=a+t(c-a)$ and $v=b+t(d-b)$ is linear on [0, 1], because $\phi$ is convex. So, $f''(t)=0$ for all $t\in (0,1)$. Using the Hessian matrix of $\phi$ in (23), we obtain

$$\begin{aligned} f''(t)&= (c-a)^2H_{11}+2(c-a)(d-b)H_{12}+(d-b)^2H_{22} \\&= \frac{\left[ (b-d)u-(c-a)v\right] ^2}{uv(u+v)}=\frac{(ad-bc)^2}{uv(u+v)}\text {,} \end{aligned}$$

and the result $ad=bc$ thus follows from $f''(t)=0$. $\square$

Lemma 6

For the function $\phi$, as defined in (12), holds that $\phi _y:t \mapsto \phi (t,y)$ is strictly decreasing and strictly convex for all $y>0$.

Proof

Using standard calculus, ${\phi _y}'(t)=\ln {t}-\ln {(t+y)}<0$ and ${\phi _y}''(t)=\frac{y}{t(t+y)}>0$, if $t>0$. $\square$

Lemma 7

If $n_{AA}>2n_{AB}$, it holds that $\psi (n_{AA}-\frac{3}{2}n_{AB},n_{AB})<0$.

Proof

Let $\alpha =n_{AA}/n_{AB}$. By (24) and Lemma 3,

$$\begin{aligned} \psi (n_{AA}-\tfrac{3}{2}n_{AB},n_{AB})=h(\alpha )n_{AB} \end{aligned}$$

where

$$\begin{aligned} h(t)=\phi _1(t-\tfrac{3}{2})+\phi _1(\tfrac{1}{2})-\phi _1(t) \end{aligned}$$

and $\phi _1$ is the function $u \mapsto \phi (u,1)$. Since $\phi _1$ is strictly convex (Lemma 6), its first derivative is strictly increasing and therefore h is strictly decreasing. Furthermore,

$$\begin{aligned} h(2)=\phi (\tfrac{1}{2},1)+\phi (\tfrac{1}{2},1)-\phi (2,1)=2\phi (\tfrac{1}{2},1)-\phi (2,1)=0 \end{aligned}$$

by Lemma 3. Consequently, $h(\alpha )<0$ since $\alpha >2$. $\square$

Lemma 8

If $n_{AA}>2n_{AB}$, it holds that $\psi (n_{AA}-\tfrac{3}{4}n_{AB},\tfrac{3}{4}n_{AB})<0$.

Proof

Let $\alpha =n_{AA}/n_{AB}$ and $\beta =3/4$. By (24) and Lemma 3,

$$\begin{aligned} \psi (n_{AA}-\beta n_{AB},\beta n_{AB})=h(\alpha )n_{AB} \end{aligned}$$

where

$$\begin{aligned} h(t)=\phi (t-\beta ,\beta )+\phi (t,1-\beta )-\phi (t,1)\text {.} \end{aligned}$$

With the use of the function $\phi _1:u \mapsto \phi (u,1)$ and Eqs. (13) and (12) we can rewrite h(t) as

$$\begin{aligned} h(t)=\phi _1(t-\beta )-\phi _1(t)+\phi (\beta ,1-\beta )\text {.} \end{aligned}$$

Since $\phi _1$ is strictly convex (Lemma 6), its first derivative is strictly increasing and therefore h is strictly decreasing. The result now follows from the fact that $h(2)<0$. $\square$

Lemma 9

If $n_{AA}\ge 3n_{AB}$, it holds that $\psi (n_{AB},n_{AB})<0$.

Proof

Let $\alpha =n_{AA}/n_{AB}$. By (24) and Lemma 3,

$$\begin{aligned} \psi (n_{AB},n_{AB})=h(\alpha )n_{AB} \end{aligned}$$

where

$$\begin{aligned} h(t)=\phi (1,1)+\phi _1(t-2)-\phi _1(t) \end{aligned}$$

and $\phi _1$ is the function $u \mapsto \phi (u,1)$. Since $\phi _1$ is strictly convex (Lemma 6), its first derivative is strictly increasing and therefore h is strictly decreasing. Furthermore, $h(3)=\ln {\frac{16}{27}}<0$. Hence, since $\alpha \ge 3$ the monotonicity of h yields $h(\alpha )<0$ and the result follows. $\square$

Lemma 10

If $n_{AA}> 2n_{AB}$, it holds that $\psi (\tfrac{n_{AA}-n_{AB}}{2},n_{AB})<0$.

Proof

Let $\alpha =n_{AA}/n_{AB}$. By (24) and Lemmas 3 and 6,

$$\begin{aligned} \psi (\tfrac{n_{AA}-n_{AB}}{2},n_{AB})=h(\alpha )n_{AB}\text {,} \end{aligned}$$

where

$$\begin{aligned} h(t)=\phi (t-1,2)-\phi (t,1)\text {.} \end{aligned}$$

Using standard calculus,

$$\begin{aligned} h'(t)=\ln {(t-1)}-\ln {t}<0\text {,} \end{aligned}$$

so that the function h is strictly decreasing. Furthermore, $h(2)=0$ which can be verified by straightforward computation. Hence, since $\alpha >2$ the monotonicity of h yields $h(\alpha )<0$ and the result follows. $\square$

Lemma 11

$\psi (0,\frac{n_{AA}n_{AB}}{n_{AA}+n_{AB}})=0$.

Proof

Denote $\rho =\frac{n_{AA}n_{AB}}{n_{AA}+n_{AB}}$. Then, $n_{AB}-\rho = \tfrac{n_{AB}}{n_{AA}}\rho$ and $n_{AA}-\rho =\tfrac{n_{AA}}{n_{AB}}\rho$, so that, using (24) and Lemma 3,

$$\begin{aligned} \psi (0,\rho )&= \phi (\theta ,n_{AB}-\rho )+\phi (n_{AA}-\rho ,\rho )-\phi (n_{AA},n_{AB}) \\&= \phi (\rho ,\tfrac{n_{AB}}{n_{AA}}\rho )+\phi (\tfrac{n_{AA}}{n_{AB}}\rho ,\rho )-\phi (n_{AA},n_{AB}) \\&=\tfrac{\rho }{n_{AA}}\phi (n_{AA},n_{AB})+\tfrac{\rho }{n_{AB}}\phi (n_{AA},n_{AB})-\phi (n_{AA},n_{AB}) \\&= \left( \tfrac{\rho }{n_{AA}}+\tfrac{\rho }{n_{AB}}-1\right) \,\phi (n_{AA},n_{AB})=0\text {.} \end{aligned}$$

$\square$

Lemma 12

$\psi (0,n_{AB})=\psi (n_{AA}-n_{AB},n_{AB})>0$.

Proof

By (24), $\psi (0,n_{AB})$ and $\psi (n_{AA}-n_{AB},n_{AB})$ are both equal to $\phi _{n_{AB}}(n_{AA}-n_{AB})-\phi _{n_{AB}}(n_{AA})$, where $\phi _{n_{AB}}$ is the function $u \mapsto \phi (u,n_{AB})$. According to Lemma 6, $\phi _{n_{AB}}$ is strictly decreasing and the result follows. $\square$

Lemma 13

For an AB-complete data set, we have that $\hat{\tau }_{23}>\hat{p}_{AB}$ is equivalent to $g(n_{11},n_{12})>0$, where

$$\begin{aligned} g(x,y)=n_{AB}x+(n_{AA}+n_{AB})y-n_{AA}n_{AB}\text {.} \end{aligned}$$

Hence, if $\hat{\tau }_{23}>\hat{p}_{AB}$, the point $(n_{11},n_{12})$ in xy-plane lies above the line through the points $(0,\tfrac{n_{AA}n_{AB}}{n_{AA}+n_{AB}})$ and $(n_{AA},0)$.

Proof

Because of AB-completeness, it holds that $n_{23}=n_{12}$. Furthermore, $n_{2}=n_{22}+n_{23}=n_{22}+n_{12}=n_{AA}-n_{11}$. Consequently, since $\hat{\tau }_{23}=n_{23}/n_{2}$ and $\hat{p}_{AB}=n_{AB}/(n_{AA}+n_{AB})$, we have

$$\begin{aligned} \hat{\tau }_{23}>\hat{p}_{AB}&\Leftrightarrow \; \frac{n_{12}}{n_{AA}-n_{11}}> \frac{n_{AB}}{n_{AA}+n_{AB}} \\&\Leftrightarrow \; n_{12}(n_{AA}+n_{AB})> (n_{AA}-n_{11})n_{AB} \\&\Leftrightarrow \; g(n_{11},n_{12})>0\text {.} \end{aligned}$$

$\square$

Proof of theorem 1

Proof

In expression (17), we can eliminate the variables $n_{13}$, $n_{22}$ and $n_{23}$, since $n_{AA}=n_{11}+n_{12}+n_{22}$, and, by the AB-completeness assumption, $n_{AB}=n_{13}+n_{23}$ and $n_{12}=n_{23}$. Hence, $\varDelta \ell \ell _{\mathrm{M}(k)\text{- }\mathrm{M}}=\psi (n_{11},n_{12})$, where

$$\begin{aligned} \psi (x,y) = \phi (x,y)+\phi (x+y,n_{AB}-y)+\phi (n_{AA}-x-y,y)-\phi (n_{AA},n_{AB}){\text {.}} \end{aligned}$$

(24)

The function $\psi$ is convex, since each of the first three terms in (24) is a convex function of (x, y), as a composition of the convex function $\phi$ (Lemma 4) and an affine function from $\mathbb {R}^2$ to $\mathbb {R}^2$.

Let $\alpha =n_{AA}-n_{AB}$. By assumption, $\alpha >0$. Let $\beta =\min \{n_{AB},\frac{\alpha }{2}\}$ and denote the following points in xy-plane: $\mathbf {a}(0,\frac{n_{AA}n_{AB}}{n_{AA}+n_{AB}})$, $\mathbf {b}(\beta ,n_{AB})$, $\mathbf {c}(\alpha ,n_{AB})$, $\mathbf {d}(0,n_{AA})$, $\mathbf {e}(n_{AA}-\tfrac{3}{2}n_{AB},n_{AB})$ and $\mathbf {f}(n_{AA}-\tfrac{3}{4}n_{AB},\tfrac{3}{4}n_{AB})$. Because $n_{AA}>2n_{AB}$, the 3-simplex $\mathbf {ecf}$ is a subset of the 4-simplex $\mathbf {abcd}$, see Fig. 3.

Fig. 3

Location of the points $\mathbf {a}$ through $\mathbf {f}$ in the xy-plane

Take $k\ge 1$ sufficiently large, so that $\hat{\tau }_{23}>\hat{p}_{AB}$. Then, using Lemma 13 and the fact that $n_{11}\ge n_{12}$ whenever $k\ge 1$, the point $\mathbf {p}(n_{11},n_{12})$ in xy-plane must be contained in the 4-simplex $\mathbf {abcd}$.

Suppose $\varDelta \ell \ell _{\mathrm{M}(k)\text{- }\mathrm{M}}\ge 0$. We prove that $\mathbf {p}$ must then belong to the 3-simplex $\mathbf {ecf}$. First, we observe that $\psi$ is non-positive on the 5-simplex $\mathbf {abefd}$, because $\psi$ is convex and $\psi$ is non-positive in all vertices of $\mathbf {abefd}$ (Lemmas 7, 8, 9, 10, 11). Hence, because $\mathbf {p}\in \mathbf {abcd}$ and $\psi (\mathbf {p})=\varDelta \ell \ell _{\mathrm{M}(k)\text{- }\mathrm{M}}\ge 0$, we have $\mathbf {p}\in \mathbf {ecf}$. Consequently, $n_{11}>n_{AA}-\tfrac{3}{2}n_{AB}$ and $n_{12}>\tfrac{3}{4}n_{AB}$. But then, (20) cannot be satisfied, since $n_{AA}-n_{11}=n_{22}+n_{12}=n_{22}+n_{23}=n_{2}$. $\square$

Proof of theorem 2

Proof

Denote $n_{11}+n_{12}=a$, $n_{13}=b$, $n_{22}=c$ and $n_{23}=d$. Then, $n_{AA}=a+c$ and $n_{AB}=b+d$. Hence, using (18), we have

$$\begin{aligned} \varDelta \ell \ell _{\widetilde{\mathrm{M}}(k)\text{- }\mathrm{M}}=\phi (a,b)+\phi (c,d)-\phi (a+c,b+d)\text {,} \end{aligned}$$

which is non-negative by Lemma 5.

Furthermore, $\varDelta \ell \ell _{\widetilde{\mathrm{M}}(k)\text{- }\mathrm{M}}=0$ is equivalent to $\phi (a,b)+\phi (c,d)=\phi (a+c,b+d)$, which in turn is equivalent to $ad=bc$ by Lemma 5. Finally, $\hat{\tau }_{13}=\hat{\tau }_{23}=\hat{p}_{AB}$ if and only if $ad=bc$, since $\hat{\tau }_{13}=\tfrac{b}{a+b}$, $\hat{\tau }_{23}=\tfrac{d}{c+d}$ and $\hat{p}_{AB}=\tfrac{b+d}{a+c+b+d}$. $\square$

Proof of theorem 3

Proof

If $n_{11}>0$ and $n_{12}>0$, we have by the binomial theorem that $n_{11}^{n_{11}}n_{12}^{n_{12}}< (n_{11}+n_{12})^{n_{11}+n_{12}}$, hence $\phi (n_{11},n_{12})< 0$ using (12). The theorem now follows from (19) and the fact that $\phi (n_{11},n_{12})=0$ if $n_{11}=0$ or $n_{12}=0$. $\square$