Markov processes with product-form stationary distribution

We study a class of Markov processes with finite state space and continuous time that have product form stationary distributions. We obtain a number of examples that can generate conjectures for diffusions with inert drift.


Introduction
This research has been inspired by several papers on processes with inert drift [5,4,3,1]. The model involves a "particle" X and an "inert drift" L, neither of which is a Markov process by itself, but the vector process (X, L) is Markov. It turns out that for some processes (X, L), the stationary measure has the product form; see [1]. The first goal of this note is to give an explicit characterization of all processes (X, L) with a finite state space for X and a product form stationary distribution-see Theorem 2.1.
The second, more philosophical, goal of this paper is to develop a simple tool that could help generate conjectures about stationary distributions for processes with continuous state space and inert drift. So far, the only paper containing a rigorous result about the stationary distribution for a process with continuous state space and inert drift, [1], was inspired by computer simulations. Examples presented in Section 3 lead to a variety of conjectures that would be hard to arrive at using pure intuition or computer simulations.

The model
Let S = {1, 2, . . . , N } for some integer N > 1 and let d ≥ 1 be an integer. We define a continuous time Markov process (X(t), L(t)) on S × R d as follows. We associate with each state j ∈ S a vector v j ∈ R d , 1 ≤ j ≤ N . Define L j (t) = µ({s ∈ [0, t] : X(s) = j}), where µ is Lebesgue measure, and let L(t) = j∈S v j L j (t). To make the "reinforcement" non-trivial, we assume that at least one of v j 's is not 0. Since L will always belong to the hyperplane spanned by v j 's, we also assume that d = dim(span{v 1 , . . . , v N }).
We also select non-negative functions a ij (l) which define the Poisson rates of jumps from state i to j. The rates depend on l = L(t). We assume that a ij 's are right-continuous with left limits. Formally speaking, the process (X, L) is defined by its generator A as follows, Af (j, l) = v j · ∇ l f (j, l) + We assume that (X, L) is irreducible in the sense of Harris, i.e., for some open set U ⊂ R d and some j 0 ∈ S, for all (x, l) ∈ S × R d , we have for some t > 0, P ((X(t), L(t)) ∈ {j 0 } × U ) > 0.
We are interested only in processes satisfying (14) below. Using that condition, it is easy to check Harris irreducibility for each of our models by a direct argument. A standard coupling argument shows that Harris irreducibility implies uniqueness of the stationary probability distribution (assuming existence of such).
The (formal) adjoint of A is given by We are interested in invariant measures of product form so suppose that g(j, l) = p j g(l), where j∈S p j = 1 and R d g(l)dl = 1. We may assume that p j > 0 for all j; otherwise some points in S are never visited. Under these assumptions, (1) becomes Theorem 2.1. Assume that for every i and j, the function l → a ij (l) is continuous. A probability measure p j g(l)djdl is invariant for the process (X, L) if and only if Proof. Recall that the state space S for X is finite. Hence v * := sup j∈S |v j | < ∞. Fix arbitrary r, t * ∈ (0, ∞). It follows that, sup i,j∈S,l∈B(0,r+2t * v * ) a ij (l) = a * < ∞.
Note that we always have |L(t)−L(u)| ≤ v * |t−u|. Hence, if |L(0)| ≤ r +t * v * and s, t > 0, s+t ≤ t * , then |L(s + t)| ≤ r + 2t * v * and, therefore, This implies that the probability of two or more jumps on the interval [s, s + t] is o(t). Assume that |l| ≤ r + t * v * and t ≤ t * . Then we have the following three estimates. First, where the remainder Let a ii (l) = − j =i a ij (l). We have where the remainder R 2 i,l (t) satisfies sup i∈S,l∈B(0,t * v * ) |R 2 i,l (t)| ≤ R 2 (t) for some R 2 (t) such that lim t→0 R 2 (t)/t = 0. Finally, where the remainder R 3 i,l (t) satisfies sup i∈S,l∈B(0,t * v * ) Suppose that |l 0 | ≤ r + v * t * , t 1 ∈ (0, t * ) and s ∈ (0, t * − t 1 ). Then We combine this formula with (3)-(5) to see that, which can be rewritten as We will analyze the limit We also have This implies that for all i and |l 0 | ≤ r + v * t * . We will argue that weakly when l → l 0 . Let T k be the time of the k-th jump of X. We have Since l → a ii (l) is continuous, we conclude that weakly as l → l 0 . This and continuity of l → a ij (l) for every j implies that weakly when l → l 0 . By the strong Markov property applied at T k 's, we obtain inductively that when l → l 0 , for every k ≥ 1. This easily implies (8), because the number of jumps is stochastically bounded on any finite interval. Since (j, l) ֒→ ∇ l f (j, l) · v j + k∈S f (k, l)a jk (l) is a continuous function, it follows from (6) and Recall We have just shown that for a fixed t ≤ t * and any j, the function l → u t (j, l) is C 1 . Hence we can apply (7) with f (j, l) = u t (j, l) to obtain, Now assume that (2) is true and let π(dj, dl) = p j g(l)djdl. In view of (10), we can change the order of integration in the following calculation. For 0 ≤ t 1 < t 2 ≤ t * , using (9), Let h(j, l) = p j g(l). For a fixed j and s ≤ t * , the function u s (j, l) = 0 outside a compact set, so we can use integration by parts to show that We combine this with the previous formula and the assumption that A * h ≡ 0 to see that It follows that t → E π f (X(t), L(t)) is constant for every C 1 function f (j, l) with compact support. This proves that the distributions of (X(t 1 ), L(t 1 )) and (X(t 2 ), L(t 2 )) are identical under π, for all 0 ≤ t 1 < t 2 ≤ t * . Conversely, assume that π(dj, dl) = p j g(l)djdl is invariant. Then the left hand side of (11) is zero for all 0 ≤ t 1 < t 2 ≤ t * . This implies that for a set of s that is dense on [0, ∞). Note that lim s↓0 u s (j, l) = f (j, l). Hence, the collection of C 1 functions u s (j, l), obtained by taking arbitrary C 1 functions f (j, l) with compact support and positive reals s dense in [0, ∞), is dense in the family of C 1 functions with compact support. This and (13) imply that A * h ≡ 0, that is, (2) holds.
Corollary 2.2. If a probability measure p j g(l)djdl is invariant for the process (X, L) then Proof. Summing (2) over j, we obtain for all l. Since g is integrable over R d , it is standard to show that there exist l 1 , l 2 , . . . , l d which span R d . Applying (15) to all l 1 , l 2 , . . . , we obtain (14).
It will be convenient to use the following notation, Corollary 2.3. A probability measure p j g(l)djdl is invariant for the process (X, L) and g(l) is the Gaussian density if and only if the following equivalent conditions hold, Proof. If g(l) is the Gaussian density then ∇g(l) = −lg(l) and (2) is equivalent to (18). Conversely, if (2) and (18) are satisfied then ∇g(l) = −lg(l), so g(l) must have the form (17).
In the rest of the paper we will consider only processes satisfying (18)-(19).
Example 2.4. We now present some choices for a ij 's. Recall the notation x + = max(x, 0), x − = − min(x, 0), and the fact that Given v j 's, p j 's and b ij 's which satisfy (19) and the condition b ij = −b ji , we may take Then as desired.
The above is a special case, in a sense, of the following. Suppose that p j = p i for all i and j. Assume that v j 's and b ij 's satisfy (19) and the condition b ij = −b ji . Fix some c > 0 and let a ij (l) = b ij (l) exp(cb ij (l)) exp(cb ij (l)) − exp(−cb ij (l)) .
It is elementary to check that with this definition, (16) is satisfied for all i and j, because b ij (l) = −b ji (l). The formula (21) arose naturally in [5]. Note that (20) (with all p i 's equal) is the limit of (21) as c → ∞.

Approximation of processes with continuous state space
This section is contains examples of processes (X, L) with finite state space for X, and conjectures concerned with processes with continuous state space. There are no proofs in this section First we will consider processes that resemble diffusions with reflection. In these models, the "inert drift" is accumulated only at the "boundary" of the domain.
We will now assume that elements of S are points in a Euclidean space R n with n ≤ N . We denote them S = {x 1 , x 2 , . . . , x N }. In other words, by abuse of notation, we switch from j to x j . We also take v j ∈ R n , i.e. d = n. Moreover, we limit ourselves to functions b ij (l) of the form b ij · l for some vector b ij ∈ R n . Then (19) becomes . . .
Consider any orthogonal transformation Λ : R n → R n . If {b ij , v j , p j } satisfy (22) then so do {Λb ij , Λv j , p j }.
Suppose that a ij (l) have the form a ij · l for some a ij ∈ R n . If {a ij , v j , p j } satisfy (18) then so do {Λa ij , Λv j , p j }. Moreover, the process with parameters {a ij , v j } has the same transition probabilities as the one with parameters {Λa ij , Λv j }.
Example 3.1. Our first example is a reflected random walk on the interval [0, 1]. Let x j = (j − 1)/(N − 1) for j = 1, . . . , N . We will construct a process with all p j 's equal to each other, i.e., p j = 1/N . We will take l ∈ R 1 , v 1 = α and v N = −α, for some α = α(N ) > 0, and all other v j = 0, so that the "inert drift" L changes only at the endpoints of the interval. We also allow jumps only between adjacent points, so b ij = 0 for |i − j| > 1. Then (22) yields Solving this, we obtain b i(i+1) = −α/N for all i.
We would like to find a family of semi-discrete models indexed by N that would converge to a continuous process with product-form stationary distribution as N → ∞. For 1 < i < N , we set a i(i+1) (l) = A R (l, N ) and a (i+1)i (l) = A L (l, N ). We would like the random walk to have variance of order 1 at time 1, for large N , so we need Since b i(i+1) = −α/N for all i, A R and A L have to satisfy When l is of order 1, we would like to have drift of order 1 at time 1, so we take α = N . Then (24) becomes Solving (23) and (25) gives Unfortunately, A L and A R given by the above formula can take negative values-this is not allowed because a ij 's have to be positive. However, for every N , the stationary distribution of L is standard normal, so l typically takes values of order 1. We are interested in large N so, intuitively speaking, A R and A L are not likely to take negative values. To make this heuristics rigorous, we modify the formulas for A R and A L as follows, Let P N denote the distribution of (X, L) with the above parameters. We conjecture that as N → ∞, P N converge to the distribution of reflected Brownian motion in [0, 1] with inert drift, as defined in [5,1]. The stationary distribution for this continuous time process is the product of the uniform measure in [0, 1] and the standard normal; see [1].
Example 3.2. This example is a semi-discrete approximation to reflected Brownian motion in a bounded Euclidean subdomain of R n , with inert drift. In this example we proceed in the reversed order, starting with b ij 's and a ij 's. Consider an open bounded connected set D ⊂ R n . Let K be a (large) integer and let D K = Z n /K ∩ D, i.e., D K is the subset of the square lattice with mesh 1/K that is inside D. We assume that D K is connected, i.e., any vertices in D K are connected by a path in D K consisting of edges of length 1/K. We take S = D K and l ∈ R n .
We will consider nearest neighbor random walk, i.e., we will take a ij (l) = 0 for |x i − x j | > 1/K. In analogy to (26), we define Let us call a point in S = D K an interior point if it has 2n neighbors in D K . We now define v j 's using (22) with p j = 1/|D K |. For all interior points x j , the vector v j is 0, by symmetry. For all boundary (that is, non-interior) points x j , the vector v j is not 0.
Fix D ⊂ R n and consider large K. Let P K denote the distribution of (X, L) constructed in this example. We conjecture that as K → ∞, P K converge to the distribution of normally reflected Brownian motion in D with inert drift, as defined in [5,1]. If D is C 2 then it is known that the stationary distribution for this continuous time process is the product of the uniform measure in D and the standard Gaussian distribution; see [1].
The next two examples are discrete counterparts of processes with continuous state space and smooth inert drift. The setting is similar to that in Example 3.2. We consider an open bounded connected set D ⊂ R n . Let K be a (large) integer and let D K = Z n /K ∩ D, i.e., D K is the subset of the square lattice with mesh 1/K that is inside D. We assume that D K is connected, i.e., any vertices in D K are connected by a path in D K consisting of edges of length 1/K. We take S = D K and l ∈ R n . Example 3.3. This example is concerned with a situation when the stationary distribution has the form p j g(l) where p j 's are not necessarily equal. We start with a C 2 "potential" V : D → R. We will write V j instead of V (x j ). Let p j = c exp(−V j ). We need an auxiliary function .
Note that d ij = d ji and for a fixed i, we have d ij K → 1 when K → ∞ and |i − j K | = 1/K. Let a ij (l) = 0 for |x i − x j | > 1/K, and for |x i − x j | = 1/K, We set If a ji (l) > 0 and a ij (l) > 0 then b ij (l) = p i a ij (l) − p j a ji (l) It follows from (28) that the above formula holds also if a ji (l) ≤ 0 or a ij (l) ≤ 0. Consider an interior point x j . For (19) to be satisfied, we have to take For large K, series expansion shows that v j ≈ −∇V.
Fix D ⊂ R n and consider large K. Let P K denote the distribution of (X, L) constructed in this example. We recall the following SDE from [1], where B is standard n-dimensional Brownian motion and V is as above. Let P * denote the distribution of (Y, S). We conjecture that as K → ∞, P K converge to P * . Under mild assumptions on V , it is known that the stationary distribution for (Y, S) is the product of the measure exp(−V (x))dx and the standard Gaussian distribution; see [1].
Example 3.4. We again consider the situation when all p j 's are equal, i.e., p j = 1/N . Consider a We set For large K, we have v j ≈ −2∇V . Fix D ⊂ R n and consider large K. Let P K denote the distribution of (X, L) constructed in this example. Consider the following SDE, where B is standard n-dimensional Brownian motion and V is as above. Let P * denote the distribution of (Y, S). We conjecture that as K → ∞, P K converge to P * , and that the stationary distribution for (Y, S) is the product of the uniform measure on D and the standard Gaussian distribution.
The next example and conjecture are devoted to examples where the inert drift is related to the curvature of the state space, in a suitable sense.
Example 3.5. In this example, we will identify R 2 and C. The imaginary unit will be denoted by i, as usual. Let S consist of N points on a circle with radius r > 0, x j = r exp(j2πi/N ), j = 1, . . . , N . We assume that the p j 's are all equal to each other.
For any pair of adjacent points x j and x k , we let with the other a kj (l) = 0. Then b j(j+1) = N (x j+1 − x j ) · l, and by (19) Let P N be the distribution of (X, L) constructed above. Let C be the circle with radius r > 0 and center 0, and let T y be the projection of R 2 onto the tangent line to C at y ∈ C. Consider the following SDE, where Y takes values in C and B is Brownian motion on this circle. Let P * be the distribution of (Y, S). We conjecture that as N → ∞, P N converge to P * , and that the stationary distribution for (Y, S) is the product of the uniform measure on the circle and the standard Gaussian distribution.
Conjecture 3.6. We propose a generalization of the conjecture stated in the previous example. We could start with an explicit discrete approximation, just like in other examples discussed so far. The notation would be complicated and the whole procedure would not be illuminating, so we skip the approximation and discuss only the continuous model.
Let S ⊂ R n be a smooth (n − 1)-dimensional surface, let T y be the projection of R n onto the tangent space to S at y ∈ S, let n(y) be the inward normal to S at y ∈ S, and let ρ be the mean curvature at y ∈ S. Consider the following SDE, where Y takes values in S and B is Brownian motion on this surface. We conjecture that for some c 0 depending only on the dimension n, the stationary distribution for (Y, S) exists, is unique and is the product of the uniform measure on S and the standard Gaussian distribution.
We end with examples of processes that are discrete approximations of continuous-space processes with jumps. It is not hard to construct examples of discrete-space processes that converge in distribution to continuous-space processes with jumps. Stable processes are a popular family of processes with jumps. These and similar examples of processes with jumps allow for jumps of arbitrary size, and this does not mesh well with our model because we assume a finite state space for X. Jump processes confined to a bounded domain have been defined (see, e.g., [2]) but their structure is not very simple. For these technical reasons, we will present approximations to processes similar to the stable process wrapped around a circle.
In both examples, we will identify R 2 and C. Let S consist of N points on the unit circle D, x j = exp(j2πi/N ), j = 1, . . . , N . We assume that the p j 's are all equal to each other, hence, p j = 1/N . In these examples, L takes values in R, not R 2 .
Example 3.7. Consider a C 3 -function V : D → R. We write V j = V (x j ). We define A(j, k) = 1 if x j and x k are adjacent on the unit circle, 0 otherwise.
For any pair of points x j and x k , not necessarily adjacent, we let where α ∈ (0, 2). We define a jk (l) = a jk (l) ∨ 0 if a kj (l) > 0, N 2 (V k − V j )A(j, k)l otherwise.
Then b jk (l) = N (V k − V j )A(j, k)l and by (19) we have v k = −N 2 Note that v k → ∆V (x) = V ′′ (x) when N → ∞ and x k → x. Let P N be the distribution of (X, L) constructed above. Let W (x) = V (e ix ) and let (Z, S) be a Markov process with the state space R × R and the following transition probabilities. The component Z is a jump process with the drift ∇W (Z)S = W ′ (Z)S. The jump density for the process Z is n∈Z |(x − y) + n2π| −1−α . We let S t = t 0 ∆W (Z s )ds. Let Y t = exp(iZ t ) and P * be the distribution of (Y, S). We conjecture that P N → P * as N → ∞ and the process (Y, S) has the stationary distribution which is the product of the uniform measure on D and the standard normal distribution. The process (Y, S) is a "stable process with index α, with inert drift, wrapped on the unit circle." Example 3.8. Consider a continuous function V : D → R with D V (x)dx = 0. Recall the notation V j = V (x j ). For any pair of points x j and x k , not necessarily adjacent, we let a jk (l) = 1 N where α ∈ (0, 2). We define a jk (l) = a jk (l) ∨ 0 if a kj (l) > 0, Note that if arg x k → y when N → ∞ then v k → V (e iy ) − D V (x)dx = V (e iy ). Let P N be the distribution of (X, L) constructed above. Let W (x) = V (e ix ) and let (Z, S) be a Markov process with the state space R×R and the following transition probabilities. The component Z is a jump process with the jump density f (x) = (W (x) − W (y))s − n∈Z ((x − y) + n2π) −1−α at time t, given {Z t = y, S t = s}. We let S t = t 0 W (Z s )ds. Let Y t = exp(iZ t ) and P * be the distribution of (Y, S). We conjecture that P N → P * as N → ∞ and the process (Y, S) has the stationary distribution which is the product of the uniform measure on D and the standard normal distribution.

Acknowledgments
We are grateful to Zhenqing Chen and Tadeusz Kulczycki for very helpful suggestions.