Nonlinear Markov Processes in Big Networks

Big networks express various large-scale networks in many practical areas such as computer networks, internet of things, cloud computation, manufacturing systems, transportation networks, and healthcare systems. This paper analyzes such big networks, and applies the mean-field theory and the nonlinear Markov processes to set up a broad class of nonlinear continuous-time block-structured Markov processes, which can be applied to deal with many practical stochastic systems. Firstly, a nonlinear Markov process is derived from a large number of interacting big networks with symmetric interactions, each of which is described as a continuous-time block-structured Markov process. Secondly, some effective algorithms are given for computing the fixed points of the nonlinear Markov process by means of the UL-type RG-factorization. Finally, the Birkhoff center, the Lyapunov functions and the relative entropy are used to analyze stability or metastability of the big network, and several interesting open problems are proposed with detailed interpretation. We believe that the results given in this paper can be useful and effective in the study of big networks.


Introduction
In this paper, we consider a large number of interacting big networks with symmetric interactions, each of which is described as a continuous-time block-structured Markov process, which can be applied to deal with many practical stochastic systems. As the The main contributions of this paper are threefold. The first one is to set up a broad class of nonlinear continuous-time block-structured Markov processes when applying the mean-field theory to analyze a large number of interacting big networks with symmetric interactions, each of which is described as a continuous-time block-structured Markov process. The second one is to propose some effective algorithms for computing the fixed points of the nonlinear Markov processes by means of the UL-type RG-factorization, and show for some big networks that there possibly exist multiple fixed points, which lead to the metastability. The third one is to use the Birkhoff center, the Lyapunov functions and the relative entropy to analyze either stability or metastability of the big networks, and to give several interesting open problems with detailed interpretation. Based on this, this paper provides some new computational lines in the study of big networks. We believe that the results given in this paper can be useful and effective in performance evaluation and optimization of the big networks.
The remainder of this paper is organized as follows. In Section 2, we derive a class of nonlinear Markov processes through an asymptotic analysis of the weakly interacting big networks, in which each big network evolves as a continuous-time block-structured Markov process. In Section 3, we provide some effective algorithms for computing the fixed points of the system of ordinary differential equations. In Section 4, we discuss the Birkhoff center of the mean-field dynamic system, and apply the Lyapunov functions and the relative entropy to study the stability or metastability of the big network. Also, we provide several interesting open problems with detailed interpretation. Some concluding remarks are given in the final section.

Nonlinear Markov Processes
In this section, we derive a class of nonlinear Markov processes through an asymptotic analysis for a collection of weakly interacting big networks, in which each big network evolves as a continuous-time block-structured Markov process, which can be applied to deal with many practical stochastic systems.
To be able to discuss a system of big networks, we assume that any individual big network evolves as a continuous-time block-structured Markov process X whose infinitesimal generator is given by where the size of the matrix Q j,j is m j for j ≥ 0, and the sizes of other matrices can be determined accordingly. It is easy to see that the matrix Q j,j is also the infinitesimal generator of a continuous-time Markov process with m j states for j ≥ 0. We assume that the continuous-time Markov process Q is irreducible, aperiodic and positive recurrent, and its state space may be expressed as a two-dimensional structure: [44] for more details.
From the continuous-time block-structured Markov chain X , the system of N weakly interacting big networks is described as an X N -valued Markov process, where the states of the N big networks are denoted as X 1,N (t), X 2,N (t), . . ., X N,N (t), respectively.
Let X N (t) = X 1,N (t) , X 2,N (t) , . . . , X N,N (t) . Then the empirical measure of the system of N big network system is given by where δ x is the Dirac measure at x.
We denote by P (Ω) the space of probability vectors on the state space Ω, which is equipped with the usual topology of weak convergence. If p ∈ P (Ω), we write p = (p 0 , p 1 , p 2 , . . .), where the size of the vector p j is m j for j ≥ 0. At the same time, it is clear that µ N (t) ∈ P (Ω) is a random variable for t ≥ 0, and µ N (t) : t ≥ 0 is a continuous-time Markov process.
For the X N -valued continuous-time block-structured Markov process, we define that the probability distribution of X N (t) is exchangeable, if for any level permutation (k i 1 , k i 2 , . . ., k i N ) of (k 1 , k 2 , . . . , k N ) and any phase permutation (j i 1 , j i 2 , . . . , j i N ) of (j 1 , j 2 , . . . , j N ), In the system of N weakly interacting big networks, the effect of a typical big network on the dynamics of the given big network is of order 1/N , and the jump intensity of any given big network depends on the configuration of other big networks only through the empirical measure µ N (t). To study the system of N weakly interacting big networks in terms of Markov processes, it is seen from probability one that at most one big network will jump, i.e., change state, at a given time, and the jump intensities of any given big network depend only on its own state and the state of the empirical measure at that time.
In addition, the jump intensities of the N big networks have the same functional form.
Based on this, for the X N -valued Markov process, if the initial probability distribution of X N (0) is exchangeable, then at any time t ≥ 0, the probability distribution of X N (t) is also exchangeable.
For the system of N weakly interacting big networks, if the probability distribution of X N (t) is exchangeable, then the N big networks are indistinguishable, thus we apply the mean-field theory to be able to analyze this system through only considering the Markov process of any given big network (such as, the first big network); while analysis of the total system will be completed by the propagation of Chaos (as N → ∞). Based on this, the infinitesimal generator of the Markov process corresponding to the first big network can be defined as follows: where the size of the matrix Γ (N ) j,j µ N (t) is m j for j ≥ 0, and the sizes of other matrices can be determined (a.s.) accordingly. Since µ N (t) is a random variable, it is clear that is a random matrix of infinite order. On the other hand, it is seen from the law of large number that the limit of the empirical measure µ N (t) is deterministic under suitable conditions.
is a probability vector, and using some probability analysis, we may obtain an infinite-dimensional dynamic system as follows: with the initial condition Obviously, the mean-field dynamic system, given in (5) and (6), is related to a nonlinear Markov process whose infinitesimal generator is given by Remark 1 To establish the infinitesimal generator Γ (p (t)) of a nonlinear Markov process, readers may also refer to some recent publications, for example, the discrete-time Markov chains by Benaim and Le Boudec [7] and Budhiraja and Majumder [16], the Markov decision processes by Gast and Bruno [31] and Gast at al. [32], the continuoustime Markov chains by Dupuis and Fischer [23] and Budhiraja et al. [14,15], and some nice practical examples include Mitzenmacher [53], Bobbio et al. [8], Li et al. [46,47], and Li and Lui [48].
In what follows, it is necessary to provide some useful interpretation or proofs for how to establish the mean-field dynamic system (5) and (6).

(a) Existence and Uniqueness
Consider the infinite-dimensional ordinary differential equation: d dt p (t) = p (t) Γ (p (t)) with p (0) = q. A solution in the classical sense is a (continuously) differential function and p (0) = q is in the interior of E, then there exists a unique global solution to the ordinary differential equation: d dt p (t) = p (t) Γ (p (t)) with p (0) = q, within E.
To deduce whether the Γ (x) is (locally) Lipschitz on a set E ⊆ P (Ω), Li et al. [46] and Li and Lui [48] gave an algorithmic method through dealing with some matrices of infinite orders.
(b) The limiting processes To discuss the limit: µ N (t) → p (t) (a.s.) for t ≥ 0, as N → ∞, we need to set up some suitable conditions in order to guarantee the existence of such a limit.
Let e k,j be the unit vector of infinite dimension in which the (k, j)th entry is one and all the others are zero. Note that the empirical measure process Hence the transition rate of the Markov process corresponding to the given big network is k,j;l,i (x). Based on this, the generator A (N ) of the Markov process µ (N ) is given by where f (x) is a real function on P N (Ω). It is easy to see that as N → ∞ converges uniformly on compact time intervals in probability to p (t) ∈ P (Ω) for t ≥ 0, where the probability vector p (t) is the unique global solution to the ordinary differential Proof: The proof may directly follow from Theorem 2.11 in Kurtz [42]. Here, we only give a simple interpretation as follows. Firstly, we notice that and and Γ (x) is (locally) Lipschitz on a set E ⊆ P (Ω). Then, for the sequence µ (N ) (t), t ≥ 0 of Markov processes, it follows from Equation (III.10.13) in Rogers and Williams [57] or page 162 in Ethier and Kurtz [24] that is a martingale with respect to N ≥ 1. Therefore, if µ (N ) (0) converges weakly to q ∈ P (Ω) as N → ∞, then µ (N ) (t), N ≥ 1 converges weakly in D F [0, +∞) endowed with the Skorohod topology to the solution p (t) to the ordinary differential equation: d dt p (t) = p (t) Γ (p (t)) with p (0) = q, within P (Ω). This completes the proof.

The Fixed Points
In this section, we use the UL-type RG-factorization to provide some effective algorithms for computing the fixed points of the ordinary differential equation: d dt p (t) = p (t) Γ (p (t)) with p (0) = q. Further, we set up a nonlinear characteristic equation of the censoring matrix to level 0, which is satisfied by the fixed points.
In this case, it is clear that which is an infinite-dimensional system of nonlinear equations. In general, there exist more difficulties and challenging due to its infinite dimensions when solving the fixed point equation (8) together with πe = 1, where e is a column vector of ones with a suitable size.
It is easy to check that for every π ∈ P (Ω), Γ (π) is the infinitesimal generator of an irreducible continuous-time Markov process. Based on Li [44], we can develop the UL-type RG-factorization of the matrix Γ (π). To that end, we partition the matrix Γ (π) as according to the level sets L ≤n and L ≥n+1 for n ≥ 0. Since the Markov chain Γ (π) is irreducible, it is clear that the two truncated chains with infinitesimal generators T (π) and W (π) are all transient, and the matrices T (π) and W (π) are all invertible. Note that the inverse of the matrix T (π) is ordinary, but the invertibility of the matrix W (π) is different under an infinite-dimensional meaning. Although the matrix W (π) of infinite size may have multiple inverses, we in general are interested in the maximal non-positive min is the minimal nonnegative inverse of −W (π). Based on this, for n ≥ 0 we write where the size of the matrix φ (n) j,j (π) is m j for 0 ≤ j ≤ n, and the sizes of other matrices can be determined accordingly. It is clear from Section 7 of Chapter 2 in Li [44] that for Let Ψ n (π) = φ (n) n,n (π) , n ≥ 0; Then the UL-type RG-factorization of the matrix Γ (π) is given by where and Based on the UL-type RG-factorization (9), it follows from Subsection 2.7.3 in Li [44] that the fixed point π is given by where x 0 (π) is the fixed point of the censored Markov chain Ψ 0 (π) to level 0, and the scalar τ is determined by ∞ k=0 π k e = 1 uniquely. Using the expression (10) of the fixed point π, we set up an important relation as follows: In what follows we consider two special cases in order to further explain the fixed point equation (11) with R-measure.

Case one: Nonlinear Markov processes of GI/M/1 type
In this case, the infinitesimal generator Γ (π) is given by Let R (π) be the minimal nonnegative solution to the nonlinear matrix equation Then where the two vectors π 0 and π 1 satisfy the following system of nonlinear matrix equations Thus, the fixed point equation (11) with R-measure is simplified as π = π 0 , π 1 , π 1 R (π) , π 1 R 2 (π) , . . . .

Case two: Nonlinear Markov processes of M/G/1 type
In this case, the infinitesimal generator Γ (π) is given by Let G (π) be the minimal nonnegative solution to the nonlinear matrix equation and for k ≥ 1 A k (π) G k−1 (π) ; and the R-measure The fixed point π is given by where x 0 (π) is the fixed point of the censored Markov chain Ψ 0 (π) to level 0 and the scalar τ is determined by ∞ k=0 π k e = 1 uniquely. Thus, the fixed point equation (11) with R-measure is simplified as Now, we write the fixed point equation (11) with R-measure as a functional form: π = F (R (π)), as shown in the above two special cases. Based on this, we can provide an approximative algorithm as follows: Algorithm I: Computation of the fixed points Step one: Taking any initial probability vector: π (0) ∈ P (Ω).
Step four: For a sufficiently small ε > 0, if π (N +1) − π (N ) < ε, then the computation is over; otherwise we go to Step three.
Note that it is possible for some big networks that there exist multiple fixed points because the infinitesimal generator Γ (π) is more general. In this case, it is a key to design a suitable initial probability vector: π (0) ∈ P (Ω), for example, for any integer m ≥ 1 we Note that for the censored Markov chain Ψ 0 (π) to level 0, we have π 0 Ψ 0 (π) = 0, π 0 e = τ ∈ (0, 1) .
Thus it is easy to see from the irreducibility of the matrix Γ (π) that for the matrix Ψ 0 (π) of size m 0 , rank(Ψ 0 (π)) = m 0 − 1 according to the irreducibility of the matrix Ψ 0 (π), and its eigenvalue with the maximal real part is equal to zero. Let the characteristic equation be f x (π) = det (xI − Ψ 0 (π)) = 0. Then the fixed points satisfy the characteristic equation f 0 (π) = det (Ψ 0 (π)) = 0. Hence the fixed points satisfy the system of nonlinear equations as follows: Note that (12) provide another algorithm for computing the fixed points as follows: Algorithm II: Computation of the fixed points Step one: Providing a numerical solution π to the nonlinear characteristic equation: det (Ψ 0 (π)) = 0.
Step two: Check whether rank(Ψ 0 ( π)) = m 0 − 1. If Yes, then π is a fixed point. If No, then going to Step one.

Stability and Metastability
In this section, we first discuss the Birkhoff center of the mean-field dynamic system: d dt p (t) = p (t) Γ (p (t)) with p (0) = q. Then we apply the Lyapunov functions and the relative entropy to study the stability or metastability of the big networks. Furthermore, we provide several interesting open problems with detailed interpretation.
To describe the isolated element structure of the set S π , we often need to use the Birkhoff center of the mean-field dynamic system, and use the Birkhoff center to check whether the fixed point is unique or not. Based on this, our discussion includes the following two cases: Case one: N → ∞. In this case, we denote by Φ (t) a solution to the system of differential equations d dt p (t) = p (t) Γ (p (t)) with p (0) = q. Thus, the Birkhoff center of the solution Φ (t) is defined as Θ = P ∈ P (Ω) : P = lim k→∞ Φ (t k ) for any scale sequence {t k } with t l ≥ 0 for l ≥ 1 and lim k→∞ t k = +∞ .
Notice that perhaps Θ contains the limit cycles or the equilibrium points (the local minimal points, or the local maximal points, or the saddle points). Thus it is clear that S π ⊂ Θ.
Obviously, the limiting empirical Markov process {Y (t) : t ≥ 0} spends most of its time in the Birkhoff center Θ, where Y (t) = lim N →∞ µ N (t) a.s..
Case two: t → +∞. In this case, we write π (N ) = lim t→+∞ µ N (t) , a.s., since for each N = 1, 2, 3, . . ., if the system of N weakly interacting big networks is stable. Let Ξ = π ∈ P (Ω) : π = lim k→∞ π (N k ) for any positive integer sequence It is easy to see that In what follows, we discuss stability and metastability of the big networks.
For the metastability in S π , a key is to determine a Lyapunov function for the meanfield dynamic system: d dt p (t) = p (t) Γ (p (t)) with p (0) = q. The Lyapunov function g defined on P (Ω) is constructed such that yΓ (y) · ∇g (y) ≤ 0, y ∈ P (Ω) .
Let |S π | be the number of elements in the set S π . If |S π | = 1, then lim N →∞ If |S π | ≥ 2, then the system of big networks exhibits a metastability property, that is, the state of the given big network switches from one stable point to the other after a long residence time. In the study of metastability, it is a key to estimate the expected value of such a residence time. See Bovier [11] and Olivieri and Vares [55] for more details.
An interesting issue in the study of big networks is to analyze stability or metastability of the corresponding nonlinear Markov processes. On this line, it is a key to construct a Lyapunov function or a local Lyapunov function. Note that the relative entropy function in some sense can define a globally attracting Lyapunov function.
For p, q ∈ P (Ω), we define the relative entropy of p with respect to q as Let Ψ (z) = z log z − z + 1. Then if p (t) and q (t) are two different solutions to the ordinary differential equation d dt where Λ is the infinitesimal generator of an irreducible continuous-time Markov process.
In this case, Dupuis and Fischer [23] indicated that and d dt R (p (t) ||q (t)) = 0 if and only if p (t) = q (t) for t ≥ 0. Obviously, d dt R (p (t) ||π) = 0 if and only if p (t) = π for t ≥ 0. [23] further demonstrated that for the ordinary differential equation: d dt p (t) = p (t) Γ (p (t)), the the relative entropy relation (14) can not be applied directly. In this case, they first defined P (N ) (t) as the state probability of the system of N big networks at time t ≥ 0, and let P (N ) (0) = ⊗ N q. Then they gave an approximate method to construct the Lyapunov function as follows:

Dupuis and Fischer
For applying the relative entropy to construct a Lyapunov function, readers may also refer to Budhiraja et al. [14,15] for more details.
In the remainder of this section, we provide several interesting open problems with detailed interpretation.
Open problem one: The mean drift condition.
We consider an irreducible QBD process whose infinitesimal generator is given by where Γ (p) e = 0, the sizes of the matrices B 1 (p) and A 1 (p) are m 0 and m, respectively, and the sizes of other matrices can be determined accordingly. We assume that for any p ∈ P (Ω), the Markov process: A (p) = A 0 (p) + A 1 (p) + A 2 (p), is irreducible, aperiodic and positive recurrent. Let θ p be the stationary probability vector of the the Markov process A (p). Then it is clear that for for any p ∈ P (Ω), the Markov process Γ (p) is positive recurrent if and only if θ p A 2 (p) e > θ p A 0 (p) e.
It is interesting to study how the mean drift condition: θ p A 2 (p) e > θ p A 0 (p) e for any p ∈ P (Ω), can influence stability or metastability of the ordinary differential equation: d dt p (t) = p (t) Γ (p (t)).
Open problem two: The censoring Markov processes.
For the infinitesimal generator Γ (p) given in (7), it is easy to give the infinitesimal generator Ψ 0 (p) of the censoring Markov processes to level 0. It is very interesting (but difficult) to set up some useful relations of stability or metastability between two ordinary differential equations: d dt p (t) = p (t) Γ (p (t)) and d dt p 0 (t) = p 0 (t) Ψ 0 (p (t)).

Concluding Remarks
This paper sets up a broad class of nonlinear continuous-time block-structured Markov processes by means of applying the mean-field theory to the study of big networks, and proposes some effective algorithms for computing the fixed points of the nonlinear Markov process by means of the UL-type RG-factorization. Furthermore, this paper considers stability or metastability of the big network, and gives several interesting open problems with detailed interpretation. Along such a line, there are a number of interesting directions for potential future research, for example: • providing algorithms for computing the fixed points of big networks with multiple stable points; • studying the influence of the censoring Markov processes on the metastability; • discussing how to apply the RG-factorizations given in Li [44] to compute the expected residence times in the study of metastability; and • analyzing some big networks with a heterogeneous geographical environment, and set up their simultaneous systems of nonlinear Markov processes.
the Fostering Plan of Innovation Team and Leading Talent in Hebei Universities under grant (# LJRC027).