SYMMETRY AND NON-EXISTENCE OF SOLUTIONS FOR A NONLINEAR SYSTEM INVOLVING THE FRACTIONAL LAPLACIAN

. In this paper, we consider the following system of pseudo-diﬀerent-ial nonlinear equations in R n where α is any real number between 0 and 2. We obtain radial symmetry in the critical case and non-existence in the subcritical case for positive solutions. To this end, we ﬁrst establish the equivalence between (1) and the corresponding integral system (cid:40) A new idea is introduced in the proof, which may hopefully be applied to many other problems. Combining this equivalence with the existing results on the integral system, we obtained much more general results on the qualitative properties of the solutions for (1).

where α is any real number between 0 and 2.
We obtain radial symmetry in the critical case and non-existence in the subcritical case for positive solutions.
A new idea is introduced in the proof, which may hopefully be applied to many other problems. Combining this equivalence with the existing results on the integral system, we obtained much more general results on the qualitative properties of the solutions for (1).
The first author is supported by China Scholarship Council (No.201206060010). The second author is partially supported by the Simons Foundation Collaboration Grant for Mathematicians 245486.
The fourth author is supported by the Fundamental Research Funds for the Central Universities (ZYGX2013J108).

1126
RAN ZHUO, WENXIONG CHEN, XUEWEI CUI AND ZIXIA YUAN 1. Introduction. The fractional Laplacian in R n is a nonlocal pseudo-differential operator, taking the form where α is any real number between 0 and 2 and PV stands for the Cauchy principal value. This operator is well defined in S, the Schwartz space of rapidly decreasing C ∞ functions in R n . In this space, it can also be defined equivalently in terms of the Fourier transform (−∆) α/2 u(ξ) = |ξ| αû (ξ) whereû is the Fourier transform of u. One can extend this operator to a wider space of distributions as the following. Let dx < ∞} (see [24]).
For u ∈ L α , we define (−∆) α/2 u as a distribution: Throughout the paper, we will consider the solutions in this distributional sense. One can verify that, when u is in S, all the above definitions coincide. In recent years, there has been a great deal of interest in using the fractional Laplacian to model diverse physical phenomena, such as anomalous diffusion and quasi-geostrophic flows, turbulence and water waves, molecular dynamics, and relativistic quantum mechanics of stars ( see [5] [6] [13] [27] and the references therein). It also has various applications in probability and finance [1] [3] [15]. In particular, the fractional Laplacian can be understood as the infinitesimal generator of a stable Lévy process [3].
In this paper, we study the nonlinear equation involving the fractional Laplacian and more generally, the system of m equations To investigate the symmetry and non-existence of solutions, the method of moving planes has been a powerful tool. However, it cannot be applied to the above nonlocal problem directly, because the method strongly relies on maximum principles of local nature. In [2], the authors used the Caffarelli-Silvestre's extension method. After being extended to one more dimension, it becomes a local problem of a second order elliptic equation which obeys a maximum principle of local nature, and hence the method of moving planes can be applied to the extended local problem. By doing so, they technically required that 1 ≤ α < 2 and u be globally bounded. In this paper, in order to extend the range of α to between 0 and 1 and to weaken the global bounded-ness condition to only local bounded-ness, we introduce a direct approach by considering the corresponding integral equation We first establish the equivalence between pseudo-differential equation (3) and integral equation (5): Theorem 1. Assume that n ≥ 2 and u ∈ L α is a nonnegative solution of (3) with 0 < p < ∞, then u also satisfies (5), and vice versa.
Combining Theorem 1 with the qualitative properties established for the integral equations in [11] and [12], one obtain immediately Corollary 1. Assume that n ≥ 2 and u is a locally bounded non-negative solution of (3) for 0 < α < 2. Then i) In the critical case when p = n+α n−α , it must assume the form ii) In the subcritical case when 1 < p < n+α n−α , we must have u ≡ 0. Remark 1. i) In [11] and [12], in order the results in Theorem 1 to hold, they required u to be in H α/2 (R n ). Here we only require u ∈ L α , a much weaker restriction.
ii) In [2], by using the extension method to obtain the same results as in part ii) of Theorem 1, the authors required that 1 ≤ α < 2 and u be globally bounded. Obviously, our condition here is much weaker.
Then more generally, we consider pseudo-differential system (4). In order to guarantee that the system contains no independent subsystem, such as, to avoid a situation like (− ) α/2 u 1 = u p 1 and (− ) α/2 u 2 = u q 2 ; we introduce the following assumption. We say system (4) is interrelated if . . , i m is a permutation of 1, 2, . . . , m.
We also assume (f) f 1 , . . . , f m are real-valued, non-negative, continuous, homogeneous functions of degree 1 < γ ≤ n+α n−α , and nondecreasing with respect to the variables u 1 , . . . , u m . Similar to the single equation, we establish the equivalence between pseudodifferential system (4) and the corresponding integral system: Theorem 2. Assume that system (4) is interrelated and condition (f) holds. Let u = (u 1 , . . . , u m ) be a positive locally bounded solution of (4). Then, in the critical case γ = n+α n−α , u = (u 1 , . . . , u m ) also satisfies integral system (6), and vice versa. The significance for establishing the equivalence between the pseudo-differential system and the integral system is that the traditional method of moving planes cannot be applied to the nonlocal fractional order pseudo-differential system. However, for integral systems, a powerful method of moving planes in integral forms had been developed in [11] to investigate properties of solutions, which works for all real values of α between 0 and n indiscriminately, and which only requires the solutions be locally integrable or locally bounded.
For integral system (6), Chen and Li [9] have proved Proposition 1. Assume that system (6) is interrelated and condition (f) holds. Then in the critical case γ = n+α n−α , every positive locally bounded solution u = (u 1 , · · · , u m ) is radially symmetric with the same center.
Combining Theorem 2 and Proposition 1, we obtain immediately Corollary 2. Assume system (4) is interrelated and condition (f) holds. Then in the critical case γ = n+α n−α , every positive locally bounded solution u = (u 1 , · · · , u m ) is radially symmetric with the same center.
Using a similar idea as in the proof of Theorem 2, we also establish non-existence of positive solutions for both (4) and (6).
Theorem 3. Assume system (4) and (6) are interrelated and condition (f) holds. Let u = (u 1 , · · · , u m ) be a nonnegative locally bounded solution of either (4) or (6). Then in the subcritical case 1 < γ < n+α n−α , we must have u ≡ 0. To prove Theorem 1, 2, and 3, we introduce the following new ideas, which may be applied to a variety of other situations.
First, we employed the Liouville Theorem for α-harmonic functions.
Proposition 2. Assume that n ≥ 2. Let u be a solution of then u ≡ C.
This result was first obtained in [4], and an alternative proof was given in [29]. In [14] and [26], similar Liouville Theorems were also established.
Applying Proposition 2 and the maximum principle for the fractional Laplacian (see [24]), we prove that, if u = (u 1 , · · · , u m ) is a solution of pseudo-differential system (4), then there exists c i ≥ 0, such that For the case of single equation (3), we have Consequently, u ≥ c. If c = 0, one can derive a contradiction with the finiteness of the integral. However, for system (4), in the case that some of c 1 , · · · , c m are zero and some are not, one cannot derive a contradiction with the finiteness of the integrals. To overcome this difficulty, we employ a Kelvin transform centered at any given point for some non-negative number β.
If c i > 0 for some i, then due to the presence of the singular term ci |x−x o | n−α , we are able to use the method of moving planes in integral forms to show thatū i must be radially symmetric about the point x o , hence so does u i . Since x o is an arbitrary point in R n , we conclude that u i is constant. This contradicts system (4). Therefore, we arrive at (6) as well as In the subcritical case, we have β > 0, and we can still utilize the other singular term 1 |y−x o | β in the above integral and the method of moving planes in integral forms to derive thatū i must be radially symmetric about the point x o , and hence arrive a contradiction as mentioned above. This establishes the non-existence of positive solutions.

Remark 2.
An integral representation formula similar to (8) has been established for poly-harmonic operator (when α is an even number) in [8], in which, instead of f i (u), they considered more general positive measure µ.
In Section 2, we establish the equivalence between single equations and hence prove Theorem 1 and Corollary 1. In Section 3, we consider systems in critical case and derive Theorem 2. In Section 4, we deal with systems in subcritical case and obtain Theorem 3.

A single equation.
In this section, we prove Theorem 1 and Corollary 1.
The Proof of Theorem 1. Assume u ∈ L α is a nonnegative locally bounded solution of Let where G R (x, y) is Green's function on the ball B R (0): . Thanks to [20], one can write . A n,α and B n,α are constants depending on n and α.
It is easy to verify that  (9) and (12), we have To continue, we need the following maximum principle: Let Ω ⊂ R n be a bounded open set, and let f be a lower-semicontinuous function inΩ such that Applying this maximum principle we derive that One can verify that, It's easy to see Denote Then by (9), (14), (15), and (16), we have From Theorem 2, we derive that w ≡ C. Then obviously, Next, we show that C = 0. Otherwise, if C > 0, then This is a contradiction. Therefore we conclude that This complete the proof of Theorem 1.
The Proof of Corollary 1. It is a direct consequence of Theorem 1 and the following results from [11] and [12]: Assume that n ≥ 2 and u is a locally bounded nonnegative solution of the integral equation (19) for 0 < α < n. Then i) In the critical case when p = n+α n−α , it must assume the form ii) In the subcritical case when 1 < p < n+α n−α , we must have u ≡ 0.
3. Systems in critical case. In this section, we present the proof of Theorem 2. Consider the following pseudo-differential system: We first show that For where G R (x, y) is the Green's function on the ball B R (0) (see (11)). It is easy to see that (20) and (23), we derive By the maximum principle (see Proposition 2.1), we have It's easy to prove that Obviously, . Using (20), (25), (26), and (27), we derive From Proposition 2, we derive that w i (x) = c i . That is, (21) holds. Next we show that c i ≡ 0, i = 1, . . . , m.
Without a lower bound assumption on f i , it requires more effort to show that all c i must be zero. To this end, we employ the Kelvin transform and the method of moving planes in integral forms. This is the first time that such a new idea has been used to prove the equivalence between the two systems, and we hope to see more applications to other situations in the near future.
Then we introduce the Kelvin transform. For any x 0 ∈ R n , we consider the Kelvin transform centered at x 0 For simplicity of arguments, we will only show the case when x 0 is the origin, while the proof for a general x 0 is entirely similar. Letū be the Kelvin transform of u i centered at origin.
From (21), we derivē where γ is the homogeneous degree of f i . In this section, we consider the critical case where γ = n+α n−α , and in this casē We will show that c i must be 0, for i = 1, . . . , m. Otherwise, there exists some i o , such that c io > 0, and we will derive a contradiction.
It's easy to see that Since u i ∈ L ∞ loc (R n ), for any domain Ω that is a positive distance away from the origin, we have By an elementary calculation, we derivē By (34), we havē We will move the plane T λ along the direction of x 1 -axis until λ = 0 to show that the solution is radially symmetric about the origin. The proof consists of two steps.
Step 2. (Move the plane to the limiting position to derive symmetry.) Inequality (36) provides a starting point to move the plane T λ . Now we start to move the plane T λ along x 1 direction as long as (36) holds. Define , a.e. ∀x ∈ Σ λ \ B (0 λ ), λ ≤ λ}. We will show that λ 0 = 0. Suppose on the contrary, λ 0 < 0, we will verify that u(x) is symmetric about T λ0 , that is If (47) does not hold, then we claim that, for all i = 1, · · · , m, In fact, by (36), we have, for some l, u λ0 l (y) >ū l (y), on a set of positive measure.
(51) Combining (50) with (51), we obtain This is impossible. Hence (48) holds. Next based on (48), we will verify that the plane can be moved further to the right. That is, for λ 0 < λ sufficiently closed to λ 0 , In fact, by inequality (46), we have By (33), for any small η > 0, we can choose R sufficiently large, such that For any τ > 0, define It's easy to see that For λ sufficiently closed to λ 0 , µ(D τ ) is very small. We will show that µ(Γ λ i ∩ E τ i ) is sufficiently small as λ close to λ 0 .
In fact,ū It follows that By Chebyshev inequality, we arrive at For fixed τ , as λ close to λ 0 , the right hand side of inequality (57) can be sufficiently small. By (55) and (56), we derive that µ(Γ λ i ∩ B R (0)) can be made as small as we wish.
For any point x 0 ∈ R n , employing the Kelvin transform centered at x 0 and applying an entirely similar argument, one can show thatū(x) is radially symmetric about x 0 . We will show that the solution u of (21) must be constant.
Let x 1 and x 2 be any two points in R n . We choose the coordinate system so that the midpoint x 0 = x 1 +x 2 2 is the origin. Applying radial symmetry ofū about x 0 , we obtainū( , and consequently u(x 1 ) = u(x 2 ). This implies that u(x) must be constant. By our positive assumption on u, we have u i (x) = b i > 0, i = 1, . . . , m.
Taking u i into (4), we have This is impossible. Hence, in (21), c i must be zero, i = 1, . . . , m. We conclude that the positive solutions of pseudo differential system (4) also satisfy integral system (6). The converse can be derived easily by applying the operator (−∆) α 2 to both sides of the integral system (6).
This completes the proof of Theorem 2 3.1. Systems in subcritical case. In the section, we prove Theorem 3.
In the previous section, we obtained where G(x, y) = cn |x−y| n−α , c i ≥ 0, i = 1, . . . , m. Again letū be the Kelvin transform of u i centered at origin.
Similar to the critical case (29), we derivē where β = 2n − (γ + 1)(n − α) > 0, and γ is the homogeneous degree of f i , with 1 < γ < n+α n−α . The difference is that, in the critical case, β = 0; while in the subcritical case β > 0. We will use the presence of the singular term 1 |z| β to show that no matter whether c i are zero or not,ū is always symmetric about the origin.
The argument is quite similar, but not entirely the same, as that in the critical case, hence we still present some details here.
We haveū Since u i ∈ L ∞ loc (R n ), for any domainΩ that is a positive distance away from the origin, we have The proof also consists of two steps.
By an elementary calculation, we derive that Similar to (45), we deduce that In this case, we still define w λ i (y) = 0, forū i (y) <ū λ i (y), u i (y) −ū λ i (y), forū i (y) >ū λ i (y), and w λ (y) = (w λ 1 (y), . . . , w λ m (y)). Combining Lemma 3.1 and Hölder inequality, we obtain Step 2. (Move the plane to the limiting position to derive symmetry.) Step 1 provides a starting point to move the plane T λ . Now we start to move the plane T λ to the right as long as (72) holds to the limiting position. Define λ 0 = sup{λ < 0|ū λ i (x) ≥ū i (x), a.e. ∀x ∈ Σ λ \ B (0 λ ), λ ≤ λ}. The rest is entirely similarly to the critical case. We only need to use Γ λ ū γ−1 i (y) |y| β n α dy instead of Γ λū 2n n−α (y)dy. We also conclude that λ 0 = 0,ū(x) =ū λ0 (x), a.e.x ∈ Σ 0 . Now it is easy to see that each positive solutionū of (64) is radially symmetric about the origin in subcritical case. Using an entirely similar argument, one can show thatū is radially symmetric about any point x 0 ∈ R n . Therefore, each positive solution u of (21) must be constant. We will further show that u(x) ≡ 0. Otherwise, if u i = d i > 0 for some i, we will deduce a contradiction as in (62). This implies that there is no positive solution of (4) in the subcritical case, and hence completes the proof of Theorem 3.