Existence of Solutions to a Class of Quasilinear Schrödinger Systems Involving the Fractional P-laplacian

The purpose of this paper is to investigate the existence of solutions to the following quasilinear Schrödinger type system driven by the fractional p-Laplacian (−∆) s p u + a(x)|u| p−2 u = H u (x, u, v) in R N , s q v + b(x)|v| q−2 v = H v (x, u, v) in R N , where 1 < q ≤ p, sp < N, (−∆) s m is the fractional m-Laplacian, the coefficients a, b are two continuous and positive functions, and H u , H v denote the partial derivatives of H with respect to the second variable and the third variable. By using the mountain pass theorem, we obtain the existence of nontrivial and nonnegative solutions for the above system. The main feature of this paper is that the nonlinearities do not necessarily satisfy the Ambrosetti–Rabinowitz condition.


Introduction
In this paper we are concerned with the following fractional Schrödinger system where N > ps with s ∈ (0, 1) and (−∆) s m is the fractional m-Laplace operator which (up to normalization factors) may be defined along any ϕ ∈ C ∞ 0 (R N ) as |x − y| N+ms dy for x ∈ R N , see [14] and the references therein for further details on the fractional m-Laplacian.
The nonlinearities H u and H v denote the partial derivative of H with respect to the second variable and the third variable, respectively, and H satisfies some hypotheses to be stated in the sequence.
In particular, (−∆) s m becomes to the fractional Laplacian (−∆) s as m = 2, and it is known that (−∆) s m reduces to the standard m-Laplacian as s ↑ 1, see [14].In fact, nonlocal and fractional operators arise in a quite natural way in many different applications, such as, continuum mechanics, phase transition phenomena, population dynamics and game theory, as they are the typical outcome of stochastically stabilization of Lévy processes, see for instance [3,10].The literature on nonlocal operators and on their applications is interesting and quite large, we refer the reader to [11,23,27,[30][31][32][33] and the references therein.For the basic properties of fractional Sobolev spaces, we refer the reader to [14].
On the one hand, in the context of fractional quantum mechanics, nonlinear fractional Schrödinger equation was first proposed by Laskin in [18,19] as a result of expanding the Feynman path integral, from the Brownian-like to the Lévy-like quantum mechanical paths.In the last years, there has been a great interest in the study of the fractional Schrödinger equation where the nonlinearity f satisfies some general conditions.For standing wave solutions of fractional Schrödinger equations in R N , see, for instance, [12,15,24,29] and the reference therein.It is worthy mentioning that Autuori and Pucci in [5] studied the following elliptic equations involving fractional Laplacian where λ ∈ R, 0 < s < 1, 2s < N, 2 < q < min{r, 2 * s }, 2 * s = 2N/(N − 2s), and (−∆) s is the fraction Laplacian operator.The authors obtained the existence and multiplicity of entire solutions of (1.2) by using the direct method in variational methods and the mountain pass theorem.The same nonlinearities were recently considered by Pucci and Saldi in [25], where the authors established existence and multiplicity of nontrivial nonnegative entire weak solutions of a stationary Kirchhoff eigenvalue problem, involving a general nonlocal integrodifferential operator.More precisely, they considered the problem where λ ∈ R, 0 < s < 1, 2s < N, and L K is an integro-differential nonlocal operator.The case with variable exponents was treated by Pucci and Zhang in [28], where the authors studied the one parameter elliptic equation where λ ∈ R and A : R N × R N → R N admits a potential A , with respect to its second variable ξ and satisfies some assumptions listed in the paper.On the other hand, Servadei and Valdinoci in [30] studied the following fractional Laplacian problem where Ω is a bounded domain.Since the fractional Laplacian operator is nonlocal, some technical difficulties arise when applying usual variational methods.In this paper, the authors used the mountain pass theorem to obtain the existence of solutions for problem (1.3).After that, many authors devoted to studying the fractional problems by using variational methods and critical point theory.Indeed, a lot of works in the literature involve the following Ambrosetti-Rabinowitz ((AR) in short) condition on the nonlinearity for some constant µ > p, see [2] for more details.In general, the (AR) condition not only ensures that the Euler-Lagrange functional associated with a problem of the form (1.1) has a mountain pass geometry, but also guarantees boundedness of the Palais-Smale sequences corresponding the functional.Although the (AR) condition is very important and widely used to get the existence of solutions for elliptic problems via variational methods, it is not fulfilled by some simple nonlinearities, such as In fact, these functions do not satisfy which is a consequence of the (AR) condition.Here we would like to sketch some advances about this aspect.For the Laplacian case as p = q = 2, the existence of nontrivial weak solutions for nonlinear elliptic problems without assuming (AR) condition was obtained in [21].For the single p-Laplacian case with p > 1, we refer the reader to [20].For the (p, q)-Laplacian with 1 < q ≤ 2 ≤ p and a = b = 0, we refer the reader to [22].For the fractional Laplacian case, the existence of infinitely many weak solutions for nonlinear elliptic problems without requiring (AR) condition was investigated in [8].Motivated by the above works, especially [12,13], we are interested in the study of solutions for system (1.1) without the (AR) condition involving the fractional p-Laplacian.For this, let V (R N ) be a subset of C(R N ) and for any V ∈ V (R N ), (A 1 ) V is bounded from below by a positive constant; where B κ (y) denotes any open ball of R N centered at y and of radius κ > 0. Note that condition (A 2 ), which is weaker than the coercivity assumption: V(x) → ∞ as |x| → ∞, was originally discussed by Bartsch and Wang in [7] to overcome the lack of compactness.
Without further mentioning, we always assume a, b ∈ V (R N ).Now we introduce some notation.Assume By the embedding E p,a → L p (R N ) (see [26]), we can define and deduce that λ * > 0.
Finally, the conditions we impose on the nonlinearity are: (H 2 ) there exist r ∈ (p, q * s ) and C > 0 such that where where Remark 1.1.Obviously, the functions f 1 (t) and f 2 (t) mentioned before satisfy (H 1 )-(H 5 ).
For the single equation, it is easy to see that (1.4) implies the more weaker condition (H 5 ).
Condition (H 4 ) together with (H 5 ), introduced by Jeanjean in [16], was often used to study the existence of nontrivial solutions for superlinear problems without (AR) condition in recent years, see for example [20,22].Now, we give the definition of weak solutions of problem (1.1).
Definition 1.2.We say that (u, v) ∈ W is a (weak) solution of problem (1.1), if The main result of this paper is the following.
This article is organized as follows.In Section 2, we give some necessary definitions and properties of the fractional Sobolev space W. In Section 3, using the mountain pass theorem, we obtain the existence of solutions for system (1.1).

Preliminaries
In this section, we give some basic results of fractional Sobolev space W that will be used in the next section.
By Lemma 10 of [26], one has E p,a = (E p,a , • E p,a ) and E q,b = (E q,b , • E q,b ) are two separable, reflexive Banach spaces.Hence, by Theorem 1.12 of [1], we have the following.Lemma 2.1.W = (W, • W ) is a separable and reflexive Banach space.

Lemma 2.2. The embedding
and Proof.By Lemma 1 of [26], there exists C ν > 0 such that Hence, Hence the lemma is proved.
Similar to Proposition A.10 of [4], we have the following lemma.
By using the same discussion as in [26, Theorem 2.1], we have the following compact embedding.
Lemma 2.4.Suppose that a, b ∈ V (R N ) and 1 < q ≤ p < q * s .Let ν ∈ [p, q * s ) be a fixed exponent.Then the embeddings E p,a → L ν (R N ) and E q,b → L ν (R N ) are compact.Moreover, the embedding

Proof of Theorem 1.3
To prove Theorem 1.3, we need the following mountain pass theorem under condition (C).
Theorem 3.1 (see [17,Theorem 6]).Let E be a real Banach space with its dual space E * , and suppose that J ∈ C 1 (E, R) satisfies = e} is the set of continuous paths joining 0 and e, then there exists a sequence {u n } n ⊂ E such that This kind of sequence is usually called a Cerami sequence.
Definition 3.2.We say that a functional J : E → R of class C 1 satisfies the Cerami condition ((C) in short) if any Cerami sequence associated with J has a strongerly convergent subsequence in E.
The Euler-Lagrange functional associated with system (1.1) is Clearly, the functional I is well-defined in W.Under conditions (H 1 )-(H 5 ), it is easy to see that the functional I is of class C 1 , and for (u, v) ∈ W for all (ϕ, ψ) ∈ W. Lemma 3.3.Any Cerami sequence associated with the functional I is bounded in W.
Proof.Let {(u n , v n )} n ⊂ W be a Cerami sequence associated with I. Then there exists C > 0 independent of n such that |I(u n , v n )| ≤ C and (1 + (u n , v n ) )I (u n , v n ) → 0 as n → ∞.In the sequel, we will use C to denote various positive constant that does not depend on n.Hence there exists ε n > 0, with ε n → 0, such that , for all (ϕ, ψ) ∈ W and n ∈ N. (3.1) Hence we have Next we show that {(u n , v n )} n is bounded in W. Arguing by contradiction, we assume that (u n , v n ) → ∞.Without loss of generality, we assume that Then there exists (X, Y) ∈ W such that, up to a subsequence, Moreover, by Lemma 2.4, we can assume that, up to a subsequence, where the last inequality follows from the following elementary inequality Hence, we deduce from (3.4) that as n → ∞ Similarly, by we obtain Set Thus, by (H 3 ) a.e. in Ω + .Then we deduce from Fatou's lemma that On the other hand, by|I(u n , v n )| ≤ C, one has which contradicts with (3.5).Hence Ω + has zero measure, that is, (X, Y) = (0, 0) a.e. in R N .By the continuity of the function t ∈ [0, 1] → I(tu n , tv n ), there exists a sequence {t n } n ⊂ [0, 1] such that where θ > 1/2.By (H 5 ), for any ε > 0, there exists δ > 0 such that For all x ∈ R N and |z| > δ, we have by (H 2 ) Hence, we obtain where Since (u n , v n ) → ∞, there exists n 0 large enough such that (2θ) 1/q / (u n , v n ) ∈ (0, 1) for all n ≥ n 0 .Hence, for all n ≥ n 0 , we obtain by q ≤ p, θ > 1 2 and Then I(t n u n , t n v n ) > θ p2 p−1 , for all n ≥ n 1 , this together with the arbitrariness of θ > 1/2 yields lim n→∞ I(t n u n , t n v n ) = ∞. (3.9) Following the same discussion as Lemma 7.3 of [6] and using the facts that I(0, 0) = 0 and I(u n , v n ) ≤ C, we can assume that t n ∈ (0, 1) for all n ≥ n 2 ≥ n 1 .Thus, for all n ≥ n 2 .From 0 ≤ t n ≤ 1 and (H 4 ), one has Combining (3.10) with (3.11), we have for all n ≥ n 2 .Then, this together with (3.9) yields On the other hand, by |I(u n , v n )| ≤ C, we have Adding this inequality to (3.3) and using (H 4 ), we obtain which contradicts with (3.12).Therefore, we conclude that {(u n , v n )} n is bounded in W.
Proof.Assume {(u n , v n )} n is a Cerami sequence.Then there exists C > 0 independent of n such that for any ν ∈ [p, q * s ).By using (3.6) corresponding ε = 1 and applying Lemma 2.4, we deduce as n → ∞, where C denotes various positive constants.Thus we get Observe that the linear functional L(u, v) : W → R defined by for all (ϕ, ψ) ∈ W, is bounded in W by using the Hölder inequality, see [26].
Now we are in position to prove Theorem 1.3.
In view of (H 2 ) and (H 3 ), for a positive constant A > 0, there exists C A > 0 such that H(x, z) ≥ A|z| p − C A for all (x, z) ∈ R N × R 2 .
sq), sq < N, and let (H 1 )-(H 5 ) hold.Assume a, b ∈ V (R N ).Then system (1.1) has at least one pair of nontrivial and nonnegative weak solution