Normalized solutions to fractional mass supercritical NLS systems with Sobolev critical nonlinearities

Abstract

In this paper, we investigate the following fractional Sobolev critical Nonlinear Schrödinger coupled systems:

$$\begin{aligned} \left\{ \begin{array}{lll} (-\Delta )^{s} u=\mu _{1} u+|u|^{2^{*}_{s}-2}u+\eta _{1}|u|^{p-2}u+\gamma \alpha |u|^{\alpha -2}u|v|^{\beta } ~ \text {in}~ {\mathbb {R}}^{N},\\ (-\Delta )^{s} v=\mu _{2} v+|v|^{2^{*}_{s}-2}v+\eta _{2}|v|^{q-2}v+\gamma \beta |u|^{\alpha }|v|^{\beta -2}v ~~\text {in}~ {\mathbb {R}}^{N},\\ \Vert u\Vert ^{2}_{L^{2}}=m_{1}^{2} ~\text {and}~ \Vert v\Vert ^{2}_{L^{2}}=m_{2}^{2}, \end{array}\right. \end{aligned}$$

where $(-\Delta )^{s}$ is the fractional Laplacian, $N>2s$, $s\in (0,1)$, $\mu _{1}, \mu _{2}\in {\mathbb {R}}$ are unknown constants, which will appear as Lagrange multipliers, $2^{*}_{s}$ is the fractional Sobolev critical index, $\eta _{1}, \eta _{2}, \gamma , m_{1}, m_{2}>0$, $\alpha>1, \beta >1$, $p, q, \alpha +\beta \in (2+4s/N,2^{*}_{s}]$. Firstly, if $p, q, \alpha +\beta <2^{*}_{s}$, we obtain the existence of positive normalized solution when $\gamma$ is big enough. Secondly, if $p=q=\alpha +\beta =2^{*}_{s}$, we show that nonexistence of positive normalized solution. The main ideas and methods of this paper are scaling transformation, classification discussion and concentration-compactness principle.

1 Introduction and main result

  The motivation for the problem studied in this article arises from finding stationary waves solutions of the following physical model:

$$\begin{aligned} \left\{ \begin{array}{lll} \displaystyle (-\Delta )^{s} \phi _{1}=-i \frac{\partial \phi _{1}}{\partial t}+|\phi _{1}|^{2^{*}_{s}-2}\phi _{1}+\eta _{1}|\phi _{1}|^{p-2}\phi _{1}+ \gamma \alpha |\phi _{1}|^{\alpha -2}\phi _{1}|\phi _{2}|^{\beta },\\ \displaystyle (-\Delta )^{s} \phi _{2}=-i \frac{\partial \phi _{2}}{\partial t}+|\phi _{2}|^{2^{*}_{s}-2}\phi _{2}+\eta _{2}|\phi _{2}|^{q-2}\phi _{2}+\gamma \beta |\phi _{1}|^{\alpha }|\phi _{2}|^{\beta -2}\phi _{2},\\ \phi _{j}(x,t)\rightarrow 0~~\text {as}~~|x|\rightarrow \infty , j=1,2, \end{array}\right. \qquad \end{aligned}$$

(1.1)

where i represents the imaginary unit and $\phi _{j}=\phi _{j}(x,t):{\mathbb {R}}^{N}\times {\mathbb {R}}^{+}\rightarrow {\mathbb {C}}$ is the wave function of the jth ($j=1,2$) component, the mass of them represents the number of particles of each component in the mean-field models for binary mixtures of Bose-Einstein condensation, see [1,2,3] and references therein, $\eta _{j}$ and $\gamma$ denote the intraspecies and interspecies scattering lengths. The sign case for $\gamma$ determines whether the interaction between the states is attractive or repulsive, i.e. the interaction is attractive if $\gamma$ is positive, the interaction is repulsive if $\gamma$ is negative.

An important solution, known as travelling or standing wave, is characterized by ansatz

$$\begin{aligned} \phi _{1}(x,t)=e^{i\mu _{1}t}u(x), ~~\phi _{2}(x,t)=e^{i\mu _{2}t}v(x) \end{aligned}$$

(1.2)

for two unknown functions $u,v: {\mathbb {R}}^{N}\rightarrow {\mathbb {R}}$, where $\mu _{1}, \mu _{2}\in {\mathbb {R}}$. Because these solutions are very similar to each other and retain their mass over time, it makes sense to seek prescribed $L^2$-norm solutions (normalized solutions). Therefore, combining (1.1) and (1.2), we arrive at the following fractional single or double Sobolev critical Schrödinger system:

$$\begin{aligned} \left\{ \begin{array}{lll} (-\Delta )^{s} u=\mu _{1} u+|u|^{2^{*}_{s}-2}u+\eta _{1}|u|^{p-2}u+\gamma \alpha |u|^{\alpha -2}u|v|^{\beta } ~ \text {in}~ {\mathbb {R}}^{N},\\ (-\Delta )^{s} v=\mu _{2} v+|v|^{2^{*}_{s}-2}v+\eta _{2}|v|^{q-2}v+\gamma \beta |u|^{\alpha }|v|^{\beta -2}v ~~\text {in}~ {\mathbb {R}}^{N},\\ \Vert u\Vert ^{2}_{L^{2}}=m_{1}^{2} ~\text {and}~ \Vert v\Vert ^{2}_{L^{2}}=m_{2}^{2}, \end{array}\right. \end{aligned}$$

(1.3)

where $N>2s$, $s\in (0,1)$, $\mu _{1}, \mu _{2}\in {\mathbb {R}}$ are unknown constants, which will appear as Lagrange multipliers, $2^{*}_{s}$ is the fractional Sobolev critical index, $\eta _{1}, \eta _{2}, \gamma , m_{1}, m_{2}>0$, $\alpha>1, \beta >1$, $p, q, \alpha +\beta \in (2+4s/N,2^{*}_{s}],$ and $(-\Delta )^{s}$ is the fractional Laplacican defined by

$$\begin{aligned} (-\Delta )^{s}u(x)=C(N,s)\lim \limits _{\varepsilon \rightarrow 0^{+}}\int _{{\mathbb {R}}^{N}\backslash B_{\varepsilon }(x)}\frac{u(x)-u(y)}{|x-y|^{N+2s}}dxdy, \end{aligned}$$

where

$$\begin{aligned} C(N, s)=\left( \int _{{\mathbb {R}}^{N}}\frac{1-\cos (\omega _{1})}{\omega }d\omega \right) ^{-1}. \end{aligned}$$

For more information about this type of operator we refer to [4].

With regard to the double Sobolev critical Schrödinger coupled systems, many experts and scholars have conducted extensive and in-depth research on this whether it is integer order or fractional order with fixed $\mu _{1}$ and $\mu _{2}$. When $s\rightarrow 1$, Zou et al. [5] considered the existence and symmetry of positive ground states for a double critical coupled systems. Moreover, they studied the limit behavior of positive ground states for another kind of double critical Schrödinger system when the interaction is repulsive in [6]. In the case of $0<s<1$, Zou and Yin [7] proved the asymptotic behaviour and existence of the positive least energy solutions for k-coupled double critical systems driven by a fractional Laplace operator by means of the idea of induction. Yang [8] dealt with a class of fractional Laplacian doubly critical coupled systems, they gave sufficient conditions for the existence of weak solutions by establishing an embedding theorem. He et al. [9] investigated the existence of least energy solution with the help of the Nehari manifold.

However, as far as we know, few papers treat parameters $\mu _{1}$ and $\mu _{2}$ as Lagrange multipliers to study the normalized solution of double Sobolev critical problems.

In particular, when $s\rightarrow 1$, problem (1.3) becomes the form:

$$\begin{aligned} \left\{ \begin{array}{lll} -\Delta u=\mu _{1} u+|u|^{2^{*}_{s}-2}u+\eta _{1}|u|^{p-2}u+\gamma \alpha |u|^{\alpha -2}u|v|^{\beta } ~ \text {in}~ {\mathbb {R}}^{N},\\ -\Delta v=\mu _{2} v+|v|^{2^{*}_{s}-2}v+\eta _{2}|v|^{q-2}v+\gamma \beta |u|^{\alpha }|v|^{\beta -2}v ~~\text {in}~ {\mathbb {R}}^{N},\\ \Vert u\Vert ^{2}_{L^{2}}=m_{1}^{2} ~\text {and}~ \Vert v\Vert ^{2}_{L^{2}}=m_{2}^{2}. \end{array}\right. \end{aligned}$$

(1.4)

Liu and Fang [10] studied the existence and nonexistence of positive normalized solution for equation (1.4) in the case of $p,q,\alpha +\beta <2^{*}_{s}$ and $p=q=\alpha +\beta =2^{*}_{s}$ respectively.

Furthermore, if let $N=4$, $p=q$, $\alpha =\beta =2$, the above problem reduces to the classical elliptic system:

$$\begin{aligned} \left\{ \begin{array}{lll} -\Delta u=\mu _{1} u+u^{3}+\eta _{1}|u|^{p-2}u+2\gamma uv^{2} ~~~\text {in}~ {\mathbb {R}}^{4},\\ -\Delta v=\mu _{2} v+v^{3}+\eta _{2}|v|^{p-2}v+2\gamma \beta u^{2}v ~~\text {in}~ {\mathbb {R}}^{4},\\ \Vert u\Vert ^{2}_{L^{2}}=m_{1}^{2} ~\text {and}~ \Vert v\Vert ^{2}_{L^{2}}=m_{2}^{2}, \end{array}\right. \end{aligned}$$

(1.5)

which was investigated by Zou et al. [11], they obtained the existence, nonexistence and asymptotic behavior of normalized ground state solutions for system (1.5) in different cases.

Of course, a great deal of work has focused on the normalized solution of integer-order nonlinear Schrödinger systems [12,13,14,15,16,17,18] or fractional-order single Schrödinger equations [19,20,21,22,23,24,25,26]. In particular, we highlight that Jeanjean and Lu [21] obtained the existence and multiplicity of normalized solutions and the asymptotic behavior of the ground state solution for a class of mass supercritical problems. The method developed in the present paper is inspired by the techniques introduced in [21]. We also point out that Soave [27] established existence and stability properties of ground state solutions for a Schrödinger equation with Sobolev critical exponent under three different assumptions, respectively mass subcritical, mass critical and mass supercritical.

So far, we have found only one paper [28] dealing with normalized solution of fractional Schrödinger coupled systems but only the subcritical case is considered. Hence it is natural to inquire what difficulties will appear if we consider single critical nonlinearity or even double critical nonlinearities.

Motivated by the work above, we will consider the existence for single critical fractional Schrödinger coupled systems and the nonexistence of normalized solutions for double critical fractional Schrödinger coupled systems. The main features of this paper is the existence of nonlocal operator and Sobolev critical nonlinearities, which makes dealing with compactness conditions more complicated. Our main methods and tools for solving these problems are scaling transformation, classification discussion and concentration-compactness principle.

A classical method for studying the normalized solution of system (1.3) is to look for critical points of the following $C^{1}$ functional

$$\begin{aligned} I(u,v):= & {} \frac{1}{2}\left( [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}\right) -\frac{1}{2_{s}^{*}}\left( \Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}+\Vert v\Vert _{2_{s}^{*}}^{2_{s}^{*}}\right) -\frac{\eta _{1}}{p}\Vert u\Vert _{p}^{p}\\{} & {} -\frac{\eta _{2}}{q}\Vert v\Vert _{q}^{q}-\gamma \int _{{\mathbb {R}}^{N}}|u|^{\alpha }|v|^{\beta } \end{aligned}$$

constrained to the set

$$\begin{aligned} S_{m_{1}}\times S_{m_{2}}=\left\{ (u,v)\in H^{s}({\mathbb {R}}^{N})\times H^{s}({\mathbb {R}}^{N}):~\Vert u\Vert ^{2}_{2}=m_{1}^{2}, ~ \Vert v\Vert ^{2}_{2}=m_{2}^{2}\right\} , \end{aligned}$$

where $H^{s}({\mathbb {R}}^{N})$ is the fractional Sobolev space defined by

$$\begin{aligned} H^{s}({\mathbb {R}}^{N})=\left\{ u\in L^{2}({\mathbb {R}}^{N})~\big |~ [u]^{2}_{H^{s}}=\int _{{\mathbb {R}}^{2N}}\frac{u(x)-u(y)}{|x-y|^{\frac{N}{2}+s}}dxdy<\infty \right\} , \end{aligned}$$

whose norm is

$$\begin{aligned} \Vert u\Vert =(\Vert u\Vert ^{2}_{L^{2}}+[u]^{2}_{H^{s}})^{\frac{1}{2}}. \end{aligned}$$

For convenience, we use $\Vert u\Vert _{p}$ to represent the norm of Lebesgue space $L^p({\mathbb {R}}^{N})$ for $p\in [1,\infty )$. We write

$$\begin{aligned}{} & {} \theta =\alpha +\beta , ~~L^{2}\text {-critical exponent}~{\overline{p}}=2+4s/N, \\{} & {} H_{rad}^{s}({\mathbb {R}}^{N})=\{u\in H^{s}({\mathbb {R}}^{N}): u(x)=u(|x|)\},~~S_{m}=\left\{ u\in H^{s}({\mathbb {R}}^{N}):\Vert u\Vert ^{2}_{2}=m^{2}\right\} , \\{} & {} S_{m_{1},r}=S_{m_{1}}\cap H_{rad}^{s}({\mathbb {R}}^{N}),~~~~S_{m_{2},r}=S_{m_{2}}\cap H_{rad}^{s}({\mathbb {R}}^{N}), \\{} & {} W_{m,r}=S_{m_{1},r}\times S_{m_{2},r},~~~~~~W_{r}=H_{rad}^{s}({\mathbb {R}}^{N})\times H_{rad}^{s}({\mathbb {R}}^{N}). \end{aligned}$$

We now present our main results:

Theorem 1.1

Assume that $N>2s$, $\alpha>1, \beta >1$, $p, q, \theta \in ({\overline{p}},2^{*}_{s})$, and $\eta _{1}, \eta _{2}, \gamma , m_{1}, m_{2}>0$. Then there exists $\gamma ^{*}=\gamma ^{*}(m_{1},m_{2})>0$ such that for any $\gamma \ge \gamma ^{*}$, problem (1.3) admits a radial normalized solution $({\widetilde{u}},{\widetilde{v}})$. Moreover, $({\widetilde{u}},{\widetilde{v}})$ is a positive solution whose associated Lagrange multipliers $\mu _{1}$ and $\mu _{2}$ are negative.

Remark 1.1

The proof of Theorem 1.1 faces some difficulties and challenges. Firstly, strong convergence of sequences in $L^{2}({\mathbb {R}}^{N})$ space is difficult to obtain because the embeddings $H^{s}({\mathbb {R}}^{N})\hookrightarrow L^{2}({\mathbb {R}}^{N})$ and $H_{rad}^{s}({\mathbb {R}}^{N})=\{u\in H^{s}({\mathbb {R}}^{N}): u(x)=u(|x|)\}\hookrightarrow L^{2}({\mathbb {R}}^{N})$ are not compact. Secondly, the lack of compactness caused by Sobolev critical index makes verifying the Palais-Smale condition more complicated. Thirdly, the idea of classification discussion is going to be used since we don’t infer which of the three indices $p,q,\theta$ is big and which is small.

Remark 1.2

This result extends the partial result in [29] in some aspects. Compared with the local case of our result, this kind of system is studied by Mederski and Schino [29] under more generalized assumptions on the nonlinear terms.

Theorem 1.2

Assume that $N>2s$, $\alpha>1, \beta >1$, $p=q=\theta =2^{*}_{s}$, and $\eta _{1}, \eta _{2}, \gamma , m_{1}, m_{2}>0$. Then problem (1.3) has no positive normalized solution.

Remark 1.3

Theorems 1.1 and 1.2 seem to be the first results of the normalized solution for fractional Sobolev critical Schrödinger coupling systems.

The paper is organized as follows. Section 2 introduces relative results of scalar equations and some preliminaries, which play an important role in the proof of Palais-Smale condition. Section 3 proves Theorem 1.1 by using the methods of scaling transformation, classification discussion and concentration-compactness principle. Section 4 gives the proof of Theorem 1.2 with the help of Pohozaev identity.

2 Relevant results for scalar equations and preliminaries

   In order to study the fractional critical Schrodinger coupling system, we first need to review related results of following scalar equations, i.e. $\gamma =0$ in (1.3):

$$\begin{aligned} \left\{ \begin{array}{lll} (-\Delta )^{s} u=\mu _{1} u+|u|^{2^{*}_{s}-2}u+\eta _{1}|u|^{p-2}u~~ in~ {\mathbb {R}}^{N}, \\ u\in H^{s}({\mathbb {R}}^{N}),~\Vert u\Vert ^{2}_{2}=m_{1}^{2}, \end{array}\right. \end{aligned}$$

(2.1)

which has been investigated in [25] by constraining on the Pohozaev manifold

$$\begin{aligned} {\mathcal {P}}_{m_{1},\eta _{1}}=\{u\in S_{m_{1}}:~[u]^{2}_{H^{s}}-\Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}-\eta _{1}\xi _{p}\Vert u\Vert _{p}^{p}=0\}, \end{aligned}$$

where

$$\begin{aligned} \xi _{p}=\frac{Np-2N}{2ps}, \ \hbox {for any} \ p\in (2,2_{s}^{*}]. \end{aligned}$$

(2.2)

A standard way to get normalized solutions of (2.1) is to look for critical points for $C^{1}$ functional

$$\begin{aligned} I_{\eta _{1}}(u)|_{S_{m_{1}}}=\frac{1}{2}[u]^{2}_{H^{s}}-\frac{1}{2_{s}^{*}}\Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}-\frac{\eta _{1}}{p}\Vert u\Vert _{p}^{p}, \end{aligned}$$

As we all know, ${\mathcal {P}}_{m_{1},\eta _{1}}$ contains every critical point of $I_{\eta _{1}}(u)|_{S_{m_{1}}}$, due to the Pohozaev identity (see [30, Proposition 4.1]).

It follows from [31] that there exists a best fractional critical Sobolev constant ${\mathcal {S}}>0$ such that

$$\begin{aligned} {\mathcal {S}}\Vert u\Vert _{2_{s}^{*}}^{2}\le [u]^{2}_{H^{s}}, \ \hbox {for all} \ u\in S_{m_{1}}, \end{aligned}$$

(2.3)

which is famous fractional Sobolev inequality.

In order to prove our result in Sect. 3, we need to obtain the following monotonicity result of scalar equations, which is necessary in the proof of Lemma 3.7.

Lemma 2.1

Assume that $N>2s$, $m_{1},\eta _{1}>0$ and $p\in ({\overline{p}},2_{s}^{*})$. Then, $m_{1}\mapsto E_{m_{1}}^{\eta _{1}}=\inf \limits _{{\mathcal {P}}_{m_{1},\eta _{1}}}I_{\eta _{1}}(u)\in (0,\frac{s{\mathcal {S}}^{\frac{N}{2s}}}{N})$ is non-increasing in $(0,+\infty )$. where ${\mathcal {S}}$ is given in (2.3).

Proof

According to [25, Theorem 1.3], we obtain that $E_{m_{1}}^{\eta _{1}}\in (0,\frac{s{\mathcal {S}}^{\frac{N}{2s}}}{N})$, Similar to the proof of [19, Lemma 9] or [10, Lemma 2.1], we only need to make a small change to get non-increasing property of the function $E_{m_{1}}^{\eta _{1}}$ with respect to $m_{1}$, so we omit it. $\square$

Lemma 2.2

[22, Lemma 1.1] There exists an optimal constant C(N, p, s) such that for $p\in (2,2_{s}^{*})$,

$$\begin{aligned} \Vert u\Vert _{p}^{p}\le C^{p}(N,p,s)[u]^{p\xi _{p}}_{H^{s}}\Vert u\Vert _{2}^{p(1-\xi _{p})},~~\forall u\in H^{s}({\mathbb {R}}^{N}), \end{aligned}$$

(2.4)

where $\xi _{p}$ is defined by (2.2).

The above fractional Gagliardo-Nirenberg inequality plays an key role in the next series of proofs.

Lemma 2.3

Let $N>2s$, $\alpha>1, \beta >1$, $\eta _{1}, \eta _{2}, \gamma >0$, $p, q, \theta \in (2,2^{*}_{s})$, and $(u,v)\in W_{r}$ is a nonnegative solution of system (1.3). Then, it follows from $u\gvertneqq 0$ that $\mu _{1}<0$; it follows from $v\gvertneqq 0$ that $\mu _{2}<0$.

Proof

On account of $u\gvertneqq 0$ and fulfills

$$\begin{aligned} (-\Delta )^{s} u=\mu _{1} u+|u|^{2^{*}_{s}-2}u+\eta _{1}|u|^{p-2}u+\gamma \alpha |u|^{\alpha -2}u|v|^{\beta } ~ \text {in}~ {\mathbb {R}}^{N}, \end{aligned}$$

thus we infer that $(-\Delta )^{s} u\ge 0$ if $\mu _{1}\ge 0$. It follows from [32, Proposition 2.17] that $u\equiv 0$. This contradicts to the condition $u\gvertneqq 0$, which means that $\mu _{1}<0$. Similarly, we also can get $\mu _{2}<0$ from $v\gvertneqq 0$. $\square$

Lemma 2.4

Let $N>2s$, $\alpha>1, \beta >1$, $\theta \in [2,2^{*}_{s}]$, $(u_{n},v_{n})\rightharpoonup (u,v)$ in $W_{r}$. Then, up to a subsequence

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{\alpha }|v_{n}|^{\beta }-|u_{n}-u|^{\alpha }|v_{n}-v|^{\beta }-|u|^{\alpha }|v|^{\beta }=0. \end{aligned}$$

Remark 2.1

Since the proof of Brézis-Lieb Lemma 2.4 is standard and classical, we would like to omit it, please refer to the literature [5, Lemma 2.3] for interested readers.

3 Proof of theorem 1.1

We will do a scaling transformation make the functional $I(\rho \star u, \rho \star v)$ satisfy the mountain pass geometry. For $(u,v)\in W$ and $\rho \in {\mathbb {R}}$, we let

$$\begin{aligned} (\rho \star u,\rho \star v) =\left( e^{\frac{N\rho }{2}}u(e^{\rho }x),e^{\frac{N\rho }{2}}v(e^{\rho }x)\right) ~~\text {for~a.e.~} x\in {\mathbb {R}}^{N}, \end{aligned}$$

which comes from the inspiration of Jeanjean [33]. The results show that the original functional I(u, v) and the transformed functional ${\widetilde{I}}=I(\rho \star u, \rho \star v)$ have the same mountain pass geometry and mountain pass level.

Lemma 3.1

Suppose that $(u,v)\in S_{m_{1}}\times S_{m_{2}}$ is arbitrary but fixed. Then we have the following conclusions:

• (1)   $[\rho \star u]^{2}_{H^{s}}+[\rho \star v]^{2}_{H^{s}}\rightarrow 0$ and  $I(\rho \star u, \rho \star v)\rightarrow 0$  as  $\rho \rightarrow -\infty ;$ 

• (2)   $[\rho \star u]^{2}_{H^{s}}+[\rho \star v]^{2}_{H^{s}}\rightarrow +\infty$ and  $I(\rho \star u, \rho \star v)\rightarrow -\infty$  as  $\rho \rightarrow +\infty .$ 

Proof

Through simple calculations, we have

$$\begin{aligned}{}[\rho \star u]^{2}_{H^{s}}+[\rho \star v]^{2}_{H^{s}}= & {} e^{2\rho s}\int \int _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{2}}{|x-y|^{N+2s}}dxdy\nonumber \\{} & {} +e^{2\rho s}\int \int _{{\mathbb {R}}^{2N}}\frac{|v(x)-v(y)|^{2}}{|x-y|^{N+2s}}dxdy \nonumber \\ {}= & {} e^{2\rho s}[u]^{2}_{H^{s}}+e^{2\rho s}[v]^{2}_{H^{s}}, \end{aligned}$$

(3.1)

$$\begin{aligned} \Vert \rho \star u\Vert _{\zeta }^{\zeta }= & {} e^{\frac{(\zeta -2)N\rho }{2}}\Vert u\Vert _{\zeta }^{\zeta },\nonumber \\ \Vert \rho \star v\Vert _{\zeta }^{\zeta }= & {} e^{\frac{(\zeta -2)N\rho }{2}}\Vert v\Vert _{\zeta }^{\zeta }\ , \end{aligned}$$

(3.2)

$$\begin{aligned} \gamma \int _{{\mathbb {R}}^{N}}|\rho \star u|^{\alpha }|\rho \star v|^{\beta }= & {} \gamma e^{\frac{(\theta -2)N\rho }{2}}\int _{{\mathbb {R}}^{N}}| u|^{\alpha }| v|^{\beta }. \end{aligned}$$

(3.3)

From (3.1)–(3.3), $\zeta \ge 2$, and $\theta >2$, we get

$$\begin{aligned}{}[\rho \star u]^{2}_{H^{s}}\rightarrow 0,~~ [\rho \star v]^{2}_{H^{s}}\rightarrow 0,~~\hbox {as}~~\rho \rightarrow -\infty , \\ \Vert \rho \star u\Vert _{\zeta }^{\zeta }\rightarrow 0, ~\Vert \rho \star v\Vert _{\zeta }^{\zeta }\rightarrow 0~~\hbox {as}~~\rho \rightarrow -\infty , \\ \gamma \int _{{\mathbb {R}}^{N}}|\rho \star u|^{\alpha }|\rho \star v|^{\beta }\rightarrow 0~~\hbox {as}~~\rho \rightarrow -\infty . \end{aligned}$$

Thus, we have

$$\begin{aligned} \begin{aligned} I(\rho \star u, \rho \star v)=&\frac{1}{2}\left( [\rho \star u]^{2}_{H^{s}}+[\rho \star v]^{2}_{H^{s}}\right) -\frac{1}{2_{s}^{*}}\left( \Vert \rho \star u\Vert _{2_{s}^{*}}^{2_{s}^{*}}+\Vert \rho \star v\Vert _{2_{s}^{*}}^{2_{s}^{*}}\right) \\&-\frac{\eta _{1}}{p}\Vert \rho \star u\Vert _{p}^{p}-\frac{\eta _{2}}{q}\Vert \rho \star v\Vert _{q}^{q}-\gamma \int _{{\mathbb {R}}^{N}}|\rho \star u|^{\alpha }|\rho \star v|^{\beta }\rightarrow 0~~\hbox {as}~~\rho \rightarrow -\infty , \end{aligned} \end{aligned}$$

which implies that (1) holds.

Again by (3.1), we obtain that $[\rho \star u]^{2}_{H^{s}}+[\rho \star v]^{2}_{H^{s}}\rightarrow +\infty$   as  $\rho \rightarrow +\infty .$ Furthermore,

$$\begin{aligned} \begin{aligned} I(\rho \star u, \rho \star v)=&\frac{1}{2}\left( [\rho \star u]^{2}_{H^{s}}+[\rho \star v]^{2}_{H^{s}}\right) -\frac{1}{2_{s}^{*}}\left( \Vert \rho \star u\Vert _{2_{s}^{*}}^{2_{s}^{*}}+\Vert \rho \star v\Vert _{2_{s}^{*}}^{2_{s}^{*}}\right) \\&-\frac{\eta _{1}}{p}\Vert \rho \star u\Vert _{p}^{p}-\frac{\eta _{2}}{q}\Vert \rho \star v\Vert _{q}^{q}-\gamma \int _{{\mathbb {R}}^{N}}|\rho \star u|^{\alpha }|\rho \star v|^{\beta }\rightarrow -\infty ~~\hbox {as}~~\rho \rightarrow +\infty \end{aligned} \end{aligned}$$

since $p,q,\theta \in (2+\frac{4s}{N},2^{*}_{s})$, which also means that (2) holds. $\square$

Lemma 3.2

There exists $A(m_{1},m_{2})>0$ small enough such that

$$\begin{aligned} 0<\sup \limits _{u\in X} I(u,v)<\inf \limits _{u\in Y} I(u,v), \end{aligned}$$

with

$$\begin{aligned} X:=\left\{ (u,v)\in S_{m_{1}}\times S_{m_{2}}:~ [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}\le A(m_{1},m_{2})\right\} , \\ Y:=\left\{ (u,v)\in S_{m_{1}}\times S_{m_{2}}:~ [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}=2A(m_{1},m_{2})\right\} . \end{aligned}$$

Proof

According to Lemma 2.2 and the Hölder inequality, for any $(u,v)\in S_{m_{1}}\times S_{m_{2}}$ we get that

$$\begin{aligned}{} & {} \frac{\eta _{1}}{p}\Vert u\Vert _{p}^{p}\le C^{p}(N,p,s,m_{1},\eta _{1})[u]^{p\xi _{p}}_{H^{s}}\le C^{p}(N,p,s,m_{1},\eta _{1})\left( [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}\right) ^{\frac{p\xi _{p}}{2}}, \end{aligned}$$

(3.4)

$$\begin{aligned}{} & {} \frac{\eta _{2}}{q}\Vert u\Vert _{q}^{q}\le C^{q}(N,q,s,m_{2},\eta _{2})[u]^{q\xi _{q}}_{H^{s}}\le C^{q}(N,q,s,m_{2},\eta _{2})\left( [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}\right) ^{\frac{q\xi _{q}}{2}}, \end{aligned}$$

(3.5)

$$\begin{aligned}{} & {} \gamma \int _{{\mathbb {R}}^{N}}|u|^{\alpha }|v|^{\beta }\le \gamma \Vert u\Vert _{\theta }^{\alpha }\Vert v\Vert _{\theta }^{\beta }\le \gamma C(N,\alpha ,\beta , s,m_{1},m_{2})\left( [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}\right) ^{\frac{\theta \xi _{\theta }}{2}}.\nonumber \\ \end{aligned}$$

(3.6)

On the one hand, if let $b=[u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}$ and $A>0$ be arbitrary but fixed, then for any $(u,v)\in X$ such that $b\le A$, it follows from (3.4)–(3.6) and (2.3) that

$$\begin{aligned} \begin{aligned} I(u,v)&\ge \frac{1}{2}b-\frac{1}{2_{s}^{*}{\mathcal {S}}^{\frac{2_{s}^{*}}{2}}}b^{\frac{2_{s}^{*}}{2}}-C^{p}(N,p,s,m_{1},\eta _{1})b^{\frac{p\xi _{p}}{2}} \\ {}&\quad -C^{q}(N,q,s,m_{2},\eta _{2})b^{\frac{q\xi _{q}}{2}}-\gamma C(N,\alpha ,\beta , s,m_{1},m_{2})b^{\frac{\theta \xi _{\theta }}{2}}\\&\ge \frac{1}{8}b>0 \end{aligned} \end{aligned}$$

for A small enough, the scaling of inequality above takes advantage of this fact that $p\xi _{p}, q\xi _{q}, \theta \xi _{\theta }>2$. On the other hand, for any $(u_{1},v_{1})\in Y$ and $(u_{2},v_{2})\in X$ such that $b_{1}=[u_{1}]^{2}_{H^{s}}+[v_{1}]^{2}_{H^{s}}=2A$ and $b_{2}=[u_{2}]^{2}_{H^{s}}+[v_{2}]^{2}_{H^{s}}\le A$, we obtain

$$\begin{aligned} \begin{aligned} I(u_{1},v_{1})-I(u_{2},v_{2})&\ge \frac{1}{2}(b_{1}-b_{2})-\frac{1}{2_{s}^{*}{\mathcal {S}}^{\frac{2_{s}^{*}}{2}}}b_{1}^{\frac{2_{s}^{*}}{2}}-C^{p}(N,p,s,m_{1},\eta _{1})b_{1}^{\frac{p\xi _{p}}{2}} \\ {}&\quad -C^{q}(N,q,s,m_{2},\eta _{2})b_{1}^{\frac{q\xi _{q}}{2}}-\gamma C(N,\alpha ,\beta , s,m_{1},m_{2})b_{1}^{\frac{\theta \xi _{\theta }}{2}}\\&\ge \frac{1}{2}A-\frac{1}{2_{s}^{*}{\mathcal {S}}^{\frac{2_{s}^{*}}{2}}}(2A)^{\frac{2_{s}^{*}}{2}}-C^{p}(N,p,s,m_{1},\eta _{1})(2A)^{\frac{p\xi _{p}}{2}} \\ {}&\quad -C^{q}(N,q,s,m_{2},\eta _{2})(2A)^{\frac{q\xi _{q}}{2}}-\gamma C(N,\alpha ,\beta , s,m_{1},m_{2})(2A)^{\frac{\theta \xi _{\theta }}{2}}\\&\ge \frac{1}{8}b \end{aligned} \end{aligned}$$

for A sufficiently small. So, we can pick A small enough for the inequality in Lemma 3.2 to be true. $\square$

Now we have obtained that the geometry of mountain pass, then we give the minimax picture: Because Zhang [25] had proved the ${\overline{u}}:=u_{m_{1},\eta _{1}}$ is a ground state of (2.1) involving parameters $p,\eta _{1},m_{1}$ and also ${\overline{v}}:=u_{m_{2},\eta _{2}}$ is a ground state of (2.1) involving parameters $p,\eta _{2},m_{2}$. Therefore, we fix $({\overline{u}},{\overline{v}})\in W_{m,r}$, according to Lemma 3.1 and Lemma 3.2, there exist two numbers $\rho _{1}\ll -1<0<1\ll \rho _{2}$ such that

$$\begin{aligned} e^{2\rho _{1} s}[{\overline{u}}]^{2}_{H^{s}}+e^{2\rho _{1} s}[{\overline{v}}]^{2}_{H^{s}}<\frac{A(m_{1},m_{2})}{2}, ~~~I(\rho _{1}\star {\overline{u}}, \rho _{1}\star {\overline{v}})>0, \\ e^{2\rho _{2} s}[{\overline{u}}]^{2}_{H^{s}}+e^{2\rho _{2} s}[{\overline{v}}]^{2}_{H^{s}}>2A(m_{1},m_{2}),~~~ I(\rho _{2}\star {\overline{u}}, \rho _{2}\star {\overline{v}})\le 0. \end{aligned}$$

We define the path

$$\begin{aligned} \Gamma =\{\chi \in C([0,1], W_{m,r}): \chi (0)=(\rho _{1}\star {\overline{u}}, \rho _{1}\star {\overline{v}}), ~ \chi (1)=(\rho _{2}\star {\overline{u}}, \rho _{2}\star {\overline{v}})\}, \end{aligned}$$

then $\Gamma$ is not empty. In fact, let $\chi _{0}(t):=([(1-t)\rho _{1}+t\rho _{2}]\star {\overline{u}}, [(1-t)\rho _{1}+t\rho _{2}]\star {\overline{v}}),$ obviously, $\chi _{0}(t)\in W_{m,r}$ and $\chi _{0}(t)\in \Gamma$.

Letting

$$\begin{aligned} c_{\gamma }(m_{1},m_{2}):=\inf \limits _{\chi \in \Gamma }\max \limits _{t\in (0,1]}I(\chi (t)), \end{aligned}$$

clearly, $c_{\gamma }(m_{1},m_{2})>0$, then we have the following asymptotic behavior of critical values:

Lemma 3.3

$\lim \limits _{\gamma \rightarrow +\infty }c_{\gamma }(m_{1},m_{2})=0$.

Proof

Fix $(u_{0},v_{0})\in W_{m,r}$ and it follows from the path $\chi _{0}(t):=([(1-t)\rho _{1}+t\rho _{2}]\star u_{0}, [(1-t)\rho _{1}+t\rho _{2}]\star v_{0})$ that

$$\begin{aligned} c_{\gamma }(m_{1},m_{2})\le & {} \max \limits _{t\in [0,1]}I(\chi _{0}(t))\\\le & {} \max \limits _{l\ge 0}\left\{ \frac{1}{2}l^{2}\left( [u_{0}]^{2}_{H^{s}}+[v_{0}]^{2}_{H^{s}}\right) -\gamma l^{\frac{N\theta -2N}{2s}}\int _{{\mathbb {R}}^{N}}|u_{0}|^{\alpha }|v_{0}|^{\beta }\right\} . \end{aligned}$$

Let $C_{1}=[u_{0}]^{2}_{H^{s}}+[v_{0}]^{2}_{H^{s}}$ and $C_{2}=\int _{{\mathbb {R}}^{N}}|u_{0}|^{\alpha }|v_{0}|^{\beta }$, we discuss the maximum value of the following function

$$\begin{aligned} g(l)=\frac{1}{2}C_{1}l^{2}-\gamma C_{2}l^{\frac{N\theta -2N}{2s}}, ~\text {for~any}~l\ge 0. \end{aligned}$$

Letting

$$\begin{aligned} g^{'}(l)=C_{1}l-\left( \frac{N\theta -2N}{2s}\right) \gamma C_{2}l^{\frac{N\theta -2N-2s}{2s}}=0, \end{aligned}$$

we can obtain the maximum point of g(l), that is

$$\begin{aligned} l_{max}=\left( \frac{2sC_{1}}{(N\theta -2N)\gamma C_{2}}\right) ^{\frac{2s}{N\theta -2N-4s}}. \end{aligned}$$

Thus,

$$\begin{aligned}{} & {} \max \limits _{l\ge 0}\left\{ \frac{1}{2}l^{2}\left( [u_{0}]^{2}_{H^{s}}+[v_{0}]^{2}_{H^{s}} \right) -\gamma l^{\frac{N\theta -2N}{2s}}\int _{{\mathbb {R}}^{N}}|u_{0}|^{\alpha }|v_{0}|^{\beta }\right\} \\{} & {} \quad =\frac{1}{2}\left( \frac{2sC_{1}}{(N\theta -2N)\gamma C_{2}}\right) ^{\frac{4s}{N\theta -2N-4s}}C_{1}-\gamma \left( \frac{2sC_{1}}{(N\theta -2N)\gamma C_{2}}\right) ^{\frac{N\theta -2N}{N\theta -2N-4s}}C_{2} \\{} & {} \quad \le \frac{1}{2}\left( \frac{2sC_{1}}{(N\theta -2N)\gamma C_{2}}\right) ^{\frac{4s}{N\theta -2N-4s}}C_{1}. \end{aligned}$$

As a result, there exists $C>0$ that don’t depend on $\gamma >0$ such that

$$\begin{aligned} c_{\gamma }(m_{1},m_{2})\le C\left( \frac{1}{\gamma }\right) ^{\frac{4s}{N\theta -2N-4s}}\rightarrow 0 ~~\text {as}~ \gamma \rightarrow \infty \end{aligned}$$

thanks to $\theta >2+4s/N$. We complete the proof of Lemma 3.3 now. $\square$

To further prove our main results, we need to define Pohozaev manifold ${\mathcal {P}}_{m_{1},m_{2},\mu _{1},\mu _{2}}$ of vector equations (1.3) in the same way as scalar equations (2.1).

$$\begin{aligned} {\mathcal {P}}_{m_{1},m_{2},\mu _{1},\mu _{2}}=\left\{ (u,v)\in S_{m_{1}}\times S_{m_{2}}:~P(u,v)=0\right\} , \end{aligned}$$

where

$$\begin{aligned} P(u,v):= & {} [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}-\Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}-\Vert v\Vert _{2_{s}^{*}}^{2_{s}^{*}} -\eta _{1}\xi _{p}\Vert u\Vert _{p}^{p}\\{} & {} -\eta _{2}\xi _{q}\Vert v\Vert _{q}^{q}-\gamma \theta \xi _{\theta }\int _{{\mathbb {R}}^{N}}|u|^{\alpha }|v|^{\beta }. \end{aligned}$$

Notice that $I(|u|,|v|)=I(u,v)$ and $P(|u|,|v|)=P(u,v)$ for any $(u,v)\in W_{m,r}$. Similar to the argument in [33, Proposition 2.2] with minor changes, replace $(0, \chi _{n})\in {\overline{\Gamma }}$ with $(0, |\chi _{n}|)\in {\overline{\Gamma }}$ if necessary, where

$$\begin{aligned} {\overline{\Gamma }}:=\left\{ {\overline{\chi }}\in ([0,1], W_{m,r}): {\overline{\chi }}(0)=(0,(\rho _{1}\star {\overline{u}}, \rho _{1}\star {\overline{v}})), ~ {\overline{\chi }}(1)=(0,(\rho _{2}\star {\overline{u}}, \rho _{2}\star {\overline{v}}))\right\} . \end{aligned}$$

Then $\{(u_{n},v_{n})\}\subset W_{m,r}$ is a Palais-Smale sequence for I(u, v) at level $c_{\gamma }(m_{1},m_{2})$, i.e.

$$\begin{aligned} I(u_{n},v_{n}))\rightarrow c_{\gamma }(m_{1},m_{2}) ~~\text {as} ~n\rightarrow \infty \ \hbox {and} \ I^{'}(u_{n},v_{n})\rightarrow 0~~\text {as} ~n\rightarrow \infty . \end{aligned}$$

Moreover,

$$\begin{aligned} u_{n}^{-}\rightarrow 0,~~ v_{n}^{-}\rightarrow 0 ~~ \hbox {a.e. ~in}~ {\mathbb {R}}^{N}~~ \hbox {as}~ n\rightarrow \infty . \end{aligned}$$

Lemma 3.4

Suppose that $\{(u_{n},v_{n})\}\subset W_{m,r}$ is a Palais-Smale sequence for I(u, v). Then $\lim \limits _{n\rightarrow \infty }P(u_{n},v_{n})=0.$

Proof

Let

$$\begin{aligned} (\rho _{n}\star u_{n}, \rho _{n}\star v_{n})=(x_{n},y_{n}). \end{aligned}$$

By direct calculations, we claim that

$$\begin{aligned} \frac{d}{d\rho _{n}}I((-\rho _{n})\star x_{n}, (-\rho _{n})\star y_{n})=-sP((-\rho _{n})\star x_{n}, (-\rho _{n})\star y_{n})=-sP(u_{n},v_{n}). \end{aligned}$$

Since $I^{'}(u_{n},v_{n})\rightarrow 0$ as $n\rightarrow \infty$, again from [22, Proposition 5.4(2)], we get that $\lim \limits _{n\rightarrow \infty }P(u_{n},v_{n})=0.$ $\square$

Lemma 3.5

If $\{(u_{n},v_{n})\}\subset W_{m,r}$ is a Palais-Smale sequence for I(u, v), then $\{(u_{n},v_{n})\}$ is bounded in $W_{m,r}$.

Proof

Observing that $\xi _{p}p, \xi _{q}q, \xi _{\theta }\theta >2$ due to the fact that $p,q,\theta >2+4s/N$. It follows from Lemma 3.4 that

$$\begin{aligned} I(u_{n},v_{n})= & {} \frac{\eta _{1}}{2p}(\xi _{p}p-2)\Vert u_{n}\Vert _{p}^{p}+\frac{\eta _{2}}{2q}(\xi _{q}q-2)\Vert v_{n}\Vert _{q}^{q}+\frac{\gamma }{2}(\xi _{\theta }\theta -2)\int _{{\mathbb {R}}^{N}}|u|^{\alpha }|v|^{\beta }\nonumber \\{} & {} +\frac{s}{N}\Vert u_{n}\Vert _{2_{s}^{*}}^{2_{s}^{*}}+\frac{s}{N}\Vert v_{n}\Vert _{2_{s}^{*}}^{2_{s}^{*}}+o(1). \end{aligned}$$

Since $I(u_{n},v_{n})$ is bounded, thus we conclude that sequences $\{\Vert u_{n}\Vert _{p}^{p}\}$, $\{\Vert v_{n}\Vert _{q}^{q}\}$, $\left\{ \int _{{\mathbb {R}}^{N}}|u_{n}|^{\alpha }|v_{n}|^{\beta }\right\}$, $\{\Vert u_{n}\Vert _{2_{s}^{*}}^{2_{s}^{*}}\}$, and $\{\Vert v_{n}\Vert _{2_{s}^{*}}^{2_{s}^{*}}\}$ are all bounded. Again by Lemma 3.4, we infer that $\{[u_{n}]^{2}_{H^{s}}\}$ and $\{[v_{n}]^{2}_{H^{s}}\}$ are also bounded, as claimed. $\square$

In view of Lemma 3.5, there exists a nonnegative $({\widetilde{u}}, {\widetilde{v}})\in W_{r}$ such that, up to a subsequence,

$$\begin{aligned} \begin{aligned} (u_{n},v_{n})\rightharpoonup&({\widetilde{u}}, {\widetilde{v}})~ \hbox {in}~W_{r}; \\(u_{n},v_{n})\rightharpoonup&({\widetilde{u}}, {\widetilde{v}})~ \hbox {in}~L^{2_{s}^{*}}({\mathbb {R}}^{N})\times L^{2_{s}^{*}}({\mathbb {R}}^{N}); \\(u_{n},v_{n})\rightarrow&({\widetilde{u}}, {\widetilde{v}})~ \hbox {in}~L^{p}({\mathbb {R}}^{N})\times L^{q}({\mathbb {R}}^{N}); \\(u_{n},v_{n})\rightarrow&({\widetilde{u}}, {\widetilde{v}})~ \hbox {in}~L^{\theta }({\mathbb {R}}^{N})\times L^{\theta }({\mathbb {R}}^{N}); \\(u_{n},v_{n})\rightarrow&({\widetilde{u}}, {\widetilde{v}})~\text {a.e.}~\hbox {in}~{\mathbb {R}}^{N} \end{aligned} \end{aligned}$$

(3.7)

as $n\rightarrow \infty$. Since $\{(u_{n},v_{n})\}\subset S_{m_{1}}\times S_{m_{2}}$ is a Palais-Smale sequence for I(u, v), on basis of the Lagrange multipliers rule, there exists a sequence $\{(\mu _{1}^{n},\mu _{2}^{n})\}\subset {\mathbb {R}}\times {\mathbb {R}}$ such that

$$\begin{aligned} I^{'}(u_{n},v_{n})+\mu _{1}^{n}(u_{n},0)+\mu _{2}^{n}(0,v_{n})\rightarrow 0~~\hbox {in}~W_{r}~~\text {as} ~n\rightarrow \infty . \end{aligned}$$

(3.8)

Next, we take $(u_{n},0)$ and $(0,v_{n})$ as test functions in (3.7), it follows from the proof of [25, Proposition 2.2] that $\{(\mu _{1}^{n},\mu _{2}^{n})\}$ is bounded in ${\mathbb {R}}$. Therefore up to a subsequence $(\mu _{1}^{n},\mu _{2}^{n})\rightarrow (\mu _{1}, \mu _{1})\in {\mathbb {R}}\times {\mathbb {R}}$.

Lemma 3.6

There exists $\gamma ^{*}=\gamma ^{*}(m_{1},m_{2})>0$ big enough, such that for any $\gamma \ge \gamma ^{*}$, $(u_{n},v_{n})\rightarrow ({\widetilde{u}}, {\widetilde{v}})~ \hbox {in}~L^{2_{s}^{*}}({\mathbb {R}}^{N})\times L^{2_{s}^{*}}({\mathbb {R}}^{N})$ and ${\widetilde{u}}, {\widetilde{v}}\ne 0$.

Proof

We first claim that $u_{n}\rightarrow {\widetilde{u}}~ \hbox {in}~L^{2_{s}^{*}}({\mathbb {R}}^{N})$. In fact, according to the concentration-compactness principle in [34], we know that there exist two nonnegative measures $\omega$, $\nu$ and a (at most countable) index set J such that $|(-\Delta )^{\frac{s}{2}}u_{n}|^{2}\rightharpoonup \omega$ in ${\mathcal {M}}({\mathbb {R}}^{N})$, $|u_{n}|^{2_{s}^{*}}\rightharpoonup \nu$ in ${\mathcal {M}}({\mathbb {R}}^{N})$ and

$$\begin{aligned} \left\{ \begin{array}{lll} \omega \ge |(-\Delta )^{\frac{s}{2}}{\widetilde{u}}|^{2}+\sum \limits _{j\in J}\omega _{j}\delta _{x_{j}}, ~~\omega _{j}\ge 0, \\ \nu =|{\widetilde{u}}|^{2_{s}^{*}}+\sum \limits _{j\in J}\nu _{j}\delta _{x_{j}},~~~~~~~~~~\nu _{j}\ge 0, \\ \nu _{j}\le {\mathcal {S}}^{\frac{-2_{s}^{*}}{2}}\omega _{j}^{\frac{2_{s}^{*}}{2}},~~~~~~~~~~~~~~~~~~\forall j\in J, \end{array}\right. \end{aligned}$$

(3.9)

where $x_{j}$ is the different point in ${\mathbb {R}}^{N}$, $\delta _{x_{j}}$ denotes the Dirac measure at $x_{j}$, ${\mathcal {S}}$ is best Sobolev constant in (2.3). Furthermore, it is possible to lose mass at infinity. i.e.,

$$\begin{aligned} \left\{ \begin{array}{lll} \limsup \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u_{n}|^{2}dx=\int _{{\mathbb {R}}^{N}}d\omega +\omega _{\infty }, \\ \limsup \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{2_{s}^{*}}dx=\int _{{\mathbb {R}}^{N}}d\nu +\nu _{\infty }, \\ \nu _{\infty }\le {\mathcal {S}}^{\frac{-2_{s}^{*}}{2}}\omega _{\infty }^{\frac{2_{s}^{*}}{2}}, \end{array}\right. \end{aligned}$$

(3.10)

where

$$\begin{aligned} \omega _{\infty }= & {} \lim \limits _{R\rightarrow \infty }\limsup \limits _{n\rightarrow \infty }\int _{|x|>R}|(-\Delta )^{\frac{s}{2}}u_{n}|^{2}dx, \\ \nu _{\infty }= & {} \lim \limits _{R\rightarrow \infty }\limsup \limits _{n\rightarrow \infty }\int _{|x|>R}|u_{n}|^{2_{s}^{*}}dx. \end{aligned}$$

$Case ~1$ Take $\psi _{\epsilon }(x)\in C^{\infty }_{0}({\mathbb {R}}^{N})$ be a cut-off function such that

$$\begin{aligned} \left\{ \begin{array}{lll} \psi _{\epsilon }(x)\equiv 1 ~~\text {in}~ B_{\epsilon }(x_{j}), \\ \psi _{\epsilon }(x)\equiv 0~~\text {in}~ B_{2\epsilon }^{c}(x_{j}), \\ 0\le \psi _{\epsilon }(x)\le 1, \end{array}\right. \end{aligned}$$

(3.11)

where $B_{\epsilon }(x_{j})$ represents the small ball with radius $\epsilon$ and center $x_{j}$. By Lemma 3.5, we note that $\{\psi _{\epsilon }(x)u_{n}\}$ is bounded in $H^{s}({\mathbb {R}}^{N})$. Next, again take $\{(\psi _{\epsilon }(x)u_{n},0)\}$ as a test function in (3.8) and letting $\epsilon \rightarrow 0$, we obtain that

$$\begin{aligned} \lim \limits _{\epsilon \rightarrow 0}\lim \limits _{n\rightarrow \infty }\langle I^{'}(u_{n},v_{n})-\mu _{1}^{n}(u_{n},0)-\mu _{2}^{n}(0,v_{n}),(\psi _{\epsilon }(x)u_{n},0)\rangle =0. \end{aligned}$$

(3.12)

From (3.7), the Hölder inequality and the absolute continuity of the Lebesgue integral, we have

$$\begin{aligned} \begin{aligned}&\lim \limits _{\epsilon \rightarrow 0}\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}\mu _{1}^{n}u_{n}^{2}\psi _{\epsilon }dx=0, \\ {}&\lim \limits _{\epsilon \rightarrow 0}\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{p}\psi _{\epsilon }dx =\lim \limits _{\epsilon \rightarrow 0}\int _{{\mathbb {R}}^{N}}|{\widetilde{u}}^{p}\psi _{\epsilon }dx=0, \\ {}&\lim \limits _{\epsilon \rightarrow 0}\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{\alpha }|v_{n}|^{\beta }\psi _{\epsilon }dx= \lim \limits _{\epsilon \rightarrow 0}\int _{{\mathbb {R}}^{N}}|{\widetilde{u}}|^{\alpha }|{\widetilde{v}}|^{\beta }\psi _{\epsilon }dx=0. \end{aligned} \end{aligned}$$

(3.13)

According to (3.12) and (3.13), we deduce that

$$\begin{aligned} \lim \limits _{\epsilon \rightarrow 0}\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u_{n}|^{2}\psi _{\epsilon }dx= \lim \limits _{\epsilon \rightarrow 0}\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{2_{s}^{*}}\psi _{\epsilon }dx, \end{aligned}$$

which means that

$$\begin{aligned} \lim \limits _{\epsilon \rightarrow 0}\int _{{\mathbb {R}}^{N}}\psi _{\epsilon }d\omega =\lim \limits _{\epsilon \rightarrow 0}\int _{{\mathbb {R}}^{N}}\psi _{\epsilon }d\nu . \end{aligned}$$

(3.14)

Therefore it follows from (3.9) and (3.14) that $\nu _{j}\ge \omega _{j}$, thereby

$$\begin{aligned} \text {either}~~\omega _{j}=0~~\text {or}~~\omega _{j}\ge {\mathcal {S}}^{\frac{N}{2s}}~~\text {for}~~j\in J, \end{aligned}$$

(3.15)

which means that J is a finite set.

$Case ~2$ Take $\varphi \in C^{\infty }_{0}({\mathbb {R}}^{N})$ be another cut-off function with

$$\begin{aligned} \left\{ \begin{array}{lll} 0\le \varphi \le 1, \\ \varphi \equiv 0 ~~\text {in}~ B_{\frac{1}{2}}(0), \\ \varphi \equiv 1~~\text {in}~ {\mathbb {R}}^{N}\setminus B_{1}(0). \end{array}\right. \end{aligned}$$

(3.16)

For any R, let $\varphi _{R}(x)=\varphi (\frac{x}{R}),$ by Lemma 3.5, we also have that $\{\varphi _{R}(x)u_{n}\}$ is bounded in $H^{s}({\mathbb {R}}^{N})$. Similarly, take $\{(\varphi _{R}(x)u_{n},0)\}$ as a test function in (3.8) and letting $R\rightarrow \infty$, we also obtain that

$$\begin{aligned} \lim \limits _{R\rightarrow \infty }\lim \limits _{n\rightarrow \infty }\langle I^{'}(u_{n},v_{n})-\mu _{1}^{n}(u_{n},0)-\mu _{2}^{n}(0,v_{n}),(\varphi _{R}(x)u_{n},0)\rangle =0. \end{aligned}$$

(3.17)

Similarly to the proof of [35, Lemma 3.3], we can also get

$$\begin{aligned} \begin{aligned}&\lim \limits _{R\rightarrow \infty }\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}\mu _{1}^{n}u_{n}^{2}\varphi _{R}dx=0, \\ {}&\lim \limits _{R\rightarrow \infty }\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{p}\varphi _{R}dx=\lim \limits _{R\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|{\widetilde{u}}|^{p}\varphi _{R}dx =0, \\ {}&\lim \limits _{R\rightarrow \infty }\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{\alpha }|v_{n}|^{\beta }\varphi _{R}dx= \lim \limits _{R\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|{\widetilde{u}}|^{\alpha }|{\widetilde{v}}|^{\beta }\varphi _{R}dx= 0. \end{aligned} \end{aligned}$$

(3.18)

By (3.17) and (3.18), we also have

$$\begin{aligned} \lim \limits _{R\rightarrow \infty }\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u_{n}|^{2}\varphi _{R}dx= \lim \limits _{R\rightarrow \infty }\lim \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|u_{n}|^{2_{s}^{*}}\varphi _{R}dx, \end{aligned}$$

that is

$$\begin{aligned} \omega _{\infty }=\nu _{\infty }. \end{aligned}$$

(3.19)

Again by (3.10), we have

$$\begin{aligned} \text {either}~~\omega _{\infty }=0~~\text {or}~~\omega _{\infty }\ge {\mathcal {S}}^{\frac{N}{2s}}. \end{aligned}$$

(3.20)

For the rest of the proof, we will only prove Case  1, because Case  2 and Case  1 are almost exactly the same.

If for any $j\in J$, $\omega _{j}=0$, then we have that $\nu _{j}=0$ since (3.9), thereby $|u_{n}|^{2_{s}^{*}}\rightarrow |{\widetilde{u}}|^{2_{s}^{*}}$. By the Brézis-Lieb lemma [36], we conclude that $u_{n}\rightarrow {\widetilde{u}}$ in $L^{2_{s}^{*}}({\mathbb {R}}^{N}$, as claimed.

If instead $\omega _{j}\ge {\mathcal {S}}^{\frac{N}{2s}}$ for some $j\in J$. In view of $I(u_{n},v_{n})\rightarrow c_{\gamma }(m_{1},m_{2})$, Lemma 3.4 and (3.11), we adopt the method of categorical discussion:

• (1) If $\theta =\min \{p,q,\theta \}$. It follows from Lemma 3.3 that there exists a positive constant $\gamma ^{1}$ big enough, such that $c_{\gamma }(m_{1},m_{2})<\left( \frac{1}{2}-\frac{1}{\theta \xi _{\theta }}{\mathcal {S}}^{\frac{N}{2s}}\right)$ for any $\gamma \ge \gamma ^{'}$. By (3.9), we have

  $$\begin{aligned} \begin{aligned} \left( \frac{1}{2}-\frac{1}{\theta \xi _{\theta }}\right) {\mathcal {S}}^{\frac{N}{2s}}>c_{\gamma }(m_{1},m_{2})&=\lim \limits _{n\rightarrow \infty }\left( I(u_{n},v_{n})-\frac{1}{\theta \xi _{\theta }}P(u_{n},v_{n})\right) \\ {}&\ge \left( \frac{1}{2}-\frac{1}{\theta \xi _{\theta }}\right) \limsup \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u_{n}|^{2}\psi _{\epsilon }dx \\ {}&=\left( \frac{1}{2}-\frac{1}{\theta \xi _{\theta }}\right) \int _{{\mathbb {R}}^{N}}\psi _{\epsilon }d\omega \\ {}&\ge \left( \frac{1}{2}-\frac{1}{\theta \xi _{\theta }}\right) \omega _{j}\ge \left( \frac{1}{2}-\frac{1}{\theta \xi _{\theta }}\right) {\mathcal {S}}^{\frac{N}{2s}}, \end{aligned} \end{aligned}$$

  which is a contradiction.

• (2) If $p=\min \{p,q,\theta \}$. Similar to $\mathbf {(1)}$, there is also a constant $\gamma ^{''}$ big enough, such that $c_{\gamma }(m_{1},m_{2})<\left( \frac{1}{2}-\frac{1}{p\xi _{p}}{\mathcal {S}}^{\frac{N}{2s}}\right)$ for any $\gamma \ge \gamma ^{''}$. We also obtain

  $$\begin{aligned} \begin{aligned} c_{\gamma }(m_{1},m_{2})&=\lim \limits _{n\rightarrow \infty }\left( I(u_{n},v_{n})-\frac{1}{p\xi _{p}}P(u_{n},v_{n})\right) \\ {}&\ge \left( \frac{1}{2}-\frac{1}{p\xi _{p}}\right) \omega _{j}\ge \left( \frac{1}{2}-\frac{1}{p\xi _{p}}\right) {\mathcal {S}}^{\frac{N}{2s}}, \end{aligned} \end{aligned}$$

  which contradicts our hypothesis.

• (3) If $q=\min \{p,q,\theta \}$. Analogously as (1) and (2), we can also get our conclusion, which we omit here.

To sum up, there exists a bigger positive constant $\gamma ^{*}(m_{1},m_{2})$ such that $\omega _{j}=0=\nu _{j}$ for any $j\in J$ and $\gamma \ge \gamma ^{*}$. Therefore, for $\gamma \ge \gamma ^{*}$, we have $u_{n}\rightarrow {\widetilde{u}}$ in $L^{2_{s}^{*}}({\mathbb {R}}^{N})$. The proof for $v_{n}\rightarrow {\widetilde{v}}$ in $L^{2_{s}^{*}}({\mathbb {R}}^{N})$ is similar.

Finally, we claim that ${\widetilde{u}}, {\widetilde{v}}\ne 0$. In fact, if not, we have $({\widetilde{u}}, {\widetilde{v}})=(0,0)$. According to (3.6), (3.7), $u_{n}\rightarrow {\widetilde{u}}, v_{n}\rightarrow {\widetilde{v}}$ in $L^{2_{s}^{*}}({\mathbb {R}}^{N})$ and Lemma 3.4, we have

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }([u_{n}]^{2}_{H^{s}}+[v_{n}]^{2}_{H^{s}})=0. \end{aligned}$$

(3.21)

It follows from (3.21) that $c_{\gamma }(m_{1},m_{2})=\lim \limits _{n\rightarrow \infty }I(u_{n},v_{n})=0$, which is impossible since $c_{\gamma }(m_{1},m_{2})>0$. As a result, we end the proof. $\square$

Set $c_{0}(m_{1},0):=E_{m_{1}}^{\eta _{1}}$ and $c_{0}(0,m_{2}):=E_{m_{2}}^{\eta _{2}}.$

Remark 3.1

For any $m_{1}, m_{2}>0$, according to Lemma 3.3, if necessary, to choose a bigger $\gamma ^{*}$ such that $c_{\gamma }(m_{1},m_{2})<\min \{c_{0}(m_{1},0),c_{0}(0,m_{2})\}$ for any $\gamma \ge \gamma ^{*}$.

Lemma 3.7

Assume that $c_{\gamma }(m_{1},m_{2})<\min \{c_{0}(m_{1},0),c_{0}(0,m_{2})\}$, then $(u_{n},v_{n})\rightarrow ({\widetilde{u}}, {\widetilde{v}})$ in $W_{r}$. Moreover, $({\widetilde{u}}, {\widetilde{v}})\in W_{r}$ is a positive normalized solution for system (1.3) associated with $\mu _{1},\mu _{2}<0$.

Proof

From Lemma 3.6, we know that ${\widetilde{u}}, {\widetilde{v}}\ne 0$. Next, we are going to classify it into two cases and prove by contradiction:

$Case ~1$ ${\widetilde{u}}\gvertneqq 0, {\widetilde{v}}\equiv 0$. From the strong maximum principle for fractional Laplace operators [32, Proposition 2.17], we have that ${\widetilde{u}}>0$. It follows from Lemma 2.1 and [25, Theorem 1.3] that ${\widetilde{u}}$ is a positive radial symmetric solution for problem (2.1) with parameters $p,\eta _{1}, m_{1}^{'}$ where $m_{1}^{'}=\Vert {\widetilde{u}}\Vert \le m_{1}^{2},$ and $c_{0}(m_{1},0)\le c_{0}(m_{1}^{'},0)\le I({\widetilde{u}},0)$.

In view of (3.7), Lemmas 2.4, 3.6, and two kinds of Brézis-Lieb Lemmas [36,37,38], we obtain that

$$\begin{aligned} \begin{aligned} 0&=\lim \limits _{n\rightarrow \infty }P(u_{n},v_{n})=\lim \limits _{n\rightarrow \infty }P(u_{n}-{\widetilde{u}},v_{n})+P({\widetilde{u}},0) \\ {}&=\lim \limits _{n\rightarrow \infty }[u_{n}-{\widetilde{u}}]^{2}_{H^{s}}+[v_{n}]^{2}_{H^{s}}, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} c_{\gamma }(m_{1},m_{2})&=\lim \limits _{n\rightarrow \infty }I(u_{n},v_{n})=\lim \limits _{n\rightarrow \infty }I(u_{n}-{\widetilde{u}},v_{n})+I({\widetilde{u}},0) \\ {}&\ge \frac{1}{2}\lim \limits _{n\rightarrow \infty }[u_{n}-{\widetilde{u}}]^{2}_{H^{s}}+[v_{n}]^{2}_{H^{s}}+c_{0}(m_{1},0) \\ {}&=c_{0}(m_{1},0). \end{aligned} \end{aligned}$$

This contradicts our hypothesis.

$Case ~2$ ${\widetilde{u}}\equiv 0, {\widetilde{v}}\gvertneqq 0$. Similarly to $\mathbf {Case ~1}$, ${\widetilde{v}}$ is a positive radial symmetric solution for problem (2.1) with parameters $q,\eta _{2}, m_{2}^{'}$ where $m_{2}^{'}=\Vert {\widetilde{v}}\Vert \le m_{2}^{2},$ and $c_{0}(0, m_{2})\le c_{0}(0, m_{2}^{'})\le I(0, {\widetilde{v}})$. We also have that

$$\begin{aligned} \begin{aligned} c_{\gamma }(m_{1},m_{2})&=\lim \limits _{n\rightarrow \infty }I(u_{n},v_{n})=\lim \limits _{n\rightarrow \infty }I(u_{n},v_{n}-{\widetilde{v}})+I(0, {\widetilde{v}}) \\ {}&\ge c_{0}(m_{1},0), \end{aligned} \end{aligned}$$

which is still a contradiction. So, we obtain that ${\widetilde{u}}\gvertneqq 0$ and ${\widetilde{v}}\gvertneqq 0$. Therefore, $\mu _{1}, \mu _{2}<0$ follows from Lemma 2.3 and (3.7) and (3.8).

On basis of (3.7) and (3.8), Lemma 3.4 and the boundedness of sequence $\{(\mu _{1}^{n},\mu _{2}^{n})\}$, we have

$$\begin{aligned} \begin{aligned} \mu _{1}m_{1}^{2}+\mu _{2}m_{2}^{2}&=\lim \limits _{n\rightarrow \infty }(\mu _{1}^{n}\Vert u_{n}\Vert _{2}^{2}+\mu _{2}^{n}\Vert v_{n}\Vert _{2}^{2}) \\ {}&=\lim \limits _{n\rightarrow \infty }\left( (\xi _{p}-1)\eta _{1}\Vert u_{n}\Vert _{p}^{p}+(\xi _{q}-1)\eta _{2}\Vert v_{n}\Vert _{q}^{q}+(\xi _{\theta }-1)\gamma \theta \int _{{\mathbb {R}}^{N}}|u_{n}|^{\alpha }|v_{n}|^{\beta }\right) \\ {}&=(\xi _{p}-1)\eta _{1}\Vert {\widetilde{u}}\Vert _{p}^{p}+(\xi _{q}-1)\eta _{2}\Vert {\widetilde{v}}\Vert _{q}^{q}+(\xi _{\theta }-1)\gamma \theta \int _{{\mathbb {R}}^{N}}|{\widetilde{u}}|^{\alpha }|{\widetilde{v}}|^{\beta } \\ {}&=\mu _{1}\Vert {\widetilde{u}}\Vert _{2}^{2}+\mu _{2}\Vert {\widetilde{v}}\Vert _{2}^{2}, \end{aligned} \end{aligned}$$

which means that

$$\begin{aligned} \mu _{1}(\Vert {\widetilde{u}}\Vert _{2}^{2}-m_{1}^{2})+\mu _{2}(\Vert {\widetilde{v}}\Vert _{2}^{2}-m_{2}^{2})=0. \end{aligned}$$

Thus,

$$\begin{aligned} \Vert {\widetilde{u}}\Vert _{2}^{2}=m_{1}^{2},~~\Vert {\widetilde{v}}\Vert _{2}^{2}=m_{2}^{2}, ~\text {i.e.}, ~(u_{n},v_{n})\rightarrow ({\widetilde{u}},{\widetilde{v}})~~\text {in}~L^{2}\times L^{2}. \end{aligned}$$

The proof is now complete.$\square$

Again by (3.7) and (3.8) and Lemma 3.4, we have

$$\begin{aligned} \begin{aligned} \lim \limits _{n\rightarrow \infty }\left( [u_{n}]^{2}_{H^{s}}+\mu _{1}\Vert u_{n} \Vert _{2}^{2}\right)&=\lim \limits _{n\rightarrow \infty }\left( \Vert u_{n}\Vert _{2_{s}^{*}}^{2_{s}^{*}}+\eta _{1}\Vert u_{n}\Vert _{p}^{p}+\gamma \alpha \int _{{\mathbb {R}}^{N}}|u_{n}|^{\alpha }|v_{n}|^{\beta }\right) \\ {}&=\Vert {\widetilde{u}}\Vert _{2_{s}^{*}}^{2_{s}^{*}}+\eta _{1}\Vert {\widetilde{u}}\Vert _{p}^{p}+\gamma \alpha \int _{{\mathbb {R}}^{N}}|{\widetilde{u}}|^{\alpha }|{\widetilde{v}}|^{\beta } \\ {}&=[{\widetilde{u}}]^{2}_{H^{s}}+\mu _{1}\Vert {\widetilde{u}}\Vert _{2}^{2}, \end{aligned} \end{aligned}$$

which implies that $\Vert u_{n}\Vert _{W_{r}}\rightarrow \Vert {\widetilde{u}}\Vert _{W_{r}}$ as $n\rightarrow \infty$. Similar to the above argument, we also obtain

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\left( [v_{n}]^{2}_{H^{s}}+\mu _{1}\Vert v_{n}\Vert _{2}^{2}\right) =[{\widetilde{v}}]^{2}_{H^{s}}+\mu _{1}\Vert {\widetilde{v}}\Vert _{2}^{2}, \end{aligned}$$

which also means that $\Vert v_{n}\Vert _{W_{r}}\rightarrow \Vert {\widetilde{v}}\Vert _{W_{r}}$ as $n\rightarrow \infty$. Therefore, we conclude that $(u_{n},v_{n})\rightarrow ({\widetilde{u}},{\widetilde{v}})$ in $W_{r}\times W_{r}$. The proof of Lemma 3.7 is complete.

Proof of Theorem 1.1

It follows from Lemmas 3.5–3.7 that the proof of Theorem 1.1 is complete. $\square$

4 Proof of theorem 1.2

Lemma 4.1

Assume that $N>2s$, $\alpha>1, \beta >1$, $p=q=\alpha +\beta +\theta =2^{*}_{s}$, and $\eta _{1}, \eta _{2}, \gamma , m_{1}, m_{2}>0$. Then the following system

$$\begin{aligned} \left\{ \begin{array}{lll} (-\Delta )^{s} u=\mu _{1} u+|u|^{2^{*}_{s}-2}u+\eta _{1}|u|^{2^{*}_{s}-2}u+\gamma \alpha |u|^{\alpha -2}u|v|^{\beta } ~ \text {in}~ {\mathbb {R}}^{N},\\ (-\Delta )^{s} v=\mu _{2} v+|v|^{2^{*}_{s}-2}v+\eta _{2}|v|^{2^{*}_{s}-2}v+\gamma \beta |u|^{\alpha }|v|^{\beta -2}v ~~\text {in}~ {\mathbb {R}}^{N},\\ \Vert u\Vert ^{2}_{L^{2}}=m_{1}^{2} ~\text {and}~ \Vert v\Vert ^{2}_{L^{2}}=m_{2}^{2}, ~~u,v\in H^{s}({\mathbb {R}}^{N}) \end{array}\right. \end{aligned}$$

(4.1)

has no positive normalized solution.

Proof

We will use the proof by contradiction. If (u, v) is a positive solution for the system (4.1) with some $\mu _{1}, \mu _{2}\in {\mathbb {R}}$. According to Lemma 2.3, we know that $\mu _{1}, \mu _{2}<0$. Then, it follows from (4.1) and the Pohozaev identity that

$$\begin{aligned} P(u,v)= & {} [u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}-\Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}-\Vert v\Vert _{2_{s}^{*}}^{2_{s}^{*}} -\eta _{1}\Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}-\eta _{2}\Vert v\Vert _{2_{s}^{*}}^{2_{s}^{*}}\\{} & {} -\gamma 2_{s}^{*}\int _{{\mathbb {R}}^{N}}|u|^{\alpha }|v|^{\beta }=0. \end{aligned}$$

Again by the definition of weak solution for the above system (4.1), we have

$$\begin{aligned}{}[u]^{2}_{H^{s}}+[v]^{2}_{H^{s}}+\mu _{1}\Vert u\Vert _{2}^{2}+\mu _{2}\Vert v\Vert _{2}^{2}= & {} \Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}+|v\Vert _{2_{s}^{*}}^{2_{s}^{*}} +\eta _{1}\Vert u\Vert _{2_{s}^{*}}^{2_{s}^{*}}+\eta _{2}\Vert v\Vert _{2_{s}^{*}}^{2_{s}^{*}}\\{} & {} +\gamma 2_{s}^{*}\int _{{\mathbb {R}}^{N}}|u|^{\alpha }|v|^{\beta }. \end{aligned}$$

The proof is now complete.$\square$

Thus, we obtain

$$\begin{aligned} \mu _{1}\Vert u\Vert _{2}^{2}+\mu _{2}\Vert v\Vert _{2}^{2}=\mu _{1}m_{1}^{2}+\mu _{2}m_{2}^{2}=0. \end{aligned}$$

This is clearly a contradiction. The proof of Lemma 4.1 is complete.

Proof of Theorem 1.2

It follows from Lemma 4.1 that the proof of Theorem 1.2 is complete. $\square$