Symmetric solutions for singular quasilinear elliptic systems involving multiple critical Hardy-Sobolev exponents

Abstract

This paper deals with the existence and multiplicity of symmetric solutions for a class of singular quasilinear elliptic systems involving multiple critical Hardy-Sobolev exponents in a bounded symmetric domain. Based upon the symmetric criticality principle of Palais and variational methods, we establish several existence and multiplicity results of G-symmetric solutions under certain appropriate hypotheses on the weighted functions and the parameters.

1 Introduction

The purpose of this paper is to investigate the existence and multiplicity of nontrivial solutions for the following singular quasilinear elliptic system:

$$\{\begin{array}{cc}{\mathcal{L}}_{a,p}^{\mu }u=\frac{K(x)}{{|x|}^{b{p}_b^{\ast }}}({|u|}^{p_b^{\ast }-2}u+{\sum }_{i=1}^m\frac{{\varsigma }_i{\alpha }_i}{p_b^{\ast }}{|u|}^{{\alpha }_i-2}u{|v|}^{{\beta }_i})+\lambda \frac{q_1{|u|}^{q_1-2}u{|v|}^{q_2}}{(q_1+q_2){|x|}^{d{p}_d^{\ast }}},\hfill & \text{in }\mathrm{\Omega },\hfill \\ {\mathcal{L}}_{a,p}^{\mu }v=\frac{K(x)}{{|x|}^{b{p}_b^{\ast }}}({|v|}^{p_b^{\ast }-2}v+{\sum }_{i=1}^m\frac{{\varsigma }_i{\beta }_i}{p_b^{\ast }}{|u|}^{{\alpha }_i}{|v|}^{{\beta }_i-2}v)+\lambda \frac{q_2{|u|}^{q_1}{|v|}^{q_2-2}v}{(q_1+q_2){|x|}^{d{p}_d^{\ast }}},\hfill & \text{in }\mathrm{\Omega },\hfill \\ u=v=0,\hfill & \text{on }\partial \mathrm{\Omega },\hfill \end{array}$$

(1.1)

where $\Omega\subset\mathbb{R}^{N}$ ($N\geq3$) is a smooth bounded domain, $0\in\Omega$ and Ω is G-symmetric with respect to a closed subgroup G of $O(\mathbb{N})$ (see Section 2 for details), ${\mathcal{L}}_{a,p}^{\mu }\triangleq -div({|x|}^{-ap}{|\mathrm{\nabla }\cdot |}^{p-2}\mathrm{\nabla }\cdot )-\mu \frac{{|\cdot |}^{p-2}\cdot }{{|x|}^{p(a+1)}}$ is a quasilinear elliptic operator, $1< p<N$, $0\leq a<\frac{N-p}{p}$, $a\leq b\leq d< a+1$, $0\leq\mu<\overline{\mu}$ with $\overline{\mu}\triangleq(\frac{N-p(a+1)}{p})^{p}$, $\lambda\geq 0$, $0< \varsigma_{i}<+\infty$ and $\alpha_{i}$, $\beta_{i}>1$ satisfy $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, m$; $1\leq m\in\mathbb{N}$), $q_{1} , q_{2}\geq1$ and $p< q_{1}+q_{2}<p_{d}^{\ast}$, $p_{b}^{\ast}\triangleq\frac{Np}{N-p(a+1-b)}$ and $p_{d}^{\ast}\triangleq\frac{Np}{N-p(a+1-d)}$ are the critical Hardy-Sobolev exponents, and $p_{a}^{\ast}=p^{\ast}\triangleq\frac{Np}{N-p}$ is the critical Sobolev exponent, $K\in\mathcal{C}(\overline{\Omega})\cap L^{\infty}(\overline{\Omega})$ satisfies some symmetry conditions which will be specified later.

Singular critical elliptic boundary value problems have been of great interest recently. This is because of both the intensive development of the theory of singular calculus itself and the applications of such constructions in various physical fields such as fluid mechanics, glaciology, molecular physics, quantum cosmology and linearization of combustion models (see [1] for example). The existence and multiplicity of solutions of scalar singular elliptic equations involving critical exponents have been studied in the literature [2–8] and the references therein. In a recent paper, Deng and Jin [9] considered the existence of nontrivial solutions for the following singular semilinear elliptic problem:

$$ -\Delta u=\mu\frac{u}{|x|^{2}}+K(x)\frac{u^{2^{\ast} (s)-1}}{|x|^{s}}, \quad\text{and}\quad u>0 \quad\text{in } \mathbb{R}^{N}, $$

(1.2)

where $N> 2$, $0\leq\mu<(\frac{N-2}{2})^{2}$, $0\leq s<2$, $2^{\ast}(s)\triangleq\frac{2(N-s)}{N-2}$ and $2^{\ast}(0)=2^{\ast}\triangleq\frac{2N}{N-2}$, and K fulfills some symmetry conditions with respect to a subgroup G of $O(\mathbb{N})$. Applying variational methods and analytic techniques, the authors obtained the existence and multiplicity of G-symmetric solutions to (1.2) under some assumptions on K. Very recently, Deng and Huang [10, 11] extended the results in [9] to the scalar weighted elliptic problems in a bounded symmetric domain. Besides, we also remark that when $\mu=s=0$ and the right-hand side term $|x|^{-s}u^{2^{\ast}(s)-1}$ is replaced by a term $f(u)$ of the pure power, such as $f(u)=u^{r-1}$ with $1< r<2^{\ast}$ or $r=2^{\ast}$, there are some elegant results on the existence and multiplicity of G-symmetric solutions of (1.2), which can be found in Refs. [12–14].

On the other hand, there have been many papers concerned with the existence and multiplicity of solutions for singular elliptic systems in recent years. Many results were obtained in these publications and these results give us a good insight into the corresponding problems. As an example, Huang and Kang [15] discussed the following singular semilinear elliptic system:

$$ \begin{cases}L_{\mu_{1}, a_{1}}u=|u|^{2^{\ast}-2}u +\frac{\varsigma\alpha}{\alpha+\beta}|u|^{\alpha-2}u|v|^{\beta} +\lambda_{1}|u|^{q_{1}-2}u, &\text{in } \Omega,\\ L_{\mu_{2}, a_{2}}v=|v|^{2^{\ast}-2}v +\frac{\varsigma\beta}{\alpha+\beta}|u|^{\alpha}|v|^{\beta-2}v +\lambda_{2}|v|^{q_{2}-2}v, &\text{in } \Omega,\\ u=v=0, &\text{on } \partial\Omega, \end{cases} $$

(1.3)

where $L_{\mu_{i}, a_{i}}=-\Delta-\mu_{i}|x-a_{i}|^{-2}$, $\varsigma>0$, $a_{i}\in\Omega$, $\lambda_{i}>0$, $\mu_{i}<(\frac{N-2}{2})^{2}$, $2\leq q_{i}<2^{\ast}$ ($i=1, 2$), and $\alpha , \beta>1$ satisfy $\alpha+\beta=2^{\ast}$. Note that $|u|^{\alpha-2}u|v|^{\beta}$ and $|u|^{\alpha}|v|^{\beta-2}v$ in (1.3) are called strongly-coupled terms, and $|u|^{2^{\ast}-2}u$, $|v|^{2^{\ast}-2}v$ are weakly-coupled terms. By employing variational methods and the Moser iteration techniques, the authors obtained the existence of positive solutions and some properties of solutions to (1.3). Recently, further studies on singular elliptic systems like (1.3) were taken up by Kang [16] and Nyamoradi [17], where the existence and multiplicity of positive solutions were proved. For more results, we refer the readers to [18–20] and the references therein.

However, as far as we know, the existence and multiplicity of G-symmetric solutions for singular elliptic systems have seldom been studied; we only find some symmetric results for singular elliptic systems in [21, 22] and when $G=O(\mathbb{N})$ some radial and nonradial results for nonsingular elliptic systems in [23]. Inspired by [9, 12, 15], in the present paper, we are concerned with the existence and multiplicity of G-symmetric solutions for system (1.1). The main difficulties lie in the fact that there are not only the singular perturbations $|x|^{-dp_{d}^{\ast}}|u|^{q_{1}-2}u|v|^{q_{2}}$ and $|x|^{-dp_{d}^{\ast}}|u|^{q_{1}}|v|^{q_{2}-2}v$ in (1.1), but also the nonlinear strong-coupled terms $\sum_{i=1}^{m} \frac{\varsigma_{i}\alpha_{i}}{p_{b}^{\ast}} |u|^{\alpha_{i}-2}u|v|^{\beta_{i}}$, $\sum_{i=1}^{m} \frac{\varsigma_{i}\beta_{i}}{p_{b}^{\ast}} |u|^{\alpha_{i}}|v|^{\beta_{i}-2}v$ and weak-coupled terms $|u|^{p_{b}^{\ast}-2}u$, $|v|^{p_{b}^{\ast}-2}v$. Compared with problems (1.2) and (1.3), the singular quasilinear elliptic problem (1.1) becomes more complicated to deal with and thus we have to face more difficulties. To our knowledge, even in the particular cases $\mu=a=b=0$, $p=2$, $\lambda\geq0$ and $\varsigma_{i}>0$ ($i=1, \ldots, m$), there are few results on the existence of G-symmetric solutions for (1.1). Many attractive and challenging topics on singular elliptic systems remain unsolved. Consequently, it makes sense for us to investigate system (1.1) thoroughly. Let $K_{0}>0$ be a constant. Note that here we will try to treat both the cases of $\lambda=0$, $K(x)\not\equiv K_{0}$ and $\lambda>0$, $K(x)\equiv K_{0}$.

This paper is divided into four sections, schemed as follows. In Section 2, we will set up the variational framework which is applicable to the discussion of problem (1.1), and we will state the main results of this paper. In Section 3, we present the proofs of several existence and multiplicity results for the cases $\lambda=0$ and $K(x)\not\equiv K_{0}$ in (1.1). In Section 4, we detail the proofs of existence results for the cases $\lambda>0$ and $K(x)\not\equiv K_{0}$ in (1.1). Our methods in this paper are mainly based upon the symmetric criticality principle of Palais (see [24]) and variational arguments.

2 Preliminaries and main results

Let $O(\mathbb{N})$ be the group of orthogonal linear transformations of $\mathbb{R}^{N}$ with natural action, and let $G\subset O(\mathbb{N})$ be a closed subgroup. For $x\neq0$ we denote the cardinality of $G_{x}=\{gx; g\in G\}$ by $|G_{x}|$ and set $|G|=\inf_{0\neq x\in\mathbb{R}^{N}}|G_{x}|$. Note that here $|G|$ may be +∞. We call Ω a G-symmetric subset of $\mathbb{R}^{N}$, if $x\in\Omega$, then $gx\in\Omega$ for all $g\in G$. For any function $f: \mathbb{R}^{N}\rightarrow \mathbb{R}$, we call $f(x)$ a G-symmetric function if for all $g\in G$ and $x\in\mathbb{R}^{N}$, $f(gx)=f(x)$ holds. In particular, if f is radially symmetric, then the corresponding group G is $O(\mathbb{N})$ and $|G|=+\infty$. Other further examples of G-symmetric functions can be found in Ref. [9].

Let $W_{a}^{1, p}(\Omega)$ denote the closure of $\mathcal {C}_{0}^{\infty}(\Omega)$ functions with respect to the norm $(\int_{\Omega}|x|^{-ap} |\nabla u|^{p}\,dx)^{1/p}$. We recall that the well-known Caffarelli-Kohn-Nirenberg inequality [25] asserts that for all $u\in W_{a}^{1, p}(\Omega)$, there is a constant $C=C(N, p, a, b)>0$ such that

$$ \biggl(\int_{\Omega}\frac{|u|^{q}}{|x|^{bp_{b}^{\ast}}}\,dx \biggr)^{p/q}\leq C\int_{\Omega}|x|^{-ap}| \nabla u|^{p}\,dx,\quad p\leq q\leq p_{b}^{\ast}, $$

(2.1)

where $-\infty< a<\frac{N-p}{p}$, $a\leq b\leq a+1$ and $p_{b}^{\ast}=\frac{Np}{N-p(a+1-b)}$. If $b=a+1$ and $q=p$, then $p_{b}^{\ast}=p$ and the following weighted Hardy inequality holds (see [25]):

$$ \int_{\Omega}\frac{|u|^{p}}{|x|^{p(a+1)}}\,dx\leq \frac{1}{\overline{\mu}}\int_{\Omega}|x|^{-ap}|\nabla u|^{p}\,dx, \quad\forall u\in W_{a}^{1, p}(\Omega), $$

(2.2)

where $\overline{\mu}=(\frac{N-p(a+1)}{p})^{p}$. Now we employ the following norm in $W_{a}^{1, p}(\Omega)$:

$$\|u\|_{\mu}\triangleq \biggl[\int_{\Omega} \biggl(|x|^{-ap}|\nabla u|^{p}-\mu\frac{|u|^{p}}{|x|^{p(a+1)}} \biggr)\,dx \biggr]^{1/p},\quad 0\leq\mu< \overline{\mu}. $$

By the weighted Hardy inequality (2.2), we find that the above norm is equivalent to the usual norm $(\int_{\Omega}|x|^{-ap}|\nabla u|^{p}\,dx)^{1/p}$. Moreover, we define the product space $(W_{a}^{1, p}(\Omega))^{2}$ endowed with the norm

$$\bigl\| (u, v)\bigr\| _{\mu}= \bigl(\|u\|_{\mu}^{p}+\|v \|_{\mu}^{p} \bigr)^{1/p}, \quad\forall(u, v)\in \bigl(W_{a}^{1, p}(\Omega)\bigr)^{2}. $$

For a bounded and G-symmetric domain $0\in\Omega\subset\mathbb{R}^{N}$, the natural functional space to study problem (1.1) is the Banach space $(W_{a, G}^{1, p}(\Omega))^{2}$, which is the subspace of $(W_{a}^{1, p}(\Omega))^{2}$ consisting of all G-symmetric functions. In this paper we are concerned with the following problems:

$$\left({\mathcal{P}}_{\lambda }^K\right)\{\begin{array}{cc}{\mathcal{L}}_{a,p}^{\mu }u=\frac{K(x)}{{|x|}^{b{p}_b^{\ast }}}({|u|}^{p_b^{\ast }-2}u+{\sum }_{i=1}^m\frac{{\varsigma }_i{\alpha }_i}{p_b^{\ast }}{|u|}^{{\alpha }_i-2}u{|v|}^{{\beta }_i})+\lambda \frac{q_1{|u|}^{q_1-2}u{|v|}^{q_2}}{(q_1+q_2){|x|}^{d{p}_d^{\ast }}},\hfill & \text{in }\mathrm{\Omega },\hfill \\ {\mathcal{L}}_{a,p}^{\mu }v=\frac{K(x)}{{|x|}^{b{p}_b^{\ast }}}({|v|}^{p_b^{\ast }-2}v+{\sum }_{i=1}^m\frac{{\varsigma }_i{\beta }_i}{p_b^{\ast }}{|u|}^{{\alpha }_i}{|v|}^{{\beta }_i-2}v)+\lambda \frac{q_2{|u|}^{q_1}{|v|}^{q_2-2}v}{(q_1+q_2){|x|}^{d{p}_d^{\ast }}},\hfill & \text{in }\mathrm{\Omega },\hfill \\ (u,v)\in {(W_{a,G}^{1,p}(\mathrm{\Omega }))}^2,\text{and}u>0,v>0,\hfill & \text{in }\mathrm{\Omega }.\hfill \end{array}$$

To mention our main results, we need to introduce two notations ${\mathcal{A}}_{\mu }$ and $y_{\epsilon}(x)$, which are, respectively, defined by

$${\mathcal{A}}_{\mu }\triangleq \underset{u\in W_a^{1,p}(\mathrm{\Omega })\mathrm{\setminus }\{0\}}{inf}\frac{{\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }u|}^p-\mu \frac{{|u|}^p}{{|x|}^{p(a+1)}})dx}{{({\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}{|u|}^{p_b^{\ast }}dx)}^{\frac{p}{p_b^{\ast }}}}$$

(2.3)

and

$$ y_{\epsilon}(x)=C\epsilon^{-\xi} U_{\mu} \biggl(\frac{|x|}{\epsilon} \biggr), $$

(2.4)

where $\epsilon>0$, $\xi\triangleq\frac{N-p(a+1)}{p}$ and the constant $C=C(N, p, a, b, \mu)>0$, depending only on N, p, a, b and μ. From [4], we find that ${\mathcal{A}}_{\mu }$ is independent of Ω and $y_{\epsilon}(x)$ satisfies the equations

$$ \int_{\mathbb{R}^{N}} \biggl(|x|^{-ap} |\nabla y_{\epsilon}|^{p}-\mu \frac{|y_{\epsilon}|^{p}}{|x|^{p(a+1)}} \biggr)\,dx=1 $$

(2.5)

and

$${\int }_{{\mathbb{R}}^N}\frac{y_{ϵ}^{p_b^{\ast }-1}\phi }{{|x|}^{b{p}_b^{\ast }}}dx={\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}{\int }_{{\mathbb{R}}^N}({|x|}^{-ap}{|\mathrm{\nabla }{y}_{ϵ}|}^{p-2}\mathrm{\nabla }{y}_{ϵ}\mathrm{\nabla }\phi -\mu \frac{{|y_{ϵ}|}^{p-2}{y}_{ϵ}\phi }{{|x|}^{p(a+1)}})dx$$

for all $\phi \in {\mathcal{D}}_a^{1,p}({\mathbb{R}}^N)$, where ${\mathcal{D}}_a^{1,p}({\mathbb{R}}^N)$ is the closure of $\mathcal{C}_{0}^{\infty}(\mathbb{R}^{N})$ functions with respect to the norm $(\int_{\mathbb{R}^{N}}|x|^{-ap}|\nabla \cdot|^{p}\,dx)^{1/p}$. In particular, we have (let $\varphi=y_{\epsilon}$)

$${\int }_{{\mathbb{R}}^N}{|x|}^{-b{p}_b^{\ast }}{y}_{ϵ}^{p_b^{\ast }}dx={\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}.$$

(2.6)

The function $U_{\mu}(x)=U_{\mu}(|x|)$ in (2.4) is the unique radial solution of the following limiting problem (see [4, Lemma 2.2]):

$$\{\begin{array}{cc}-div({|x|}^{-ap}{|\mathrm{\nabla }u|}^{p-2}\mathrm{\nabla }u)=\mu \frac{u^{p-1}}{{|x|}^{p(a+1)}}+\frac{u^{p_b^{\ast }-1}}{{|x|}^{b{p}_b^{\ast }}},\hfill & \text{in }{\mathbb{R}}^N\mathrm{\setminus }\{0\},\hfill \\ u\in {\mathcal{D}}_a^{1,p}({\mathbb{R}}^N),\text{and}u>0,\hfill & \text{in }{\mathbb{R}}^N\mathrm{\setminus }\{0\},\hfill \end{array}$$

satisfying $U_{\mu}(1)=(p_{b}^{\ast}(\overline{\mu}-\mu)/p) ^{1/(p_{b}^{\ast}-p)}$. Moreover, the following asymptotic properties at the origin and infinity for $U_{\mu}(r)$ and $U_{\mu}^{\prime}(r)$ hold:

$$\begin{aligned}& \lim_{r\rightarrow 0}r^{l_{1}}U_{\mu}(r)= \widetilde{C}_{1}>0, \quad\quad\lim_{r\rightarrow 0}r^{l_{1}+1}\bigl|U_{\mu}^{\prime}(r)\bigr|= \widetilde{C}_{1}l_{1}\geq0, \end{aligned}$$

(2.7)

$$\begin{aligned}& \lim_{r\rightarrow +\infty}r^{l_{2}}U_{\mu}(r)= \widetilde{C}_{2}>0, \quad\quad\lim_{r\rightarrow +\infty}r^{l_{2}+1}\bigl|U_{\mu}^{\prime}(r)\bigr|= \widetilde{C}_{2}l_{2}>0, \end{aligned}$$

(2.8)

where $\widetilde{C}_{1}$, $\widetilde{C}_{2}$ are positive constants and $l_{1}=l_{1}(N, a, p, \mu)$, $l_{2}=l_{2}(N, a, p, \mu)$ are the zeros of the function

$$\mathcal{L}(t)=(p-1)t^{p}- \bigl(N-p(a+1) \bigr)t^{p-1}+\mu,\quad t\geq0, 0\leq\mu< \overline{\mu}, $$

which satisfy

$$ 0\leq l_{1}< \xi<l_{2}\leq\frac{N-p(a+1)}{p-1}. $$

(2.9)

Furthermore, there exist the positive constants $\widetilde{C}_{3}=\widetilde{C}_{3}(p, \mu, a, b)$ and $\widetilde{C}_{4}=\widetilde{C}_{4}(p, \mu, a, b)$ such that

$$ \widetilde{C}_{3}\leq U_{\mu}(x) \bigl(|x|^{\frac{l_{1}}{\xi}} +|x|^{\frac{l_{2}}{\xi}} \bigr)^{\xi}\leq \widetilde{C}_{4},\quad \xi=\frac{N-p(a+1)}{p}. $$

(2.10)

We suppose that the function $K(x)$ verifies the following hypotheses.

• (K.1) $K\in\mathcal{C}(\overline{\Omega})\cap L^{\infty}(\overline{\Omega})$, and K is G-symmetric.

• (K.2) $K_{+}\not\equiv0$, where $K_{+}=\max\{0, K\}$.

Since $0\in\Omega$, we can choose $\varrho>0$ small enough such that $B_{2\varrho}(0)\subset\Omega$ and define a function $\phi\in\mathcal{C}_{0}^{1}(\Omega)$ such that $\phi(x)=1$ on $B_{\varrho}(0)$, $\phi(x)=0$ on $\Omega\backslash B_{2\varrho}(0)$. Setting $V_{\epsilon}=\phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$, we get (see (3.15) for details)

$${\parallel V_{ϵ}\parallel }_{\mu }=1\text{and}{\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}{|V_{ϵ}|}^{p_b^{\ast }}dx={\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}+O\left({ϵ}^{p(l_2+a+1)-N}\right).$$

The main results of this paper are summarized in the following.

Theorem 2.1

Suppose that (K.1) and (K.2) hold. If

$${\int }_{\mathrm{\Omega }}K(x)\frac{{|V_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}dx\ge {\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}max\{K_{+}(0),{|G|}^{\frac{p-p_b^{\ast }}{p}}{({\mathcal{A}}_0/{\mathcal{A}}_{\mu })}^{-\frac{p_b^{\ast }}{p}}{\parallel K_{+}\parallel }_{\mathrm{\infty }}\}>0$$

(2.11)

for some $\epsilon>0$, then problem $({\mathcal{P}}_0^K)$ has at least one positive solution in $(W_{a, G}^{1, p}(\Omega))^{2}$.

Corollary 2.1

Suppose that (K.1) and (K.2) hold. Then problem $({\mathcal{P}}_0^K)$ has at least one positive solution in $(W_{a, G}^{1, p}(\Omega))^{2}$ if

$$K(0)>0,K(0)\ge {|G|}^{\frac{p-p_b^{\ast }}{p}}{({\mathcal{A}}_0/{\mathcal{A}}_{\mu })}^{-\frac{p_b^{\ast }}{p}}{\parallel K_{+}\parallel }_{\mathrm{\infty }}$$

(2.12)

and $K(x)\geq K(0)+\gamma_{0} |x|^{\vartheta}$ for some $\gamma_{0}>0$, $\vartheta\in (0, p(l_{2}+a+1)-N )$ and $|x|$ small.

Theorem 2.2

Suppose that $K_{+}(0)=0$ and $|G|=+\infty$. Then problem $({\mathcal{P}}_0^K)$ has infinitely many G-symmetric solutions.

Corollary 2.2

If K is a radially symmetric function such that $K_{+}(0)=0$, then problem $({\mathcal{P}}_0^K)$ has infinitely many solutions which are radially symmetric.

Theorem 2.3

Let $\lambda>0$ and $K(x)\equiv K_{0}>0$ be a constant. If $q_{1} , q_{2}\geq1$ satisfy

$$ \max \biggl\{ p, \frac{N- dp_{d}^{\ast}}{l_{2}}, p_{d}^{\ast}+p- \frac{p^{2}l_{2}}{N-p(a+1)} \biggr\} < q_{1}+q_{2}<p_{d}^{\ast}, $$

(2.13)

then $({\mathcal{P}}_{\lambda }^{K_0})$ possesses at least one positive solution in $(W_{a, G}^{1, p}(\Omega))^{2}$.

In what follows, we denote by $(W_{a, G}^{1, p}(\Omega))^{2}$ the subspace of $(W_{a}^{1, p}(\Omega))^{2}$ consisting of all G-symmetric functions. The dual space of $(W_{a, G}^{1, p}(\Omega))^{2}$ ($(W_{a}^{1, p}(\Omega))^{2}$, resp.) is denoted by $(W_{a, G}^{-1, p^{\prime}}(\Omega))^{2}$ ($(W_{a}^{-1, p^{\prime}}(\Omega))^{2}$, resp.), where $\frac{1}{p}+\frac{1}{p^{\prime}}=1$. The ball of center x and radius r is denoted by $B_{r}(x)$. We employ C, $C_{i}$ ($i=1,2,\ldots$) to denote the positive constants, and denote by ‘→’ convergence in norm in a given Banach space X and by ‘⇀’ weak convergence. A functional $\mathcal{F}\in {\mathcal{C}}^1(X,\mathbb{R})$ is said to satisfy the $(PS)_{c}$ condition if each sequence $\{w_{n}\}$ in X satisfying $\mathcal{F}(w_n)\to c$, ${\mathcal{F}}^{\prime }(w_n)\to 0$ in $X^{\ast}$ has a subsequence which strongly converges to some element in X. Hereafter, $o_{n}(1)$ denotes a datum which tends to 0 as $n\rightarrow\infty$. $L^{q}(\Omega, |x|^{-\varsigma})$ denotes the weighted $L^{q}(\Omega)$ space with the norm $(\int_{\Omega}|x|^{-\varsigma} |u|^{q}\,dx)^{1/q}$.

3 Existence and multiplicity results for problem $({\mathcal{P}}_0^K)$

The energy functional corresponding to problem $({\mathcal{P}}_0^K)$ is defined on $(W_{a, G}^{1, p}(\Omega))^{2}$ by

$$\mathcal{E}(u,v)=\frac{1}{p}{\parallel (u,v)\parallel }_{\mu }^p-\frac{1}{p_b^{\ast }}{\int }_{\mathrm{\Omega }}K(x){|x|}^{-b{p}_b^{\ast }}({|u|}^{p_b^{\ast }}+{|v|}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{|u|}^{{\alpha }_i}{|v|}^{{\beta }_i})dx.$$

(3.1)

By (K.1), (2.1) and the Young inequality, we easily verify that $\mathcal{E}\in {\mathcal{C}}^1({(W_{a,G}^{1,p}(\mathrm{\Omega }))}^2,\mathbb{R})$. It is well known that there exists a one-to-one correspondence between the weak solutions of problem $({\mathcal{P}}_0^K)$ and the critical points of ℰ. More precisely, the weak solutions of $({\mathcal{P}}_0^K)$ are exactly the critical points of ℰ by the following principle of symmetric criticality due to Palais (see Lemma 3.1), namely $(u, v)\in(W_{a, G}^{1, p}(\Omega))^{2}$ satisfies $({\mathcal{P}}_0^K)$ if and only if for all $(\varphi_{1}, \varphi_{2})\in(W_{a}^{1, p}(\Omega))^{2}$, there holds

$$\begin{aligned} 0 =&\int_{\Omega} \Biggl\{ |x|^{-ap}|\nabla u|^{p-2}\nabla u\nabla\varphi_{1}+|x|^{-ap}|\nabla v|^{p-2}\nabla v\nabla\varphi_{2}-\mu\frac{|u|^{p-2}u\varphi_{1} +|v|^{p-2}v\varphi_{2}}{|x|^{p(a+1)}} \\ &{}-\frac{K(x)}{|x|^{bp_{b}^{\ast}}} \Biggl[|u|^{p_{b}^{\ast}-2}u\varphi_{1} +|v|^{p_{b}^{\ast}-2}v\varphi_{2} \\ &{}+ \sum_{i=1}^{m} \frac{\varsigma_{i} (\alpha_{i}|u|^{\alpha_{i}-2}u|v|^{\beta_{i}}\varphi_{1} +\beta_{i}|u|^{\alpha_{i}}|v|^{\beta_{i}-2}v\varphi_{2}) }{p_{b}^{\ast}} \Biggr] \Biggr\} \,dx. \end{aligned}$$

(3.2)

Lemma 3.1

Let $K(x)$ be a G-symmetric function; ${\mathcal{E}}^{\prime }(u,v)=0$ in $(W_{a, G}^{-1, p^{\prime}}(\Omega))^{2}$ implies ${\mathcal{E}}^{\prime }(u,v)=0$ in $(W_{a}^{-1, p^{\prime}}(\Omega))^{2}$.

Proof

Similar to the proof of [12, Lemma 1] (see also [23, Proposition 2.8]). □

For all $\mu\in[0, \overline{\mu})$, $0<\varsigma_{i}<+\infty$, $\alpha_{i}$, $\beta_{i}>1$ and $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, m$), we define

$${\mathcal{A}}_{\mu ,{\varsigma }_i}\triangleq \underset{(u,v)\in {(W_a^{1,p}(\mathrm{\Omega })\mathrm{\setminus }\{0\})}^2}{inf}\frac{{\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }u|}^p+{|x|}^{-ap}{|\mathrm{\nabla }v|}^p-\mu \frac{{|u|}^p+{|v|}^p}{{|x|}^{p(a+1)}})dx}{{[{\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}({|u|}^{p_b^{\ast }}+{|v|}^{p_b^{\ast }}+{\sum }_{i=1}^m{\varsigma }_i{|u|}^{{\alpha }_i}{|v|}^{{\beta }_i})dx]}^{\frac{p}{p_b^{\ast }}}},$$

(3.3)

$$\begin{aligned}& h(\tau)\triangleq\frac{1+\tau^{p}}{ (1+\tau^{p_{b}^{\ast}} +\sum_{i=1}^{m}\varsigma_{i}\tau^{\beta_{i}} )^{\frac{p}{p_{b}^{\ast}}}}, \quad\tau\geq0, \end{aligned}$$

(3.4)

$$\begin{aligned}& h(\tau_{\min})\triangleq\min_{\tau\geq0}h( \tau)>0, \end{aligned}$$

(3.5)

where $\tau_{\min}>0$ is a minimal point of $h(\tau)$ and therefore a root of the equation

$$ \tau^{p_{b}^{\ast}-p}+\sum_{i=1}^{m} \frac{\varsigma_{i}\beta_{i}}{p_{b}^{\ast}}\tau^{\beta_{i}-p} -\sum_{i=1}^{m} \frac{\varsigma_{i}\alpha_{i}}{ p_{b}^{\ast}}\tau^{\beta_{i}}-1=0,\quad \tau\geq0. $$

(3.6)

Lemma 3.2

Let $y_{\epsilon}(x)$ be the extremal function of ${\mathcal{A}}_{\mu }$ defined in (2.4), $0<\varsigma_{i}<+\infty$, $\alpha_{i}$, $\beta_{i}>1$ and $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, m$). Then we have the following statements:

• (i) ${\mathcal{A}}_{\mu ,{\varsigma }_i}=h({\tau }_{min}){\mathcal{A}}_{\mu }$;

• (ii) ${\mathcal{A}}_{\mu ,{\varsigma }_i}$ has the minimizer $(y_{\epsilon}(x), \tau_{\min}y_{\epsilon}(x))$, $\forall\epsilon>0$.

Proof

The proof is similar to that of [17, Theorem 2] and is omitted here. □

Lemma 3.3

Let $\{(u_{n}, v_{n})\}$ be a weakly convergent sequence to $(u, v)$ in $(W_{a, G}^{1, p}(\Omega))^{2}$ such that $|x|^{-ap}|\nabla u_{n}|^{p}\rightharpoonup\eta^{(1)}$, $|x|^{-ap}|\nabla v_{n}|^{p}\rightharpoonup\eta^{(2)}$, $|x|^{-bp_{b}^{\ast}}|u_{n}|^{p_{b}^{\ast}}\rightharpoonup \sigma^{(1)}$, $|x|^{-bp_{b}^{\ast}}|v_{n}|^{p_{b}^{\ast}}\rightharpoonup \sigma^{(2)}$, $|x|^{-bp_{b}^{\ast}}|u_{n}|^{\alpha_{i}}|v_{n}|^{\beta_{i}} \rightharpoonup\nu^{(i)}$ ($i=1,\ldots,m$), $|x|^{-p(a+1)}|u_{n}|^{p}\rightharpoonup\gamma^{(1)}$, $|x|^{-p(a+1)}|v_{n}|^{p}\rightharpoonup\gamma^{(2)}$ in the sense of measures. Then there exists some at most countable set $\mathcal{J}$, ${\{{\eta }_j^{(1)}\ge 0\}}_{j\in \mathcal{J}\cup \{0\}}$, ${\{{\eta }_j^{(2)}\ge 0\}}_{j\in \mathcal{J}\cup \{0\}}$, ${\{{\sigma }_j^{(1)}\ge 0\}}_{j\in \mathcal{J}\cup \{0\}}$, ${\{{\sigma }_j^{(2)}\ge 0\}}_{j\in \mathcal{J}\cup \{0\}}$, ${\{{\nu }_j^{(i)}\ge 0\}}_{j\in \mathcal{J}\cup \{0\}}$, $\gamma_{0}^{(1)}\geq0$, $\gamma_{0}^{(2)}\geq0$, ${\{x_j\}}_{j\in \mathcal{J}}\subset \overline{\mathrm{\Omega }}\mathrm{\setminus }\{0\}$ such that

• (a) ${\eta }^{(1)}\ge \frac{{|\mathrm{\nabla }u|}^p}{{|x|}^{ap}}+{\sum }_{j\in \mathcal{J}}{\eta }_j^{(1)}{\delta }_{x_j}+{\eta }_0^{(1)}{\delta }_0$, ${\eta }^{(2)}\ge \frac{{|\mathrm{\nabla }v|}^p}{{|x|}^{ap}}+{\sum }_{j\in \mathcal{J}}{\eta }_j^{(2)}{\delta }_{x_j}+{\eta }_0^{(2)}{\delta }_0$,

• (b) ${\sigma }^{(1)}=\frac{{|u|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}+{\sum }_{j\in \mathcal{J}}{\sigma }_j^{(1)}{\delta }_{x_j}+{\sigma }_0^{(1)}{\delta }_0$, ${\sigma }^{(2)}=\frac{{|v|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}+{\sum }_{j\in \mathcal{J}}{\sigma }_j^{(2)}{\delta }_{x_j}+{\sigma }_0^{(2)}{\delta }_0$, ${\nu }^{(i)}=\frac{{|u|}^{{\alpha }_i}{|v|}^{{\beta }_i}}{{|x|}^{b{p}_b^{\ast }}}+{\sum }_{j\in \mathcal{J}}{\nu }_j^{(i)}{\delta }_{x_j}+{\nu }_0^{(i)}{\delta }_0$, $i=1,\ldots,m$,

• (c) $\gamma^{(1)}= |x|^{-p(a+1)}|u|^{p}+\gamma_{0}^{(1)}\delta_{0}$, $\gamma^{(2)}=|x|^{-p(a+1)}|v|^{p}+\gamma_{0}^{(2)}\delta_{0}$,

• (d) ${\mathcal{A}}_{0,{\varsigma }_i}{({\sigma }_j^{(1)}+{\sigma }_j^{(2)}+{\sum }_{i=1}^m{\varsigma }_i{\nu }_j^{(i)})}^{p/p_b^{\ast }}\le {\eta }_j^{(1)}+{\eta }_j^{(2)}$, ${\mathcal{A}}_0{({\sigma }_j^{(1)})}^{p/p_b^{\ast }}\le {\eta }_j^{(1)}$, ${\mathcal{A}}_0{({\sigma }_j^{(2)})}^{p/p_b^{\ast }}\le {\eta }_j^{(2)}$,

• (e) ${\mathcal{A}}_{\mu ,{\varsigma }_i}{({\sigma }_0^{(1)}+{\sigma }_0^{(2)}+{\sum }_{i=1}^m{\varsigma }_i{\nu }_0^{(i)})}^{p/p_b^{\ast }}\le {\eta }_0^{(1)}+{\eta }_0^{(2)}-\mu ({\gamma }_0^{(1)}+{\gamma }_0^{(2)})$, ${\mathcal{A}}_{\mu }{({\sigma }_0^{(1)})}^{p/p_b^{\ast }}\le {\eta }_0^{(1)}-\mu {\gamma }_0^{(1)}$, ${\mathcal{A}}_{\mu }{({\sigma }_0^{(2)})}^{p/p_b^{\ast }}\le {\eta }_0^{(2)}-\mu {\gamma }_0^{(2)}$,

where $\delta_{x_{j}}$, $j\in \mathcal{J}\cup \{0\}$, is the Dirac mass of 1 concentrated at $x_{j}\in \overline{\Omega}$.

Proof

The proof is similar to that of the concentration compactness principle in Refs. [26, 27] (see also [19, Lemma 2.2]) and is omitted here. □

In order to find critical points of ℰ, we need the following local $(PS)_{c}$ condition.

Lemma 3.4

Suppose that (K.1) and (K.2) hold. Then the $(PS)_{c}$ condition in $(W_{a, G}^{1, p}(\Omega))^{2}$ holds for ℰ if

$$c<c_0^{\ast }\triangleq \frac{a+1-b}{N}min\{{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}{K}_{+}{(0)}^{\frac{-p}{p_b^{\ast }-p}},|G|{\mathcal{A}}_{0,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}{\parallel K_{+}\parallel }_{\mathrm{\infty }}^{\frac{-p}{p_b^{\ast }-p}}\}.$$

(3.7)

Proof

The proof is similar to that in [12, Proposition 2]. We sketch the argument here for completeness. Let $\{(u_{n}, v_{n})\}\subset(W_{a, G}^{1, p}(\Omega))^{2}$ be a $(PS)_{c}$ sequence for ℰ with $c< c_{0}^{\ast}$. Then we easily see from (2.1) and (3.7) that $\{(u_{n}, v_{n})\}$ is bounded in $(W_{a, G}^{1, p}(\Omega))^{2}$, and we may assume that $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(W_{a, G}^{1, p}(\Omega))^{2}$. In view of Lemma 3.3, there exist measures $\eta^{(1)}$, $\eta^{(2)}$, $\sigma^{(1)}$, $\sigma^{(2)}$, $\nu^{(i)}$ ($i=1,\ldots,m$), $\gamma^{(1)}$ and $\gamma^{(2)}$ such that relations (a)-(e) of this lemma hold. Let $x_{j}\neq0$ be a singular point of measures $\eta^{(1)}$, $\eta^{(2)}$, $\sigma^{(1)}$, $\sigma^{(2)}$ and $\nu^{(i)}$ ($i=1,\ldots,m$). We define two functions $\phi_{1}, \phi_{2}\in \mathcal{C}_{0}^{\infty}(\Omega)$ such that $\phi_{1}=\phi_{2}=1$ in $B_{\epsilon}(x_{j})$, $\phi_{1}=\phi_{2}=0$ on $\Omega\backslash B_{2\epsilon}(x_{j})$ and $|\nabla\phi_{1}|\leq 2/\epsilon$, $|\nabla\phi_{2}|\leq2/\epsilon$. By Lemma 3.1, ${lim}_{n\to \mathrm{\infty }}〈{\mathcal{E}}^{\prime }(u_n,v_n),(u_n{\varphi }_1,v_n{\varphi }_2)〉=0$, and consequently, combining (2.1) and the Hölder inequality, we have

$$\begin{aligned}& \int_{\Omega} \bigl(\phi_{1}d \eta^{(1)}+\phi_{2}d\eta^{(2)} \bigr) -\int _{\Omega}K(x) \Biggl\{ \phi_{1}\,d\sigma^{(1)}+ \phi_{2}\,d\sigma^{(2)} +\sum_{i=1}^{m} \frac{\varsigma_{i} }{p_{b}^{\ast}} (\alpha_{i}\phi_{1} + \beta_{i}\phi_{2} )\,d\nu \Biggr\} \\ & \quad\quad{}-\int_{\Omega}\mu \bigl(\phi_{1}\,d \gamma^{(1)}+\phi_{2}\,d\gamma^{(2)} \bigr) \\ & \quad\leq\limsup _{n\rightarrow\infty}\int_{\Omega}\frac {|u_{n}||\nabla u_{n}|^{p-1}|\nabla\phi_{1}|+|v_{n}||\nabla v_{n}|^{p-1}|\nabla\phi_{2}|}{|x|^{ap}}\,dx \\ & \quad\leq\sup_{n\geq1} \biggl(\int_{\Omega}|x|^{-ap}| \nabla u_{n}|^{p}\,dx \biggr)^{\frac{p-1}{p}}\limsup _{n\rightarrow\infty} \biggl(\int_{\Omega}|x|^{-ap}|u_{n}|^{p}| \nabla\phi_{1}|^{p}\,dx \biggr)^{\frac {1}{p}} \\ & \quad\quad{}+\sup_{n\geq1} \biggl(\int_{\Omega}|x|^{-ap}| \nabla v_{n}|^{p}\,dx \biggr)^{\frac{p-1}{p}}\limsup _{n\rightarrow\infty} \biggl(\int_{\Omega}|x|^{-ap}|v_{n}|^{p}| \nabla\phi_{2}|^{p}\,dx \biggr)^{\frac {1}{p}} \\ & \quad\leq C \biggl\{ \biggl(\int_{\Omega} |x|^{-ap}|u|^{p}| \nabla\phi_{1}|^{p}\,dx \biggr)^{\frac{1}{p}}+ \biggl(\int _{\Omega} |x|^{-ap}|v|^{p}|\nabla \phi_{2}|^{p}\,dx \biggr)^{\frac{1}{p}} \biggr\} \\ & \quad\leq C \biggl\{ \biggl(\int_{B_{2\epsilon}(x_{j})} \frac{|u|^{p^{\ast}}}{|x|^{ap^{\ast}}}\,dx \biggr)^{\frac{1}{p^{\ast}}} \biggl(\int_{\Omega} |\nabla \phi_{1}|^{N} \biggr)^{\frac{1}{N}} \\ & \quad\quad{}+ \biggl(\int _{B_{2\epsilon}(x_{j})} \frac{|v|^{p^{\ast}}}{|x|^{ap^{\ast}}}\,dx \biggr)^{\frac{1}{p^{\ast}}} \biggl(\int _{\Omega} |\nabla\phi_{2}|^{N} \biggr)^{\frac{1}{N}} \biggr\} \\ & \quad\leq C \biggl\{ \biggl(\int_{B_{2\epsilon}(x_{j})}|x|^{-ap}|\nabla u|^{p}\,dx \biggr)^{\frac{1}{p}}+ \biggl(\int_{B_{2\epsilon}(x_{j})} |x|^{-ap}|\nabla v|^{p}\,dx \biggr)^{\frac{1}{p}} \biggr\} . \end{aligned}$$

(3.8)

Taking limits as $\epsilon\rightarrow0$, we conclude from (3.8) and the relations (a)-(c) of Lemma 3.3 that

$$ K(x_{j}) \Biggl(\sigma_{j}^{(1)}+ \sigma_{j}^{(2)} +\sum_{i=1}^{m} \varsigma_{i}\nu_{j}^{(i)} \Biggr)\geq \eta_{j}^{(1)}+\eta_{j}^{(2)}. $$

(3.9)

The above inequality means that the concentration of the measures $\sigma^{(1)}$, $\sigma^{(2)}$ and $\nu^{(i)}$ cannot occur at points where $K(x_{j})\leq0$. More exactly, if $K(x_{j})\leq0$ then $\eta_{j}^{(1)}=\eta_{j}^{(2)}=\sigma_{j}^{(1)}=\sigma_{j}^{(2)} =\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)}=0$. Consequently, we deduce from (3.9) and (d) of Lemma 3.3 that either

• (i) $\sigma_{j}^{(1)}=\sigma_{j}^{(2)} =\sum_{i=1}^{m}\varsigma_{i}\nu_{j}^{(i)}=0$ or

• (ii) ${\sigma }_j^{(1)}+{\sigma }_j^{(2)}+{\sum }_{i=1}^m{\varsigma }_i{\nu }_j^{(i)}\ge {({\mathcal{A}}_{0,{\varsigma }_i}/K(x_j))}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}\ge {({\mathcal{A}}_{0,{\varsigma }_i}/{\parallel K_{+}\parallel }_{\mathrm{\infty }})}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}$.

For the point $x=0$, similarly to the case $x_{j}\neq0$, we get

$$\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu \bigl(\gamma_{0}^{(1)}+\gamma_{0}^{(2)} \bigr) -K(0) \Biggl(\sigma_{0}^{(1)}+\sigma_{0}^{(2)} +\sum_{i=1}^{m}\varsigma_{i} \nu_{0}^{(i)} \Biggr)\leq0. $$

This, combined with (e) of Lemma 3.3, implies that either

• (iii) $\sigma_{0}^{(1)}=\sigma_{0}^{(2)} =\sum_{i=1}^{m}\varsigma_{i}\nu_{0}^{(i)}=0$ or

• (iv) ${\sigma }_0^{(1)}+{\sigma }_0^{(2)}+{\sum }_{i=1}^m{\varsigma }_i{\nu }_0^{(i)}\ge {({\mathcal{A}}_{\mu ,{\varsigma }_i}/K_{+}(0))}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}$.

We now rule out cases (ii) and (iv). For every continuous nonnegative function ψ such that $0\leq\psi(x)\leq1$ on Ω, we obtain from (3.1) and (3.2) that

$$\begin{array}{rcl}c& =& \underset{n\to \mathrm{\infty }}{lim}(\mathcal{E}(u_n,v_n)-\frac{1}{p_b^{\ast }}〈{\mathcal{E}}^{\prime }(u_n,v_n),(u_n,v_n)〉)\\ & =& (\frac{1}{p}-\frac{1}{p_b^{\ast }})\underset{n\to \mathrm{\infty }}{lim}{\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }{u}_n|}^p+{|x|}^{-ap}{|\mathrm{\nabla }{v}_n|}^p-\mu \frac{{|u_n|}^p+{|v_n|}^p}{{|x|}^{p(a+1)}})dx\\ & \ge & \frac{a+1-b}{N}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }{u}_n|}^p+{|x|}^{-ap}{|\mathrm{\nabla }{v}_n|}^p-\mu \frac{{|u_n|}^p+{|v_n|}^p}{{|x|}^{p(a+1)}})\psi (x)dx.\end{array}$$

If (ii) occurs, then the set $\mathcal{J}$ must be finite because the measures $\sigma^{(1)}$, $\sigma^{(2)}$ and $\nu^{(i)}$ ($i=1,\ldots,m$) are bounded. Since functions $(u_{n}, v_{n})$ are G-symmetric, the measures $\sigma^{(1)}$, $\sigma^{(2)}$ and $\nu^{(i)}$ ($i=1,\ldots,m$) must be G-invariant. This implies that if $x_{j}\neq0$ is a singular point of $\sigma^{(1)}$, $\sigma^{(2)}$ and $\nu^{(i)}$ ($i=1,\ldots,m$), so is $gx_{j}$ for each $g\in G$, and the mass of $\sigma^{(1)}$, $\sigma^{(2)}$ and $\nu^{(i)}$ ($i=1,\ldots,m$) concentrated at $gx_{j}$ is the same for each $g\in G$. If we assume the existence of $j\in \mathcal{J}$ with $x_{j}\neq0$ such that (ii) holds, then we choose ψ with compact support so that $\psi(gx_{j})=1$ for each $g\in G$ and we get

$$\begin{array}{rcl}c& \ge & \frac{a+1-b}{N}|G|({\eta }_j^{(1)}+{\eta }_j^{(2)})\ge \frac{a+1-b}{N}|G|{\mathcal{A}}_{0,{\varsigma }_i}{({\sigma }_j^{(1)}+{\sigma }_j^{(2)}+\sum _{i=1}^m{\varsigma }_i{\nu }_j^{(i)})}^{\frac{p}{p_b^{\ast }}}\\ & \ge & \frac{a+1-b}{N}|G|{\mathcal{A}}_{0,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}{\parallel K_{+}\parallel }_{\mathrm{\infty }}^{\frac{-p}{p_b^{\ast }-p}},\end{array}$$

a contradiction with (3.7). Similarly, if (iv) holds for $x=0$, we choose ψ with compact support so that $\psi(0)=1$ and we find

$$\begin{array}{rcl}c& \ge & \frac{a+1-b}{N}({\eta }_0^{(1)}+{\eta }_0^{(2)}-\mu {\gamma }_0^{(1)}-\mu {\gamma }_0^{(2)})\ge \frac{a+1-b}{N}{\mathcal{A}}_{\mu ,{\varsigma }_i}{({\sigma }_0^{(1)}+{\sigma }_0^{(2)}+\sum _{i=1}^m{\varsigma }_i{\nu }_0^{(i)})}^{\frac{p}{p_b^{\ast }}}\\ & \ge & \frac{a+1-b}{N}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}{K}_{+}{(0)}^{\frac{-p}{p_b^{\ast }-p}},\end{array}$$

which contradicts (3.7). Thus $\sigma_{j}^{(1)}=\sigma_{j}^{(2)}=\nu_{j}^{(i)}=0$ ($i=1,\ldots,m$) for all $j\in \mathcal{J}\cup \{0\}$, and therefore we get

$$\lim_{n\rightarrow\infty}\int_{\Omega} \frac{|u_{n}|^{p_{b}^{\ast}}+|v_{n}|^{p_{b}^{\ast}} +\sum_{i=1}^{m}\varsigma_{i}|u_{n}|^{\alpha_{i}}|v_{n}|^{\beta_{i}}}{ |x|^{bp_{b}^{\ast}}}\,dx =\int_{\Omega}\frac{|u|^{p_{b}^{\ast}}+|v|^{p_{b}^{\ast}} +\sum_{i=1}^{m}\varsigma_{i}|u|^{\alpha_{i}}|v|^{\beta_{i}}}{ |x|^{bp_{b}^{\ast}}}\,dx. $$

Finally, since ${lim}_{n\to \mathrm{\infty }}〈{\mathcal{E}}^{\prime }(u_n,v_n)-{\mathcal{E}}^{\prime }(u,v),(u_n-u,v_n-v)〉=0$, we naturally deduce that $(u_{n}, v_{n})\rightarrow(u, v)$ in $(W_{a}^{1, p}(\Omega))^{2}$. □

As an immediate consequence of Lemma 3.4 we have the following result.

Corollary 3.1

If $K_{+}(0)=0$ and $|G|=+\infty$, then the functional ℰ satisfies $(PS)_{c}$ condition for every $c\in\mathbb{R}$.

Proof of Theorem 2.1

Firstly, we choose $\epsilon>0$ such that assumption (2.11) holds, where $V_{\epsilon}=\phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$ fulfills (3.12)-(3.15). In view of (K.1), we find from (3.1) and (3.3) that

$$\mathcal{E}(u,v)\ge \frac{1}{p}{\parallel (u,v)\parallel }_{\mu }^p-\frac{1}{p_b^{\ast }}{\parallel K\parallel }_{\mathrm{\infty }}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{-\frac{p_b^{\ast }}{p}}{\parallel (u,v)\parallel }_{\mu }^{p_b^{\ast }}.$$

Therefore there exist constants $\alpha_{0}>0$ and $\rho>0$ such that $\mathcal{E}(u,v)\ge {\alpha }_0$ for all $\|(u, v)\|_{\mu}=\rho$. Furthermore, if we set $u=V_{\epsilon}$, $v=\tau_{\min}V_{\epsilon}$ and

$$\begin{array}{rcl}\mathrm{\Phi }(t)& =& \mathcal{E}(t{V}_{ϵ},t{\tau }_{min}{V}_{ϵ})\\ & =& \frac{t^p}{p}(1+{\tau }_{min}^p){\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }{V}_{ϵ}|}^p-\mu \frac{{|V_{ϵ}|}^p}{{|x|}^{p(a+1)}})dx\\ & & -\frac{t^{p_b^{\ast }}}{p_b^{\ast }}(1+{\tau }_{min}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{\tau }_{min}^{{\beta }_i}){\int }_{\mathrm{\Omega }}K(x){|x|}^{-b{p}_b^{\ast }}{|V_{ϵ}|}^{p_b^{\ast }}dx\end{array}$$

with $t\geq0$, then we easily deduce that $\Phi(t)$ has a unique maximum in positive t at some $\overline{t}$ at which $d\Phi(t)/dt$ becomes zero. A simple computation gives us this value

$$\overline{t}= \biggl\{ \frac{ (1+\tau_{\min}^{p} ) \int_{\Omega} (|x|^{-ap}|\nabla V_{\epsilon}|^{p}-\mu\frac{|V_{\epsilon}|^{p}}{|x|^{p(a+1)}} )\,dx}{ (1+\tau_{\min}^{p_{b}^{\ast}} +\sum_{i=1}^{m}\varsigma_{i}\tau_{\min}^{\beta_{i}} ) \int_{\Omega}K(x)|x|^{-bp_{b}^{\ast}}|V_{\epsilon}|^{p_{b}^{\ast}}\,dx} \biggr\} ^{\frac{1}{p_{b}^{\ast}-p}}. $$

Consequently, we obtain from (3.4) and (3.5) that

$$\begin{array}{rcl}\underset{t\ge 0}{max}\mathrm{\Phi }(t)& =& \mathcal{E}(\overline{t}{V}_{ϵ},\overline{t}{\tau }_{min}{V}_{ϵ})\\ & =& \frac{a+1-b}{N}{\left\{\frac{(1+{\tau }_{min}^p){\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }{V}_{ϵ}|}^p-\mu \frac{{|V_{ϵ}|}^p}{{|x|}^{p(a+1)}})dx}{{[(1+{\tau }_{min}^{p_b^{\ast }}+{\sum }_{i=1}^m{\varsigma }_i{\tau }_{min}^{{\beta }_i}){\int }_{\mathrm{\Omega }}K(x){|x|}^{-b{p}_b^{\ast }}{|V_{ϵ}|}^{p_b^{\ast }}dx]}^{\frac{p}{p_b^{\ast }}}}\right\}}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}\\ & =& \frac{a+1-b}{N}{\left\{\frac{h({\tau }_{min}){\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }{V}_{ϵ}|}^p-\mu \frac{{|V_{ϵ}|}^p}{{|x|}^{p(a+1)}})dx}{{({\int }_{\mathrm{\Omega }}K(x){|x|}^{-b{p}_b^{\ast }}{|V_{ϵ}|}^{p_b^{\ast }}dx)}^{\frac{p}{p_b^{\ast }}}}\right\}}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}.\end{array}$$

(3.10)

On the other hand, since $\mathcal{E}(t{V}_{ϵ},t{\tau }_{min}{V}_{ϵ})\to -\mathrm{\infty }$ as $t\rightarrow\infty$, there exists $t_{0}>0$ such that $\|(t_{0}V_{\epsilon}, t_{0}\tau_{\min}V_{\epsilon})\|_{\mu}>\rho$ and $\mathcal{E}(t_0{V}_{ϵ},t_0{\tau }_{min}{V}_{ϵ})<0$. Now we define

$$c_0=\underset{\gamma \in \mathrm{\Gamma }}{inf}\underset{t\in [0,1]}{max}\mathcal{E}(\gamma (t)),$$

(3.11)

where $\mathrm{\Gamma }=\{\gamma \in \mathcal{C}([0,1],{(W_{a,G}^{1,p}(\mathrm{\Omega }))}^2);\gamma (0)=(0,0),\mathcal{E}(\gamma (1))<0,{\parallel \gamma (1)\parallel }_{\mu }>\rho \}$. In view of Lemma 3.2, we obtain from (2.11), (3.7), (3.10) and (3.11) that

$$\begin{array}{rcl}{c}_0& \le & \mathcal{E}(\overline{t}{V}_{ϵ},\overline{t}{\tau }_{min}{V}_{ϵ})\\ & =& \frac{a+1-b}{N}{\left\{\frac{h({\tau }_{min}){\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }{V}_{ϵ}|}^p-\mu \frac{{|V_{ϵ}|}^p}{{|x|}^{p(a+1)}})dx}{{({\int }_{\mathrm{\Omega }}K(x){|x|}^{-b{p}_b^{\ast }}{|V_{ϵ}|}^{p_b^{\ast }}dx)}^{\frac{p}{p_b^{\ast }}}}\right\}}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}\\ & \le & \frac{a+1-b}{N}{\left\{\frac{h({\tau }_{min}){\int }_{\mathrm{\Omega }}({|x|}^{-ap}{|\mathrm{\nabla }{V}_{ϵ}|}^p-\mu \frac{{|V_{ϵ}|}^p}{{|x|}^{p(a+1)}})dx}{{({\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}max\{K_{+}(0),{|G|}^{\frac{p-p_b^{\ast }}{p}}{({\mathcal{A}}_0/{\mathcal{A}}_{\mu })}^{-\frac{p_b^{\ast }}{p}}{\parallel K_{+}\parallel }_{\mathrm{\infty }}\})}^{\frac{p}{p_b^{\ast }}}}\right\}}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}\\ & =& \frac{a+1-b}{N}min\{{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}{K}_{+}{(0)}^{\frac{-p}{p_b^{\ast }-p}},|G|{\mathcal{A}}_{0,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}{\parallel K_{+}\parallel }_{\mathrm{\infty }}^{\frac{-p}{p_b^{\ast }-p}}\}=c_0^{\ast }.\end{array}$$

If $c_{0}< c_{0}^{\ast}$, then by Lemma 3.4 the $(PS)_{c}$ condition holds and the conclusion follows by the mountain pass theorem in Ref. [28] (see also [29]). If $c_{0}=c_{0}^{\ast}$, then $\gamma(t)=(tt_{0}V_{\epsilon}, tt_{0}\tau_{\min}V_{\epsilon})$, with $0\leq t\leq1$, is a path in Γ such that ${max}_{t\in [0,1]}\mathcal{E}(\gamma (t))=c_0$. Consequently, either ${\mathrm{\Phi }}^{\prime }(\overline{t})={\mathcal{E}}^{\prime }(\overline{t}{V}_{ϵ},\overline{t}{\tau }_{min}{V}_{ϵ})=0$ and we are done, or γ can be deformed to a path $\widetilde{\gamma}\in\Gamma$ with ${max}_{t\in [0,1]}\mathcal{E}(\tilde{\gamma }(t))<c_0$, which is impossible. Hence we obtain a nontrivial solution $(u_{0}, v_{0})\in(W_{a, G}^{1, p}(\Omega))^{2}$ of problem $({\mathcal{P}}_0^K)$. In the following, we have just to show that the solution $(u_{0}, v_{0})$ can be chosen to be positive on Ω. Since $\mathcal{E}(u_0,v_0)=\mathcal{E}(|u_0|,|v_0|)$ and

$$\begin{array}{rcl}0& =& 〈{\mathcal{E}}^{\prime }(u_0,v_0),(u_0,v_0)〉\\ & =& {\parallel (u_0,v_0)\parallel }_{\mu }^p-{\int }_{\mathrm{\Omega }}\frac{K(x)}{{|x|}^{b{p}_b^{\ast }}}({|u_0|}^{p_b^{\ast }}+{|v_0|}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{|u_0|}^{{\alpha }_i}{|v_0|}^{{\beta }_i})dx,\end{array}$$

we find that $\int_{\Omega}K(x)|x|^{-bp_{b}^{\ast}} (|u_{0}|^{p_{b}^{\ast}}+|v_{0}|^{p_{b}^{\ast}} +\sum_{i=1}^{m}\varsigma_{i} |u_{0}|^{\alpha_{i}}|v_{0}|^{\beta_{i}})\,dx=\|(u_{0}, v_{0})\|_{\mu}^{p}>0$. This means $c_0=\mathcal{E}(|u_0|,|v_0|)={max}_{t\ge 0}\mathcal{E}(t|u_0|,t|v_0|)$. Consequently, either $(|u_{0}|, |v_{0}|)$ is a critical point of ℰ or $\gamma(t)=(tt_{0}|u_{0}|, tt_{0}|v_{0}|)$, with $\mathcal{E}(t_0|u_0|,t_0|v_0|)<0$, can be deformed, as the first part of the proof, to a path $\widetilde{\gamma}(t)$ with ${max}_{t\in [0,1]}\mathcal{E}(\tilde{\gamma }(t))<c_0$, which is impossible. Therefore, we may assume that $u_{0}\geq0$, $v_{0}\geq0$ on Ω and the fact that $u_{0}>0$, $v_{0}>0$ on Ω follows by the strong maximum principle. □

Proof of Corollary 2.1

Let $y_{\epsilon}(x)$ be the extremal function satisfying (2.4)-(2.10). Choose $\phi\in\mathcal{C}_{0}^{1}(\Omega)$ so that $\phi\geq0$ on Ω and $\phi(x)=1$ on $B_{\varrho}(0)$, with $\varrho>0$ to be determined. Applying the methods in Refs. [3, 29], we get from (2.4)-(2.10) that

$$\begin{aligned}& \|\phi y_{\epsilon}\|_{\mu}^{p}=\int _{\Omega} \biggl(|x|^{-ap}\bigl|\nabla(\phi y_{\epsilon})\bigr|^{p}-\mu\frac{|\phi y_{\epsilon}|^{p}}{|x|^{p(a+1)}} \biggr)\,dx=1+O \bigl( \epsilon ^{p(l_{2}+a+1)-N} \bigr), \end{aligned}$$

(3.12)

$${\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}{|\varphi y_{ϵ}|}^{p_b^{\ast }}dx={\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}+O\left({ϵ}^{p_b^{\ast }(l_2+b)-N}\right),$$

(3.13)

$$\begin{aligned}& \int_{\Omega}|x|^{-dp_{d}^{\ast}}|\phi y_{\epsilon}|^{q}\,dx= \begin{cases} O (\epsilon^{q(l_{2}+a+1-\frac{N}{p})} ),& 1\leq q< \frac{N-dp_{d}^{\ast}}{l_{2}}, \\ O (\epsilon^{q(l_{2}+a+1-\frac{N}{p})}|\ln\epsilon| ), &q=\frac{N-dp_{d}^{\ast}}{l_{2}}, \\ O (\epsilon^{(p_{d}^{\ast}-q)(\frac{N}{p}-a-1)} ),& \frac{N-dp_{d}^{\ast}}{l_{2}}<q<p_{d}^{\ast}. \end{cases} \end{aligned}$$

(3.14)

Set $V_{\epsilon}=\phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$; then by (3.12) and (3.13) we derive

$${\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}dx={\int }_{\mathrm{\Omega }}\frac{{|\varphi y_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}{\parallel \varphi y_{ϵ}\parallel }_{\mu }^{p_b^{\ast }}}dx={\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}+O\left({ϵ}^{p(l_2+a+1)-N}\right).$$

(3.15)

Let us now choose $\varrho>0$ so that $K(x)\geq K(0)+\gamma_{0}|x|^{\vartheta}$ for $|x|\leq\varrho$. Then we obtain from (3.15) that

$${\int }_{\mathrm{\Omega }}K(x)\frac{{|V_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}dx={\int }_{\mathrm{\Omega }}(K(x)-K(0)){|x|}^{-b{p}_b^{\ast }}{|V_{ϵ}|}^{p_b^{\ast }}dx+K(0){\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}+O\left({ϵ}^{p(l_2+a+1)-N}\right).$$

It is sufficient to show that

$$ \int_{\Omega} \bigl(K(x)-K(0) \bigr)|x|^{-bp_{b}^{\ast}}|V_{\epsilon }|^{p_{b}^{\ast}}\,dx +O \bigl( \epsilon^{p(l_{2}+a+1)-N} \bigr)\geq0 $$

(3.16)

for sufficiently small $\epsilon>0$. We find that

$$\begin{aligned}& \int_{\Omega} \bigl(K(x)-K(0) \bigr)|x|^{-bp_{b}^{\ast}}|V_{\epsilon }|^{p_{b}^{\ast}}\,dx \\& \quad=\int_{|x|\leq\varrho} \bigl(K(x)-K(0) \bigr)|x|^{-bp_{b}^{\ast }}|V_{\epsilon}|^{p_{b}^{\ast}}\,dx +\int_{|x|\geq\varrho} \bigl(K(x)-K(0) \bigr)|x|^{-bp_{b}^{\ast }}|V_{\epsilon}|^{p_{b}^{\ast}}\,dx \\& \quad\geq\gamma_{0}\int_{|x|\leq\varrho} \frac{|x|^{\vartheta-bp_{b}^{\ast}}|y_{\epsilon}|^{p_{b}^{\ast}}}{ \|\phi y_{\epsilon}\|_{\mu}^{p_{b}^{\ast}}}\,dx+ \int_{|x|\geq\varrho}\frac{(K(x)-K(0))|\phi y_{\epsilon}|^{p_{b}^{\ast}}}{|x|^{bp_{b}^{\ast}}\|\phi y_{\epsilon}\|_{\mu}^{p_{b}^{\ast}}}\,dx=J_{1}+J_{2}. \end{aligned}$$

For $\epsilon>0$ small enough, we deduce from (2.4)-(2.10), (3.12) and the fact $N-1+\vartheta-(b+l_{1})p_{b}^{\ast}>-1$, $N-1+\vartheta-(b+l_{2})p_{b}^{\ast}<-1$ that

$$\begin{aligned} J_{1} =&\gamma_{0}\int_{|x|\leq\varrho} \frac{|x|^{\vartheta-bp_{b}^{\ast}}|y_{\epsilon}|^{p_{b}^{\ast}}}{ \|\phi y_{\epsilon}\|_{\mu}^{p_{b}^{\ast}}}\,dx \\ =&\frac{\gamma_{0}C^{p_{b}^{\ast}}\epsilon^{-\xi p_{b}^{\ast}}}{ (1+O (\epsilon^{p(l_{2}+a+1)-N} ) ) ^{\frac{p_{b}^{\ast}}{p}}}\int_{|x|\leq\varrho} |x|^{\vartheta-b p_{b}^{\ast}} \biggl[U_{\mu} \biggl(\frac{|x|}{\epsilon} \biggr) \biggr]^{p_{b}^{\ast}}\,dx \\ =&\frac{C\epsilon^{\vartheta}}{ (1+O (\epsilon^{p(l_{2}+a+1)-N} ) )^{\frac{p_{b}^{\ast}}{p}}} \int_{\frac{|x|}{\epsilon}\leq\frac{\varrho}{\epsilon}} \biggl(\frac{|x|}{\epsilon} \biggr) ^{\vartheta-(b+l_{2})p_{b}^{\ast}} \biggl[ \biggl(\frac{|x|}{\epsilon} \biggr)^{l_{2}} U_{\mu} \biggl(\frac{|x|}{\epsilon} \biggr) \biggr]^{p_{b}^{\ast}} \,d \biggl(\frac{x}{\epsilon} \biggr) \\ \geq& C\epsilon^{\vartheta} \biggl\{ \int_{|x|\leq 1} \frac{ (|x|^{l_{1}} U_{\mu}(|x|) )^{p_{b}^{\ast}}}{ |x|^{-\vartheta+(b+l_{1})p_{b}^{\ast}}} \,dx+\int_{1< |x|\leq \frac{\varrho}{\epsilon}}\frac{ (|x|^{l_{2}} U_{\mu}(|x|) )^{p_{b}^{\ast}}}{ |x|^{-\vartheta+(b+l_{2})p_{b}^{\ast}}} \,dx \biggr\} \\ \geq&\overline{C}_{1}\epsilon^{\vartheta},\quad \vartheta\in \bigl(0, p(l_{2}+a+1)-N \bigr) \end{aligned}$$

and

$$\begin{aligned} |J_{2}| \leq&\int_{|x|\geq\varrho}\frac {|K(x)-K(0)||\phi y_{\epsilon}|^{p_{b}^{\ast}}}{|x|^{bp_{b}^{\ast}}\|\phi y_{\epsilon}\|_{\mu}^{p_{b}^{\ast}}}\,dx \\ \leq& C\int_{\frac{|x|}{\epsilon}\geq\frac{\varrho}{\epsilon}} \biggl(\frac{|x|}{\epsilon} \biggr) ^{-(b+l_{2})p_{b}^{\ast}} \biggl[ \biggl(\frac{|x|}{\epsilon} \biggr)^{l_{2}} U_{\mu} \biggl(\frac{|x|}{\epsilon} \biggr) \biggr]^{p_{b}^{\ast}} \,d \biggl(\frac{x}{\epsilon} \biggr) \\ \leq& C\int_{\varrho\epsilon^{-1}}^{+\infty}r^{N-1 -(b+l_{2})p_{b}^{\ast}}\,dr \leq \overline{C}_{2}\epsilon^{p_{b}^{\ast}(l_{2}+a+1-\frac{N}{p})}, \end{aligned}$$

where $\overline{C}_{1}>0$ and $\overline{C}_{2}>0$ are constants independent of ϵ. In view of $0<\vartheta<p(l_{2}+a+1)-N<p_{b}^{\ast}(l_{2}+a+1-\frac{N}{p})$, we find that inequality (3.16) holds as $\epsilon>0$ sufficiently small. Therefore, we conclude from (2.12), (3.15) and (3.16) that

$$\begin{array}{rcl}{\int }_{\mathrm{\Omega }}K(x)\frac{{|V_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}dx& =& {\int }_{\mathrm{\Omega }}(K(x)-K(0))\frac{{|V_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}dx+K(0){\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}+O\left({ϵ}^{p(l_2+a+1)-N}\right)\\ & \ge & K(0){\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}\ge {\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}max\{K_{+}(0),{|G|}^{\frac{p-p_b^{\ast }}{p}}{({\mathcal{A}}_0/{\mathcal{A}}_{\mu })}^{-\frac{p_b^{\ast }}{p}}{\parallel K_{+}\parallel }_{\mathrm{\infty }}\}>0.\end{array}$$

This, combined with Theorem 2.1, implies the conclusion. □

To prove Theorem 2.2 we need the following version of the symmetric mountain pass theorem (see [30, Theorem 9.12]).

Lemma 3.5

Let X be an infinite dimensional Banach space, and let $\mathcal{E}\in {\mathcal{C}}^1(X,\mathbb{R})$ be an even functional satisfying $(PS)_{c}$ condition for each c and $\mathcal{E}(0)=0$. Furthermore, we suppose that:

• (i) there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathcal{E}(w)\ge \overline{\alpha }$ for all $\|w\|=\rho$;

• (ii) there exists an increasing sequence of subspaces $\{X_{k}\}$ of X, with $\dim X_{k}=k$, such that for every k one can find a constant $R_{k}>0$ such that $\mathcal{E}(w)\le 0$ for all $w\in X_{k}$ with $\|w\|\geq R_{k}$.

Then ℰ possesses a sequence of critical values $\{c_{k}\}$ tending to ∞ as $k\rightarrow\infty$.

Proof of Theorem 2.2

Applying Lemma 3.5 with $X=(W_{a, G}^{1, p}(\Omega))^{2}$ and $(u, v)=w\in X$, we see from (K.1), (3.1) and (3.3) that

$$\mathcal{E}(u,v)\ge \frac{1}{p}{\parallel (u,v)\parallel }_{\mu }^p-\frac{1}{p_b^{\ast }}{\parallel K\parallel }_{\mathrm{\infty }}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{-\frac{p_b^{\ast }}{p}}{\parallel (u,v)\parallel }_{\mu }^{p_b^{\ast }}.$$

Since $p_{b}^{\ast}>p>1$, there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathcal{E}(u,v)\ge \overline{\alpha }$ for all $(u, v)$ with $\|(u, v)\|_{\mu}=\rho$. To find a suitable sequence of finite dimensional subspaces of $(W_{a, G}^{1, p}(\Omega))^{2}$, we set $\Omega_{+}=\{x\in\Omega; K(x)>0\}$. Since the set $\Omega_{+}$ is G-symmetric, we can define $(W_{a, G}^{1, p}(\Omega_{+}))^{2}$, which is the subspace of G-symmetric functions of $(W_{a}^{1, p}(\Omega_{+}))^{2}$ (see Section 2). By extending functions in $(W_{a, G}^{1, p}(\Omega_{+}))^{2}$ by 0 outside $\Omega_{+}$ we can assume that $(W_{a, G}^{1, p}(\Omega_{+}))^{2}\subset(W_{a, G}^{1, p}(\Omega))^{2}$. Let $\{X_{k}\}$ be an increasing sequence of subspaces of $(W_{a, G}^{1, p}(\Omega_{+}))^{2}$ with $\dim X_{k}=k$ for each k. Then there exists a constant $\iota(k)>0$ such that

$$\frac{1}{p_{b}^{\ast}}\int_{\Omega_{+}}\frac{K(x)}{|x|^{bp_{b}^{\ast}}} \Biggl(| \tilde{u}|^{p_{b}^{\ast}}+|\tilde{v}|^{p_{b}^{\ast}} +\sum _{i=1}^{m}\varsigma_{i} | \tilde{u}|^{\alpha_{i}}|\tilde{v}|^{\beta_{i}} \Biggr)\,dx\geq\iota(k) $$

for all $(\tilde{u}, \tilde{v})\in X_{k}$, with $\|(\tilde{u}, \tilde{v})\|_{\mu}=1$. Hence, if $(u, v)\in X_{k}\backslash\{(0, 0)\}$ then we write $(u, v)=t(\tilde{u}, \tilde{v})$ with $t=\|(u, v)\|_{\mu}$ and $\|(\tilde{u}, \tilde{v})\|_{\mu}=1$. Thus we obtain

$$\mathcal{E}(u,v)=\frac{t^p}{p}-\frac{t^{p_b^{\ast }}}{p_b^{\ast }}{\int }_{{\mathrm{\Omega }}_{+}}\frac{K(x)}{{|x|}^{b{p}_b^{\ast }}}({|\tilde{u}|}^{p_b^{\ast }}+{|\tilde{v}|}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{|\tilde{u}|}^{{\alpha }_i}{|\tilde{v}|}^{{\beta }_i})dx\le \frac{t^p}{p}-\iota (k)t^{p_b^{\ast }}\le 0$$

for t large enough. By Lemma 3.5 and Corollary 3.1 we conclude that there exists a sequence of critical values $c_{k}\rightarrow\infty$ as $k\rightarrow\infty$ and the results follow. □

Proof of Corollary 2.2

Since $K(x)$ is radially symmetric, namely $K(x)=K(|x|)$, we find that the corresponding group $G=O(\mathbb{N})$ and $|G|=+\infty$. According to Corollary 3.1, ℰ satisfies the $(PS)_{c}$ condition for every $c\in\mathbb{R}$. Consequently, by Theorem 2.2 we obtain the conclusion. □

4 Existence results for problem $({\mathcal{P}}_{\lambda }^{K_0})$

Throughout this section we assume that $\lambda>0$ and $K(x)\equiv K_{0}>0$ is a constant. Since we are interested in positive G-symmetric solutions of problem $({\mathcal{P}}_{\lambda }^{K_0})$, we define a functional ${\mathcal{F}}_{\lambda }:{(W_{a,G}^{1,p}(\mathrm{\Omega }))}^2\to \mathbb{R}$ given by

$$\begin{array}{rcl}{\mathcal{F}}_{\lambda }(u,v)& =& \frac{1}{p}{\int }_{\mathrm{\Omega }}(\frac{{|\mathrm{\nabla }u|}^p+{|\mathrm{\nabla }v|}^p}{{|x|}^{ap}}-\mu \frac{{|u|}^p+{|v|}^p}{{|x|}^{p(a+1)}})dx-\frac{\lambda }{q_1+q_2}{\int }_{\mathrm{\Omega }}\frac{{|u^{+}|}^{q_1}{|v^{+}|}^{q_2}}{{|x|}^{d{p}_d^{\ast }}}dx\\ & & -\frac{K_0}{p_b^{\ast }}{\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}({\left|u^{+}\right|}^{p_b^{\ast }}+{\left|v^{+}\right|}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{\left|u^{+}\right|}^{{\alpha }_i}{\left|v^{+}\right|}^{{\beta }_i})dx,\end{array}$$

(4.1)

where $q_{1} , q_{2}\geq1$, $1< p<q_{1}+q_{2}<p_{d}^{\ast}$, $u^{+}=\max\{0, u\}$ and $v^{+}=\max\{0, v\}$. By (2.1) and the Young inequality, we obtain

$$\begin{aligned} \int_{\Omega}\frac{|u^{+}| ^{q_{1}}|v^{+}|^{q_{2}}}{|x|^{dp_{d}^{\ast}}}\,dx \leq& \frac{q_{1}}{q_{1}+q_{2}}\int_{\Omega} \frac{|u|^{q_{1}+q_{2}}}{|x|^{dp_{d}^{\ast}}}\,dx+ \frac {q_{2}}{q_{1}+q_{2}}\int_{\Omega} \frac{|v|^{q_{1}+q_{2}}}{|x|^{dp_{d}^{\ast}}}\,dx \\ \leq& C \biggl[ \biggl(\int_{\Omega}|x|^{-ap}|\nabla u|^{p}\,dx \biggr)^{\frac{q_{1}+q_{2}}{p}}+ \biggl(\int_{\Omega}|x|^{-ap}| \nabla v|^{p}\,dx \biggr)^{\frac{q_{1}+q_{2}}{p}} \biggr] \\ \leq& C\bigl\| (u, v) \bigr\| _{\mu}^{q_{1}+q_{2}}. \end{aligned}$$

(4.2)

Hence we find from (4.2) that ${\mathcal{F}}_{\lambda }$ is well defined, ${\mathcal{F}}_{\lambda }\in {\mathcal{C}}^1({(W_{a,G}^{1,p}(\mathrm{\Omega }))}^2,\mathbb{R})$ and there exists a one-to-one correspondence between the weak solutions of $({\mathcal{P}}_{\lambda }^{K_0})$ and the critical points of ${\mathcal{F}}_{\lambda }$. Furthermore, an analogously symmetric criticality principle of Lemma 3.1 clearly holds; thus the weak solutions of problem $({\mathcal{P}}_{\lambda }^{K_0})$ are exactly the critical points of the functional ${\mathcal{F}}_{\lambda }$.

Lemma 4.1

Suppose that $\lambda>0$, $q_{1} , q_{2}\geq1$, $q_{1}+q_{2}\in (p, p_{d}^{\ast})$, $0<\varsigma_{i}<+\infty$, and $\alpha_{i} , \beta_{i}>1$ satisfy $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, m$). Then the $(PS)_{c}$ condition in $(W_{a, G}^{1, p}(\Omega))^{2}$ holds for ${\mathcal{F}}_{\lambda }$ if

$$c<\frac{a+1-b}{N}{K}_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}.$$

(4.3)

Proof

Let $\{(u_{n}, v_{n})\}\subset(W_{a, G}^{1, p}(\Omega))^{2}$ be a $(PS)_{c}$ sequence for ${\mathcal{F}}_{\lambda }$ with c satisfying (4.3). Note that $p< q_{1}+q_{2}<p_{d}^{\ast}\leq p_{b}^{\ast}$. Then by (4.1), there exists $n_{0}\geq1$ such that for $n\geq n_{0}$, we obtain

$$\begin{array}{rcl}c+1& \ge & {\mathcal{F}}_{\lambda }(u_n,v_n)-\frac{1}{q_1+q_2}〈{\mathcal{F}}_{\lambda }^{\prime }(u_n,v_n),(u_n,v_n)〉+\frac{1}{q_1+q_2}〈{\mathcal{F}}_{\lambda }^{\prime }(u_n,v_n),(u_n,v_n)〉\\ & =& (\frac{1}{p}-\frac{1}{q_1+q_2}){\parallel (u_n,v_n)\parallel }_{\mu }^p+o_n(1){\parallel (u_n,v_n)\parallel }_{\mu }\\ & & +(\frac{1}{q_1+q_2}-\frac{1}{p_b^{\ast }})K_0{\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}({\left|u_n^{+}\right|}^{p_b^{\ast }}+{\left|v_n^{+}\right|}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{\left|u_n^{+}\right|}^{{\alpha }_i}{\left|v_n^{+}\right|}^{{\beta }_i})dx\\ & \ge & (\frac{1}{p}-\frac{1}{q_1+q_2}){\parallel (u_n,v_n)\parallel }_{\mu }^p+o_n(1){\parallel (u_n,v_n)\parallel }_{\mu }.\end{array}$$

This implies that $\{(u_{n}, v_{n})\}$ is bounded in $(W_{a, G}^{1, p}(\Omega))^{2}$. Therefore, just as in Lemma 3.4, we may assume that $u_{n}\rightharpoonup u$, $v_{n}\rightharpoonup v$ in $W_{a, G}^{1, p}(\Omega)$ and in $L^{p_{b}^{\ast}}(\Omega, |x|^{-bp_{b}^{\ast}})$; moreover, $u_{n}\rightarrow u$, $v_{n}\rightarrow v$ in $L^{q_{1}+q_{2}}(\Omega, |x|^{-dp_{d}^{\ast}})$ for all $p< q_{1}+q_{2}<p_{d}^{\ast}$ (see [31, Theorem 2.1]) and a.e. on Ω. This means

$$ \int_{\Omega}\frac {|u_{n}^{+}|^{q_{1}}|v_{n}^{+}|^{q_{2}}}{|x|^{dp_{d}^{\ast}}}\,dx =\int _{\Omega}\frac{|u^{+}|^{q_{1}}|v^{+}|^{q_{2}}}{|x|^{dp_{d}^{\ast }}}\,dx+o_{n}(1). $$

(4.4)

A standard argument shows that $(u, v)$ is a critical point of ${\mathcal{F}}_{\lambda }$, and hence

$$\begin{array}{rcl}{\mathcal{F}}_{\lambda }(u,v)& =& (\frac{1}{p}-\frac{1}{p_b^{\ast }})K_0{\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}({\left|u^{+}\right|}^{p_b^{\ast }}+{\left|v^{+}\right|}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{\left|u^{+}\right|}^{{\alpha }_i}{\left|v^{+}\right|}^{{\beta }_i})dx\\ & & +(\frac{1}{p}-\frac{1}{q_1+q_2})\lambda {\int }_{\mathrm{\Omega }}\frac{{|u^{+}|}^{q_1}{|v^{+}|}^{q_2}}{{|x|}^{d{p}_d^{\ast }}}dx\ge 0.\end{array}$$

(4.5)

Now we set $\tilde{u}_{n}=u_{n}-u$ and $\tilde{v}_{n}=v_{n}-v$. Then, by the Brezis-Lieb lemma [32] and arguing as in [33, Lemma 2.1], we get

$$\begin{aligned}& \bigl\| (\tilde{u}_{n}, \tilde{v}_{n}) \bigr\| _{\mu}^{p}=\bigl\| (u_{n}, v_{n}) \bigr\| _{\mu}^{p}-\bigl\| (u, v)\bigr\| _{\mu}^{p}+o_{n}(1), \end{aligned}$$

(4.6)

$$\begin{aligned}& \int_{\Omega}|x|^{-bp_{b}^{\ast}}\bigl| \tilde{u}_{n}^{+}\bigr|^{p_{b}^{\ast}}\,dx =\int _{\Omega}|x|^{-bp_{b}^{\ast}}\bigl|u_{n}^{+}\bigr|^{p_{b}^{\ast}}\,dx -\int_{\Omega}|x|^{-bp_{b}^{\ast}}\bigl|u^{+}\bigr|^{p_{b}^{\ast}}\,dx+o_{n}(1), \end{aligned}$$

(4.7)

$$\begin{aligned}& \int_{\Omega}|x|^{-bp_{b}^{\ast}}\bigl| \tilde{v}_{n}^{+}\bigr|^{p_{b}^{\ast}}\,dx =\int _{\Omega}|x|^{-bp_{b}^{\ast}}\bigl|v_{n}^{+}\bigr|^{p_{b}^{\ast}}\,dx -\int_{\Omega}|x|^{-bp_{b}^{\ast}}\bigl|v^{+}\bigr|^{p_{b}^{\ast}}\,dx+o_{n}(1), \end{aligned}$$

(4.8)

$$\begin{aligned}& \int_{\Omega}\frac{|\tilde{u}_{n}^{+}|^{\alpha} |\tilde{v}_{n}^{+}|^{\beta}}{|x|^{bp_{b}^{\ast}}}\,dx =\int _{\Omega}\frac{|u_{n}^{+}|^{\alpha} |v_{n}^{+}|^{\beta}}{|x|^{bp_{b}^{\ast}}}\,dx -\int_{\Omega} \frac{|u^{+}|^{\alpha} |v^{+}|^{\beta}}{|x|^{bp_{b}^{\ast}}}\,dx+o_{n}(1). \end{aligned}$$

(4.9)

In view of ${\mathcal{F}}_{\lambda }(u_n,v_n)=c+o_n(1)$ and ${\mathcal{F}}_{\lambda }^{\prime }(u_n,v_n)=o_n(1)$, we get from (4.1), (4.4) and (4.6)-(4.9) that

$$\begin{array}{rcl}c+o_n(1)& =& {\mathcal{F}}_{\lambda }(u_n,v_n)\\ & =& {\mathcal{F}}_{\lambda }(u,v)+\frac{1}{p}{\parallel ({\tilde{u}}_n,{\tilde{v}}_n)\parallel }_{\mu }^p\\ & & -\frac{K_0}{p_b^{\ast }}{\int }_{\mathrm{\Omega }}{|x|}^{-b{p}_b^{\ast }}({\left|{\tilde{u}}_n^{+}\right|}^{p_b^{\ast }}+{\left|{\tilde{v}}_n^{+}\right|}^{p_b^{\ast }}+\sum _{i=1}^m{\varsigma }_i{\left|{\tilde{u}}_n^{+}\right|}^{{\alpha }_i}{\left|{\tilde{v}}_n^{+}\right|}^{{\beta }_i})dx+o_n(1)\end{array}$$

(4.10)

and

$$ \bigl\| (\tilde{u}_{n}, \tilde{v}_{n}) \bigr\| _{\mu}^{p} -K_{0}\int_{\Omega} |x|^{-bp_{b}^{\ast}} \Biggl(\bigl|\tilde{u}_{n}^{+}\bigr|^{p_{b}^{\ast}} +\bigl|\tilde{v}_{n}^{+}\bigr|^{p_{b}^{\ast}} +\sum _{i=1}^{m}\varsigma_{i} \bigl| \tilde{u}_{n}^{+}\bigr|^{\alpha_{i}}\bigl| \tilde{v}_{n}^{+}\bigr|^{\beta_{i}} \Biggr)\,dx=o_{n}(1). $$

(4.11)

Hence, for a subsequence $\{(\tilde{u}_{n}, \tilde{v}_{n})\}$, we have

$$\begin{aligned}& \bigl\| (\tilde{u}_{n}, \tilde{v}_{n}) \bigr\| _{\mu}^{p} \rightarrow\widetilde{l}\geq0 \quad\text{and}\\& K_{0}\int_{\Omega} |x|^{-bp_{b}^{\ast}} \Biggl(\bigl| \tilde{u}_{n}^{+}\bigr|^{p_{b}^{\ast}} +\bigl| \tilde{v}_{n}^{+}\bigr|^{p_{b}^{\ast}} +\sum _{i=1}^{m}\varsigma_{i} \bigl| \tilde{u}_{n}^{+}\bigr|^{\alpha_{i}}\bigl| \tilde{v}_{n}^{+}\bigr|^{\beta_{i}} \Biggr)\,dx\rightarrow \widetilde{l} \end{aligned}$$

as $n\rightarrow\infty$. It follows from (3.3) that ${\mathcal{A}}_{\mu ,{\varsigma }_i}{(\tilde{l}/K_0)}^{\frac{p}{p_b^{\ast }}}\le \tilde{l}$, which means either $\widetilde{l}=0$ or $\tilde{l}\ge K_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}$. If $\tilde{l}\ge K_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}$, then we deduce from (4.5), (4.10) and (4.11) that

$$c={\mathcal{F}}_{\lambda }(u,v)+(\frac{1}{p}-\frac{1}{p_b^{\ast }})\tilde{l}\ge \frac{a+1-b}{N}{K}_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}},$$

which contradicts (4.3). Consequently, we obtain $\|(\tilde{u}_{n}, \tilde{v}_{n})\|_{\mu}^{p}\rightarrow 0$ as $n\rightarrow\infty$, and hence $(u_{n}, v_{n})\rightarrow (u, v)$ in $(W_{a, G}^{1, p}(\Omega))^{2}$. The conclusion of this lemma follows. □

Lemma 4.2

Suppose that $\lambda>0$, $0<\varsigma_{i}<+\infty$, $\alpha_{i}$, $\beta_{i}>1$, $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, m$), and $q_{1} , q_{2}\geq1$ satisfy (2.13). Then there exists a pair of functions $(\overline{u}, \overline{v})\in(W_{a, G}^{1, p}(\Omega)\backslash\{0\})^{2}$ such that

$$\underset{t\ge 0}{sup}{\mathcal{F}}_{\lambda }(t\overline{u},t\overline{v})<\frac{a+1-b}{N}{K}_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}.$$

(4.12)

Proof

Recall that $\tau_{\min}>0$ in Lemma 3.2 and $V_{\epsilon}=\phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$, which satisfies (3.12)-(3.15). In the following, we will show that $(V_{\epsilon}, \tau_{\min}V_{\epsilon})$ satisfies (4.12) for $\epsilon>0$ sufficiently small. First, we consider the functions

$$\begin{array}{rcl}\mathrm{\Psi }(t)& =& {\mathcal{F}}_{\lambda }(t{V}_{ϵ},t{\tau }_{min}{V}_{ϵ})\\ & =& \frac{t^p}{p}(1+{\tau }_{min}^p)-\frac{t^{p_b^{\ast }}}{p_b^{\ast }}(1+\sum _{i=1}^m{\varsigma }_i{\tau }_{min}^{{\beta }_i}+{\tau }_{min}^{p_b^{\ast }})K_0{\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}dx\\ & & -\frac{\lambda {\tau }_{min}^{q_2}}{q_1+q_2}{t}^{q_1+q_2}{\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{q_1+q_2}}{{|x|}^{d{p}_d^{\ast }}}dx,t\ge 0\end{array}$$

(4.13)

and

$$ \widetilde{\Psi}(t)=\frac{t^{p}}{p} \bigl(1+ \tau_{\min}^{p} \bigr) -\frac{t^{p_{b}^{\ast}}}{p_{b}^{\ast}} \Biggl(1+ \sum _{i=1}^{m}\varsigma_{i} \tau_{\min}^{\beta_{i}} +\tau_{\min}^{p_{b}^{\ast}} \Biggr)K_{0} \int_{\Omega} \frac{|V_{\epsilon}|^{p_{b}^{\ast}}}{|x|^{bp_{b}^{\ast}}}\,dx,\quad t \geq0. $$

(4.14)

Note that $\Psi(0)=0$, $\Psi(t)>0$ for $t\rightarrow0^{+}$, and $\lim_{t\rightarrow+\infty}\Psi(t)=-\infty$. Therefore $\sup_{t\geq0}\Psi(t)$ can be achieved at some $t_{\epsilon}>0$ for which we get

$$\begin{aligned}& \bigl(1+\tau_{\min}^{p} \bigr)t_{\epsilon}^{p-1} -K_{0} \Biggl(1+ \sum_{i=1}^{m} \varsigma_{i}\tau_{\min}^{\beta_{i}} +\tau_{\min}^{p_{b}^{\ast}} \Biggr)t_{\epsilon}^{p_{b}^{\ast}-1} \int_{\Omega}|x|^{-bp_{b}^{\ast}} |V_{\epsilon}|^{p_{b}^{\ast}}\,dx \\& \quad{}-\lambda\tau_{\min}^{q_{2}}t_{\epsilon}^{q_{1}+q_{2}-1} \int_{\Omega}|x|^{-dp_{d}^{\ast}}|V_{\epsilon}|^{q_{1}+q_{2}}\,dx=0. \end{aligned}$$

(4.15)

In view of $V_{\epsilon}=\phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$, we find from (3.12) and (3.14) that

$$ \begin{cases} C_{1}\epsilon^{q(l_{2}+a+1-\frac{N}{p})}\leq \int_{\Omega}\frac{|V_{\epsilon}|^{q}}{|x|^{dp_{d}^{\ast}}}\,dx\leq C_{2}\epsilon^{q(l_{2}+a+1-\frac{N}{p})},& 1\leq q< \frac{N-dp_{d}^{\ast}}{l_{2}},\\ C_{3}\epsilon^{q(l_{2}+a+1-\frac{N}{p})}|\ln\epsilon|\leq \int_{\Omega}\frac{|V_{\epsilon}|^{q}}{|x|^{dp_{d}^{\ast}}}\,dx\leq C_{4}\epsilon^{q(l_{2}+a+1-\frac{N}{p})}|\ln\epsilon|,& q=\frac{N-dp_{d}^{\ast}}{l_{2}},\\ C_{5}\epsilon^{(p_{d}^{\ast}-q)(\frac{N}{p}-a-1)}\leq \int_{\Omega}\frac{|V_{\epsilon}|^{q}}{|x|^{dp_{d}^{\ast}}}\,dx\leq C_{6}\epsilon^{(p_{d}^{\ast}-q)(\frac{N}{p}-a-1)},& \frac{N-dp_{d}^{\ast}}{l_{2}} <q<p_{d}^{\ast}. \end{cases} $$

(4.16)

Therefore, for $\epsilon>0$ small enough, we deduce from (3.15), (4.15) and (4.16) and the fact $\lambda> 0$, $\tau_{\min}>0$ that

$$ 0< \overline{C}_{3}\leq t_{\epsilon}\leq \biggl\{ \frac{1+\tau_{\min}^{p}}{K_{0} (1+ \sum_{i=1}^{m}\varsigma_{i}\tau_{\min}^{\beta_{i}} +\tau_{\min}^{p_{b}^{\ast}} )\int_{\Omega} |x|^{-bp_{b}^{\ast}}|V_{\epsilon}|^{p_{b}^{\ast}}\,dx} \biggr\} ^{\frac{1}{p_{b}^{\ast}-p}}\triangleq t_{\epsilon}^{0} \leq \overline{C}_{4}, $$

(4.17)

where $\overline{C}_{3}>0$, $\overline{C}_{4}>0$ are constants independent of ϵ. On the other hand, the function $\widetilde{\Psi}(t)$ defined by (4.14) attains its maximum at $t_{\epsilon}^{0}$ and is increasing in the interval $[0, t_{\epsilon}^{0}]$, together with Lemma 3.2, (3.15) and (4.13)-(4.17), we obtain

$$\begin{array}{rcl}\mathrm{\Psi }(t_{ϵ})& =& \tilde{\mathrm{\Psi }}(t_{ϵ})-\frac{\lambda {\tau }_{min}^{q_2}}{q_1+q_2}{t}_{ϵ}^{q_1+q_2}{\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{q_1+q_2}}{{|x|}^{d{p}_d^{\ast }}}dx\le \tilde{\mathrm{\Psi }}\left(t_{ϵ}^0\right)-C{\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{q_1+q_2}}{{|x|}^{d{p}_d^{\ast }}}dx\\ & =& \frac{a+1-b}{N}{\left\{\frac{1+{\tau }_{min}^p}{{[K_0(1+{\sum }_{i=1}^m{\varsigma }_i{\tau }_{min}^{{\beta }_i}+{\tau }_{min}^{p_b^{\ast }}){\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{p_b^{\ast }}}{{|x|}^{b{p}_b^{\ast }}}dx]}^{\frac{p}{p_b^{\ast }}}}\right\}}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}-C{\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{q_1+q_2}}{{|x|}^{d{p}_d^{\ast }}}dx\\ & =& \frac{a+1-b}{N}{K}_0^{\frac{p}{p-p_b^{\ast }}}{\left\{\frac{h({\tau }_{min})}{{[{\mathcal{A}}_{\mu }^{-\frac{p_b^{\ast }}{p}}+O({ϵ}^{p(l_2+a+1)-N})]}^{\frac{p}{p_b^{\ast }}}}\right\}}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}-C{\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{q_1+q_2}}{{|x|}^{d{p}_d^{\ast }}}dx\\ & =& \frac{a+1-b}{N}{K}_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}+O\left({ϵ}^{p(l_2+a+1)-N}\right)-C{\int }_{\mathrm{\Omega }}\frac{{|V_{ϵ}|}^{q_1+q_2}}{{|x|}^{d{p}_d^{\ast }}}dx.\end{array}$$

(4.18)

Furthermore, we easily check from (2.13) that

$$ p (l_{2}+a+1 )-N> \bigl[p_{d}^{\ast}- (q_{1}+q_{2} ) \bigr] \biggl(\frac{N}{p}-a-1 \biggr). $$

(4.19)

Choosing $\epsilon>0$ sufficiently small, we deduce from (4.16), (4.18) and (4.19) that

$$\underset{t\ge 0}{sup}{\mathcal{F}}_{\lambda }(t{V}_{ϵ},t{\tau }_{min}{V}_{ϵ})=\mathrm{\Psi }(t_{ϵ})<\frac{a+1-b}{N}{K}_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}.$$

Thus we conclude that $(V_{\epsilon}, \tau_{\min}V_{\epsilon})$ satisfies (4.12) for $\epsilon>0$ sufficiently small and the results follow. □

Proof of Theorem 2.3

For any $(u, v)\in(W_{a, G}^{1, p}(\Omega)\backslash\{0\})^{2}$, we deduce from (K.1), (3.3), (4.1) and (4.2) that

$${\mathcal{F}}_{\lambda }(u,v)\ge \frac{1}{p}{\parallel (u,v)\parallel }_{\mu }^p-\frac{K_0}{p_b^{\ast }}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{-\frac{p_b^{\ast }}{p}}{\parallel (u,v)\parallel }_{\mu }^{p_b^{\ast }}-C{\parallel (u,v)\parallel }_{\mu }^{q_1+q_2}.$$

Since $p< q_{1}+q_{2}<p_{d}^{\ast}\leq p_{b}^{\ast}$, we conclude that there exist constants $\widetilde{\alpha}>0$ and $\rho>0$ such that ${\mathcal{F}}_{\lambda }(u,v)\ge \tilde{\alpha }$ for all $\|(u, v)\|_{\mu}=\rho$. Moreover, in view of ${lim}_{t\to \mathrm{\infty }}{\mathcal{F}}_{\lambda }(tu,tv)=-\mathrm{\infty }$, we find that there exists $t_{0}>0$ such that $\|(t_{0}u, t_{0}v)\|_{\mu}>\rho$ and ${\mathcal{F}}_{\lambda }(t_{0}u,t_{0}v)<0$. Now we set

$$c_1=\underset{\gamma \in \mathrm{\Gamma }}{inf}\underset{t\in [0,1]}{max}{\mathcal{F}}_{\lambda }(\gamma (t)),$$

where $\mathrm{\Gamma }=\{\gamma \in \mathcal{C}([0,1],{(W_{a,G}^{1,p}(\mathrm{\Omega }))}^2);\gamma (0)=(0,0),{\mathcal{F}}_{\lambda }(\gamma (1))<0\}$. By the mountain pass theorem (see [28]), we deduce that there exists a sequence $\{(u_{n}, v_{n})\}\subset(W_{a, G}^{1, p}(\Omega))^{2}$ such that ${\mathcal{F}}_{\lambda }(u_n,v_n)\to c_1\ge \tilde{\alpha }$, ${\mathcal{F}}_{\lambda }^{\prime }(u_n,v_n)\to 0$ as $n\rightarrow\infty$. Let $(\overline{u}, \overline{v})$ be the functions obtained in Lemma 4.2. Then we get

$$0<\tilde{\alpha }\le c_1\le \underset{t\in [0,1]}{sup}{\mathcal{F}}_{\lambda }(t{t}_0\overline{u},t{t}_0\overline{v})<\frac{a+1-b}{N}{K}_0^{\frac{p}{p-p_b^{\ast }}}{\mathcal{A}}_{\mu ,{\varsigma }_i}^{\frac{p_b^{\ast }}{p_b^{\ast }-p}}.$$

According to the above inequality and Lemma 4.1, we obtain a critical point $(u_{1}, v_{1})$ of ${\mathcal{F}}_{\lambda }$ satisfying problem $({\mathcal{P}}_{\lambda }^{K_0})$. Taking $(u_{1}^{-}, v_{1}^{-})$ as the test functions, where $u_{1}^{-}=\min\{0, u_{1}\}$ and $v_{1}^{-}=\min\{0, v_{1}\}$, we have

$$0=〈{\mathcal{F}}_{\lambda }^{\prime }(u_1,v_1),(u_1^{-},v_1^{-})〉={\parallel (u_1^{-},v_1^{-})\parallel }_{\mu }^p.$$

This means $u_{1}\geq0$ and $v_{1}\geq 0$ on Ω. Using the strong maximum principle, we find $u_{1}> 0$ and $v_{1}> 0$ on Ω. Finally, by the symmetric criticality principle, we conclude that $(u_{1}, v_{1})$ is a positive G-symmetric solution of problem $({\mathcal{P}}_{\lambda }^{K_0})$. □