In order to deduce Theorem 1.1, the following lemmas are important. Borrowing an idea from Lemma 3.6 in [14], we obtain the first result.
Lemma 3.1
For
\(s, t>0\), the system
$$ \textstyle\begin{cases} f(t,s)=t-aS(\frac{s+t}{\lambda})^{\frac{1}{3}}=0, \\ g(t,s)=s-bS^{2}(\frac{s+t}{\lambda})^{\frac{2}{3}}=0, \end{cases} $$
has a unique solution
\((t_{0},s_{0})\), where
\(\lambda>0\)
is a constant. Moreover, if
$$ \textstyle\begin{cases} f(t,s)\geq0, \\ g(t,s)\geq0, \end{cases} $$
then
\(t\geq t_{0}\)
and
\(s\geq s_{0}\), where
\(t_{0}=\frac{abS^{3}+a\sqrt {b^{2}S^{6}+4\lambda aS^{3}}}{2\lambda}\)
and
\(s_{0}=\frac{bS^{6}+2\lambda abS^{3}+b^{2}S^{3}\sqrt{b^{3}S^{6}+4\lambda aS^{3}}}{2\lambda^{2}}\).
Lemma 3.2
Assume that (\(\mu_{1}\)), (k1), and (h1) hold. Let a sequence
\(\{u_{n}\}\subset{N}\)
be such that
\(u_{n}\rightharpoonup u\)
in
\(H^{1}(\mathrm{R}^{3})\)
and
\(I(u_{n})\rightarrow c\), but any subsequence of
\(\{u_{n}\}\)
does not converge strongly to u. Then one of the following results holds:
-
(1)
\(c>I(t(u)u)\)
in the case
\(u\neq0\)
and
\(\langle I'(u),u\rangle <0\);
-
(2)
\(c\geq c^{*}\)
in the case
\(u=0\);
-
(3)
\(c>c^{*}\)
in the case
\(u\neq0\)
and
\(\langle I'(u),u\rangle\geq0\);
where
\(c^{*}=\frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\| ^{2}_{\infty}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\| ^{2}_{\infty}}\), and
\(t(u)\)
is defined as in Lemma
2.4.
Proof
Part of the proof is similar to that of [22], Lemma 3.1 or [25], Proposition 3.3. For the reader’s convenience, we only sketch the proof. Since \(u_{n}\rightharpoonup u\) in \(H^{1}(\mathrm{R}^{3})\), we have \(u_{n}-u\rightharpoonup0\). Then by Lemma 2.1 we obtain that
$$ \int_{\mathrm{R}^{3}}h(x)|u_{n}-u|^{2}\,dx \rightarrow0. $$
(3.1)
We obtain from the Brézis-Lieb lemma [26], (3.1), and \(u_{n}\in{N}\) that
$$\begin{aligned} c+o(1) =&I(u_{n}) \\ =&I(u)+\frac{1}{2}\|u_{n}-u\|^{2}+ \frac{b}{4} \biggl( \int _{\mathrm{R}^{3}}\bigl\vert \nabla(u_{n}-u)\bigr\vert ^{2}\,dx \biggr)^{2} \\ &{}-\frac{1}{6} \int_{\mathrm {R}^{3}}k(x)|u_{n}-u|^{6}\,dx+o(1) \end{aligned}$$
(3.2)
and
$$\begin{aligned} 0 =&\bigl\langle I'(u_{n}),u_{n}\bigr\rangle \\ =&\bigl\langle I'(u),u\bigr\rangle +\|u_{n}-u \|^{2}+b \biggl( \int_{\mathrm {R}^{3}}\bigl\vert \nabla(u_{n}-u)\bigr\vert ^{2}\,dx \biggr)^{2} \\ &{}- \int_{\mathrm{R}^{3}}k(x)|u_{n}-u|^{6}\,dx+o(1). \end{aligned}$$
(3.3)
Up to a subsequence, we may assume that there exist \(l_{i}\geq 0\), \(i=1,2,3\), such that
$$ \begin{aligned} &\|u_{n}-u\|^{2}\rightarrow l_{1},\qquad b \biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla (u_{n}-u)\bigr\vert ^{2}\,dx \biggr)^{2}\rightarrow l_{2}, \\ &\int_{\mathrm {R}^{3}}k(x)|u_{n}-u|^{6}\,dx\rightarrow l_{3}. \end{aligned} $$
(3.4)
Since any subsequence of \(\{u_{n}\}\) does not converge strongly to u, we have \(l_{1}>0\). Set \(\gamma(t)=\frac{l_{1}}{2}t^{2}+\frac {l_{2}}{4}t^{4}-\frac{l_{3}}{6}t^{6}\) and \(\eta(t)=g(t)+\gamma(t)\). By (3.3) and (3.4) we have \(\eta' (1)=g'(1)+\gamma'(1)=0\), and \(t=1\) is the only critical point of \(\eta (t)\) in \((0,+\infty)\), which implies that
$$ \eta(1)= \max_{t>0}\eta(t). $$
(3.5)
We consider three situations:
(1) \(u\neq0\) and \(\langle I'(u),u\rangle<0\). Then by (3.3) and (3.4) we have
$$ l_{1}+l_{2}-l_{3}>0. $$
(3.6)
Then,
$$ \gamma '(t)=l_{1}t+l_{2}t^{3}-l_{3}t^{5}>l_{1}t+l_{2}t^{3}-(l_{1}+l_{2})t^{5}= \bigl(1-t^{2}\bigr)\bigl[l_{1}t+(l_{1}+l_{2})t^{3} \bigr]\geq 0 $$
(3.7)
for any \(0< t<1\), which implies that
$$ \gamma(t)>\gamma(0)=0 \quad \text{for any } t\in(0,1). $$
(3.8)
Since \(\langle I'(u),u\rangle<0\), by Lemma 2.4 there exists \(t(u)>0\) such that \(0< t(u)<1\). Then it follows from (3.8) that \(\gamma (t(u))>0\). Therefore, we obtain from (3.2) and (3.5) that \(c=\eta(1)>\eta (t(u))=g(t(u))+\gamma(t(u))>I(t(u)u)\), which implies that (1) holds.
(2) \(u=0\). Then by (3.2), (3.3), and (3.4) we get
$$ \textstyle\begin{cases} l_{1}+l_{2}-l_{3}=0, \\ \frac{1}{2}l_{1}+\frac{1}{4}l_{2}-\frac{1}{6}l_{3}=c. \end{cases} $$
By the definition of S we see that
$$\begin{aligned}& \int_{\mathrm{R}^{3}}|\nabla u_{n}|^{2}\,dx\geq \frac{S}{\|k\| _{\infty}^{1/3}} \biggl( \int_{\mathrm{R}^{3}}k(x)|u_{n}|^{6}\,dx \biggr)^{\frac {1}{3}}, \\& b \biggl( \int_{\mathrm{R}^{3}}|\nabla u_{n}|^{2}\,dx \biggr)^{2}\geq b\frac{ S^{2}}{\|k\|_{\infty}^{2/3}} \biggl( \int_{\mathrm{R}^{3}}k(x)|u_{n}|^{6}\,dx \biggr)^{\frac{2}{3}}. \end{aligned}$$
Then
$$l_{1}\geq a S\biggl(\frac{l_{1}+l_{2}}{\|k\|_{\infty}}\biggr)^{\frac {1}{3}} \quad \mbox{and}\quad l_{2}\geq b S^{2}\biggl(\frac{l_{1}+l_{2}}{\|k\|_{\infty}} \biggr)^{\frac{2}{3}}. $$
Obviously, if \(l_{1}>0\), then \(l_{2}, l_{3}>0\). It follows from Lemma 3.1 that
$$\begin{aligned} c =&\frac{1}{3}l_{1}+\frac{1}{12}l_{2} \\ \geq&\frac{1}{3}\frac{abS^{3}+a\sqrt{b^{2}S^{6}+4\|k\|_{\infty}aS^{3}}}{2\| k\|_{\infty}}+\frac{1}{12}\frac{bS^{6}+2\|k\|_{\infty}abS^{3}+b^{2}S^{3}\sqrt {b^{3}S^{6}+4\|k\|_{\infty}aS^{3}}}{ 2\|k\|_{\infty}^{2}} \\ =&\frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\|_{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\|_{\infty}^{2}}:=c^{*}. \end{aligned}$$
(3.9)
(3) \(u\neq0\) and \(\langle I'(u),u\rangle\geq0\). We prove this case in two steps. Firstly, we consider \(u\neq0\) and \(\langle I'(u),u\rangle=0\). Then from Lemma 2.3 and Lemma 2.4 we get
$$ I(u)= \max_{t>0}I(tu)>0. $$
(3.10)
Since \(u\neq0\) and \(\langle I'(u),u\rangle=0\), as in (3.9), we obtain that
$$ c=\eta(1)=I(u)+\frac{l_{1}}{3}+\frac{l_{2}}{12}>c^{*}. $$
(3.11)
Secondly, we prove the case \(u\neq0\) and \(\langle I'(u),u\rangle>0\). Set \(t^{**}=(\frac{l_{2}+\sqrt{l_{2}^{2}+4l_{1}l_{3}}}{2l_{3}})^{\frac{1}{2}}\). Then, \(\gamma(t)\) attains its maximum at \(t^{**}\), that is,
$$\begin{aligned} \gamma\bigl(t^{**}\bigr) =& \max_{t>0}\gamma(t) \\ =&\frac{l_{1}l_{2}}{4l_{3}}+\frac{l_{2}^{2}}{24l_{3}^{2}}+\frac {(l_{2}^{2}+4l_{1}l_{3})^{\frac{3}{2}}}{24l_{3}^{2}} \\ \geq&\frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\| _{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\| _{\infty}^{2}}=c^{*}. \end{aligned}$$
(3.12)
It follows from Lemma 2.4 that \(0< t^{**}<1\). Then \(I(t^{**}u)\geq0\). Therefore, by (3.2), (3.5), and (3.12) we obtain
$$c=\eta(1)>\eta\bigl(t^{**}\bigr)=I\bigl(t^{**}u\bigr)+\gamma \bigl(t^{**}\bigr)\geq c^{*}. $$
The proof of Lemma 3.2 is complete. □
Lemma 3.3
If the hypotheses of Theorem
1.1
hold with
\(1<\beta <3\), then
$$c_{1}< \frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\|_{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\|_{\infty}^{2}}=c^{*}, $$
where
\(c_{1}\)
is defined by
\(\inf_{u\in{N}}I(u)\).
Proof
To prove this lemma, we borrow an idea employed in [22]. For \(\varepsilon,r>0\), define \(w_{\varepsilon}(x)=\frac{C\varphi (x)\varepsilon^{\frac{1}{4}}}{(\varepsilon+|x-x_{0}|^{2})^{\frac {1}{2}}}\), where C is a normalizing constant, \(x_{0}\) is given in (k2), and \(\varphi\in C_{0}^{\infty}(\mathrm{R}^{3})\), \(0\leq\varphi\leq1\), \(\varphi|_{B_{r}(0)}\equiv1\), and \(\operatorname{supp}\varphi\subset B_{2r}(0)\). Using the method of [25], we obtain
$$ \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2} \,dx=K_{1}+O\bigl(\varepsilon ^{\frac{1}{2}}\bigr), \qquad \int_{\mathrm{R}^{3}}|w_{\varepsilon}|^{6}\,dx=K_{2}+O \bigl(\varepsilon^{\frac{3}{2}}\bigr), $$
(3.13)
and
$$ \int_{\mathrm{R}^{3}}|w_{\varepsilon}|^{s}\, dx= \textstyle\begin{cases} K\varepsilon^{\frac{s}{4}}, & s\in[2,3), \\ K\varepsilon^{\frac{3}{4}}|\ln\varepsilon|, & s=3, \\ K\varepsilon^{\frac{6-s}{4}}, & s\in(3,6), \end{cases} $$
(3.14)
where \(K_{1}\), \(K_{2}\), K are positive constants. Moreover, the best Sobolev constant is \(S=K_{1}K_{2}^{-\frac{1}{3}}\). By (3.13) we have
$$ \frac{\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2} \,dx}{(\int_{\mathrm{R}^{3}} w_{\varepsilon}^{6} \,dx)^{\frac {1}{3}}}=S+O\bigl(\varepsilon^{\frac{1}{2}}\bigr). $$
(3.15)
By Lemma 2.4, for this \(w_{\varepsilon}\), there exists a unique \(t(w_{\varepsilon})>0\) such that \(t(w_{\varepsilon})w_{\varepsilon}\in {N}\). Thus, \(c_{1}< I(t(w_{\varepsilon})w_{\varepsilon})\). Using (2.1), for \(t>0\), since \(I(tw_{\varepsilon})\rightarrow-\infty\) as \(t\rightarrow \infty\), we easily see that \(I(tw_{\varepsilon})\) has a unique critical \(t(w_{\varepsilon})>0\) that corresponds to its maximum, that is, \(I(t_{\varepsilon}w_{\varepsilon})=\max_{t>0}I(tw_{\varepsilon})\). It follows from (1) of Lemma 2.3, \(I(tw_{\varepsilon})\rightarrow -\infty\) as \(t\rightarrow\infty\), and the continuity of I that there exist two positive constants \(t_{0}\) and \(T_{0}\) such that \(t_{0}< t_{\varepsilon}< T_{0}\). Let \(I(t_{\varepsilon}w_{\varepsilon})=F(\varepsilon)+G(\varepsilon)+H(\varepsilon)\), where
$$\begin{aligned}& F(\varepsilon)=\frac{at_{\varepsilon}^{2}}{2} \int _{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx+ \frac{bt_{\varepsilon}^{4}}{4}\biggl( \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx \biggr)^{2}-\frac{ t_{\varepsilon}^{6}}{6} \int_{\mathrm{R}^{3}}k(x_{0})|w_{\varepsilon}|^{6} \,dx, \\& G(\varepsilon)=\frac{ t_{\varepsilon}^{6}}{6} \int_{\mathrm {R}^{3}}k(x_{0})|w_{\varepsilon}|^{6} \,dx-\frac{ t_{\varepsilon}^{6}}{6} \int _{\mathrm{R}^{3}}k(x)|w_{\varepsilon}|^{6}\,dx, \end{aligned}$$
and
$$H(\varepsilon)=\frac{t_{\varepsilon}^{2}}{2} \int_{\mathrm {R}^{3}}|w_{\varepsilon}|^{2}\,dx- \frac{\mu t_{\varepsilon}^{2}}{2} \int _{\mathrm{R}^{3}}h(x)|w_{\varepsilon}|^{2}\,dx. $$
Set
$$\Phi(t)=\frac{at^{2}}{2} \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx+ \frac{bt^{4}}{4}\biggl( \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx \biggr)^{2}-\frac{ t^{6}}{6} \int_{\mathrm{R}^{3}}k(x_{0})|w_{\varepsilon}|^{6} \,dx. $$
Note that \(\Phi(t)\) attains its maximum at
$$t^{*}_{0}= \biggl(\frac{b(\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx)^{2}+\sqrt{b^{2}(\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx)^{4}+4a(\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx)^{2}\int _{\mathrm{R}^{3}}k(x_{0})|w_{\varepsilon}|^{6}\,dx}}{2\int_{\mathrm {R}^{3}}k(x_{0})|w_{\varepsilon}|^{6}\,dx} \biggr)^{\frac{1}{2}}. $$
Then
$$ \max_{t\geq0}\Phi(t)=\Phi\bigl(t^{*}_{0}\bigr)= \frac{abS^{3}}{4\|k\| _{\infty}}+\frac{b^{3}S^{6}}{24\|k\|_{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\| _{\infty}S)^{\frac{3}{2}}}{24\|k\|_{\infty}^{2}}+O\bigl( \varepsilon^{\frac{1}{2}}\bigr) $$
(3.16)
for \(\varepsilon>0\) small enough. Then we have
$$ F(\varepsilon)\leq c^{*}+O\bigl(\varepsilon^{\frac{1}{2}}\bigr). $$
(3.17)
By (3.36) of [22] we have
$$ G(\varepsilon)\leq C\varepsilon^{\frac{1}{2}}. $$
(3.18)
From (3.38) of [22], (3.14), and the boundedness of \(t_{\varepsilon}\) we obtain
$$\begin{aligned} \begin{aligned}[b] H(\varepsilon)&=\frac{t_{\varepsilon}^{2}}{2} \int_{\mathrm {R}^{3}}|w_{\varepsilon}|^{2}\,dx- \frac{\mu t_{\varepsilon}^{2}}{2} \int _{\mathrm{R}^{3}}h(x)|w_{\varepsilon}|^{2}\,dx \\ &\leq C\varepsilon^{\frac{1}{2}}-\mu C \varepsilon^{1-\frac {\beta}{2}}. \end{aligned} \end{aligned}$$
(3.19)
Since \(1<\beta<3\), for fixed \(\mu>0\), we obtain
$$ \frac{H(\varepsilon)}{\varepsilon^{\frac{1}{2}}}\rightarrow-\infty \quad \mbox{as } \varepsilon\rightarrow0. $$
(3.20)
It follows from (3.17), (3.18), and (3.20) that the proof of Lemma 3.3 is complete. □
Proof of Theorem 1.1
By the definition of \(c_{1}\) there exists a sequence \(\{u_{n}\}\subset N \) such that \(I(u_{n})\rightarrow c_{1}\) as \(n\rightarrow\infty\). Then we obtain that
$$ \|u_{n}\|^{2}+b\biggl( \int_{\mathrm{R}^{3}}|\nabla u_{n}|^{2}\,dx \biggr)^{2}- \int _{\mathrm{R}^{3}}\mu h(x)|u_{n}|^{2}\,dx= \int_{\mathrm{R}^{3}}k(x)|u_{n}|^{6}\,dx. $$
(3.21)
It follows from (3.21) and Lemma 2.2 that
$$\begin{aligned} c_{1}+o(1) =&\frac{1}{3}\biggl(\|u_{n} \|^{2}-\mu \int_{\mathrm {R}^{3}}h(x)|u_{n}|^{2}\,dx\biggr)+ \biggl(\frac{b}{4}-\frac{b}{6}\biggr) \biggl( \int_{\mathrm {R}^{3}}|\nabla u_{n}|^{2}\,dx \biggr)^{2} \\ \geq&\frac{1}{3}\biggl(1-\frac{\mu}{\tilde{\mu}}\biggr)\|u_{n} \|^{2}, \end{aligned}$$
(3.22)
which implies the boundedness of \(\{u_{n}\}\) in \(H^{1}(\mathrm{R}^{3})\) since \(0<\mu<\tilde{\mu}\). Then there exists a subsequence of \(\{ u_{n}\} \), still denoted by \(\{u_{n}\} \), such that \(u_{n}\rightharpoonup u\) in \(H^{1}(\mathrm{R}^{3})\). By (2) of Lemma 3.2 and Lemma 3.3 we have \(u\neq0\). By the definition of \(t(u)\) we get \(t(u)u\in{N}\). So \(I(t(u)u)\geq c_{1}\). We claim that \(u_{n}\rightarrow u\) in \(H^{1}(\mathrm {R}^{3})\). Otherwise, by (1) and (3) of Lemma 3.2, we would get that \(c_{1}>I(t(u)u)\) or \(c_{1}>c^{*}\). In any case, we get a contradiction since \(c_{1}< c^{*}\). Therefore, \(\{u_{n}\}\) converges strongly to u. Thus, \(u\in {N}\) and \(I(u)=c_{1}\). By the Lagrange multiplier rule there exists \(\theta\in \mathrm{R}\) such that \(I'(u)=\theta G'(u)\) and thus
$$0=\bigl\langle I'(u),u\bigr\rangle =\theta \biggl(2\|u \|^{2}+4b\biggl( \int _{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2}-6 \int_{\mathrm{R}^{3}} k(x)|u|^{6}\,dx-2\mu \int_{\mathrm{R}^{3}}h(x)|u|^{2}\,dx \biggr). $$
Since \(u\in N\), we get
$$0=\theta \biggl(-4\biggl(\|u\|^{2}-\mu \int_{\mathrm {R}^{3}}h(x)|u|^{2}\,dx\biggr)-2b\biggl( \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \biggr), $$
which implies that \(\theta=0\) and u is a nontrivial critical point of the functional I in \(H^{1}(\mathrm{R}^{3})\). Therefore, the nonzero function u can solve Eq. (1.1), that is,
$$ -\biggl(a+b \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)\triangle u+u= k(x)|u|^{2^{*}-2}u+\mu h(x)u. $$
(3.23)
In (3.23), using \(u^{-}=\max\{-u,0\}\) as a test function and integrating by parts, by (k1), (h2), and (\(\mu_{1}\)) we obtain
$$\begin{aligned} 0 =& \int_{\mathrm{R}^{3}}a\bigl\vert \nabla u^{-}\bigr\vert ^{2} \,dx+ \int_{\mathrm {R}^{3}}\bigl\vert u^{-}\bigr\vert ^{2}\,dx+b \int_{\mathrm{R}^{3}}\vert \nabla u\vert ^{2}\,dx \int_{\mathrm {R}^{3}}\bigl\vert \nabla u^{-}\bigr\vert ^{2}\,dx \\ &{}+ \int_{\mathrm{R}^{3}}k(x)\bigl\vert u^{-}\bigr\vert ^{2^{*}-2}\bigl\vert u^{-}\bigr\vert ^{2}\,dx+ \int_{\mathrm {R}^{3}}\mu h(x)\bigl\vert u^{-}\bigr\vert ^{2} \,dx\geq0. \end{aligned}$$
Then \(u^{-}=0\) and \(u\geq0\). From Harnack’s inequality [27] we can infer that \(u>0\) for all \(x\in \mathrm{R}^{3}\). Therefore, u is a positive solution of (1.1). The proof is complete by choosing \(\omega_{0}=u\). □