RETRACTED ARTICLE: Sign-changing solutions to Schrödinger-Kirchhoff-type equations with critical exponent

Abstract

In this paper, we study the following Schrödinger-Kirchhoff-type equation:

$$ \textstyle\begin{cases} -(a+b\int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx)\triangle u+u= k(x)|u|^{2^{*}-2}u+\mu h(x)u \quad \text{in } \mathrm{R}^{3}, \\ u\in H^{1}(\mathrm{R}^{3}), \end{cases} $$

where $a, b, \mu>0$ are constants, $2^{*}=6$ is the critical Sobolev exponent in three spatial dimensions. Under appropriate assumptions on nonnegative functions $k(x)$ and $h(x)$, we establish the existence of positive and sign-changing solutions by variational methods.

1 Introduction

In this paper, we investigate the following Schrödinger-Kirchhoff-type problem:

$$ \textstyle\begin{cases}-(a+b\int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx)\triangle u+u= k(x)|u|^{2^{*}-2}u+\mu h(x)u \quad \text{in } \mathrm{R}^{3}, \\ u\in H^{1}(\mathrm{R}^{3}), \end{cases} $$

(1.1)

where $a, b>0$ are constants, and $2^{*}=6$ is the critical Sobolev exponent in dimension three. We assume that μ and the functions $k(x)$ and $h(x)$ satisfy the following hypotheses:

($\mu_{1}$):

$0<\mu<\tilde{\mu}$, where μ̃ is defined by

$$\tilde{\mu}:=\inf_{u\in H^{1}(\mathrm{R}^{3})\setminus\{0\}}\biggl\{ \int _{\mathrm{R}^{3}}\bigl(a|\nabla u|^{2}+|u|^{2} \bigr)\,dx: \int_{\mathrm {R}^{3}}h(x)|u|^{2}\,dx=1\biggr\} ; $$

(k₁):

$k(x)\geq0$, $\forall x\in\mathrm{R}^{3}$;

(k₂):

there exist $x_{0}\in\mathrm{R}^{3}$, $\sigma_{1}>0$, $\rho_{1}>0$, and $1\leq\alpha<3$ such that $k(x_{0})= \max_{x\in \mathrm{R}^{3}}k(x)$ and

$$\bigl\vert k(x)-k(x_{0})\bigr\vert \leq\sigma_{1} \vert x-x_{0}\vert ^{\alpha}\quad \text{for } \vert x-x_{0}\vert < \rho_{1}; $$

(h₁):

$h(x)\geq0$ for any $x\in \mathrm{R}^{3}$ and $h(x)\in L^{\frac {3}{2}}(\mathrm{R}^{3})$;

(h₂):

there exist $\sigma_{2}>0$ and $\rho_{2}>0$ such that $h(x)\geq \sigma_{2}|x-x_{0}|^{-\beta}$ for $|x-x_{0}|<\rho_{2}$.

The Kirchhof-type problem is related to the stationary analogue of the equation

$$u_{tt}-\biggl(a+b \int_{\Omega}|\nabla u|^{2}\,dx\biggr)\triangle u= f(x,u) \quad \text{in } \Omega, $$

where Ω is a bounded domain in $\mathrm{R}^{N}$, u denotes the displacement, $f(x,u)$ is the external force, and b is the initial tension, whereas a is related to the intrinsic properties of the string (such as Young’s modulus). Equations of this type arise in the study of string or membrane vibration and were proposed by Kirchhoff in 1883 (see [1]) to describe the transversal oscillations of a stretched string, particularly, taking into account the subsequent change in string length caused by oscillations.

Kirchhoff-type problems are often referred to as being nonlocal because of the presence of the integral over the entire domain Ω, which provokes some mathematical difficulties. Similar nonlocal problems also model several physical and biological systems where u describes a process that depends on the average of itself, for example, the population density; see [2, 3]. Kirchhoff-type problems have received much attention. Some important and interesting results can be found in, for example, [4–6] and the references therein.

The solvability of the following Schrödinger-Kirchoff-type equation (1.2) has also been well studied in general dimension by various authors:

$$ -\biggl(a+b \int_{\mathrm{R}^{N}}|\nabla u|^{2}\,dx\biggr)\triangle u+V(x)u=f(x,u) \quad \text{in } \mathrm{R}^{N}. $$

(1.2)

For example, Wu [7] and many others [8–13], using variational methods, proved the existence of nontrivial solutions to (1.2) with subcritical nonlinearities. Li and Ye [14] obtained the existence of a positive solution for (1.2) with critical exponents. More recently, Wang et al. [15] and Liang and Zhang [16] proved the existence and multiplicity of positive solutions of (1.2) with critical growth and a small positive parameters.

The problem of finding sign-changing solutions is a very classical problem. In general, this problem is much more difficult than finding a mere solution. There were several abstract theories or methods to study sign-changing solutions; see, for example, [17, 18] and the references therein. In recent years, Zhang and Perera [19] obtained sign-changing solutions of (1.2) with superlinear or asymptotically linear terms. More recently, Mao and Zhang [20] use minimax methods and invariant sets of descent flow to prove the existence of nontrivial solutions and sign-changing solutions for (1.2) without the P.S. condition. Motivated by the works described, in this paper, our aim is to study the existence of positive and sign-changing solutions for problem (1.1). The method is inspired by Hirano and Shioji [21] and Huang et al. [22]; however, their arguments cannot be directly applied here. To our best knowledge, there are very few works up to now studying sign-changing solutions for Schrödinger-Kirchhoff-type problem with critical exponent, that is, problem (1.1). Our main results are as follows.

Theorem 1.1

Assume that ($\mu_{1}$), (k₁), (k₂), and (h₁)-(h₂) hold. Then, for $1<\beta<3$, problem (1.1) possesses at least one positive solution.

Theorem 1.2

Assume that ($\mu_{1}$), (k₁), (k₂), and (h₁)-(h₂) hold. Then, for $\frac{3}{2}<\beta<3$, problem (1.1) possesses at least one sign-changing solution.

Notation

• $H^{1}(\mathrm{R}^{3})$ is the Sobolev space equipped with the norm $\|u\|^{2}_{H^{1}(\mathrm{R}^{3})}=\int_{\mathrm {R}^{3}}{(|\nabla u|^{2}+|u|^{2})\,dx}$.

• We define $\|u\|^{2}:=\int_{\mathrm{R}^{3}}{(a|\nabla u|^{2}+|u|^{2})\,dx}$ for $u\in H^{1}(\mathrm{R}^{3})$. Note that $\|\cdot\|$ is an equivalent norm on $H^{1}(\mathrm{R}^{3})$.

• For any $1\leq s\leq\infty$, $\|u\|_{L^{s}}:=(\int_{\mathrm {R}^{3}}|u|^{s} \,dx)^{\frac{1}{s}}$ denotes the usual norm of the Lebesgue space $L^{s}(\mathrm{R}^{3})$.

• By $D^{1,2}(\mathrm{R}^{3})$ we denote the completion of $C_{0}^{\infty}(\mathrm{R}^{3})$ with respect to the norm $\|u\| ^{2}_{D^{1,2}(\mathrm{R}^{3})}:=\int_{\mathrm{R}^{3}}|\nabla u|^{2} \,dx$.

• S denotes the best Sobolev constant defined by $S=\inf_{u\in D^{1,2}(\mathrm{R}^{3})\setminus\{0\}}\frac{\int _{\mathrm{R}^{3}}|\nabla u|^{2} \,dx}{(\int_{\mathrm{R}^{3}} u^{6} \,dx)^{\frac {1}{3}}}$.

• $C>0$ denotes various positive constants.

The outline of the paper is given as follows. In Section 2, we present some preliminary results. In Sections 3 and 4, we give proofs of Theorems 1.1 and 1.2, respectively.

2 The variational framework and preliminary

In this section, we give some preliminary lemmas and the variational setting for (1.1). It is clear that system (1.1) is the Euler-Lagrange equations of the functional $I:H^{1}(\mathrm{R}^{3})\rightarrow\mathrm{R}$ defined by

$$ I(u)=\frac{1}{2}\|u\|^{2}+\frac{b}{4}\biggl( \int_{\mathrm {R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2}- \frac{1}{6} \int_{\mathrm {R}^{3}}k(x)|u|^{6}\,dx-\frac{\mu}{2} \int_{\mathrm{R}^{3}}h(x)|u|^{2}\,dx. $$

(2.1)

Obviously, I is a well-defined $C^{1}$ functional and satisfies

$$\begin{aligned} \bigl\langle I'(u),v\bigr\rangle =& \int_{\mathrm{R}^{3}}{(a\nabla u\nabla v+uv)}\,dx+b \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx \int_{\mathrm {R}^{3}}\nabla u\nabla v\,dx \\ &{}- \int_{\mathrm{R}^{3}}\bigl({k(x)|u|^{4}uv}+\mu h(x)uv\bigr)\,dx \end{aligned}$$

(2.2)

for $v\in H^{1}(\mathrm{R}^{3})$. It is well known that $u\in H^{1}(\mathrm {R}^{3})$ is a critical point of the functional I if and only if u is a weak solution of (1.1).

Lemma 2.1

Assume that (h₁) holds. Then the function $\psi _{h}:u\in H^{1}(\mathrm{R}^{3})\mapsto\int_{\mathrm{R}^{3}} h(x)u^{2}\,dx$ is weakly continuous, and for each $v\in H^{1}(\mathrm{R}^{3})$, $\varphi_{h}:u\in H^{1}(\mathrm{R}^{3})\mapsto\int_{\mathrm {R}^{3}}h(x)uv\,dx$ is also weakly continuous.

The proof of Lemma 2.1 is a direct conclusion of [23], Lemma 2.13.

Lemma 2.2

Assume that (h₁) holds. Then the infimum

$$\tilde{\mu}:= \inf_{u\in H^{1}(\mathrm {R}^{3})\setminus\{0\}} \biggl\{ \int_{\mathrm{R}^{3}}\bigl(a|\nabla u|^{2}+|u|^{2} \bigr)\,dx: \int_{\mathrm {R}^{3}}h(x)|u|^{2}\,dx=1\biggr\} $$

is achieved.

Proof

The proof of Lemma 2.2 is the same as that of [24], Lemma 2.5. Here we omit it for simplicity. □

Lemma 2.3

Assume that (k₁), (h₁), and ($\mu_{1}$) hold. Then the functional I possesses the following properties.

• (1) There exist $\rho, \gamma>0$ such that $I(u)\geq\gamma$ for $\| u\|=\rho$.

• (2) There exists $e\in H^{1}(\mathrm{R}^{3})$ with $\|e\|>\rho$ such that $I(e)<0$.

Proof

By Lemma 2.2 and the Sobolev inequality we obtain

$$I(u)\geq\frac{1}{2}\|u\|^{2}-C\|u\|^{6}- \frac{\mu}{2\tilde{\mu}}\|u\| ^{2}=\|u\|^{2}\biggl( \frac{1}{2}-\frac{\mu}{2\tilde{\mu}}-C\|u\|^{4}\biggr). $$

Set $\|u\|=\rho$ small enough such that $C\rho^{4}\leq\frac {1}{4}(1-\frac{\mu}{\tilde{\mu}})$. Then we have

$$ I(u)\geq\frac{1}{4}\biggl(1-\frac{\mu}{\tilde{\mu}}\biggr)\rho^{2}. $$

(2.3)

Choosing $\gamma=\frac{1}{4}(1-\frac{\mu}{\tilde{\mu}})\rho^{2}$, we complete the proof of (1).

For $t>0$ and some $u_{0}\in H^{1}(\mathrm{R}^{3})$ with $\|u_{0}\|=1$, it follows from (h₁) and ($\mu_{1}$) that

$$I(tu_{0})\leq\frac{1}{2}t^{2}\|u_{0} \|^{2}+\frac{b}{4}t^{4}\biggl( \int_{\mathrm {R}^{3}}|\nabla u_{0}|^{2}\,dx \biggr)^{2}-\frac{t^{6}}{6} \int_{\mathrm{R}^{3}}k(x)|u_{0}|^{6}\,dx, $$

which implies that $I(tu_{0})<0$ for $t>0$ large enough. Hence, we can take an $e=t_{1}u_{0}$ for some $t_{1}>0$ large enough, and (2) follows. □

Next, we define the Nehari manifold N associated with I by

$${N}:=\bigl\{ u\in H^{1}\bigl(\mathrm{R}^{3}\bigr)\setminus\{0 \}:G(u)=0\bigr\} ,\quad \text{where } G(u)=\bigl\langle I'(u),u\bigr\rangle . $$

Now we state some properties of N.

Lemma 2.4

Assume that ($\mu_{1}$) is satisfied. Then the following conclusions hold.

• (1) For all $u\in H^{1}(\mathrm{R}^{3})\setminus\{0\}$, there exists a unique $t(u)>0$ such that $t(u)u\in{N}$. Moreover, $I(t(u))u= \max_{t\geq0}I(tu)$.

• (2) $0< t(u)<1$ in the case $\langle I'(u),u\rangle<0$; $t(u)>1$ in the case $\langle I'(u),u\rangle>0$.

• (3) $t(u)$ is a continuous functional with respect to u in $H^{1}(\mathrm{R}^{3})$.

• (4) $t(u)\rightarrow+\infty$ as $\|u\|\rightarrow0$.

Proof

The proof is similar to that of [22], Lemma 2.4, and is omitted here. □

3 Positive solution

In order to deduce Theorem 1.1, the following lemmas are important. Borrowing an idea from Lemma 3.6 in [14], we obtain the first result.

Lemma 3.1

For $s, t>0$, the system

$$ \textstyle\begin{cases} f(t,s)=t-aS(\frac{s+t}{\lambda})^{\frac{1}{3}}=0, \\ g(t,s)=s-bS^{2}(\frac{s+t}{\lambda})^{\frac{2}{3}}=0, \end{cases} $$

has a unique solution $(t_{0},s_{0})$, where $\lambda>0$ is a constant. Moreover, if

$$ \textstyle\begin{cases} f(t,s)\geq0, \\ g(t,s)\geq0, \end{cases} $$

then $t\geq t_{0}$ and $s\geq s_{0}$, where $t_{0}=\frac{abS^{3}+a\sqrt {b^{2}S^{6}+4\lambda aS^{3}}}{2\lambda}$ and $s_{0}=\frac{bS^{6}+2\lambda abS^{3}+b^{2}S^{3}\sqrt{b^{3}S^{6}+4\lambda aS^{3}}}{2\lambda^{2}}$.

Lemma 3.2

Assume that ($\mu_{1}$), (k₁), and (h₁) hold. Let a sequence $\{u_{n}\}\subset{N}$ be such that $u_{n}\rightharpoonup u$ in $H^{1}(\mathrm{R}^{3})$ and $I(u_{n})\rightarrow c$, but any subsequence of $\{u_{n}\}$ does not converge strongly to u. Then one of the following results holds:

• (1) $c>I(t(u)u)$ in the case $u\neq0$ and $\langle I'(u),u\rangle <0$;

• (2) $c\geq c^{*}$ in the case $u=0$;

• (3) $c>c^{*}$ in the case $u\neq0$ and $\langle I'(u),u\rangle\geq0$;

where $c^{*}=\frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\| ^{2}_{\infty}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\| ^{2}_{\infty}}$, and $t(u)$ is defined as in Lemma  2.4.

Proof

Part of the proof is similar to that of [22], Lemma 3.1 or [25], Proposition 3.3. For the reader’s convenience, we only sketch the proof. Since $u_{n}\rightharpoonup u$ in $H^{1}(\mathrm{R}^{3})$, we have $u_{n}-u\rightharpoonup0$. Then by Lemma 2.1 we obtain that

$$ \int_{\mathrm{R}^{3}}h(x)|u_{n}-u|^{2}\,dx \rightarrow0. $$

(3.1)

We obtain from the Brézis-Lieb lemma [26], (3.1), and $u_{n}\in{N}$ that

$$\begin{aligned} c+o(1) =&I(u_{n}) \\ =&I(u)+\frac{1}{2}\|u_{n}-u\|^{2}+ \frac{b}{4} \biggl( \int _{\mathrm{R}^{3}}\bigl\vert \nabla(u_{n}-u)\bigr\vert ^{2}\,dx \biggr)^{2} \\ &{}-\frac{1}{6} \int_{\mathrm {R}^{3}}k(x)|u_{n}-u|^{6}\,dx+o(1) \end{aligned}$$

(3.2)

and

$$\begin{aligned} 0 =&\bigl\langle I'(u_{n}),u_{n}\bigr\rangle \\ =&\bigl\langle I'(u),u\bigr\rangle +\|u_{n}-u \|^{2}+b \biggl( \int_{\mathrm {R}^{3}}\bigl\vert \nabla(u_{n}-u)\bigr\vert ^{2}\,dx \biggr)^{2} \\ &{}- \int_{\mathrm{R}^{3}}k(x)|u_{n}-u|^{6}\,dx+o(1). \end{aligned}$$

(3.3)

Up to a subsequence, we may assume that there exist $l_{i}\geq 0$, $i=1,2,3$, such that

$$ \begin{aligned} &\|u_{n}-u\|^{2}\rightarrow l_{1},\qquad b \biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla (u_{n}-u)\bigr\vert ^{2}\,dx \biggr)^{2}\rightarrow l_{2}, \\ &\int_{\mathrm {R}^{3}}k(x)|u_{n}-u|^{6}\,dx\rightarrow l_{3}. \end{aligned} $$

(3.4)

Since any subsequence of $\{u_{n}\}$ does not converge strongly to u, we have $l_{1}>0$. Set $\gamma(t)=\frac{l_{1}}{2}t^{2}+\frac {l_{2}}{4}t^{4}-\frac{l_{3}}{6}t^{6}$ and $\eta(t)=g(t)+\gamma(t)$. By (3.3) and (3.4) we have $\eta' (1)=g'(1)+\gamma'(1)=0$, and $t=1$ is the only critical point of $\eta (t)$ in $(0,+\infty)$, which implies that

$$ \eta(1)= \max_{t>0}\eta(t). $$

(3.5)

We consider three situations:

(1) $u\neq0$ and $\langle I'(u),u\rangle<0$. Then by (3.3) and (3.4) we have

$$ l_{1}+l_{2}-l_{3}>0. $$

(3.6)

Then,

$$ \gamma '(t)=l_{1}t+l_{2}t^{3}-l_{3}t^{5}>l_{1}t+l_{2}t^{3}-(l_{1}+l_{2})t^{5}= \bigl(1-t^{2}\bigr)\bigl[l_{1}t+(l_{1}+l_{2})t^{3} \bigr]\geq 0 $$

(3.7)

for any $0< t<1$, which implies that

$$ \gamma(t)>\gamma(0)=0 \quad \text{for any } t\in(0,1). $$

(3.8)

Since $\langle I'(u),u\rangle<0$, by Lemma 2.4 there exists $t(u)>0$ such that $0< t(u)<1$. Then it follows from (3.8) that $\gamma (t(u))>0$. Therefore, we obtain from (3.2) and (3.5) that $c=\eta(1)>\eta (t(u))=g(t(u))+\gamma(t(u))>I(t(u)u)$, which implies that (1) holds.

(2) $u=0$. Then by (3.2), (3.3), and (3.4) we get

$$ \textstyle\begin{cases} l_{1}+l_{2}-l_{3}=0, \\ \frac{1}{2}l_{1}+\frac{1}{4}l_{2}-\frac{1}{6}l_{3}=c. \end{cases} $$

By the definition of S we see that

$$\begin{aligned}& \int_{\mathrm{R}^{3}}|\nabla u_{n}|^{2}\,dx\geq \frac{S}{\|k\| _{\infty}^{1/3}} \biggl( \int_{\mathrm{R}^{3}}k(x)|u_{n}|^{6}\,dx \biggr)^{\frac {1}{3}}, \\& b \biggl( \int_{\mathrm{R}^{3}}|\nabla u_{n}|^{2}\,dx \biggr)^{2}\geq b\frac{ S^{2}}{\|k\|_{\infty}^{2/3}} \biggl( \int_{\mathrm{R}^{3}}k(x)|u_{n}|^{6}\,dx \biggr)^{\frac{2}{3}}. \end{aligned}$$

Then

$$l_{1}\geq a S\biggl(\frac{l_{1}+l_{2}}{\|k\|_{\infty}}\biggr)^{\frac {1}{3}} \quad \mbox{and}\quad l_{2}\geq b S^{2}\biggl(\frac{l_{1}+l_{2}}{\|k\|_{\infty}} \biggr)^{\frac{2}{3}}. $$

Obviously, if $l_{1}>0$, then $l_{2}, l_{3}>0$. It follows from Lemma 3.1 that

$$\begin{aligned} c =&\frac{1}{3}l_{1}+\frac{1}{12}l_{2} \\ \geq&\frac{1}{3}\frac{abS^{3}+a\sqrt{b^{2}S^{6}+4\|k\|_{\infty}aS^{3}}}{2\| k\|_{\infty}}+\frac{1}{12}\frac{bS^{6}+2\|k\|_{\infty}abS^{3}+b^{2}S^{3}\sqrt {b^{3}S^{6}+4\|k\|_{\infty}aS^{3}}}{ 2\|k\|_{\infty}^{2}} \\ =&\frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\|_{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\|_{\infty}^{2}}:=c^{*}. \end{aligned}$$

(3.9)

(3) $u\neq0$ and $\langle I'(u),u\rangle\geq0$. We prove this case in two steps. Firstly, we consider $u\neq0$ and $\langle I'(u),u\rangle=0$. Then from Lemma 2.3 and Lemma 2.4 we get

$$ I(u)= \max_{t>0}I(tu)>0. $$

(3.10)

Since $u\neq0$ and $\langle I'(u),u\rangle=0$, as in (3.9), we obtain that

$$ c=\eta(1)=I(u)+\frac{l_{1}}{3}+\frac{l_{2}}{12}>c^{*}. $$

(3.11)

Secondly, we prove the case $u\neq0$ and $\langle I'(u),u\rangle>0$. Set $t^{**}=(\frac{l_{2}+\sqrt{l_{2}^{2}+4l_{1}l_{3}}}{2l_{3}})^{\frac{1}{2}}$. Then, $\gamma(t)$ attains its maximum at $t^{**}$, that is,

$$\begin{aligned} \gamma\bigl(t^{**}\bigr) =& \max_{t>0}\gamma(t) \\ =&\frac{l_{1}l_{2}}{4l_{3}}+\frac{l_{2}^{2}}{24l_{3}^{2}}+\frac {(l_{2}^{2}+4l_{1}l_{3})^{\frac{3}{2}}}{24l_{3}^{2}} \\ \geq&\frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\| _{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\| _{\infty}^{2}}=c^{*}. \end{aligned}$$

(3.12)

It follows from Lemma 2.4 that $0< t^{**}<1$. Then $I(t^{**}u)\geq0$. Therefore, by (3.2), (3.5), and (3.12) we obtain

$$c=\eta(1)>\eta\bigl(t^{**}\bigr)=I\bigl(t^{**}u\bigr)+\gamma \bigl(t^{**}\bigr)\geq c^{*}. $$

The proof of Lemma 3.2 is complete. □

Lemma 3.3

If the hypotheses of Theorem  1.1 hold with $1<\beta <3$, then

$$c_{1}< \frac{abS^{3}}{4\|k\|_{\infty}}+\frac{b^{3}S^{6}}{24\|k\|_{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\|_{\infty}S)^{\frac{3}{2}}}{24\|k\|_{\infty}^{2}}=c^{*}, $$

where $c_{1}$ is defined by $\inf_{u\in{N}}I(u)$.

Proof

To prove this lemma, we borrow an idea employed in [22]. For $\varepsilon,r>0$, define $w_{\varepsilon}(x)=\frac{C\varphi (x)\varepsilon^{\frac{1}{4}}}{(\varepsilon+|x-x_{0}|^{2})^{\frac {1}{2}}}$, where C is a normalizing constant, $x_{0}$ is given in (k₂), and $\varphi\in C_{0}^{\infty}(\mathrm{R}^{3})$, $0\leq\varphi\leq1$, $\varphi|_{B_{r}(0)}\equiv1$, and $\operatorname{supp}\varphi\subset B_{2r}(0)$. Using the method of [25], we obtain

$$ \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2} \,dx=K_{1}+O\bigl(\varepsilon ^{\frac{1}{2}}\bigr), \qquad \int_{\mathrm{R}^{3}}|w_{\varepsilon}|^{6}\,dx=K_{2}+O \bigl(\varepsilon^{\frac{3}{2}}\bigr), $$

(3.13)

and

$$ \int_{\mathrm{R}^{3}}|w_{\varepsilon}|^{s}\, dx= \textstyle\begin{cases} K\varepsilon^{\frac{s}{4}}, & s\in[2,3), \\ K\varepsilon^{\frac{3}{4}}|\ln\varepsilon|, & s=3, \\ K\varepsilon^{\frac{6-s}{4}}, & s\in(3,6), \end{cases} $$

(3.14)

where $K_{1}$, $K_{2}$, K are positive constants. Moreover, the best Sobolev constant is $S=K_{1}K_{2}^{-\frac{1}{3}}$. By (3.13) we have

$$ \frac{\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2} \,dx}{(\int_{\mathrm{R}^{3}} w_{\varepsilon}^{6} \,dx)^{\frac {1}{3}}}=S+O\bigl(\varepsilon^{\frac{1}{2}}\bigr). $$

(3.15)

By Lemma 2.4, for this $w_{\varepsilon}$, there exists a unique $t(w_{\varepsilon})>0$ such that $t(w_{\varepsilon})w_{\varepsilon}\in {N}$. Thus, $c_{1}< I(t(w_{\varepsilon})w_{\varepsilon})$. Using (2.1), for $t>0$, since $I(tw_{\varepsilon})\rightarrow-\infty$ as $t\rightarrow \infty$, we easily see that $I(tw_{\varepsilon})$ has a unique critical $t(w_{\varepsilon})>0$ that corresponds to its maximum, that is, $I(t_{\varepsilon}w_{\varepsilon})=\max_{t>0}I(tw_{\varepsilon})$. It follows from (1) of Lemma 2.3, $I(tw_{\varepsilon})\rightarrow -\infty$ as $t\rightarrow\infty$, and the continuity of I that there exist two positive constants $t_{0}$ and $T_{0}$ such that $t_{0}< t_{\varepsilon}< T_{0}$. Let $I(t_{\varepsilon}w_{\varepsilon})=F(\varepsilon)+G(\varepsilon)+H(\varepsilon)$, where

$$\begin{aligned}& F(\varepsilon)=\frac{at_{\varepsilon}^{2}}{2} \int _{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx+ \frac{bt_{\varepsilon}^{4}}{4}\biggl( \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx \biggr)^{2}-\frac{ t_{\varepsilon}^{6}}{6} \int_{\mathrm{R}^{3}}k(x_{0})|w_{\varepsilon}|^{6} \,dx, \\& G(\varepsilon)=\frac{ t_{\varepsilon}^{6}}{6} \int_{\mathrm {R}^{3}}k(x_{0})|w_{\varepsilon}|^{6} \,dx-\frac{ t_{\varepsilon}^{6}}{6} \int _{\mathrm{R}^{3}}k(x)|w_{\varepsilon}|^{6}\,dx, \end{aligned}$$

and

$$H(\varepsilon)=\frac{t_{\varepsilon}^{2}}{2} \int_{\mathrm {R}^{3}}|w_{\varepsilon}|^{2}\,dx- \frac{\mu t_{\varepsilon}^{2}}{2} \int _{\mathrm{R}^{3}}h(x)|w_{\varepsilon}|^{2}\,dx. $$

Set

$$\Phi(t)=\frac{at^{2}}{2} \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx+ \frac{bt^{4}}{4}\biggl( \int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx \biggr)^{2}-\frac{ t^{6}}{6} \int_{\mathrm{R}^{3}}k(x_{0})|w_{\varepsilon}|^{6} \,dx. $$

Note that $\Phi(t)$ attains its maximum at

$$t^{*}_{0}= \biggl(\frac{b(\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx)^{2}+\sqrt{b^{2}(\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx)^{4}+4a(\int_{\mathrm{R}^{3}}|\nabla w_{\varepsilon}|^{2}\,dx)^{2}\int _{\mathrm{R}^{3}}k(x_{0})|w_{\varepsilon}|^{6}\,dx}}{2\int_{\mathrm {R}^{3}}k(x_{0})|w_{\varepsilon}|^{6}\,dx} \biggr)^{\frac{1}{2}}. $$

Then

$$ \max_{t\geq0}\Phi(t)=\Phi\bigl(t^{*}_{0}\bigr)= \frac{abS^{3}}{4\|k\| _{\infty}}+\frac{b^{3}S^{6}}{24\|k\|_{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\| _{\infty}S)^{\frac{3}{2}}}{24\|k\|_{\infty}^{2}}+O\bigl( \varepsilon^{\frac{1}{2}}\bigr) $$

(3.16)

for $\varepsilon>0$ small enough. Then we have

$$ F(\varepsilon)\leq c^{*}+O\bigl(\varepsilon^{\frac{1}{2}}\bigr). $$

(3.17)

By (3.36) of [22] we have

$$ G(\varepsilon)\leq C\varepsilon^{\frac{1}{2}}. $$

(3.18)

From (3.38) of [22], (3.14), and the boundedness of $t_{\varepsilon}$ we obtain

$$\begin{aligned} \begin{aligned}[b] H(\varepsilon)&=\frac{t_{\varepsilon}^{2}}{2} \int_{\mathrm {R}^{3}}|w_{\varepsilon}|^{2}\,dx- \frac{\mu t_{\varepsilon}^{2}}{2} \int _{\mathrm{R}^{3}}h(x)|w_{\varepsilon}|^{2}\,dx \\ &\leq C\varepsilon^{\frac{1}{2}}-\mu C \varepsilon^{1-\frac {\beta}{2}}. \end{aligned} \end{aligned}$$

(3.19)

Since $1<\beta<3$, for fixed $\mu>0$, we obtain

$$ \frac{H(\varepsilon)}{\varepsilon^{\frac{1}{2}}}\rightarrow-\infty \quad \mbox{as } \varepsilon\rightarrow0. $$

(3.20)

It follows from (3.17), (3.18), and (3.20) that the proof of Lemma 3.3 is complete. □

Proof of Theorem 1.1

By the definition of $c_{1}$ there exists a sequence $\{u_{n}\}\subset N$ such that $I(u_{n})\rightarrow c_{1}$ as $n\rightarrow\infty$. Then we obtain that

$$ \|u_{n}\|^{2}+b\biggl( \int_{\mathrm{R}^{3}}|\nabla u_{n}|^{2}\,dx \biggr)^{2}- \int _{\mathrm{R}^{3}}\mu h(x)|u_{n}|^{2}\,dx= \int_{\mathrm{R}^{3}}k(x)|u_{n}|^{6}\,dx. $$

(3.21)

It follows from (3.21) and Lemma 2.2 that

$$\begin{aligned} c_{1}+o(1) =&\frac{1}{3}\biggl(\|u_{n} \|^{2}-\mu \int_{\mathrm {R}^{3}}h(x)|u_{n}|^{2}\,dx\biggr)+ \biggl(\frac{b}{4}-\frac{b}{6}\biggr) \biggl( \int_{\mathrm {R}^{3}}|\nabla u_{n}|^{2}\,dx \biggr)^{2} \\ \geq&\frac{1}{3}\biggl(1-\frac{\mu}{\tilde{\mu}}\biggr)\|u_{n} \|^{2}, \end{aligned}$$

(3.22)

which implies the boundedness of $\{u_{n}\}$ in $H^{1}(\mathrm{R}^{3})$ since $0<\mu<\tilde{\mu}$. Then there exists a subsequence of $\{ u_{n}\}$, still denoted by $\{u_{n}\}$, such that $u_{n}\rightharpoonup u$ in $H^{1}(\mathrm{R}^{3})$. By (2) of Lemma 3.2 and Lemma 3.3 we have $u\neq0$. By the definition of $t(u)$ we get $t(u)u\in{N}$. So $I(t(u)u)\geq c_{1}$. We claim that $u_{n}\rightarrow u$ in $H^{1}(\mathrm {R}^{3})$. Otherwise, by (1) and (3) of Lemma 3.2, we would get that $c_{1}>I(t(u)u)$ or $c_{1}>c^{*}$. In any case, we get a contradiction since $c_{1}< c^{*}$. Therefore, $\{u_{n}\}$ converges strongly to u. Thus, $u\in {N}$ and $I(u)=c_{1}$. By the Lagrange multiplier rule there exists $\theta\in \mathrm{R}$ such that $I'(u)=\theta G'(u)$ and thus

$$0=\bigl\langle I'(u),u\bigr\rangle =\theta \biggl(2\|u \|^{2}+4b\biggl( \int _{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2}-6 \int_{\mathrm{R}^{3}} k(x)|u|^{6}\,dx-2\mu \int_{\mathrm{R}^{3}}h(x)|u|^{2}\,dx \biggr). $$

Since $u\in N$, we get

$$0=\theta \biggl(-4\biggl(\|u\|^{2}-\mu \int_{\mathrm {R}^{3}}h(x)|u|^{2}\,dx\biggr)-2b\biggl( \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \biggr), $$

which implies that $\theta=0$ and u is a nontrivial critical point of the functional I in $H^{1}(\mathrm{R}^{3})$. Therefore, the nonzero function u can solve Eq. (1.1), that is,

$$ -\biggl(a+b \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)\triangle u+u= k(x)|u|^{2^{*}-2}u+\mu h(x)u. $$

(3.23)

In (3.23), using $u^{-}=\max\{-u,0\}$ as a test function and integrating by parts, by (k₁), (h₂), and ($\mu_{1}$) we obtain

$$\begin{aligned} 0 =& \int_{\mathrm{R}^{3}}a\bigl\vert \nabla u^{-}\bigr\vert ^{2} \,dx+ \int_{\mathrm {R}^{3}}\bigl\vert u^{-}\bigr\vert ^{2}\,dx+b \int_{\mathrm{R}^{3}}\vert \nabla u\vert ^{2}\,dx \int_{\mathrm {R}^{3}}\bigl\vert \nabla u^{-}\bigr\vert ^{2}\,dx \\ &{}+ \int_{\mathrm{R}^{3}}k(x)\bigl\vert u^{-}\bigr\vert ^{2^{*}-2}\bigl\vert u^{-}\bigr\vert ^{2}\,dx+ \int_{\mathrm {R}^{3}}\mu h(x)\bigl\vert u^{-}\bigr\vert ^{2} \,dx\geq0. \end{aligned}$$

Then $u^{-}=0$ and $u\geq0$. From Harnack’s inequality [27] we can infer that $u>0$ for all $x\in \mathrm{R}^{3}$. Therefore, u is a positive solution of (1.1). The proof is complete by choosing $\omega_{0}=u$. □

4 Sign-changing solution

This subsection is devoted to proving the existence of sign-changing solution of Eq. (1.1). Let $\overline{{{N}}}=\{u=u^{+}-u^{-}\in H^{1}(\mathrm{R}^{3}):u^{+}\in{N}, u^{-}\in{N}\}$, where $u^{\pm}=\max\{ \pm u,0\}$. If $u^{+}\neq0$ and $u^{-}\neq0$, then u is called a sign-changing function. We define $c_{2}= \inf_{u\in\overline{{{N}}}}I(u)$.

Lemma 4.1

Assume that ($\mu_{1}$), (k₁)-(k₂), and (h₁)-(h₂) hold. Then for $\frac{3}{2}<\beta<3$, $c_{2}< c_{1}+c^{*}$.

Proof

By Lemma 2.4, using first the same argument as in [22] or [28], we have that there are $s_{1}>0$ and $s_{2}\in \mathrm{R}$ such that

$$ s_{1}\omega_{0}+s_{2}\omega_{\varepsilon}\in \overline{{N}}. $$

(4.1)

Next, we prove that there exists $\varepsilon>0$ small enough such that

$$ \sup_{s_{1}>0,s_{2}\in\mathrm{R}}I(s_{1}\omega_{0}+s_{2} \omega _{\varepsilon})< c_{1}+c^{*}. $$

(4.2)

Obviously, it follows from (2) of Lemma 2.3 that, for any $s_{1}>0$ and $s_{2}\in \mathrm{R}$ satisfying $\|s_{1}\psi_{1}+s_{2}\omega_{\varepsilon}\|>\rho$, $I(s_{1}\omega_{0}+s_{2}\omega_{\varepsilon})<0$. We only estimate $I(s_{1}\omega_{0}+s_{2}\omega_{\varepsilon})$ for all $\|s_{1}\omega _{0}+s_{2}\omega_{\varepsilon}\|\leq\rho$. By calculation we see that

$$ I(s_{1}\omega_{0}+s_{2}\omega_{\varepsilon})=I(s_{1} \omega _{0})+\Pi_{1}+\Pi_{2}+\Pi_{3}+ \Pi_{4}+\Pi_{5}+\Pi_{6}, $$

(4.3)

where

$$\begin{aligned}& \Pi_{1}=\frac{as_{2}^{2}}{2} \int_{\mathrm {R}^{3}}\vert \nabla w_{\varepsilon} \vert ^{2} \,dx+\frac{bs_{2}^{4}}{4}\biggl( \int_{\mathrm {R}^{3}}\vert \nabla w_{\varepsilon} \vert ^{2} \,dx\biggr)^{2}-\frac{ s_{2}^{6}}{6} \int_{\mathrm {R}^{3}}k(x_{0})\vert w_{\varepsilon} \vert ^{6}\,dx, \\& \Pi_{2}=\frac{ s_{2}^{6}}{6} \int_{\mathrm{R}^{3}}k(x_{0})\vert w_{\varepsilon} \vert ^{6}\,dx-\frac{ s_{2}^{6}}{6} \int_{\mathrm{R}^{3}}k(x)\vert w_{\varepsilon} \vert ^{6} \,dx, \\& \Pi_{3}=\frac{1}{6} \int_{\mathrm{R}^{3}}k(x) \bigl(\vert s_{1}\omega _{0}\vert ^{6}+\vert s_{2}w_{\varepsilon} \vert ^{6}-\vert s_{1}\omega_{0}+s_{2}w_{\varepsilon} \vert ^{6}\bigr)\,dx, \\& \Pi_{4}=\frac{s_{2}^{2}}{2} \int_{\mathrm {R}^{3}}\vert w_{\varepsilon} \vert ^{2}\,dx- \frac{\mu s_{2}^{2}}{2} \int_{\mathrm {R}^{3}}h(x)\vert w_{\varepsilon} \vert ^{2} \,dx, \\& \Pi_{5}=\frac{b}{4}\biggl[\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla (s_{1} \omega_{0}+s_{2}\omega_{\varepsilon})\bigr\vert ^{2}\,dx\biggr)^{2}-\biggl( \int_{\mathrm {R}^{3}}\bigl\vert \nabla(s_{1} \omega_{0})\bigr\vert ^{2}\,dx\biggr)^{2} \\& \hphantom{\Pi_{5}={}}{}-\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla (s_{2} \omega_{\varepsilon})\bigr\vert ^{2}\,dx\biggr)^{2}\biggr], \end{aligned}$$

and

$$\Pi_{6}= \int_{\mathrm{R}^{3}}\bigl(a\nabla(s_{1}\omega_{0}) \nabla ( s_{2}\omega_{\varepsilon})+(s_{1} \omega_{0}) (s_{2}\omega_{\varepsilon})-\mu h(x) (s_{1}\omega_{0}) (s_{2}\omega_{\varepsilon}) \bigr)\,dx. $$

By (3.16) we obtain that

$$ \sup_{s_{2}\in\mathrm{R}}\Pi_{1}=\frac{abS^{3}}{4\|k\| _{\infty}}+ \frac{b^{3}S^{6}}{24\|k\|_{\infty}^{2}}+\frac{(b^{2}S^{4}+4a\|k\| _{\infty}S)^{\frac{3}{2}}}{24\|k\|_{\infty}^{2}}+O\bigl(\varepsilon^{\frac{1}{2}}\bigr). $$

(4.4)

It follows from (3.18) that

$$ \Pi_{2}\leq C\varepsilon^{\frac{1}{2}}. $$

(4.5)

From the elementary inequality

$$|s+t|^{q}\geq|s|^{q}+|t|^{q}-C \bigl(|s|^{q-1}t+|t|^{q-1}s\bigr) \quad \text{for any } q\geq1, $$

the fact that $\omega_{0}\in H^{1}(\mathrm{R}^{3})\cap L^{\infty}(\mathrm {R}^{3})$, and from (3.14) we have

$$\begin{aligned} \Pi_{3} \leq& C \int_{\mathrm{R}^{3}}k(x) \bigl(|\omega_{0}|^{5}\omega _{\varepsilon}+\omega_{0}|w_{\varepsilon}|^{5}\bigr)\,dx \\ \leq&\|k\|_{\infty}\|\omega_{0}\|_{\infty}\int_{\mathrm {R}^{3}}|w_{\varepsilon}|^{5}\,dx+\|k \|_{\infty}\bigl\| \omega_{0}^{5}\bigr\| _{\infty}\int _{\mathrm{R}^{3}} w_{\varepsilon}\,dx \\ \leq& C\varepsilon^{\frac{1}{4}}. \end{aligned}$$

(4.6)

By (3.19) we have

$$ \Pi_{4}\leq C\varepsilon^{\frac{1}{2}}-C\varepsilon ^{1-\frac{\beta}{2}}, $$

(4.7)

and using (3.13), we have

$$\begin{aligned} \Pi_{5} \leq&\frac{b}{4}\biggl[4\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla (s_{1} \omega_{0})\bigr\vert ^{2}\,dx\biggr)^{2}+4 \biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla(s_{2}\omega _{\varepsilon})\bigr\vert ^{2}\,dx\biggr)^{2} \\ &{}-\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla(s_{1} \omega_{0})\bigr\vert ^{2}\,dx\biggr)^{2}-\biggl( \int _{\mathrm{R}^{3}}\bigl\vert \nabla(s_{2} \omega_{\varepsilon})\bigr\vert ^{2}\,dx\biggr)^{2}\biggr] \\ =&\frac{3b}{4}\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla(s_{1}\omega _{0})\bigr\vert ^{2}\,dx\biggr)^{2}+ \frac{3b}{4}\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla(s_{2}\omega _{\varepsilon})\bigr\vert ^{2}\,dx\biggr)^{2} \\ \leq& C+C\varepsilon^{\frac{1}{2}}. \end{aligned}$$

(4.8)

Since $\omega_{0}$ is a positive solution of (1.1), by the Sobolev inequality we obtain

$$\begin{aligned} \Pi_{6} =&s_{1}s_{2} \int_{\mathrm{R}^{3}}k(x)|\omega_{0}|^{5}\omega _{\varepsilon}\,dx-b \int_{\mathrm{R}^{3}}\bigl\vert \nabla(s_{1} \omega_{0})\bigr\vert ^{2}\,dx \int _{\mathrm{R}^{3}}\nabla(s_{1}\omega_{0}) \nabla(s_{2}\omega_{\varepsilon}) \,dx \\ \leq&\|k\|_{\infty}\bigl\| \omega_{0}^{5}\bigr\| _{\infty}\int_{\mathrm{R}^{3}} w_{\varepsilon}\,dx+b\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla(s_{1}\omega _{0})\bigr\vert ^{2}\,dx\biggr)^{\frac{3}{2}}\biggl( \int_{\mathrm{R}^{3}}\bigl\vert \nabla(s_{2}\omega _{\varepsilon}) \bigr\vert ^{2} \,dx\biggr)^{\frac{1}{2}} \\ \leq& C\varepsilon^{\frac{1}{4}}. \end{aligned}$$

(4.9)

It follows from (4.3)-(4.9) that, for $\frac{3}{2}<\beta<3$,

$$\begin{aligned} I(s_{1}\omega_{0}+s_{2}\omega_{\varepsilon}) \leq& I(s_{1}\omega _{0})+c^{*}+C+C\varepsilon^{\frac{1}{4}}+C \varepsilon^{\frac {1}{2}}-C\varepsilon^{1-\frac{\beta}{2}} \\ < &I(s_{1}\omega_{0})+c^{*}=c_{1}+c^{*} \end{aligned}$$

as $\varepsilon\rightarrow0$, which implies that (4.2) holds. This finishes the proof of Lemma 4.1. □

Lemma 4.2

Suppose that ($\mu_{1}$), (k₁)-(k₂), and (h₁)-(h₂) hold. Then, for $\frac{3}{2}<\beta<3$, there exists $\omega_{1}\in\overline{{N}}$ such that $I(\omega_{1})=c_{2}$.

Proof

Let $\{u_{n}\}\subset\overline{{N}}$ be such that $I(u_{n})\rightarrow c_{2}$. Since $u_{n}\in\overline{{N}}$, we may assume that there exist constants $d_{1}$ and $d_{2}$ such that $I(u^{+}_{n})\rightarrow d_{1}$ and $I(u^{-}_{n})\rightarrow d_{2}$ and $d_{1}+d_{2}=c_{2}$. Then

$$ d_{1}\geq c_{1},\qquad d_{2}\geq c_{1}. $$

(4.10)

Just as the proof of (3.22), we can prove the boundedness of $\{u^{+}_{n}\}$ and $\{u^{-}_{n}\}$. Going, if necessary, to a subsequence, we may assume that $u_{n}^{\pm}\rightharpoonup u^{\pm}$ in $H^{1}(\mathrm{R}^{3})$ as $n\rightarrow\infty$.

We claim $u^{+}\neq0$ and $u^{-}\neq0$. Arguing by contradiction, if $u^{+}=0$ or $u^{-}=0$, then by (4.10) and Lemma 3.2,

$$c_{1}+c^{*}\leq d_{2}+d_{1}=c_{2}, $$

which contradicts Lemma 4.1. Hence, $u^{+}\neq0$ and $u^{-}\neq0$. We claim that $u^{\pm}_{n}\rightarrow u^{\pm}$ strongly in $H^{1}(\mathrm {R}^{3})$. Indeed, according to Lemma 3.2, we get one of the following:

• (i) $\{u^{+}_{n}\}$ converges strongly to $u^{+}$;

• (ii) $d_{1}>I(t(u^{+})u^{+})$;

• (iii) $d_{1}> c^{*}$;

and we also have one of the following:

• (iv) $\{u^{-}_{n}\}$ converges strongly to $u^{-}$;

• (v) $d_{2}>I(t(u^{-})u^{-})$;

• (vi) $d_{2}> c^{*}$.

We will prove that only cases (i) and (iv) hold. For example, in cases (i) and (v) or (ii) and (v), from $u^{+}-t(u^{-})u^{-}\in\overline{{N}}$ or $t(u^{+})u^{+}-t(u^{-})u^{-}\in\overline{{N}}$ we have

$$c_{2}\leq I\bigl(u^{+}-t\bigl(u^{-}\bigr)u^{-}\bigr)=I\bigl(u^{+}\bigr)+I \bigl(-t\bigl(u^{-}\bigr)u^{-}\bigr)< d_{1}+d_{2}=c_{2} $$

or

$$c_{2}\leq I\bigl(t\bigl(u^{+}\bigr)u^{+}-t\bigl(u^{-}\bigr)u^{-}\bigr)=I \bigl(t\bigl(u^{+}\bigr)u^{+}\bigr)+I\bigl(-t\bigl(u^{-}\bigr)u^{-}\bigr)< d_{1}+d_{2}=c_{2}. $$

Any one of the two inequalities is impossible. In cases (i) and (vi) or (ii) and (vi) or (iii) and (vi), we have

$$\begin{aligned}& c_{1}+c^{*}\leq I\bigl(u^{+}\bigr)+c^{*}< d_{1}+d_{2}=c_{2}, \\& c_{1}+c^{*}\leq I\bigl(t\bigl(u^{+}\bigr)u^{+}\bigr)+c^{*}< d_{1}+d_{2}=c_{2}, \\& c_{1}+c^{*}\leq c^{*}+c^{*}< d_{1}+d_{2}=c_{2}, \end{aligned}$$

and any one of the three inequalities is a contradiction. Therefore, we prove that only (i) and (iv) hold. Hence, we obtain that $\{u^{+}_{n}\}$ and $\{u^{-}_{n}\}$ converge strongly to $u^{+}$ and $u^{-}$, respectively, and we obtain $u^{+}, u^{-}\in{N}$. Denote $\omega_{1}=u^{+}-u^{-}$. Then $\omega_{1}\in\overline{{N}}$ and $I(\omega_{1})=d_{1}+d_{2}=c_{2}$. □

Proof of Theorem 1.2

Now we show that $\omega_{1}$ is a critical point of I in $H^{1}(\mathrm{R}^{3})$. Arguing by contradiction, assume that $I'(\omega_{1})\neq0$. For any $u\in{N}$, we claim that $\|G'(u)\|_{H^{-1}}=\sup_{\|v\|=1}|\langle G'(u),v\rangle|\neq0$. In fact, by the definition of N and Lemma 2.2, for any $u\in{N}$, we have

$$\begin{aligned} \bigl\langle G'(u),u\bigr\rangle =&2 \biggl(\|u\|^{2}- \mu \int _{\mathrm{R}^{3}}h(x)|u|^{2}\,dx+b\biggl( \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \biggr)+2b\biggl( \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \\ &{}-6 \int_{\mathrm{R}^{3}}k(x)|u|^{6}\,dx \\ =&2\biggl(\|u\|^{2}-\mu \int_{\mathrm{R}^{3}}h(x)|u|^{2}\,dx+b\biggl( \int_{\mathrm {R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \biggr)+2b\biggl( \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \\ &{}-6\biggl(\|u\|^{2}-\mu \int_{\mathrm{R}^{3}}h(x)|u|^{2}\,dx+b\biggl( \int_{\mathrm {R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \biggr) \\ =&-4 \biggl(\|u\|^{2}-\mu \int_{\mathrm{R}^{3}}h(x)|u|^{2}\,dx+b\biggl( \int_{\mathrm {R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \biggr)+2b\biggl( \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \\ \leq&-4\biggl[\biggl(1-\frac{\mu}{\tilde{\mu}}\biggr)\|u\|^{2}+b\biggl( \int_{\mathrm {R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2} \biggr]+2b\biggl( \int_{\mathrm{R}^{3}}|\nabla u|^{2}\,dx\biggr)^{2}< 0. \end{aligned}$$

Then we define

$$\Phi(u)=I'(u)- \biggl\langle I'(u), \frac{G'(u)}{\|G'(u)\| } \biggr\rangle \frac{G'(u)}{\|G'(u)\|},\quad u\in{N}. $$

Choose $\lambda\in(0, \min\{\|u^{+}\|,\|u^{-}\|\}/3)$ such that $\|\Phi(v)-\Phi(u)\|\leq\frac{1}{2}\|\Phi(\omega_{1})\|$ for any $v\in N$ with $\|v-\omega_{1}\|\leq2\lambda$. Let $\chi:N\rightarrow [0,1]$ be a Lipschitz mapping such that

$$\chi(v)= \textstyle\begin{cases} 0, & v\in N \text{ with } \|v-\omega_{1}\|\geq2\lambda, \\ 1, & v\in N \text{ with } \|v-\omega_{1}\|\leq\lambda, \end{cases} $$

and for positive constant $s_{0}$, let $\eta:[0,s_{0}]\times N\rightarrow N$ be the solution of the differential equation

$$\eta(0,v)=0, \qquad \frac{d\eta(s,v)}{ds}=-\chi\bigl(\eta(s,v)\bigr)\Phi\bigl(\eta (s,v)\bigr) \quad \text{for } (s,v)\in[0,s_{0}]\times N. $$

We set

$$\psi(\tau)=t\bigl((1-\tau)\omega_{1}^{+}+\tau\omega_{1}^{-} \bigr) \bigl((1-\tau)\omega _{1}^{+}+\tau\omega_{1}^{-}, \xi( \tau)=\eta\bigl(s_{0},\psi(\tau)\bigr)\bigr)\quad \text{for } 0\leq \tau\leq1. $$

We now give the proof of the fact that $I(\xi(\tau))< I(u)$ for some $\tau\in(0,1)$. Obviously, if $\tau\in(0,\frac{1}{2})\cup (\frac{1}{2},1)$, then we have $I(\xi(\frac{1}{2}))< I(\psi(\frac {1}{2}))< I(\omega_{1})$ and $I(\xi(\tau))\leq I(\psi(\tau))< I(\omega_{1})$.

Since $t(\xi^{+}(\tau))-t(\xi^{-}(\tau))\rightarrow-\infty$ as $\tau \rightarrow0+0$ and $t(\xi^{+}(\tau))-t(\xi^{-}(\tau))\rightarrow +\infty$ as $\tau\rightarrow1-0$, there exists $\tau_{1}\in(0,1)$ such that $t(\xi^{+}(\tau))=t(\xi^{-}(\tau))$. Thus, $\xi(\tau_{1})\in \overline{{N}}$ and $I(\xi(\tau_{1}))< I(\omega_{1})$, which contradicts to the definition of $c_{2}$. Hence, we get that $I'(\omega_{1})=0$ and $\omega_{1}$ is a sign-changing solution of problem (1.1). The proof of Theorem 1.2 is complete. □