Improved results of nontrivial solutions for a nonlinear nonhomogeneous Klein–Gordon–Maxwell system involving sign-changing potential

Abstract

This paper is concerned with the following system:

$$ \textstyle\begin{cases} {-\Delta u+\lambda A(x) u-K(x)(2 \omega +\phi ) \phi u=f(x, u)+h(x), \quad x \in \mathbb{R}^{3}}, \\ {\Delta \phi =K(x)(\omega +\phi ) u^{2}, \quad x \in \mathbb{R}^{3}}, \end{cases} $$

where $\lambda \geq 1$ is a parameter, $\omega >0$ is a constant and the potential A is sign-changing. Under the classic Ambrosetti–Rabinowitz condition and other suitable conditions, nontrivial solutions are obtained via the linking theorem and Ekeland’s variational principle. Especially speaking, we use a super-quadratic condition to replace the 4-superlinear condition which is usually used to show the existence of nontrivial solutions in many references. Our results improve the previous results in the literature.

1 Introduction

The following type of Klein–Gordon–Maxwell system is considered:

$$ \textstyle\begin{cases} {-\Delta u+\lambda A(x) u-K(x)(2 \omega +\phi ) \phi u=f(x, u)+h(x), \quad x \in \mathbb{R}^{3}}, \\ \Delta \phi =K(x)(\omega +\phi ) u^{2}, \quad x \in \mathbb{R}^{3}, \end{cases} $$

(1.1)

where $\lambda \geq 1$ is a parameter, $\omega >0$ is a constant, $A \in C (\mathbb{R}^{3}, \mathbb{R} )$, $f \in C (\mathbb{R}^{3} \times \mathbb{R}, \mathbb{R} )$, and f satisfies the following basic condition:

(F1):

$f(x, t)=o(|t|)$uniformly inxas$t \rightarrow 0$, there exists a constant$C>0$such that$|f(x, t)| \leq C (|t|+|t|^{q} )$,$2< q<6$, for all$(x, t)$, and$F(x, t)=\int _{0}^{t} f(x, s) \,ds \geq 0$for all$(x, t)$.

The Klein–Gordon–Maxwell system was first introduced by Benci and Fortunato [1] as a model to describe a nonlinear Klein–Gordon equation interacting with an electromagnetic field.

When $h(x)\equiv 0$, $K(x)\equiv 1$ and $\lambda A(x)\equiv m^{2}-\omega ^{2}$, system (1.1) becomes to the following homogeneous system (1.2). We now first recall some results about homogeneous case. In 2002, Benci and Fortunato proved that system (1.2) with constant potential has infinitely many radially symmetric nontrivial solutions when $f(x, u)=|u|^{q-2} u$, ω and m are constants and $|m|>|\omega |$, $4< q<6$. For more physical background, see [2]. We have

$$ \textstyle\begin{cases} {-\Delta u+ [m^{2}-(\omega +\phi )^{2} ] \phi u=f(x, u), \quad x \in \mathbb{R}^{3},} \\ {\Delta \phi =(\omega +\phi ) u^{2}, \quad x \in \mathbb{R}^{3}}. \end{cases} $$

(1.2)

The authors in [3] investigated the case $2< q<4$ and $0<\omega <\sqrt{\frac{q}{2}-1} m$. Later, the existence of a ground state solution for (1.2) was obtained in [4] either $4 \leq q<6$ and $\omega < m$, or $2< q<4$ and $\omega < m \sqrt{(q-2)/(6-q)}$.

When the potential A was an external Coulomb function, or a steep function, or a periodic function, or a sign-changing function, etc., the Klein–Gordon–Maxwell system had been extensively studied in the past decades. For example, positive ground state solutions for the following system were obtained by Cunha [5]:

$$ \textstyle\begin{cases} {-\Delta u+A(x) u+ [m^{2}-(\omega +\phi )^{2} ] \phi u= \xi \vert u \vert ^{q-2} u+ \vert u \vert ^{2^{*}-2} u, \quad x \in \mathbb{R}^{3}}, \\ {\Delta \phi =(\omega +\phi ) u^{2}, \quad x \in \mathbb{R}^{3}}, \end{cases} $$

(1.3)

where A is a periodic potential.

In [6], Georgiev and Visciglia investigated a homogeneous system with a small external Coulomb potential and $\lambda =1$. In [7], Chen and Tang considered the geometrically distinct solutions for Klein–Gordon–Maxwell systems by using Lusternik–Schnirelmann theory. In [8], Ding and Li proved that the Klein–Gordon–Maxwell system with sign-changing potential had infinitely many standing wave solutions. Liu, Chen and Tang [9] studied the ground state solutions for Klein–Gordon–Maxwell system with steep potential well. In [10], Wang improved the results of [2].

Next, let us present some results for the nonhomogeneous case.

When $\lambda \equiv 1$ and $K(x)\equiv 1$, Shi and Chen [11] established the multiplicity of solutions for nonhomogeneous system (1.1). Wang and Chen [12] investigated the system (1.1) with sign-changing potential A and f satisfies the following crucial assumptions:

(F2)’:

$F(x, t) / t^{4} \rightarrow +\infty$as$|t| \rightarrow +\infty$uniformly inx;

(F3)’:

$\mathcal{F}(x, t) :=\frac{1}{4} f(x, t) t-F(x, t) \geq 0$for all$(x, t) \in \mathbb{R}^{3} \times \mathbb{R}$;

Existence and multiplicity of solutions for a type of Klein–Gordon–Maxwell system with sign-changing potentials were got via the symmetric mountain pass theorem in [13]. In [14], under a variant super-quadratic condition, two solutions for a nonhomogeneous Klein–Gordon–Maxwell system were got by Wang via the mountain pass theorem and Ekeland’s variational principle. Via Ekeland’s variational principle and the mountain pass theorem, the author in [15] studied the nonhomogeneous Klein–Gordon–Maxwell system with constant potential.

Finally, we mention some recent work also related to the Klein–Gordon–Maxwell system. In [16], the authors investigated positive ground state solutions for a kind of fractional Klein–Gordon–Maxwell system. In [17, 18], some results on reaction-diffusion equations involving fractional operators were obtained. In [19], some results on Klein–Gordon equations involving fractional operators were obtained. The authors in [20] studied a nonlinear heat equation and obtained some results. In [21], the authors investigated the numerical computation of Klein–Gordon equation by using a homotopy analysis transform method.

Inspired by the above-mentioned work, we will make the following conditions which are weaker than conditions (F2)’ and (F3)’. Otherwise, we consider a more general potential $A(x)$:

(F2):

$F(x, t) / t^{2} \rightarrow +\infty$as$|t| \rightarrow +\infty$uniformly inx;

(F3):

There exists$\mu >2$such that$\mu F(x, t) \leq f(x, t) t$for all$(x, t) \in \mathbb{R}^{3} \times \mathbb{R}$;

(A1):

There is$a>0$such that$\operatorname{meas} \{ x \in \mathbb{R}^{3} : A(x) \leq a \} <+\infty$;

(A2):

$A \in C (\mathbb{R}^{3}, \mathbb{R} )$is bounded from below;

(A3):

$\varOmega =\operatorname{int} A^{-1}(0)$is nonempty,∂Ωis smooth boundary,$\overline{\varOmega }=A^{-1}(0)$.

In our assumptions, the nonlinearity f just needs to satisfy a super-quadratic condition at infinity. The 4-superlinear assumption is not necessary. Conditions (A1)–(A3) were first introduced in [22]. Since the potential in (1.1) is sign-changing, the usual way of verifying the compactness is invalid. Following [12, 23], we establish the parameter which is dependent on compactness conditions to recover the compactness. The following assumptions will be needed throughout the paper.

(F4):

There exist$a_{1}$,$L_{0}>0$and$\sigma \in (3 / 2,2)$such that

$$ \bigl\vert f(x, t) \bigr\vert ^{\sigma } \leq a_{1} \mathcal{F}(x, t) \vert t \vert ^{\sigma }, \quad \text{for all } x \in \mathbb{R}^{3} \text{ and } \vert t \vert \geq L_{0}, $$

where$\mathcal{F}(x, t) :=\frac{1}{\mu } f(x, t) t-F(x, t)$;

(K):

$K(x) \in L^{3} (\mathbb{R}^{3} ) \cup L^{\infty } ( \mathbb{R}^{3} )$,$K(x) \geq 0$and$K(x) \not \equiv 0$for a.e.$x \in \mathbb{R}^{3}$;

(H):

$h(x) \in L^{2} (\mathbb{R}^{3} )$and$h(x) \geq 0$for a.e.$x \in \mathbb{R}^{3}$.

Remark 1.1

It is not difficult to see that (F2) and (F3) are much weaker than (F2)’ and (F3)’, respectively. The following function satisfies (F2) and (F3) but not (F2)’ and (F3)’:

$$ f(x, t)=c(x) \vert t \vert ^{\varrho -2}, \quad \forall (x, t) \in \mathbb{R}^{3} \times \mathbb{R}, $$

where $\varrho \in (2,4)$, c is a continuous function with $\inf_{x \in \mathbb{R}^{3}} c(x)>0$.

Theorem 1.2

Let (F1)–(F4), (A1)–(A2), (K) and (H) hold. If there exists$x_{0} \in \mathbb{R}^{3}$such that$A(x_{0})<0$, then$\forall n \in \mathbb{N}$, there exist$\lambda _{n}>n$, $b_{n}>0$and$\eta _{n}>0$such that system (1.1) admits at least two nontrivial solutions for every$\lambda =\lambda _{n}$, $|K|_{\infty }< b_{n}$ (or$|K|_{3}< b_{n}$) and$|h|_{2} \leq \eta _{n}$.

Theorem 1.3

Suppose (F1)–(F4), (A1)–(A3), (K) and (H) hold. If the interior of$A^{-1}(0)$is nonempty, then there exist$\bar{\varLambda }>0$, $b_{\lambda }>0$and$\eta _{\lambda }>0$such that problem (1.1) possesses at least two nontrivial solutions for every$\lambda >\bar{\varLambda }$, $|h|_{2} \leq \eta _{\lambda }$and$|K|_{\infty }< b_{\lambda }$ (or$|K|_{3}< b_{\lambda }$).

Theorem 1.4

Assume$A \geq 0$and let (F1)–(F4), (A1)–(A3), (K) and (H) hold. If the interior of$A^{-1}(0)$is nonempty and$h \neq 0$, then there exist$\tilde{\varLambda }>0$and$\eta >0$such that system (1.1) has at least two nontrivial solutions for every$\lambda >\tilde{\varLambda }$and$|h|_{2} \leq \eta$.

In the present paper, we use weaker conditions than the previous literature to show the boundedness of Palais–Smale sequence and obtain nontrivial solutions for the nonhomogeneous Klein–Gordon–Maxwell system involving sign-changing potential, which extend and generalize the related results in the literature.

In this paper, C denotes different positive constant in different place. The paper is organized by four sections. Some preliminary results are stated in Sect. 2. The proofs of main results are given in Sect. 3. The conclusion is given in Sect. 4.

2 Preliminaries

Some notations are given first. For $1 \leq s \leq +\infty$, $L^{s}(\varOmega )$ denotes a Lebesgue space with the norm given by $|\cdot |_{s}$. Let $D^{1,2} (\mathbb{R}^{3} )$ be the completion of $C_{0}^{\infty } (\mathbb{R}^{3} )$ endowed with the norm

$$ \Vert u \Vert _{D}^{2} := \Vert u \Vert _{D^{1,2} (\mathbb{R}^{3} )}^{2}= \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2} \,d x. $$

The space $H^{1} (\mathbb{R}^{3} )$ is endowed with the following standard product and norm, respectively:

$$ (u, v)= \int _{\mathbb{R}^{3}}(\nabla u \nabla v+u v) \,d x ; \qquad \Vert u \Vert ^{2}= \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+ \vert u \vert ^{2} \bigr) \,d x. $$

The best Sobolev constant S is given by

$$ S=\inf_{u \in D^{1,2} (\mathbb{R}^{3} ) \backslash \{0\}} \frac{ \Vert u \Vert _{D}}{ \vert u \vert _{6}}. $$

For any $\rho >0$ and $x\in \mathbb{R}^{3}$, $B_{\rho }(x)$ denotes the ball of radius ρ centered at x.

As pointed out in [4], the existence of solutions are not related to the signs of ω, so one can assume that $\omega >0$. Similar to [12], we now first give the variational structure of system (1.1).

Let $A^{+}=\max \{ A(x), 0\}$ and $A^{-}=\max \{-A(x), 0\}$, then $A(x)=A^{+}(x)-A^{-}(x)$. Let

$$ H= \biggl\{ u \in H^{1} \bigl(\mathbb{R}^{3} \bigr) : \int _{ \mathbb{R}^{3}}\bigl[ \vert \nabla u \vert ^{2}+A^{+}(x) u^{2} \bigr]\,d x< \infty \biggr\} $$

be a Hilbert space, whose inner product and norm are given by

$$ (u, v)= \int _{\mathbb{R}^{3}} \bigl(\nabla u \nabla v+A^{+}(x) u v \bigr) \,d x\quad \text{and}\quad \Vert u \Vert =(u, u)^{1 / 2}, $$

respectively. For $\lambda \geq 1$, the inner product and norm are defined as

$$ (u, v)_{\lambda }= \int _{\mathbb{R}^{3}} \bigl(\nabla u \nabla v+ \lambda A^{+}(x) u v \bigr) \,d x, \qquad \Vert u \Vert _{\lambda }=(u, u)_{ \lambda }^{1 / 2}. $$

It is obvious $\|u\| \leq \|u\|_{\lambda }$ for $\lambda \geq 1$. Let $H_{\lambda }= (H,\|\cdot \|_{\lambda } )$. By (A1)–(A2) and the Poincaré inequality, the embedding $H_{\lambda } \hookrightarrow H^{1} (\mathbb{R}^{3} )$ is continuous. Thus, for $s \in [2,6]$, there exists $\gamma _{s}>0$ which is independent of λ such that

$$ \vert u \vert _{s} \leq \gamma _{s} \Vert u \Vert _{\lambda }, \quad \forall u \in H_{ \lambda }. $$

(2.1)

Let $E_{\lambda }= \{ u \in H_{\lambda } : \operatorname{supp} u \subset A^{-1}([0, \infty )) \}$, $E_{\lambda }^{\perp }$ be the orthogonal complement of $E_{\lambda }$ in $H_{\lambda }$. Obviously, if $A \geq 0$, then $E_{\lambda }=H_{\lambda }$, otherwise $E_{\lambda }^{\perp } \neq \{0\}$. Define

$$ A_{\lambda } :=-\Delta +\lambda A, $$

the corresponding bilinear form of $A_{\lambda }$ is defined as

$$ a_{\lambda }(u, v)= \int _{\mathbb{R}^{3}}\bigl(\nabla u \nabla v+\lambda A(x) u v\bigr) \,d x, $$

it is obvious that $A_{\lambda }$ is self-adjoint in $L^{2} (\mathbb{R}^{3} )$ and $a_{\lambda }(u, v)$ is continuous in $H_{\lambda }$. As in [23], for fixed $\lambda >0$, consider the following eigenvalue problem:

$$ -\Delta u+\lambda A^{+}(x) u=\xi \lambda A^{-}(x) u, \quad u \in E_{ \lambda }^{\perp }. $$

(2.2)

From (A1)–(A2), the mapping $u \mapsto \int _{\mathbb{R}^{3}} \lambda A^{-}(x) u^{2} \,d x$ is weakly continuous. Hence, following [24], the following proposition is obtained in [12].

Proposition 2.1

([12])

Let (A1), (A2) be satisfied, then, for any$\tilde{\varLambda }>0$, there exists a sequence of positive eigenvalues$\{ \xi _{i}(\lambda ) \}$for problem (2.2), which is characterized by

$$ \xi _{i}(\lambda )=\inf_{\operatorname{dim} N \geq i, N \subset E_{ \lambda }^{\perp }} \sup \biggl\{ \Vert u \Vert _{\lambda }^{2} : u \in N, \int _{\mathbb{R}^{3}} \lambda A^{-}(x) u^{2} \,d x=1 \biggr\} , $$

where$i=1,2,3, \ldots$ . Furthermore, $\xi _{1}(\lambda ) \leq \xi _{2}(\lambda ) \leq \cdots \leq \xi _{i}( \lambda ) \rightarrow +\infty$as$i \rightarrow +\infty$, the corresponding eigenfunctions$\{ g_{i}(\lambda ) \}$can be chosen such that$(g_{i}(\lambda ), g_{j}(\lambda ) )_{\lambda }=\delta _{i j}$, then$\{ g_{i}(\lambda ) \}$is a basis of$E_{\lambda }^{\perp }$.

Proposition 2.2

([23])

Suppose that (A1), (A2) are satisfied and$A^{-} \not \equiv \{0\}$. Then, for$i \in \mathbb{N}$, $\xi _{i}(\lambda ) \rightarrow 0$as$\lambda \rightarrow +\infty$and$\xi _{i}(\lambda )$is non-increasing with respect toλ.

Denote

$$ H_{\lambda }^{-} :=\operatorname{span} \bigl\{ g_{i}( \lambda ) : \xi _{i}( \lambda ) \leq 1 \bigr\} \quad \text{and} \quad H_{\lambda }^{+} := \operatorname{span} \bigl\{ g_{i}( \lambda ) : \xi _{i}(\lambda )>1 \bigr\} . $$

Then $H_{\lambda }=H_{\lambda }^{-} \oplus H_{\lambda }^{+} \oplus E_{ \lambda }$. Moreover, $a_{\lambda }\leq 0$ on $H_{\lambda }^{-}$, and $a_{\lambda }\geq 0$ on $H_{\lambda }^{+} \oplus E_{\lambda }$. It can easily be verified that $a_{\lambda }(u, v)=0$ if u and v come from different subspaces of $H_{\lambda }$, respectively.

By Proposition 2.2, there exists $\tilde{\varLambda }_{0}>0$ such that $\xi _{1}(\lambda ) \leq 1$ for all $\lambda >\tilde{\varLambda }_{0}$, so one has $\operatorname{dim} H_{\lambda }^{-} \geq 1$ when $\lambda >\tilde{\varLambda }_{0}$. Moreover, for every fixed $\lambda >0$, $\operatorname{dim} H_{\lambda }^{-}<+\infty$ since $\xi _{i}(\lambda ) \rightarrow +\infty$ as $i \rightarrow +\infty$.

System (1.1) has a variational formulation. Actually, the corresponding functional $\varphi _{\lambda } : H_{\lambda } \times D^{1,2} (\mathbb{R}^{3} ) \rightarrow \mathbb{R}$ is defined by

$$ \begin{aligned}[b] \varphi _{\lambda }(u, \phi )&= \frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+ \lambda A(x) u^{2} \bigr) \,d x-\frac{1}{2} \int _{\mathbb{R}^{3}} \vert \nabla \phi \vert ^{2} \,d x- \int _{\mathbb{R}^{3}} h(x) u \,d x \\ &\quad{}-\frac{1}{2} \int _{\mathbb{R}^{3}} K(x) (2 \omega +\phi ) \phi u^{2} \,d x- \int _{\mathbb{R}^{3}} F(x, u) \,d x. \end{aligned} $$

The pair $(u, \phi ) \in H_{\lambda } \times D^{1,2} (\mathbb{R}^{3} )$ is a solution of system (1.1) if and only if it is a critical point of $\varphi _{\lambda }$. By borrowing the reduction method used in [25], we can study $\varphi _{\lambda }(u, \phi )$ with only one variable u. The following technical result comes from [12].

Proposition 2.3

([12])

Let$K(x)$satisfy the condition (K). Then, for any$u \in H_{\lambda }$, there exists a unique$\phi =\phi _{u} \in D^{1,2} (\mathbb{R}^{3} )$which satisfies

$$ \Delta \phi =K(x) (\phi +\omega ) u^{2} \quad \textit{in } \mathbb{R}^{3}. $$

Moreover, the map$\varPhi : u \in H_{\lambda } \mapsto \phi _{u} \in D^{1,2} ( \mathbb{R}^{3} )$is continuously differentiable, and

• (i) $-\omega \leq \phi _{u} \leq 0$on the set$\{ x \in \mathbb{R}^{3} | u(x) \neq 0 \}$;

• (ii) $\Vert \phi _{u} \Vert _{D} \leq C_{1}|K|_{3}\|u\|_{\lambda }^{2}$and$\int _{\mathbb{R}^{3}} K(x) \phi _{u} u^{2} \,d x \leq C_{2}|K|_{3}^{2} \|u\|_{\lambda }^{4}$, if$K \in L^{3} (\mathbb{R}^{3} )$;

• (iii) $\Vert \phi _{u} \Vert _{D} \leq C_{3}|K|_{\infty }\|u\|_{ \lambda }^{2}$and$\int _{\mathbb{R}^{3}} K(x) \phi _{u} u^{2} \,d x \leq C_{4}|K|_{ \infty }^{2}\|u\|_{\lambda }^{4}$, if$K \in L^{\infty } (\mathbb{R}^{3} )$.

Remark 2.4

It is pointed out in [12] that the condition (K) can be replaced by

(K)’:

$K(x) \in L^{q_{1}} (\mathbb{R}^{3} ) \cup L^{\infty } (\mathbb{R}^{3} )$, $K(x) \geq 0$, and $K(x) \not \equiv 0$ for a.e. $x \in \mathbb{R}^{3}$, where $q_{1} \geq 3$.

Multiplying both sides of the equation $-\Delta \phi _{u}+K(x) \phi _{u} u^{2}=-\omega K(x) u^{2}$ by $\phi _{u}$ and integrating by parts, we get

$$ \int _{\mathbb{R}^{3}} \bigl( \vert \nabla \phi _{u} \vert ^{2}+K(x) \phi _{u}^{2} u^{2} \bigr) \,d x=- \int _{\mathbb{R}^{3}} \omega K(x) \phi _{u} u^{2} \,d x. $$

(2.3)

From (2.3), we obtain a $C^{1}$ functional $\varPsi _{\lambda } : H_{\lambda } \rightarrow \mathbb{R}$ which is given by

$$ \begin{aligned}[b] \varPsi _{\lambda }(u)&= \varphi _{\lambda } (u, \phi _{u} ) \\ &= \frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+ \lambda A(x) u^{2} \bigr) \,d x-\frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla \phi _{u} \vert ^{2}+K(x) \phi _{u}^{2} u^{2} \bigr) \,d x \\ &\quad{}- \int _{\mathbb{R}^{3}} \omega K(x) \phi _{u} u^{2} \,d x- \int _{ \mathbb{R}^{3}} F(x, u) \,d x- \int _{\mathbb{R}^{3}} h(x) u \,d x \\ &= \frac{1}{2} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla u \vert ^{2}+ \lambda A(x) u^{2} \bigr) \,d x-\frac{1}{2} \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x \\ &\quad{}- \int _{\mathbb{R}^{3}} F(x, u) \,d x- \int _{\mathbb{R}^{3}} h(x) u \,d x. \end{aligned} $$

(2.4)

By $\phi _{u}= (\triangle -K(x) u^{2} )^{-1} [\omega K(x) u^{2} ]$, the Gateaux derivative of $\varPsi _{\lambda }(u)$ is given by

$$ \begin{aligned}[b] \bigl\langle \varPsi _{\lambda }^{\prime }(u), v \bigr\rangle &= \int _{ \mathbb{R}^{3}}\bigl(\nabla u \cdot \nabla v+\lambda A(x) u v\bigr) \,d x- \int _{ \mathbb{R}^{3}} K(x) (2 \omega +\phi _{u} ) \phi _{u} u v \,d x \\ &\quad{}- \int _{\mathbb{R}^{3}} f(x, u) v \,d x- \int _{\mathbb{R}^{3}} h(x) v \,d x \end{aligned} $$

(2.5)

for all $v \in H_{\lambda }$. Set

$$ G(u)= \int _{\mathbb{R}^{3}}-\omega K(x) \phi _{u} u^{2} \,d x. $$

The properties of the functional G is given by [12], the derivative $G^{\prime }$ possesses the Brezis–Lieb-splitting (written for BL-splitting) property, which is similar to the Brezis–Lieb lemma [26].

Next, the compactness conditions for the functional $\varPsi _{\lambda }$ is considered. It is well known that a $C^{1}$ functional I satisfying Palais–Smale ($(\mathit{PS})$ for short) condition at level c if for any sequence $\{ u_{n} \} \subset H$ such that $I (u_{n} ) \rightarrow c$ and $I^{\prime } (u_{n} ) \rightarrow 0$, there exists a convergent subsequence of H, which is called a $(P S)_{c}$ sequence. Since $K \in L^{\infty } (\mathbb{R}^{3} )$ and $K \in L^{3} (\mathbb{R}^{3} )$ are similar, in the following, we just consider the case $K \in L^{\infty } (\mathbb{R}^{3} )$.

Lemma 2.5

Let (A1)–(A2), (F1)–(F4), (K) and (H) hold. Then, for each$c \in \mathbb{R}$, every$(P S)_{c}$sequence is bounded in$H_{\lambda }$.

Proof

Let $\{ u_{n} \} \subset H_{\lambda }$ be a $(P S)_{c}$ sequence of $\varPsi _{\lambda }$. Arguing indirectly, suppose $\|u_{n}\|_{\lambda } \rightarrow \infty$ such that

$$ \varPsi _{\lambda } (u_{n} ) \rightarrow c, \qquad \varPsi _{ \lambda }^{\prime } (u_{n} ) \rightarrow 0, \quad n \rightarrow \infty , $$

(2.6)

after passing to a subsequence. Denote $w_{n} :=u_{n} / \Vert u_{n} \Vert _{\lambda }$. Then $\Vert w_{n} \Vert _{\lambda }=1$, $w_{n} \rightharpoonup w_{0}$ in $H_{\lambda }$ and $w_{n}(x) \rightarrow w_{0}(x)$ for a.e. $x \in \mathbb{R}^{3}$.

If $w_{0}=0$, by the fact $w_{n} \rightarrow 0$ in $L^{2} ( \{ x \in \mathbb{R}^{3} : A(x)<0 \} )$, (2.4), (2.5), (2.6), (F3), (F4) and Proposition 2.3, there are two cases to consider.

Case (1). $2<\mu <4$. From (2.4), (2.5) and (2.6), we derive

$$ \begin{aligned}[b] o(1) & = \frac{1}{ \Vert u_{n} \Vert _{\lambda }^{2}} \bigl(\mu \varPsi _{\lambda } (u_{n} )- \bigl\langle \varPsi _{\lambda }^{ \prime } (u_{n} ), u_{n} \bigr\rangle \bigr) \\ & = \biggl(\frac{\mu }{2}-1 \biggr) \Vert w_{n} \Vert _{\lambda }^{2} - \biggl(\frac{\mu }{2}-1 \biggr) \int _{\mathbb{R}^{3}} \lambda A^{-}(x) w_{n}^{2} \,d x+\frac{1}{ \Vert u_{n} \Vert _{\lambda }^{2}} \int _{ \mathbb{R}^{3}} K(x) \phi _{u_{n}}^{2} u_{n}^{2} \,d x \\ &{\quad {}+ \frac{ (2-\frac{\mu }{2} )}{ \Vert u_{n} \Vert _{\lambda }^{2}} \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u_{n}} u_{n}^{2} \,d x + \frac{1}{ \Vert u_{n} \Vert _{\lambda }^{2}} \int _{\mathbb{R}^{3}} \mathcal{F} (x, u_{n} ) \,d x + \frac{(1-\mu )}{ \Vert u_{n} \Vert _{\lambda }^{2}} \int _{ \mathbb{R}^{3}} h(x) u_{n} \,d x} \\ & \ge \biggl(\frac{\mu }{2}-1 \biggr)- \biggl(\frac{\mu }{2}-1 \biggr) \lambda \bigl\vert A^{-} \bigr\vert _{\infty } \int _{\operatorname{supp} A^{-}} w_{n}^{2} \,d x \\ &{\quad {}- \biggl(2-\frac{\mu }{2} \biggr) \vert K \vert _{\infty } \omega ^{2} { \vert w_{n} \vert _{ \lambda }^{2}}+ (1-\mu ) \vert h \vert _{2} \gamma _{2} \frac{1}{ \Vert u_{n} \Vert _{\lambda }}} \\ & = \biggl(\frac{\mu }{2}-1 \biggr)+o(1), \end{aligned} $$

then $0\ge \frac{\mu }{2}-1$, which contradicts $\mu >2$.

Case (2). $\mu \ge 4$. In this case, by (2.4), (2.5) and (2.6), we have

$$ \begin{aligned}[b] o(1) & = \frac{1}{ \Vert u_{n} \Vert _{\lambda }^{2}} \bigl(\mu \varPsi _{\lambda } (u_{n} )- \bigl\langle \varPsi _{\lambda }^{ \prime } (u_{n} ), u_{n} \bigr\rangle \bigr) \\ & \ge \biggl(\frac{\mu }{2}-1 \biggr)- \biggl(\frac{\mu }{2}-1 \biggr) \lambda \bigl\vert A^{-} \bigr\vert _{\infty } \int _{\operatorname{supp} A^{-}} w_{n}^{2} \,d x+ \frac{ (2-\frac{\mu }{2} )}{ \Vert u_{n} \Vert _{\lambda }^{2}} \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u_{n}} u_{n}^{2} \,d x \\ &{\quad {}+ (1-\mu ) \vert h \vert _{2} d_{2} \frac{1}{ \Vert u_{n} \Vert _{\lambda }}} \\ & \ge \biggl(\frac{\mu }{2}-1 \biggr)- \biggl(\frac{\mu }{2}-1 \biggr) \lambda \bigl\vert A^{-} \bigr\vert _{\infty } \int _{\operatorname{supp} A^{-}} w_{n}^{2} \,d x+ (1-\mu ) \vert h \vert _{2} d_{2} \frac{1}{ \Vert u_{n} \Vert _{\lambda }} \\ & = \biggl(\frac{\mu }{2}-1 \biggr)+o(1), \end{aligned} $$

then $0\ge \frac{\mu }{2}-1$, which contradicts $\mu \ge 4$.

If $w_{0} \neq 0$, then the Lebesgue measure of $\varOmega _{1} := \{ x \in \mathbb{R}^{3} : w_{0}(x) \neq 0 \}$ is positive. For $x \in \varOmega _{1}$, one has $\vert u_{n}(x) \vert \rightarrow \infty$ as $n \rightarrow \infty$, and then, by (F2),

$$ \frac{F (x, u_{n}(x) )}{u_{n}^{2}(x)} w_{n}^{2}(x) \rightarrow +\infty \quad \text{as } n \rightarrow \infty . $$

(2.7)

From (2.7) and Fatou’s lemma, it yields

$$ \int _{\varOmega _{1}} \frac{F (x, u_{n} )}{u_{n}^{2}} w_{n}^{2} \,d x \rightarrow +\infty \quad \text{as } n \rightarrow \infty . $$

(2.8)

From (2.4), (2.6), (2.8), (K), (H) and Proposition 2.3, we have

$$ \begin{aligned}[b] 0 &=\lim_{n \rightarrow +\infty } \frac{\varPsi _{\lambda } (u_{n} )}{ \Vert u_{n} \Vert _{\lambda }^{2}} \\ &=\lim_{n \rightarrow +\infty } \biggl[\frac{1}{2}-\frac{1}{2} \int _{ \mathbb{R}^{3}} \frac{K(x)\omega \phi _{u_{n}}u_{n}^{2}}{ \Vert u_{n} \Vert _{\lambda }^{2}} \,d x- \int _{\mathbb{R}^{3}} \frac{F (x, u_{n} )}{ \Vert u_{n} \Vert _{\lambda }^{2}} \,d x- \int _{\mathbb{R}^{3}} \frac{h(x) u_{n}}{ \Vert u_{n} \Vert _{\lambda }^{2}} \,d x \biggr] \\ &=\frac{1}{2}+o(1)-\lim_{n \rightarrow +\infty } \int _{\mathbb{R}^{3}} \frac{F (x, u_{n} )}{ \Vert u_{n} \Vert _{\lambda }^{2}} \,d x \\ &\le \frac{1}{2}+o(1)-\lim_{n \rightarrow +\infty } \int _{\varOmega _{1}} \frac{F (x, u_{n} )}{u_{n}^{2}}w_{n}^{2} \,d x \\ &=-\infty , \end{aligned} $$

a contradiction. Hence, the boundedness of $\{ u_{n} \}$ in $H_{\lambda }$ is obtained.

The case $K \in L^{3} (\mathbb{R}^{3} )$ can be proved in a similar way as shown above. □

Lemma 2.6

Assume$A \ge 0$and let (A1)–(A2), (F1)–(F4), (K) and (H) hold. Then, for any$M>0$, there exists$\tilde{\varLambda }=\tilde{\varLambda }(M)>0$such that, for all$c< M$and$\lambda >\tilde{\varLambda }$, $\varPsi _{\lambda }$satisfies$(\mathit{PS})_{c}$condition.

Proof

Let $\{ u_{n} \} \subset H_{\lambda }$ be a $(\mathit{PS})_{c}$ sequence with $c< M$. According to Lemma 2.5, $\{ u_{n} \}$ is bounded in $H_{\lambda }$, and there exists $C>0$ such that $\Vert u_{n} \Vert _{\lambda } \leq C$. Hence, passing to a subsequence, we have

$$ \begin{gathered} {u_{n} \rightharpoonup u \quad \text{in } H_{\lambda }}; \\ {u_{n} \rightarrow u \quad \text{in } L_{\mathrm{loc}}^{s} \bigl( \mathbb{R}^{3} \bigr)\mbox{ } \bigl(1 \leq s< 2^{*} \bigr)}; \\ {u_{n}(x) \rightarrow u(x) \quad \text{a.e. } x \in \mathbb{R}^{3}}. \end{gathered} $$

(2.9)

For $\lambda >0$ large enough, we should prove $u_{n} \rightarrow u$ in $H_{\lambda }$. Let $w_{n} :=u_{n}-u$, then $w_{n} \rightharpoonup 0$ in $H_{\lambda }$. By [12, Lemma 2.8], we know

$$\begin{aligned}& \varPsi _{\lambda } (u_{n} )=\varPsi _{\lambda } (w_{n} )+\varPsi _{\lambda }(u)+o(1), \end{aligned}$$

(2.10)

$$\begin{aligned}& \bigl\langle \varPsi _{\lambda }^{\prime } (u_{n} ), v \bigr\rangle = \bigl\langle \varPsi _{\lambda }^{\prime } (w_{n} ), v \bigr\rangle + \bigl\langle \varPsi _{\lambda }^{\prime }(u), v \bigr\rangle + o(1), \quad \text{uniformly for all } v \in H_{ \lambda } \end{aligned}$$

(2.11)

as $n \rightarrow \infty$, in particular, if $\varPsi _{\lambda } (u_{n} ) \rightarrow c\in \mathbb{R}$ and $\varPsi _{\lambda }^{\prime } (u_{n} ) \rightarrow 0$ in $H_{\lambda }^{*}$, then $\varPsi _{\lambda }^{\prime }(u)=0$ and up to passing to a subsequence, we have

$$ \begin{gathered} {\varPsi _{\lambda } (w_{n} ) \rightarrow c-\varPsi _{\lambda }(u)}, \\ { \bigl\langle \varPsi _{\lambda }^{\prime } (w_{n} ), \varphi \bigr\rangle \rightarrow 0, \quad \text{uniformly for all } \varphi \in H_{\lambda }}.\end{gathered} $$

(2.12)

From (2.12), we have $\varPsi _{\lambda }^{\prime }(u)=0$, and

$$ \varPsi _{\lambda } (w_{n} ) \rightarrow c-\varPsi _{\lambda }(u), \qquad \varPsi _{\lambda }^{\prime } (w_{n} ) \rightarrow 0 \quad \text{as } n \rightarrow \infty . $$

(2.13)

From $A\ge 0$ and (F3), we get

$$ \begin{aligned}[b] \varPsi _{\lambda }(u) &=\varPsi _{\lambda }(u)-\frac{1}{\mu } \bigl\langle \varPsi _{\lambda }^{\prime }(u), u \bigr\rangle \\ &= \biggl(\frac{1}{2}-\frac{1}{\mu } \biggr) \Vert u \Vert _{\lambda }^{2}+ \frac{1}{\mu } \int _{\mathbb{R}^{3}} K(x) \phi _{u}^{2} u^{2} \,d x+ \biggl(\frac{2}{\mu }-\frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x \\ &{\quad {}+ \int _{\mathbb{R}^{3}} \mathcal{F}(x, u) \,d x+ \biggl( \frac{1}{\mu }-1 \biggr) \int _{\mathbb{R}^{3}} h u \,d x} \\ &=\varPhi _{\lambda }(u)+ \biggl(\frac{2}{\mu }-\frac{1}{2} \biggr) \int _{ \mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x+ \biggl(\frac{1}{\mu }-1 \biggr) \int _{\mathbb{R}^{3}} h u \,d x, \end{aligned} $$

where

$$ \varPhi _{\lambda }(u)= \biggl(\frac{1}{2}- \frac{1}{\mu } \biggr) \Vert u \Vert _{ \lambda }^{2}+ \frac{1}{\mu } \int _{\mathbb{R}^{3}} K(x) \phi _{u}^{2} u^{2} \,d x+ \int _{\mathbb{R}^{3}} \mathcal{F}(x, u) \,d x \geq 0. $$

(2.14)

From (2.1), we get

$$ \begin{aligned}[b] - \biggl(\frac{1}{\mu }-1 \biggr) \vert h \vert _{2} \vert u \vert _{2} &\leq - \biggl( \frac{1}{\mu }-1 \biggr) \vert h \vert _{2} \gamma _{2} \Vert u \Vert _{\lambda } \\ &\leq - \biggl(\frac{1}{\mu }-1 \biggr) \vert h \vert _{2} \gamma _{2} \liminf_{n \rightarrow \infty } \Vert u_{n} \Vert _{\lambda } \\ & \leq \vert h \vert _{2} \gamma _{2} C \leq \widetilde{M}, \end{aligned} $$

(2.15)

where $\widetilde{M}>0$ is independent of λ. We have two cases to consider.

Case (1). $2<\mu <4$. From (2.9), (2.13), (2.14), (2.15), $c< M$ and Proposition 2.3, we have

$$\begin{aligned} & \biggl(\frac{1}{2}- \frac{1}{\mu } \biggr) \Vert w_{n} \Vert _{ \lambda }^{2}+ \int _{\mathbb{R}^{3}} \mathcal{F} (x, w_{n} ) \,d x \\ &\quad =\varPsi _{\lambda } (w_{n} )-\frac{1}{\mu } \bigl\langle \varPsi _{\lambda }^{\prime } (w_{n} ), w_{n} \bigr\rangle \\ &{\qquad {}- \biggl(\frac{2}{\mu }-\frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{w_{n}} w_{n}^{2} \,d x+ \biggl(1-\frac{1}{\mu } \biggr) \int _{\mathbb{R}^{3}} h w_{n} \,d x+o(1)} \\ &\quad \leq c-\varPsi _{\lambda }(u)+o(1)+ \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega ^{2} w_{n}^{2} \,d x \\ &\quad \leq c-\varPsi _{\lambda }(u)+o(1)+ \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \vert K \vert _{\infty } \omega ^{2} \vert w_{n} \vert _{\lambda }^{2} \\ &\quad \leq c-\varPsi _{\lambda }(u)+o(1) \\ &\quad =c- \biggl[\varPhi _{\lambda }(u)+ \biggl(\frac{1}{\mu }-1 \biggr) \int _{ \mathbb{R}^{3}} h u \,d x+ \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x \biggr]+o(1) \\ &\quad =c-\varPhi _{\lambda }(u)- \biggl(\frac{1}{\mu }-1 \biggr) \int _{ \mathbb{R}^{3}} h u \,d x+o(1)- \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x \\ &\quad \leq c-\varPhi _{\lambda }(u)- \biggl(\frac{1}{\mu }-1 \biggr) \int _{ \mathbb{R}^{3}} h u \,d x+o(1)+ \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \omega ^{2} \vert K \vert _{\infty } \vert u \vert _{2}^{2} \\ &\quad \leq M+\widetilde{M}+C+o(1). \end{aligned}$$

(2.16)

Case (2). $\mu \ge 4$. By (2.9), (2.13), (2.14), (2.15), $c< M$ and Proposition 2.3, we have

$$ \begin{aligned}[b] & \biggl(\frac{1}{2}- \frac{1}{\mu } \biggr) \Vert w_{n} \Vert _{ \lambda }^{2}+ \int _{\mathbb{R}^{3}} \mathcal{F} (x, w_{n} ) \,d x \\ &\quad =\varPsi _{\lambda } (w_{n} )-\frac{1}{\mu } \bigl\langle \varPsi _{\lambda }^{\prime } (w_{n} ), w_{n} \bigr\rangle - \biggl(\frac{2}{\mu }-\frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{w_{n}} w_{n}^{2} \,d x \\ &{\qquad {}+ \biggl(1-\frac{1}{\mu } \biggr) \int _{\mathbb{R}^{3}} h w_{n} \,d x+o(1)} \\ &\quad \leq c-\varPsi _{\lambda }(u)+o(1) \\ &\quad =c- \biggl[\varPsi _{\lambda }(u)+ \biggl(\frac{1}{\mu }-1 \biggr) \int _{ \mathbb{R}^{3}} h u \,d x+ \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x \biggr]+o(1) \\ &\quad =c-\varPsi _{\lambda }(u)- \biggl(\frac{1}{\mu }-1 \biggr) \int _{ \mathbb{R}^{3}} h u \,d x+o(1)- \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x \\ &\quad \leq M+\widetilde{M}+o(1). \end{aligned} $$

(2.17)

Hence,

$$ \int _{\mathbb{R}^{3}} \mathcal{F} (x, w_{n} ) \,d x \leq M+ \widetilde{M}+C+o(1). $$

(2.18)

From (A1) and $w_{n} \rightharpoonup 0$, we get

$$ \vert w_{n} \vert _{2}^{2} \leq \frac{1}{\lambda a} \int _{A(x) \geq a} \lambda A^{+}(x) w_{n}^{2} \,d x+ \int _{A(x)< a} w_{n}^{2} \,d x \leq \frac{1}{\lambda a} \Vert w_{n} \Vert _{\lambda }^{2}+o(1). $$

(2.19)

For $2< s<6$, by the Hölder inequality, Sobolev inequality and (2.19),

$$ \begin{aligned}[b] \vert w_{n} \vert _{s}^{s} &= \int _{\mathbb{R}^{3}} \vert w_{n} \vert ^{s} \,d x \\ & \leq \biggl( \int _{\mathbb{R}^{3}} \vert w_{n} \vert ^{2} \,d x \biggr)^{\frac{6-s}{s}} \biggl( \int _{\mathbb{R}^{3}} \vert w_{n} \vert ^{6} \,d x \biggr)^{\frac{9 s-18}{s}} \\ & \leq \biggl[\frac{1}{\lambda a} \int _{\mathbb{R}^{3}} \bigl( \vert \nabla w_{n} \vert ^{2}+\lambda A^{+} w_{n}^{2} \bigr) \,d x \biggr]^{ \frac{6-s}{s}} \biggl(\overline{S}^{-6} \biggl[ \int _{\mathbb{R}^{3}} \vert \nabla w_{n} \vert ^{2} \,d x \biggr]^{3} \biggr)^{ \frac{9 s-18}{s}} +o(1) \\ & \leq \biggl(\frac{1}{\lambda a} \biggr)^{\frac{6-s}{4}} \overline{S}^{-\frac{3(s-2)}{2}} \Vert w_{n} \Vert _{\lambda }^{s}+o(1). \end{aligned} $$

(2.20)

According to (F1), for any $\varepsilon >0$, there exists $C=C(\varepsilon )>0$ such that $|f(x, t)| \leq \varepsilon |t|$ for all $x \in \mathbb{R}^{3}$ and $|t| \leq C(\varepsilon )$, and (F4) is satisfied for $|t| \geq C(\varepsilon )$ (with the same σ but possibly larger than $a_{1}$). Thus, by (F4), (2.18), (2.20) and the Hölder inequality, we obtain

$$ \int _{ \vert w_{n} \vert \leq C} f (x, w_{n} ) w_{n} \,d x \leq \varepsilon \int _{ \vert w_{n} \vert \leq C} w_{n}^{2} \,d x \leq \frac{\varepsilon }{\lambda a} \Vert w_{n} \Vert _{\lambda }^{2}+o(1) $$

(2.21)

and

$$ \begin{aligned}[b] \int _{ \vert w_{n} \vert \geq C} f (x, w_{n} ) w_{n} \,d x & \leq \biggl( \int _{ \vert w_{n} \vert \geq C} \biggl\vert \frac{f (x, w_{n} )}{w_{n}} \biggr\vert ^{\sigma } \,d x \biggr)^{1 / \sigma } \vert w_{n} \vert _{s}^{2} \\ & \leq \biggl( \int _{ \vert w_{n} \vert \geq C} a_{1} \mathcal{F} (x, w_{n} ) \,d x \biggr)^{1 / \sigma } \vert w_{n} \vert _{s}^{2} \\ & \leq \bigl[a_{1}(M+\widetilde{M}+C) \bigr]^{1 / \sigma } \overline{S}^{-\frac{3(2 s-4)}{2 s}} \biggl(\frac{1}{\lambda a} \biggr)^{ \theta } \Vert w_{n} \Vert _{\lambda }^{2}+o(1), \end{aligned} $$

(2.22)

where $s=2 \sigma /(\sigma -1)$ and $\theta =\frac{6-s}{2s}>0$.

Since $u_{n} \rightharpoonup u$ in $L^{2} (\mathbb{R}^{3} )$ and $h \in L^{2} (\mathbb{R}^{3} )$, we obtain

$$ \int _{\mathbb{R}^{3}} h (u_{n}-u ) \,d x \rightarrow 0 \quad \text{as } n \rightarrow \infty . $$

(2.23)

Therefore, by (2.21), (2.22), (2.23) and Proposition 2.3, we have

$$ \begin{aligned}[b] o(1) &= \bigl\langle \varPsi _{\lambda }^{\prime } (w_{n} ), w_{n} \bigr\rangle \\ &\geq \Vert w_{n} \Vert _{\lambda }^{2}- \int _{\mathbb{R}^{3}} K(x) (2 \omega +\phi _{w_{n}} ) \phi _{w_{n}} w_{n}^{2} \,d x- \int _{\mathbb{R}^{3}} f (x, w_{n} ) w_{n} \,d x - \int _{ \mathbb{R}^{3}} h w_{n} \,d x \\ & \geq \biggl[1-\frac{\varepsilon }{\lambda a}- \bigl[a_{1}(M+ \widetilde{M}+C) \bigr]^{1 / \tau } \overline{S}^{- \frac{3(2 s-4)}{2 s}} \biggl(\frac{1}{\lambda a} \biggr)^{\theta } \biggr] \Vert w_{n} \Vert _{\lambda }^{2}+o(1). \end{aligned} $$

(2.24)

It follows from (2.24) that there exists $\tilde{\varLambda }=\tilde{\varLambda }(M)>0$ such that $w_{n} \rightarrow 0$ in $H_{\lambda }$ when $\lambda >\tilde{\varLambda }$. Since $w_{n}=u_{n}-u$, so $u_{n} \rightarrow u$ in $H_{\lambda }$. □

Lemma 2.7

Assume (A1)–(A2), (F1)–(F4), (K) and (H) hold. Let$\{ u_{n} \}$be a$(P S)_{c}$sequence of$\varPsi _{\lambda }$with level$c>0$. Then, for any$M>0$, up to a subsequence, there exists$\tilde{\varLambda }=\tilde{\varLambda }(M)>0$such that$u_{n} \rightharpoonup u\neq 0$in$H_{\lambda }$satisfying$\varPsi '_{\lambda }(u)=0$and$\varPsi _{\lambda }(u) \leq c$for all$c< M$and$\lambda >\tilde{\varLambda }$.

Proof

By Lemma 2.6 and [12, Lemma 2.8], we obtain

$$ \varPsi _{\lambda }^{\prime }(u)=0, \qquad \varPsi _{\lambda } (w_{n} ) \rightarrow c-\varPsi _{\lambda }(u), \qquad \varPsi _{\lambda }^{ \prime } (u_{n} ) \rightarrow 0 \quad \text{as } n \rightarrow \infty . $$

(2.25)

However, since the appearance of the nonlinear term h in $\varPsi _{\lambda }(u)$ and A is sign-changing, we cannot deduce that $\varPsi _{\lambda }(u) \geq 0$ from

$$ \begin{aligned}[b] \varPsi _{\lambda }(u)&= \varPsi _{\lambda }(u)-\frac{1}{\mu } \bigl\langle \varPsi _{\lambda }^{\prime }(u), u \bigr\rangle \\ &= \biggl(\frac{1}{2}-\frac{1}{\mu } \biggr) \Vert u \Vert _{\lambda }^{2}+ \biggl(\frac{1}{\mu }-\frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} \lambda A^{-}(x) u^{2} \,d x+ \frac{1}{\mu } \int _{\mathbb{R}^{3}} K(x) \phi _{u}^{2} u^{2} \,d x \\ &\quad{}+ \biggl(\frac{2}{\mu }-\frac{1}{2} \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{u} u^{2} \,d x+ \int _{\mathbb{R}^{3}} \mathcal{F}(x, u) \,d x+ \biggl(\frac{1}{\mu }-1 \biggr) \int _{\mathbb{R}^{3}} h u \,d x. \end{aligned} $$

So there are two situations to consider: (i) $\varPsi _{\lambda }(u)<0$; (ii) $\varPsi _{\lambda }(u) \geq 0$.

If $\varPsi _{\lambda }(u)<0$, then $u\neq 0$, thus, the proof is complete. If $\varPsi _{\lambda }(u) \geq 0$, following the proof of Lemma 2.6, we can deduce $u_{n} \rightarrow u$ in $H_{\lambda }$. Indeed, from (A1) and $w_{n} \rightarrow 0$ in $L^{2} ( \{ x \in \mathbb{R}^{3} : A(x)< a \} )$, we obtain

$$ \biggl\vert \int _{\mathbb{R}^{3}} A^{-}(x) w_{n}^{2}(x) \,d x \biggr\vert \leq \bigl\vert A^{-} \bigr\vert _{\infty } \int _{\operatorname{supp} A^{-}} w_{n}^{2} \,d x=o(1), $$

(2.26)

We have two cases to consider again.

Case (1). $2<\mu <4$. In this case, from (2.25) and (2.26), we have

$$ \begin{aligned}[b] &{ \int _{\mathbb{R}^{3}}} {\mathcal{F} (x, w_{n} ) \,d x} \\ &\quad = \varPsi _{\lambda } (w_{n} )-\frac{1}{\mu } \bigl\langle \varPsi _{\lambda }^{\prime } (w_{n} ), w_{n} \bigr\rangle + \biggl(\frac{1}{\mu }-\frac{1}{2} \biggr) \Vert w_{n} \Vert _{ \lambda }^{2} + \biggl( \frac{1}{2}-\frac{1}{\mu } \biggr) \int _{ \mathbb{R}^{3}} \lambda A^{-}(x) w_{n}^{2} \,d x \\ &{\qquad {}-\frac{1}{\mu } \int _{\mathbb{R}^{3}} K(x) \phi _{w_{n}}^{2} w_{n}^{2} \,d x+ \biggl(\frac{1}{2}-\frac{2}{\mu } \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{w_{n}} w_{n}^{2} \,d x+ \biggl(1-\frac{1}{\mu } \biggr) \int _{\mathbb{R}^{3}} h w_{n} \,d x} \\ &\quad \leq c-\varPsi _{\lambda }(u)+o(1)+ \biggl(\frac{2}{\mu }- \frac{1}{2} \biggr) \vert K \vert _{\infty } \omega ^{2} \vert w_{n} \vert _{\lambda }^{2} \\ &\quad \leq c-\varPsi _{\lambda }(u)+o(1) \leq M+o(1). \end{aligned} $$

Case (2). $\mu \geq 4$. In this case, from (2.25) and (2.26), one has

$$ \begin{aligned}[b] &{ \int _{\mathbb{R}^{3}}} {\mathcal{F} (x, w_{n} ) \,d x} \\ &\quad = \varPsi _{\lambda } (w_{n} )-\frac{1}{\mu } \bigl\langle \varPsi _{\lambda }^{\prime } (w_{n} ), w_{n} \bigr\rangle + \biggl(\frac{1}{\mu }-\frac{1}{2} \biggr) \Vert w_{n} \Vert _{ \lambda }^{2} + \biggl( \frac{1}{2}-\frac{1}{\mu } \biggr) \int _{ \mathbb{R}^{3}} \lambda A^{-}(x) w_{n}^{2} \,d x \\ &{\qquad {}-\frac{1}{\mu } \int _{\mathbb{R}^{3}} K(x) \phi _{w_{n}}^{2} w_{n}^{2} \,d x+ \biggl(\frac{1}{2}-\frac{2}{\mu } \biggr) \int _{\mathbb{R}^{3}} K(x) \omega \phi _{w_{n}} w_{n}^{2} \,d x+ \biggl(1-\frac{1}{\mu } \biggr) \int _{\mathbb{R}^{3}} h w_{n} \,d x} \\ &\quad \leq c-\varPsi _{\lambda }(u)+o(1) \leq M+o(1). \end{aligned} $$

Hence, (2.22), (2.23) and (2.24) still remain valid. Therefore, $u_{n} \rightarrow u$ in $H_{\lambda }$, $\varPsi '_{\lambda }(u)=0$ and $\varPsi _{\lambda }(u)=c>0$. □

3 Proof of main results

Lemma 3.1

([27])

Let$E=A_{1} \oplus A_{2}$be a Banach space, $\operatorname{dim} A_{2}<\infty$, $\varPhi \in C^{1} (A, \mathbb{R}^{3} )$. If there exist$\alpha >0$, $R>\rho >0$and$e \in A_{1}$such that

$$ \alpha :=\inf \varPhi (A_{1} \cap L_{\rho } )>\sup \varPhi ( \partial S), $$

where$L_{\rho }=\{u \in E :\|u\|=\rho \}$, $S= \{ u=z+t e : v \in A_{2}, t \geq 0,\|u\| \leq R \}$. ThenΦhas a$(P S)_{c}$sequence with$c \in [\alpha , \sup \varPhi (S)]$.

Let $A_{1}=H_{\lambda }^{+} \oplus E_{\lambda }$ and $A_{2}=H_{\lambda }^{-}$. By Proposition 2.2, $\xi _{i}(\lambda ) \rightarrow 0$ as $\lambda \rightarrow \infty$ for every fixed i, and there exists $\varLambda _{1}>0$ such that $H_{\lambda }^{-} \neq \emptyset$ and $\operatorname{dim} H_{\lambda }^{-}<\infty$ for $\lambda >\varLambda _{1}$. The following lemma comes from [12].

Lemma 3.2

([12])

Assume that (A1)–(A2), (K), (H) and (F1) are satisfied. Then, for each$\lambda >\varLambda _{1}$, there exist$\alpha _{\lambda }$, $\rho _{\lambda }$and$\eta _{\lambda }>0$such that

$$ \varPsi _{\lambda }(u) \geq \alpha _{\lambda }\quad \textit{for all } u \in H_{ \lambda }^{+} \oplus E_{\lambda } \textit{ with } \Vert u \Vert _{\lambda }=\rho _{ \lambda } \textit{ and } \vert h \vert _{2}< \eta _{\lambda }. $$

(3.1)

Furthermore, if$A \geq 0$, we can choose$\alpha , \rho , \eta >0$independent ofλ.

Lemma 3.3

Let (A1), (A2), (F1), (F2), (K) and (H) hold. Then, for any subspace$\tilde{E}_{\lambda } \subset H_{\lambda }$with finite dimension,

$$ \varPsi _{\lambda }(u) \rightarrow -\infty \quad \textit{as } \Vert u \Vert _{ \lambda } \rightarrow \infty , u \in \tilde{E}_{\lambda }. $$

Proof

Suppose by contradiction, it can be assumed that there is a sequence $(u_{n} ) \subset \tilde{E}_{\lambda }$ with $\Vert u_{n} \Vert _{\lambda } \rightarrow \infty$ such that

$$ \inf_{n} \varPsi _{\lambda } (u_{n} )>-\infty . $$

(3.2)

Let $v_{n} :=u_{n} / \Vert u_{n} \Vert _{\lambda }$. Since $\operatorname{dim} \tilde{E}_{\lambda }<+\infty$, after passing to a subsequence, there is $v_{0} \in \tilde{E}_{\lambda } \backslash \{0\}$ such that

$$ v_{n} \rightarrow v_{0}\quad \text{in } \tilde{E}_{\lambda ,}, \qquad v_{n}(x) \rightarrow v_{0}(x) \quad \text{a.e. } x \in \mathbb{R}^{3}. $$

If $v_{0}(x) \neq 0$, then $\vert u_{n}(x) \vert \rightarrow +\infty$ as $n \rightarrow \infty$. Thus, it follows from (F2) that

$$ \frac{F (x, u_{n}(x) )}{u_{n}^{2}(x)} v_{n}^{2}(x) \rightarrow +\infty \quad \text{as } n \rightarrow \infty , $$

which coupled with (F1), (2.4), Proposition 2.3 and Fatou’s lemma yields

$$ \begin{aligned}[b] \frac{\varPsi _{\lambda } (u_{n} )}{ \Vert u_{n} \Vert _{\lambda }^{2}} & \leq \frac{1}{2}- \frac{1}{2 \Vert u_{n} \Vert _{\lambda }^{2}} \int _{\mathbb{R}^{3}}K(x) \omega \phi _{u_{n}} u_{n}^{2} \,d x- \int _{ \mathbb{R}^{3}} \frac{F (x, u_{n} )}{ \Vert u_{n} \Vert _{\lambda }^{2}} \,d x- \int _{\mathbb{R}^{3}} h(x) \frac{u_{n}}{ \Vert u_{n} \Vert _{\lambda }^{2}} \,d x \\ & \leq \frac{1}{2}+\frac{1}{2} \vert K \vert _{\infty } \omega ^{2} \vert v_{n} \vert _{ \lambda }^{2}- \biggl( \int _{v_{0}=0}+ \int _{v_{0} \neq 0} \biggr) \frac{F (x, u_{n} )}{u_{n}^{2}} v_{n}^{2} \,d x+ \frac{ \vert h \vert _{2} \gamma _{2}}{ \Vert u_{n} \Vert _{\lambda }} \\ & \leq \frac{1}{2}+\frac{1}{2} \vert K \vert _{\infty } \omega ^{2} \vert v_{n} \vert _{ \lambda }^{2}- \int _{v_{0} \neq 0} \frac{F (x, u_{n} )}{u_{n}^{2}} v_{n}^{2} \,d x+ \frac{ \vert h \vert _{2} \gamma _{2}}{ \Vert u_{n} \Vert _{\lambda }} \\ & \rightarrow -\infty . \end{aligned} $$

This contradicts (3.2). □

Proof of Theorem 1.2

Firstly, we prove that there is a function $u_{0,\lambda } \in H_{\lambda }$ satisfying $\varPsi _{\lambda }^{\prime } (u_{0,\lambda } )=0$ and $\varPsi _{\lambda } (u_{0,\lambda } )<0$. Since $h \in L^{2} (\mathbb{R}^{3} )$ and $h \geq 0(\not \equiv 0)$, we choose a function $\zeta \in H_{\lambda }$ satisfying

$$ \int _{\mathbb{R}^{3}} h(x) \psi (x) \,d x>0. $$

Therefore, it follows from $-\omega \leq \phi _{u} \leq 0$ that

$$ \begin{aligned}[b] \varPsi _{\lambda }(t \zeta ) &= \frac{t^{2}}{2} \Vert \zeta \Vert _{\lambda }^{2}- \frac{\lambda t^{2}}{2} \int _{\mathbb{R}^{3}} A^{-}(x) \zeta ^{2} \,d x- \frac{1}{2} \int _{\mathbb{R}^{3}} K(x) \omega \phi _{t \zeta }(t \zeta )^{2} \,d x \\ &\quad{}- \int _{\mathbb{R}^{3}} F(x, t \zeta ) \,d x-t \int _{\mathbb{R}^{3}} h(x) \zeta \,d x \\ &\leq \frac{t^{2}}{2} \Vert \zeta \Vert _{\lambda }^{2}+ \frac{t^{2}}{2} \int _{ \mathbb{R}^{3}} \omega ^{2} \zeta ^{2} \,d x-t \int _{\mathbb{R}^{3}} h(x) \zeta \,d x \\ &< 0 \quad \text{for } t>0 \text{ small enough.} \end{aligned} $$

Let $u_{0,\lambda }=t \zeta$ small enough such that $\varPsi _{\lambda } (u_{0,\lambda } )<0$. For $\rho _{\lambda }>0$, which is given by Lemma 3.2, from Lemma 3.3, we get

$$ c_{0, \lambda }=\inf \bigl\{ \varPsi _{\lambda }(u) : u \in \overline{B}_{ \rho _{\lambda }} \bigr\} < 0. $$

From Ekeland’s variational principle, there is a minimizing sequence $\{ u_{n, \lambda } \} \subset \overline{B}_{\rho _{ \lambda }}$ such that

$$ \begin{gathered} {c_{0, \lambda } \leq \varPsi _{\lambda } (u_{n, \lambda } )< c_{0, \lambda }+\frac{1}{n_{\lambda }}}, \\ {\varPsi _{\lambda } (w_{\lambda } ) \geq \varPsi _{\lambda } (u_{n, \lambda } )-\frac{1}{n_{\lambda }} \Vert w_{ \lambda }-u_{n, \lambda } \Vert _{\lambda }},\end{gathered} $$

for all $w_{\lambda } \in \overline{B}_{\rho _{\lambda }}$. Hence, the boundedness of $\{ u_{n, \lambda } \}$ is obtained. By a standard argument, from Lemma 2.6 and [12, Lemma 2.8], there is a function $u_{0,\lambda } \in H_{\lambda }$ such that $\varPsi _{\lambda }^{\prime } (u_{0,\lambda } )=0$ and $\varPsi _{\lambda } (u_{0,\lambda } )=c_{0, \lambda }<0$.

Next, we prove that there is a function $\bar{u}_{\lambda } \in H_{\lambda }$ satisfying $\varPsi _{\lambda }^{\prime } (\bar{u}_{\lambda } )=0$ and $\varPsi _{\lambda } (\bar{u}_{\lambda } )>0$. By Lemma 3.1, and [12, Lemma 3.2, Lemma 3.4], for each $k \in \mathbb{N}$, $\lambda =\lambda _{k}$ and $|h|_{2}<\eta _{k}$, $\varPsi _{\lambda _{k}}$ has a $(P S)_{c}$ sequence with $c \in [\alpha _{\lambda _{k}}, \sup \varPsi _{\lambda _{k}} (S_{k} ) ]$. Let $M :=\sup \varPsi _{\lambda _{k}} (S_{k} )$, then, by Lemmas 2.6, 2.7 and 3.1, $\varPsi _{\lambda _{k}}$ has a nontrivial critical point $\bar{u}_{\lambda } \in H_{\lambda }$ such that $\varPsi _{\lambda }^{\prime } (\bar{u}_{\lambda } )=0$ and $\varPsi _{\lambda } (\bar{u}_{\lambda } )=c \geq \alpha _{ \lambda _{k}}>0$. □

Proof of Theorem 1.3

The first solution can be proved in the same way as shown in Theorem 1.2. The second solution follows from [12, Lemma 3.2, Lemma 3.5], Lemma 2.6, Lemma 2.7 and Lemma 3.1. □

Proof of Theorem 1.4

If $A\geq 0$, from the proof of Theorem 1.2, we can check that $\rho _{\lambda }$, $c_{0, \lambda }$, $u_{0, \lambda }$ are independent of λ. So we choose $c_{0}=c_{0,\lambda }$, $B_{\rho }=B_{\rho ,\lambda }$, α, ρ, and η are independent of λ, then, by the mountain pass theorem [27, 28], the proof of Theorem 1.4 is complete. □

4 Conclusion

In this paper, we first obtained a Palais–Smale sequence by using super-quadratic condition. Then we establish the parameter which depends on compactness conditions to recover the compactness. Finally, the existence of nontrivial solutions is proved by the linking theorem and Ekeland’s variational principle. Obviously, the super-quadratic condition has been successfully applied to find the solutions of the nonhomogeneous Klein–Gordon–Maxwell system with sign-changing potential, we hope that these results can be widely used in fractional systems as discussed in [29] and [30].