Coexistence states for a modified Leslie-Gower type predator-prey model with diffusion

Abstract

This paper is concerned with a modified Leslie-Gower predator-prey model with general functional response under homogeneous Robin boundary conditions. We establish the existence of coexistence states by the fixed index theory on positive cones. As an example, we apply the obtained results to this model with Holling-type II functional response. Our results show that the intrinsic growth rates and the principle eigenvalues of the corresponding elliptic problems with respect to the Robin boundary conditions play more important roles than other parameters for the existence of positive solutions.

MSC: 35J55, 37B25, 92D25.

1 Introduction

Population ecology is dominated by a focus on interspecific interaction such as competition, cooperation and predation in recent research papers. In particular, predator-prey systems are very important to describe the interactions in the multi-species population dynamics. Because of the differences in capturing food and consuming energy, a major trend in theoretical work on predator-prey dynamics has been launched so as to derive more realistic models and functional responses and understand the interactions among the predators and the preys such as Lotka-Volterra type [1, 2], Holling type [3], Beddington-DeAngelis type [4, 5] and so on. In order to model the predator-prey mite outbreak interactions on fruit trees, Wollkind et al. [6] adapted the following ordinary differential equations based on the model due to May [7]:

$$\{\begin{array}{c}\frac{du}{dt}=ru(1-\frac{u}{K})-p(u)v,\hfill \\ \frac{dv}{dt}=v[s(1-\frac{hv}{u})],\hfill \end{array}$$

(1.1)

where u and v represent the densities of the prey and predator respectively. In system (1.1), it is assumed that the prey grows logistically with carrying capacity K and intrinsic growth rate r in the absence of predators. The predator consumes the prey according to the functional response $p(u)$ and grows logistically with intrinsic growth rate s and carrying capacity proportional to the population size of prey. The parameter h is the numbers of prey required to support one predator at equilibrium when v equals $u/h$. In recent years, the Leslie-Gower type predator-prey model (1.1) has been widely studied by many authors; see [8–11]. If $p(u)$ is of Holling-type II functional response in (1.1), then it is the so-called Holling-Tanner predator-prey system as follows:

$$\{\begin{array}{c}\frac{du}{dt}=ru(1-\frac{u}{K})-\frac{uv}{r_1+u},\hfill \\ \frac{dv}{dt}=v[s(1-\frac{hv}{u})].\hfill \end{array}$$

(1.2)

As pointed out in [12, 13], in the case of severe scarcity, the predator v can switch to other populations, but its growth will be limited by the fact that its most favorite food, the prey u, is not available in abundance. To model the phenomena in population dynamics, a positive constant is added to the denominator of the predator equation. Based on such a reason, (1.2) becomes the following system with modified Leslie-Gower functional response [12]:

$$\{\begin{array}{c}\frac{du}{dt}=u(a_1-b_{1}u)-\frac{cuv}{r_1+u},\hfill \\ \frac{dv}{dt}=v(a_2-\frac{ev}{r_2+u}),\hfill \end{array}$$

(1.3)

where $a_1$, $a_2$, $b_1$, $r_1$, $r_2$, c, e are positive constants in a biological viewpoint. More precisely, in [12], Aziz-Alaoui et al. investigated the boundedness of solutions, the existence of positive invariance attracting set and global stability of the coexisting interior equilibrium. Later, Nindjin et al. [13] gave the qualitative analysis of the corresponding delayed system.

In the evolutionary process of the species, the individuals do not remain fixed in space, and their spatial distribution changes continuously due to the impact of many reasons (the environment factors, food supplies, etc.). Therefore, spatial effects such as diffusion and dispersal should be introduced into population models. In particular, introducing the spatial effects is not trivial in many works. For example, the famous Turing instability was observed in many nature processes, and it was also proved in some mathematical models with diffusion. Clearly, such a Turing instability cannot be formulated by the ordinary differential equations.

Particularly, the spatial diffusion in predator-prey models was also considered by many authors. For example, Chen and Wang [14] studied system (1.3) with diffusion under homogeneous Neumann boundary conditions, while Peng and Wang [15] focused on system (1.3) with diffusion under homogeneous Dirichlet boundary conditions. Ryu and Ahn [16] and Ko and Ryu [17], respectively, investigated diffusive Gause-type predator-prey systems with ratio-dependent Holling-type II functional response and nonmonotonic functional response under Robin boundary conditions. The authors of these works mentioned above mainly discussed the existence and nonexistence of positive solutions of the stationary problem. For more works on diffusive predator-prey systems, one can see [18–26] and the references cited therein. Motivated by the previous works, in this paper, we introduce diffusion into system (1.3) and consider the following partial differential equations equipped with homogeneous Robin boundary conditions:

$$\{\begin{array}{c}{u}_t-d_1\mathrm{\Delta }u=ug(u)-p(u)v,\hfill \\ v_t-d_2\mathrm{\Delta }v=v(a_2-\frac{ev}{r_2+u}),(x,t)\in D_T=\mathrm{\Omega }\times (0,T],\hfill \\ {\kappa }_1\frac{\partial u}{\partial \nu }+u=0,{\kappa }_2\frac{\partial v}{\partial \nu }+v=0,(x,t)\in S_T=\partial \mathrm{\Omega }\times (0,T],T\in (0,\mathrm{\infty }),\hfill \\ u(x,0)=u_0(x)\ge 0,v(x,0)=v_0(x)\ge 0,x\in \overline{\mathrm{\Omega }}.\hfill \end{array}$$

(1.4)

In the above, u and v, respectively, stand for the population densities of prey and predator; $\mathrm{\Omega }\subset {\mathbb{R}}^N$ is a bounded domain with smooth boundary ∂ Ω; ν denotes the outward unit normal vector of the boundary ∂ Ω; ${\kappa }_1$, ${\kappa }_2$ are nonnegative constants. $g(u)$ is the birth function of the prey u. It should be pointed out that system (1.4) has the function response with a general formation $p(u)$, which is different from the models in the related papers.

For evolutionary systems, steady state solutions play an important role in understanding the long-time behavior of the corresponding Cauchy-type problem. For example, from the viewpoint of monotone dynamical systems, the steady state solutions of an evolutionary system can often determine the eventual state of the system. Some problems concerned with the steady states of evolutionary systems such as the traveling wave solutions and the positive solutions of elliptic equations have been widely studied. To understand the dynamics of system (1.4), we first consider its stationary problem in this paper. More precisely, we shall establish the existence of positive solutions to the following elliptic system:

$$\{\begin{array}{cc}-\mathrm{\Delta }u=ug(u)-p(u)v,\hfill & x\in \mathrm{\Omega },\hfill \\ -\mathrm{\Delta }v=v(a_2-\frac{ev}{r_2+u}),\hfill & x\in \mathrm{\Omega },\hfill \\ {\kappa }_1\frac{\partial u}{\partial \nu }+u=0,{\kappa }_2\frac{\partial v}{\partial \nu }+v=0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

(1.5)

The rest of this paper is arranged as follows. In Section 2, we collect some known results including the eigenvalue problem and the fixed point index on positive cones. In Section 3, we establish the existence of positive solutions for system (1.5). In Section 4, as an example, we apply the obtained results to system (1.5) with Holling-type II functional response.

2 Preliminaries

In this section, we give some preliminaries, which will serve as the basic tools in the sequel. First, we introduce the fixed point index of compact maps on positive cones; see [27–29].

Let E be a real Banach space and $W\subset E$ be a closed convex set. Then W is called a wedge if $\beta W\subset W$ for all $\beta \ge 0$, and a wedge W is said to be a cone if $W\cap (-W)=\{0\}$. For $y\in W$, define $W_y=\{x\in E:y+\gamma x\in W\text{ for some }\gamma >0\}$ and $S_y=\{x\in {\overline{W}}_y:-x\in {\overline{W}}_y\}$. It is evident that ${\overline{W}}_y$ is a wedge containing W, y, −y, while $S_y$ is a closed subspace of E containing y. In what follows, we always assume that $E=\overline{W-W}$. Let $\mathcal{T}:{\overline{W}}_y\to {\overline{W}}_y$ be a compact linear operator on E. We say that $\mathcal{T}$ has property α on ${\overline{W}}_y$ if there exist $t\in (0,1)$ and $w\in {\overline{W}}_y\mathrm{\setminus }{S}_y$ such that $w-t\mathcal{T}w\in S_y$. Suppose that $\mathcal{F}:W\to W$ is a compact operator with a fixed point $y\in W$. If ℱ is Fréchet differential at y, then the derivative ${\mathcal{F}}^{\prime }(y)$ has the property that ${\mathcal{F}}^{\prime }(y):{\overline{W}}_y\to {\overline{W}}_y$. For an open subset $U\subset W$, define ${index}_W(\mathcal{F},U)=index(\mathcal{F},U,W)={deg}_W(I-\mathcal{F},U,0)$, where I is the identity map. If y is an isolated fixed point of ℱ, then the fixed point index of ℱ at y related to W is defined by ${index}_W(\mathcal{F},y)=index(\mathcal{F},y,W)={index}_W(\mathcal{F},U(y),W)$, herein $U(y)$ is a small open neighborhood of y in W. The following results of fixed point index can be obtained from [16, 27–29].

Lemma 2.1 Assume that $I-{\mathcal{F}}^{\prime }(y)$ is invertible on ${\overline{W}}_y$.

(i) If ${\mathcal{F}}^{\prime }(y)$ has property α, then ${index}_W(\mathcal{F},y)=0$.

(ii) If ${\mathcal{F}}^{\prime }(y)$ does not have property α, then ${index}_W(\mathcal{F},y)={(-1)}^{\sigma }$, where σ is the sum of multiplicities of all eigenvalues of ${\mathcal{F}}^{\prime }(y)$ which are greater than one.

Now, we introduce some known results about the eigenvalue problem equipped with Robin boundary conditions. For $q(x)\in C^{\alpha }(\overline{\mathrm{\Omega }})$ and $\kappa \ge 0$, let ${\lambda }_{1,\kappa }(q(x))$ be the principle eigenvalue of the following problem:

$$\{\begin{array}{cc}-\mathrm{\Delta }u+q(x)u=\lambda u,\hfill & x\in \mathrm{\Omega },\hfill \\ \kappa \frac{\partial u}{\partial \nu }+u=0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

(2.1)

In particular, we denote ${\lambda }_{1,\kappa }(0)$ by ${\lambda }_{1,\kappa }$ for the sake of convenience. It is well known that ${\lambda }_{1,\kappa }(q(x))$ is strictly increasing with respect to $q(x)$, namely, ${\lambda }_{1,\kappa }(q_1(x))<{\lambda }_{1,\kappa }(q_2(x))$ if $q_1(x)\le q_2(x)$ and $q_1(x)\not\equiv q_2(x)$. Furthermore, the eigenfunction ${\varphi }_1$ of (2.1) corresponding to the eigenvalue ${\lambda }_{1,\kappa }(q(x))$ is unique and positive. In [30, 31], the authors discussed the eigenvalue problem (2.1) in detail and established the existence and comparison results for (2.1). Furthermore, we cite the following lemma on the eigenvalue of (2.1), which can be found in [16, 32].

Lemma 2.2 Let $q(x)\in C^{\alpha }(\overline{\mathrm{\Omega }})$ and $u\ge 0$, $u\not\equiv 0$ in Ω.

(a1) If $0\not\equiv -\mathrm{\Delta }u+q(x)u\le 0$, then ${\lambda }_{1,\kappa }(q(x))<0$.

(b1) If $0\not\equiv -\mathrm{\Delta }u+q(x)u\ge 0$, then ${\lambda }_{1,\kappa }(q(x))>0$.

(c1) If $-\mathrm{\Delta }u+q(x)u\equiv 0$, then ${\lambda }_{1,\kappa }(q(x))=0$.

In addition, if M is a positive constant such that $-q(x)+M>0$ on $\overline{\mathrm{\Omega }}$, then we have the following conclusions:

(a2) ${\lambda }_{1,\kappa }(q(x))<0\Rightarrow r[{(-\mathrm{\Delta }+M)}^{-1}(-q(x)+M)]>1$.

(b2) ${\lambda }_{1,\kappa }(q(x))>0\Rightarrow r[{(-\mathrm{\Delta }+M)}^{-1}(-q(x)+M)]<1$.

(c2) ${\lambda }_{1,\kappa }(q(x))=0\Rightarrow r[{(-\mathrm{\Delta }+M)}^{-1}(-q(x)+M)]=1$, where $r(\cdot )$ is the spectral radius of an operator.

Consider the following scalar equation:

$$\{\begin{array}{cc}-\mathrm{\Delta }u=uf(x,u),\hfill & x\in \mathrm{\Omega },\hfill \\ \kappa \frac{\partial u}{\partial \nu }+u=0,\hfill & x\in \partial \mathrm{\Omega },\hfill \end{array}$$

(2.2)

where $f(x,u):\overline{\mathrm{\Omega }}\times [0,\mathrm{\infty })\to \mathbb{R}$ is $C^{\alpha }$ in x for $0<\alpha <1$ and $C^1$ in u. The following lemma can be obtained from[16, 30].

Lemma 2.3 Assume that $f_u(x,u)<0$ for all $(x,u)\in \overline{\mathrm{\Omega }}\times [0,\mathrm{\infty })$ and $f(x,u)\le 0$ on $(x,u)\in \overline{\mathrm{\Omega }}\times [C,\mathrm{\infty })$ for some positive constant C.

(a) If ${\lambda }_{1,\kappa }(-f(x,0))\ge 0$, then (2.2) has no positive solutions. Moreover, the trivial solution is globally asymptotically stable.

(b) If ${\lambda }_{1,\kappa }(-f(x,0))<0$, then (2.2) has a unique positive solution $u(x)$ which is globally asymptotically stable and satisfies $u(x)\le C$ for all $x\in \overline{\mathrm{\Omega }}$.

3 Existence of positive solutions for system (1.5)

In order to establish the existence of positive solutions of system (1.5), we give the following hypotheses.

(H1) $g\in C^1([0,\mathrm{\infty }))$, $g(0)>0$, $g(K)=0$, $-\beta \le g_u(u)<0$, for any $u\ge 0$, where the constants $K>0$ and $\beta >0$.

(H2) $p\in C^2([0,\mathrm{\infty }))$, $p(0)=0$ and $0<p_u(u)\le \gamma$, for any $u\ge 0$, where the constant $\gamma >0$.

Firstly, we give a priori estimates of positive solutions of system (1.5). For the purpose, consider the following equation:

$$\{\begin{array}{cc}-\mathrm{\Delta }u=ug(u),\hfill & x\in \mathrm{\Omega },\hfill \\ \kappa \frac{\partial u}{\partial \nu }+u=0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

(3.1)

By Lemma 2.3, when $g(0)>{\lambda }_{1,\kappa }$, (3.1) has a unique positive solution $u_0$. When $a_2>{\lambda }_{1,{\kappa }_2}$,

$$\{\begin{array}{cc}-\mathrm{\Delta }v=v(a_2-\frac{ev}{r_2}),\hfill & x\in \mathrm{\Omega },\hfill \\ \kappa \frac{\partial v}{\partial \nu }+v=0,\hfill & x\in \partial \mathrm{\Omega }\hfill \end{array}$$

(3.2)

has a unique positive solution $v_0$. Hence, when $g(0)>{\lambda }_{1,{\kappa }_1}$ and $a_2>{\lambda }_{1,{\kappa }_2}$, system (1.5) has two semi-trivial solutions $(u_0,0)$ and $(0,v_0)$. By virtue of the maximum principle and Hopf’s lemma, we obtain the following results on the boundedness of the nonnegative solutions of (1.5), of which the proof is omitted here.

Proposition 3.1 Any nonnegative solution $(u,v)$ of (1.5) satisfies

$$u(x)\le K,v(x)\le R_0:=\frac{a_2(r_2+K)}{e}.$$

For the calculation of the fixed point index, we introduce the following notations:

$$\begin{array}{c}E=C_{{\kappa }_1}(\overline{\mathrm{\Omega }})\times C_{{\kappa }_2}(\overline{\mathrm{\Omega }});C_{{\kappa }_i}(\overline{\mathrm{\Omega }})=\{w\in C(\overline{\mathrm{\Omega }}):{\kappa }_i\frac{\partial w}{\partial \nu }+w=0,x\in \partial \mathrm{\Omega }\};\hfill \\ W=K_1\times K_2;K_i=\{w\in C_{{\kappa }_i}(\overline{\mathrm{\Omega }}):0\le w(x),x\in \overline{\mathrm{\Omega }}\};\hfill \\ D=\{(u,v)\in E:u\le K+1,v\le R_0+1\};D^{\mathrm{\prime }}=(intD)\cap W.\hfill \end{array}$$

It is easy to verify that

$$\begin{array}{c}{\overline{W}}_{(0,0)}=K_1\times K_2;S_{(0,0)}=\{(0,0)\};\hfill \\ {\overline{W}}_{(u_0,0)}=C_{{\kappa }_1}(\overline{\mathrm{\Omega }})\times K_2;S_{(u_0,0)}=C_{{\kappa }_1}(\overline{\mathrm{\Omega }})\times \{0\};\hfill \\ {\overline{W}}_{(0,v_0)}=K_1\times C_{{\kappa }_2}(\overline{\mathrm{\Omega }});S_{(0,v_0)}=\{0\}\times C_{{\kappa }_2}(\overline{\mathrm{\Omega }}).\hfill \end{array}$$

From Proposition 3.1, we can see that the nonnegative solution of (1.5) must lie in $D^{\mathrm{\prime }}$.

Choosing

$$M>max\{g(0)+\beta K+\gamma R_0,a_2+\frac{2e{R}_0}{r_2}\},$$

then

$$u(a_1-b_{1}u-\frac{cv}{r_1+u})+Mu,v(a_2-\frac{ev}{r_2+u})+Mv$$

are nonnegative for all $(u,v)\in [0,K]\times [0,R_0]$.

Define an operator $\mathcal{F}:E\to E$ by

$$\mathcal{F}(u,v)={(-\mathrm{\Delta }+M)}^{-1}{}^{\left(\begin{array}{c}ug(u)-p(u)v+Mu\\ v(a_2-\frac{ev}{r_2+u})+Mv\end{array}\right)T}.$$

By the strong maximum principle, ${(-\mathrm{\Delta }+M)}^{-1}$ is a compact linear operator, and ℱ is a direct sum of compact positive operators. Clearly, system (1.5) is equivalent to $\mathcal{F}(u,v)=(u,v)$ (it should be noted that this is independent of the choice of M as long as M is large enough). Thus, finding a positive solution of system (1.5) is equivalent to proving that ℱ has a nontrivial fixed point in $D^{\mathrm{\prime }}$. Without loss of generality, we may assume that $(0,0)$, $(u_0,0)$, and $(0,v_0)$ are isolated fixed points of ℱ if they exist, and so the corresponding indices related to W are well defined. For $t\in [0,1]$, define a homotopy

$${\mathcal{F}}_t(u,v)={(-\mathrm{\Delta }+M)}^{-1}{}^{\left(\begin{array}{c}t(ug(u)-p(u)v)+Mu\\ tv(a_2-\frac{ev}{r_2+u})+Mv\end{array}\right)T},$$

then $\mathcal{F}={\mathcal{F}}_1$.

Lemma 3.2 For any open set $D^{\mathrm{\prime }}$ in W, ${index}_W(\mathcal{F},D^{\mathrm{\prime }})=1$.

Proof Firstly, we can see that ${index}_W(\mathcal{F},D^{\mathrm{\prime }})$ is well defined since ℱ has no fixed point on $\partial D^{\mathrm{\prime }}$. For $t\in [0,1]$, a fixed point of ${\mathcal{F}}_t$ is a solution of the following problem:

$$\{\begin{array}{cc}-\mathrm{\Delta }u=t(ug(u)-p(u)v),\hfill & x\in \mathrm{\Omega },\hfill \\ -\mathrm{\Delta }v=tv(a_2-\frac{ev}{r_2+u}),\hfill & x\in \mathrm{\Omega },\hfill \\ {\kappa }_1\frac{\partial u}{\partial \nu }+u=0,{\kappa }_2\frac{\partial v}{\partial \nu }+v=0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

(3.3)

In view of Proposition 3.1, the fixed point of ${\mathcal{F}}_t$ satisfies $u(x)\le K$ and $v(x)\le R_0$ on $\overline{\mathrm{\Omega }}$ for all $t\in [0,1]$, and so all the fixed points of ${\mathcal{F}}_t$ must lie in $D^{\mathrm{\prime }}$, and ${index}_W({\mathcal{F}}_t,D^{\mathrm{\prime }})$ is independent of t. Hence, by the homotopy invariance,

$${index}_W(\mathcal{F},D^{\mathrm{\prime }})={index}_W({\mathcal{F}}_1,D^{\mathrm{\prime }})={index}_W({\mathcal{F}}_0,D^{\mathrm{\prime }}).$$

Since problem (3.3) with $t=0$ has only the trivial solution $(0,0)$, we have

$${index}_W({\mathcal{F}}_0,D^{\mathrm{\prime }})={index}_W({\mathcal{F}}_0,(0,0)).$$

Denote

$${\mathcal{L}}_0:={\mathcal{F}}_0^{\prime }(0,0)={(-\mathrm{\Delta }+M)}^{-1}\left(\begin{array}{cc}M& 0\\ 0& M\end{array}\right).$$

Thus, it follows from Lemma 2.2 that $r({\mathcal{L}}_0)<1$, which indicates that $I-{\mathcal{L}}_0$ is invertible on ${\overline{W}}_{(0,0)}$ and ${\mathcal{L}}_0$ does not have property α on ${\overline{W}}_{(0,0)}$. So, we may conclude that ${index}_W({\mathcal{F}}_0,(0,0))=1$ by Lemma 2.1. The proof is completed. □

Lemma 3.3 Assume that $g(0)>{\lambda }_{1,{\kappa }_1}$ and $a_2\ne {\lambda }_{1,{\kappa }_2}$. Then ${index}_W(\mathcal{F},(0,0))=0$.

Proof Note that $\mathcal{F}(0,0)=(0,0)$ and ℱ is compact. By calculating, we get

$${\mathcal{L}}_1:={\mathcal{F}}^{\mathrm{\prime }}(0,0)={(-\mathrm{\Delta }+M)}^{-1}\left(\begin{array}{cc}g(0)+M& 0\\ 0& a_2+M\end{array}\right).$$

Suppose that ${\mathcal{L}}_1(\xi ,\eta )=(\xi ,\eta )\in {\overline{W}}_{(0,0)}$. Then

$$\{\begin{array}{cc}-\mathrm{\Delta }\xi =g(0)\xi ,\hfill & x\in \mathrm{\Omega },\hfill \\ {\kappa }_1\frac{\partial \xi }{\partial \nu }+\xi =0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

If $\xi >0$, then $g(0)={\lambda }_{1,{\kappa }_1}$ by the lemma, which is a contradiction to the assumption. Thus, $\xi \equiv 0$. Similarly, since $a_2\ne {\lambda }_{1,{\kappa }_2}$, then $\eta \equiv 0$. Therefore, $I-{\mathcal{L}}_1$ is invertible on ${\overline{W}}_{(0,0)}$.

Since $g(0)>{\lambda }_{1,{\kappa }_1}$, we have $r_1:=r[{(-\mathrm{\Delta }+M)}^{-1}(g(0)+M)]>1$ by Lemma 2.2. From the Krein-Rutman theorem, $r_1$ is the principle eigenvalue of the operator ${(-\mathrm{\Delta }+M)}^{-1}(g(0)+M)$ with a corresponding eigenfunction $\varphi \in K_1\mathrm{\setminus }\{0\}$. Set $t_0=1/r_1$. Then we have $0<t_0<1$ and

$$(I-t_0{\mathcal{L}}_1)(\varphi ,0)=(0,0)\in S_{(0,0)}.$$

This implies that ${\mathcal{L}}_1$ has property α. It follows from Lemma 2.1 that ${index}_W(\mathcal{F},(0,0))=0$. The proof is completed. □

Lemma 3.4 Assume that $g(0)>{\lambda }_{1,{\kappa }_1}$ and $a_2>{\lambda }_{1,{\kappa }_2}$. Then

$${index}_W(\mathcal{F},(u_0,0))=0.$$

Proof By a direct computation, we have

$${\mathcal{L}}_2:={\mathcal{F}}^{\mathrm{\prime }}(u_0,0)={(-\mathrm{\Delta }+M)}^{-1}\left(\begin{array}{cc}g(u_0)+u_0{g}^{\prime }(u_0)+M& -p(u_0)\\ 0& a_2+M\end{array}\right).$$

Suppose that ${\mathcal{L}}_2(\xi ,\eta )=(\xi ,\eta )$ for some $(\xi ,\eta )\in {\overline{W}}_{(u_0,0)}$. Then

$$\{\begin{array}{cc}-\mathrm{\Delta }\xi -(g(u_0)+u_0{g}^{\prime }(u_0))\xi =-p(u_0)\eta ,\hfill & x\in \mathrm{\Omega },\hfill \\ -\mathrm{\Delta }\eta =a_2\eta ,\hfill & x\in \mathrm{\Omega },\hfill \\ {\kappa }_1\frac{\partial \xi }{\partial \nu }+\xi =0,{\kappa }_2\frac{\partial \eta }{\partial \nu }+\eta =0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

(3.4)

For $\eta \in K_2$, in the second equation of (3.4), $a_2={\lambda }_{1,{\kappa }_2}$, if $\eta \not\equiv 0$, by Lemma 2.2. Since $a_2>{\lambda }_{1,{\kappa }_2}$, we have $\eta \equiv 0$. If $\xi \not\equiv 0$, then 0 is an eigenvalue of the following problem:

$$\{\begin{array}{cc}-\mathrm{\Delta }\xi -(g(u_0)+u_0{g}^{\prime }(u_0))\xi =\lambda \xi ,\hfill & x\in \mathrm{\Omega },\hfill \\ {\kappa }_1\frac{\partial \xi }{\partial \nu }+\xi =0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

Thus, ${\lambda }_{1,{\kappa }_1}(-g(u_0)-u_0{g}^{\prime }(u_0))<0$, ${\lambda }_{1,{\kappa }_1}(-g(u_0))=0$. Since $(u_0,0)$ is the semi-trivial solution of system (1.5), using the comparison property of the eigenvalue, we have

$${\lambda }_{1,{\kappa }_1}(-g(u_0)-u_0{g}^{\prime }(u_0))>{\lambda }_{1,{\kappa }_1}(-g(u_0))=0.$$

This yields a contradiction. Hence, $(\xi ,\eta )=(0,0)$. This indicates that $I-{\mathcal{L}}_2$ is invertible on ${\overline{W}}_{(u_0,0)}$.

Now, we shall prove that ${\mathcal{L}}_2$ has property α on ${\overline{W}}_{(u_0,0)}$. In fact, since $a_2>{\lambda }_{1,{\kappa }_2}$, then $r_2:=r({(-\mathrm{\Delta }+M)}^{-1}(a_2+M))>1$ by Lemma 2.2. From the Krein-Rutman theorem, $r_2$ is the principle eigenvalue of the operator ${(-\mathrm{\Delta }+M)}^{-1}(a_2+M)$ with a corresponding eigenfunction $\varphi \in K_2\mathrm{\setminus }\{0\}$. Set $t_0=1/r_2$. Then $0<t_0<1$. For $(0,\varphi )\in {\overline{W}}_{(u_0,0)}\mathrm{\setminus }{S}_{(u_0,0)}$, it is easy to verify that

$$(I-t_1{\mathcal{L}}_2)(0,\varphi )\in S_{(u_0,0)}.$$

This implies that ${\mathcal{L}}_2$ has property α. By Lemma 2.1, we know ${index}_W(\mathcal{F},(u_0,0))=0$. The proof is completed. □

Similarly, we have the following lemma, the proof of which is a slight modification of the above.

Lemma 3.5 Assume that $a_2>{\lambda }_{1,{\kappa }_2}$ and $g(0)>{\lambda }_{1,{\kappa }_1}(p^{\prime }(0)v_0)$. Then

$${index}_W(\mathcal{F},(0,v_0))=0.$$

Now, we establish the existence of positive solutions of the system based on the above results about the fixed index.

Theorem 3.6 Assume that $a_2>{\lambda }_{1,{\kappa }_2}$, $g(0)>{\lambda }_{1,{\kappa }_1}(p^{\prime }(0)v_0)$. Then system (1.5) has a positive solution.

Proof By Lemmas 3.2-3.5, we have

$${index}_W(\mathcal{F},(0,0))+{index}_W(\mathcal{F},(u_0,0))+{index}_W(\mathcal{F},(0,v_0))=0$$

and ${index}_W(\mathcal{F},D^{\mathrm{\prime }})=1$. Therefore, system (1.5) has a positive solution in $D^{\mathrm{\prime }}$. The proof is completed. □

4 Applications

In this section, as an example, we apply the above results to system (1.5) with Holling II type functional response and establish the existence of positive solutions for the following predator-prey system:

$$\{\begin{array}{cc}-\mathrm{\Delta }u=u(a_1-b_{1}u)-\frac{cuv}{r_1+u},\hfill & x\in \mathrm{\Omega },\hfill \\ -\mathrm{\Delta }v=v(a_2-\frac{ev}{r_2+u}),\hfill & x\in \mathrm{\Omega },\hfill \\ {\kappa }_1\frac{\partial u}{\partial \nu }+u=0,{\kappa }_2\frac{\partial v}{\partial \nu }+v=0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

(4.1)

Consider the following equation:

$$\{\begin{array}{cc}-\mathrm{\Delta }\mathrm{\Theta }=\mathrm{\Theta }(\rho -\mathrm{\Theta }),\hfill & x\in \mathrm{\Omega },\hfill \\ \kappa \frac{\partial \mathrm{\Theta }}{\partial \nu }+\mathrm{\Theta }=0,\hfill & x\in \partial \mathrm{\Omega }.\hfill \end{array}$$

(4.2)

By virtue of Lemma 2.3, we see that (4.2) has a unique positive solution ${\mathrm{\Theta }}_{[\rho ]}$ with ${\mathrm{\Theta }}_{[\rho ]}\le \rho$ when $\rho >{\lambda }_{1,\kappa }$, where ρ is a positive constant. Hence, system (4.1) admits two semi-trivial solutions $({\mathrm{\Theta }}_{[a_1]},0)$ and $(0,{\mathrm{\Theta }}_{[a_2]})$ if $a_1>{\lambda }_{1,{\kappa }_1}$ and $a_2>{\lambda }_{1,{\kappa }_2}$.

By virtue of the maximum principle and Hopf’s lemma, we can give a priori estimates of positive solutions of (4.1), the proof of which is omitted here.

Proposition 4.1 Any nonnegative solution $(u,v)$ of (4.1) satisfies

$$u(x)\le Q:=\frac{a_1}{b_1},v(x)\le R:=\frac{a_2(b_1{r}_2+a_1)}{e{b}_1}.$$

In order to calculate the fixed point index, we introduce the following notations:

$$\begin{array}{c}E=C_{{\kappa }_1}(\overline{\mathrm{\Omega }})\times C_{{\kappa }_2}(\overline{\mathrm{\Omega }}),\text{where }{C}_{{\kappa }_i}(\overline{\mathrm{\Omega }})=\{w\in C(\overline{\mathrm{\Omega }}):{\kappa }_i\frac{\partial w}{\partial \nu }+w=0,x\in \partial \mathrm{\Omega }\};\hfill \\ W=K_1\times K_2,\text{where }{K}_i=\{w\in C_{{\kappa }_i}(\overline{\mathrm{\Omega }}):0\le w(x),x\in \overline{\mathrm{\Omega }}\};\hfill \\ D=\{(u,v)\in E:u\le Q+1,v\le R+1\},D^{\mathrm{\prime }}=(intD)\cap W.\hfill \end{array}$$

It is easy to verify that

$$\begin{array}{c}{\overline{W}}_{(0,0)}=K_1\times K_2;S_{(0,0)}=\{(0,0)\};\hfill \\ {\overline{W}}_{({\mathrm{\Theta }}_{[a_1]},0)}=C_{{\kappa }_1}(\overline{\mathrm{\Omega }})\times K_2;S_{({\mathrm{\Theta }}_{[a_1]},0)}=C_{{\kappa }_1}(\overline{\mathrm{\Omega }})\times \{0\};\hfill \\ {\overline{W}}_{(0,{\mathrm{\Theta }}_{[a_2]})}=K_1\times C_{{\kappa }_2}(\overline{\mathrm{\Omega }});S_{(0,{\mathrm{\Theta }}_{[a_2]})}=\{0\}\times C_{{\kappa }_2}(\overline{\mathrm{\Omega }}).\hfill \end{array}$$

From Proposition 4.1, we can see that the nonnegative solution of (4.1) must lie in $D^{\mathrm{\prime }}$.

Choosing

$$M>max\{|-a_1+(2b_1+\frac{c}{r_1})Q|,|-a_2+\frac{2e}{r}R|\},$$

then

$$u(a_1-b_{1}u-\frac{cv}{r_1+u})+Mu\text{and}v(a_2-\frac{ev}{r_2+u})+Mv$$

are nonnegative for all $(u,v)\in [0,Q]\times [0,R]$. Define an operator $\mathcal{F}:E\to E$ by

$$\mathcal{F}(u,v)={(-\mathrm{\Delta }+M)}^{-1}(u(a_1-b_{1}u-\frac{cv}{r_1+u})+Mu,v(a_2-\frac{ev}{r_2+u})+Mv).$$

By the strong maximum principle, ${(-\mathrm{\Delta }+M)}^{-1}$ is a compact linear operator, and ℱ is a direct sum of compact positive operators. Clearly, system (4.1) is equivalent to $\mathcal{F}(u,v)=(u,v)$ (it should be noted that this is independent of the choice of M as long as M is large enough). Thus, finding a positive solution of system (4.1) is equivalent to proving that ℱ has a nontrivial fixed point in $D^{\mathrm{\prime }}$. Without loss of generality, we may assume that $(0,0)$, $({\mathrm{\Theta }}_{[a_1]},0)$ and $(0,{\mathrm{\Theta }}_{[a_2]})$ are isolated fixed points of ℱ if they exist, and so the corresponding indices related to W are well defined. For $t\in [0,1]$, define a homotopy

$${\mathcal{F}}_t(u,v)={(-\mathrm{\Delta }+M)}^{-1}(tu(a_1-b_{1}u-\frac{cv}{r_1+u})+Mu,tv(a_2-\frac{ev}{r_2+u})+Mv),$$

then $\mathcal{F}={\mathcal{F}}_1$.

Similar to the discussion in Section 3, we have the following lemmas.

Lemma 4.2 Assume that $a_1>{\lambda }_{1,{\kappa }_1}$. Then

(i) ${index}_W(\mathcal{F},D^{\mathrm{\prime }})=1$, for an open set $D^{\mathrm{\prime }}$ in W.

(ii) ${index}_W(\mathcal{F},(0,0))=0$, if $a_2\ne {\lambda }_{1,{\kappa }_2}$.

(iii) ${index}_W(\mathcal{F},({\mathrm{\Theta }}_{[a_1]},0))=0$, if $a_2>{\lambda }_{1,{\kappa }_2}$.

(iv) ${index}_W(\mathcal{F},({\mathrm{\Theta }}_{[a_1]},0))=1$, if $a_2<{\lambda }_{1,{\kappa }_2}$.

Lemma 4.3 Assume that $a_2>{\lambda }_{1,{\kappa }_2}$ holds. Then the following items hold:

(i) ${index}_W(\mathcal{F},(0,{\mathrm{\Theta }}_{[a_2]}))=0$, if $a_1>{\lambda }_{1,{\kappa }_1}(\frac{c{\mathrm{\Theta }}_{[a_2]}}{r_1})$.

(ii) ${index}_W(\mathcal{F},(0,{\mathrm{\Theta }}_{[a_2]}))=1$, if $a_1<{\lambda }_{1,{\kappa }_1}(\frac{c{\mathrm{\Theta }}_{[a_2]}}{r_1})$.

Theorem 4.4 For system (4.1), the following results hold:

(i) If $a_1\le {\lambda }_{1,{\kappa }_1}$, then (4.1) has no positive solution and, in addition, if $a_2\le {\lambda }_{1,{\kappa }_2}$, then (4.1) has no nonnegative nonzero solution.

(ii) If $a_2>{\lambda }_{1,{\kappa }_2}$ and $a_1>{\lambda }_{1,{\kappa }_1}(\frac{c{\mathrm{\Theta }}_{[a_2]}}{r_1})$, then (4.1) admits a positive solution.

(iii) If $a_2>{\lambda }_{1,{\kappa }_2}$ and (4.1) has a positive solution, then ${\lambda }_{1,{\kappa }_1}(-a_1+\frac{c{\mathrm{\Theta }}_{[a_2]}}{r_1+{\mathrm{\Theta }}_{[a_1]}})<0$.

Proof Firstly, suppose on the contrary that $(\overline{u},\overline{v})$ is a positive solution of (4.1), then $(\overline{u},\overline{v})$ satisfies the equation

$$\{\begin{array}{cc}-\mathrm{\Delta }\overline{u}=\overline{u}(a_1-b_1\overline{u}-\frac{c\overline{v}}{r_1+\overline{u}}),\hfill & x\in \mathrm{\Omega },\hfill \\ {\kappa }_1\frac{\partial \overline{u}}{\partial \nu }+\overline{u}=0,\hfill & x\in \partial \mathrm{\Omega },\hfill \end{array}$$

and so ${\lambda }_{1,{\kappa }_1}(-a_1+b_1\overline{u}+\frac{c\overline{v}}{r_1+\overline{u}})=0$ by Lemma 2.2. Using the comparison property of an eigenvalue, it follows that $a_1>{\lambda }_{1,{\kappa }_1}$, which is a contradiction. Next, assume that $(\overline{u},\overline{v})$ is a nonnegative nonzero solution of (4.1). If $\overline{u}\not\equiv 0$ and $\overline{v}\equiv 0$, then $a_1>{\lambda }_{1,{\kappa }_1}$. Similarly, if $\overline{u}\equiv 0$ and $\overline{v}\not\equiv 0$, then $a_2>{\lambda }_{1,{\kappa }_2}$. A contradiction occurs. This completes the proof of (i).

For (ii), by Lemmas 4.2 and 4.3, we obtain

$${index}_W(\mathcal{F},(0,0))+{index}_W(\mathcal{F},({\mathrm{\Theta }}_{[a_1]},0))+{index}_W(\mathcal{F},(0,{\mathrm{\Theta }}_{[a_2]}))=0$$

and ${index}_W(\mathcal{F},D^{\mathrm{\prime }})=1$. So, (4.1) has a positive solution in $D^{\mathrm{\prime }}$, which shows that the second statement is true.

Finally, we prove (iii). Let $(\overline{u},\overline{v})$ be a positive solution of (4.1). Then $a_1>{\lambda }_{1,{\kappa }_1}$ holds so that (4.1) has a semi-trivial solution $({\mathrm{\Theta }}_{[a_1]},0)$. Because of $a_2>{\lambda }_{1,{\kappa }_2}$, (4.1) has a semi-trivial solution $(0,{\mathrm{\Theta }}_{[a_2]})$ such that $\overline{u}\le {\mathrm{\Theta }}_{[a_1]}$ and ${\mathrm{\Theta }}_{[a_2]}\le \overline{v}$ by the uniqueness of ${\mathrm{\Theta }}_{[a_1]}$ and ${\mathrm{\Theta }}_{[a_2]}$. Applying the comparison property of the eigenvalue, it is evident that

$${\lambda }_{1,{\kappa }_1}(-a_1+\frac{c{\mathrm{\Theta }}_{[a_2]}}{r_1+{\mathrm{\Theta }}_{[a_1]}})<{\lambda }_{1,{\kappa }_1}(-a_1+b_1\overline{u}+\frac{c\overline{v}}{r_1+\overline{u}})=0.$$

The proof is complete. □