Permanence, stability, and coexistence of a diffusive predator–prey model with modified Leslie–Gower and B–D functional response

Abstract

This paper investigates a diffusive predator–prey system with modified Leslie–Gower and B–D (Beddington–DeAngelis) schemes. Firstly, we discuss stability analysis of the equilibrium for a corresponding ODE system. Secondly, we prove that the system is permanent by the comparison argument of parabolic equations. Thirdly, sufficient conditions for the global asymptotic stability of the unique positive equilibrium of the system are proved by using the method of Lyapunov function. Finally, by using the maximum principle, Poincare inequality, and Leray–Schauder degree theory, we establish the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system, which indicates the effect of large diffusivity.

1 Introduction and model formulation

Population ecology is an important branch of ecology. Due to the complexity of ecological relations, mathematical methods and results have been increasingly used in ecology and population ecology, which is the most widely used in-depth in mathematical ecology. In recent years, because of the widespread application of biological models, such as a predator–prey model in population ecology, the research on them has aroused the attention of many scientists and biologists. Moreover, the predator–prey model is an important branch of reaction-diffusion equations. The dynamic relationship between predators and their prey is one of the dominant themes in ecology and mathematical ecology. During these thirty years, the investigation on the predator–prey models has been developed [1–10], and more realistic models have been derived in view of laboratory experiments. In particular, the research on the predator–prey models has been conducted from various views and many good results have been obtained (see [11–17] and the references therein).

Functional response is a reaction term in which the predation rate of each predator varies with the density of prey, that is, the predator’s predation effect on prey. Many scholars have studied predator–prey models with different functional responses from different views. For example, Wang et al. [18] and Cheng et al. [19] considered the Holling I type functional response in a predator–prey model, Ko and Ryu [20] focused on the Holling II type functional response, Zhuo and Zhang [21], Pang and Wang [22], and Wang [23] investigated the ratio-dependent predator–prey system, and other functional response, please see [24–29].

In references [30] and [31], the authors considered a predator–prey model incorporating a modified version of the Leslie–Gower functional response as well as the Holling-type II functional response:

$$ \textstyle\begin{cases} \frac{dx}{dt} = x ( a-bx-\frac{c_{1}y}{x+k_{1}} ) , \\ \frac{dy}{dt} = y ( d- \frac{c_{2}y}{x+k_{2}} ) , \\ x(0) > 0, \quad\quad y(0) > 0. \end{cases} $$

(1)

Model (1) describes a prey population x which serves as food for a predator with population y. The parameters a, d, b, $c_{1}$, $c _{2}$, $k_{1}$, $k_{2}$ are assumed to be only positive constants. In this paper, for model (1), we take into account the inhomogeneous distribution of predators and prey in different spatial locations with B–D functional response within a fixed bounded domain $\Omega \subset R^{N}$ with smooth boundary at any given time, and the natural tendency of each species to diffuse to areas of smaller population concentration [32–34]. Hence, we will investigate the reaction-diffusion system under the homogeneous Neumann boundary conditions [35] as follows:

$$ \textstyle\begin{cases} u_{t} - d_{1}\Delta u = u ( a- bu - \frac{c_{1}v}{u+mv+k_{1}} ) , \quad x\in \Omega ,t>0, \\ v_{t} - d_{2}\Delta v = v ( d- \frac{c_{2}v}{u+k_{2}} ) , \quad x\in \Omega ,t>0, \\ \frac{\partial u}{\partial n} = \frac{\partial u}{\partial n} = 0, \quad x\in \partial \Omega ,t>0, \\ u(x,0) = u_{0}(x)\geq 0, \quad\quad v(x,0)= v_{0}(x)\geq 0, \quad x\in \Omega , \end{cases} $$

(2)

where n is the outward unit normal vector of the boundary ∂Ω, the positive constants $d_{1}$, $d_{2}$ are the diffusion coefficients, $u_{0}$, $v_{0}$ are continuous functions of x. u and v stand for the densities of prey and predator, respectively. The parameters a, b, d, m, $c_{1}$, $c_{2}$, $k_{1}$, $k_{2}$ are assumed to be only positive constants. a and d denote the intrinsic growth rate of prey $u\triangleq u(x,t)$ and predator $v\triangleq v(x,t)$, respectively. $c_{1}$ and $c_{2}$ stand for a capturing rate to predator and a conversion rate of prey captured by predator, respectively. $f(u,v)=\frac{uv}{u+mv+k_{1}}$ stands for Beddington–DeAngelis functional response [36, 37]. For more biological background of system (2), one could refer to [38–42] and the references cited therein. Thus, this response function can better simulate the transformation law of two species.

The rest of this paper is arranged as follows. In Sect. 2, we discuss stability analysis of the equilibrium of ODE system corresponding to system (2). In Sect. 3, the permanence of the system is obtained by the comparison principle of parabolic equations. In Sect. 4, by using the method of Lyapunov function, we get sufficient conditions for the global asymptotic stability of the unique positive equilibrium of the system. In Sect. 5, the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system are established by using the Leray–Schauder degree theory, which demonstrates the effect of large diffusivity.

2 Stability analysis of ODE model

The goal of this section is to discuss stability analysis of ODE model for the reaction-diffusion predator–prey system (2).

Firstly, we give the ordinary differential equation of system (2) as follows:

$$ \textstyle\begin{cases} \frac{du}{dt} = u ( a-bu-\frac{c_{1} v}{u+mv+k_{1}} ) , \\ \frac{dv}{dt} = v ( d-\frac{c_{2}v}{u+k_{2}} ) . \end{cases} $$

(3)

We can easily get the three trivial equilibria of system (3) as follows: $P_{1}=(0,0)$, $P_{2}=(0,\frac{a}{b})$, $P_{3}=(0,\frac{dk _{2}}{c_{2}})$. Moreover, we can establish the existence and uniqueness of the positive nontrivial equilibrium by the following proposition.

Lemma 2.1

If $c_{1}< ma$, then system (3) has an interior equilibrium $P_{4}=(\tilde{u},\tilde{v})$.

Proof

Let

$$ \textstyle\begin{cases} a- bu - \frac{c_{1}v}{u+mv+k_{1}}=0, \\ d- \frac{c_{2}v}{u+k_{2}}=0, \end{cases} $$

(4)

by calculating, we get the following equation:

$$ b \biggl( 1+\frac{md}{c_{2}} \biggr) u^{2} + \biggl[ b \biggl( \frac{mk_{2}d}{c _{2}}+k_{1} \biggr) +\frac{c_{1}d}{c_{2}}-a \biggl( 1+ \frac{md}{c_{2}} \biggr) \biggr] u+\frac{dk _{2}(c_{1}-ma)}{c_{2}}-ak_{1}=0. $$

Consider the following quadratic function on u:

$$ F(u)=b \biggl( 1+\frac{md}{c_{2}} \biggr) u^{2} + \biggl[ b \biggl( \frac{mk _{2}d}{c_{2}}+k_{1} \biggr) +\frac{c_{1}d}{c_{2}}-a \biggl( 1+ \frac{md}{c _{2}} \biggr) \biggr] u+\frac{dk_{2}(c_{1}-ma)}{c_{2}}-ak_{1}. $$

Since $b(1+\frac{md}{c_{2}})>0$, and $F(0)=\frac{dk_{2}(c_{1}-ma)}{c _{2}}-ak_{1}<0$ if $c_{1}< ma$. $F(u)=0$ has a unique positive solution denoted by ũ, and $\tilde{v}=\frac{d(\tilde{u}+k_{2})}{c _{2}}$. Thus, the existence of the interior equilibrium is demonstrated. The proof is completed. □

It is easy to see that the equilibria of system (3) consist of three trivial critical points $P_{1}=(0,0)$, $P_{2}=(\frac{a}{b},0)$, $P _{3}=(0,\frac{dk_{2}}{c_{2}})$ on the boundary of $\Omega =\{(u,v):u\geq 0, v\geq 0\}$, and (4) has a nontrivial critical point $P_{4}=(\tilde{u},\tilde{v})$. Firstly, we give the Jacobian matrix of system (3) at some point $(u,v)$:

$$J_{(u,v)}=\left(\begin{array}{cc}a-2u-\frac{c_{1}v(mv+k_1)}{{(u+mv+k_1)}^2}& -\frac{c_{1}u(u+k_1)}{{(u+mv+k_1)}^2}\\ \frac{c_2{v}^2}{{(u+k_2)}^2}& d-\frac{2c_{2}v}{u+k_2}\end{array}\right).$$

Next, by using the ODE stability theory, we establish the following results of the stability on four points $P_{1}$, $P_{2}$, $P_{3}$, $P_{4}$, respectively.

• (i) For $P_{1}=(0,0)$, the corresponding Jacobian matrix is

  $$J_{P_1}=\left(\begin{array}{cc}a& 0\\ 0& d\end{array}\right),$$

  it follows that $J_{P_{1}}$ has two eigenvalues $\lambda_{1}=a>0$, $\lambda_{2}=d>0$, then $P_{1}=(0,0)$ is an unstable node.

• (ii) For $P_{2}=(\frac{a}{b}, 0)$, the corresponding Jacobian matrix is

  $$J_{P_2}=\left(\begin{array}{cc}a& -\frac{c_{1}a}{a+b{k}_1}\\ 0& d\end{array}\right),$$

  it follows that $J_{P_{2}}$ has two eigenvalues $\lambda_{1}=-a<0$, $\lambda_{2}=d>0$, then $P_{2}=(\frac{a}{b}, 0)$ is a saddle point.

• (iii) For $P_{3}=(0,\frac{dk_{2}}{c_{2}})$, the corresponding Jacobian matrix is

  $$J_{P_3}=\left(\begin{array}{cc}a& 0\\ \frac{d^2}{c_2}& d\end{array}\right),$$

  by (ii), obviously, $J_{P_{3}}$ is a saddle point.

• (iv) For $P_{4}=(\tilde{u},\tilde{v})$, the corresponding Jacobian matrix is

  $$\begin{array}{c}{J}_{P_4}=\left(\begin{array}{cc}-b\tilde{u}+\frac{c_1\tilde{u}\tilde{v}}{{(\tilde{u}+m\tilde{v}+k_1)}^2}& -\frac{c_1\tilde{u}(\tilde{u}+k_1)}{{(\tilde{u}+m\tilde{v}+k_1)}^2}\\ \frac{d^2}{c_2}& -d\end{array}\right),\hfill \\ det(\mu I-J_{P_4})={\mu }^2-\mu \cdot trac(J_{P_4})+det{J}_{P_4},\hfill \end{array}$$

  where

  $$\begin{aligned}& \det J_{P_{4}}=\biggl(-b\tilde{u}+ {\frac{c_{1}\tilde{u}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}\biggr) (-d)+ \frac{d ^{2}}{c_{2}}\cdot \frac{c_{1}\tilde{u}(\tilde{u}+k_{1})}{(\tilde{u}+m \tilde{v}+k_{1})^{2}}, \\& \operatorname{trac} (J_{P_{4}})= -d-b\tilde{u}+ {\frac{c_{1}\tilde{u}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}. \end{aligned}$$

It is easy to get that $\det J_{P_{4}}> 0$, and $\operatorname{trac} (J_{P _{4}}) < 0$, if $-b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}<0$. Then two eigenvalues of the matrix $J_{P_{4}}$ have negative real parts, therefore, the equilibrium $P_{4}=(\tilde{u},\tilde{v})$ is locally asymptotically stable.

From (i)–(iv), we obtained the type of stability on four equilibria points $P_{1}$, $P_{2}$, $P_{3}$, $P_{4}$, which indicates the law of long time change on the solutions of ODE system.

3 Permanence

The goal of this section is to show that any nonnegative solution $(u(x, t), v(x, t))$ of system (2) lies in a certain bounded region as $t\rightarrow \infty$ for all $x \in \Omega$.

Firstly, a well-known conclusion on the logistic equation is given as follows.

Lemma 3.1

([43])

Suppose that $u(x, t)$ is determined by the following problem:

$$ \textstyle\begin{cases} u_{t} =d_{1}\Delta u +\gamma ( 1- \frac{u}{K} ) , \quad x \in \Omega ,t>0, \\ \frac{\partial u}{\partial n} = 0, \quad x \in \Omega ,t>0, \\ u(x,0)=u_{0}(x)>0, \quad x \in \Omega , \end{cases} $$

(5)

then $\lim_{t\rightarrow \infty }u(x,t)=K$.

Theorem 3.1

Any solution of system (2) is nonnegative and defined for all $t > 0$ when $c_{1}< am$. Moreover, the nonnegative solution $(u(x, t), v(x, t))$ of system (2) yields

$$ \limsup_{t\rightarrow \infty }\max_{x\in \bar{\Omega }}u(x,t) \leq \frac{a}{b}, \quad\quad \limsup_{t\rightarrow \infty }\max_{x\in \bar{\Omega }}v(x,t) \leq \frac{d( \frac{a}{b} +k_{2})}{c_{2}}\doteq \hat{v}. $$

(6)

Proof

Since the initial value is nonnegative, it is easy to see that any solution of system (2) is nonnegative. (For the specific proof process, we refer to Theorem 2.2 in [40], Theorem 2.1 in [44], and Theorem 2.1 in [45].) Now we mainly discuss the remaining part of the theorem. It follows from the first equation of system (2) that

$$ u_{t} \leq d_{1} \Delta u+ u (a- bu). $$

(7)

Hence, by the comparison argument of parabolic equations [46] and Lemma 3.1, we know that, for any arbitrary $\epsilon > 0$, there exists $T_{1}> 0$ such that, for any $t > T_{1}$,

$$ u(x,t)\leq \frac{a}{b}+\epsilon . $$

(8)

It follows that

$$ \limsup_{t\rightarrow \infty }\max_{x\in \bar{\Omega }}u(x,t)\leq \frac{a}{b}. $$

According to the second equation of system (2) and system (8), we deduce

$$ v_{t} \leq d_{2} \Delta v+ v \biggl( d- \frac{c_{2}v}{\frac{a}{b}+\epsilon +k_{2}} \biggr) . $$

(9)

Hence, by the comparison argument of parabolic equations [46] and Lemma 3.1, we know that, for any arbitrary $\epsilon > 0$, there exists $T_{2}>T_{1}$ such that, for any $t > T_{2}$,

$$ v(x,t) \leq \frac{d(\frac{a}{b}+\epsilon +k_{2})}{c_{2}}+\epsilon \doteq \hat{v}_{\epsilon }. $$

(10)

It follows that

$$ \limsup_{t\rightarrow \infty }\max_{x\in \bar{\Omega }}v(x,t) \leq \frac{d( \frac{a}{b} +k_{2})}{c_{2}}\doteq \hat{v}. $$

(11)

The proof is completed. □

Theorem 3.2

If $c_{1}< am$, then the nonnegative solution $(u(x, t), v(x, t))$ of system (2) meets

$$ \liminf_{t\rightarrow \infty }\max_{x\in \bar{\Omega }}u(x,t) \geq \frac{a(m \hat{v}+k_{1})-c_{1}\hat{v}}{b}, \quad\quad \liminf_{t\rightarrow \infty } \max_{x\in \bar{\Omega }}v(x,t) \geq \frac{dk_{2}}{c_{2}}. $$

Proof

It follows from the first equation of system (2) and system (10) that

$$\begin{aligned} u_{t} &= d_{1}\Delta u + u \biggl( a- bu - \frac{c_{1}v}{u+mv+k_{1}} \biggr) \\ &\geq d_{1}\Delta u + u \biggl( a- bu - \frac{c_{1}v}{mv+k_{1}} \biggr) \\ &\geq d_{1}\Delta u + u \biggl( a- bu - \frac{c_{1}\hat{v}_{\epsilon }}{m\hat{v}_{\epsilon }+k_{1}} \biggr) \\ & = d_{1}\Delta u + u \biggl[ \biggl( a -\frac{c_{1}\hat{v}_{\epsilon }}{m\hat{v}_{\epsilon }+k_{1}} \biggr) -bu \biggr] . \end{aligned}$$

Hence, by the comparison argument of parabolic equations [46] and Lemma 3.1, it is easy to see that

$$ \liminf_{t\rightarrow \infty }\max_{x\in \bar{\Omega }}u(x,t)\geq \frac{a(m \hat{v}+k_{1})-c_{1}\hat{v}}{b}>0 \quad (c_{1}< am). $$

According to the second equation of system (2), we deduce

$$ v_{t}\geq d_{2} \Delta v+ v \biggl( d- \frac{c_{2}v}{k_{2}} \biggr) . $$

Hence, by the comparison argument of parabolic equations [46] and Lemma 3.1, we know that, for any arbitrary $\epsilon > 0$, there exists $T_{3}>T_{2}$ such that, for any $t > T_{3}$,

$$ v(x,t)\geq \frac{dk_{2}}{c_{2}}-\epsilon . $$

(12)

It follows that

$$ \liminf_{t\rightarrow \infty }\max_{x\in \bar{\Omega }}v(x,t) \geq \frac{dk _{2}}{c_{2}}. $$

(13)

The proof is completed. □

From Theorem 3.1 and Theorem 3.2, we can easily establish the following conclusion.

Theorem 3.3

Suppose $c_{1}< am$. The permanence of system (2) is valid.

Remark 3.1

According to Theorem 3.3, if $c_{1}< am$, we can easily deduce that there exist positive constants $\underline{u}$, u̅, $\underline{v}$, v̅ which satisfy $\underline{u}\leq u(x,t) \leq \overline{u}$, $\underline{v}\leq v(x,t)\leq \overline{v}$ with t large enough.

This section established that any nonnegative solution $(u(x, t), v(x, t))$ of system (2) lies in a certain bounded region as $t\rightarrow \infty$ for all $x \in \Omega$ when $c_{1}< am$, which demonstrates the density of the two species is in a bounded interval when the system parameters meet certain conditions.

4 Stability on the constant equilibrium of system (2)

The goal of this section is to investigate the local and global stability of the positive constant steady state $(\tilde{u},\tilde{v})= \tilde{U}$.

Now, we give some notations for developing our result.

• (i) Let $0 = \mu_{1} < \mu_{2}< \mu_{3}< \cdots <\cdots < \infty$ be the eigenvalues of −Δ on Ω with the homogeneous Neumann boundary condition.

• (ii) Set $S(\mu_{i})$ be the space of eigenfunctions corresponding to $\mu_{i}$ with $i = 0, 1, 2,\ldots$ .

• (iii) $X_{ij}:=\{c:\phi_{ij}: c\in R^{2}\}$, where $\phi_{ij}$ is an orthonormal basis of $S(\mu_{i})$ for $i = 0, 1, 2,\ldots$ , $\dim[S( \mu_{i})]$.

• (iv) $X := \{W = (u, v)\in [C^{1}(\bar{\Omega })]^{2}: \partial_{n}u = \partial_{n}v=0\}$, and so $X =\bigoplus_{i=1}^{\infty }X_{i}$, where $X_{i}=\bigoplus_{j=1}^{\dim[S(\mu_{i})]}X_{ij}$.

We first discuss the local stability of Ũ.

Theorem 4.1

Assume that $-b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}<0$, then the positive constant steady state $(\tilde{u},\tilde{v})=\tilde{U}$ of system (2) is locally asymptotically stable.

Proof

The linearization of system (2) at the positive constant solution $(\tilde{u},\tilde{v})=\tilde{W}$ can be written as

$$ W_{t}=\bigl(D\Delta +F_{W}(\tilde{W})\bigr)W, $$

here $W=(u(x,t),v(x,t))^{T}$, $D=\operatorname{diag} (d_{1}, d_{2})$,

$$\begin{array}{c}F(W)=(u(a-bu-\frac{c_{1}v}{u+mv+k_1}),v(d-\frac{c_{2}v}{u+k_2})),\hfill \\ F_W(\tilde{W})=\left(\begin{array}{cc}-b\tilde{u}+\frac{c_1\tilde{u}\tilde{v}}{{(\tilde{u}+m\tilde{v}+k_1)}^2}& -\frac{c_1\tilde{u}(\tilde{u}+k_1)}{{(\tilde{u}+m\tilde{v}+k_1)}^2}\\ \frac{d^2}{c_2}& -d\end{array}\right).\hfill \end{array}$$

For each $i = 0, 1, 2,\ldots$ , $X_{i}$ is invariant under the operator $D\Delta +F_{W}(\tilde{W})$, and λ is an eigenvalue of this operator on $X_{i}$ if and only if it is an eigenvalue of the matrix

$$\begin{array}{c}{A}_i=-{\mu }_{i}D+F_W(\tilde{W})=\left(\begin{array}{cc}-d_1{\mu }_i-b\tilde{u}+\frac{c_1\tilde{u}\tilde{v}}{{(\tilde{u}+m\tilde{v}+k_1)}^2}& -\frac{c_1\tilde{u}(\tilde{u}+k_1)}{{(\tilde{u}+m\tilde{v}+k_1)}^2}\\ \frac{d^2}{c_2}& -d_2{\mu }_i-d\end{array}\right),\hfill \\ det(\mu I-A_i)={\lambda }^2-\lambda \cdot trac(A_i)+det{A}_i,\hfill \end{array}$$

where

$$\begin{aligned}& \det A_{i}=(d_{2}\mu_{i}+d) \biggl[ d_{1}\mu_{i}-\tilde{u} \biggl( -b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}} \biggr) \biggr] + \frac{d^{2}}{c_{2}}\cdot \frac{c_{1}\tilde{u}(\tilde{u}+k_{1})}{( \tilde{u}+m\tilde{v}+k_{1})^{2}}, \\& \operatorname{trac} (A_{i})= -(d_{1}+d_{2}) \mu_{i} -d+\tilde{u} \biggl( -b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}} \biggr) . \end{aligned}$$

It follows that $\det A_{i} > 0$, and $\operatorname{trac} A_{i} < 0$ if $-b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}<0$. Then two eigenvalues of the matrix $A_{i}$ have negative real parts. Thus, the equilibrium $\tilde{W}=(\tilde{u},\tilde{v})$ is locally asymptotically stable. □

Next, we present the result of global stability of the unique positive equilibrium $\tilde{W}=(\tilde{u},\tilde{v})$ of system (2) by using the method of Lyapunov function.

Theorem 4.2

The unique positive equilibrium $\tilde{W}=(\tilde{u},\tilde{v})$ of system (2) is globally asymptotically stable if

$$ \textstyle\begin{cases} {b+\frac{c_{1}\tilde{v}}{(\overline{u}+m\overline{v}+k_{1})(\tilde{u}+m \tilde{v}+k_{1})} -\frac{c_{1}\tilde{u}+c_{1}k_{1}}{2(\underline{u}+m \underline{v}+k_{1})(\tilde{u}+m\tilde{v}+k_{1})}} \\ {-\frac{c_{2}\tilde{v}}{2(\underline{u}+k_{2})(\tilde{u}+k_{2})}>0,} \\ {\frac{c_{2}\tilde{u}+c_{2}k_{2}}{(\overline{u}+k_{2})(\tilde{u}+k _{2})}-\frac{c_{2}\tilde{v}}{2(\underline{u}+k_{2})(\tilde{u}+k_{2})} -\frac{c_{1}\tilde{u}+c_{1}k_{1}}{2(\underline{u}+m\underline{v}+k _{1})(\tilde{u}+m\tilde{v}+k_{1})}>0.} \end{cases} $$

(14)

Proof

Set $(u(x, t), v(x, t))$ be the solution of system (2). To demonstrate our claim, we construct a Lyapunov function defined as follows:

$$ \begin{gathered} \hat{E}(u,v)=u-\tilde{u}-\tilde{u} \ln \biggl(\frac{u}{\tilde{u}} \biggr)+v- \tilde{v}-\tilde{v}\ln \biggl(\frac{v}{\tilde{v}}\biggr), \\ V(t)= \int_{\Omega } \hat{E}\bigl(u(x,t),v(x,t)\bigr)\,dx.\end{gathered} $$

(15)

Obviously, we know that $V(t)\geq 0$ with all $t>0$. Differentiating $V(t)$ along the solutions of system (2), we get

$$ \begin{aligned}[b] \frac{dV(t)}{dt} &= \int_{\Omega }\biggl[\biggl(1-\frac{\tilde{u}}{u} \biggr)u_{t}+\biggl(1-\frac{ \tilde{v}}{v}\biggr)v_{t}\biggr] \,dx \\ & = \int_{\Omega }\biggl[\biggl(1-\frac{\tilde{u}}{u}\biggr)d _{1}\Delta u +\biggl(1-\frac{\tilde{v}}{v}\biggr)d_{2}\Delta v \biggr]\,dx \\ & \quad {} + \int_{ \Omega }\biggl[(u-\tilde{u}) \biggl(a- bu - \frac{c_{1}v}{u+mv+k_{1}} \biggr) +(v- \tilde{v}) \biggl(d- \frac{c_{2}v}{u+k_{2}}\biggr)\biggr]\,dx \\ &\doteq N_{1}(t)+N_{2}(t). \end{aligned} $$

(16)

For $N_{1}(t)$, according to the Neumann boundary condition of system (2), we obtain

$$ N_{1}(t)=-d_{1}\tilde{u} \int_{\Omega }\frac{ \vert \nabla u \vert ^{2}}{u^{2}}\,dx-d _{2}\tilde{v} \int_{\Omega }\frac{ \vert \nabla v \vert ^{2}}{v^{2}}\,dx\leq 0. $$

For $N_{2}(t)$, according to Remark 3.1, we know that there exist $\underline{u}$, u̅, $\underline{v}$, v̅, and $\underline{u}\leq u(x,t)\leq \overline{u}$, $\underline{v}\leq v(x,t) \leq \overline{v}$ with t large enough. It follows that

$$\begin{aligned} N_{2}(t) &= \int_{\Omega }(u-\tilde{u}) \biggl( a- bu - \frac{c_{1}v}{u+mv+k _{1}}-a+ b \tilde{u}+\frac{c_{1}\tilde{v}}{\tilde{u}+m\tilde{v}+k_{1}} \biggr) \\ & \quad {} +(v- \tilde{v}) \biggl( d- \frac{c_{2}v}{u+k_{2}}-d+\frac{c_{2} \tilde{v}}{\tilde{u}+k_{2}} \biggr) \,dx \\ &\leq \int_{\Omega }[(u- \tilde{u}) \biggl(- b(u-\tilde{u})- \frac{c_{1}v\tilde{u}-c_{1}\tilde{v}u+c _{1}k_{1}(v-\tilde{v})}{(u+mv+k_{1})(\tilde{u}+m\tilde{v}+k_{1})}\biggr) \\ & \quad {} -(v- \tilde{v}) \biggl[ \frac{c_{2}(v\tilde{u}-\tilde{v}u)+c_{2}k_{2}(v- \tilde{v})}{(u+k_{2})(\tilde{u}+k_{2})} \biggr] \,dx \\ &\leq \int_{\Omega } \biggl\{ - \biggl[ b+\frac{c_{1}\tilde{v}}{(\overline{u}+m\overline{v}+k_{1})( \tilde{u} +m\tilde{v}+k_{1})}- \frac{c_{1}\tilde{u}+c_{1}k_{1}}{2( \underline{u}+m\underline{v}+k_{1})(\tilde{u}+m\tilde{v}+k_{1})} \\ & \quad {} -\frac{c _{2}\tilde{v}}{2(\underline{u}+k_{2})(\tilde{u}+k_{2})}\biggr](u-\tilde{u})^{2} -\biggl[ \frac{c_{2}\tilde{u}+c_{2}k_{2}}{(\overline{u}+k_{2})(\tilde{u}+k _{2})}-\frac{c_{2}\tilde{v}}{2(\underline{u}+k_{2})(\tilde{u}+k_{2})} \\ &\quad {} -\frac{c_{1}\tilde{u}+c_{1}k_{1}}{2(\underline{u}+m\underline{v}+k _{1})(\tilde{u}+m\tilde{v}+k_{1})}\biggr](v-\tilde{v})^{2}\biggr\} \,dx. \end{aligned}$$

Observe that from our assumption in system (14), we get $N_{2}(t)<0$, it follows that

$$ \frac{dV(t)}{dt}=N_{1}(t)+N_{2}(t)< 0. $$

Thus, $\tilde{W}=(\tilde{u},\tilde{v})$ is globally asymptotically stable, the whole proof is completed. □

5 Nonconstant positive steady states of system (2)

The main purpose of this section is to provide some sufficient conditions for the existence and nonexistence of a nonconstant positive solution of the steady states of system (2) by using the maximum principle, Poincaré inequality, and Leray–Schauder degree theory. The corresponding steady-state problem of system (2) is the elliptic system as follows:

$$ \textstyle\begin{cases} - d_{1}\Delta u = u ( a- bu - \frac{c_{1}v}{u+mv+k_{1}} ) , \quad x\in \Omega , \\ -d_{2} \Delta v = v ( d- \frac{c_{2}v}{u+k_{2}} ) , \quad x \in \Omega , \\ \frac{\partial u}{\partial n} = \frac{\partial u}{\partial n} = 0, \quad x\in \partial \Omega . \end{cases} $$

(17)

Firstly, we establish a priori positive upper and lower bounds for the positive solution of system (17). For this goal, we cite a known result which is due to Lou and Ni [47].

Proposition 5.1

(Maximum principle [47], Proposition 2.2)

Let $g\in C(\bar{\Omega }\times R)$.

• (i) If $\omega \in C^{2}(\Omega )\times C^{1}(\bar{\Omega })$ satisfies

  $$ \Delta \omega (x) +g\bigl(x,\omega (x)\bigr) \geq 0, \quad x\in \Omega ,\quad\quad \frac{ \partial \omega }{\partial n} \leq 0, \quad x\in \partial \Omega , $$

  and $\omega (x_{0})=\max_{\bar{\Omega }}\omega$, then $g(x_{0}, \omega (x_{0})) \leq 0$.

• (ii) If $\omega \in C^{2}(\Omega )\times C^{1}(\bar{\Omega })$ satisfies

  $$ \textstyle\begin{cases} \Delta \omega (x) +g(x,\omega (x)) \leq 0, \quad x\in \Omega , \\ \frac{\partial \omega }{\partial n} \geq 0, \quad x\in \partial \Omega , \end{cases} $$

  and $\omega (x_{0})=\min_{\bar{\Omega }}\omega$, then $g(x_{0}, \omega (x_{0})) \leq 0$.

In the rest of this paper, we say the positive solution of system (17) belongs to $C^{2}(\bar{\Omega })\times C^{2}(\bar{ \Omega })$ by the standard regularity theory for elliptic equations [46, 48]. For notational convenience, we shall define $\Lambda = \Lambda (a, d, b, c_{1}, c_{2}, k_{1}, k_{2}, m)$ in the last part.

Theorem 5.1

(Upper bound)

Any positive solution $(u,v)$ of system (17) satisfies

$$ \max_{x\in \bar{\Omega }}u(x)\leq \frac{a}{b}, \quad\quad \max _{x\in \bar{\Omega }}v(x)\leq \frac{d(\frac{a}{b} +k_{2})}{c_{2}} \doteq \hat{v}. $$

(18)

Proof

Applying Proposition 5.1 to the first equation of system (17) meets the first inequality of system (18). For the second equation of system (17), by Proposition 5.1, we similarly get

$$ \max_{x\in \bar{\Omega }}v(x)\leq \frac{d(u(x_{0})+k_{2})}{c_{2}} \leq \frac{d(\frac{a}{b} +k_{2})}{c_{2}} \doteq \hat{v}. $$

 □

Theorem 5.2

(Lower bound)

Any positive solution $(u,v)$ of system (17) satisfies

$$ \textstyle\begin{cases} \min_{x\in \bar{\Omega }}u(x)\geq \frac{a}{b+ \frac{c_{1}d}{k_{1}c_{2}}(\frac{a}{b}+k_{2})}\doteq \ell , \\ \min_{x\in \bar{\Omega }}v(x)\leq \frac{d(\ell +k_{2})}{c_{2}}. \end{cases} $$

(19)

Proof

Set $u(x_{0})=\min_{x\in \bar{\Omega }}u(x)$, applying Proposition 5.1 to the first equation of system (17), we have

$$ \min_{x\in \bar{\Omega }}u(x)\geq \frac{a}{b+\frac{c_{1}\omega (x_{0})}{k _{1}}} \geq \frac{a}{b+\frac{c_{1}d}{k_{1}c_{2}}(\frac{a}{b}+k_{2})} \doteq \ell . $$

For the second equation of system (17), let $v(y_{0})= \min_{x\in \bar{\Omega }}v(x)$, by Proposition 5.1, we similarly get

$$ \min_{x\in \bar{\Omega }}v(x)\geq \frac{d(u(y_{0})+k_{2})}{c_{2}} \leq \frac{d(\ell +k_{2})}{c_{2}}. $$

The proof is completed. □

In the following, we provide some sufficient conditions for the existence and nonexistence of a nonconstant positive solution of the steady states of system (17) by dividing into the following two subsections.

5.1 Nonexistence of nonconstant positive solutions

The goal of this part is to provide the nonexistence of nonconstant positive solutions of system (17) by the effect of large diffusivity. For some related research on the effect of small/large diffusivity on reaction-diffusion equations, we can refer to [46, 48]. For ease of notation, we set

$$ g_{1}(u,v) = u \biggl( a- bu - \frac{c_{1}v}{u+mv+k_{1}} \biggr) , \quad\quad g_{2}(u,v) = v \biggl( d- \frac{c_{2}v}{u+k_{2}} \biggr) . $$

Theorem 5.3

Assume that $\tilde{d}_{2}>\frac{d}{\mu_{1}}$ is a fixed positive constant. Then there exists a positive constant $\tilde{d}_{1} = \tilde{d}_{1}(\tilde{d}_{2}, \Lambda )$ such that system (17) has no nonconstant positive solution for $d_{1}\geq \tilde{d}_{1}$, $d_{2}\geq \tilde{d}_{2}$.

Proof

Suppose that $(u, v)$ is a positive solution of system (17). Set $\bar{u}=\frac{1}{ \vert \Omega \vert }\int_{\Omega }u\,dx$ and $\bar{v}=\frac{1}{ \vert \Omega \vert }\int_{\Omega }v\,dx$. Multiplying the first equation of system (17) by $u-\bar{u}$, and integrating over Ω by parts, by calculating, we obtain

$$\begin{aligned} &d_{1} \int_{\Omega } \bigl\vert \nabla (u-\bar{u}) \bigr\vert ^{2}\,dx \\ &\quad = \int_{\Omega }g _{1}(u,v) (u-\bar{u})\,dx \\ &\quad = \int_{\Omega }\bigl[g_{1}(u,v)-g_{1}(\bar{u}, \bar{v})\bigr](u-\bar{u})\,dx \\ &\quad = \int_{\Omega }\biggl[u\biggl(a- bu - \frac{c_{1}v}{u+mv+k _{1}}\biggr) - \bar{u}\biggl(a- b\bar{u} - \frac{c_{1}\bar{v}}{\bar{u}+m\bar{v}+k _{1}}\biggr)\biggr](u-\bar{u})\,dx \\ &\quad = \int_{\Omega }\biggl[a(u-\bar{u})- b(u+\bar{u}) (u- \bar{u})- \frac{c_{1}vu}{u+mv+k_{1}}+\frac{c_{1}\bar{v}\bar{u}}{ \bar{u}+m\bar{v}+k_{1}}\biggr](u-\tilde{u})\,dx \\ &\quad \leq \int_{\Omega }\biggl[\bigl(a- b(u+ \bar{u})\bigr) (u- \bar{u})^{2} +2c_{1} \vert u-\bar{u} \vert \vert v- \bar{v} \vert + \frac{2c_{1}}{m}(u-\bar{u})^{2}\biggr]\,dx \\ &\quad \leq \int_{\Omega }\biggl[\biggl(a+\frac{2c _{1}}{m}\biggr) (u- \bar{u})^{2} +2c_{1} \vert u-\bar{u} \vert \vert v- \bar{v} \vert \biggr]\,dx. \end{aligned}$$

(20)

For the second equation of system (17), by the similar method, we have

$$ \begin{aligned}[b] d_{2} \int_{\Omega } \bigl\vert \nabla (v-\bar{v}) \bigr\vert ^{2}\,dx&= \int_{\Omega }g_{2}(u,v) (u- \bar{u})\,dx= \int_{\Omega }\bigl[g_{2}(u,v)-g_{2}(\bar{u}, \bar{v})\bigr](v- \bar{v})\,dx \\ &= \int_{\Omega }\biggl[v\biggl(d - \frac{c_{2}v}{u+k_{2}}\biggr) -\bar{v} \biggl(d - \frac{c_{2}\bar{v}}{\bar{u}+k_{2}}\biggr)\biggr](v-\bar{v})\,dx \\ &\leq \int_{ \Omega }\biggl[d(v-\bar{v})^{2} +\frac{c_{2}\bar{v}^{2}}{(\ell +k_{2})( \bar{u}+k_{2})} \vert u-\bar{u} \vert \vert v-\bar{v} \vert \biggr]\,dx. \end{aligned} $$

(21)

Adding system (20) and system (21), we get

$$ \begin{aligned}[b] &d_{1} \int_{\Omega } \bigl\vert \nabla (u-\bar{u}) \bigr\vert ^{2}\,dx+d_{2} \int_{\Omega } \bigl\vert \nabla (v-\bar{v}) \bigr\vert ^{2}\,dx\\ &\quad = \int_{\Omega }d_{1} \vert \nabla u \vert ^{2}+d_{2} \vert \nabla v \vert ^{2}\,dx \\ &\quad \leq \int_{\Omega }\biggl[\biggl(a+\frac{2c_{1}}{m}\biggr) (u- \bar{u})^{2} +2\hbar \vert u-\bar{u} \vert \vert v-\bar{v} \vert +d(v-\bar{v})^{2}\biggr]\,dx \\ &\quad \leq \int_{\Omega }\biggl[\biggl(a+\frac{2c_{1}}{m}+\frac{\hbar }{\epsilon } \biggr) (u- \bar{u})^{2} +(d+\epsilon \hbar ) (v-\bar{v})^{2} \biggr]\,dx, \end{aligned} $$

(22)

where $\hbar =c_{1}+\frac{c_{2}\bar{v}^{2}}{2(\ell +k_{2})(\bar{u}+k _{2})}$, and ϵ is an arbitrary small positive constant, the last inequality of system (22) can be deduced from the following fact:

$$ 2\hbar \vert u-\bar{u} \vert \vert v-\bar{v} \vert =2\sqrt{ \frac{\hbar }{\epsilon }} \vert u- \bar{u} \vert \sqrt{\epsilon \hbar } \vert v- \bar{v} \vert \leq \frac{\hbar }{\epsilon }(u-\bar{u})^{2}+\epsilon \hbar (v-\bar{v})^{2}. $$

It follows from the Poincare inequality that

$$ \begin{aligned}[b] &\int_{\Omega }\bigl[d_{1}\mu_{1}(u- \bar{u})^{2} +d_{2}\mu_{2}(v-\bar{v})^{2} \bigr]\,dx \\&\quad \leq \int_{\Omega }\biggl[\biggl(a+\frac{2c_{1}}{m}+\frac{\hbar }{\epsilon } \biggr) (u- \bar{u})^{2}+(d+\epsilon \hbar ) (v-\bar{v})^{2} \biggr]\,dx.\end{aligned} $$

(23)

Since $d_{2}\mu_{1}>d$, we may choose $\epsilon_{0}$, ($\epsilon_{0} \ll 1$) such that $d_{2}>\tilde{d}_{2}\doteq \frac{1}{\mu_{1}} (d+\epsilon _{0}\hbar )$. Consequently, by (23),

$$ \int_{\Omega }d_{1}\mu_{1}(u- \bar{u})^{2} \,dx\leq \int_{\Omega }\biggl[\biggl(a+\frac{2c _{1}}{m}+\frac{\hbar }{\epsilon_{0}} \biggr) (u-\bar{u})^{2} \biggr]\,dx, $$

which implies that $u=\bar{u}=\mathrm{constant}$, and in turn $v=\bar{v}=\mathrm{constant}$, if $d_{1}\mu_{1}>a+\frac{2c_{1}}{m}+\frac{ \hbar }{\epsilon_{0}}$, i.e., $d_{1}>\tilde{d}_{1}\doteq \frac{1}{\mu _{1}}(a+\frac{2c_{1}}{m}+\frac{\hbar }{\epsilon_{0}})$. The proof is completed. □

Theorem 5.3 obtained the sufficient condition on the nonexistence of nonconstant positive solutions of system (17) by the effect of large diffusivity, which shows that the two species cannot coexist in the bounded region Ω when the diffusivity is large enough.

5.2 Global existence of nonconstant positive solutions

The goal of this subsection is to establish the global existence of nonconstant positive solutions to system (17) when the diffusion coefficients $d_{1}$ and $d_{2}$ vary while the parameters a, d, b, $c_{1}$, $c_{2}$, $k_{1}$, $k_{2}$, and m are kept fixed.

Theorem 4.1 indicates that system (17) has no nonconstant positive solutions when $-b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}\leq 0$. In view of this reason, we shall restrict this discussion to the case $-b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}} > 0$.

For simplicity, we write $W=(u,v)$ and $\tilde{W}=(\tilde{u}, \tilde{v})$. Let θ, β, δ be defined as follows:

$$\begin{array}{c}\varrho =-b\tilde{u}+\frac{c_1\tilde{u}\tilde{v}}{{(\tilde{u}+m\tilde{v}+k_1)}^2},\beta =\frac{c_1\tilde{u}(\tilde{u}+k_1)}{{(\tilde{u}+m\tilde{v}+k_1)}^2},\delta =\frac{d^2}{c_2},\hfill \\ D=\left(\begin{array}{cc}{d}_1& 0\\ 0& d_2\end{array}\right),F(W)=\left(\begin{array}{c}u(a-bu-\frac{c_{1}y}{u+mv+k_1}\\ v(d-\frac{c_{2}v}{u+k_2})\end{array}\right),\hfill \\ Q=\left(\begin{array}{cc}\varrho & -\beta \\ \delta & -d\end{array}\right)=F_W(\tilde{W}).\hfill \end{array}$$

We note that system (17) can be written as

$$ -\Delta W=D^{-1}F(W), \quad x\in \Omega ,\quad\quad \frac{\partial W}{\partial n}=0, \quad x\in \partial \Omega . $$

(24)

Furthermore, W̃ solves system (24) if and only if it satisfies

$$ f(d_{1},d_{2}; W)=W-(I-\Delta )^{-1} \bigl\{ D^{-1}F(W)+W\bigr\} =0, $$

(25)

where $(I-\Delta )^{-1}$ is the inverse of $I-\Delta$ with the homogeneous Neumann boundary condition. Direct computation gives

$$ D_{W}f(d_{1},d_{2}; W):=I-(I- \Delta )^{-1}\bigl\{ D^{-1}Q+I\bigr\} . $$

(26)

As in the proof of Theorem 4.2, we note that γ is an eigenvalue of $D_{W}f(d_{1},d_{2}; W)$ for each $X_{i}$ if and only if $\gamma (1+\mu_{i})$ is an eigenvalue of the matrix

$$Q={\mu }_{i}I-D^{-1}Q=\left(\begin{array}{cc}{\mu }_i-d_1^{-1}\varrho & d_1^{-1}\beta \\ -d_2^{-1}\delta & {\mu }_i-d_2^{-1}d\end{array}\right).$$

It follows that

$$ \det M_{i}=\frac{1}{d_{1}d_{2}}\bigl[d_{1}d_{2} \mu_{i}^{2}+(d_{1}d-d_{2} \varrho ) \mu_{i}+\beta \delta -\varrho d\bigr], \quad\quad \operatorname{trac} M_{i}=2\mu_{i}+\frac{d}{d_{2}}-\frac{\varrho }{d_{1}} . $$

Write

$$ G(d_{1},d_{2}; \mu )=d_{1}d_{2} \mu_{i}^{2}+(d_{1}d-d_{2}\varrho )\mu _{i}+\beta \delta -\varrho d. $$

Then $G(d_{1},d_{2}; \mu )=d_{1}d_{2}\det M_{i}$. If

$$ (d_{1}d-d_{2}\varrho )^{2}>4d_{1}d_{2}( \beta \delta -\varrho d), $$

(27)

then $G(d_{1},d_{2}; \mu )=0$ has two real roots, namely

$$ \textstyle\begin{cases} \mu_{+}(d_{1},d_{2}) =\frac{d_{2}\varrho -d_{1}d+\sqrt{(d_{1}d-d _{2}\varrho )^{2}-4d_{1}d_{2}(\beta \delta -\varrho d)}}{2d_{1}d_{2}}, \\ \mu_{-}(d_{1},d_{2}) =\frac{d_{2}\varrho -d_{1}d-\sqrt{(d_{1}d-d _{2}\varrho )^{2}-4d_{1}d_{2}(\beta \delta -\varrho d)}}{2d_{1}d_{2}}. \end{cases} $$

(28)

Let

$$ \Phi =\Phi (d_{1},d_{2})=\bigl\{ \mu : \mu \geq 0, \mu_{-}(d_{1},d_{2})< \mu < \mu_{+}(d_{1},d_{2}) \bigr\} , \quad\quad S_{q}=\{\mu_{0}, \mu_{1}, \mu_{2}, \mu_{3},\ldots \}, $$

and set $m(\mu_{i})$ be the multiplicity of $\mu_{i}$. Next we will calculate the index of $f(d_{1},d_{2}; \cdot )$ at W̃, we firstly state a lemma whose proof can be found in [22].

Lemma 5.1

Suppose $G(d_{1},d_{2}; \mu )\neq 0$ for all $\mu_{i}\in S_{q}$, then

$$ \bigl(f(d_{1},d_{2}; \tilde{W})\bigr)=(-1)^{\tau }, $$

where

$$ \tau = \textstyle\begin{cases} \sum_{\mu_{i}\in \Phi \cap S_{q}} m(\mu_{i}) & \Phi \cap S_{q}\neq \emptyset , \\ 0 & \Phi \cap S_{q} = \emptyset . \end{cases} $$

(29)

Moreover, if $G(d_{1},d_{2}; \mu )> 0$ for all $\mu_{i}\geq 0$, then $\tau =0$.

According to Lemma 5.1, we are going to calculate the index of $f(d_{1},d_{2}; \cdot )$ at W̃, but the key step is to determine the range of μ for which $G(d_{1},d_{2}; \mu )< 0$.

Theorem 5.4

If $-b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}>0$, $\varrho /d_{1}\in (\mu_{l},\mu_{l+1})$ for some $l\geq 1$, and $\tau_{l}=\sum_{i=1}^{l}m(\mu_{i})$ is odd, then there exists a positive constant d̂ such that system (17) has at least one nonconstant positive solution for all $d_{2}\geq \hat{d}$.

Proof

Since $-b+ {\frac{c_{1}\tilde{v}}{(\tilde{u}+m\tilde{v}+k_{1})^{2}}}>0$, it is easy to know $\varrho >0$. Thus, we can deduce that if $d_{2}$ is large enough then system (27) holds and $\mu_{+}(d_{1},d_{2})>\mu _{-}(d_{1},d_{2}) > 0$. Furthermore,

$$ \lim_{d_{2}\rightarrow \infty }\mu_{+}(d_{1},d_{2})= \frac{\varrho }{d _{1}}, \quad\quad \lim_{d_{2}\rightarrow \infty }\mu_{-}(d_{1},d_{2})=0. $$

Notice that $\varrho /d_{1}\in (\mu_{l},\mu_{l+1})$, we get that there exists $d_{0} \gg 1$ such that

$$ \mu_{+}(d_{1},d_{2})\in ( \mu_{l},\mu_{l+1}),\quad\quad 0< \mu_{-}(d_{1},d _{2})< \mu_{1}\quad \text{for all } d_{2}\geq d_{0}. $$

(30)

Thanks to Theorem 5.3, we obtain that there exists $\bar{d} \geq d_{0}$ such that system (17) with $d_{1}=\bar{d}$ and $d_{2}\geq \bar{d}$ has no nonconstant positive solution. Moreover, we can choose d̄ so large that $\varrho /\bar{d} < \mu_{1}$. It follows that there exists $\hat{d} > \bar{d}$ such that

$$ 0 < \mu_{-}(d_{1},d_{2})< \mu_{+}(d_{1},d_{2})< \mu_{1}\quad \text{for all } d_{2}\geq \hat{d}. $$

(31)

We shall prove that, for any $d_{2}\geq \hat{d}$, system (17) has at least one nonconstant positive solution. By the method of contradiction, suppose that this claim is not true for some $\hat{d}_{2}\geq \hat{d}$. Thanks to the homotopy argument, we can demonstrate a contradiction as follows.

Let $d_{2}=\hat{d}_{2}$ be fixed, for $t\in [0,1]$, we define

$$D(t)=\left(\begin{array}{cc}t{d}_1+(1-t)\overline{d}& 0\\ 0& t{d}_2+(1-t)\hat{d}\end{array}\right)$$

and investigate the following problem:

$$ \textstyle\begin{cases} -\Delta W=D^{-1}(t)F(W), \quad x\in \Omega , \\ \frac{\partial W}{\partial n}=0, \quad x\in \partial \Omega . \end{cases} $$

(32)

Note that W is a nonconstant positive solution of system (17) if and only if it is such a solution of system (32) for ${t = 1}$. It is obvious that W̃ is the unique positive constant solution of system (32). For any $0 < t < 1$, W is a nonconstant positive solution of system (32) if and only if it is such a solution of the problem

$$ \aleph (W; t)=W-(I-\Delta )^{-1}\bigl\{ D^{-1}(t)F(W)+W\bigr\} =0. $$

(33)

Our above arguments have shown that system (33) has no nonconstant positive solution for $t=0$, and we have assumed that there is no such solution for $t=1$ at $d_{2}= \hat{d}_{2}$. It is easy to see that

$$ \aleph (W; 1)=f(d_{1},d_{2}; W), \quad\quad \aleph (W; 0)=f(\bar{d}, \hat{d}; W) $$

and

$$ \textstyle\begin{cases} D_{W}f(d_{1},d_{2}; \tilde{W}):=I-(I-\Delta )^{-1}\{D^{-1}Q+I\}, \\ D_{W}f(\bar{d},\hat{d}; \tilde{W}):=I-(I-\Delta )^{-1}\{\hat{D}^{-1}Q+I\} \end{cases} $$

(34)

with $\hat{D}=\operatorname{diag} \{\bar{d}, \hat{d} \}$. According to system (30) and system (31), we get

$$ \Phi (d_{1},d_{2}) \cap S_{l}=\{ \mu_{1},\mu_{2},\mu_{3},\ldots, \mu _{l} \}\quad \text{and}\quad \Phi (d_{1},d_{2}) \cap S_{l}=\emptyset . $$

Observe that $\tau_{l}$ is odd, Lemma 5.1 gives

$$ \textstyle\begin{cases} \operatorname{index}(\aleph (\cdot ; 1), \tilde{W})=\operatorname{index}(f(d_{1},d _{2}; \cdot ), \tilde{W}):=(-1)^{\tau_{l}}=-1, \\ \operatorname{index}(\aleph (\cdot ; 0), \tilde{W})=\operatorname{index}(f(\bar{d} , \hat{d} ; \cdot ), \tilde{W}):=(-1)^{0}=1. \end{cases} $$

(35)

Now, by Theorem 4.2 and Theorem 5.1, there exist positive constants $\underline{B}$ and B̅ such that, for all $t\in [0,1]$, the positive solution of system (33) satisfies $\underline{B}\leq u(x),v(x)\leq \overline{B}$ on Ω̄. Set

$$ \Gamma =\bigl\{ W\in X: \underline{B}\leq u(x),v(x)\leq \overline{B}\bigr\} . $$

It follows that $\aleph (W; t)\neq 0$ for all $W\in \partial \Gamma$ and $t\in [0,1]$. By the homotopy invariance of the Leray–Schauder degree [49], we obtain

$$ \operatorname{deg}\bigl(\aleph (\cdot ; 1),\Gamma , 0\bigr)= \operatorname{deg}\bigl(\aleph (\cdot ; 0), \Gamma , 0\bigr). $$

(36)

Since both equations $\aleph (W ; 1) = 0$ and $\aleph (W ; 0) = 0$ have the unique positive solution W̃ in Γ, it follows that

$$ \textstyle\begin{cases} \operatorname{deg}(\aleph (\cdot ; 1),\Gamma , 0)=\operatorname{index}(\aleph (\cdot ; 1), \tilde{W})=-1, \\ \operatorname{deg}(\aleph (\cdot ; 0),\Gamma , 0)=\operatorname{index}(\aleph (\cdot ; 0), \tilde{W})=1. \end{cases} $$

This contradicts system (36) and our proof is completed. □

By the similar method, we establish the following conclusion whose proof was omitted.

Theorem 5.5

Suppose that the pair $(d_{1},d_{2})$ yields

$$ \Phi (d_{1},d_{2}) \cap S_{q}=\{ \mu_{r},\mu_{r+1},\mu_{r+2},\ldots, \mu_{r+l}\} $$

for some $r\geq 1$ and $l\geq 1$. If $\tau_{l}=\sum_{j=0}^{l}m(\mu _{r+j})$ is odd, then there exists at least one nonconstant positive solution.

Remark 5.1

For system (28), it is easy to see that

$$ \lim_{d_{1}\rightarrow 0^{+}}\mu_{+}(d_{1},d_{2})= \frac{\beta \delta -\varrho d}{\varrho d_{2}}, \quad\quad \lim_{d_{1}\rightarrow 0^{+}}\mu_{-}(d _{1},d_{2})=\infty . $$

If all $\mu_{i}$, $i = 0, 1, 2,\ldots$ , are simple and $(\beta \delta - \varrho d)/(\varrho d_{2}) \in S_{q}$, Theorem 5.5 implies that there exists a sequence of intervals $\{(d_{-}^{j}, d_{+}^{j})\}_{j=1} ^{\infty }$, with $d_{+}^{j+1}< d_{-}^{j}$ and $d_{-}^{j}\searrow 0^{+}$ as $j\rightarrow \infty$, such that system (17) has at least one nonconstant positive classical solution for every $d_{1}\in (d_{-}^{j}, d_{+}^{j})$.

Theorem 5.4 and Theorem 5.5 established the global existence of nonconstant positive solutions to system (17) when the diffusion coefficients $d_{1}$ and $d_{2}$ satisfy a suitable condition while the parameters a, d, b, $c_{1}$, $c_{2}$, $k_{1}$, $k_{2}$, and m are kept fixed. This shows that the two species can globally coexist in the bounded region Ω when the parameters of system (17) meet some suitable conditions.

6 Conclusion

This paper investigates the diffusive predator–prey system with modified Leslie–Gower and B–D (Beddington–DeAngelis) schemes under homogeneous Neumann boundary conditions. We have come to the following conclusions. Firstly, we discussed stability analysis of the equilibrium of ODE system corresponding to system (2) and obtained the results of the stability on four critical points $P_{1}$, $P_{2}$, $P_{3}$, $P_{4}$. Secondly, the permanence of system was obtained by the comparison principle of parabolic equations (see Theorem 3.1, Theorem 3.2, and Theorem 3.3). Thirdly, by using the method of Lyapunov function, we got sufficient conditions for the global asymptotic stability of the unique positive equilibrium of the system (see Theorem 4.2). Finally, by using the maximum principle, Poincaré inequality, and the Leray–Schauder degree theory, we have established the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system, which demonstrates the effect of large diffusivity (see Theorem 5.3, Theorem 5.4, and Theorem 5.5). The research result shows that the coexistence states of two species of organisms depend on certain ranges of the parameters.