Backward bifurcation of predator–prey model with anti-predator behaviors

Abstract

In this study, we consider a predator–prey model with stage structure and anti-predator behavior such that the adult prey can counterattack their predators. We first investigate the existence and stability of the equilibria. Especially, we verify that there can exist at most one positive equilibrium, which is always stable whenever it exists, if the predator only feed on one age class. We then prove that the system can undergo either a forward bifurcation or a backward bifurcation. Numerical analyses show that anti-predator behavior is beneficial to the growth of prey population, especially helps the equilibrium level of the prey population increase, by enhancing the pressure on the predator. Moreover, anti-predator behavior makes the coexistence of the predator and prey less likely by shrinking the coexistence region with respect to the initial conditions or weakening the existence and stability of the positive equilibrium.

1 Introduction

Although biologists routinely label the animals as predator or prey, there are many examples of role reversals in predators and prey (anti-predator behaviors) [1,2,3]. That is, juvenile prey that escape from predation and become adult can counterattack juvenile predators, and adults just kill the juveniles but do not consume them, which can serve to reduce future predation risk [4]. Therefore, it is very important to evaluate the cyclic dominance for predator–prey interactions when anti-predator behaviors occur.

The dynamical relationship between predators and their preys has been considered in depth in a mass of studies [5,6,7,8,9,10]. In these studies, both predators and preys are assumed to be homogeneous. However, the anti-predator behavior of adult prey for juvenile predators indicates that we should take the age class structure of both predators and preys into consideration. Actually, it has been recognized for a long time that the age class structure of both predators and preys has a great influence on the dynamics of the interactions between these species [11, 12]. For example, Dörner et al. reported that Perca fluviatilis play an essential role in structuring the fish community because it is important in controlling the juvenile fish abundance [13].

Many studies investigated the dynamics of the predator–prey system with stage structure. In 1990, Aiello et al. studied a single specie model with stage structure [14]. Then the dynamics of the single population with stage structure, especially the bifurcation phenomenon, has been discussed in depth in [15,16,17]. By dividing the prey into multure and immature subpopulations, many researchers investigated the predator–prey system through autonomous models [18, 19], periodical models [20,21,22], delay differential equations [23,24,25] and partial differential equations [26]. Many other researchers also considered how the stage structure for predator [27,28,29,30,31] or for both predator and prey [32,33,34] affects the dynamics of the predator–prey system. Particularly, in 2000, Zhang et al. proposed a basic predator–prey model with stage structure for the prey [35], where they assume that the birth rate of the adult predator is proportional to its existing population. In 2015, Falconi et al. considered the carrying capacity of the habitat for the juvenile class [36]. Recently, Coast et al. introduced an exponential density dependence for the fecundity of adult preys [37] with a particular focus on the impact of culling predators for the prey. In this study, the authors have assumed the density of the predator population to be a constant.

Although several publications modeled the anti-predator behavior, the authors just assume that the prey is homogeneous to conduct the anti-predator behaviors [38, 39]. In this paper, to better understand the impact of anti-predator behavior on the dynamics of the predator–prey system, we proposed the following predator–prey model with stage structure for prey:

$$ \textstyle\begin{cases} \dot{x_{1}} = bx_{2}\exp (-ax_{2})-\beta _{1}x_{1}x_{3}-\gamma x_{1}-m _{J}x_{1}, \\ \dot{x_{2}} = \gamma x_{1}-\beta _{2}x_{2}x_{3}-m_{A}x_{2}, \\ \dot{x_{3}} = k\beta _{1}x_{1}x_{3}+k\beta _{2}x_{2}x_{3}-\eta x_{2}x _{3}-\delta x_{3}. \end{cases} $$

(1)

Here, $x_{1}$, $x_{2}$ and $x_{3}$ represent the density of the juvenile prey, the adult prey and the predator, respectively. η is the rate of anti-predator behavior of adult prey to the predator population, the term $be^{-ax_{2}}$ is the exponential density dependence birth rate of the prey, $m_{A}$ and $m_{J}$ denote the death rate of the adult prey and the juvenile prey, respectively, γ represents the maturation rate of the juvenile prey, δ is the natural death rate of the predator and k is transformation rate from the prey to the predator, $\beta _{1}$ and $\beta _{2}$ are the predation rates of the predator to the juvenile prey and adult prey, respectively.

The paper is organized as follows. In Sect. 2, we mainly discuss the existence and stability of the equilibria, especially, discuss the existence and stability of the equilibria for two subcases that the predator only feed on the juvenile prey or the adult prey. In Sect. 3, we prove that the proposed model can undergo a forward bifurcation or a backward bifurcation, which are the common bifurcations in epidemic systems discussed in [40, 41]. In Sect. 4, we provide some numerical simulations to show how the anti-predator behavior affects the dynamics of the predator–prey system. In Sect. 5, we make the conclusion and discussion of this study.

2 The existence and stability for the equilibria

In this section, we analyze the dynamics of system (1) by investigating the existence and the stability of the equilibria. It is easy to see that system (1) always has a trivial equilibrium $E_{0}(0,0,0)$. Meanwhile, if $R_{0}>1$ holds true, then system (1) has a predator-extinction equilibrium $\hat{E}(\hat{x} _{1},\hat{x}_{2},0)$, where

$$ \hat{x}_{1}=-\frac{m_{A}}{a\gamma }\ln \biggl(\frac{1}{R_{0}} \biggr), \quad \quad \hat{x}_{2}=-\frac{1}{a}\ln \biggl( \frac{1}{R_{0}} \biggr), \quad \quad R_{0}=\frac{b\gamma }{m_{A}(\gamma +m_{J})}. $$

The Jacobian matrix of system (1) gives

$$J=\left(\begin{array}{ccc}-{\beta }_1{x}_3-\gamma -m_J& b(1-a{x}_2)exp(-a{x}_2)& -{\beta }_1{x}_1\\ \gamma & -{\beta }_2{x}_3-m_A& -{\beta }_2{x}_2\\ k{\beta }_1{x}_3& k{\beta }_2{x}_3-\eta x_3& k{\beta }_1{x}_1+k{\beta }_2{x}_2-\eta x_2-\delta \end{array}\right).$$

Thus, the Jacobian matrix at the trivial equilibrium $E_{0}$ is

$$J{|}_{E_0}=\left(\begin{array}{ccc}-\gamma -m_J& b& 0\\ \gamma & -m_A& 0\\ 0& 0& -\delta \end{array}\right).$$

That is, the characteristic equation of system (1) at the trivial equilibrium $E_{0}$ is

$$ (\lambda +\delta ) \bigl[\lambda ^{2}+(\gamma +m_{A}+m_{J})\lambda +m _{A}(\gamma +m_{J})-b\gamma \bigr]=0. $$

(2)

It follows from Eq. (2) that if $0< R_{0}<1$, the equilibrium $E_{0}$ is locally asymptotically stable; if $R_{0}>1$, then the equilibrium $E_{0}$ is unstable.

Similarly, we can calculate the Jacobian matrix of system (1) at the predator-extinction equilibrium as

$$J{|}_{\hat{E}}=\left(\begin{array}{ccc}-\gamma -m_J& \frac{b}{R_0}[1+ln(\frac{1}{R_0})]& \frac{{\beta }_1{m}_A}{a\gamma }ln(\frac{1}{R_0})\\ \gamma & -m_A& \frac{{\beta }_2}{a}ln(\frac{1}{R_0})\\ 0& 0& -ln(\frac{1}{R_0})(k\frac{{\beta }_1{m}_A}{a\gamma }+k\frac{{\beta }_2}{a}-\frac{\eta }{a})-\delta \end{array}\right).$$

Thus, the characteristic equation of system (1) at Ê is $\varPhi \varPsi =0$, where

$$ \begin{gathered} \varPhi =\lambda +\ln \biggl( \frac{1}{R_{0}} \biggr) \biggl(k\frac{\beta _{1}m_{A}}{a\gamma }+ k\frac{\beta _{2}}{a}- \frac{\eta }{a} \biggr)+ \delta , \\ \varPsi =\lambda ^{2}+(\gamma +m_{A}+m_{J}) \lambda +m_{A}(m_{J}+\gamma )-\frac{ \gamma b}{R_{0}} \biggl[1+ \ln \biggl(\frac{1}{R_{0}} \biggr) \biggr]. \end{gathered} $$

(3)

It is easy to verify that if $R_{0}>1$,

$$ m_{A}(m_{J}+\gamma )-\frac{\gamma b}{R_{0}} \biggl[1+\ln \biggl(\frac{1}{R _{0}} \biggr) \biggr]>0, $$

so the equilibrium Ê has two eigenvalues with negative real parts, which are the two roots of the following equation:

$$ \lambda ^{2}+(\gamma +m_{A}+m_{J})\lambda +m_{A}(m_{J}+\gamma )-\frac{ \gamma b}{R_{0}} \biggl[1+\ln \biggl(\frac{1}{R_{0}} \biggr) \biggr]=0. $$

The other eigenvalue of Ê gives

$$ \lambda _{1}=-\ln \biggl(\frac{1}{R_{0}} \biggr) \biggl(k \frac{\beta _{1}m_{A}}{a\gamma }+ k\frac{\beta _{2}}{a}-\frac{\eta }{a} \biggr)- \delta . $$

Obviously, if the inequality

$$ k\frac{\beta _{1}m_{A}}{a\gamma }+ k\frac{\beta _{2}}{a}- \frac{\eta }{a}< 0\quad \biggl( \text{i.e.,}\quad \eta >\frac{k\beta _{1}m _{A}+k\beta _{2}\gamma }{\gamma }\doteq \eta ^{*} \biggr) $$

holds true, then we have $\lambda _{1}<0$, thus the equilibrium Ê is locally asymptotically stable. Denote

$$ R^{*}=\exp {\frac{a\delta \gamma }{k\beta _{1}m_{A}+ k\beta _{2}\gamma -\eta \gamma }}. $$

Further, if $\eta <\eta ^{*}$ holds true, then $\lambda _{1}<0$ and $\lambda _{1}>0$ for $R_{0}< R^{*}$ and $R_{0}>R^{*}$, respectively, which means that if $\eta <\eta ^{*}$, then the equilibrium Ê is locally asymptotically stable and unstable provided that $1< R_{0}< R ^{*}$ and $R_{0}>R^{*}$, respectively. Thus, we have derived the following result.

Theorem 2.1

System (1) always has a trivial equilibrium $E_{0}$, and it is locally asymptotically stable when $R_{0}<1$. Meanwhile, if the inequality $R_{0}>1$ holds true, there exists a predator-extinction equilibrium Ê. Furthermore, if $\eta >\eta ^{*}$ (or $\eta <\eta ^{*}$ and $1< R_{0}< R^{*}$), then the predator-extinction equilibrium Ê is locally asymptotically stable.

Next, we mainly consider the existence and the stability of the positive equilibrium of system (1). We first consider two special cases that the predator only feed on the juvenile prey or the adult prey. If we assume that the predator only feeds on the adult prey, i.e. $\beta _{1}=0$, then system (1) becomes

$$ \textstyle\begin{cases} \dot{x_{1}} = bx_{2}\exp (-ax_{2})-\gamma x_{1}-m_{J}x_{1}, \\ \dot{x_{2}} = \gamma x_{1}-\beta _{2}x_{2}x_{3}-m_{A}x_{2}, \\ \dot{x_{3}} = k\beta _{2}x_{2}x_{3}-\eta x_{2}x_{3}-\delta x_{3}. \end{cases} $$

(4)

The positive equilibrium of system (4) $E_{1}^{*}(x^{*}_{11},x ^{*}_{12},x^{*}_{13})$ gives

$$\begin{aligned}& x^{*}_{11}=\frac{\delta b\exp [-\frac{a\delta }{k\beta _{2}-\eta } ]}{(k\beta _{2}-\eta )(\gamma +m_{J})},\quad \quad x^{*}_{12}=\frac{ \delta }{k\beta _{2}-\eta }, \\& x^{*}_{13}=\frac{\gamma b\exp [-\frac{a\delta }{k\beta _{2}-\eta } ]-m_{A}(\gamma +m_{J})}{ \beta _{2}(\gamma +m_{J})}. \end{aligned}$$

Therefore, if $\eta < k\beta _{2}$ and $R_{0}>\exp ({\frac{a\delta }{k \beta _{2}-\eta }})$ hold true, then system (4) has a unique positive equilibrium $E_{1}^{*}(x^{*}_{11},x^{*}_{12},x^{*}_{13})$. By easy calculation we can obtain the characteristic equation of system (4) at the positive equilibrium $E_{1}^{*}$,

$$ \lambda ^{3}+A_{1}\lambda ^{2}+B_{1} \lambda +C_{1}=0, $$

(5)

where

$$\begin{aligned}& A_{1}=\gamma +m_{J}+m_{A}+\beta _{2}x_{13}^{*},\quad\quad C_{1}=\beta _{2}(k\beta _{2}-\eta ) (\gamma +m_{J})x_{12}^{*}x_{13}^{*}, \\& B_{1}=\beta _{2}(k\beta _{2}-\eta )x_{12}^{*}x_{13}^{*}+(\gamma +m_{J}) \bigl( \beta _{2}x_{13}^{*}+m_{A} \bigr)-\gamma b(1-ax_{2})\exp \bigl(-ax_{12}^{*} \bigr). \end{aligned}$$

According to the Hurwitz criterion, to make the positive equilibrium stable, we just need $A_{1}>0$, $C_{1}>0$ and $A_{1}B_{1}-C _{1}>0$. It is easy to see that $A_{1}>0$ and $C_{1}>0$ always hold true, so we just need to verify $A_{1}B_{1}-C_{1}>0$, and we note that

$$\begin{aligned} A_{1}B_{1}-C_{1}={} &\bigl(\gamma +m_{J}+\beta _{2}x^{*}_{13}+m_{A} \bigr) \bigl(\bigl(ax_{12} ^{*}-1\bigr)b\gamma \exp \bigl({-ax^{*}_{12}}\bigr)+(\gamma +m_{J}) \bigl( \beta _{2}x^{*} _{13} \\ &{}+m_{A}\bigr)\bigr)+\delta \beta _{2}x^{*}_{13} \bigl(\beta _{2}x^{*}_{13}+m_{A}\bigr) \\ ={} &\beta _{2}x^{*}_{13}\bigl(\beta _{2}x^{*}_{13}+m_{A}\bigr) (\gamma +m_{J}+ \delta )+(\gamma +m_{J}+m_{A}) (\gamma +m_{J}) \\ &{}\times\bigl(\beta _{2}x^{*}_{13}+m_{A} \bigr)+b\gamma \bigl(ax_{12}^{*}-1\bigr) \bigl(\gamma +m_{J}+ \beta _{2}x^{*}_{13}+m_{A} \bigr)\exp \bigl({-ax_{12}^{*}}\bigr) \\ ={} &\beta _{2}x^{*}_{13}\bigl(\beta _{2}x^{*}_{13}+m_{A}\bigr) (\gamma +m_{J}+ \delta )+(\gamma +m_{J}+m_{A}) (\gamma +m_{J}) \\ &{}\times\bigl(\beta _{2}x^{*}_{13}+m_{A} \bigr)+\bigl(ax_{12}^{*}-1\bigr) \bigl(\gamma +m_{J}+\beta _{2}x ^{*}_{13}+m_{A} \bigr) \\ &{}\times\bigl(\beta _{2}x^{*}_{13}(\gamma +m_{J})+m_{A}(\gamma +m_{J})\bigr) \\ ={} &ax_{12}^{*}\bigl(\gamma +m_{J}+\beta _{2}x^{*}_{13}+m_{A}\bigr) \bigl(\beta _{2}x ^{*}_{13}(\gamma +m_{J})+m_{A}( \gamma +m_{J})\bigr) \\ &{}+\beta _{2}\delta x^{*}_{13}\bigl(\beta _{2}x^{*}_{13}+m_{A}\bigr). \end{aligned}$$

(6)

Obviously, the inequality $A_{1}B_{1}-C_{1}>0$ is always true since $x^{*}_{13}>0$ and $x_{12}^{*}>0$. That is, the positive equilibrium $E_{1}^{*}$ is always locally asymptotically stable whenever it exists. Note that, when $\beta _{1}=0$, we have $R_{1}^{*}=\exp (a\delta /(k \beta _{2}-\eta ) )$ and $\eta _{1}^{*}=k\beta _{2}$.

Analogously, assume that the predator just feed on the juvenile prey, system (1) becomes

$$ \textstyle\begin{cases} \dot{x_{1}} = bx_{2}\exp (-ax_{2})-\beta _{1}x_{1}x_{3}-\gamma x_{1}-m _{J}x_{1}, \\ \dot{x_{2}} = \gamma x_{1}-m_{A}x_{2}, \\ \dot{x_{3}} = k\beta _{1}x_{1}x_{3}-\eta x_{2}x_{3}-\delta x_{3}, \end{cases} $$

(7)

and the positive equilibrium of system (7) $E_{2}^{*}(x^{*} _{21},x^{*}_{22},x^{*}_{23})$ gives

$$\begin{aligned}& x^{*}_{21}=\frac{\delta m_{A}}{k\beta _{1}m_{A}-\eta \gamma }, \quad \quad x^{*}_{22}=\frac{\gamma \delta }{k\beta _{1}m_{A}-\eta \gamma }, \\& x^{*}_{23}=\frac{b\gamma }{\beta _{1} m_{A}}\exp \biggl({- \frac{a\gamma \delta }{k\beta _{1}m_{A}-\eta \gamma }} \biggr)-\frac{\gamma +m_{J}}{ \beta _{1}}. \end{aligned}$$

If the inequalities

$$ \eta < \frac{k\beta _{1}m_{A}}{\gamma }\quad \text{and}\quad R_{0}> \exp \biggl( \frac{a\gamma \delta }{m_{A} k\beta _{1}-\eta \gamma } \biggr) $$

hold true, then there is a unique positive equilibrium $E_{2}^{*}$ for system (7). Through a similar process, we can prove that the positive equilibrium $E_{2}^{*}$ is always locally stable whenever it exists. As $\beta _{2}=0$,

$$ R_{2}^{*}=\exp \biggl(\frac{a\gamma \delta }{m_{A} k\beta _{1}-\eta \gamma } \biggr)\quad \text{and}\quad \eta _{2}^{*}=\frac{k\beta _{1}m _{A}}{\gamma }. $$

Therefore, the dynamics of system (4) and system (7) is concluded as follows.

Theorem 2.2

There is always a trivial equilibrium $E_{0}$ of system (4) (or system (7)) which is locally asymptotically stable if $R_{0}<1$. Meanwhile, if the inequality $R_{0}>1$ holds true, system (4) (or system (7)) has a predator-extinction equilibrium Ê, which is locally stable if any one of the following conditions holds true:

$$\begin{aligned}& (\mathrm{a})\quad 1< R_{0}< R_{1}^{*}\quad \textit{and} \quad \eta < \eta _{1}^{*}; \\& (\mathrm{b})\quad 1< R_{0}< R_{2}^{*}\quad \textit{and} \quad \eta < \eta _{2}^{*}; \\& (\mathrm{c})\quad \eta >\eta _{1}^{*}; \\& (\mathrm{d})\quad \eta >\eta _{2}^{*}. \end{aligned}$$

Furthermore, if $R_{0}>R_{1}^{*}$ and $\eta <\eta _{1}^{*}$ (or $R_{0}>R_{2}^{*}$ and $\eta <\eta _{2}^{*}$), then there is a unique positive equilibrium $E_{1}^{*}$ for system (4) (or $E_{2}^{*}$ for system (7)) which is locally asymptotically stable.

Different from subsystem (4) and subsystem (7), the existence of the positive equilibrium for system (1) becomes very complex. Similarly the positive equilibrium of system (1) $E^{*}(x_{1}^{*},x_{2}^{*},x_{3}^{*})$ satisfies

$$\begin{aligned} &bx_{2}^{*}e^{-ax_{2}^{*}}-\beta _{1}x_{1}^{*}x_{3}^{*}- \gamma x_{1} ^{*}-m_{J}x_{1}^{*}=0, \end{aligned}$$

(8)

$$\begin{aligned} &\gamma x_{1}^{*}-\beta _{2}x_{2}^{*}x_{3}^{*}-m_{A}x_{2}^{*}=0, \end{aligned}$$

(9)

$$\begin{aligned} &k\beta _{1}x_{1}^{*}+k\beta _{2}x_{2}^{*}- \eta x_{2}^{*}-\delta =0. \end{aligned}$$

(10)

From Eq. (9) and Eq. (10), we can easily get

$$ x_{1}^{*}= \frac{\delta +\eta x_{2}^{*}-k\beta _{2}x_{2}^{*}}{k\beta _{1}},\quad \text{and}\quad x_{3}^{*}=\frac{\gamma (\delta +\eta x _{2}^{*}-k\beta _{2} x_{2}^{*})}{k\beta _{1} \beta _{2}x_{2}^{*}}-\frac{m _{A}}{\beta _{2}}. $$

Let $x_{1}^{*}>0$, we see that if $\eta < k\beta _{2}$ and $\eta >k\beta _{2}$,

$$ x_{2}^{*}< -\frac{\delta }{\eta -k\beta _{2}}\quad \text{and}\quad x _{2}^{*}>-\frac{\delta }{\eta -k\beta _{2}}, $$

respectively. Similarly, to guarantee the positivity of $x_{3}^{*}$, we should have

$$ x_{2}^{*}< \frac{\gamma \delta }{k\beta _{2}\gamma +k\beta _{1} m_{A}- \gamma \eta }\quad \text{and}\quad x_{2}^{*}>\frac{\gamma \delta }{k \beta _{2}\gamma +k\beta _{1} m_{A}-\gamma \eta } $$

provided that $\eta <\eta ^{*}$ and $\eta >\eta ^{*}$, respectively. Since all the parameters are positive,

$$ \frac{\delta }{k\beta _{2}-\eta }>\frac{\gamma \delta }{k\beta _{2} \gamma +k\beta _{1} m_{A}-\eta \gamma }>0 $$

provided that $\eta < k\beta _{2}$. As a conclusion, to guarantee the positivity of both $x_{1}^{*}$ and $x_{3}^{*}$, we consider two cases, one case is that if $\eta <\eta ^{*}$ holds true, we need

$$ 0< x_{2}^{*}< \frac{\gamma \delta }{k\beta _{2}\gamma +k\beta _{1} m_{A}- \eta \gamma }. $$

The other case is that if $\eta >\eta ^{*}$ holds true, we need $x_{2}^{*}>0$. Then substituting $x_{1}^{*}$ and $x_{3}^{*}$ into Eq. (8), we have the following equation for $x_{2}^{*}$:

$$ A{x_{2}^{*}}^{2}\exp \bigl(-ax_{2}^{*} \bigr)+Bx_{2}^{*}+C{x_{2}^{*}}^{2}+D=0, $$

(11)

where

$$\begin{aligned}& A=k^{2}b\beta _{1}\beta _{2}, \qquad B=\delta (k \beta _{1}m_{A}-k\beta _{2}m_{J}-2\eta \gamma +k\beta _{2} \gamma ), \\& C=(\eta -k\beta _{2}) (k\beta _{1}m_{A}-k\beta _{2}m_{J}-\eta \gamma ), \qquad D=-\gamma \delta ^{2}. \end{aligned}$$

As we can see, Eq. (11) is a transcendental equation with high nonlinearity, thus it is difficult to calculate the roots of Eq. (11). Let

$$ F(x)=Ax^{2}\exp ({-ax})+Bx+Cx^{2}+D. $$

(12)

The changing curve of the function $F(x)$ with different values of η are plotted in Fig. 1. It is easy to verify that system (1) may have one positive equilibrium (shown in Fig. 1(A)) or two positive equilibria (shown in Fig. 1(B)) or no positive equilibrium (shown in Fig. 1(C)), respectively. This implies that system (1) may undergo the equilibrium bifurcation. Therefore, in the next section, we mainly prove that system (1) can undergo the backward bifurcation or the forward bifurcation.

Figure 1

The curve of the function $F(x)$ with $\eta =0.01$ in (A), $\eta =0.04$ in (B) and $\eta =0.18$ in (C). The other parameter values are $b=50$, $\delta =2$, $a=0.22$, $\beta _{1}=0.2$, $\beta _{2}=0.1$, $k=0.4$, $\gamma =0.45$, $m _{A}=0.45$, $m_{J}=0.8$. Note that here $\eta ^{*}=0.12$

3 Bifurcation analysis

From the last section, we find that if $\eta <\eta ^{*}$, there can be $\lambda _{1}=0$, which means that the predator-extinction equilibrium has a single zero eigenvalue. Thus, system (1) may undergo a bifurcation at $R_{0}=R^{*}$. Since $R_{0}$ is inconvenient to use directly as a bifurcation parameter, here we take the parameter δ as a bifurcation parameter. There exists a $\delta ^{*}$ such that $R_{0}=R^{*}$ if we fix other parameters. Definitely, we have $R_{0}>R^{*}$ if $\delta <\delta ^{*}$ and $R_{0}< R^{*}$ if $\delta > \delta ^{*}$. Then the Jacobian matrix at $(\hat{E},\delta ^{*})$ is

$$J{|}_{(\hat{E},{\delta }^{\ast })}=\left(\begin{array}{ccc}-\gamma -m_J& \frac{b}{R_0}(1+ln(\frac{1}{R_0}))& \frac{{\beta }_1{m}_A}{a\gamma }ln(\frac{1}{R_0})\\ \gamma & -m_A& \frac{{\beta }_2}{a}ln(\frac{1}{R_0})\\ 0& 0& 0\end{array}\right).$$

It is easy to see that $V=(v_{1},v_{2},v_{3})=(0,0,1)$ is a left eigenvector of the matrix $J\vert _{(\hat{E},\delta ^{*})}$. Let $W=(w_{1},w _{2},w_{3})$ with $w_{3}=1$ be a right eigenvector of the matrix $J\vert _{(\hat{E},\delta ^{*})}$. Then,

$$ \textstyle\begin{cases} -(\gamma +m_{J})w_{1}+\frac{b}{R_{0}} (1+\ln ( \frac{1}{R_{0}} ) )w_{2} +\frac{\beta _{1}m_{A}}{a\gamma } \ln (\frac{1}{R_{0}} )=0, \\ \gamma w_{1}-m_{A}w_{2}+\frac{\beta _{2}}{a}\ln (\frac{1}{R_{0}} )=0. \end{cases} $$

(13)

Solving the above equation we obtain

$$ w_{1}=-\frac{\beta _{1} m_{A}}{a\gamma (\gamma +m_{J})}-\frac{\beta _{2}}{a\gamma } \biggl(1+\ln \biggl( \frac{1}{R_{0}} \biggr) \biggr), \quad \quad w_{2}=- \frac{1}{am _{A}(\gamma +m_{J})}\bigl(\beta _{1} m_{A}+\beta _{2}(\gamma +m_{J})\bigr). $$

Definitely, we have $VW=1$. Furthermore,

$$\begin{aligned}& \frac{\partial ^{2}f_{3}}{\partial x_{1}\partial x_{3}}\bigg\vert _{ (\hat{E},\delta ^{*} )}=k\beta _{1}, \quad\quad \frac{\partial ^{2}f_{3}}{\partial x_{2}\partial x_{3}}\bigg\vert _{ (\hat{E},\delta ^{*} )}=k\beta _{2}-\eta , \\& \frac{\partial ^{2}f_{3}}{\partial x_{3}\partial x_{1}}\bigg\vert _{ (\hat{E},\delta ^{*} )}=k\beta _{1}, \quad\quad \frac{\partial ^{2}f_{3}}{\partial x_{3}\partial x_{2}}\bigg\vert _{ (\hat{E},\delta ^{*} )}=k\beta _{2}-\eta . \end{aligned}$$

Thus,

$$ \begin{aligned} \varLambda &=\frac{1}{2} \biggl(w_{1}w_{3} \frac{\partial ^{2}f_{3}}{ \partial x_{1}\partial x_{3}}+w_{2}w_{3}\frac{\partial ^{2}f_{3}}{ \partial x_{2}\partial x_{3}}+w_{3}w_{1} \frac{\partial ^{2}f_{3}}{ \partial x_{3}\partial x_{1}}+w_{3}w_{2}\frac{\partial ^{2}f_{3}}{ \partial x_{3}\partial x_{2}} \biggr) \bigg\vert _{ (\hat{E},\delta ^{*} )} \\ &=k\beta _{1}w_{1}+(k\beta _{2}-\eta )w_{2}. \end{aligned} $$

Furthermore,

$$ \kappa =v_{3}w_{3}\frac{\partial ^{2} f_{3}}{\partial x_{3}\partial \delta }=-1\neq 0. $$

Thus, according to Theorem 4 in [42], we have the following conclusion.

Theorem 3.1

If $\varLambda <0$, then there is a stable positive equilibrium near Ê for $\delta ^{*}-\varepsilon _{1}< \delta <\delta ^{*}$ and system (1) undergoes a forward bifurcation at $\delta =\delta ^{*}$. If $\varLambda >0$, then there is an unstable positive equilibrium near Ê for $\delta ^{*}<\delta < \delta ^{*}+\varepsilon _{2}$ and system (1) undergo a backward bifurcation at $\delta =\delta ^{*}$.

Remark 3.2

If $\beta _{1}=0$ (or $\beta _{2}=0$) holds true, then

$$ \varLambda =-\frac{\beta _{2}(k\beta _{2}-\eta )}{am_{A}}< 0\quad \biggl(\text{or}\quad \varLambda =- \frac{\beta _{1}}{a\gamma (\gamma +m_{J})}(k\beta _{1}m _{A}-\eta \gamma )< 0\biggr) $$

for $\eta <\eta _{1}^{*}$ (or $\eta <\eta _{2}^{*}$). Therefore, the two systems (4) and (7) undergo a forward bifurcation at $\delta =\delta ^{*}$.

Remark 3.3

Noting that the trivial equilibrium $E_{0}$ also has s single zero eigenvalue at ${R_{0}=1}$. Taking b as a bifurcation parameter, there is a $b^{*}$ such that $R_{0}>1$ for $b>b^{*}$ and $R_{0}<1$ for $b< b^{*}$. Then, we can calculate that

$$ W_{1}=(1,\gamma /m_{A},0), \quad\quad V_{1}= \biggl(\frac{m_{A}^{2}}{m_{A} ^{2}+b\gamma },\frac{bm_{A}}{m_{A}^{2}+b\gamma },0 \biggr) $$

are a right eigenvalue and a left eigenvalue of the matrix $J\vert _{(E _{0},b^{*})}$, respectively, which satisfy $V_{1}W_{1}=1$. And we have

$$ \varLambda =-\frac{ab\gamma ^{2}}{m_{A}^{2}+\gamma b}< 0,\quad \text{and}\quad \kappa = \frac{\gamma m_{A}}{m_{A}^{2}+\gamma b} \neq 0. $$

Therefore, system (1) undergoes a forward bifurcation at $R_{0}=1$.

4 Numerical simulations

In this section, we mainly analysis the dynamics of system (1) through numerical simulations by using Xpp-Auto. In the last section, we proved that system (1) can undergo the backward bifurcation and the forward bifurcation if we choose δ as a bifurcation parameter. In Figs. 2(A) and (B), we set $b=50$ and $b=10$, respectively, and fix all the other parameter values as

$$\begin{aligned}& \eta =0.01,\quad\quad a=0.22,\quad\quad \beta _{1}=0.2,\quad\quad \beta _{2}=0.1,\quad\quad k=0.4,\quad\quad \gamma =0.45,\\& m _{A}=0.45,\quad\quad m_{J}=0.8. \end{aligned}$$

Let $R_{0}=R^{*}$, we can easily calculate $\delta ^{*}=1.844$ for Fig. 2(A) and $\delta ^{*}=1.039$ for Fig. 2(B). Then, we have $\varLambda =0.107>0$ in Fig. 2(A) and $\varLambda =-0.023<0$ in Fig. 2(B). Therefore, Figs. 2(A) and (B) show that system (1) undergoes a backward bifurcation and a forward bifurcation, respectively.

Figure 2

One parameter bifurcation diagram of δ. ‘SN’ is the saddle-node bifurcation. The dash curve represents unstable equilibrium while the solid curve represent stable equilibrium. Here we set $b=50$ for (A) and $b=10$ for (B). The other parameter values are: $\eta =0.01$, $a=0.22$, $\beta _{1}=0.2$, $\beta _{2}=0.1$, $k=0.4$, $\gamma =0.45$, $m_{A}=0.45$, $m_{J}=0.8$

Next, we mainly investigate how the anti-predator behavior affects the dynamics of system (1). In Figs. 3(A)–(C), we plot the bifurcation diagram of the equilibria with respect to η. It follows from Figs. 3(A)–(C) that when η is relatively small (i.e. below the backward bifurcation point), a stable positive equilibrium exists, particularly, as the parameter η increases, the levels of both the juvenile prey and the adult prey increase while the population of the predator at the steady state decreases. As η continuously increases, the positive equilibrium becomes bistable with the predator-extinction equilibrium and the population of the predator at the steady state further decreases. Furthermore, when η exceeds the saddle-node bifurcation point, the stable positive equilibrium disappears and the predator will finally die out due to the high pressure of anti-predator behaviors.

Figure 3

(A)–(C) One parameter bifurcation diagram with respect to η, here $b=90$; (D)–(F) One parameter bifurcation diagram with respect to b, here $\eta =0.01$. ‘SN’ represents saddle-node bifurcation, ‘BB’ means backward bifurcation and ‘FB’ is for forward bifurcation. Also, the dash curve is unstable equilibrium and the solid curve is the stable equilibrium. Here, $\delta =2$ and the other parameter values are the same as those in Fig. 2

Because $R_{0}$ is independent on the parameters δ and η, they cannot change the stability of the trivial equilibrium. Thus, we then let b be a bifurcation parameter and fix $\delta =2$ and $\eta =0.01$, Figs. 3(D)–(F) show that system (1) first undergoes a forward bifurcation at $b=1.25$ with the trivial equilibrium losing its stability and a stable predator-extinction equilibrium emerging. When b continues increases to 43.56, a saddle-node bifurcation occurs and then there exist an unstable positive equilibrium and a stable positive equilibrium (which is bistable with the predator-extinction equilibrium). Then, system (1) undergoes a backward bifurcation (transcritical bifurcation) at $b=68.2$, here, the predator-extinction equilibrium loses its stability while the unstable positive equilibrium disappears. As we can see from Figs. 3(D)–(F), increasing b can increase the population of the prey. However, different from the anti-predator behavior, increase the birth rate of the prey can also increase the population of the predator.

Then, we plot the two parameter bifurcation diagram of η with respect to b in Fig. 4. In region $\omega _{1}$, there is only a trivial equilibrium which is stable. In region $\omega _{2}$, a stable predator-extinction equilibrium appears while the trivial equilibrium loss its’ stability. If the parameters cross the SN curve from region $\omega _{2}$ to region $\omega _{3}$, system (1) undergoes a saddle-node bifurcation, and there are two positive equilibria, one of which is bistable with the predator-extinction equilibrium. If the parameters further cross the BB curve from region $\omega _{3}$ to region $\omega _{4}$, then system (1) undergoes a backward bifurcation. As we can see from Fig. 4, when the birth rate of the prey is small (i.e. $0< b<38$), the anti-predator behavior cannot affect the dynamics of system, while the predator will always die out. However, when the birth rate b is high, the predator and the prey can always coexist if the degree of the anti-predator behavior is relatively low (corresponding to the region $\omega _{4}$), and if the degree of the anti-predator behavior becomes high, the positive equilibrium bistable with the predator-extinction equilibrium, that is, depending on the initial conditions, the predator can coexist with the prey or die out. If the degree of the anti-predator behavior is higher, then the predator will always die out as well.

Figure 4

Two parameter bifurcation diagram of b and η. ‘SN’ represents saddle-node bifurcation, ‘BB’ means backward bifurcation and ‘FB’ is for forward bifurcation. The other parameter values are the same as those in Fig. 3

It follows from Figs. 3(A)–(C) that if $\eta \in (0.0171,0.1095)$, the positive equilibrium and the predator-extinction equilibrium are bistable. Thus, in Fig. 5(A)–(B), we plot the basin of attraction of system (1) with respect to $x_{1}$ and $x_{3}$. When the initial values of $x_{1}$ and $x_{3}$ are located at the red region, the solution trajectories tend to the predator-extinction equilibrium, while they go to the positive equilibrium when the initial values of $x_{1}$ and $x_{3}$ located at the green region. As we can see, if we fix the initial condition of $x_{3}$ between 6.5 and 7 in Fig. 5(A), then there is a critical value of $x_{1}$ determining the stability of the positive equilibrium. This means that if the predator is an endangered species, we can make it persistent by introducing more juvenile preys. Combining Figs. 5(A) and (B), we find that increasing η will enlarge the stable region of the predator-extinction equilibrium, which means the possibility of the coexistence of the predator and prey will decrease. Correspondingly, in Figs. 5(C)–(D), with different values of η, we plot the solution trajectories of system (1). It is easy to see that the higher degree of the anti-predator behavior, the more numerous the population of the prey at the steady state is and the less the population of the predator at the steady state is.

Figure 5

The basin of attraction domain of system (1) when $\eta =0.09$ in (A) and $\eta =0.1$ in (B), and we fix the initial value of $x_{2}$ as 5. (C) and (D) Solutions of system (1) with the initial condition being $(6,5,10)$. Here, $b=90$ and the other parameter values are the same as those in Fig. 3

5 Conclusion and discussion

This paper proposed a predator–prey model with stage structure for prey such that the adult prey can counterattack their predators. Firstly, the existence and the stability of the equilibria was discussed through exploring the characteristic equations. It found that there is always a trivial equilibrium which is stable when $R_{0}<1$ and becomes unstable if $R_{0}>1$. Correspondingly, there emerges a predator-extinction equilibrium when $R_{0}>1$. Meanwhile, it is verified that system (1) undergoes a forward bifurcation at $R_{0}=1$. If $\eta <\eta ^{*}$ holds true, the predator-extinction equilibrium has single zero eigenvalue while the other two eigenvalues have negative real parts at $R_{0}=R^{*}$. Based on this condition, system (1) may undergo either a forward bifurcation or a backward bifurcation by choosing the death rate of the predator as a bifurcation parameter. Furthermore, we also discussed the existence and stability of the equilibria for two special cases. The results show that if the predator only feeds on one age class, the backward bifurcation could not happen, and the system can have at most one positive equilibrium, which is stable whenever it exists.

Numerical analysis shows that the predator can coexist with the prey in the term of a stable positive equilibrium if the rate of anti-predator behavior is relatively small, as shown in Figs. 3(A)–(C). As expected, the anti-predator behavior is beneficial to the growth of both the juvenile and the adult prey population through inhibiting the growth of the predator population. Also, the anti-predator behavior can weaken the stability of the positive equilibrium, while it enhances the stability of the predator-extinction equilibrium in terms of increasing the attraction area. These results are in agreement with the main results obtained in [38]. Correspondingly, when η exceeds the backward bifurcation point, the positive equilibrium is bistable with the predator-extinction equilibrium. We then showed that anti-predator behavior can make the coexistence of the prey and predator less likely by shrinking the stable region of the positive equilibrium when it is bistable with the predator-extinction equilibrium. Moreover, if the prey can further improve their anti-predator behavior, the predator population would become extinct with the stable positive equilibrium disappearing. It should be noticed that the impact of the anti-predator behavior also depends on the characters of the predator–prey system. For example, if the birth rate of the adult prey is not enough to support the coexistence of the prey and the predator, then the anti-predator behavior would not affect their dynamics.

Our model uses the simple bilinear terms to represent the anti-predator behavior and the functional response of the predation. The dynamics of the predator–prey system can be very complex if we take the other Holling type functional response into consideration. Our model should also incorporate the seasonal factor and the delay effect between the reproduction and the predation of the predator when analyzing the impact of the anti-predator behavior. Addressing these issues needs more future work.