Skip to main content

Theory and Modern Applications

Analysis of a stochastic predator–prey system with fear effect and Lévy noise

Abstract

This paper studies a stochastic predator–prey model with Beddington–DeAngelis functional response, fear effect, and Lévy noise, where the fear is of prey induced by predator. First, we use Itô’s formula to prove the existence and uniqueness of a global positive solution and its moment boundedness. Next, sufficient conditions for the persistence and extinction of both species are given. We further investigate the stability in distribution of our system. Finally, we verify our analytical results by exhaustive numerical simulations.

1 Introduction

As an important part of ecology, population models have been widely studied and explored because of their rich dynamic behaviors, aiming to provide theoretical guidance for the protection, development, and utilization of biological resources [1]. Among the most important population models, the predator–prey model plays an important role in understanding the interactions of different species in unstable natural environments and has been extensively studied [24].

In the past few decades, most studies have only considered the direct effects of predator species on prey species, because the effect is easy to observe in any case. However, some actual data suggest that indirect effects of predator species on prey species also have a significant effect on population dynamics, and in some cases are even greater than direct predation [57]. Although we only observe direct predation behavior in nature, any prey will respond to perceived predation risk and therefore exhibit different types of anti-predation behaviors such as new selection of habitat, foraging behaviors, vigilance, and several psychological changes [811]. This activity against predators can be thought of as beneficial to adult survival, but the long-term cost will reduce the prey’s basic reproduction. In addition, when some prey are fully aware of impending predation risks, the choice of a new habitat can sometimes negatively affect an individual’s lifetime reproductive success. Poor habitat selection, i.e., poor quality of new habitat, affects not only adult reproduction, but also adult survival [4]. In 2011, experiments by Zanette et al. [12] showed that song sparrows (Melospiza Melodia) produced 40% less offspring due to fear of predators. This reduction is due to the effect of anti-predator behavior on both the birth rate and the survival of offspring. Thus, theoretical biologists and evolutionary ecologists have realized that it is not only the direct killing or shock that needs to be taken into account, but also the indirect shock or fear costs that must be incorporated into the model of predator and prey populations.

The behavioral characteristics of predation can be called functional responses, which play a dominant role in some complex dynamical behaviors, such as the steady states, bistability, periodic oscillations, chaos, and bifurcation phenomena. The functional response depends on many factors, such as different prey densities, the efficiency with which predators search for and kill prey, processing time, competition between predators, etc. The most typical prey-dependent functional responses are (i) Lotka–Volterra type; \(g(x)=px\) (linear); (ii) Holling type II, \(g(x)=\frac{px}{c+x}\) (concave increasing); (iii) Holling type III, \(g(x)=\frac{px^{2}}{c+x^{2}}\) (sigmoid increasing), where x is the density of prey, p is the maximum predation rate, and c is the half-satiation constant. However, prey-dependent functional responses fail to mimic predator–predator interference and face challenges from biological and physiological communities. Some biologists have argued that in many cases, especially when predators must search for food (and therefore must share or compete for food), the functional response in the predator–prey model should be predator-dependent, and numerous experiments and observations have shown this to be the case. In order to reconcile the theoretical and experimental views, Beddington and DeAngelis et al. considered a functional form of prey consumption rate and proposed the following form: \(g(x,y)=\frac{px}{ax+by+c}\), which is similar to Holling-type II functional response, but there is an extra term “by” in the denominator, which is interpreted as an interference between predators. The function \(g(x,y)=\frac{px}{ax+by+c}\) is called the Beddington–DeAngelis function response, where x and y represent the populations of prey and predator, respectively [1315].

Let \(x(t)\) be the prey density at time t and \(y(t)\) be the predator population density at time t. It is assumed that the predator preys on prey according to the functional response of Beddington–DeAngelis. In the presence of direct predation and fear factor, the prey population follows a logical growth, then we get the following differential system:

$$ \textstyle\begin{cases} \frac{d x}{dt} = \frac{\alpha x}{1+ky}-bx^{2}-\frac{\beta xy}{1+mx+ny}, \\ \frac{d y}{dt} = -cy-dy^{2}+ \frac{\theta \beta xy}{1+mx+ny}, \end{cases} $$
(1.1)

where α, b, c, d, θ, β, m, n are positive constants, α is the intrinsic growth rate of prey, c is the natural death rate of the predator, b and d respectively represent the mortality rates of the prey and predator species due to intraspecific competition between individuals. The constants m, n, θ, and β are the half-saturation constant of prey, the half-saturation constant of predator, the conversion rate of prey biomass to predator biomass, and the rate of predation, respectively [16, 17]. Function \(g(k, y)=\frac{1}{1+ky}\) represents the fear function, which stands for the cost of anti-predator defence of prey due to fear induced by predator, and k is the level of fear. The function \(g(k, y)\) has some special properties, as several field data show that the effect of fear reduces the reproductive process of prey species. For more details on the fear function \(g(k, y)\), see [18].

On the other hand, population dynamics in the real world is inevitably affected by environmental noise. To capture how environmental fluctuations affect system (1.1), it is necessary to take into account the stochastic differential equation model corresponding to the deterministic model (1.1) [1922]. Applying the technique used in [23] to include stochastic effects, we can obtain the stochastic version of model (1.1) as follows (the proof can be shown by the similar procedure as in [4]):

$$ \textstyle\begin{cases} dx(t) = \{\frac{\alpha x(t)}{1+ky(t)}-bx^{2}(t)- \frac{\beta x(t)y(t)}{1+mx(t)+ny(t)} \}\,dt+\sigma _{1}x(t)\,dB_{1}(t), \\ dy(t) = \{-cy(t)-dy^{2}(t)+ \frac{\theta \beta x(t)y(t)}{1+mx(t)+ny(t)} \}\,dt+\sigma _{2}y(t)\,dB_{2}(t), \end{cases} $$
(1.2)

where \(\sigma _{j}^{2}( j=1,2)\) stands for the intensity of white noise. Throughout this paper, let \((\Omega,\mathcal{F},\{\mathcal{F}_{t}\}_{t\geq 0},\mathbb{P})\) be a complete probability space with a filtration \(\{\mathcal{F}_{t}\}_{t\geq 0}\) satisfying the usual conditions (i.e. it is right continuous and \(\mathcal{F}_{0}\) contains all \(\mathbb{P}-\text{\textit{null}}\) sets) and \(B_{j}(t)(t\geq 0)( j=1,2)\) be a scalar standard Brownian motion defined on this probability space.

Furthermore, the population system may suffer sudden environmental shocks, e.g., earthquakes, hurricanes, epidemics, etc. However, the stochastic Lotka–Volterra model (1.2) cannot explain such phenomena [2426]. Just as Scheffer et al. [27] pointed out that studies on lakes, coral reefs, oceans, forests, and arid lands had shown that smooth change could be interrupted by sudden drastic switches to a contrasting state. For example, an abrupt climate change, whether warming or cooling, wetting or drying, could have lasting and profound impacts on natural ecosystems [28]. Zhou et al. [29] studied the dynamics of a stochastic SIS models with Lévy jumps and found that Lévy noise could suppress disease outbreak. Zhao et al. [30] analyzed a two-species Lotka–Volterra competition model in an impulsive polluted environment and showed that Lévy noise could change the persistence and extinction of each species significantly, and the Lévy noise was harmful to the survival of one of the species and was advantageous to the survival of the other. Therefore it is interesting and beneficial to treat differential systems with Lévy noise. In this paper, we develop a stochastic predator–prey model with Beddington–DeAngelis functional response, fear effect, and Lévy noise as follows:

$$ \textstyle\begin{cases} dx(t) = x(t) \{ [\frac{\alpha}{1+ky(t)}-bx(t)- \frac{\beta y(t)}{1+mx(t)+ny(t)} ]\,dt+\sigma _{1}\,dB_{1}(t) \}\\ \phantom{dx(t) = }{}+ \int _{\mathbb{Z}}\gamma _{1}(u)x(t^{-}) \tilde{N}(dt,du), \\ dy(t) = y(t) \{ [-c-dy(t)+ \frac{\theta \beta x(t)}{1+mx(t)+ny(t)} ]\,dt+\sigma _{2}\,dB_{2}(t) \}\\ \phantom{dy(t) =}{} +\int _{\mathbb{Z}}\gamma _{2}(u)y(t^{-})\tilde{N}(dt,du), \end{cases} $$
(1.3)

with initial data \(X(0)=(x(0),y(0))\in \mathbb{R}_{+}^{2}\), where \(x(t^{-})\) and \(y(t^{-})\) represent the left limit of \(x(t)\) and \(y(t)\), respectively; N is a Poisson counting measure with compensator Ñ and characteristic measure λ on a measurable subset \(\mathbb{Z}\) of \((0,\infty )\) with \(\lambda (\mathbb{Z})<\infty \) and \(\tilde{N}(dt,du)=N(dt,du)-\lambda (du)\,dt\). The parameter \(\gamma _{i}(u)\) characterizes the effect of Lévy noise on the ith species. For biological reasons, we suppose that \(1+\gamma _{i}(u)>0\), where \(\gamma _{i}(u)>0\) means the increasing of the species (e.g., planting) and \(-1<\gamma _{i}(u)<0\) means the decreasing of the species (e.g., harvesting and epidemics), \(u\in \mathbb{Z},i=1,2\). For more details of the Lévy jumps, see  [31]. The Brownian motion and Lévy jumps are assumed to be mutually independent.

Based on the above discussion, this paper intends to study the dynamical properties of (1.3). The organization of this paper is as follows. In the next section, we present some necessary notations and preliminary results of model (1.3). In Sect. 3, we prove the existence and boundedness of global positive unique solutions. Then, in Sect. 4, we establish some sufficient conditions for the persistence and extinction of both species. We further investigate the stability in distribution of our system in Sect. 5. Some numerical simulations are carried out in Sect. 6. We close the paper with a conclusion in Sect. 7.

2 Preliminaries

In this section, we begin with the introduction of the generalized Itô formula with jumps. Let \(x(t)\in \mathbb{R}^{n} \) be a solution of the following stochastic differential equation with Lévy jumps:

$$ dx(t) = F\bigl(x\bigl(t^{-}\bigr),t^{-}\bigr)\,dt+G\bigl(x \bigl(t^{-}\bigr),t^{-}\bigr)\,dB(t)+ \int _{\mathbb{Z}}H\bigl(x\bigl(t^{-}\bigr),t^{-},u \bigr) \tilde{N}(dt,du), $$
(2.1)

where \(F:\mathbb{R}^{n}\times \mathbb{R}_{+}\to \mathbb{R}^{n}\), \(G:\mathbb{R}^{n}\times \mathbb{R}_{+}\to \mathbb{R}^{n}\), and \(H: \mathbb{R}^{n}\times \mathbb{R}_{+}\times \mathbb{Z}\to \mathbb{R}^{n}\) are measurable functions. Given \(V\in C^{2,1}(\mathbb{R}^{n}\times \mathbb{R}_{+};\mathbb{R})\), we define the operator LV by

$$\begin{aligned} LV(x,t) ={}& V_{t}(x,t)+V_{x}(x,t)F(x,t)+\frac {1}{2} \mathrm{trace}\bigl[G^{T}(x,t)V_{xx}(x,t)G(x,t)\bigr] \\ &{}+ \int _{\mathbb{Z}} \bigl\{ V\bigl(x+H(x,t,u),t\bigr)-V(x,t)-V_{x}(x,t)H(x,t,u) \bigr\} \lambda (du), \end{aligned}$$
(2.2)

where \(V_{t}(x,t)=\frac{\partial V(x,t)}{\partial t}\), \(V_{x}(x,t)= (\frac{\partial V(x,t)}{\partial x_{1}},\ldots, \frac{\partial V(x,t)}{\partial x_{n}} )\), \(V_{xx}(x,t)= ( \frac{\partial ^{2} V(x,t)}{\partial x_{i} \partial x_{j}} )_{n \times n}\). Then the generalized Itô formula with jumps is as follows:

$$\begin{aligned} dV(x,t) ={}& LV(x,t)\,dt+V_{x}(x,t)G(x,t)\,dB(t) \\ &{}+ \int _{\mathbb{Z}}\bigl\{ V\bigl(x+H(x,t,u),t\bigr)-V(x,t) \bigr\} \tilde{N}(dt,du). \end{aligned}$$
(2.3)

For more details on Itô’s formula with jumps, see, e.g., Reference [32].

From now on, we make the following fundamental assumptions on the jump-diffusion coefficients of model (1.3).

Assumption 1

For any \(p>0\) and \(i=1,2\), there exist constants \(K_{j}>0\ (j=1,2,3)\) such that

$$\begin{aligned} & \int _{\mathbb{Z}} \bigl\{ \bigl\vert \gamma _{i}(u) \bigr\vert ^{2}\vee \bigl[\ln \bigl(1+\gamma _{i}(u)\bigr) \bigr]^{2} \bigr\} \lambda (du)\leq K_{1}< \infty, \\ & \int _{\mathbb{Z}} \bigl\{ \gamma _{i}(u)-\ln \bigl(1+\gamma _{i}(u)\bigr) \bigr\} \lambda (du)\leq K_{2}< \infty, \\ & \int _{\mathbb{Z}} \bigl\{ \bigl(1+\gamma _{i}(u) \bigr)^{p}-1-p\gamma _{i}(u) \bigr\} \lambda (du)\leq K_{3} < \infty, \end{aligned}$$

which implies that the intensity of Lévy noise cannot be too strong, otherwise the solution of system (1.3) may explode in some finite time.

For the continuous and bounded function \(f(t)\) defined on \([0,+\infty )\), we cite the following notions:

$$\begin{aligned} \bigl\langle f(t)\bigr\rangle =\frac {1}{t} \int _{0}^{t}f(s)\,ds, \qquad\bigl\langle f(t) \bigr\rangle ^{*}=\lim_{t\to \infty}\sup \bigl\langle f(t)\bigr\rangle ,\qquad \bigl\langle f(t)\bigr\rangle _{*}=\lim_{t\to \infty} \inf \bigl\langle f(t)\bigr\rangle . \end{aligned}$$

3 Existence and boundedness of solutions

In this section, under Assumption 1, we show that the solution of system (1.3) is globally nonnegative, and for any \(n>0\), the solution admits a uniformly finite nth moment.

Theorem 3.1

Let Assumption 1hold. Then, for any given initial value \((x_{0},y_{0})\in \mathbb{R}_{+}^{2}\), system (1.3) will possess a unique solution \((x(t),y(t))\) for all \(t\geq 0\), and the solution will remain in \(\mathbb{R}_{+}^{2}\) with probability 1.

Proof

Since the coefficients of (1.3) are locally Lipschitz continuous, for any given initial condition \((x_{0},y_{0})\in \mathbb{R}_{+}^{2}\), there is a unique local solution \((x(t),y(t))\) for \(t\in [0,\tau _{e})\), where \(\tau _{e}\) is the explosion time (see [33]). To show that the solution is global, we need to prove \(\tau _{e}=\infty \) a.s. Let \(k_{0}>0\) be sufficiently large such that \(x_{0}\) and \(y_{0}\) are lying within the interval \([1/k_{0},k_{0}]\). For each integer \(k\geq k_{0}\), we define the stopping time by

$$ \tau _{k}=\inf \biggl\{ t\in [0,\tau _{e}):x(t)\notin \biggl( \frac {1}{k},k \biggr) {\text{ or }} y(t)\notin \biggl( \frac {1}{k},k \biggr) \biggr\} , $$

where \(\inf \emptyset =\infty \) (as usual, denotes the empty set). Denote \(\tau _{\infty}=\lim_{k\to \infty}\tau _{k}\), and since \(\tau _{k}\) is nondecreasing as \(k\to \infty \), then \(\tau _{\infty}\leq \tau _{e}\) a.s. Now we show that \(\tau _{\infty}=\infty \) a.s. If not, then there exist \(T>0\) and \(\varepsilon \in (0,1)\) such that \(\mathbb{P}\{\tau _{\infty}\leq T\}>\varepsilon \). Thus, by denoting \(\varOmega _{k}=\{\tau _{k}\leq T\}\), there exists \(k_{1}\geq k_{0}\) such that

$$ \mathbb{P} (\varOmega _{k} )\geq \varepsilon \quad{\text{for all }} k\geq k_{1}. $$
(3.1)

Define a \(C^{2}\)-function V: \(\mathbb{R}_{+}^{2}\to \mathbb{R}_{+}^{2}\) by \(V(x,y)=x-1-\ln x+y-1-\ln y\), which is nonnegative. If \((x(t),y(t))\in \mathbb{R}_{+}^{2}\), by using Itô’s formula, we get

$$\begin{aligned} dV(x,y) ={}& LV(x,y)\,dt+\sigma _{1}(x-1)\,dB_{1}(t)+\sigma _{2}(y-1)\,dB_{2}(t) \\ &{}+ \int _{\mathbb{Z}} \bigl\{ \bigl[\gamma _{1}(u)x-\ln \bigl(1+ \gamma _{1}(u)\bigr)\bigr]+\bigl[ \gamma _{2}(u)y-\ln \bigl(1+\gamma _{2}(u)\bigr)\bigr] \bigr\} \tilde{N}(dt,du), \end{aligned}$$
(3.2)

where

$$\begin{aligned} LV(x,y) ={}& (x-1) \biggl(\frac {\alpha}{1+ky}-bx- \frac {\beta y}{1+mx+ny} \biggr) \\ &{}+ \biggl(1-\frac {1}{y} \biggr) \frac {\theta \beta xy}{1+mx+ny}-(y-1) (c+dy)+ \frac {\sigma _{1}^{2}+\sigma _{2}^{2}}{2} \\ & {}+ \int _{\mathbb{Z}} \bigl[\gamma _{1}(u)-\ln \bigl(1+\gamma _{1}(u)\bigr) \bigr]\lambda (du)+ \int _{\mathbb{Z}} \bigl[\gamma _{2}(u)-\ln \bigl(1+ \gamma _{2}(u)\bigr) \bigr]\lambda (du) \\ \leq {}& x \biggl(\frac {\alpha}{1+ky}-bx-\frac {\beta y}{1+mx+ny} \biggr)- \biggl( \frac {\alpha}{1+ky}-bx-\frac {\beta y}{1+mx+ny} \biggr) \\ &{}-y(c+dy)+(c+dy) \\ &{} +\frac {\theta \beta xy}{1+mx+ny}+ \frac {\sigma _{1}^{2}+\sigma _{2}^{2}}{2}+ \int _{\mathbb{Z}} \bigl[ \gamma _{1}(u)-\ln \bigl(1+\gamma _{1}(u)\bigr) \bigr]\lambda (du) \\ &{}+ \int _{ \mathbb{Z}} \bigl[\gamma _{2}(u)-\ln \bigl(1+\gamma _{2}(u)\bigr) \bigr] \lambda (du) \\ \leq{} & -bx^{2}+(\alpha +b)x-dy^{2}+\biggl(d-c+\beta + \frac {\theta \beta}{m}\biggr)y+c+ \frac {\sigma _{1}^{2}+\sigma _{2}^{2}}{2}+2K_{2} \\ \leq {}& \frac {(\alpha +b)^{2}}{4b}+ \frac {(d-c+\beta +\frac {\theta \beta}{m})^{2}}{4d}+c+ \frac {\sigma _{1}^{2}+\sigma _{2}^{2}}{2}+2K_{2} \\ := {}& K>0, \end{aligned}$$
(3.3)

where K is a positive constant. Thus

$$\begin{aligned} dV(x,y) \leq {}& K\,dt+\sigma _{1}(x-1) \,dB_{1}(t)+\sigma _{2}(y-1)\,dB_{2}(t) \\ &{}+ \int _{\mathbb{Z}} \bigl[\gamma _{1}(u)x-\ln \bigl(1+\gamma _{1}(u)\bigr) \bigr]\tilde{N}(dt,du) \\ &{}+ \int _{\mathbb{Z}} \bigl[\gamma _{2}(u)y- \ln \bigl(1+\gamma _{2}(u)\bigr) \bigr]\tilde{N}(dt,du). \end{aligned}$$
(3.4)

Integrating both sides of (3.4) from 0 to \(\tau _{k}\wedge T\), we obtain

$$\begin{aligned} \int _{0}^{\tau _{k}\wedge T}\,dV\bigl(x(t),y(t)\bigr) \leq {}& \int _{0}^{\tau _{k} \wedge T}K\,dt+ \int _{0}^{\tau _{k}\wedge T}\sigma _{1}(x-1) \,dB_{1}(t)+ \int _{0}^{\tau _{k}\wedge T}\sigma _{2}(y-1) \,dB_{2}(t) \\ &{}+ \int _{0}^{\tau _{k}\wedge T} \int _{\mathbb{Z}} \bigl[\gamma _{1}(u)x- \ln \bigl(1+\gamma _{1}(u)\bigr) \bigr]\tilde{N}(dt,du) \\ &{}+ \int _{0}^{\tau _{k}\wedge T} \int _{\mathbb{Z}} \bigl[\gamma _{2}(u)y- \ln \bigl(1+\gamma _{2}(u)\bigr) \bigr]\tilde{N}(dt,du). \end{aligned}$$

Taking expectations of the above inequality leads to

$$\begin{aligned} EV(x_{\tau _{k}\wedge T},y_{\tau _{k}\wedge T})\leq V(x_{0},y_{0})+KE( \tau _{k}\wedge T)\leq V(x_{0},y_{0})+KT. \end{aligned}$$

On the other hand, by (3.1), we get \(\mathbb{P} (\varOmega _{k} )\geq \varepsilon \). Noting that, for every \(\omega \in \varOmega _{k}\), either \(x_{\tau _{k}}(\omega )\) or \(y_{\tau _{k}}(\omega )\) equals either k or \(1/k\), we have

$$\begin{aligned} V\bigl(x_{\tau _{k}\wedge T}(\omega ),y_{\tau _{k}\wedge T}(\omega )\bigr)\geq \varepsilon (k-1-\ln k)\wedge \biggl(\frac {1}{k}-1+\ln k\biggr). \end{aligned}$$

Setting \(k\to \infty \) leads to the contradiction

$$ \infty >V(x_{0},y_{0})+KT=\infty. $$

Therefore, we have \(\tau _{\infty}=\infty \) a.s. The proof is complete. □

Now we prove the boundedness of the moments of \(x(t)\) and \(y(t)\).

Theorem 3.2

Let \(X(t)=(x(t),y(t))\) be a solution of system (1.3). For any initial value \(X_{0}=(x_{0},y_{0}) \in \mathbb{R}_{+}^{2}\), then for \(n\geq 1\) we have

$$ E\bigl(x^{n}(t)\bigr)\leq M_{1}(n),\quad {\textit{where }} M_{1}(n)= \biggl({\frac{n}{n+1}} \biggr)^{n+1} \frac{ [\frac{1+\tilde{\gamma}_{1n}}{n}+ \alpha +\frac{n-1}{2}\sigma _{1}^{2} ]^{n+1}}{b^{n}}. $$

Also, for \(n\geq 1\),

$$ E\bigl(y^{n}(t)\bigr)\leq M_{2}(n),\quad {\textit{where }} M_{2}(n)= \biggl({\frac{n}{n+1}} \biggr)^{n+1} \frac{ [\frac{1+\tilde{\gamma}_{2n}}{n}+ \frac{\theta \beta}{m}+\frac{n-1}{2}\sigma _{2}^{2} ]^{n+1}}{d^{n}}. $$

Proof

Applying Itô’s formula to the first equation of (1.3), we can easily obtain

$$\begin{aligned} d\bigl(e^{t}x^{n}(t)\bigr)={}& ne^{t}x^{n}(t) \biggl[ \frac {1+\tilde{\gamma}_{1n}}{n}+\frac {\alpha}{1+ky(t)}-bx(t)- \frac {\beta y(t)}{1+mx(t)+ny(t)}+ \frac {n-1}{2}\sigma _{1}^{2} \biggr]\,dt \\ &{}+ne^{t}x^{n}(t)\sigma _{1}\,dB_{1}(t)+e^{t}x^{n}(t) \int _{\mathbb{Z}} \bigl[\bigl(1+\gamma _{1}(u) \bigr)^{n}-1 \bigr]\tilde{N}(dt,du), \end{aligned}$$
(3.5)

where \(\tilde{\gamma}_{1n}=\int _{\mathbb{Z}} \{(1+\gamma _{1}(u))^{n}-1-n \gamma _{1}(u) \}\lambda (du)\). Integrating the two sides of (3.5) and taking expectations leads to

$$\begin{aligned} &E\bigl(e^{t}x^{n}(t)\bigr) \\ &\quad= x_{0}^{n}+n \int _{0}^{t} \biggl\{ E\bigl(e^{s}x^{n}(s) \bigr) \biggl[\frac {1+\tilde{\gamma}_{1n}}{n}+\frac {\alpha}{1+ky(t)}- \frac {\beta y(t)}{1+mx(t)+ny(t)}+ \frac {n-1}{2}\sigma _{1}^{2} \biggr] \\ &\qquad{}-bE \bigl(e^{s}x^{n+1}(s)\bigr) \biggr\} \,ds \\ &\quad \leq x_{0}^{n}+n \int _{0}^{t} \biggl\{ E\bigl(e^{s}x^{n}(s) \bigr) \biggl[ \frac {1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac {n-1}{2}\sigma _{1}^{2} \biggr]-bE\bigl(e^{s}x^{n+1}(s) \bigr) \biggr\} \,ds. \end{aligned}$$

Now let \(h(x)=x^{n} \{ [\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha + \frac{n-1}{2}\sigma _{1}^{2} ]-bx \}\). In order to find the maximum value of \(h(x)\), we first calculate \(h'(x)\) and obtain

$$\begin{aligned} h'(x)={}& nx^{n-1} \biggl\{ \biggl[\frac {1+\tilde{\gamma}_{1n}}{n}+ \alpha +\frac {n-1}{2}\sigma _{1}^{2} \biggr]-bx \biggr\} +x^{n}(-b) \\ ={}& x^{n-1} \biggl\{ n \biggl[\frac {1+\tilde{\gamma}_{1n}}{n}+\alpha + \frac {n-1}{2}\sigma _{1}^{2} \biggr]-b(n+1)x \biggr\} . \end{aligned}$$

When \(x= \frac{n [\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac{n-1}{2}\sigma _{1}^{2} ]}{b(n+1)}\), we get the critical point by getting \(h'(x)=0\). Further, we notice that \(h''(x)<0\) at the critical point, and maximum value at the critical point is given by

$$ h_{\mathrm{max}}= \biggl(\frac {n}{b} \biggr)^{n} \biggl( \frac {\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac{n-1}{2}\sigma _{1}^{2}}{n+1} \biggr)^{n+1}. $$

Therefore,

$$\begin{aligned} E\bigl(e^{t}x^{n}(t)\bigr)\leq{} & x_{0}^{n}+nE \int _{0}^{t}e^{s} \biggl( \frac {n}{b} \biggr)^{n} \biggl( \frac {\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac{n-1}{2}\sigma _{1}^{2}}{n+1} \biggr)^{n+1}\,ds \\ \leq {}& x_{0}^{n}+ \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac{n-1}{2}\sigma _{1}^{2} ]^{n+1}}{b^{n}}\bigl(e^{t}-1\bigr), \end{aligned}$$

i.e.,

$$\begin{aligned} E\bigl(x^{n}(t)\bigr)\leq {}& \biggl\{ x_{0}^{n}- \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac{n-1}{2}\sigma _{1}^{2} ]^{n+1}}{b^{n}} \biggr\} e^{-t} \\ &{} + \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac{n-1}{2}\sigma _{1}^{2} ]^{n+1}}{b^{n}}. \end{aligned}$$

It is clear that \(E(x^{n}(t))=x_{0}^{n}\) if \(t=0\), and if \(t\to \infty \), then

$$ E\bigl(x^{n}(t)\bigr)\leq \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{1n}}{n}+\alpha +\frac{n-1}{2}\sigma _{1}^{2} ]^{n+1}}{b^{n}}:=M_{1}(n). $$

Therefore, we conclude that \(E(x^{n}(t))\leq M_{1}(n)\) for \(n\geq 1\).

Similarly for predator species, we have

$$\begin{aligned} d\bigl(e^{t}y^{n}(t)\bigr)={}& ne^{t}y^{n}(t) \biggl[ \frac {1+\tilde{\gamma}_{2n}}{n}-c-dy(t)+ \frac {\theta \beta x(t)}{1+mx(t)+ny(t)}+\frac {n-1}{2}\sigma _{2}^{2} \biggr]\,dt \\ &{}+ne^{t}y^{n}(t)\sigma _{2}\,dB_{2}(t)+e^{t}y^{n}(t) \int _{\mathbb{Z}} \bigl[\bigl(1+\gamma _{2}(u) \bigr)^{n}-1 \bigr]\tilde{N}(dt,du), \end{aligned}$$
(3.6)

where \(\tilde{\gamma}_{2n}=\int _{\mathbb{Z}} \{(1+\gamma _{2}(u))^{n}-1-n \gamma _{2}(u) \}\lambda (du)\). Integrating both sides of (3.6) from 0 to t and taking expectation, we have

$$\begin{aligned} &E\bigl(e^{t}y^{n}(t)\bigr) \\ &\quad= y_{0}^{n}+n \int _{0}^{t}E\bigl(e^{s}y^{n}(s) \bigr) \biggl[ \frac {1+\tilde{\gamma}_{2n}}{n}-c-dy(t)+ \frac {\theta \beta x(t)}{1+mx(t)+ny(t)}+\frac {n-1}{2} \sigma _{2}^{2} \biggr]\,ds \\ &\quad\leq y_{0}^{n}+n \int _{0}^{t}E\bigl(e^{s}y^{n}(s) \bigr) \biggl\{ \biggl[ \frac {1+\tilde{\gamma}_{2n}}{n}+\frac {\theta \beta}{m}+ \frac {n-1}{2} \sigma _{2}^{2} \biggr]-dy(t) \biggr\} \,ds \\ &\quad \leq y_{0}^{n}+nE \int _{0}^{t}e^{s} \biggl( \frac {n}{d} \biggr)^{n} \biggl( \frac {\frac{1+\tilde{\gamma}_{2n}}{n}+\frac{\theta \beta}{m}+\frac{n-1}{2}\sigma _{2}^{2}}{n+1} \biggr)^{n+1}\,ds \\ &\quad \leq y_{0}^{n}+ \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{2n}}{n}+\frac{\theta \beta}{m}+\frac{n-1}{2}\sigma _{2}^{2} ]^{n+1}}{d^{n}}\bigl(e^{t}-1\bigr). \end{aligned}$$

That is,

$$\begin{aligned} E\bigl(y^{n}(t)\bigr)\leq{} & \biggl\{ y_{0}^{n}- \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{2n}}{n}+\frac{\theta \beta}{m}+\frac{n-1}{2}\sigma _{2}^{2} ]^{n+1}}{d^{n}} \biggr\} e^{-t} \\ &{} + \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{2n}}{n}+\frac{\theta \beta}{m}+\frac{n-1}{2}\sigma _{2}^{2} ]^{n+1}}{d^{n}}. \end{aligned}$$

By a similar reason as above, we have

$$ E\bigl(y^{n}(t)\bigr)\leq \biggl(\frac {n}{n+1} \biggr)^{n+1} \frac { [\frac{1+\tilde{\gamma}_{2n}}{n}+\frac{\theta \beta}{m}+\frac{n-1}{2}\sigma _{2}^{2} ]^{n+1}}{d^{n}}:=M_{2}(n),\quad n \geq 1. $$

The proof is now complete. □

4 Stochastic persistence and extinction scenarios

In this current section we are motivated to investigate the persistence and extinction criterion of system (1.3). Before going to the main results, we want to highlight some frequently used definitions and lemmas [34].

Definition 4.1

For system (1.3), we say that:

  1. (a)

    The population \(x(t)\) goes to extinction if \(\lim_{t\to \infty}x(t)=0\);

  2. (b)

    The population \(x(t)\) is nonpersistent in mean if \(\langle x(t)\rangle ^{*}=0\);

  3. (c)

    The population \(x(t)\) is weakly persistent in mean if \(\langle x(t)\rangle ^{*}>0\).

For later applications, we cite a strong law of large numbers for local martingales as the first lemma below, see Theorem 1 in [35].

Lemma 4.1

Let \(J(t)(t\geq 0)\) be a local martingale vanishing at time 0 and define

$$ \rho _{J}(t):= \int _{0}^{t}\frac {d\langle J\rangle (s)}{(1+s)^{2}},\quad t\geq 0,$$

where \(\langle J\rangle (t):=\langle J,J\rangle (t)\) is Meyer’s angle bracket process. Then

$$ \lim_{t\to \infty}\frac {J(t)}{t}=0 a.s. \quad\textit{provided that } \lim_{t\to \infty}\rho _{J}(t)< \infty \textit{ a.s.}$$

Lemma 4.2

For the solution \((x(t),y(t))\) of system (1.3) initiated from \((x_{0},y_{0})\in \mathbb{R}_{+}^{2}\), the below mentioned properties hold:

$$ \lim_{t\to \infty}\sup \frac{\ln x(t)}{t}\leq 0\quad\textit{and}\quad \lim _{t\to \infty}\sup \frac{\ln y(t)}{t}\leq 0\quad \textit{almost surely}.$$

Proof

The proof can be shown by a similar procedure as in Cheng [36]. So we omit it here. □

Lemma 4.3

(See Lemma 7.3 in [18])

Let \(x(t)\in C[\varOmega \times \mathbb{R}_{+},\mathbb{R}_{+}^{0}]\), where \(\mathbb{R}_{+}^{0}:=\{r:r>0,r\in \mathbb{R}\}\).

  1. (1)

    If it is possible to find positive constants \(\alpha _{0},T,\alpha \geq 0\) such that

    $$ \ln x(t)\leq \alpha T-\alpha _{0} \int _{0}^{t}x(s)\,ds+\sum _{i=1}^{n} \beta _{i}(t)B_{i}(t)$$

    for \(t\geq T\), where \(\beta _{i}(t)\ (1\leq i\leq n)\) are continuous bounded functions on \(\mathbb{R}_{+}\) and \(B_{i}(t)\) are independent standard Brownian motions, then we have \(\langle x(t)\rangle ^{*}\leq \frac{\alpha}{\alpha _{0}}\) almost surely.

  2. (2)

    If it is possible to find positive constants \(\alpha _{0},T,\alpha \geq 0\) such that

    $$ \ln x(t)\geq \alpha T-\alpha _{0} \int _{0}^{t}x(s)\,ds+\sum _{i=1}^{n} \beta _{i}(t)B_{i}(t)$$

    for \(t\geq T\), where \(\beta _{i}(t) (1\leq i\leq n)\) are continuous bounded functions on \(\mathbb{R}_{+}\) and \(B_{i}(t)\) are independent standard Brownian motions, then we have \(\langle x(t)\rangle _{*}\geq \frac{\alpha}{\alpha _{0}}\) almost surely.

Now we are going to present the main results of this section. Applying Itô’s formula to (1.3) results in

$$\begin{aligned} d\ln x ={}& \biggl\{ \frac {\alpha}{1+ky(t)}-bx(t)- \frac {\beta y(t)}{1+mx(t)+ny(t)}- \frac {\sigma _{1}^{2}}{2} \\ &{}+ \int _{ \mathbb{Z}} \bigl[\ln \bigl(1+\gamma _{1}(u)\bigr)- \gamma _{1}(u) \bigr] \lambda (du) \biggr\} \,dt \\ & {}+\sigma _{1}\,dB_{1}(t)+ \int _{\mathbb{Z}}\ln \bigl(1+\gamma _{1}(u)\bigr) \tilde{N}(dt,du), \end{aligned}$$
(4.1)
$$\begin{aligned} d\ln y ={}& \biggl\{ -c-dy(t)+\frac {\theta \beta x(t)}{1+mx(t)+ny(t)}- \frac {\sigma _{2}^{2}}{2}+ \int _{\mathbb{Z}} \bigl[\ln \bigl(1+\gamma _{2}(u)\bigr)- \gamma _{2}(u) \bigr]\lambda (du) \biggr\} \,dt \\ &{} +\sigma _{2}\,dB_{2}(t)+ \int _{\mathbb{Z}}\ln \bigl(1+\gamma _{2}(u)\bigr) \tilde{N}(dt,du). \end{aligned}$$
(4.2)

For convenience, we denote

$$ Q_{1}=\alpha -\frac {\sigma _{1} ^{2}}{2},\qquad Q_{2}=-c- \frac {\sigma _{2} ^{2}}{2}, \qquad Q_{3}=\frac {\alpha}{1+kM_{2}(1)}- \frac {\sigma _{1} ^{2}}{2},$$

where \(M_{2}(1)=\frac{ (1+\tilde{\gamma}_{21}+\frac{\theta \beta}{m} )^{n+1}}{4d}\), \(Q_{2}<0\). We give the dynamics of prey population \(x(t)\) of (1.3) as follows.

Theorem 4.1

For the prey population \(x(t)\) of (1.3), we have the following properties:

  1. (1)

    If \(Q_{1}<0\), then \(x(t)\) goes to extinction almost surely.

  2. (2)

    If \(Q_{1}=0\), then \(x(t)\) is nonpersistent in the mean almost surely.

  3. (3)

    If \(Q_{3}>0\), then \(x(t)\) is weakly persistent in the mean almost surely.

Proof

Now we prove them point by point.

(1) Integrating both sides of (4.1) from 0 to t leads to

$$\begin{aligned} \ln x(t)-\ln x_{0} \leq {}& \int _{0}^{t} \biggl\{ Q_{1}-bx(s)- \frac {\beta y(s)}{1+mx(s)+ny(s)} \\ &{}- \int _{\mathbb{Z}} \bigl[\gamma _{1}(u)- \ln \bigl(1+\gamma _{1}(u)\bigr) \bigr]\lambda (du) \biggr\} \,ds+P_{1}(t), \end{aligned}$$
(4.3)

where \(P_{1}(t)=\int _{0}^{t}\sigma _{1}\,dB_{1}(s)+\int _{0}^{t}\int _{ \mathbb{Z}}\ln (1+\gamma _{1}(u))\tilde{N}(ds,du)\). From (4.3) we get

$$ \ln x(t)-\ln x_{0}\leq tQ_{1}+P_{1}(t).$$

By Lemma 4.1 and the strong law of large numbers, we obtain

$$\begin{aligned} \lim_{t\to \infty}\frac {1}{t} \int _{0}^{t}\sigma _{1} \,dB_{1}(s)=0\quad {\text{and}}\quad \lim_{t\to \infty} \frac {1}{t} \int _{0}^{t} \int _{\mathbb{Z}}\ln \bigl(1+\gamma _{1}(u)\bigr) \tilde{N}(ds,du)=0\quad \text{a.s.}, \end{aligned}$$

i.e.,

$$\begin{aligned} \lim_{t\to \infty}\frac {P_{1}(t)}{t}=0\quad \text{a.s.} \end{aligned}$$
(4.4)

Thus

$$\begin{aligned} \frac {\ln x(t)-\ln x_{0}}{t}\leq Q_{1}+\frac {P_{1}(t)}{t}. \end{aligned}$$
(4.5)

By taking the superior limit on both sides of (4.5), we get

$$ \lim_{t\to \infty}\sup \frac {\ln x(t)}{t}\leq Q_{1}< 0.$$

So \(\lim_{t\to \infty}x(t)=0\).

(2) From (4.3) we get

$$\begin{aligned} \frac {\ln x(t)-\ln x_{0}}{t}\leq Q_{1}-b\bigl\langle x(t)\bigr\rangle + \frac {P_{1}(t)}{t}. \end{aligned}$$
(4.6)

By the specific property of the superior limit and from (4.4), we can easily notice that for arbitrarily given and sufficiently small \(\varepsilon >0\), there exists \(T>0\) such that \(P_{1}(t)/t\leq \varepsilon \) for all \(t\geq T\). Now, by substituting the above inequality in (4.6), we have

$$ \ln x(t)-\ln x_{0}\leq (Q_{1}+\varepsilon )t-b \int _{0}^{t}x(s)\,ds.$$

Under the condition that \(Q_{1}=0\), we have

$$ \ln \frac {x(t)}{x_{0}}\leq \varepsilon t-bx_{0} \int _{0}^{t} \frac {x(s)}{x_{0}}\,ds.$$

As \(b>0\), we derive from Lemma 4.3 that

$$ \bigl\langle x(t)\bigr\rangle ^{*}\leq \frac {\varepsilon}{b}.$$

By the arbitrariness of ε, due to the fact that the solution of (1.3) is nonnegative, we have \(\langle x(t)\rangle ^{*}=0\). From Definition 4.1, the prey species \(x(t)\) is nonpersistent in the mean a.s.

(3) By Definition 4.1, we only need to show that there exists a constant \(m_{1}>0\) such that

$$ \bigl\langle x(t)\bigr\rangle ^{*}\geq m_{1}$$

for any solution \((x(t),y(t))\) of (1.3) with initial value \((x_{0},y_{0})\in \mathbb{R}_{+}^{2}\) almost surely. If not, then for any arbitrary \(\varepsilon _{1}>0\), there exists a solution \((\tilde{x}(t),\tilde{y}(t))\) with positive initial values \(\tilde{x}_{0}>0\) and \(\tilde{y}_{0}>0\) satisfying \(P\{\langle \tilde{x}(t)\rangle ^{*}<\varepsilon _{1}\}>0\). Let \(\varepsilon _{1}\) be sufficiently small so that

$$\begin{aligned} \textstyle\begin{cases} Q_{3}- b\varepsilon _{1} >0, \\ Q_{2}+\theta \beta \varepsilon _{1}^{2}< 0. \end{cases}\displaystyle \end{aligned}$$
(4.7)

From (4.2) we have

$$\begin{aligned} \frac {\ln \tilde{y}(t)-\ln y_{0}}{t}\leq Q_{2}-d\bigl\langle \tilde{y}(t) \bigr\rangle + \theta \beta \bigl\langle \tilde{x}(t)\bigr\rangle + \frac {P_{2}(t)}{t}, \end{aligned}$$
(4.8)

where \(P_{2}(t)=\int _{0}^{t}\sigma _{2}\,dB_{2}(s)+\int _{0}^{t}\int _{ \mathbb{Z}}\ln (1+\gamma _{2}(u))\tilde{N}(ds,du)\), then we get \(\langle t^{-1}\ln \tilde{y}(t)\rangle ^{*}=Q_{2}+\theta \beta \varepsilon _{1}<0\). Therefore,

$$\begin{aligned} \lim_{t\to \infty}\tilde{y}(t)=0. \end{aligned}$$
(4.9)

From (4.3) we have

$$ \frac {\ln \tilde{x}(t)-\ln x_{0}}{t}\geq Q_{3}-b\bigl\langle \tilde{x}(t) \bigr\rangle -\beta \bigl\langle \tilde{y}(t)\bigr\rangle +\frac {P_{1}(t)}{t}.$$

Taking the superior limit on both sides of the above inequality and using (4.4), (4.7), and (4.9), we get

$$ \bigl\langle t^{-1}\ln \tilde{x}(t)\bigr\rangle ^{*}\geq Q_{3}-b\varepsilon _{1}>0. $$

Now we have shown that \(P\{\langle t^{-1}\ln \tilde{x}(t)\rangle ^{*}>0\}>0\), which contradicts Lemma 4.2. Hence \(\langle x(t)\rangle ^{*}>0\), that is, the prey population \(x(t)\) is weakly persistent in the mean almost surely. The proof is complete. □

In the case of predator species of system (1.3), we have the following theorem.

Theorem 4.2

For the predator population of system (1.3), we have the following properties:

  1. (1)

    If \(bQ_{2}+\theta \beta Q_{1}<0\), then \(y(t)\) goes to extinction almost surely.

  2. (2)

    If \(bQ_{2}+\theta \beta Q_{1}=0\), then \(y(t)\) is nonpersistent in the mean almost surely.

  3. (3)

    If \(Q_{2}+ \langle \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)} \rangle ^{*}>0\), then \(y(t)\) is weakly persistent in the mean almost surely, where \(\bar{x}(t)\) and \(\bar{y}(t)\) are defined later.

Proof

We prove them point by point as follows.

(1) If \(bQ_{2}+\theta \beta Q_{1}<0\), because \(Q_{2}=-c-\frac {\sigma _{2} ^{2}}{2}<0\) is known and \(b,\theta,\beta \) are all positive constants, we consider the following two cases of \(Q_{1}\).

If \(Q_{1}\leq 0\), then it follows from Theorem 4.1 that \(\langle x(t)\rangle ^{*}=0\). On the other hand, by using (4.2), we have

$$ \frac {\ln y(t)-\ln y_{0}}{t}\leq Q_{2}+\theta \beta \bigl\langle x(t) \bigr\rangle +\frac {P_{2}(t)}{t}.$$

Then we get \([t^{-1}\ln y(t)]^{*}\leq Q_{2}<0\), and hence \(\lim_{t\to \infty}y(t)=0\) holds.

If \(Q_{1}>0\), from the properties of the superior and inferior limits combined with (4.4), we obtain that there exists a constant \(T>0\) such that

$$ \frac {\ln x(t)-\ln x_{0}}{t}\leq Q_{1}-b\bigl\langle x(t)\bigr\rangle + \varepsilon $$

for arbitrarily sufficiently small \(\varepsilon >0\) and \(t>T\). Applying Lemma 4.3 and the arbitrariness of ε, we have

$$\begin{aligned} \bigl\langle x(t)\bigr\rangle ^{*}\leq \frac {Q_{1}}{b}. \end{aligned}$$
(4.10)

Now by (4.2) we get

$$\begin{aligned} \bigl( t^{-1}\ln y(t)\bigr)^{*}\leq Q_{2}+\theta \beta \frac {Q_{1}}{b}. \end{aligned}$$
(4.11)

By the condition \(bQ_{2}+\theta \beta Q_{1}<0\), we have

$$ \bigl( t^{-1}\ln y(t)\bigr)^{*}\leq \frac {bQ_{2}+\theta \beta Q_{1}}{b}< 0.$$

Therefore, \(\lim_{t\to \infty}y(t)=0\) almost surely.

(2) In the previous case we have shown that \(\lim_{t\to \infty}y(t)=0\) under the condition \(Q_{1}\leq 0\), so we only need to show that \(\langle y(t)\rangle ^{*}=0\) if \(Q_{1}>0\). If not, then \(\langle y(t)\rangle ^{*}>0\) under \(Q_{1}>0\). By Lemma 4.3 and (4.11), we have

$$\begin{aligned} \bigl[t^{-1}\ln y(t)\bigr]^{*}\leq Q_{2}+\theta \beta \bigl\langle x(t)\bigr\rangle ^{*}. \end{aligned}$$
(4.12)

For any arbitrarily small \(\varepsilon >0\), we get \(T>0\) such that, for any \(t>T\), the following inequalities hold:

$$\begin{aligned} &\bigl\langle \theta \beta x(t)\bigr\rangle \leq \theta \beta \bigl\langle x(t) \bigr\rangle ^{*}+\frac {\varepsilon}{2}, \\ &\frac {P_{2}(t)}{t}< \frac {\varepsilon}{2}. \end{aligned}$$

Substituting them in (4.2) leads to

$$ \frac {\ln y(t)-\ln y_{0}}{t}\leq Q_{2}+\varepsilon +\theta \beta \bigl\langle x(t)\bigr\rangle ^{*}-d\bigl\langle y(t)\bigr\rangle \quad\forall t>T.$$

We can obtain from Lemma 4.3 and (4.12) that

$$ \bigl\langle y(t)\bigr\rangle ^{*}\leq \frac {Q_{2}+\varepsilon +\theta \beta \langle x(t)\rangle ^{*}}{d}.$$

Since ε is arbitrarily small, we get from (4.10) that

$$ \bigl\langle y(t)\bigr\rangle ^{*}\leq \frac {bQ_{2}+\theta \beta Q_{1}}{bd}=0,$$

which is a contradiction. Therefore, \(\langle y(t)\rangle ^{*}=0\) holds almost surely.

(3) By Definition 4.1, we only need to show that \(\langle y(t)\rangle ^{*}>0\) almost surely. If it is not true, then for any arbitrarily small \(\varepsilon _{2}>0\), there exists a solution \((\tilde{x}(t),\tilde{y}(t))\) of system (1.3), with positive initial value, \((\tilde{x}_{0},\tilde{y}_{0})\in \mathbb{R}_{+}^{2}\) such that \(\langle \tilde{y}(t)\rangle ^{*}<\varepsilon _{2}\). Let \(\varepsilon _{2}\) be sufficiently small so that

$$\begin{aligned} Q_{2}+ \biggl\langle \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)} \biggr\rangle ^{*}> \biggl(d+ \frac {\theta \beta (k\alpha +\theta \beta )}{mb} \biggr)\varepsilon _{2}, \end{aligned}$$
(4.13)

where \((\bar{x}(t),\bar{y}(t))\) is the solution of the following system:

$$\begin{aligned} \textstyle\begin{cases} d\bar{x}(t)=\bar{x}(t) (\alpha -b\bar{x}(t) )\,dt+\sigma _{1} \bar{x}(t)\,dB_{1}(t), \\ d\bar{y}(t)=\bar{y}(t) (\frac{\theta \beta}{m}-d\bar{y}(t) )\,dt+ \sigma _{2}\bar{y}(t)\,dB_{2}(t), \end{cases}\displaystyle \end{aligned}$$
(4.14)

with initial value \(\bar{x}_{0}>0\) and \(\bar{y}_{0}>0\). By the comparison theorem, we have \(\tilde{x}(t)\leq \bar{x}(t),\tilde{y}(t)\leq \bar{y}(t)\) a.s. for \(t\in [0,\infty )\). From (4.2), we have

$$\begin{aligned} \frac {\ln \tilde{y}(t)-\ln y_{0}}{t}={}&Q_{2}+ \biggl\langle \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)} \biggr\rangle -\bigl\langle \,d\tilde{y}(t)\bigr\rangle +\frac {P_{2}(t)}{t}\\ &{}+ \biggl\langle \frac {\theta \beta \tilde{x}(t)}{1+m\tilde{x}(t)+n\tilde{y}(t)}- \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)} \biggr\rangle . \end{aligned}$$

Then

$$\begin{aligned} \frac {\theta \beta \tilde{x}(t)}{1+m\tilde{x}(t)+n\tilde{y}(t)}- \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)}={}& \frac {\theta \beta (\tilde{x}(t)-\bar{x}(t))+\theta \beta n(\tilde{x}(t)\bar{y}(t)-\bar{x}(t)\tilde{y}(t))}{(1+m\tilde{x}(t)+n\tilde{y}(t))(1+m\bar{x}(t)+n\bar{y}(t))} \\ \geq {}& {-} \frac {\theta \beta (\bar{x}(t)-\tilde{x}(t))+\theta \beta n(\bar{x}(t)-\tilde{x}(t))\tilde{y}(t)}{(1+m\tilde{x}(t)+n\tilde{y}(t))(1+m\bar{x}(t)+n\bar{y}(t))} \\ \geq {}& {-}\frac {\theta \beta (\bar{x}(t)-\tilde{x}(t))}{m}. \end{aligned}$$

After calculation we get

$$\begin{aligned} \frac {\ln \tilde{y}(t)-\ln y_{0}}{t}\geq Q_{2}+ \biggl\langle \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)} \biggr\rangle -\bigl\langle \,d\tilde{y}(t)\bigr\rangle +\frac {P_{2}(t)}{t}+ \biggl\langle \frac {\theta \beta (\bar{x}(t)-\tilde{x}(t))}{m} \biggr\rangle . \end{aligned}$$
(4.15)

Consider the Lyapunov function \(V_{2}(t)=|\ln \bar{x}(t)-\ln \tilde{x}(t)|\). Obviously, \(V_{2}(t)\) is a positive function on \(\mathbb{R}_{+}\). After using Itô’s formula and by (4.14) and (4.15), we get

$$\begin{aligned} d^{+}V_{2}(t)={}& \biggl[ \biggl(\alpha -\frac {\sigma _{1}^{2}}{2}-b \bar{x}(t) \biggr)\,dt+\sigma _{1}\,dB_{1}(t) \biggr] \\ &{}- \biggl[ \biggl(\frac {\alpha}{1+k\tilde{y}(t)}- \frac {\sigma _{1}^{2}}{2}-b\tilde{x}(t)- \frac {\theta \beta \tilde{y}(t)}{1+m\tilde{x}(t)+n\tilde{y}(t)} \biggr)\,dt+\sigma _{1}\,dB_{1}(t) \biggr] \\ \leq{} & \bigl[(k\alpha +\theta \beta )\tilde{y}(t)-b\bigl(\bar{x}(t)- \tilde{x}(t)\bigr) \bigr]\,dt. \end{aligned}$$
(4.16)

Integrating first both sides of (4.16) from 0 to t and then dividing by t, we have

$$ \frac {V_{2}(t)-V_{2}(0)}{t}\leq (k\alpha +\theta \beta )\bigl\langle \tilde{y}(t)\bigr\rangle -b\bigl\langle \bar{x}(t)-\tilde{x}(t)\bigr\rangle .$$

Using \(V_{2}(t)/t\geq 0\) and \(V_{2}(0)=0\) yields

$$ \bigl\langle \bar{x}(t)-\tilde{x}(t)\bigr\rangle \leq \frac {k\alpha +\theta \beta}{b}\bigl\langle \tilde{y}(t)\bigr\rangle .$$

We can derive from the above inequality and (4.15) that

$$ \frac {\ln \tilde{y}(t)-\ln y_{0}}{t}\geq Q_{2}+ \biggl\langle \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)} \biggr\rangle -\bigl\langle d\tilde{y}(t)\bigr\rangle +\frac {P_{2}(t)}{t}- \frac {\theta \beta (k\alpha +\theta \beta ) }{mb}\bigl\langle \tilde{y}(t) \bigr\rangle .$$

Taking the superior limit of the above inequality and by (4.13), we obtain

$$ \bigl[t^{-1}\ln \tilde{y}(t)\bigr]^{*}\geq Q_{2}+ \biggl\langle \frac {\theta \beta \bar{x}(t)}{1+m\bar{x}(t)+n\bar{y}(t)} \biggr\rangle ^{*}- \biggl(d+ \frac {\theta \beta (k\alpha +\theta \beta )}{mb} \biggr)\varepsilon _{2}>0,$$

which contradicts Lemma 4.2. Therefore \(\langle y(t)\rangle ^{*}>0\) almost surely. This implies that \(y(t)\) is weakly persistent in the mean almost surely. The proof is now complete. □

5 Stability in distribution

In this section, let us turn to establishing sufficient criteria for the stability in distribution of system (1.3). First we give the following important lemmas [37, 38].

Lemma 5.1

Let \(X(t)\) be an n-dimensional stochastic process on \(t\ge 0\). Suppose that there exist positive constants \(\tilde{\alpha},\tilde{\beta }, \tilde{\xi} \) such that

$$ E \bigl\vert X ( t ) -X(s) \bigr\vert ^{\tilde{\alpha}}\le \tilde{\xi} \vert t-s \vert ^{1+ \tilde{\beta}},\quad 0\le s,t< \infty.$$

Then there exists a continuous modification \(\tilde{X}(t)\) of \(X(t)\), and almost every sample path of \(\tilde{X}(t)\) is local but uniformly \(H\ddot{o}lder\) continuous with exponent \(\kappa <\frac{\tilde{\alpha}}{\tilde{\beta}}\). In other words, the continuous modification \(\tilde{X} ( t ) \) of \(X(t)\) has the property that, for every \(\kappa \in (0,\frac{\tilde{\alpha}}{\tilde{\beta}})\),

$$ P \biggl\{ \zeta: \underset{0< \vert t-s \vert < f ( \zeta ),0\le s,t< \infty}{\sup} \frac{|\tilde{X} ( t,\zeta ) -\tilde{X} ( s,\zeta )}{ \vert t-s \vert ^{\kappa}} \le \frac{2}{1-2^{-\kappa}} \biggr\} =1.$$

Lemma 5.2

Let \(f(t)\) be a nonnegative function defined on \([0, \infty )\) such that \(f(t)\) is integrable on \([0, \infty )\) and is uniformly continuous on \([0, \infty )\), then \(\lim_{t \to +\infty} f(t)=0\).

Definition 5.1

Let \(X_{1}(t)=(x_{1}(t), y_{1}(t))\) be a positive solution of (1.3) with initial value \(X_{1}(0)\in \mathbb{R}_{+}^{2}\). \(X_{1}(t)\) is said to be globally asymptotically stable in expectation if for any other solution \(X_{2}(t)=(x_{2}(t), y_{2}(t))\) of (1.3), we have

$$ P \Bigl\{ \lim_{t\rightarrow +\infty}E \bigl( \bigl\vert X_{1}(t) -X_{2}(t) \bigr\vert \bigr) =0 \Bigr\} =1.$$

Theorem 5.1

If \(b-\theta \beta -(\beta m+\theta \beta n) G_{2} >0\) and \(d-\alpha k-\beta -(\beta m+\theta \beta n) G_{1} >0\), then system (1.3) is asymptotically stable in distribution, where \(G_{1}\) and \(G_{2}\) are defined later. That is, there exists a unique probability measure \(\mu (\cdot )\) such that, for any initial value \(X(0)=(x(0),y(0))\), the transition probability \(p(t,X(0),\cdot )\) of \({X}(t)\) weakly converges to \(\mu (\cdot )\) as \(t\rightarrow \infty \).

Proof

Consider the following stochastic integral equation of (1.3):

$$\begin{aligned} x ( t ) ={}&x ( 0 ) + \int _{0}^{t}{x(s) \biggl[ \frac{\alpha}{1+ky}-bx- \frac {\beta y}{1+mx+ny} \biggr]}\,ds+ \int _{0}^{t}{x(s) \sigma _{1}}\,dB(s)\\ &{}+ \int _{0}^{t}{ \int _{\mathbb{Z}}{x(s) \gamma _{1} ( u )}\tilde{N} ( ds,du )}. \end{aligned}$$

Let \(f_{1}=x(t) [ \frac{\alpha}{1+ky(t)}-bx ( t ) - \frac{\beta y}{1+mx+ny} ], g_{1}=x ( t ) \sigma _{1},h_{1}=x ( t ) \gamma _{1} ( u ) \). By Theorem 3.2, there is a positive constant \(M_{1}(n)\) such that \(E(x^{n}(t))\le M_{1}(n)\) on \(t\ge 0\). Then we can derive that

$$ \begin{aligned} E \bigl( \vert f_{1} \vert ^{n} \bigr) ={}& E \biggl( x^{n}(s) \biggl\vert \frac{\alpha}{1+ky(s)}-bx(s)-\frac{\beta y(s)}{1+mx(s)+ny(s)} \biggr\vert ^{n} \biggr) \\ \le {}& \frac{1}{2}E \bigl( x^{2n}(s) \bigr) +\frac{1}{2}E \biggl[ \biggl( \frac{\alpha}{1+ky(s)}-bx(s)-\frac{\beta y(s)}{1+mx(s)+ny(s)} \biggr) ^{2n} \biggr] \\ \le {}& \frac{1}{2}E \bigl(x^{2n}(s) \bigr)+2^{2n-2} \bigl(E \bigl( \alpha -bx(s) \bigr) ^{2n}+ (\beta ) ^{2n}E \bigl(y^{2n}(s) \bigr) \bigr) \\ \le {}& \frac{1}{2}E \bigl( x^{2n}(s) \bigr) +2^{4n-3} \bigl(\alpha ^{2n}+b^{2n}E \bigl( x^{2n}(s) \bigr) \bigr) +2^{2n-2} (\beta ) ^{2n}E \bigl( y^{2n}(s) \bigr) \\ \le {}& \frac{1}{2}M_{1} ( 2n ) +2^{4n-3} \bigl(\alpha ^{2n}+b^{2n}M_{1} ( 2n ) \bigr) +2^{2n-2} (\beta ) ^{2n}M_{2} ( 2n ) \\ :={}& R_{1}(n) \end{aligned} $$
(5.1)

and

$$ \begin{aligned} E \bigl( \vert g_{1} \vert ^{n} \bigr) =E \bigl( x^{n}(t)\sigma _{1}^{n} \bigr) = \sigma _{1}^{n}E \bigl(x^{n} (t ) \bigr) \le \sigma _{1}^{n}M_{1} ( n ) :=R_{2}(n). \end{aligned} $$
(5.2)

We assume \(n>2\). For \(0\le s< t<\infty \), the moment inequality (cf. Friedman [39]) on (5.2) leads to

$$ \begin{aligned} E \biggl\vert \int _{s}^{t}{g_{1}}\,dB ( v ) \biggr\vert ^{n}\le \biggl[ \frac {n ( n-1 )}{2} \biggr] ^{\frac{n}{2}} ( t-s ) ^{\frac{n-2}{2}} \int _{s}^{t}{E|g_{1}}|^{n} \,dB ( v ). \end{aligned} $$
(5.3)

Under the conditions of Theorem 5.1, with Kunita’s inequalities (see Lemma 2.3, [40]), we have

$$ \begin{aligned} E \biggl[| \int _{s}^{t}{ \int _{\mathbb{Z}}{h_{1}\tilde{N} ( dv,du )}|^{n}} \biggr] \le {}& 2^{n-1} \biggl\{ E \biggl[ \int _{s}^{t}{ \int _{\mathbb{Z}}{ \bigl\vert x(s) \gamma _{1}(u) \bigr\vert ^{2}}\lambda ( du )}\,dv \biggr] ^{\frac{n}{2}}\\ &{}+E \biggl[ \int _{s}^{t}{ \int _{ \mathbb{Z}}{ \bigl\vert x ( s ) \gamma _{1}(u) \bigr\vert ^{n}\lambda ( du )}\,dv} \biggr] \biggr\} \\ \le{} & 2^{n-1} \bigl\{ ( t-s ) ^{\frac{n}{2}}K_{3}^{ \frac{n}{2}}M_{1} ( n ) + ( t-s ) K_{3}^{ \frac{n}{2}}M_{1} ( n ) \bigr\} . \end{aligned} $$
(5.4)

Let \(0< s< T<\infty \), \(t-s\le 1\), \(\frac {1}{n}+1/q=1\), then from (5.1)–(5.4), we obtain

$$\begin{aligned} & E \bigl\vert x ( t ) -x ( s ) \bigr\vert ^{n}\\ &\quad\le 2^{n-1}E \biggl( \int _{s}^{t}{ \vert f_{1} \vert \,dv} \biggr) ^{n}+2^{n-1}E \biggl( \int _{0}^{t}{ \vert g_{1} \vert \,dB ( v )} \biggr) ^{n}+2^{n-1}E \biggl(\biggl| \int _{s}^{t}{ \int _{\mathbb{Z}}{h_{1}\tilde{N} ( dv,du )}\biggr|^{n}} \biggr) \\ &\quad\le 2^{n-1} \biggl( \int _{s}^{t}{1^{q}}\,dv \biggr) ^{\frac{n}{q}}E \biggl( \int _{s}^{t}{ \vert f_{1} \vert ^{n}\,dv} \biggr) +2^{n-1} \biggl[ \frac{n ( n-1 )}{2} \biggr]^{\frac{n}{2}} (t-s )^{\frac{n-2}{2}} \int _{s}^{t}{E|g_{1}}|^{n} \,dB ( v ) \\ &\qquad{}+2^{n-1} \bigl\{ 2^{n-1} ( t-s ) ^{\frac{p}{2}}K_{3}^{ \frac{n}{2}}M_{1} ( n ) +2^{n-1} ( t-s ) K_{3}^{ \frac{n}{2}}M_{1} ( n ) \bigr\} \\ &\quad= 2^{n-1} ( t-s ) ^{ ( n-1 ) +1}R_{1} ( n ) +2^{n-1} \biggl[ \frac{n ( n-1 )}{2} \biggr] ^{ \frac{n}{2}} ( t-s ) ^{\frac{n-2}{2}+1}R_{2} ( n ) \\ &\qquad{}+2^{n-1} \bigl\{ 2^{n-1} ( t-s ) ^{\frac{n}{2}}K_{3}^{ \frac{n}{2}}M_{1} ( n ) +2^{n-1} ( t-s ) K_{3}^{ \frac{p}{2}}M_{1} ( p ) \bigr\} \\ &\quad \le 2^{n-1} ( t-s ) ^{\frac{n}{2}} \biggl\{ ( t-s ) ^{\frac{n}{2}}R_{1} ( n ) + \biggl[ \frac{n ( n-1 )}{2} \biggr] ^{\frac{p}{2}}R_{2} ( n )+2^{n-1}K_{3}^{\frac{n}{2}}M_{1} (n ) \\ &\qquad{}+2^{n-1} ( t-s ) ^{\frac{n}{2}}K_{3}^{\frac{n}{2}}M_{1} (n ) \biggr\} \\ &\quad\le 2^{n-1} ( t-s ) ^{\frac{n}{2}}R(n), \end{aligned}$$

where \(R(n)= ( t-s ) ^{\frac{n}{2}}R_{1}(n)+ [ \frac{n ( n-1 )}{2} ] ^{\frac{n}{2}}R_{2}(n)+2^{n-1}K_{3}^{ \frac{n}{2}}M_{1} ( n ) +2^{n-1} ( t-s ) ^{ \frac{n}{2}}K_{3}^{\frac{n}{2}}M_{1} ( n )< \infty \). Then it follows from Lemma 5.1 that almost every sample path of \(x(t)\) is locally but uniformly Hölder continuous with exponent κ for every \(\kappa \in (0,(n-2)/(2n))\). Therefore almost every sample path of \(x(t)\) is uniformly continuous on \(t\ge 0\). From a similar discussion about \(E|y(t)-y(s)|^{n}\), we can conclude that almost every sample path of \(x(t)\) and \(y(t)\) is uniformly continuous on \(t\ge 0\). Next, let \(X_{1}(t)= (x_{1}(t),y_{1}(t) )\) and \(X_{2}(t)= (x_{2}(t),y_{2}(t) )\) be any two solutions of system (1.3) with positive initial data. Consider a Lyapunov function \(V_{3}(t)\) defined by

$$\begin{aligned} V_{3}(t)= \bigl\vert \ln x_{1}(t)-\ln x_{2}(t) \bigr\vert + \bigl\vert \ln y_{1}(t)-\ln y_{2}(t) \bigr\vert ,\quad t \ge 0. \end{aligned}$$
(5.5)

Making use of Itô’s formula with jumps for (5.5), one can deduce that

$$ \begin{aligned} d^{+}V_{3}(t)={}& \operatorname{sgn} \bigl(x_{1}(t)-x_{2}(t) \bigr) \biggl\{ \alpha \biggl( \frac{1}{1+ky_{1}(t)}-\frac{1}{1+ky_{2}(t)} \biggr)-b \bigl( x_{1}(t)-x_{2}(t) \bigr) \\ &{}-\beta \biggl( \frac{y_{1}(t)}{1+mx_{1}(t)+ny_{1}(t)}- \frac{y_{2}(t)}{1+mx_{2}(t)+ny_{2}(t)} \biggr) \biggr\} \,dt \\ &{}+\operatorname{sgn} \bigl( y_{1}(t) -y_{2}(t) \bigr) \biggl\{ -d \bigl( y_{1}(t) -y_{2}(t) \bigr) \\ &{}+\theta \beta \biggl( \frac{x_{1}(t)}{1+mx_{1}(t)+ny_{1}(t)}- \frac{x_{2}(t)}{1+mx_{2}(t)+ny_{2}(t)} \biggr) \biggr\} \,dt. \end{aligned} $$

Integrating from 0 to t and taking expectations yields

$$ \begin{aligned} &E \bigl( V_{3}(t) \bigr)-E \bigl( V_{3} ( 0 ) \bigr) \\ &\quad= E \int _{0}^{t} \biggl\{ \operatorname{sgn} \bigl( x_{1}(s) -x_{2}(s) \bigr) \biggl[ \alpha \biggl( \frac{1}{1+ky_{1}(s)}- \frac{1}{1+ky_{2}(s)} \biggr)-b \bigl( x_{1}(s) -x_{2}(s) \bigr) \\ &\qquad{}-\beta \biggl(\frac{y_{1}(s)}{1+mx_{1}(s)+ny_{1}(s)}- \frac{y_{2}(s)}{1+mx_{2}(s)+ny_{2}(s)} \biggr) \biggr] \\ &\qquad{}+\operatorname{sgn} \bigl( y_{1}(s) -y_{2}(s) \bigr) \biggl[ -d \bigl( y_{1}(s) -y_{2}(s) \bigr) \\ &\qquad{}+\theta \beta \biggl(\frac{x_{1}(s)}{1+mx_{1}(s)+ny_{1}(s)}- \frac{x_{2}(s)}{1+mx_{2}(s)+ny_{2}(s)} \biggr) \biggr] \biggr\} \,ds. \end{aligned} $$

Thus,

$$ \begin{aligned} dE \bigl( V(t) \bigr) \le {}& \alpha E \biggl( \frac{1}{1+ky_{1}(t)}- \frac{1}{1+ky_{2}(t)} \biggr) -bE \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr)-dE \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr) \\ &{}-\beta E \biggl(\frac{y_{1}(t)}{1+mx_{1}(t)+ny_{1}(t)}- \frac{y_{2}(t)}{1+mx_{2}(t)+ny_{2}(t)} \biggr) \\ &{}+\theta \beta E \biggl(\frac{x_{1}(t)}{1+mx_{1}(t)+ny_{1}(t)}- \frac{x_{2}(t)}{1+mx_{2}(t)+ny_{2}(t)} \biggr) \\ \le{} & \alpha kE \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr) -bE \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr)-dE \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr) \\ &{}+\beta E \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr) +\beta m E \bigl( \bigl\vert x_{2}(t)y_{1}(t) -x_{1}(t) y_{2}(t) \bigr\vert \bigr) \\ &{}+\theta \beta E \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr)+\theta \beta nE \bigl( \bigl\vert x_{1}(t)y_{2}(t) -x_{2}(t) y_{1}(t) \bigr\vert \bigr) \\ \le {}& \bigl( -b+\theta \beta +(\beta m+\theta \beta n)E \bigl( y_{1}(t) \bigr) \bigr) E \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr) \\ &{}+ \bigl( -d+\alpha k+\beta +(\beta m+\theta \beta n)E \bigl( x_{1}(t) \bigr) \bigr) E \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr) \\ \le{} & \bigl( -b+\theta \beta +(\beta m+\theta \beta n)E \bigl( y_{1}^{3}(t) \bigr) ^{\frac{1}{3}} \bigr) E \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr) \\ &{}+ \bigl( -d+\alpha k+\beta +(\beta m+\theta \beta n)E \bigl( x_{1}^{3}(t) \bigr) ^{\frac{1}{3}} \bigr) E \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr). \end{aligned} $$

By Theorem 3.2, we have

$$ \begin{aligned} &E \bigl( x_{1}^{3}(t) \bigr) ^{\frac{1}{3}}\le \frac{ [ \frac{3}{4} ( \frac{1+\tilde{\gamma}_{13}}{3}+\alpha +\sigma _{1}^{2} ) ] ^{\frac{4}{3}}}{b}:=G_{1}, \\ &E \bigl( y_{1}^{3}(t) \bigr) ^{\frac{1}{3}}\le \frac{ [ \frac{3}{4} ( \frac{1+\tilde{\gamma}_{23}}{3}+\frac{\theta \beta}{m}+\sigma _{2}^{2} ) ] ^{\frac{4}{3}}}{d}:=G_{2}. \end{aligned} $$

So,

$$ \begin{aligned} \frac{dE ( V_{3}(t) )}{dt}\le & \bigl( -b+\theta \beta +( \beta m+\theta \beta n)G_{2} \bigr) E \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr) \\ &+ \bigl( -d+\alpha k+\beta +(\beta m+\theta \beta n)G_{1} \bigr) E \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr). \end{aligned} $$

By the condition of Theorem 5.1,

$$ \begin{aligned} E \bigl( V_{3}(t) \bigr) \le{} & V_{3} ( 0 ) - \bigl(b- \theta \beta -(\beta m+\theta \beta n)G_{2} \bigr) \int _{0}^{t}{E \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr)}\,ds \\ &{}- \bigl(d-\alpha k-\beta -(\beta m+\theta \beta n)G_{1} \bigr) \int _{0}^{t}{E \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr)}\,ds \\ < {}& \infty. \end{aligned} $$

According to \(V(t) \ge 0\), \(E|x_{1}(t)-x_{2}(t)|\in L^{1} [ 0,\infty )\) and \(E|y_{1}(t)-y_{2}(t)|\in L^{1} [ 0,\infty )\), therefore,

$$ \begin{aligned} E \bigl( \bigl\vert \bigl( x_{1}(t),y_{1} ( y ) \bigr) - \bigl( x_{2}(t) ,y_{2}(t) \bigr) \bigr\vert \bigr) =& E \bigl\{ \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert ^{2}+ \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert ^{2} \bigr) ^{\frac{1}{2}} \bigr\} \\ \le & E \bigl( \bigl\vert x_{1}(t) -x_{2}(t) \bigr\vert \bigr) +E \bigl( \bigl\vert y_{1}(t) -y_{2}(t) \bigr\vert \bigr) \\ \in & L^{1} [0,\infty ). \end{aligned} $$

Then, by Lemma 5.1 and \(X_{1}(t)=(x_{1}(t), y_{1}(t))\) is uniformly continuous on \(t\ge 0\), we have

$$ \begin{aligned} \underset{t\rightarrow +\infty}{\lim}E \bigl( \bigl\vert X_{1}(t)-X_{2}(t) \bigr\vert \bigr) =0. \end{aligned} $$
(5.6)

And finally, let \(p(t,X(0),\mathcal{B})\) denote the transition probability of the event \({X(t;X(0))\in \mathcal{B}}\), where \(\mathcal{B}\) is a Borel measurable set of \(\mathbb{R}_{+}^{2}\). Let \(\mathcal{P} ( \mathbb{R}_{+}^{2} )\) denote all probability measures on \(\mathbb{R}_{+}^{2}\). For any \(\mathcal{P}_{1}, \mathcal{P}_{2} \in \mathcal{P} ( \mathbb{R}_{+}^{2} )\), we define metric \(d_{\mathcal{BL}}\) as follows:

$$ d_{\mathcal{BL}} ( \mathcal{P}_{1},\mathcal{P}_{2} ) = \underset{g\in \mathcal{BL}}{\sup} \biggl\vert \int _{\mathbb{R}_{+}^{2}}{g ( X )}\mathcal{P}_{1} ( dX ) - \int _{ \mathbb{R}_{+}^{2}}{g ( X )}\mathcal{P}_{2} ( dX ) \biggr\vert , $$

where \(\mathcal{BL}= \{ g:\mathbb{R}_{+}^{2}\rightarrow \mathbb{R}:|g ( X ) -g ( Y ) |\le \Vert X-Y\Vert,|g ( \cdot ) \le 1 \} \). First, we prove \(p((t,X(0),\mathcal{B}):t\ge 0)\) is Cauchy in the space \(\mathcal{P} ( \mathbb{R}_{+}^{2} )\) with metric \(d_{\mathcal{BL}}\). According to Theorem 3.2 and Chebyshev inequality, \({p((t,X(0),\mathcal{B}):t\ge 0)}\) is tight. For any \(g\in \mathcal{BL}\) and \(t,s >0\), we have

$$ \begin{aligned} &\bigl\vert Eg \bigl( X\bigl( X(0);t+s \bigr) \bigr) -Eg \bigl( X\bigl(X(0);t\bigr) \bigr) \bigr\vert \\ &\quad= \bigl\vert E \bigl[ E\bigl( g\bigl( X\bigl( X( 0 );t+s \bigr) \bigr) | \mathcal{F}_{s}\bigr) \bigr] -Eg\bigl( X\bigl( X( 0 );t \bigr) \bigr) \bigr\vert \\ &\quad= \biggl\vert \int _{\mathbb{R}_{+}^{2}}{Eg\bigl( X\bigl( \tilde{X}( 0 );t \bigr) \bigr)}p \bigl( s,X( 0),d\tilde{X}(0) \bigr) -Eg\bigl( X\bigl( X( 0);t \bigr) \bigr) \biggr\vert \\ &\quad\le \int _{\mathbb{R}_{+}^{2}}{|Eg\bigl( X\bigl( \tilde{X}( 0 );t\bigr) \bigr)}-Eg \bigl( X\bigl( X( 0);t \bigr) \bigr)|p\bigl( s,X( 0 ),d\tilde{X}( 0 ) \bigr). \end{aligned} $$

It follows from (5.6) that there is a constant \(T\ge 0\) such that

$$ \begin{aligned} \underset{g\in \mathcal{BL}}{\sup} \bigl\vert Eg \bigl(X\bigl(\tilde{X}(0);t\bigr)\bigr) -Eg\bigl( X\bigl( X(0);t \bigr) \bigr) \bigr\vert \le \varepsilon,\quad \forall t\ge T. \end{aligned} $$

Thanks to the arbitrariness of g, we have

$$ \begin{aligned} \underset{\mathcal{BL}}{\sup} \bigl\vert Eg \bigl( X(0);t+s \bigr) -Eg \bigl( X(0);t \bigr) \bigr\vert \le \varepsilon _{1},\quad \forall t\ge T,s>0. \end{aligned} $$
(5.7)

(5.7) is equivalent to

$$ d_{\mathcal{BL}}\bigl(p\bigl(t+s,X(0),\cdot \bigr),p\bigl(t,X(0),\cdot \bigr) \bigr) \le \varepsilon,\quad \forall t\ge T,s>0.$$

Therefore, the transition probability \(p((t,X(0),\cdot ):t\ge 0)\) of the solution of system (1.3) is Cauchy in the space \(\mathcal{P}( \mathbb{R}_{+}^{2} )\) with metric \(d_{\mathcal{BL}}\). So there is a unique probability measure \(\mu (\cdot )\) such that

$$ \begin{aligned} \lim_{t\rightarrow \infty}d_{\mathcal{BL}}\bigl( \mathcal{P}(t,0,\cdot ), \mu (\cdot )\bigr) =0. \end{aligned} $$
(5.8)

Then, for any fixed \(X( 0 ) \in \mathbb{R}_{+}^{2}\), combining with (5.7) and (5.8), we have

$$ \lim_{t\rightarrow \infty}d_{\mathcal{BL}}\bigl(\mathcal{P}\bigl(t,X(0), \cdot \bigr),\mu (\cdot )\bigr) \le \lim_{t\rightarrow \infty} \bigl[ d_{ \mathcal{BL}}\bigl( \mathcal{P}(t,0,\cdot ),\mu (\cdot )\bigr) +d_{ \mathcal{BL}}\bigl(\mathcal{P}(t, 0,\cdot ),\mathcal{P}\bigl( t,X(0),\cdot \bigr)\bigr) \bigr].$$

That is,

$$ \begin{aligned} \lim_{t\rightarrow \infty}d_{\mathcal{BL}}\bigl( \mathcal{P}\bigl( t,X( 0 ), \cdot \bigr),\mu ( \cdot ) \bigr) =0. \end{aligned} $$

The proof is complete. □

6 Numerical simulations

In this section, computer-based simulations are performed to validate our analytical findings obtained in the previous sections. The numerical simulations and figures have been done using MATLAB R2013a. In this section, we always take the following parameter values:

$$\begin{aligned} &\alpha =1,\qquad b=0.3,\qquad c=0.1,\qquad d=0.4, \\ & k=0.1,\qquad \beta =0.4, \qquad\theta =0.7,\qquad m=1.5,\\ & n=0.5,\qquad \gamma _{1}=\gamma _{2}=0.15, \end{aligned}$$
(6.1)

and \(\mathbb{Z}=(0,+\infty )\), \(\lambda (\mathbb{Z})=1\) with initial value \((x(0),y(0))=(0.6,0.6)\). First, we chose \(\sigma _{1}=\sigma _{2}=0.08\), then verified that \(Q_{3} =0.7364>0\), i.e., the condition of weak persistence of prey of Theorem 4.1 holds here. Similarly, the condition of weak persistence of predator species of Theorem 4.2 also holds. Hence system (1.3) is weakly persistent. Figure 1 depicts the fact.

Figure 1
figure 1

Mean weakly persistent images of two species in system (1.3). (a) and (b) represent the trajectories for the prey and predator populations respectively of the stochastic system (1.3)

Secondly, in order to obtain deep insights of the influences of Lévy noise, we keep the model parameter values the same but let \(\gamma _{1}=\gamma _{2}=0\), that is, without Lévy jumps. We can obtain that system (1.3) is persistent. Comparing Fig. 1 with Fig. 2, it is found that the Lévy jumps promote the survival of both prey and predator populations to a certain extent. In the absence of Lévy noise, the number of predator populations remains at a low level and the fluctuation frequency is small.

Figure 2
figure 2

Population state diagram of system (1.2). (a) and (b) represent time evolution of the prey and predator populations respectively of system (1.2); the only difference from system (1.3) is that there is no Lévy noise. Compared with Fig. 1, we can find that Lévy noise promotes the survival of species

Now we consider some other cases. We consider system (1.3) with the intensity of white noise as \(\sigma _{1}=1.5\), \(\sigma _{2}=0.5\) and keep the rest of the parameters consistent with (6.1). Then we get \(Q_{1}=-0.125<0\) and \(b Q_{2} +\theta \beta Q_{1}=-0.1025<0\). So the conditions of prey and predator extinction in Theorems 4.1 and 4.2 are satisfied. We exhibit the fact by Fig. 3.

Figure 3
figure 3

Time evolution of the stochastic predator–prey system (1.3) when the intensity of noise is high. These figures clearly show that high intensity of noise leads to the extinction of both species

Next, in order to illustrate the influence of fear effect on model (1.3) through numerical simulation, we choose different values of k, say \(k=0\), \(k=0.5\), and \(k=1\). For the remaining parameter values, we keep them the same as in (6.1). We can check that these parameters satisfy the condition of population persistence, which is depicted in Fig. 4. From Fig. 4 we find that the increase of fear effect will reduce the density of prey, the number of predators also decreased, but the magnitude of the change was less than that of the prey.

Figure 4
figure 4

Population state diagrams under different fear levels. (a) and (b) represent the numerical simulation for model (1.3) with initial value \((x(0),y(0))=(0.6,0.6)\) and different \(k=0\), \(k=0.5\), and \(k=1\), respectively. Other parameters are taken as in (6.1)

Finally, we numerically simulate the stability in distribution of system (1.3). We choose the parameters \(\alpha =0.8\), \(b=0.5\), \(d=1\), \(\theta =0.5\), \(m=0.5\), and for the remaining parameter values, we keep them the same as in (6.1). We can compute that \(b-\theta \beta -(\beta m+\theta \beta n) G_{2}=0.1574>0\) and \(d-\alpha k-\beta -(\beta m+\theta \beta n) G_{1}=0.0198>0\), which means that these parameters satisfy the condition of Theorem 5.1. So we obtain that system (1.3) is asymptotically stable in distribution by Theorem 5.1, shown in Fig. 5.

Figure 5
figure 5

The population dynamics of system (1.3), which shows that (1.3) is asymptotically stable in distribution when the condition of Theorem 5.1 is satisfied. The parameters \(\alpha =0.8\), \(b=0.5\), \(d=1\), \(\theta =0.5\), \(m=0.5\), others are the same as in (6.1)

7 Conclusion

This paper focuses on a stochastic Beddington–DeAngelis prey–predator model with fear effect and Lévy noise. Mathematically, we have shown that the solution of the stochastic system will not explode at a finite time, and without any parametric restriction the system will possess a unique global solution starting from any interior of the positive quadrant. We also establish some sufficient conditions for the stochastic persistence and the extinction of both species in view of parametric restrictions and noise intensity. The sufficient criteria for the asymptotic stability in distribution of the model have been obtained (see Theorem 5.1 and Fig. 5). Ecologically, we get the following conclusions:

  1. (1)

    By Theorems 4.1 and 4.2, a random disturbance may change the dynamical behaviors of the population. Especially when the noise is large, it may lead to the extinction of the prey and predator populations, see Fig. 3.

  2. (2)

    The Lévy jumps promote the survival of species. In the absence of the Lévy noise, the number of predator populations remain at a low level and the fluctuation frequency is small, see Fig. 1 and Fig. 2.

  3. (3)

    The level of fear also has significant effects on the dynamics of system (1.3). We find that the increasing of the fear effect k will lead to a decrease in the density of prey and predator, but the range of change of predator is less than that of prey, see Fig. 4.

There are some interesting themes worthy of further research. On the one hand, we can consider some other functional response into model (1.3), such as a ratio-dependent functional response and so on. On the other hand, to make model (1.3) be more realistic, we can further consider the factors such as the influence of impulsive perturbations and delay. In addition, the Lévy jump process is very useful in other scenarios as well (for example, foragers have maximized abundances when individuals perform scale-free Lévy flights [41]). We leave these for future investigations.

Availability of data and materials

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Brauer, F., Castillo-Chvez, C.: Mathematical Models in Population Biology and Epidemiology. Springer, New York (2001)

    Book  MATH  Google Scholar 

  2. Cresswell, W.: Predation in bird populations. J. Ornithol. 152(S1), 251–263 (2011)

    Article  Google Scholar 

  3. Jia, D., Zhang, T., Yuan, S.: Pattern dynamics of a diffusive toxin producing phytoplankton-zooplankton model with three-dimensional patch. Int. J. Bifurc. Chaos 29, 1930011 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  4. Xia, Y., Yuan, S.: Survival analysis of a stochastic predator-prey model with prey refuge and fear effect. J. Biol. Dyn. 14(1), 871–892 (2020)

    Article  MathSciNet  Google Scholar 

  5. Creel, S., Christianson, D.: Relationships between direct predation and risk effects. Trends Ecol. Evol. 23, 194–201 (2008)

    Article  Google Scholar 

  6. Lima, S.L.: Nonlethal effects in the ecology of predator-prey interactions. Bioscience 48, 25–34 (1998)

    Article  Google Scholar 

  7. Lima, S.L.: Predators and the breeding bird: behavioural and reproductive flexibility under the risk of predation. Biol. Rev. 84, 485–513 (2009)

    Article  Google Scholar 

  8. Creel, S., Christianson, D., Liley, S., Winnie, J.A.: Predation risk affects reproductive physiology and demography of elk. Science 315, 960 (2007)

    Article  Google Scholar 

  9. Peacor, S.D., Peckarsky, B.L., Trussell, G.C., Vonesh, J.R.: Costs of predator-induced phenotypic plasticity: a graphical model for predicting the contribution of nonconsumptive and consumptive effects of predators on prey. Oecologia 171, 1–10 (2013)

    Article  Google Scholar 

  10. Preisser, E.L., Bolnick, D.I.: The many faces of fear: comparing the pathways and impacts of nonconsumptive predator effects on prey populations. PLoS ONE 3, e2465 (2008)

    Article  Google Scholar 

  11. Pettorelli, N., Coulson, T., Durant, S.M., Gaillard, J.M.: Predation, individual variability and vertebrate population dynamics. Oecologia 167, 305–314 (2011)

    Article  Google Scholar 

  12. Zanette, L.Y., White, A.F., Allen, M.C., Clinchy, M.: Perceived predation risk reduces the number of offspring songbirds produce per year. Science 334, 1398–1401 (2011)

    Article  Google Scholar 

  13. DeAngelis, D.L., Goldstein, R.A., O’Neill, R.V.: A model for tropic interaction. Ecology 56, 881–892 (1975)

    Article  Google Scholar 

  14. Beddington, J.R.: Mutual interference between parasites or predators and its effect on searching efficiency. J. Anim. Ecol. 44, 331–340 (1975)

    Article  Google Scholar 

  15. Pal, S., Majhi, S., Mandal, S., Pal, N.: Role of fear in a predator-prey model with Beddington–DeAngelis functional response. Z. Naturforsch. A 74(7), 581–595 (2019)

    Article  Google Scholar 

  16. Han, R., Guin, L.N., Dai, B.: Consequences of refuge and diffusion in a spatiotemporal predator-prey model. Nonlinear Anal., Real World Appl. 60, 103311 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  17. Sarkar, K., Khajanchi, S.: Impact of fear effect on the growth of prey in a predator-prey interaction model. Ecol. Complex. 42, 100826 (2020)

    Article  Google Scholar 

  18. Roy, J., Alam, S.: Fear factor in a predator-prey system in deterministic and stochastic environment. Phys. A, Stat. Mech. Appl. 541, 123359 (2020)

    Article  MATH  Google Scholar 

  19. Meng, L., Ke, W.: Population dynamical behavior of Lotka–Volterra cooperative systems with random perturbations. Discrete Contin. Dyn. Syst., Ser. A 33(6), 2495–2522 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  20. Mao, X., Li, X.: Population dynamical behavior of non-autonomous Lotka–Volterra competitive system with random perturbation. Discrete Contin. Dyn. Syst. 24, 523–545 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  21. Ji, C., Jiang, D.Q., Shi, N.Z.: Analysis of a predator-prey model with modified Leslie–Gower and Holling-type II schemes with stochastic perturbation. J. Math. Anal. Appl. 359, 482–489 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  22. Jiang, D.Q., Shi, N.Z., Li, X.Y.: Global stability and stochastic permanence of a non-autonomous logistic equation with random perturbation. J. Math. Anal. Appl. 340, 588–597 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  23. Durrett, H.: Stochastic Calculus: A Practical Introduction, Probability Stochastics, 1st edn. CRC Press, Boca Raton (1996)

    MATH  Google Scholar 

  24. Liu, Q., Jiang, D., Shi, N., Hayat, T., Alsaedi, A.: Stochastic mutualism model with Lévy jumps. Commun. Nonlinear Sci. Numer. Simul. 43, 78–90 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  25. Guo, Y.: Stochastic regime switching SIR model driven by Lévy noise. Physica A 479, 1–11 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  26. Bao, J., Mao, X., Yin, G., Yuan, C.: Competitive Lotka–Volterra population dynamics with jumps. Nonlinear Anal. 74, 6601–6616 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  27. Scheffer, M., Carpenter, S., Foley, J.A., Folke, C.: Catastrophic shifts in ecosystems. Nature 413, 591–596 (2001)

    Article  Google Scholar 

  28. Alley, R.B.: Abrupt climate change. Science 299, 2005–2010 (5615)

    Article  Google Scholar 

  29. Zhou, Y., Yuan, S.: Threshold behavior of a stochastic SIS model with Lévy jumps. Appl. Math. Comput. 275, 255–267 (2016)

    MathSciNet  MATH  Google Scholar 

  30. Zhao, Y., Yuan, S.: The effect of Lévy noise on the survival of a stochastic competitive model in an impulsive polluted environment. Appl. Math. Model. 40(17–18), 7583–7600 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  31. Zhao, Y.: Stability in distribution of a stochastic hybrid competitive Lotka–Volterra model with Lévy jumps. Chaos Solitons Fractals 85, 98–109 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  32. Kunita, H.: Itô’s stochastic calculus: its surprising power for applications. Stoch. Process. Appl. 120, 622–652 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  33. Applebaum, H.: Lévy Processes and Stochastic Calculus, 2nd edn. Cambridge University Press, Cambridge (2009)

    Book  MATH  Google Scholar 

  34. Liu, M., Wang, K., Wu, Q.: Survival analysis of stochastic competitive models in a polluted environment and stochastic competitive exclusion principle. Bull. Math. Biol. 73, 1969–2012 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  35. Liptser, R.: A strong law of large numbers for local martingales. Stochastics 3, 217–228 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  36. Cheng, H.: Stochastic population systems. Stoch. Anal. Appl. 27, 854–874 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  37. Mao, X.: Stochastic versions of the LaSalle theorem. J. Differ. Equ. 153(1), 175–195 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  38. Karatzas, I., Shreve, S.E.: Brownian Motion and Stochastic Calculus. Springer, Berlin (1991)

    MATH  Google Scholar 

  39. Friedman, A.: Stochastic Differential Equations and Their Applications. Academic Press, New York (1976)

    MATH  Google Scholar 

  40. Privault, N.: Stochastic SIR Lévy jump model with heavy-tailed increments. J. Nonlinear Sci. 31, 15 (2021)

    Article  MATH  Google Scholar 

  41. Dannemann, T., Boyer, D.: Lévy flight movements prevent extinctions and maximize population abundances in fragile Lotka Volterra systems. Proc. Natl. Acad. Sci. 115(15), 3794–3799 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful to the anonymous reviewers for their helpful, valuable comments and suggestions in the improvement of this manuscript.

Funding

This work is funded by NSF of China (No. 11861027).

Author information

Authors and Affiliations

Authors

Contributions

RX carried out all the analyses and drafted the manuscript. MC helped in the analyses. YS participated in the conception of the study. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yuanfu Shao.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

All authors provide consent for publications.

Competing interests

The authors declare that they have no competing interests in this paper.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, R., Shao, Y. & Cui, M. Analysis of a stochastic predator–prey system with fear effect and Lévy noise. Adv Cont Discr Mod 2022, 72 (2022). https://doi.org/10.1186/s13662-022-03749-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13662-022-03749-x

Keywords