According to Theorem 3, we can easily find that if only free diffusion is introduced to the ODE system (9), the uniform positive stationary solution is also locally stable, which means that under conditions (10), only the PDE with no selfdiffusion and crossdiffusion cannot induce Turing instability. In this section, we will mainly discuss the case which is a wellknown phenomenon of crossdiffusion driven instability. Now let us investigate the effects of selfdiffusion and crossdiffusion for system (1).
For simplicity, we denote that \(\mathbf{K(u)}=(d_{1} u_{1} +k_{11}u _{1}^{2}+k_{12}u_{1} u_{2}+\frac{k_{13}u_{1}}{\beta + u_{3}}, d_{2} u _{2}+k_{21}u_{1} u_{2}+k_{22}u_{2}^{2}+k_{23}u_{2} u_{3}, d_{3} u_{3}+\frac{k _{31}}{m+u_{1}}u_{2} u_{3}+k_{33}u_{3}^{2} )^{T}\). Linearizing system (1) at \(\mathbf{u^{*}}\), we have
$$\begin{aligned} \textstyle\begin{cases} \frac{\mathbf{\partial u}}{\mathbf{\partial t}}=(\mathbf{K_{u}}\Delta +\mathbf{F_{u}(u^{*})})\mathbf{u}, &x\in \Omega,t>0, \\ \mathbf{\partial _{n} u}=0, &x\in \partial \Omega, t>0, \\ \mathbf{u}(x,0)=(u_{10}(x),u_{20}(x),u_{30}(x))^{T}, &x\in \Omega, \end{cases}\displaystyle \end{aligned}$$
(14)
where
$$\begin{aligned} &\mathbf{K_{u} \bigl(u^{*} \bigr)} \\ &\quad\!\!= \begin{pmatrix} d_{1}+2k_{11} u_{1}^{*}+k_{12}u_{2}^{*}+\frac{k_{13}}{\beta +u_{3} ^{*}} & k_{12}u_{1}^{*} &  \frac{k_{13} u_{1}^{*}}{(\beta + u_{3}^{*})^{2}} \\ k_{21} u_{2}^{*} & d_{2}+k_{21}u_{1}^{*}+2k_{22}u_{2}^{*}+k_{23}u _{3}^{*} & k_{23} u_{2}^{*} \\ \frac{k_{31}u_{2}^{*} u_{3}^{*}}{(m+ u_{1}^{*})^{2}} & \frac{k_{31}u _{3}^{*}}{m+u_{1}^{*}} & d_{3}+\frac{k_{31}u_{2}^{*}}{m+u_{1}^{*}}+2k _{33}u_{3}^{*}\! \end{pmatrix} . \end{aligned}$$
It is well known that if for each \(i\ge 1\), X is invariant under the operator \(\mathbf{K_{u}(u^{*})}\Delta + \mathbf{F_{u}(u^{*})}\), then problem (1) has a nontrivial solution of the form \(\boldsymbol{\psi }=\mathbf{c}\phi \exp (\lambda t)\) if and only if \((\lambda,\mathbf{c})\) is an eigenpair for the matrix \(\mu_{i} \mathbf{K_{u}(u^{*})}+\mathbf{F_{u}(u^{*})}\), where c is a constant vector. Therefore the equilibrium \(\mathbf{u^{*}}\) is unstable if at least one eigenvalue λ has a positive real part for some \(\mu_{i}\).
By some computations, the characteristic polynomial of \(\mu_{i} \mathbf{K_{u}(u^{*})}+\mathbf{F_{u}(u^{*})}\) is given by
$$ \psi_{i}(\lambda )=\lambda^{3}+C_{2} \lambda^{2}+C_{1} \lambda +C_{0}, $$
(15)
where
$$\begin{aligned}& C_{2}=\mu_{i} \biggl[d_{1}+d_{2}+d_{3}+2(k_{11}+k_{21})u_{1}^{*}+ \biggl(k_{12}+2k _{22}+\frac{k_{31}}{m+u_{1}^{*}} \biggr)u_{2}^{*} \\& \hphantom{ C_{2}=}{}+(k_{23}+2k_{33})u_{3}^{*}+ \frac{k _{13}}{\beta +u_{3}^{*}} \biggr]+u_{1}^{*}+u_{2}^{*}+ \frac{u_{3}^{*}}{1+ \delta u_{1}^{*}}, \\& C_{1}= \bigl(\mu_{i} L_{1}+u_{1}^{*} \bigr) \bigl(\mu_{i} L_{2}+u_{2}^{*} \bigr)+ \bigl(\mu_{i} L_{1}+u_{1}^{*} \bigr) \biggl(\mu_{i} L_{3}+\frac{u_{3}^{*}}{1+\delta u_{1}^{*}} \biggr) \\& \hphantom{ C_{1}=}{}+ \bigl(\mu_{i} L_{2}+u_{2}^{*} \bigr) \biggl(\mu_{i} L_{3}+\frac{u_{3}^{*}}{1+\delta u _{1}^{*}} \biggr) \\& \hphantom{ C_{1}=}{} \biggl(\frac{k_{13}\mu_{i} u_{1}^{*}}{(\beta +u_{3}^{*})^{2}}+\frac{\alpha \gamma u_{1}^{*} u_{2}^{*}}{(1+\gamma u_{3}^{*})^{2}} \biggr) \biggl( \frac{\mu_{i} k_{31}u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}}+\frac{\delta {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \biggr) \\& \hphantom{ C_{1}=}{} \biggl(\mu_{i} k_{12} u_{1}^{*}+ \frac{ \alpha u_{1}^{*}}{1+\gamma u_{3}^{*}} \biggr) \bigl( \mu_{i} k_{21} u_{2}^{*} \mu u_{2}^{*} \bigr) \bigl( \mu_{i} k_{23} u_{2}^{*}\eta u_{2}^{*} \bigr) \frac{\mu_{i} k_{31 u_{3}^{*}}}{m+u_{1}^{*}}, \\& C_{0}= \biggl(\mu_{i} k_{12} u_{1}^{*}+ \frac{\alpha u_{1}^{*}}{1+\gamma u _{3}^{*}} \biggr) \bigl(\mu u_{2}^{*} \mu_{i} k_{21} u_{2}^{*} \bigr) \biggl( \mu_{i} L_{3}+ \frac{u _{3}^{*}}{1+\delta u_{1}^{*}} \biggr) \\& \hphantom{C_{0}=}{}+ \bigl(\mu_{i} L_{1}+u_{1}^{*} \bigr) \bigl(\eta u_{2} ^{*}\mu_{i} k_{23}u_{2}^{*} \bigr)\frac{\mu_{i} k_{31}u_{3}^{*}}{m+u_{1} ^{*}} \\& \hphantom{C_{0}=}{} + \bigl( \mu_{i} L_{1}+u_{1}^{*} \bigr) \bigl( \mu_{i} L_{2}+u_{2}^{*} \bigr) \biggl( \mu_{i} L _{3}+\frac{u_{3}^{*}}{1+\delta u_{1}^{*}} \biggr) \\& \hphantom{C_{0}=}{}+ \biggl(\mu_{i} k_{12}u_{1}^{*}+ \frac{ \alpha u_{1}^{*}}{1+\gamma u_{3}^{*}} \biggr) \bigl(\eta u_{2}^{*} \mu_{i} k_{23}u _{2}^{*} \bigr) \biggl( \frac{\mu_{i} k_{31}u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}}+\frac{ \delta {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \biggr) \\& \hphantom{C_{0}=}{}+\frac{\mu_{i} k_{31} u_{3}^{*}}{m+u_{1}^{*}} \bigl(\mu u_{2}^{*} \mu_{i} k_{21} u_{2}^{*} \bigr) \biggl( \frac{ \mu_{i} k_{13} u_{1}^{*}}{(\beta +u_{3}^{*})^{2}}+\frac{\alpha \gamma u_{1}^{*} u_{2}^{*}}{(1+\gamma u_{3}^{*})^{2}} \biggr) \\& \hphantom{ C_{0}=}{} \bigl(\mu_{i} L_{2}+u_{2} ^{*} \bigr) \biggl(\frac{\mu_{i} k_{13} u_{1}^{*}}{(\beta +u_{3}^{*})^{2}}+\frac{ \alpha \gamma u_{1}^{*} u_{2}^{*}}{(1+\gamma u_{3}^{*})^{2}} \biggr) \biggl( \frac{ \mu_{i} k_{31}u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}}+\frac{\delta {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \biggr) . \end{aligned}$$
While
$$\begin{aligned}& L_{1}=d_{1}+2k_{11}u_{1}^{*}+k_{12}u_{2}^{*}+ \frac{k_{13}}{\beta +u _{3}^{*}}, \\& L_{2}=d_{2}+k_{21}u_{1}^{*}+2k_{22}u_{2}^{*}+k_{23}u_{3}^{*} , \\& L_{3}=d_{3}+2k_{33}u_{3}^{*}+ \frac{k_{31}u_{2}^{*}}{m+u_{1}^{*}} . \end{aligned}$$
Let \(\lambda_{1}(\mu_{i})\), \(\lambda_{2} (\mu_{i})\), \(\lambda_{3} (\mu _{i})\) be the three roots of \(\psi_{i} (\lambda )=0\), then \(\lambda _{1}(\mu_{i})\lambda_{2}(\mu_{i})\lambda_{3}(\mu_{i})=C_{0}\). In order to have at least one \(\operatorname{Re}\lambda_{j}(\mu_{i})>0\), it is sufficient that \(C_{0}<0\).
In the following we shall find out the conditions such that \(C_{0}<0\). In fact \(C_{0}=\operatorname{det}(\mu_{i}\mathbf{K_{u}(u^{*})} \mathbf{F_{u}(u^{*})})\). By simple computation, it follows that
$$ C_{0}=P_{3} \mu_{i}^{3}+P_{2} \mu_{i}^{2}+P_{1}\mu_{i} \operatorname{det} \bigl( \mathbf{F_{u} \bigl(u^{*} \bigr)} \bigr), $$
(16)
where
$$\begin{aligned}& P_{3}=L_{1} L_{2} L_{3} \frac{k_{12} k_{23} k_{31}u_{1}^{*} {u_{2} ^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}} \\& \hphantom{P_{3}=}{}\frac{ k_{13} k_{21}k_{31} u _{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})^{2}}\frac{ k_{13}k_{31} L _{2} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}(\beta +u_{3} ^{*})^{2}}k_{12} k_{21}L_{3} u_{1}^{*} u_{2}^{*}\frac{k_{23} k_{31} L_{1} u_{2}^{*} u_{3}^{*}}{m+u_{1}^{*}}, \\& P_{2}=L_{1} L_{3} u_{2}^{*}+L_{2} L_{3} u_{1}^{*} +\frac{L_{1} L_{2} u_{3}^{*}}{1+\delta u_{1}^{*}} + \frac{\eta k_{12} k_{31}u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}} \frac{\delta k_{12} k_{23} u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \\& \hphantom{P_{2}=}{}\frac{k_{23}k_{31} \alpha u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(1+\gamma u _{3}^{*})(m+u_{1}^{*})^{2}} \frac{k_{21} k_{31}\alpha \gamma u_{1} ^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u_{3}^{*})} +\frac{ \mu k_{13} k_{31} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})^{2}} \\& \hphantom{P_{2}=}{} \frac{k_{31} \alpha \gamma L_{2} u_{1}^{*} {u_{2} ^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2} (1+\gamma u_{3}^{*})^{2}} \frac{k _{13} k_{31} u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2} ( \beta +u_{3}^{*})^{2}} \frac{\delta k_{13} L_{2} u_{1}^{*} {u_{3}^{*}} ^{2}}{(1+\delta u_{1}^{*})^{2} (\beta +u_{3}^{*})^{2}} \\& \hphantom{P_{2}=}{}\frac{k_{12} k _{21} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}\frac{\alpha k_{21} L_{3} u_{1}^{*} u_{2}^{*}}{1+\gamma u_{3}^{*}}+\mu k_{12} L _{3} u_{1}^{*} u_{2}^{*}+ \frac{\eta k_{31}L_{1} u_{2}^{*} u_{3}^{*}}{m+u _{1}^{*}}\frac{k_{23} k_{31} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{m+u_{1} ^{*}}, \\& P_{1}=\frac{L_{1} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}} +\frac{L _{2} u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}+L_{3} u_{1}^{*} u_{2} ^{*} +\frac{k_{12}\eta \delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+ \delta u_{1}^{*})^{2}} \frac{k_{23}\alpha \delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\gamma u_{3}^{*})(1+\delta u_{1}^{*})^{2}} \\& \hphantom{P_{1}=}{}+\frac{k _{31}\alpha \eta u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(1+\gamma u_{3} ^{*})(m+u_{1}^{*})^{2}}+\frac{\alpha \gamma \mu k_{31}u_{1}^{*} {u _{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u_{3}^{*})^{2}} \frac{k _{31}\alpha \gamma u_{1}^{*} {u_{2}^{*}}^{3} u_{3}^{*}}{(m+u_{1}^{*})^{2} (1+\gamma u_{3}^{*})^{2}} \\& \hphantom{P_{3}=}{}\frac{\alpha \gamma \delta L_{2} u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u _{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} \frac{\delta k_{13} u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} \frac{ \alpha k_{21}u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})(1+ \delta u_{1}^{*})} \\& \hphantom{P_{3}=}{}+\frac{k_{12} \mu u_{1}^{*} u_{2}^{*} u_{3}^{*}}{1+ \delta u_{1}^{*}} +\frac{\mu \alpha L_{3} u_{1}^{*} u_{2}^{*}}{1+ \gamma u_{3}^{*}} +\frac{k_{31} \eta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{m+u _{1}^{*}}, \\& \operatorname{det} \bigl(\mathbf{F_{u} \bigl(u^{*} \bigr)} \bigr) =\frac{u_{1}^{*} u_{2}^{*}u_{3}^{*}}{1+ \delta u_{1}^{*}} +\frac{\delta \alpha \eta u_{1}^{*} u_{2}^{*} {u_{3} ^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(1+\gamma u_{3}^{*})} \\& \hphantom{\operatorname{det}(\mathbf{F_{u}(u^{*})})=}{}+\frac{\alpha \mu u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u _{3}^{*})}\frac{\alpha \delta \gamma u_{1}^{*} {u_{2}^{*}}^{2} {u _{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} . \end{aligned}$$
Let \(\tilde{P}(\nu )=P_{3} \nu^{3}+P_{2} \nu^{2}+P_{1}\nu \operatorname{det}(\mathbf{F_{u}(u^{*})})\), and let \(\tilde{\mu _{1}}\), \(\tilde{\mu _{2}}\), and \(\tilde{\mu _{3}}\) be the three roots of \(\tilde{P}(\nu )=0\) with \(\operatorname{Re}(\tilde{\mu }_{1})\leq \operatorname{Re}(\tilde{\mu }_{2})\leq \operatorname{Re}(\tilde{\mu } _{3})\). Then \(\tilde{\mu }_{1} \tilde{\mu }_{2}\tilde{\mu }_{3}=\operatorname{det}( \mathbf{F_{u}(u^{*})})=A_{0}<0\). Notice that direct calculations show that \(P_{3}>0\), thus one of \(\tilde{\mu }_{1}\), \(\tilde{\mu }_{2}\), \(\tilde{\mu }_{3}\) is real and negative and the product of the other two is positive.
Our following discussion will cover five cases.
Case 1
\(k_{21}\) is a variation parameter and other parameters are fixed.
Consider the following limits:
$$\begin{aligned}& \lim_{k_{21}\to \infty } \frac{P_{3}}{k_{21}}=d_{1} L_{3} u _{1}^{*}+2k_{11}L_{3} {u_{1}^{*} }^{2}+\frac{ k_{13} d_{3} u_{1}^{*}}{ \beta +u_{3}^{*}}+ \frac{2 k_{13} k_{33}u_{1}^{*} u_{3}^{*}}{\beta +u _{3}^{*}}+\frac{k_{13} k_{31}\beta u_{1}^{*} u_{2}^{*}}{(\beta +u_{3} ^{*})^{2}(m+u_{1}^{*})} \\& \hphantom{\lim_{k_{21}\to \infty } \frac{P_{3}}{k_{21}}=}{}\frac{k_{13} k_{31}{u_{1}^{*}}^{2} u_{2}^{*} u _{3}^{*}}{(m+u_{1}^{*})^{2} (\beta +u_{3}^{*})^{2}}:=h_{3}>0, \\& \lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=L_{3} {u_{1} ^{*}}^{2}+\frac{(d_{1}+2k_{11}u_{1}^{*}+\frac{k_{13}}{\beta +u_{3} ^{*}})u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}} \\& \hphantom{ \lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=}{}\frac{k_{31}\alpha \gamma u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u_{3}^{*})}  \frac{k_{31}\alpha \gamma {u_{1}^{*}}^{2} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} \\& \hphantom{ \lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=}{}\frac{k_{13}\delta {u_{1}^{*}}^{2} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} \frac{\alpha L_{3} u_{1}^{*} u_{2}^{*}}{1+ \gamma u_{3}^{*}}:=h_{2}, \\& \lim_{k_{21}\to \infty }{\frac{P_{1}}{k_{21}}= \biggl(u_{1}^{*} \frac{ \alpha \gamma \delta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}\frac{\alpha u_{2}^{*}}{1+\gamma u_{3} ^{*}} \biggr)\frac{u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}}:=h_{1}. \end{aligned}$$
Note that \(\lim_{k_{21}\to \infty }{\tilde{P}(\nu )=\nu (h_{3} \nu^{2}+h_{2} \nu +h_{1})}\). Thus, when \(u_{1}^{*}<\frac{\alpha \gamma \delta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}}+\frac{\alpha u_{2}^{*}}{1+\gamma u_{3}^{*}}\), we show that \(h_{1}<0<h_{3} \). A continuity argument shows that there exists a positive constant \(k_{21}^{*}\) such that, for \(k_{21}\geq k _{21}^{*}\), the three roots \(\tilde{\mu }_{1}\), \(\tilde{\mu }_{2}\), and \(\tilde{\mu }_{3}\) of \(\tilde{P}(\nu )=0\) are all real and satisfy
$$\begin{aligned}& \lim_{k_{21}\to \infty } {\tilde{\mu }_{1}= \frac{h_{2}\sqrt{h _{2}^{2}4h_{1} h_{3}}}{2h_{3}}}, \\& \lim_{k_{21}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{21}\to \infty } {\tilde{\mu } _{3}=\frac{h_{2}+\sqrt{h_{2}^{2}4h_{1} h_{3}}}{2h_{3}}}, \end{aligned}$$
and we can conclude that \(\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}\). Furthermore, we can obtain
$$ \tilde{P}(\nu )< 0,\quad \mbox{when }\nu \in (\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$
It is well known that since at least one eigenvalue \(\mu_{i}\) of −Δ satisfies \(\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})\) for some i, we have \(C_{0}<0\) and the number of sign changes for the characteristic polynomial \(\psi (\lambda )\) is either one or three. By using Descartes’ rule, \(\psi (\lambda )\) has at least one positive root. Therefore, we can conclude the following theorem.
Theorem 4
Suppose that
$$ u_{1}^{*}< \frac{\alpha \gamma \delta u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(1+ \delta u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}+ \frac{\alpha u_{2}^{*}}{1+ \gamma u_{3}^{*}} $$
(17)
and (10) hold. Then there exists a positive constant
\(k_{21}^{*}\)
such that, for
\(k_{21}\geq k_{21}^{*}\), the uniform stationary solution
\(\mathbf{u^{*}}\)
of system (1) is unstable.
Case 2
\(k_{23}\) is a variation parameter and other parameters are fixed.
Consider the following limits:
$$\begin{aligned}& \lim_{k_{23}\to \infty } {\frac{P_{3}}{k_{23}}=L_{1} u_{3} \bigl( d _{3}+2k_{33} u_{3}^{*} \bigr)\frac{ k_{12}k_{31} u_{1}^{*}{u_{2}^{*}}^{2} u _{3}^{*}}{(m+u_{1}^{*})^{2}}\frac{k_{13} k_{31}u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(m+u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}}}:=e_{3}>0, \\& \lim_{k_{23}\to \infty }\frac{P_{2}}{K_{23}}= \bigl(d_{3}+2k_{33}u _{3}^{*} \bigr)u_{1}^{*} u_{3}^{*}+\frac{L_{1} {u_{3}^{*}}^{2}}{1+\delta u _{1}^{*}}\frac{k_{12}\delta u_{1}^{*} u_{2}^{*} {u_{3}^{*}}^{2}}{(1+ \delta u_{1}^{*})^{2}} \\& \hphantom{\lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}=}{}\frac{(1+2\gamma u_{3}^{*})k_{31}\alpha u_{1} ^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}(m+u_{1}^{*})^{2}}\frac{k _{13}\delta u_{1}^{*} {u_{3}^{*}}^{3}}{(1+\delta u_{1}^{*})^{2}( \beta +u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{21}\to \infty }\frac{P_{2}}{K_{21}}}:=e_{2}, \\& \lim_{k_{23}\to \infty }{\frac{P_{1}}{k_{23}}= \biggl(1\frac{\alpha \delta u_{2}^{*}}{(1+\gamma u_{3}^{*})(1+\delta u_{1}^{*})} \frac{ \alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}} \biggr)\frac{u_{1}^{*} {u_{3}^{*}}^{2}}{1+\delta u _{1}^{*}}}:=e_{1} . \end{aligned}$$
Noticing \(\lim_{k_{23}\to \infty }{\tilde{P}(\nu )=\nu (e_{3} \nu^{2}+e_{2} \nu +e_{1})}\). If the parameters satisfy \(\frac{\alpha \delta u_{2}^{*}}{(1+\gamma u_{3}^{*})(1+\delta u_{1}^{*})}+\frac{ \alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}}>1\), then \(e_{1}<0<e_{3} \). A continuity argument shows that there exists a positive constant \(k_{23}^{*}\) such that, for \(k_{23}\geq k_{23}^{*}\), the three roots \(\tilde{\mu }_{1}\), \(\tilde{\mu }_{2}\), and \(\tilde{\mu }_{3}\) of \(\tilde{P}(\nu )=0\) are all real and satisfy
$$\begin{aligned}& \lim_{k_{23}\to \infty } {\tilde{\mu }_{1}= \frac{e_{2}\sqrt{e _{2}^{2}4e_{1} e_{3}}}{2e_{3}}}, \\& \lim_{k_{23}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{23}\to \infty } {\tilde{\mu } _{3}=\frac{e_{2}+\sqrt{e_{2}^{2}4e_{1} e_{3}}}{2e_{3}}}, \end{aligned}$$
and we can conclude that \(\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}\). Furthermore, we can obtain
$$ \tilde{P}(\nu )< 0, \quad \mbox{when }\nu \in (\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$
It is worth noting that since at least one eigenvalue \(\mu_{i}\) of −Δ satisfies \(\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})\) for some i, therefore \(C_{0}<0\) and the number of sign changes for the characteristic polynomial \(\psi (\lambda )\) is either one or three. By using Descartes’ rule, \(\psi (\lambda )\) has at least one positive root. Hence, we have the following theorem.
Theorem 5
If the condition
$$ \frac{\alpha \delta u_{2}^{*}}{(1+\gamma u_{3}^{*})(1+\delta u_{1} ^{*})}+\frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u _{1}^{*})(1+\gamma u_{3}^{*})^{2}}>1 $$
(18)
and (10) are fulfilled, then there exists a positive constant
\(k_{23}^{*}\)
such that the uniform stationary solution
\(\mathbf{u^{*}}\)
of system (1) is unstable provided
\(k_{23}\geq k_{23}^{*}\).
Case 3
\(k_{31}\) is a variation parameter and other parameters are fixed.
Consider the following limits:
$$\begin{aligned}& \lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=\frac{L_{1} L _{2} u_{2}^{*}}{m+u_{1}^{*}} \frac{ k_{12} k_{23} u_{1}^{*} {u_{2} ^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}}\frac{k_{13} k_{21}u_{1}^{*} u _{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})^{2}}\frac{k_{13} L _{2} u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(\beta +u_{3}^{*})^{2}(m+u_{1} ^{*})^{2}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=}{}\frac{k_{12} k_{21}u_{1}^{*} {u_{2}^{*}}^{2}}{m+u_{1}^{*}}\frac{k _{23}L_{1} u_{2}^{*} u_{3}^{*}}{m+u_{1}^{*}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=f_{3}>0, \\& \lim_{k_{31}\to \infty }\frac{P_{2}}{K_{31}}=L_{3} {u_{1} ^{*}}^{2}+\frac{(d_{1}+2k_{11}u_{1}^{*}+\frac{k_{13}}{\beta +u_{3} ^{*}})u_{1}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}\frac{k_{31}\alpha \gamma u_{1}^{*} {u_{2}^{*}}^{2} u_{3}^{*}}{(m+u_{1}^{*})(1+\gamma u _{3}^{*})} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=}{}\frac{k_{31}\alpha \gamma {u_{1}^{*}}^{2} {u_{2}^{*}}^{2} u _{3}^{*}}{(m+u_{1}^{*})^{2}(1+\gamma u_{3}^{*})^{2}} \frac{k_{13} \delta {u_{1}^{*}}^{2} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}( \beta +u_{3}^{*})^{2}}  \frac{\alpha L_{3} u_{1}^{*} u_{2}^{*}}{1+ \gamma u_{3}^{*}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=f_{2}, \\& \lim_{k_{31}\to \infty }\frac{P_{1}}{k_{31}}= \biggl(u_{2}^{*}+ \eta u_{3}^{*}+\frac{\alpha \eta u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3} ^{*})(m+u_{1}^{*})}+\frac{\mu \alpha u_{2}^{*}+2\alpha \gamma \mu u _{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}=}{} \frac{\alpha \gamma {u_{2}^{*}}^{2} u_{3}^{*}}{(m+ u_{1}^{*})(1+\gamma u_{3}^{*})^{2}} \biggr)\frac{u_{1}^{*} u_{2}^{*}}{m+u_{1}^{*}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=f_{1}. \end{aligned}$$
It is worth mentioning that \(\lim_{k_{31}\to \infty }{\tilde{P}( \nu )=\nu (f_{3}\nu^{2}+f_{2} \nu +f_{1})}\). Thus, when \(u_{2}^{*}+ \eta u_{3}^{*}+\frac{\alpha \eta u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3} ^{*})(m+u_{1}^{*})}+\frac{\mu \alpha u_{2}^{*}+2\alpha \gamma \mu u _{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}}<\frac{\alpha \gamma {u_{2}^{*}}^{2} u_{3}^{*}}{(m+ u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}\), we show that \(f_{1}<0<f_{3} \). A continuity argument shows that there exists a positive constant \(k_{31}^{*}\) such that, for \(k_{31}\geq k _{31}^{*}\), the three roots \(\tilde{\mu }_{1}\), \(\tilde{\mu }_{2}\), and \(\tilde{\mu }_{3}\) of \(\tilde{P}(\nu )=0\) are all real and satisfy
$$\begin{aligned}& \lim_{k_{31}\to \infty } {\tilde{\mu }_{1}= \frac{f_{2}\sqrt{f _{2}^{2}4f_{1} f_{3}}}{2f_{3}}}, \\& \lim_{k_{31}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{31}\to \infty } {\tilde{\mu } _{3}=\frac{f_{2}+\sqrt{f_{2}^{2}4f_{1} f_{3}}}{2f_{3}}}, \end{aligned}$$
and we can conclude that \(\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}\). Furthermore, we can obtain
$$ \tilde{P}(\nu )< 0, \quad \mbox{when }\nu \in (\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$
It is well known that since at least one eigenvalue \(\mu_{i}\) of −Δ satisfies \(\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})\) for some i, thus \(C_{0}<0\) and the number of sign changes for the characteristic polynomial \(\psi (\lambda )\) is either one or three. By using Descartes’ rule, \(\psi (\lambda )\) has at least one positive root. Therefore, we can conclude the following theorem.
Theorem 6
Suppose that
$$ u_{2}^{*}+\eta u_{3}^{*}+ \frac{\alpha \eta u_{2}^{*} u_{3}^{*}}{(1+ \gamma u_{3}^{*})(m+u_{1}^{*})}+\frac{\mu \alpha u_{2}^{*}+2\alpha \gamma \mu u_{2}^{*} u_{3}^{*}}{(1+\gamma u_{3}^{*})^{2}}< \frac{ \alpha \gamma {u_{2}^{*}}^{2} u_{3}^{*}}{(m+ u_{1}^{*})(1+\gamma u _{3}^{*})^{2}} $$
(19)
and (10) are satisfied. Then there exists a positive constant
\(k_{31}^{*}\)
such that, for
\(k_{31}\geq k_{31}^{*}\), the uniform stationary solution
\(\mathbf{u^{*}}\)
of crossdiffusion predator–prey system (1) is unstable.
Case 4
\(k_{13}\) is a variation parameter and other parameters are fixed.
Perform the following limits:
$$\begin{aligned}& \lim_{k_{13}\to \infty } \frac{P_{3}}{k_{13}}=\frac{1}{ \beta +u_{3}^{*}} \biggl(L_{2} L_{3}\frac{k_{21}k_{31}u_{1}^{*} u_{2}^{*} u _{3}^{*}}{(m+u_{1}^{*})(\beta +u_{3}^{*})}\frac{k_{31} L_{2} u_{1} ^{*} u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})^{2}(\beta +u_{3}^{*})} \frac{k _{23} k_{31}u_{2}^{*} u_{3}^{*}}{m+u_{1}^{*}} \biggr) \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}}{}:=g_{3}>0, \\& \lim_{k_{13}\to \infty }\frac{P_{2}}{k_{13}}=\frac{L_{3} u_{2}^{*}}{\beta +u_{3}^{*}}+ \frac{L_{2} u_{3}^{*}}{(\beta +u_{3}^{*})(1+ \delta u_{1}^{*})}+\frac{k_{31}\eta u_{2}^{*} u_{3}^{*}}{(m+u_{1}^{*})( \beta +u_{3}^{*})}+\frac{k_{31}\mu u_{1}^{*} u_{2}^{*} u_{3}^{*}}{(m+u _{1}^{*})(\beta +u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{13}\to \infty }\frac{P_{2}}{k_{13}}=}{}\frac{k_{31} u_{1}^{*} {u_{2}^{*}} ^{2} u_{3}^{*}}{(m+u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} \frac{\delta L_{2} u_{1}^{*} {u_{3}^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}} \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}}{} :=g_{2}, \\& \lim_{k_{13}\to \infty }{\frac{P_{1}}{k_{13}}=\frac{u_{2}^{*} u_{3}^{*}}{\beta +u_{3}^{*}} \biggl( \frac{1}{1+\delta u_{1}^{*}}+\frac{k_{31} \eta }{m+u_{1}^{*}}\frac{\delta u_{1}^{*} u_{3}^{*} }{1+\delta u_{1} ^{*}} \biggr)}:=g_{1}. \end{aligned}$$
Also notice that \(\lim_{k_{13}\to \infty }{\tilde{P}(\nu )= \nu (g_{3}\nu^{2}+g_{2} \nu +g_{1})}\). Thus, when \(\frac{k_{31}\eta (1+ \delta u_{1}^{*})}{m+ u_{1}^{*}}+1<\delta u_{1}^{*}u_{3}^{*}\), we show that \(g_{1}<0<g_{3} \). A continuity argument shows that there exists a positive constant \(k_{13}^{*}\) such that, for \(k_{13}\geq k_{13}^{*}\), the three roots \(\tilde{\mu }_{1}\), \(\tilde{\mu }_{2}\), and \(\tilde{\mu }_{3}\) of \(\tilde{P}(\nu )=0\) are all real and satisfy
$$\begin{aligned}& \lim_{k_{13}\to \infty } {\tilde{\mu }_{1}= \frac{g_{2}\sqrt{g _{2}^{2}4g_{1} g_{3}}}{2g_{3}}}, \\& \lim_{k_{13}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{13}\to \infty } {\tilde{\mu } _{3}=\frac{g_{2}+\sqrt{g_{2}^{2}4g_{1} g_{3}}}{2g_{3}}}, \end{aligned}$$
and we can conclude that \(\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}\). Furthermore, we can obtain
$$ \tilde{P}(\nu )< 0,\quad \mbox{ when }\nu \in (\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$
It is remarkable that since at least one eigenvalue \(\mu_{i}\) of −Δ satisfies \(\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})\) for some i, hence \(C_{0}<0\) and the number of sign changes for the characteristic polynomial \(\psi (\lambda )\) is either one or three. By using Descartes’ rule, \(\psi (\lambda )\) has at least one positive root. Therefore, we can obtain the following instability theorem.
Theorem 7
Assume that the parameters satisfy
$$ \frac{k_{31}\eta (1+\delta u_{1}^{*})}{m+ u_{1}^{*}}+1< \delta u_{1} ^{*}u_{3}^{*} $$
(20)
and (10). Then there exists a positive constant
\(k_{13}^{*}\)
such that, for
\(k_{13}\geq k_{13}^{*}\), the uniform stationary solution
\(\mathbf{u^{*}}\)
of crossdiffusion system with generalist (1) is unstable.
Case 5
\(k_{22}\) is a variation parameter and other parameters are fixed.
Perform the following limits:
$$\begin{aligned}& \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}=2u_{2}^{*} \bigl(d _{1}+2k_{11}u_{1}^{*}+k_{12}u_{2}^{*} \bigr) \bigl(d_{3}+2k_{33}u_{3}^{*}\bigr) +\frac{2k _{31}{u_{2}^{*}}^{2}(d_{1}+2k_{11}u_{1}^{*}+k_{12}u_{2}^{*})}{m+u_{1} ^{*}} \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}=}{}+\frac{2k_{13}u_{2}^{*}(d_{3}+2k_{33}u_{3}^{*})}{\beta +u_{3} ^{*}}+\frac{2k_{13}k_{31}{u_{2}^{*}}^{2}(m\beta +mu_{3}^{*}+\beta u_{1}^{*})}{(m+u _{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} \\& \hphantom{ \lim_{k_{22}\to \infty }\frac{P_{3}}{k_{22}}}{}:=s_{3}>0, \\& \lim_{k_{22}\to \infty }{\frac{P_{2}}{k_{22}}}=2L_{3} u_{1} ^{*} u_{2}^{*}+\frac{2L_{1} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}} \frac{k _{31}\alpha \gamma u_{1}^{*}{u_{2}^{*}}^{3} u_{3}^{*}}{(m+u_{1}^{*})^{2}(1+ \gamma u_{3}^{*})^{2}}\frac{\delta k_{13}u_{1}^{*} u_{2}^{*} {u_{3} ^{*}}^{2}}{(1+\delta u_{1}^{*})^{2}(\beta +u_{3}^{*})^{2}} \\& \hphantom{\lim_{k_{31}\to \infty }\frac{P_{3}}{k_{31}}}:=s_{2}, \\& \lim_{k_{22}\to \infty }{\frac{P_{1}}{k_{22}}=\frac{2 u_{1} ^{*} u_{2}^{*} u_{3}^{*}}{1+\delta u_{1}^{*}}\biggl(1 \frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{(1+\delta u_{1}^{*})(1+\gamma u_{3}^{*})^{2}}\biggr)}:=s _{1} . \end{aligned}$$
It is worth pointing out that \(\lim_{k_{22}\to \infty }{\tilde{P}( \nu )=\nu (s_{3}\nu^{2}+s_{2} \nu +s_{1})}\). Thus, when \(\frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{ (1+\delta u_{1}^{*})(1+\gamma u _{3}^{*})^{2}}>1\), we show that \(s_{1}<0<s_{3} \). A continuity argument shows that there exists a positive constant \(k_{22}^{*}\) such that, for \(k_{22}\geq k_{22}^{*}\), the three roots \(\tilde{\mu }_{1}\), \(\tilde{\mu }_{2}\), and \(\tilde{\mu }_{3}\) of \(\tilde{P}(\nu )=0\) are all real and satisfy
$$\begin{aligned}& \lim_{k_{22}\to \infty } {\tilde{\mu }_{1}= \frac{s_{2}\sqrt{s _{2}^{2}4s_{1} s_{3}}}{2s_{3}}}, \\& \lim_{k_{22}\to \infty } {\tilde{\mu }_{2}=0}, \\& \lim_{k_{22}\to \infty } {\tilde{\mu } _{3}=\frac{s_{2}+\sqrt{s_{2}^{2}4s_{1} s_{3}}}{2s_{3}}}, \end{aligned}$$
and we can conclude that \(\infty <\tilde{\mu }_{1}<0<\tilde{\mu } _{2}<\tilde{\mu }_{3}\). Furthermore, we can obtain
$$ \tilde{P}(\nu )< 0, \quad \mbox{when }\nu \in (\infty, \tilde{\mu }_{1}) \cup ( \tilde{\mu }_{2}, \tilde{\mu }_{3}). $$
It is clear that since at least one eigenvalue \(\mu_{i}\) of −Δ satisfies \(\mu_{i}\in (\tilde{\mu }_{2}, \tilde{\mu }_{3})\) for some i, therefore \(C_{0}<0\) and the number of sign changes for the characteristic polynomial \(\psi (\lambda )\) is either one or three. By using Descartes’ rule, \(\psi (\lambda )\) has at least one positive root. So, we can get the following instability theorem.
Theorem 8
Suppose that
$$ \frac{\alpha \gamma \delta u_{2}^{*} u_{3}^{*}}{ (1+\delta u_{1}^{*})(1+ \gamma u_{3}^{*})^{2}}>1 $$
(21)
and (10) are fulfilled. Then there exists a positive constant
\(k_{22}^{*}\)
such that, for
\(k_{22}\geq k_{22}^{*}\), the uniform stationary solution
\(\mathbf{u^{*}}\)
of (1) is unstable.
Remark

(A)
By using the arguments similar to above, we can easily know that if the other constants are fixed, whereas selfdiffusion coefficient \(k_{11}\) or \(k_{33}\) is sufficiently large, then the selfdiffusion predator–prey system cannot induce Turing instability.

(B)
\(k_{13}\), \(k_{21}\), \(k_{22}\), \(k_{23}\), \(k_{31}\) can be chosen as variation parameters since the number of sign changes for the polynomial (16) could be bigger than one for large values of \(k_{13}\), \(k _{21}\), \(k_{22}\), \(k_{23}\), \(k_{31}\). By using Descartes’ rule, polynomial (16) could have at least a positive root, which leads to linear instability.