Dynamics of a two-dimensional competitive system of rational difference equations with quadratic terms

Abstract

We investigate global dynamics of the following systems of difference equations:

$$\{\begin{array}{l}{x}_{n+1}=\frac{b_1{x}_n^2}{A_1+y_n^2},\\ y_{n+1}=\frac{a_2+c_2{y}_n^2}{x_n^2},\end{array}n=0,1,2,\dots ,$$

where the parameters $b_1$, $a_2$, $A_1$, $c_2$ are positive numbers and the initial condition $y_0$ is an arbitrary nonnegative number and $x_0$ is a positive number. We show that this system has rich dynamics which depends on the part of a parametric space. We find precisely the basins of attraction of all attractors including the points at ∞.

MSC:39A10, 39A30, 37E99, 37D10.

1 Introduction

In this paper we study the global dynamics of the following rational system of difference equations:

$$\{\begin{array}{l}{x}_{n+1}=\frac{b_1{x}_n^2}{A_1+y_n^2},\\ y_{n+1}=\frac{a_2+c_2{y}_n^2}{x_n^2},\end{array}n=0,1,2,\dots ,$$

(1)

where the parameters $b_1$, $a_2$, $A_1$, $c_2$ are positive numbers and the initial condition $y_0$ is an arbitrary nonnegative number and $x_0$ is a positive number.

The related system of difference equations

$$x_{n+1}=\frac{{\beta }_1{x}_n}{A_1+y_n},y_{n+1}=\frac{{\alpha }_2+{\gamma }_2{y}_n}{x_n},n=0,1,\dots ,$$

(2)

where the parameters $A_1$, ${\beta }_1$, ${\alpha }_2$ and ${\gamma }_2$ are positive numbers and the initial conditions $x_0>0$, $y_0\ge 0$, was considered in [1], where it was shown that this system has simple dynamics. Precisely, it was shown that system (2) has no equilibrium points if ${\beta }_1\le A_1$ and that it has a unique equilibrium point if ${\beta }_1>A_1$, in which case this equilibrium point is a saddle point. Furthermore, the following result describes the global dynamics of system (2).

Theorem 1 Consider system (2).

(1) Assume that ${\beta }_1>A_1$ and ${\gamma }_2{A}_1\ne {\alpha }_2$. Then there exist a set $\mathcal{C}\subset \mathcal{R}$ which is invariant and a subset of the basin of attraction of E. The set $\mathcal{C}$ is a graph of a strictly increasing continuous function of the first variable on an interval (and so is a manifold) and separates ℛ into two connected and invariant components, namely

$${\mathcal{W}}_{-}:=\{x\in \mathcal{R}\mathrm{\setminus }\mathcal{C}:\mathrm{\exists }y\in \mathcal{C}\mathit{\text{ with }}x{⪯}_{se}y\}\mathit{\text{and}}{\mathcal{W}}_{+}:=\{x\in \mathcal{R}\mathrm{\setminus }\mathcal{C}:\mathrm{\exists }y\in \mathcal{C}\mathit{\text{ with }}y{⪯}_{se}x\},$$

which satisfy:

$$\underset{n\to \mathrm{\infty }}{lim}(x_n,y_n)=(0,\mathrm{\infty })\mathit{\text{for every }}(x_0,y_0)\in {\mathcal{W}}_{-},$$

and

$$\underset{n\to \mathrm{\infty }}{lim}(x_n,y_n)=(\mathrm{\infty },0)\mathit{\text{for every }}(x_0,y_0)\in {\mathcal{W}}_{+}.$$

(2) Assume that ${\gamma }_2{A}_1={\alpha }_2$. Then system (2) can be decoupled as follows:

$$x_{n+1}=\frac{{\beta }_1{x}_n^2}{A_1{x}_n+{\beta }_1{\gamma }_2},y_{n+1}=\frac{1}{{\beta }_1}{y}_n(A_1+y_n),n=0,1,\dots ,$$

and every solution of this system (depending of the choice of the initial condition $(x_0,y_0)$) is either bounded and converges to an equilibrium point, or increases monotonically to infinity.

(3) Assume that ${\beta }_1\le A_1$ and ${\gamma }_2{A}_1\ne {\alpha }_2$. Every solution $\{(x_n,y_n)\}$ of system (2), with $x_0>0$, $y_0\ge 0$, satisfies

$$\underset{n\to \mathrm{\infty }}{lim}{x}_n=0\mathit{\text{and}}\underset{n\to \mathrm{\infty }}{lim}{y}_n=\mathrm{\infty }.$$

Thus every solution of system (2) either converges to the unique equilibrium point or is asymptotic to one of the points at infinity, precisely to either $(0,\mathrm{\infty })$ or to $(\mathrm{\infty },0)$. In all cases, either solution is eventually monotonic or the subsequences of even indexed and odd indexed terms are eventually monotonic. Introduction of quadratic terms into the system will substantially change the dynamics by introducing new equilibrium points (up to three) with different local character and minimal period-two solutions (up to 11). Again, most of the solutions of system (1) will be asymptotic to $(\mathrm{\infty },0)$ or $(0,\mathrm{\infty })$, but the separatrix between the two basins of attraction may consist of several global stable manifolds of either saddle point equilibrium points or non-hyperbolic equilibrium points or minimal period-two solutions. In one case, when there exists a unique non-hyperbolic equilibrium point, it is possible that this point will have a basin of attraction of positive Lebesgue measure.

System (1) is a competitive system, and our results are based on recent results about competitive systems in the plane, see [2, 3]. System (1) can be used as a mathematical model for competition in population dynamics. The first systematic study of a specific competitive system with quadratic terms was performed in [4] where system of the form

$$x_{n+1}=\frac{x_n}{a+y_n^2},y_{n+1}=\frac{y_n}{b+x_n^2},n=0,1,\dots ,$$

(3)

where the parameters $a,b>0$ and the initial conditions $x_0,y_0\ge 0$, was considered. It was shown that the dynamics of system (3) is very similar to the dynamics of the corresponding linear fractional system

$$x_{n+1}=\frac{x_n}{a+y_n},y_{n+1}=\frac{y_n}{b+x_n},n=0,1,\dots ,$$

with the same conditions on parameters and initial conditions. Both systems have nine parametric regions with different dynamical behavior.

As noted, the introduction of quadratic terms in system (2) dramatically changes the dynamics. The techniques used to study system (2) were straightforward calculations, while the techniques which will be used to study system (1) are a combination of techniques for studying real algebraic curves and implicit function theorem as neither equilibrium points nor period-two solutions are explicitly computable. Some of our calculations are performed by using Mathematica and outputs are included in the Appendix.

The paper is organized as follows. Section 2 contains some necessary results on competitive systems in the plane. Section 3 provides some basic facts about the equilibrium points and injectivity of the map associated with system (1). Section 4 contains local stability analysis of both equilibrium solutions and minimal period-two solutions. Section 5 gives global dynamics in different cases.

2 Preliminaries

A first-order system of difference equations

$$\{\begin{array}{l}{x}_{n+1}=f(x_n,y_n),\\ y_{n+1}=g(x_n,y_n),\end{array}n=0,1,2,\dots ,$$

(4)

where $\mathcal{S}\subset {\mathbb{R}}^2$, $(f,g):\mathcal{S}\to \mathcal{S}$, f, g are continuous functions, is competitive if $f(x,y)$ is non-decreasing in x and non-increasing in y, and $g(x,y)$ is non-increasing in x and non-decreasing in y. If both f and g are non-decreasing in x and y, system (4) is cooperative. Competitive and cooperative maps are defined similarly. Strongly competitive systems of difference equations or strongly competitive maps are those for which the functions f and g are coordinate-wise strictly monotone. Competitive and cooperative systems have been investigated by many authors, see [2, 3, 5–14]. Special attention to discrete competitive and cooperative systems in the plane was given in [2, 3, 5, 6, 9, 10, 14–23]. One of the reasons for paying special attention to two-dimensional discrete competitive and cooperative systems is their applicability to mathematical models in biology and economics, the former involves competition or cooperation between two species. Another reason is that the theory of two-dimensional discrete competitive and cooperative systems is very well developed, unlike such theory for three and higher dimensional systems. Part of the reason for this situation is de Mottoni-Schiaffino theorem given below, which provides relatively simple scenarios for possible behavior of many two-dimensional discrete competitive and cooperative systems. However, this does not mean that one cannot encounter chaos in such systems as has been shown by Smith, see [14].

If $\mathbf{v}=(u,v)\in {\mathbb{R}}^2$, we denote by ${\mathcal{Q}}_{\ell }(\mathbf{v})$, $\ell \in \{1,2,3,4\}$ the four quadrants in ${\mathbb{R}}^2$ relative to v, i.e., ${\mathcal{Q}}_1(\mathbf{v})=\{(x,y)\in {\mathbb{R}}^2:x\ge u,y\ge v\}$, ${\mathcal{Q}}_2(\mathbf{v})=\{(x,y)\in {\mathbb{R}}^2:x\le u,y\ge v\}$, and so on. Define the south-east partial order ${⪯}_{se}$ on ${\mathbb{R}}^2$ by $(x,y){⪯}_{se}(s,t)$ if and only if $x\le s$ and $y\ge t$. Similarly, we define the north-east partial order ${⪯}_{ne}$ on ${\mathbb{R}}^2$ by $(x,y){⪯}_{ne}(s,t)$ if and only if $x\le s$ and $y\le t$. For $\mathcal{A}\subset {\mathbb{R}}^2$ and $\mathrm{x}\in {\mathbb{R}}^2$, define the distance from x to $\mathcal{A}$ as $dist(\mathrm{x},\mathcal{A}):=inf\{\parallel \mathrm{x}-\mathrm{y}\parallel :\mathrm{y}\in \mathcal{A}\}$. By $int\mathcal{A}$ we denote the interior of a set $\mathcal{A}$.

It is easy to show that a map F is competitive if it is non-decreasing with respect to the south-east partial order, that is, if the following holds:

$$\left(\begin{array}{c}{x}^1\\ y^1\end{array}\right){⪯}_{se}\left(\begin{array}{c}{x}^2\\ y^2\end{array}\right)\Rightarrow F\left(\begin{array}{c}{x}^1\\ y^1\end{array}\right){⪯}_{se}F\left(\begin{array}{c}{x}^2\\ y^2\end{array}\right).$$

For standard definitions of attracting fixed point, saddle point, stable manifold, and related notions, see [8, 24].

We now state three results for competitive maps in the plane. The following definition is from [14].

Definition 1 Let $\mathcal{S}$ be a nonempty subset of ${\mathbb{R}}^2$. A competitive map $T:\mathcal{S}\to \mathcal{S}$ is said to satisfy condition $(O+)$ if for every x, y in $\mathcal{S}$, $T(x){⪯}_{ne}T(y)$ implies $x{⪯}_{ne}y$, and T is said to satisfy condition $(O-)$ if for every x, y in $\mathcal{S}$, $T(x){⪯}_{ne}T(y)$ implies $y{⪯}_{ne}x$.

The following theorem was proved by de Mottoni and Schiaffino [19] for the Poincaré map of a periodic competitive Lotka-Volterra system of differential equations. Smith generalized the proof to competitive and cooperative maps [12].

Theorem 2 Let $\mathcal{S}$ be a nonempty subset of ${\mathbb{R}}^2$. If T is a competitive map for which $(O+)$ holds, then, for all $x\in \mathcal{S}$, $\{T^n(x)\}$ is eventually component-wise monotone. If the orbit of x has compact closure, then it converges to a fixed point of T. If instead $(O-)$ holds, then, for all $x\in \mathcal{S}$, $\{T^{2n}(x)\}$ is eventually component-wise monotone. If the orbit of x has compact closure in $\mathcal{S}$, then its omega limit set is either a period-two orbit or a fixed point.

The following result is from [14], with the domain of the map specialized to be the cartesian product of intervals of real numbers. It gives a sufficient condition for conditions $(O+)$ and $(O-)$.

Theorem 3 Let $\mathcal{R}\subset {\mathbb{R}}^2$ be the cartesian product of two intervals in ℝ. Let $T:\mathcal{R}\to \mathcal{R}$ be a $C^1$ competitive map. If T is injective and $det{J}_T(x)>0$ for all $x\in \mathcal{R}$, then T satisfies $(O+)$. If T is injective and $det{J}_T(x)<0$ for all $x\in \mathcal{R}$, then T satisfies $(O-)$.

The following result is a direct consequence of the trichotomy theorem of Dancer and Hess, see [2] and [25], and is helpful for determining the basins of attraction of the equilibrium points.

Corollary 1 If the nonnegative cone of ⪯ is a generalized quadrant in ${\mathbb{R}}^n$, and if T has no fixed points in $〚u_1,u_2〛$ other than $u_1$ and $u_2$, then the interior of $〚u_1,u_2〛$ is either a subset of the basin of attraction of $u_1$ or a subset of the basin of attraction of $u_2$.

The next result is a well-known global attractivity result which holds in partially ordered Banach spaces as well, see [25].

Theorem 4 Let T be a monotone map on a closed and bounded rectangular region $\mathcal{R}\subset {\mathbb{R}}^2$. Suppose that T has a unique fixed point $\overline{\mathbf{e}}$ in ℛ. Then $\overline{\mathbf{e}}$ is a global attractor of T on ℛ.

The following theorems were proved by Kulenović and Merino [3] for competitive systems in the plane, when one of the eigenvalues of the linearized system at an equilibrium (hyperbolic or non-hyperbolic) is by an absolute value smaller than 1, while the other has an arbitrary value. These results are useful for determining basins of attraction of fixed points of competitive maps.

Theorem 5 Let T be a competitive map on a rectangular region $\mathcal{R}\subset {\mathbb{R}}^2$. Let $\overline{\mathrm{x}}\in \mathcal{R}$ be a fixed point of T such that $\mathrm{\Delta }:=\mathcal{R}\cap int({\mathcal{Q}}_1(\overline{x})\cup {\mathcal{Q}}_3(\overline{\mathrm{x}}))$ is nonempty (i.e., $\overline{\mathrm{x}}$ is not the NW or SE vertex of ℛ), and T is strongly competitive on Δ. Suppose that the following statements are true.

• (a) The map T has a $C^1$ extension to a neighborhood of $\overline{\mathrm{x}}$.

• (b) The Jacobian $J_T(\overline{\mathrm{x}})$ of T at $\overline{\mathrm{x}}$ has real eigenvalues λ, μ such that $0<|\lambda |<\mu$, where $|\lambda |<1$, and the eigenspace $E^{\lambda }$ associated with λ is not a coordinate axis.

Then there exists a curve $\mathcal{C}\subset \mathcal{R}$ through $\overline{\mathrm{x}}$ that is invariant and a subset of the basin of attraction of $\overline{\mathrm{x}}$ such that $\mathcal{C}$ is tangential to the eigenspace $E^{\lambda }$ at $\overline{\mathrm{x}}$, and $\mathcal{C}$ is the graph of a strictly increasing continuous function of the first coordinate on an interval. Any endpoints of $\mathcal{C}$ in the interior of ℛ are either fixed points or minimal period-two points. In the latter case, the set of endpoints of $\mathcal{C}$ is a minimal period-two orbit of T.

The situation where the endpoints of $\mathcal{C}$ are boundary points of ℛ is of interest. The following result gives a sufficient condition for this case.

Theorem 6 For the curve $\mathcal{C}$ of Theorem  5 to have endpoints in $\partial \mathcal{R}$, it is sufficient that at least one of the following conditions is satisfied.

• (i) The map T has no fixed points nor periodic points of minimal period two in Δ.

• (ii) The map T has no fixed points in Δ, $det{J}_T(\overline{\mathrm{x}})>0$, and $T(\mathrm{x})=\overline{\mathrm{x}}$ has no solutions $\mathrm{x}\in \mathrm{\Delta }$.

• (iii) The map T has no points of minimal period two in Δ, $det{J}_T(\overline{\mathrm{x}})<0$, and $T(\mathrm{x})=\overline{\mathrm{x}}$ has no solutions $\mathrm{x}\in \mathrm{\Delta }$.

The next result is useful for determining basins of attraction of fixed points of competitive maps.

Theorem 7 (A) Assume the hypotheses of Theorem  5, and let $\mathcal{C}$ be the curve whose existence is guaranteed by Theorem  5. If the endpoints of $\mathcal{C}$ belong to $\partial \mathcal{R}$, then $\mathcal{C}$ separates ℛ into two connected components, namely

$${\mathcal{W}}_{-}:=\{\mathrm{x}\in \mathcal{R}\setminus \mathcal{C}:\mathrm{\exists }\mathrm{y}\in \mathcal{C}\mathit{\text{ with }}\mathrm{x}{⪯}_{se}\mathrm{y}\}\mathit{\text{and}}{\mathcal{W}}_{+}:=\{\mathrm{x}\in \mathcal{R}\setminus \mathcal{C}:\mathrm{\exists }\mathrm{y}\in \mathcal{C}\mathit{\text{ with }}\mathrm{y}{⪯}_{se}\mathrm{x}\},$$

such that the following statements are true.

• (i) ${\mathcal{W}}_{-}$ is invariant, and $dist(T^n(\mathrm{x}),{\mathcal{Q}}_2(\overline{\mathrm{x}}))\to 0$ as $n\to \mathrm{\infty }$ for every $\mathrm{x}\in {\mathcal{W}}_{-}$.

• (ii) ${\mathcal{W}}_{+}$ is invariant, and $dist(T^n(\mathrm{x}),{\mathcal{Q}}_4(\overline{\mathrm{x}}))\to 0$ as $n\to \mathrm{\infty }$ for every $\mathrm{x}\in {\mathcal{W}}_{+}$.

• (B) If, in addition to the hypotheses of part (A), $\overline{\mathrm{x}}$ is an interior point of ℛ and T is $C^2$ and strongly competitive in a neighborhood of $\overline{\mathrm{x}}$, then T has no periodic points in the boundary of ${\mathcal{Q}}_1(\overline{\mathrm{x}})\cup {\mathcal{Q}}_3(\overline{\mathrm{x}})$ except for $\overline{\mathrm{x}}$, and the following statements are true.

• (iii) For every $\mathrm{x}\in {\mathcal{W}}_{-}$, there exists $n_0\in \mathbb{N}$ such that $T^n(\mathrm{x})\in int{\mathcal{Q}}_2(\overline{\mathrm{x}})$ for $n\ge n_0$.

• (iv) For every $\mathrm{x}\in {\mathcal{W}}_{+}$, there exists $n_0\in \mathbb{N}$ such that $T^n(\mathrm{x})\in int{\mathcal{Q}}_4(\overline{\mathrm{x}})$ for $n\ge n_0$.

If T is a map on a set ℛ and if $\overline{\mathrm{x}}$ is a fixed point of T, the stable set ${\mathcal{W}}^s(\overline{\mathrm{x}})$ of $\overline{\mathrm{x}}$ is the set $\{x\in \mathcal{R}:T^n(\mathrm{x})\to \overline{\mathrm{x}}\}$ and unstable set ${\mathcal{W}}^u(\overline{\mathrm{x}})$ of $\overline{\mathrm{x}}$ is the set

$$\{\mathrm{x}\in \mathcal{R}:\text{there exists }{\{{\mathrm{x}}_n\}}_{n=-\mathrm{\infty }}^0\subset \mathcal{R}\text{ s.t. }T({\mathrm{x}}_n)={\mathrm{x}}_{n+1},{\mathrm{x}}_0=\mathrm{x},\text{ and }\underset{n\to -\mathrm{\infty }}{lim}{\mathrm{x}}_n=\overline{\mathrm{x}}\}.$$

When T is non-invertible, the set ${\mathcal{W}}^s(\overline{\mathrm{x}})$ may not be connected and made up of infinitely many curves, or ${\mathcal{W}}^u(\overline{\mathrm{x}})$ may not be a manifold. The following result gives a description of the stable and unstable sets of a saddle point of a competitive map. If the map is a diffeomorphism on ℛ, the sets ${\mathcal{W}}^s(\overline{\mathrm{x}})$ and ${\mathcal{W}}^u(\overline{\mathrm{x}})$ are the stable and unstable manifolds of $\overline{x}$.

Theorem 8 In addition to the hypotheses of part (B) of Theorem  7, suppose that $\mu >1$ and that the eigenspace $E^{\mu }$ associated with μ is not a coordinate axis. If the curve $\mathcal{C}$ of Theorem  5 has endpoints in $\partial \mathcal{R}$, then $\mathcal{C}$ is the stable set ${\mathcal{W}}^s(\overline{\mathrm{x}})$ of $\overline{\mathrm{x}}$, and the unstable set ${\mathcal{W}}^u(\overline{\mathrm{x}})$ of $\overline{x}$ is a curve in ℛ that is tangential to $E^{\mu }$ at $\overline{\mathrm{x}}$ and such that it is the graph of a strictly decreasing function of the first coordinate on an interval. Any endpoints of ${\mathcal{W}}^u(\overline{\mathrm{x}})$ in ℛ are fixed points of T.

The following result gives information on local dynamics near a fixed point of a map when there exists a characteristic vector whose coordinates have negative product and such that the associated eigenvalue is hyperbolic. This is a well-known result, valid in a much more general setting: we include it here for completeness. A point $(x,y)$ is a subsolution if $T(x,y){⪯}_{se}(x,y)$, and $(x,y)$ is a supersolution if $(x,y){⪯}_{se}T(x,y)$. An order interval $〚(a,b),(c,d)〛$ is the cartesian product of the two compact intervals $[a,c]$ and $[b,d]$.

Theorem 9 Let T be a competitive map on a rectangular set $\mathcal{R}\subset {\mathbb{R}}^2$ with an isolated fixed point $\overline{\mathrm{x}}\in \mathcal{R}$ such that $\mathcal{R}\cap int({\mathcal{Q}}_2(\overline{\mathrm{x}})\cup {\mathcal{Q}}_4(\overline{\mathrm{x}}))\ne \mathrm{\varnothing }$. Suppose that T has a $C^1$ extension to a neighborhood of $\overline{\mathrm{x}}$. Let $\mathrm{v}=({\mathrm{v}}^{(1)},{\mathrm{v}}^{(2)})\in {\mathbb{R}}^2$ be an eigenvector of the Jacobian of T at $\overline{\mathrm{x}}$, with associated eigenvalue $\mu \in \mathbb{R}$. If ${\mathrm{v}}^{(1)}{\mathrm{v}}^{(2)}<0$, then there exists an order interval ℐ which is also a relative neighborhood of $\overline{\mathrm{x}}$ such that for every relative neighborhood $\mathcal{U}\subset \mathcal{I}$ of $\overline{\mathrm{x}}$ the following statements are true.

• (i) If $\mu >1$, then $\mathcal{U}\cap int{\mathcal{Q}}_2(\overline{\mathrm{x}})$ contains a subsolution and $\mathcal{U}\cap int{\mathcal{Q}}_4(\overline{\mathrm{x}})$ contains a supersolution. In this case, for every $\mathrm{x}\in \mathcal{I}\cap int({\mathcal{Q}}_2(\overline{\mathrm{x}})\cup {\mathcal{Q}}_4(\overline{\mathrm{x}}))$, there exists N such that $T^n(\mathrm{x})\notin \mathcal{I}$ for $n\ge N$.

• (ii) If $\mu <1$, then $\mathcal{U}\cap int{\mathcal{Q}}_2(\overline{\mathrm{x}})$ contains a supersolution and $\mathcal{U}\cap int{\mathcal{Q}}_4(\overline{\mathrm{x}})$ contains a subsolution. In this case $T^n(\mathrm{x})\to \overline{\mathrm{x}}$ for every $\mathrm{x}\in \mathcal{I}$.

3 Some basic facts

In this section we give some basic facts which will be used later. The map T associated to system (1) is given by

$$T(x,y)=(f(x,y),g(x,y))=(\frac{b_1{x}^2}{A_1+y^2},\frac{a_2+c_2{y}^2}{x^2}).$$

(5)

Let

$$\mathcal{R}={\mathbb{R}}_{+}^2\setminus \{(0,y):y\ge 0\}.$$

3.1 Equilibrium points

The equilibrium points $(\overline{x},\overline{y})$ of system (1) satisfy the equations

$$\frac{b_1{\overline{x}}^2}{{\overline{y}}^2+A_1}=\overline{x},\frac{c_2{\overline{y}}^2+a_2}{{\overline{x}}^2}=\overline{y}.$$

(6)

By eliminating $\overline{x}\ne 0$ from (6), we get

$${\overline{y}}^5+2A_1{\overline{y}}^3-b_1^2{c}_2{\overline{y}}^2+A_1^2\overline{y}-a_2{b}_1^2=0.$$

(7)

Similarly, we can eliminate variable $\overline{y}$ from system (6) to obtain

$$b_1{\overline{x}}^5-A_1{\overline{x}}^4-b_1^2{c}_2^2{\overline{x}}^2-2b_1{c}_2(a_2-A_1{c}_2)\overline{x}-{(a_2-A_1{c}_2)}^2=0.$$

(8)

Lemma 1 Let

$$\begin{array}{rl}{\mathrm{\Delta }}_1=& -7,500a_2^3{A}_1{b}_1^4{c}_2+a_2^2(5,550A_1^2{b}_1^4{c}_2^2+256A_1^5)\\ -4a_2(283A_1^3{b}_1^4{c}_2^3+128A_1^6{c}_2-27b_1^8{c}_2^5)+3,125a_2^4{b}_1^4-27A_1^4{b}_1^4{c}_2^4+256A_1^7{c}_2^2\end{array}$$

and

$${\mathrm{\Delta }}_2=-460a_2{A}_1^2{c}_2+250a_2^2{A}_1+202A_1^3{c}_2^2-27b_1^4{c}_2^4.$$

Then the following statements hold:

• (a) Consider equation (7). Then all its real roots are positive numbers. Furthermore, equation (7) has one, two, or three real roots.

• (b) If ${\mathrm{\Delta }}_1>0$, then equation (7) has one real root and two pairs of distinct conjugate imaginary roots.

• (c) If ${\mathrm{\Delta }}_1<0$, then equation (7) has three distinct real roots and one pair of conjugate imaginary roots.

• (d) If ${\mathrm{\Delta }}_1=0$ and ${\mathrm{\Delta }}_2\ne 0$, then equation (7) has one pair of conjugate imaginary roots and two real roots, one real root of multiplicity one and other one of multiplicity two.

• (e) If ${\mathrm{\Delta }}_1=0$ and ${\mathrm{\Delta }}_2=0$, then equation (7) has one pair of conjugate imaginary roots and one real root of multiplicity three.

Proof The proof of (a) follows from Descartes’ rule of signs.

Let

$$\tilde{f}(y)=y^5+2A_1{y}^3-b_1^2{c}_2{y}^2+A_1^{2}y-a_2{b}_1^2.$$

The following matrix, called the discrimination matrix of $\tilde{f}(y)$ and ${\tilde{f}}^{\prime }(y)$ in [26], is actually the Sylvester matrix of $\tilde{f}(y)$ and ${\tilde{f}}^{\prime }(y)$ with some permuted rows.

$$Discr(\tilde{f})=\left(\begin{array}{cccccccccc}1& 0& 2A_1& -b_1^2{c}_2& A_1^2& -a_2{b}_1^2& 0& 0& 0& 0\\ 0& 5& 0& 6A_1& -2b_1^2{c}_2& A_1^2& 0& 0& 0& 0\\ 0& 1& 0& 2A_1& -b_1^2{c}_2& A_1^2& -a_2{b}_1^2& 0& 0& 0\\ 0& 0& 5& 0& 6A_1& -2b_1^2{c}_2& A_1^2& 0& 0& 0\\ 0& 0& 1& 0& 2A_1& -b_1^2{c}_2& A_1^2& -a_2{b}_1^2& 0& 0\\ 0& 0& 0& 5& 0& 6A_1& -2b_1^2{c}_2& A_1^2& 0& 0\\ 0& 0& 0& 1& 0& 2A_1& -b_1^2{c}_2& A_1^2& -a_2{b}_1^2& 0\\ 0& 0& 0& 0& 5& 0& 6A_1& -2b_1^2{c}_2& A_1^2& 0\\ 0& 0& 0& 0& 1& 0& 2A_1& -b_1^2{c}_2& A_1^2& -a_2{b}_1^2\\ 0& 0& 0& 0& 0& 5& 0& 6A_1& -2b_1^2{c}_2& A_1^2\end{array}\right).$$

Let $D_k$ denote the determinant of the submatrix of $Discr(\tilde{f})$, formed by the first 2k row and the first 2k columns, for $k=1,\dots ,m$. So, by a straightforward calculation, one can see that

$$\begin{array}{c}{D}_1=5,\hfill \\ D_2=-20A_1,\hfill \\ D_3=-16A_1^3-45b_1^4{c}_2^2,\hfill \\ D_4=2b_1^4(-460a_2{A}_1^2{c}_2+250a_2^2{A}_1+202A_1^3{c}_2^2-27b_1^4{c}_2^4)=2b_1^4{\mathrm{\Delta }}_2,\hfill \\ \begin{array}{rl}{D}_5=& b_1^4(-7,500a_2^3{A}_1{b}_1^4{c}_2+a_2^2(5,550A_1^2{b}_1^4{c}_2^2+256A_1^5)\\ -4a_2(283A_1^3{b}_1^4{c}_2^3+128A_1^6{c}_2-27b_1^8{c}_2^5)\\ +3,125a_2^4{b}_1^4-27A_1^4{b}_1^4{c}_2^4+256A_1^7{c}_2^2)=b_1^4{\mathrm{\Delta }}_1.\end{array}\hfill \end{array}$$

Assume that $D_5>0$. The sign list of the sequence $\{D_1,D_2,D_3,D_4,D_5\}$ is given by

$$[1,-1,-1,sign(D_4),1],$$

(9)

from which it follows that the number of sign changes of the revised sign list of list (9) is two. Now, statement (b) follows in view of Theorem 1 [26]. Assume that $D_5<0$. If $D_4\ge 0$, then we obtain that $\tilde{f}(y)$ has three pairs of conjugate imaginary roots, which is a contradiction. Hence, $D_4<0$. The sign list of the sequence $\{D_1,D_2,D_3,D_4,D_5\}$ is given by

$$[1,-1,-1,-1,-1],$$

(10)

which implies that the number of sign changes of the revised sign list of (10) is one. Now, statement (c) follows in view of Theorem 1 [26]. Similarly, one can prove statements (d) and (e). □

3.2 Injectivity, $(O+)$ and $(O-)$

Lemma 2 Assume that $(\overline{x},\overline{y})$ is an equilibrium of the map T. Then the following hold:

• (1) If $a_2\ne A_1{c}_2$, then T is injective.

• (2) If $a_2=A_1{c}_2$, then the curve

  $$b_1{x}^2=\overline{x}(A_1+y^2)$$

  is invariant under the map T. Furthermore, the following holds:

  $$T(x,\sqrt{\frac{b_1{x}^2-A_1\overline{x}}{\overline{x}}})=(\overline{x},\overline{y})\mathit{\text{for }}x\ge \sqrt{\frac{A_1\overline{x}}{b_1}}.$$

• (3) If $A_1{c}_2>a_2$, then T satisfies $(O+)$, in which case $\{T^n(x_0,y_0)\}$ is asymptotic to either $(0,\mathrm{\infty })$ or $(\mathrm{\infty },0)$, or to an equilibrium point, for all $(x_0,y_0)\in \mathcal{R}$.

• (4) If $A_1{c}_2<a_2$, then T satisfies $(O-)$, in which case $\{T^n(x_0,y_0)\}$ is asymptotic to either $(0,\mathrm{\infty })$ or $(\mathrm{\infty },0)$, or to a period-two point, for all $(x_0,y_0)\in \mathcal{R}$.

Proof (1) Assume that $T(x_1,y_1)=T(x_2,y_2)$. Then we have

$$(\frac{A_1{b}_1{x}_1^2-A_1{b}_1{x}_2^2+b_1{x}_1^2{y}_2^2-b_1{x}_2^2{y}_1^2}{(A_1+y_1^2)(A_1+y_2^2)},\frac{-a_2{x}_1^2+a_2{x}_2^2-c_2{x}_1^2{y}_2^2+c_2{x}_2^2{y}_1^2}{x_1^2{x}_2^2})=(0,0).$$

(11)

Equation (11) is equivalent to

$$x_1^2(A_1{b}_1+b_1{y}_2^2)-A_1{b}_1{x}_2^2-b_1{x}_2^2{y}_1^2=0,$$

(12)

$$x_1^2(-a_2-c_2{y}_2^2)+a_2{x}_2^2+c_2{x}_2^2{y}_1^2=0.$$

(13)

Equation (12) implies

$$x_1^2=\frac{x_2^2(A_1+y_1^2)}{A_1+y_2^2}.$$

(14)

By substituting this into equation (13), we obtain

$$\frac{x_2^2(y_1^2-y_2^2)(a_2-A_1{c}_2)}{A_1+y_2^2}=0,$$

from which it follows that $y_1=y_2$ since $a_2\ne A_1{c}_2$. From (14) we have $x_1=x_2$, which completes the proof of statement (a).

(2)One can see that

$$T(x,\sqrt{\frac{b_1{x}^2-A_1\overline{x}}{\overline{x}}})-(\overline{x},\overline{y})=(0,\frac{b_1{c}_2-\overline{x}\overline{y}}{\overline{x}}).$$

Since $a_2=A_1{c}_2$, equations (7) and (8) become

$$c_2({\overline{y}}^2+A_1)(A_1\overline{y}+{\overline{y}}^3-b_1^2{c}_2)=0$$

(15)

and

$$c_2-{\overline{x}}^2(-A_1{\overline{x}}^2+b_1{\overline{x}}^3-b_1^2{c}_2^2)=0.$$

(16)

From (15) we have $A_1=\frac{b_1^2{c}_2-{\overline{y}}^3}{\overline{y}}$. By substituting this into (16) we get

$$\frac{(\overline{x}\overline{y}-b_1{c}_2)(b_1(c_2\overline{y}+{\overline{x}}^2)+\overline{x}{\overline{y}}^2)}{\overline{y}}=0,$$

which implies $\overline{x}\overline{y}-b_1{c}_2=0$, from which the proof follows.

(3) The Jacobian matrix of the map T has the form

$$J_T=\left(\begin{array}{cc}\frac{2x{b}_1}{y^2+A_1}& -\frac{2x^{2}y{b}_1}{{(y^2+A_1)}^2}\\ -\frac{2(c_2{y}^2+a_2)}{x^3}& \frac{2y{c}_2}{x^2}\end{array}\right).$$

(17)

The determinant of (17) at any point is equal to

$$J_T(x,y)=\frac{4b_1{x}^{2}y(A_1{c}_2-a_2)}{x^3{(y^2+A_1)}^2}.$$

The proof of (3) and (4) follows from Theorem 3. □

4 Linearized stability analysis

The determinant of (17) at the equilibrium point is given by

$$det{J}_T(\overline{x},\overline{y})=\frac{4b_1\overline{y}(A_1{c}_2-a_2)}{\overline{x}{({\overline{y}}^2+A_1)}^2}.$$

(18)

The trace of (17) at the equilibrium point is given by

$$tr{J}_T(\overline{x},\overline{y})=\frac{2c_2\overline{y}}{{\overline{x}}^2}+2.$$

The characteristic equation has the form

$${\lambda }^2-\lambda (\frac{2c_2\overline{y}}{{\overline{x}}^2}+2)+\frac{4b_1\overline{y}(A_1{c}_2-a_2)}{\overline{x}{({\overline{y}}^2+A_1)}^2}=0.$$

Equilibrium curves ${\mathcal{C}}_f=\{(x,y)\in \mathcal{R}:f(x,y)=x\}$ and ${\mathcal{C}}_g=\{(x,y)\in \mathcal{R}:g(x,y)=y\}$ can be given explicitly as functions of y:

$$\begin{array}{c}{\mathcal{C}}_f:x_f(y)=\frac{A_1+y^2}{b_1},\hfill \\ {\mathcal{C}}_g:\{\begin{array}{ll}{x}_{g^{+}}(y)=+\sqrt{\frac{a_2+c_2{y}^2}{y}},& y>0,\\ x_{g^{-}}(y)=-\sqrt{\frac{a_2+c_2{y}^2}{y}},& y>0.\end{array}\hfill \end{array}$$

Note that $x_{g^{-}}(x)$ is always negative. Let $x_g(y)$ denote $x_{g^{+}}(y)$. We consider only $x_f(y)$ and $x_g(y)$. Let

$$\tilde{x}(y)=x_f(y)-x_g(y).$$

Lemma 3 Let $T=(f,g)$ be the map defined by (5). Then $f_x^{\prime }(x,y)>1$, and the following is true:

$$sign(\tilde{x}(y))=sign(y^5+2A_1{y}^3-b_1^2{c}_2{y}^2+A_1^{2}y-a_2{b}_1^2).$$

Proof The first derivative of $x_f(y)$ is given by

$$x_f^{\prime }(y)=\frac{f_y^{\prime }(x,y)}{1-f_x^{\prime }(x,y)}=\frac{2y}{b_1}>0.$$

Since $f_y^{\prime }(x,y)<0$, we get $f_x^{\prime }(x,y)>1$. Further,

$$\tilde{x}(y)=x_f(y)-x_g(y)=\frac{A_1+y^2}{b_1}-\sqrt{\frac{a_2+c_2{y}^2}{y}}=\frac{\sqrt{y}(A_1+y^2)-b_1\sqrt{a_2+c_2{y}^2}}{b_1\sqrt{y}}.$$

Now, the proof follows from

$${\left(\sqrt{y}(A_1+y^2)\right)}^2-{\left(b_1\sqrt{a_2+c_2{y}^2}\right)}^2=y^5+2A_1{y}^3-b_1^2{c}_2{y}^2+A_1^{2}y-a_2{b}_1^2.$$

 □

Lemma 4 Let T be the map defined by (5), and let

$$J_T(\overline{x},\overline{y})=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)$$

(19)

be the Jacobian matrix of T at a fixed point $(\overline{x},\overline{y})$. Then the Jacobian matrix (19) has real and distinct eigenvalues ${\lambda }_1$ and ${\lambda }_2$ such that $|{\lambda }_1|<{\lambda }_2$ and ${\lambda }_2>1$. Furthermore, the following holds:

$$sign({\tilde{x}}^{\prime }(\overline{y}))=sign(1-{\lambda }_1).$$

Proof Implicit differentiation of the equations defining $C_f$ and $C_g$ at $(\overline{x},\overline{y})$ gives

$$x_f^{\prime }(\overline{y})=\frac{f_y^{\prime }(\overline{x},\overline{y})}{1-f_x^{\prime }(\overline{x},\overline{y})},x_g^{\prime }(\overline{y})=\frac{1-g_y^{\prime }(\overline{x},\overline{y})}{g_x^{\prime }(\overline{x},\overline{y})}.$$

(20)

The characteristic equation associated with the Jacobian matrix of T at $(\overline{x},\overline{y})$ is given by

$$\begin{array}{rl}p(\lambda )& ={\lambda }^2-[f_x^{\prime }(\overline{x},\overline{y})+g_y^{\prime }(\overline{x},\overline{y})]\lambda +[f_x^{\prime }(\overline{x},\overline{y})g_y^{\prime }(\overline{x},\overline{y})-f_y^{\prime }(\overline{x},\overline{y})g_x^{\prime }(\overline{x},\overline{y})]\\ ={\lambda }^2-(a+d)\lambda +(ad-bc).\end{array}$$

Since the map T is competitive, then the eigenvalues of the Jacobian matrix of the map T at the equilibrium $(\overline{x},\overline{y})$ are real and distinct and $|{\lambda }_1|<{\lambda }_2$. By (20), we have

$$\begin{array}{rl}{\tilde{x}}^{\prime }(\overline{y})& =x_f^{\prime }(\overline{y})-x_g^{\prime }(\overline{y})=\frac{f_y^{\prime }(\overline{x},\overline{y})}{1-f_x^{\prime }(\overline{x},\overline{y})}-\frac{1-g_y^{\prime }(\overline{x},\overline{y})}{g_x^{\prime }(\overline{x},\overline{y})}\\ =\frac{b}{1-a}-\frac{1-d}{c}=\frac{-1+(a+d)-(ad-bc)}{c(1-a)}\\ =\frac{-p(1)}{c(1-a)}=\frac{(1-{\lambda }_1)(1-{\lambda }_2)}{c(a-1)}.\end{array}$$

From $tr{J}_T(\overline{x},\overline{y})={\lambda }_1+{\lambda }_2>2$ we get ${\lambda }_2>1$. The map T is competitive, which implies $c=g_x^{\prime }(\overline{x},\overline{y})<0$. In view of Lemma 3, we get $a=f_x^{\prime }(\overline{x},\overline{y})>1$, from which it follows that $sign({\tilde{x}}^{\prime }(\overline{y}))=sign(1-{\lambda }_1)$. □

Theorem 10 Assume that ${\mathrm{\Delta }}_1>0$ and ${\lambda }_1$ and ${\lambda }_2$ are eigenvalues of $J_T(\overline{x},\overline{y})$. Then there exists the unique equilibrium point $E=(\overline{x},\overline{y})$ and the following hold:

• (a) If $a_2<A_1{c}_2$, then E is a saddle point and $0<{\lambda }_1<1$, ${\lambda }_2>1$.

• (b) Assume that $a_2>A_1{c}_2$. Let

  $$\mathrm{\Gamma }(\overline{y})=5c_2{\overline{y}}^4+{\overline{y}}^2(9A_1{c}_2-a_2)+3a_2{A}_1.$$

  • (b1) If $\mathrm{\Gamma }(\overline{y})>0$, then E is a saddle point. Furthermore, the following hold: $-1<{\lambda }_1<0$, ${\lambda }_2>1$.

  • (b2) If $\mathrm{\Gamma }(\overline{y})<0$, then E is a repeller. Furthermore, the following hold: ${\lambda }_1<-1$, ${\lambda }_2>1$; $|{\lambda }_1|<{\lambda }_2$.

  • (b3) If $\mathrm{\Gamma }(\overline{y})=0$, then E is a non-hyperbolic equilibrium point. Furthermore, the following hold: ${\lambda }_1=-1$, ${\lambda }_2>1$.

Proof In view of (7) and Lemma 1, we have that the function

$$\tilde{f}(y)=y^5+2A_1{y}^3-b_1^2{c}_2{y}^2+A_1^{2}y-a_2{b}_1^2$$

has one zero $\overline{y}$ of multiplicity one. In view of Lemma 1, the map T has a unique equilibrium point. Since $\tilde{f}(0)=-a_2{b}_1^2<0$ and ${lim}_{y\to +\mathrm{\infty }}\tilde{f}(y)=+\mathrm{\infty }$, we have $\tilde{f}(y)<0$ for $y<\overline{y}$ and $\tilde{f}(y)>0$ for $y>\overline{y}$. By Lemmas 6 and 7 from [27], the equilibrium curves ${\mathcal{C}}_f$ and ${\mathcal{C}}_g$ intersect transversally at $(\overline{x},\overline{y})$, i.e., ${\tilde{x}}^{\prime }(\overline{y})\ne 0$. In view of Lemma 3 and by the continuity of function $\tilde{x}(y)$, there exists a neighborhood $U_{\overline{y}}$ of $\overline{y}$ such that ${\tilde{x}}^{\prime }(y)>0$ for $y\in U_{\overline{y}}$, which implies

$${\tilde{x}}^{\prime }(\overline{y})=x_f^{\prime }(\overline{y})-x_g^{\prime }(\overline{y})>0.$$

(21)

From (21) and Lemma 4 we obtain ${\lambda }_1<1$ and ${\lambda }_2>1$.

If $a_2<A_1{c}_2$, then $det{J}_T(\overline{x},\overline{y})={\lambda }_1{\lambda }_2>0$, which implies that ${\lambda }_1\in (0,1)$.

Now, assume that $a_2>A_1{c}_2$. By using

$$b_1=\frac{{\overline{y}}^2+A_1}{\overline{x}},\overline{x}=\sqrt{\frac{c_2{\overline{y}}^2+a_2}{\overline{y}}},$$

one can see that

$$\begin{array}{rl}p(-1)& =1+det{J}_T(\overline{x},\overline{y})+tr{J}_T(\overline{x},\overline{y})=\frac{5c_2{\overline{y}}^4+{\overline{y}}^2(9A_1{c}_2-a_2)+3a_2{A}_1}{({\overline{y}}^2+A_1)(c_2{\overline{y}}^2+a_2)}\\ =\frac{\mathrm{\Gamma }(\overline{y})}{({\overline{y}}^2+A_1)(c_2{\overline{y}}^2+a_2)}\end{array}$$

and

$$det{J}_T(\overline{x},\overline{y})={\lambda }_1{\lambda }_2=\frac{4b_1\overline{y}(A_1{c}_2-a_2)}{\overline{x}{({\overline{y}}^2+A_1)}^2},$$

where $p(\lambda )=(\lambda -{\lambda }_1)(\lambda -{\lambda }_2)$. In view of (18) and $p(-1)=({\lambda }_1+1)({\lambda }_2+1)$, we obtain statement (b) of the theorem. □

Lemma 5 Suppose that all the assumptions of Theorem  10 are satisfied. Let

$$y_{\pm }=\sqrt{\frac{a_2-9A_1{c}_2\pm \sqrt{-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2}}{10c_2}}.$$

Then the following statements are true.

• (a) $\mathrm{\Gamma }(\overline{y})>0$ if and only if one of the following inequalities holds:

  $$\begin{array}{c}9A_1{c}_2-a_2\ge 0,\hfill \\ 9A_1{c}_2-a_2<0\mathit{\text{and}}-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2<0,\hfill \\ 9A_1{c}_2-a_2<0\mathit{\text{and}}-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2\ge 0\mathit{\text{and}}\hfill \\ (\tilde{f}(y_{-})>0\mathit{\text{ or }}\tilde{f}(y_{+})<0);\hfill \end{array}$$

• (b) $\mathrm{\Gamma }(\overline{y})<0$ if and only if the following hold:

  $$9A_1{c}_2-a_2<0\mathit{\text{and}}-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2\ge 0\mathit{\text{and}}(\tilde{f}(y_{-})<0\mathit{\text{ and }}\tilde{f}(y_{+})>0);$$

• (c) $\mathrm{\Gamma }(\overline{y})=0$ if and only if

  $$9A_1{c}_2-a_2<0\mathit{\text{and}}-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2\ge 0\mathit{\text{and}}(\tilde{f}(y_{-})=0\mathit{\text{ or }}\tilde{f}(y_{+})=0).$$

Proof The function $\tilde{f}(y)$ has one simple zero $\overline{y}$, which implies $\tilde{f}(y)<0$ for $0\le y<\overline{y}$ and $\tilde{f}(y)>0$ for $y>\overline{y}$. Then

$$\overline{y}>\alpha \text{if and only if}\tilde{f}(\alpha )<0,$$

while

$$\overline{y}<\beta \text{if and only if}\tilde{f}(\beta )>0$$

for some $\alpha ,\beta \in [0,\mathrm{\infty })$. Now the proof follows from the fact that $\tilde{f}(y)=0$ has real roots

$$\begin{array}{c}\{\sqrt{\frac{a_2-9A_1{c}_2\pm \sqrt{-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2}}{10c_2}},\hfill \\ -\sqrt{\frac{a_2-9A_1{c}_2\pm \sqrt{-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2}}{10c_2}}\}\hfill \end{array}$$

if and only if

$$9A_1{c}_2-a_2<0\text{and}-78a_2{A}_1{c}_2+a_2^2+81A_1^2{c}_2^2\ge 0.$$

 □

Theorem 11 Assume that ${\mathrm{\Delta }}_1<0$. Then there exist three distinct equilibrium points in the positive quadrant: $E_1=({\overline{x}}_1,{\overline{y}}_1)$, $E_2=({\overline{x}}_2,{\overline{y}}_2)$ and $E_3=({\overline{x}}_3,{\overline{y}}_3)$ such that $E_1{\ll }_{ne}{E}_2{\ll }_{ne}{E}_3$ and the following hold:

• (a) $E_1$ and $E_3$ are saddle points. If ${\lambda }_1^{(i)}$ and ${\lambda }_2^{(i)}$ are the eigenvalues of $J_T(E_i)$, $i=1,3$, then $0<{\lambda }_1^{(i)}<1$, ${\lambda }_2^{(i)}>1$.

• (b) The equilibrium point $E_2$ is a repeller. If ${\lambda }_1^{(2)}$ and ${\lambda }_2^{(2)}$ are the eigenvalues of $J_T(E_2)$, then $1<{\lambda }_1^{(2)}<{\lambda }_2^{(2)}$.

Proof In view of Lemma 1, equation (7) has three positive roots of multiplicity one. Since

$$\overline{x}=\frac{A_1+{\overline{y}}^2}{b_1}>0,$$

then by (a) of Lemma 1 we obtain that the map T has three equilibrium points that we denote by $E_1$, $E_2$ and $E_3$. Given points lie on the increasing curve

$$x_f(y)=\frac{A_1+y^2}{b_1},$$

which implies that the points are in the north-east ordering. Descartes’ rule of signs and (8) imply that $det{J}_T(x,y)>0$ when $a_2<A_1{c}_2$. In view of (7) and Lemma 1, we have that the polynomial

$$\tilde{f}(y)=y^5+2A_1{y}^3-b_1^2{c}_2{y}^2+A_1^{2}y-a_2{b}_1^2$$

has three zeros ${\overline{y}}_i$, $i=1,2,3$, of multiplicity one. Since $\tilde{f}(0)=-a_2{b}_1^2<0$ and ${lim}_{y\to +\mathrm{\infty }}\tilde{f}(y)=+\mathrm{\infty }$, we have $\tilde{f}(y)<0$ for $y\in (0,{\overline{y}}_1)\cup ({\overline{y}}_2,{\overline{y}}_3)$ and $\tilde{f}(y)>0$ for $y\in ({\overline{y}}_1,{\overline{y}}_2)\cup ({\overline{y}}_3,+\mathrm{\infty })$.

By Lemmas 6 and 7 from [27], the equilibrium curves ${\mathcal{C}}_f$ and ${\mathcal{C}}_g$ intersect transversally at $E_1$, $E_2$ and $E_3$, i.e., ${\tilde{x}}^{\prime }({\overline{y}}_i)\ne 0$, $i=1,2,3$. By this and Lemma 3 and by the continuity of function $\tilde{x}(y)$, there exists a neighborhood $U_{{\overline{y}}_i}^{(i)}$ of ${\overline{y}}_i$ such that ${\tilde{x}}^{\prime }(y)>0$ for $y\in U_{{\overline{y}}_i}^{(i)}$ for $i=1,3$ and ${\tilde{x}}^{\prime }(y)<0$ for $y\in U_{{\overline{y}}_i}^{(2)}$. Using this we get

$${\tilde{x}}^{\prime }({\overline{y}}_i)>0\text{for }i=1,3\text{and}{\tilde{x}}^{\prime }({\overline{y}}_i)<0\text{for }i=2.$$

Let

$$J_T(E_i)=\left(\begin{array}{cc}{a}_i& b_i\\ c_i& d_i\end{array}\right),i=1,2,3.$$

In view of (18), we have $det{J}_T(E_i)={\lambda }_1^{(i)}{\lambda }_2^{(i)}>0$, $i=1,2,3$. By Lemma 4 we obtain $0<{\lambda }_1^{(i)}<1$ and ${\lambda }_2^{(i)}>1$ for $i=1,3$. Since ${\tilde{x}}^{\prime }({\overline{y}}_2)<0$, by Lemma 4 we have $1<{\lambda }_1^{(2)}<{\lambda }_2^{(2)}$. This completes the proof. □

Theorem 12 Assume that ${\mathrm{\Delta }}_1=0$ and ${\mathrm{\Delta }}_2\ne 0$ Then there exist two distinct equilibrium points in the positive quadrant $E_1=({\overline{x}}_1,{\overline{y}}_1)$ and $E_3=({\overline{x}}_3,{\overline{y}}_3)$ such that $E_1{\ll }_{ne}{E}_3$. Let ${\lambda }_1^{(i)}$ and ${\lambda }_2^{(i)}$ be the eigenvalues of $J_T(E_i)$, $i=1,3$. Then the following hold:

• (a) Exactly one of the roots ${\overline{y}}_1$ or ${\overline{y}}_3$ of (7) has multiplicity two.

• (b) If ${\overline{y}}_1$ is a root of (7) of multiplicity two, then the equilibrium point $E_1$ is non-hyperbolic and $E_3$ is a saddle point. Furthermore, ${\lambda }_1^{(1)}=1$, ${\lambda }_2^{(1)}>1$ and $0<{\lambda }_1^{(3)}<1$, ${\lambda }_2^{(3)}>1$.

• (c) If ${\overline{y}}_3$ is a root of (7) of multiplicity two, then the equilibrium point $E_3$ is non-hyperbolic and $E_1$ is a saddle point. Furthermore, ${\lambda }_1^{(3)}=1$, ${\lambda }_2^{(3)}>1$ and $0<{\lambda }_1^{(1)}<1$, ${\lambda }_2^{(1)}>1$.

Proof In view of Lemma 1, equation (7) has two positive zeros, one of multiplicity one and another one of multiplicity two, which implies statement (a). Since $\overline{x}=(A_1+{\overline{y}}^2)/b_1>0$, we obtain that the map T has two equilibrium points that we denote by $E_1$ and $E_3$. Descartes’ rule of signs and (8) imply that $a_2<A_1{c}_2\Rightarrow det{J}_T(x,y)>0$. Now, we prove statement (b). Similarly as in the proof of Theorem 11, one can see that $E_3$ is a saddle point. In view of Lemmas 6 and 7, from [27] we have that ${\tilde{x}}^{\prime }({\overline{y}}_1)=0$, since ${\overline{y}}_1$ is the root of (7) of multiplicity two. By Lemma 4 we obtain ${\lambda }_1^{(1)}=1$, ${\lambda }_2^{(1)}>1$. The proof of statement (c) is similar and we will skip it. □

Theorem 13 Assume that ${\mathrm{\Delta }}_1=0$ and ${\mathrm{\Delta }}_2=0$. Then there exists one equilibrium point in the positive quadrant $E_1=({\overline{x}}_1,{\overline{y}}_1)$ which is non-hyperbolic. If ${\lambda }_1^{(1)}$ and ${\lambda }_2^{(1)}$ are eigenvalues of $J_T(E_1)$, then ${\lambda }_1^{(1)}=1$, ${\lambda }_2^{(1)}>1$.

Proof In view of Lemma 1, ${\overline{y}}_1$ is zero of (7) of multiplicity three. In view of Lemmas 6 and 7, from [27] we have that ${\tilde{x}}^{\prime }({\overline{y}}_1)=0$. The rest of the proof is similar to that in Theorem 12 and we skip it. □

4.1 Period-two solution

Let

$$T^2(x,y)=T(T(x,y))=(F(x,y),G(x,y)),$$

where

$$F(x,y)=\frac{b_1^3{x}^8}{{(A_1+y^2)}^2(2a_2{c}_2{y}^2+a_2^2+A_1{x}^4+c_2^2{y}^4)}$$

and

$$G(x,y)=\frac{{(A_1+y^2)}^2(2a_2{c}_2^2{y}^2+a_2^2{c}_2+a_2{x}^4+c_2^3{y}^4)}{b_1^2{x}^8}.$$

Period-two solution $\{(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\}$ satisfies the system

$$F(\mathrm{\Phi },\mathrm{\Psi })=\mathrm{\Phi },G(\mathrm{\Phi },\mathrm{\Psi })=\mathrm{\Psi },$$

which is equivalent to

$$\begin{array}{r}{b}_1^3{\mathrm{\Phi }}^7-A_1{\mathrm{\Phi }}^4{(A_1+{\mathrm{\Psi }}^2)}^2-{(A_1+{\mathrm{\Psi }}^2)}^2{(a_2+c_2{\mathrm{\Psi }}^2)}^2=0,\\ b_1^2\mathrm{\Psi }{\mathrm{\Phi }}^8-a_2{\mathrm{\Phi }}^4{(A_1+{\mathrm{\Psi }}^2)}^2-c_2{(A_1+{\mathrm{\Psi }}^2)}^2{(a_2+c_2{\mathrm{\Psi }}^2)}^2=0.\end{array}$$

(22)

The Jacobian matrix of the map $T^2$ at $(x,y)$ has the form

$$J_{T^2}(x,y)=\left(\begin{array}{cc}\frac{4x^7{b}_1^3(A_1{x}^4+2{(c_2{y}^2+a_2)}^2)}{{(y^2+A_1)}^2{(A_1{x}^4+{(c_2{y}^2+a_2)}^2)}^2}& -\frac{4x^{8}y{b}_1^3(A_1{x}^4+(c_2{y}^2+a_2)(a_2+(2y^2+A_1)c_2))}{{(y^2+A_1)}^3{(A_1{x}^4+{(c_2{y}^2+a_2)}^2)}^2}\\ -\frac{4{(y^2+A_1)}^2(2c_2^3{y}^4+2a_2^2{c}_2+a_2(x^4+4y^2{c}_2^2))}{x^9{b}_1^2}& \frac{4y(y^2+A_1)(y^2(2y^2+A_1)c_2^3+a_2^2{c}_2+a_2(x^4+(3y^2+A_1)c_2^2))}{x^8{b}_1^2}\end{array}\right).$$

(23)

The determinant of (23) at $(x,y)$ is given by

$$det{J}_{T^2}(x,y)=\frac{16b_1{x}^{3}y{(a_2-A_1{c}_2)}^2(a_2+c_2{y}^2)}{(A_1+y^2){({(a_2+c_2{y}^2)}^2+A_1{x}^4)}^2}.$$

The trace of (23) at $(x,y)$ is given by

$$\begin{array}{rl}tr{J}_{T^2}(x,y)=& \frac{4(\frac{b_1^5{x}^{15}(2{(a_2+c_2{y}^2)}^2+A_1{x}^4)}{{({(a_2+c_2{y}^2)}^2+A_1{x}^4)}^2}+a_{2}y{(A_1+y^2)}^3(c_2^2(A_1+3y^2)+x^4))}{b_1^2{x}^8{(A_1+y^2)}^2}\\ +\frac{4(a_2^2{c}_{2}y{(A_1+y^2)}^3+c_2^3{y}^3(A_1+2y^2){(A_1+y^2)}^3)}{b_1^2{x}^8{(A_1+y^2)}^2}.\end{array}$$

(24)

Lemma 6 Let ${\mathcal{C}}_F:=\{(x,y):F(x,y)=x\}$ and ${\mathcal{C}}_G:=\{(x,y):G(x,y)=y\}$ be the period-two curves, that is, the curves the intersection of which is a period-two solution. Then, for all $y>0$, there exist exactly one $x_F(y)>0$ and exactly one $x_G(y)>0$ such that $F(x_F(y),y)=x$ and $G(x_G(y),y)=y$. Furthermore, $x_F(y)$ and $x_G(y)$ are continuous functions and $x_F^{\prime }(y)>0$.

Proof Since $F(x,y)=x$ and $G(x,y)=y$ if and only if

$$\begin{array}{c}{b}_1^3{x}^7-A_1{x}^4{(A_1+y^2)}^2-{(A_1+y^2)}^2{(a_2+c_2{y}^2)}^2=0,\hfill \\ b_1^{2}y{x}^8-a_2{x}^4{(A_1+y^2)}^2-c_2{(A_1+y^2)}^2{(a_2+c_2{y}^2)}^2=0,\hfill \end{array}$$

respectively, in view of Descartes’ rule of signs, we have that for all $y>0$ there exist exactly one $x_F(y)>0$ and exactly one $x_G(y)>0$ such that $F(x_F(y),y)=x$ and $G(x_G(y),y)=y$. Taking derivatives of $F(x,y)=x$ with respect to y, we get

$$x_F^{\prime }(y)=\frac{F_y^{\prime }(x,y)}{1-F_x^{\prime }(x,y)}.$$

From $F(x,y)=x$ we have that ${(A_1+y^2)}^2=\frac{b_1^3{x}^7}{{(a_2+c_2{y}^2)}^2+A_1{x}^4}$, which implies

$$F_x^{\prime }(x,y)=\frac{4x^7{b}_1^3(A_1{x}^4+2{(c_2{y}^2+a_2)}^2)}{{(y^2+A_1)}^2{(A_1{x}^4+{(c_2{y}^2+a_2)}^2)}^2}=8-\frac{4A_1{x}^4}{{(a_2+c_2{y}^2)}^2+A_1{x}^4}>4.$$

Since $F_y^{\prime }(x,y)<0$, we get $x_F^{\prime }(y)>0$. □

Theorem 14 If $a_2\le A_1{c}_2$, then T has no minimal period-two solution. If $a_2>A_1{c}_2$ and T has a minimal period-two solution $\{(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\}$, then $\{(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\}$ is unstable. If ${\mu }_1$ and ${\mu }_2$ (${\mu }_1<{\mu }_2$) are the eigenvalues of $J_{T^2}(\mathrm{\Phi },\mathrm{\Psi })$, then ${\mu }_1>0$ and ${\mu }_2>1$. All period-two solutions are ordered with respect to the north-east ordering.

Proof If $a_2\le A_1{c}_2$, the statement follows from Lemma 2. If $a_2>A_1{c}_2$, then from the first equation of (22) we have that

$${(A_1+{\mathrm{\Psi }}^2)}^2=\frac{b_1^3{\mathrm{\Phi }}^7}{{(a_2+c_2{\mathrm{\Psi }}^2)}^2+A_1{\mathrm{\Phi }}^4}.$$

By substituting this into (24) we obtain

$$\begin{array}{rl}tr{J}_{T^2}(\mathrm{\Phi },\mathrm{\Psi })=& \frac{4\mathrm{\Psi }(A_1+{\mathrm{\Psi }}^2)(a_2(c_2^2(A_1+3{\mathrm{\Psi }}^2)+{\mathrm{\Phi }}^4)+a_2^2{c}_2+c_2^3{\mathrm{\Psi }}^2(A_1+2{\mathrm{\Psi }}^2))}{b_1^2{\mathrm{\Phi }}^8}\\ +8-\frac{4A_1{\mathrm{\Phi }}^4}{{(a_2+c_2{\mathrm{\Psi }}^2)}^2+A_1{\mathrm{\Phi }}^4}>4.\end{array}$$

The rest of the proof follows from the fact that $tr{J}_{T^2}(\mathrm{\Phi },\mathrm{\Psi })={\mu }_1+{\mu }_2>4$, $det{J}_{T^2}(\mathrm{\Phi },\mathrm{\Psi })={\mu }_1{\mu }_2>0$ and Lemma 6. □

Theorem 15 If the map T has a minimal period-two solution $\{({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1),T({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)\}$, which is non-hyperbolic, then $D(p)=0$, where $D(p)$ is the discriminant of the polynomial

$$p(y):=p_{22}{y}^{22}+p_{21}{y}^{21}+\cdots +p_{1}y+p_0,$$

where the coefficients $p_i$, $i=0,\dots ,22$, are in the Appendix. If $\{({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1),T({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)\}$ and $\{({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2),T({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2)\}$ are two minimal period-two solutions such that T has no other minimal period-two solutions in $〚({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1),({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2)〛=\{(x,y):({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1){⪯}_{ne}(x,y){⪯}_{ne}({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2)\}$ and $D(p)\ne 0$, then one of them is a saddle point and the other is a repeller.

Proof Period-two solution curves ${\mathcal{C}}_F=\{(x,y)\in \mathcal{R}:\tilde{F}(x,y)=0\}$ and ${\mathcal{C}}_G=\{(x,y)\in \mathcal{R}:\tilde{G}(x,y)=0\}$, where

$$\begin{array}{c}\tilde{F}(x,y)=b_1^3{x}^7-A_1{x}^4{(A_1+y^2)}^2-{(A_1+y^2)}^2{(a_2+c_2{y}^2)}^2\text{and }\hfill \\ \tilde{G}(x,y)=b_1^{2}y{x}^8-a_2{x}^4{(A_1+y^2)}^2-c_2{(A_1+y^2)}^2{(a_2+c_2{y}^2)}^2,\hfill \end{array}$$

are algebraic curves. By using software Mathematica, one can see that the resultant of the polynomials $\tilde{F}(x,y)$ and $\tilde{G}(x,y)$ in variable x is given by

$$\begin{array}{rl}R(\tilde{F},\tilde{G})& =-b_1^6{(A_1+y^2)}^{14}{(a_2+c_2{y}^2)}^8(-a_2{b}_1^2+2A_1{y}^3+A_1^{2}y-b_1^2{c}_2{y}^2+y^5)p(y)\\ =-b_1^6{(A_1+y^2)}^{14}{(a_2+c_2{y}^2)}^8\tilde{f}(y)p(y).\end{array}$$

Suppose that $\{({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1),T({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)\}$ is a non-hyperbolic minimal period-two solution. This implies that $\tilde{F}({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)=0$, $\tilde{G}({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)=0$. By Theorem 9.3 [28], $\tilde{F}$ and $\tilde{G}$ have a common non-constant factor if and only $R(\tilde{F},\tilde{G})=0$, which implies that system $\tilde{F}(x,y)=0$, $\tilde{G}(x,y)=0$ has a solution if and only if $R(\tilde{F},\tilde{G})=0$. Since $\tilde{f}({\mathrm{\Psi }}_1)\ne 0$, it must be $p({\mathrm{\Psi }}_1)=0$. Similarly as in Lemma 4, one can see that

$$\begin{array}{rl}{x}_F^{\prime }({\mathrm{\Psi }}_1)-x_G^{\prime }({\mathrm{\Psi }}_1)& =\frac{F_y^{\prime }({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)}{1-F_x^{\prime }({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)}-\frac{1-G_y^{\prime }({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)}{G_x^{\prime }({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)}\\ =\frac{f_1}{1-e_1}-\frac{1-h_1}{g_1}=\frac{-1+(e_1+g_1)-(e_1{h}_1-f_1{g}_1)}{g_1(1-e_1)}\\ =\frac{-p_1(1)}{g_1(1-e_1)}=\frac{(1-{\mu }_1)(1-{\mu }_2)}{g_1(e_1-1)},\end{array}$$

where $p_1(\mu )$ is the characteristic equation of the matrix

$$J_{T^2}({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)=\left(\begin{array}{cc}{e}_1& f_1\\ g_1& h_1\end{array}\right).$$

From Theorem 14 we have that $0<{\mu }_1<{\mu }_2$ and ${\mu }_2>1$. Since $\{({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1),T({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)\}$ is non-hyperbolic, we obtain that ${\mu }_1=1$, from which it follows that $x_F^{\prime }({\mathrm{\Psi }}_1)-x_G^{\prime }({\mathrm{\Psi }}_1)=0$. Since $R(\tilde{F},\tilde{G})\not\equiv 0$, we have that ${\mathcal{C}}_F$ and ${\mathcal{C}}_G$ have no common component. By Lemmas 6 and 7, from [27], the curves ${\mathcal{C}}_F$ and ${\mathcal{C}}_G$ intersect transversally at $({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)$ (i.e., $y_{\tilde{F}}^{\prime }({\mathrm{\Psi }}_1)-y_{\tilde{G}}^{\prime }({\mathrm{\Psi }}_1)\ne 0$) if and only if ${\mathrm{\Psi }}_1$ is zero of $p(y)$ of multiplicity one. By Theorem 9.4 [28], $p(y)$ has zeros of multiplicity greater than one if and only if the discriminant $D(p)$ of the polynomial $p(y)$ is equal to zero, which proves the first statement of the lemma.

Assume that $\{({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1),T({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1)\}$ and $\{({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2),T({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2)\}$ are two minimal period-two solutions such that T has no other minimal period-two solutions in $〚({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1),({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2)〛=\{(x,y):({\mathrm{\Phi }}_1,{\mathrm{\Psi }}_1){⪯}_{ne}(x,y){⪯}_{ne}({\mathrm{\Phi }}_2,{\mathrm{\Psi }}_2)\}$ and $D(p)\ne 0$. From the previous discussion we have $x_F^{\prime }({\mathrm{\Psi }}_i)-x_G^{\prime }({\mathrm{\Psi }}_i)\ne 0$, $i=1,2$. Since $x_F({\mathrm{\Psi }}_i)-x_G({\mathrm{\Psi }}_i)=0$, $i=1,2$, it follows that $(x_F^{\prime }({\mathrm{\Psi }}_1)-x_G^{\prime }({\mathrm{\Psi }}_1))(x_F^{\prime }({\mathrm{\Psi }}_2)-x_G^{\prime }({\mathrm{\Psi }}_2))<0$. Indeed assume, for example, that $x_F^{\prime }({\mathrm{\Psi }}_1)-x_G^{\prime }({\mathrm{\Psi }}_1)<0$ and $x_F^{\prime }({\mathrm{\Psi }}_2)-x_G^{\prime }({\mathrm{\Psi }}_2)<0$. Then there exists $ϵ>0$ such that $x_F(y)-x_G(y)<0$ for $y\in ({\mathrm{\Psi }}_1,{\mathrm{\Psi }}_1+ϵ)$ and $x_F(y)-x_G(y)>0$ for $y\in ({\mathrm{\Psi }}_2-ϵ,{\mathrm{\Psi }}_2)$. Since $x_F(y)-x_G(y)$ is a continuous function, this implies that there exists $\mathrm{\Psi }\in ({\mathrm{\Psi }}_1,{\mathrm{\Psi }}_2)$ such that $x_F(\mathrm{\Psi })-x_G(\mathrm{\Psi })=0$, which is a contradiction. The rest of the proof follows from the fact that $e_i>4$ and $g_i<0$, $i=1,2$. □

Notice that

$$D(p)=-\frac{1}{p_{22}}R(p,p^{\prime }),$$

where $R(p,p^{\prime })=detSyl(p,p^{\prime })$, the determinant of the Sylvester matrix $Syl(p,p^{\prime })$, see [28, 29], and

$$Syl(p,p^{\prime })=\left(\begin{array}{cccccccc}{p}_{22}& p_{21}& \cdots & p_1& p_0& 0& \cdots & 0\\ 0& p_{22}& p_{21}& \cdots & p_1& p_0& \cdots & 0\\ ⋮\\ 0& \cdots & p_{22}& p_{21}& p_{20}& \cdots & p_1& p_0\\ 22p_{21}& 21p_{20}& \cdots & p_1& 0& 0& \cdots & 0\\ 0& 22p_{21}& 21p_{20}& \cdots & p_1& 0& \cdots & 0\\ ⋮\\ 0& 0& \cdots & 22p_{21}& 21p_{20}& 20p_{19}& \cdots & p_1\end{array}\right).$$

Theorem 16 If $a_2>A_1{c}_2$ and $\mathrm{\Gamma }(\overline{y})<0$, then T has one equilibrium point $E(\overline{x},\overline{y})$, which is a repeller, and there exists at least one minimal period-two solution $\{(\psi ,\varphi ),T(\psi ,\varphi )\}$ which is non-hyperbolic or a saddle point. If T has no minimal period-two solutions which are non-hyperbolic, then $(\psi ,\varphi ){\ll }_{ne}E{\ll }_{ne}T(\psi ,\varphi )$.

Proof By Theorem 10 we have that T has one equilibrium point $E(\overline{x},\overline{y})$, which is a repeller. This and Lemma 2 imply that $T^n(x_0,y_0)$ is asymptotic to either $(0,\mathrm{\infty })$ or $(\mathrm{\infty },0)$, or a minimal period-two solution, for all $(x_0,y_0)\in \mathcal{R}$. Let $\mathcal{B}(0,\mathrm{\infty })$ be the basin of attraction of $(0,\mathrm{\infty })$, and let $\mathcal{B}(\mathrm{\infty },0)$ be the basin of attraction of $(\mathrm{\infty },0)$. By using Theorem 9 one can prove that $int(Q_2(E))\subset \mathcal{B}(0,\mathrm{\infty })$ and $int(Q_4(E))\subset \mathcal{B}(\mathrm{\infty },0)$. Let ${\mathcal{S}}_1$ denote the boundary of $\mathcal{B}(\mathrm{\infty },0)$ considered as a subset of $Q_1(E)$, and let ${\mathcal{S}}_2$ denote the boundary of $\mathcal{B}(\mathrm{\infty },0)$ considered as a subset of $Q_3(E)$. It is easy to see that $E\in {\mathcal{S}}_1$, $E\in {\mathcal{S}}_2$ and $T(\mathcal{R})\subset int(\mathcal{R})$. Now we prove the following claim.

Claim 1 Let ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$ be the sets defined as above. Then

• (a) If $(x_0,y_0)\in \mathcal{B}(\mathrm{\infty },0)$, then $(x_1,y_1)\in \mathcal{B}(\mathrm{\infty },0)$ for all $(x_0,y_0){⪯}_{se}(x_1,y_1)$.

• (b) If $(x_0,y_0)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2$, then $(x_1,y_1)\in int(\mathcal{B})(\mathrm{\infty },0)$ for all $(x_0,y_0){\ll }_{se}(x_1,y_1)$.

• (c) ${\mathcal{S}}_1\cap int(Q_1(E))\ne \mathrm{\varnothing }$ and ${\mathcal{S}}_2\cap int(Q_3(E))\ne \mathrm{\varnothing }$.

• (d) $T({\mathcal{S}}_1\cup {\mathcal{S}}_2)\subseteq {\mathcal{S}}_1\cup {\mathcal{S}}_2$.

• (e) $(x_0,y_0),(x_1,y_1)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2\Rightarrow (x_0,y_0){\ll }_{ne}(x_1,y_1)$ or $(x_1,y_1){\ll }_{ne}(x_0,y_0)$.

Proof

• (a) The statement follows from $T^n(x_0,y_0){⪯}_{se}{T}^n(x_1,y_1){⪯}_{se}(\mathrm{\infty },0)$ and $T^n(x_0,y_0)\to (\mathrm{\infty },0)$ as $n\to \mathrm{\infty }$.

• (b) The claim (b) follows from the observation that there exists a ball centered at $(x_0,y_0)$ with the property that all its points $(x,y)$ satisfy $(x,y){\ll }_{se}(x_1,y_1)$. But one of these points necessarily lies in $\mathcal{B}(\mathrm{\infty },0)$, so by (a) there exists $(x_1,y_1)\in \mathcal{B}(\mathrm{\infty },0)$. Furthermore, there exists a ball centered at $(x_1,y_1)$ with the property that all its points $(x,y)$ satisfy $(x,y)\in \mathcal{B}(\mathrm{\infty },0)$, which implies $(x_1,y_1)\in int(\mathcal{B})(\mathrm{\infty },0)$.

• (c) Take $y^{\prime }>\overline{y}$ arbitrary (but fixed). Since T is strongly competitive, we have $T(\overline{x},y^{\prime }){\ll }_{se}T(\overline{x},\overline{y})$, which implies $T(\overline{x},y^{\prime })\in int(Q_2(E))$. This implies that there exists a ball $B_{\epsilon }(T(\overline{x},y^{\prime }))$ with the property $B_{\epsilon }(T(\overline{x},y^{\prime }))\subset int(Q_2(E))$. Since T is a continuous map on a set ${\mathbb{R}}_{+}^2\setminus \{(0,y):y\ge 0\}$, then there exists a ball $B_{{\delta }_1}(\overline{x},y^{\prime })$ such that $T(B_{{\delta }_1}(\overline{x},y^{\prime }))\subset B_{\epsilon }(T(\overline{x},y^{\prime }))\subset int(Q_2(E))$, which implies $T^n(x,y)\to (0,\mathrm{\infty })$ as $n\to \mathrm{\infty }$ for all $(x,y)\in B_{{\delta }_1}(\overline{x},y^{\prime })$. Similarly, one can prove that then there exists a ball $B_{{\delta }_2}(\overline{x}+{\delta }_1/2,\overline{y})$ such that $T^n(x,y)\to (\mathrm{\infty },0)$ as $n\to \mathrm{\infty }$ for all $(x,y)\in B_{{\delta }_2}(\overline{x}+{\delta }_1/2,\overline{y})$. Let $y^{″}=sup\{y:{lim}_{n\to \mathrm{\infty }}{T}^n(\overline{x}+{\delta }_1/2,y)=(\mathrm{\infty },0)\}$. It is easy to see that $(\overline{x}+{\delta }_1/2,y^{″})\in {\mathcal{S}}_1\cap int(Q_1(E))$. The assertion concerning ${\mathcal{S}}_2$ is proved in a similar fashion.

• (d) Take $(x,y)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2$. Assume that $T(x,y)\notin {\mathcal{S}}_1\cup {\mathcal{S}}_2$. Since ${\mathcal{S}}_1\cup {\mathcal{S}}_2=\partial \mathcal{B}(\mathrm{\infty },0)=\overline{\mathcal{B}(\mathrm{\infty },0)}\setminus int(\mathcal{B}(\mathrm{\infty },0))$, then either $T(x,y)\in int(\mathcal{B}(\mathrm{\infty },0))$ or $T(x,y)\notin \overline{\mathcal{B}(\mathrm{\infty },0)}$. Assume that $T(x,y)\in int(\mathcal{B}(\mathrm{\infty },0))$. This implies that there exists a ball $B_{\epsilon }(T(x,y))$ with the property $B_{\epsilon }(T(x,y))\subset int(\mathcal{B}(\mathrm{\infty },0))$. Since T is a continuous map on the set ${\mathbb{R}}_{+}^2\setminus \{(0,y):y\ge 0\}$, then there exists a ball $B_{\delta }(x,y)$, $\delta >0$ such that $T(B_{\delta }(x,y))\subset B_{\epsilon }(T(x,y))$, which implies $B_{\delta }(x,y)\subset \mathcal{B}(\mathrm{\infty },0)$. This is in contradiction with $(x,y)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2=\partial \mathcal{B}(\mathrm{\infty },0)$. Hence $T(x,y)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2$ in this case. Similarly, one can prove that $T(x,y)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2$ if $T(x,y)\notin \overline{\mathcal{B}(\mathrm{\infty },0)}$. This implies that $T({\mathcal{S}}_1\cup {\mathcal{S}}_2)\subseteq ({\mathcal{S}}_1\cup {\mathcal{S}}_2)$.

• (e) Assume that $(x_0,y_0),(x_1,y_1)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2\Rightarrow (x_0,y_0){⪯}_{se}(x_1,y_1)$ and $(x_0,y_0)\ne (x_1,y_1)$. Since T is strongly competitive, we get $T(x_0,y_0){\ll }_{ne}T(x_1,y_1)$. This contradicts (e) and (b), which completes the proof. □

In view of Claim 1, we have that $({\mathcal{S}}_1\cup {\mathcal{S}}_2,{\ll }_{ne})$ is a totally ordered set which is invariant under T. If $(x_0,y_0)\in {\mathcal{S}}_1\cup {\mathcal{S}}_2$, then $\{T^{(2n)}(x_0,y_0)\}$ is eventually component-wise monotone. Then there exists a minimal period-two solution $\{(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\}\in {\mathcal{S}}_1\cup {\mathcal{S}}_2\subset Q_1(E)\cup Q_3(E)$ such that $T^{(2n)}(x_0,y_0)\to (\mathrm{\Phi },\mathrm{\Psi })$ as $n\to \mathrm{\infty }$. By Theorem 14, $\{(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\}$ is a non-hyperbolic or a saddle point. Assume that T has no minimal period-two solutions which are non-hyperbolic points and, for example, that $(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\in {\mathcal{S}}_1$ such that $E{\ll }_{ne}(\mathrm{\Phi },\mathrm{\Psi }){\ll }_{ne}T(\mathrm{\Phi },\mathrm{\Psi })$. Since $\{(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\}$ is a saddle point, in view of Theorems 6, 7 and 8, we have that the global stable manifolds ${\mathcal{W}}^s(\{(\mathrm{\Phi },\mathrm{\Psi }),T(\mathrm{\Phi },\mathrm{\Psi })\})$ are the union of two curves ${\mathcal{W}}^s(\mathrm{\Phi },\mathrm{\Psi })$ and ${\mathcal{W}}^s(T(\mathrm{\Phi },\mathrm{\Psi }))$ whose endpoints are repeller points such that $T({\mathcal{W}}^s(\mathrm{\Phi },\mathrm{\Psi }))={\mathcal{W}}^s(T(\mathrm{\Phi },\mathrm{\Psi }))$ and $E{\ll }_{ne}\overline{{\mathcal{W}}^s(\mathrm{\Phi },\mathrm{\Psi })}{\ll }_{ne}\overline{{\mathcal{W}}^s(T(\mathrm{\Phi },\mathrm{\Psi }))}$. If $P_1$ and $P_2$ ($P_1{⪯}_{ne}{P}_2$) are endpoints of ${\mathcal{W}}^s(\mathrm{\Phi },\mathrm{\Psi })$, then $T(P_1)$ and $T(P_2)$ are endpoints of $T({\mathcal{W}}^s(\mathrm{\Phi },\mathrm{\Psi }))$ and either $T(P_1){⪯}_{ne}T(P_2)$ or $T(P_2){⪯}_{ne}T(P_1)$. Assume, for example, that $P_1{\ll }_{ne}{\mathcal{W}}^s(\mathrm{\Phi },\mathrm{\Psi }){\ll }_{ne}{P}_2{\ll }_{ne}T(P_2){\ll }_{ne}{\mathcal{W}}^s(T(\mathrm{\Phi },\mathrm{\Psi })){\ll }_{ne}T(P_1)$. By Theorem 15 between two repellers $P_2$ and $T(P_2)$, there exists a saddle point $S_1$ where its stable manifold is the union of two invariant curves ${\mathcal{W}}^s(S_1)$ and ${\mathcal{W}}^s(T(S_1))$ whose endpoints are repellers such that $P_2{\ll }_{ne}\overline{{\mathcal{W}}^s(S_1)}{\ll }_{ne}T(P_2)$. Continuing in this way, we obtain that T has infinitely many minimal period-two solutions $\{P_i,T(P_i)\}$, which is in contradiction with the fact that T has at most eleven minimal period-two solutions. Hence $(\mathrm{\Phi },\mathrm{\Psi }){\ll }_{ne}E{\ll }_{ne}T(\mathrm{\Phi },\mathrm{\Psi })$.  □

Corollary 2 Assume that $a_2>A_1{c}_2$, $\mathrm{\Gamma }(\overline{y})<0$ and $D(p)\ne 0$. Then there exists one equilibrium point E which is a repeller. Further, the set $int(Q_1(E))\cup int(Q_3(E))$ contains an odd number of minimal period-two solutions $\{({\mathrm{\Phi }}_i,{\mathrm{\Psi }}_i),({\tilde{\mathrm{\Phi }}}_i,{\tilde{\mathrm{\Psi }}}_i)\}$, $i=1,\dots ,2n+1$, such that $({\mathrm{\Phi }}_{i+1},{\mathrm{\Psi }}_{i+1}){\ll }_{ne}({\mathrm{\Phi }}_i,{\mathrm{\Psi }}_i){\ll }_{ne}E$ and $E{\ll }_{ne}({\tilde{\mathrm{\Phi }}}_i,{\tilde{\mathrm{\Psi }}}_i){\ll }_{ne}({\tilde{\mathrm{\Phi }}}_{i+1},{\tilde{\mathrm{\Psi }}}_{i+1})$, where $({\tilde{\mathrm{\Phi }}}_i,{\tilde{\mathrm{\Psi }}}_i)=T({\mathrm{\Phi }}_i,{\mathrm{\Psi }}_i)$. Furthermore, odd indexed period-two points are saddles and even indexed period-two points are repellers.

Proof By Theorem 10 we have that T has one equilibrium point $E(\overline{x},\overline{y})$, which is a repeller. By Theorem 15 all minimal period-two solutions are hyperbolic and the number of minimal period-two solutions is finite. In view of Claim 1, T has at least one minimal period-two solution which is a saddle point. Let $\{P_1,T(P_1)\}$ be a minimal period-two solution which is a saddle point such that $P_1{\ll }_{ne}E{\ll }_{ne}T(P_1)$ and T has no minimal period-two solutions in $〚E,T(P_1)〛$ and $〚P_1,E〛$. Such a minimal period-two solution exists in view of Theorem 15. The map $T^2$ satisfies all conditions of Theorems 6, 7 and 8, which yields the existence of the global stable manifolds ${\mathcal{W}}^s(\{P_1,{\tilde{P}}_1\})$ which are the union of two curves ${\mathcal{W}}^s(P_1)$ and ${\mathcal{W}}^s({\tilde{P}}_1)$ that have a common endpoint E. If T has minimal period-two solutions in $int(Q_1(T(P_1)))\cup int(Q_3(P_1))$, let $\{P_2,{\tilde{P}}_2\}$ ($P_2{\ll }_{ne}{\tilde{P}}_2$) denote minimal period-two solutions such that T has no other minimal period-two solutions in $〚T(P_1),T(P_2)〛$ and $〚P_2,P_1〛$. Then ${\mathcal{W}}^s(P_1)$ has the second endpoint at $P_2$ and ${\mathcal{W}}^s(T(P_1))$ has the second endpoint at $T(P_2)$ and $P_2{\ll }_{ne}{P}_1{\ll }_{ne}E{\ll }_{ne}T(P_1){\ll }_{ne}T(P_2)$. Furthermore, a minimal period-two solution $\{P_2,T(P_2)\}$ is a repeller. Similarly as in Theorem 16, one can prove that $int(Q_1(T(P_2)))\cup int(Q_3(P_2))$ contains at least one minimal period-two solution which is a saddle point. Since the number of minimal period-two solutions is finite, continuing in this way, we will end with a minimal period-two solution which is a saddle point, from which the proof follows. □

Corollary 3 If $a_2>A_1{c}_2$ and $\mathrm{\Gamma }(\overline{y})>0$, then there exists one equilibrium point E which is a saddle point. If $D(p)\ne 0$, then $int(Q_1(E))\cup int(Q_3(E))$ contains an even number of minimal period-two solutions $\{({\mathrm{\Phi }}_i,{\mathrm{\Psi }}_i),({\tilde{\mathrm{\Phi }}}_i,{\tilde{\mathrm{\Psi }}}_i)\}$, $i=1,\dots ,2n$, such that $({\mathrm{\Phi }}_{i+1},{\mathrm{\Psi }}_{i+1}){\ll }_{ne}({\mathrm{\Phi }}_i,{\mathrm{\Psi }}_i){\ll }_{ne}E$ and $E{\ll }_{ne}({\tilde{\mathrm{\Phi }}}_i,{\tilde{\mathrm{\Psi }}}_i){\ll }_{ne}({\tilde{\mathrm{\Phi }}}_{i+1},{\tilde{\mathrm{\Psi }}}_{i+1})$ and $({\tilde{\mathrm{\Phi }}}_i,{\tilde{\mathrm{\Psi }}}_i)=T({\mathrm{\Phi }}_i,{\mathrm{\Psi }}_i)$. Furthermore, even indexed period-two points are saddles and odd indexed period-two points are repellers.

Proof The proof is similar as the proof of Corollary 2 and it will be omitted. □

Based on a series of numerical simulations, we propose the following conjecture.

Conjecture 1 System (1) has at most one minimal period-two solution.

5 Global behavior

In this section we present global dynamics of system (1) in different parametric regions. We have five parametric regions with different dynamics which will be characterized by the following five theorems.

5.1 The case where the equilibrium points and period-two solutions are hyperbolic points (${\mathrm{\Delta }}_1\ne 0$ and $D(p)\ne 0$)

Theorem 17 Assume that ${\mathrm{\Delta }}_1<0$. Then system (1) has three equilibrium solutions $E_1{\ll }_{ne}{E}_2{\ll }_{ne}{E}_3$, where $E_1$ and $E_3$ are saddle points and $E_2$ is a repeller. In this case there exist four invariant continuous curves ${\mathcal{W}}^s(E_1)$, ${\mathcal{W}}^s(E_3)$, ${\mathcal{W}}^u(E_1)$, ${\mathcal{W}}^u(E_3)$, where ${\mathcal{W}}^s(E_1)$, ${\mathcal{W}}^s(E_3)$ have end points at $E_2$, and are graphs of increasing functions. The curves ${\mathcal{W}}^u(E_1)$, ${\mathcal{W}}^u(E_3)$ are the graphs of decreasing functions. Every solution $\{(x_n,y_n)\}$ which starts below ${\mathcal{W}}^s(E_1)\cup {\mathcal{W}}^s(E_3)$ in the south-east ordering is asymptotic to $(0,\mathrm{\infty })$, and every solution $\{(x_n,y_n)\}$ which starts above ${\mathcal{W}}^s(E_1)\cup {\mathcal{W}}^s(E_3)$ in the south-east ordering is asymptotic to $(\mathrm{\infty },0)$. The first quadrant of initial condition $Q_1=\{(x_0,y_0):x_0>0,y_0\ge 0\}$ is the union of five disjoint basins of attraction, i.e.,

$$Q_1=\mathcal{B}(0,\mathrm{\infty })\cup \mathcal{B}(\mathrm{\infty },0)\cup \mathcal{B}(E_1)\cup \mathcal{B}(E_3)\cup \mathcal{B}(E_2),$$

where $\mathcal{B}(E_2)=\{E_2\}$, $\mathcal{B}(E_1)={\mathcal{W}}^s(E_1)$, $\mathcal{B}(E_3)={\mathcal{W}}^s(E_3)$ and

$$\begin{array}{c}\mathcal{B}(0,\mathrm{\infty })=\{(x,y)|(x,y){⪯}_{se}({\tilde{x}}_0,{\tilde{y}}_0)\mathit{\text{ for some }}({\tilde{x}}_0,{\tilde{y}}_0)\in {\mathcal{W}}^s(E_1)\cup {\mathcal{W}}^s(E_3)\},\hfill \\ \mathcal{B}(\mathrm{\infty },0)=\{(x,y)|({\tilde{x}}_1,{\tilde{y}}_1){⪯}_{se}(x,y)\mathit{\text{ for some }}({\tilde{x}}_1,{\tilde{y}}_1)\in {\mathcal{W}}^s(E_1)\cup {\mathcal{W}}^s(E_3)\}.\hfill \end{array}$$

Proof Theorem 11 implies that there exist three equilibrium points, namely $E_1$, $E_2$ and $E_3$, such that $E_1{\ll }_{ne}{E}_2{\ll }_{ne}{E}_3$. In this case, $E_1$ and $E_3$ are saddle points and $E_2$ is a repeller. In view of (17), the map T is competitive on ℛ and strongly competitive on $int(\mathcal{R})$. It follows from the Perron-Frobenius theorem and a change of variables [14] that at each point the Jacobian matrix of a strongly competitive map has two real and distinct eigenvalues, the larger one in absolute value being positive, and that corresponding eigenvectors may be chosen to point in the direction of the second and first quadrant, respectively. Also, one can show that if the map is strongly competitive, then no eigenvector is aligned with a coordinate axis.

Since ${\mathrm{\Delta }}_1<0$ implies $a_2<A_1{c}_2$, we have that $det{J}_T(E_i)>0$, $i=1,3$. Hence, all conditions of Theorems 6, 7 and 8 are satisfied, which yields the existence of the global stable manifolds ${\mathcal{W}}^s(E_1)$, ${\mathcal{W}}^s(E_3)$ and the global unstable manifolds ${\mathcal{W}}^u(E_1)$, ${\mathcal{W}}^u(E_3)$, where ${\mathcal{W}}^s(E_1)$, ${\mathcal{W}}^s(E_3)$ are passing through the point $E_2$ and are graphs of increasing functions. The curves ${\mathcal{W}}^u(E_1)$, ${\mathcal{W}}^u(E_3)$ are the graphs of decreasing functions. Let

$$\begin{array}{c}{\mathcal{W}}^{-}=\{(x,y)|(x,y){⪯}_{se}({\tilde{x}}_0,{\tilde{y}}_0)\mathit{\text{ for some }}({\tilde{x}}_0,{\tilde{y}}_0)\in {\mathcal{W}}^s(E_1)\cup {\mathcal{W}}^s(E_3)\},\hfill \\ {\mathcal{W}}^{+}=\{(x,y)|({\tilde{x}}_1,{\tilde{y}}_1){⪯}_{se}(x,y)\mathit{\text{ for some }}({\tilde{x}}_1,{\tilde{y}}_1)\in {\mathcal{W}}^s(E_1)\cup {\mathcal{W}}^s(E_3)\}.\hfill \end{array}$$

Take $(x_0,y_0)\in {\mathcal{W}}^{-}\cap \mathcal{R}$. By Theorem 7 we have that there exists $n_0>0$ such that $T^n(x_0,y_0)\in int((Q_2(E_1)\cup Q_2(E_3))\cap \mathcal{R})$, $n>n_0$. In view of Theorem 9, since $E_1$ and $E_3$ are saddle points, we obtain that for all $(x_0,y_0)\in int((Q_2(E_1)\cup Q_2(E_3))\cap \mathcal{R})$, there exists a subsolution $(u_0,v_0)$ such that $(x_0,y_0){⪯}_{se}(u_0,v_0)$. By monotonicity we have that $(0,\mathrm{\infty }){⪯}_{se}{T}^n(x_0,y_0){⪯}_{se}{T}^n(u_0,v_0)\ll E_3\text{ if }(x_0,y_0)\in int(Q_2(E_3)\cap \mathcal{R})$ and $(0,\mathrm{\infty }){⪯}_{se}{T}^n(x_0,y_0){⪯}_{se}{T}^n$