Global dynamics of cubic second order difference equation in the first quadrant

Abstract

We investigate the global behavior of a cubic second order difference equation $x_{n+1}=Ax_{n}^{3}+ Bx_{n}^{2}x_{n-1}+Cx_{n}x_{n-1}^{2}+Dx_{n-1}^{3}+Ex_{n}^{2} +Fx_{n}x_{n-1}+Gx_{n-1}^{2}+Hx_{n}+Ix_{n-1}+J$, $n=0,1,\ldots$ , with nonnegative parameters and initial conditions. We establish the relations for the local stability of equilibriums and the existence of period-two solutions. We then use this result to give global behavior results for special ranges of the parameters and determine the basins of attraction of all equilibrium points. We give a class of examples of second order difference equations with quadratic terms for which a discrete version of the 16th Hilbert problem does not hold. We also give the class of second order difference equations with quadratic terms for which the Julia set can be found explicitly and represent a planar quadratic curve.

1 Introduction and preliminaries

In this paper we study the global dynamics of the following polynomial difference equation:

$$ x_{n+1}=Ax_{n}^{3}+Bx_{n}^{2}x_{n-1}+Cx_{n}x_{n-1}^{2} +Dx_{n-1}^{3}+Ex_{n}^{2}+Fx_{n}x_{n-1}+Gx_{n-1}^{2}+Hx_{n}+Ix_{n-1}+J, $$

(1)

where the parameters A, B, C, D, E, F, G, H, I, J are nonnegative numbers with condition $A+B+C+D>0$ and initial conditions $x_{-1}$ and $x_{0}$ are arbitrary nonnegative numbers. The condition $A+B+C+D>0$ is necessary in order to avoid the quadratic case, which was completely studied in [1]. Polynomial difference equations and corresponding maps have been studied in both the real and the complex domain and many results have been obtained; see [1–3]. One of the major problems in the dynamics of polynomial maps is determining the basin of attraction of the point at ∞ and in particular the boundary of that basin known as the Julia set. In [1] we precisely determined the Julia set for a second order quadratic polynomial equation, that is, (1) where $A+B+C+D=0$, and we obtained the global dynamics in the interior of the Julia set, which includes all the point for which solutions are not asymptotic to the point at ∞. It turned out that the Julia set for (1), where $A+B+C+D=0$, is the union of the stable manifolds of some saddle equilibrium points or nonhyperbolic equilibrium points or period-two points. The asymptotic formulas for these manifolds were obtained in [4]. The advantage of our results is that these manifolds are continuous decreasing functions of which the parametrization is simple and so their asymptotic formulas can easily be derived by using the method of undetermined coefficients.

In this paper we restrict our attention to nonnegative initial conditions and nonnegative parameters, which will make our results more special but also more precise and applicable. Our results are based on a number of theorems which hold for monotone difference equations, which will be described in the next section. Our principal tool is the theory of monotone maps, and in particular cooperative maps, which guarantee the existence and uniqueness of the stable and unstable manifolds for the fixed points and periodic points. Our results can be extended to (1) to hold in the whole plane, when $B=C=E=F=G=0$.

Remark 1

The following result is a consequence of our results in [1]: If $A=B=C=D=E=H=J=0$, $F=G>0$ and $I<1$, then (1), which reduces to the quadratic second order difference equation

$$x_{n+1} = F\bigl(x_{n} x_{n-1} + x_{n-1}^{2}\bigr) + I x_{n-1}, \quad n=0,1,\ldots, $$

has infinitely many minimal period-two solutions, which are given by

$$\mathcal{H}=\bigl\{ (x,y):F(x+y)+I=1,x>0,y>0\bigr\} . $$

The curve $\mathcal{H}$ separates the first quadrant into two regions: the region below the curve $\mathcal{H}$ is the basin of attraction of $E_{0} (0,0 )$ and the region above the curve is the basin of attraction of point at infinity. Thus, the curve $\mathcal{H}$ is the Julia set for (1) in this case. This result shows that the discrete version of the 16th Hilbert problem does not hold, which is the problem if there exists a quadratic system of difference equations in the plane with an infinite number of periodic solutions. It is well known that in the case of quadratic systems of differential equations the number of periodic solutions is finite; see [5, 6]. In this paper we give the explicit formula for the Julia set for a whole class of difference equations with cubic terms. The Julia set consists of an infinite number of period-two solutions and thus provides the whole class of examples of the second order difference equations with the cubic terms with an infinite number of a period-two solutions; see Theorem 17.

The rest of this section presents some results as regards monotone difference equations in the plane. The second section presents the local stability analysis of the equilibrium solutions. The third section describes the local stability analysis of the period-two points in all cases. The fourth section gives the global dynamics, which includes the basins of attraction of all equilibrium points and the period-two points. Some Mathematica outputs are given in the Appendix.

Consider the difference equation

$$ x_{n+1}=f(x_{n},x_{n-1}),\quad n=0,1,\ldots, $$

(2)

where f is a continuous and increasing function in both variables.

Here we list some of the results that will be needed in this paper. The first result was obtained in [7] and it was extended to the case of higher order difference equations and systems in [8, 9].

Theorem 1

Let $f:[a,b]\times [ a,b]\rightarrow [ a,b]$ be a continuous function satisfying the following properties:

• (i) $f(x,y)$ is nondecreasing in each of its arguments, i.e. $x\rightarrow f(x,y)$ is nondecreasing for every y and $y\rightarrow f(x,y)$ is nondecreasing for every x;

• (ii) (2) has a unique equilibrium $\overline{x}\in [ a,b]$.

Then every solution of (2) converges to $\overline{x}$.

The following result was obtained in [10].

Theorem 2

Let $I\subseteq\mathbb{R}$ be an interval (finite or infinite) and let $f\in C[I\times I,I]$ be a function which increases in both variables. Then for every solution of (2) the subsequences $\{x_{2n}\}_{n=0}^{\infty}$ and $\{x_{2n+1}\} _{n=-1}^{\infty}$ of even and odd terms of the solution do exactly one of the following:

• (i) Eventually they are both monotonically increasing.

• (ii) Eventually they are both monotonically decreasing.

• (iii) One of them is monotonically increasing and the other is monotonically decreasing.

As a consequence of Theorem 2 every bounded solution of (1) approaches either an equilibrium solution or period-two solution or the singular point on the boundary and every unbounded solution is asymptotic to the point at infinity in a monotonic way; see [11]. Thus the major problem in the dynamics of (1) is the problem of determining the basins of attraction of three different types of attractors: the equilibrium solutions, period-two solution(s), and the point(s) at infinity. The following two results can be proved by using the techniques of the proof of Theorem 11 in [12].

Theorem 3

Consider (2) where $I\subseteq\mathbb{R}$ is an interval (finite or infinite) and $f\in C[I\times I,I]$ is an increasing function in its arguments and assume that there is no minimal period-two solution. Assume that $E_{1}(x_{1},y_{1})$ and $E_{2}(x_{2},y_{2})$ are two consecutive equilibrium points in North-East ordering that satisfy

$$ (x_{1},y_{1})\preceq_{ne}(x_{2},y_{2}) $$

and that $E_{1}$ is a local attractor and $E_{2}$ is a saddle point or a nonhyperbolic point with second characteristic root in the interval $(-1,1)$, with the neighborhoods where f is strictly increasing. Then the basin of attraction $\mathcal{B}(E_{1})$ of $E_{1}$ is the region below the global stable manifold $\mathcal{W}^{s}(E_{2})$. More precisely

$$ {\mathcal{B}}(E_{1})=\bigl\{ (x,y):\exists y_{u}: y< y_{u}, (x,y_{u})\in { \mathcal{W}}^{s}(E_{2}) \bigr\} . $$

The basin of attraction $\mathcal{B}(E_{2})=\mathcal{W}^{s}(E_{2})$ is exactly the global stable manifold of $E_{2}$. The global stable manifold extends to the boundary of the domain of (2). If there exists a period-two solution, then the end points of the global stable manifold are exactly the period-two solution.

Also, we will use the following theorem from [12].

Theorem 4

Consider (2) where $I\subseteq\mathbb{R}$ is an interval (finite or infinite) and $f\in C[I\times I,I]$ is an increasing function in its arguments and assume that there is no minimal period-two solution. Assume that $E_{1}(x_{1},y_{1})$, $E_{2}(x_{2},y_{2})$, and $E_{3}(x_{2},y_{2})$ are three consecutive equilibrium points in North-East ordering that satisfy

$$ (x_{1},y_{1})\preceq_{ne}(x_{2},y_{2}) \preceq_{ne}(x_{3},y_{3}) $$

and that $E_{1}$ and $E_{3}$ are saddle points with the neighborhoods where f is strictly increasing, and $E_{2}$ is a local attractor. Then the basin of attraction $\mathcal{B}(E_{2})$ of $E_{2}$ is the region between the global stable manifolds $\mathcal{W}^{s}(E_{1})$ and $\mathcal{W}^{s}(E_{3})$. More precisely

$$ {\mathcal{B}}(E_{2})=\bigl\{ (x,y):\exists y_{u},y_{l}: y_{l}< y< y_{u}, (x,y_{l})\in{\mathcal{W}}^{s}(E_{1}),(x,y_{u})\in {\mathcal{W}} ^{s}(E_{3})\bigr\} . $$

The basins of attraction $\mathcal{B}(E_{1})=\mathcal{W}^{s}(E_{1})$ and $\mathcal{B}(E_{1})=\mathcal{W}^{s}(E_{1})$ are exactly the global stable manifolds of $E_{1}$ and $E_{3}$.

The following result gives the necessary and sufficient condition for the local stability of (2) when f is nondecreasing in all its arguments; see [13].

Theorem 5

Let

$$ z_{n+1}= p_{0}z_{n} + p_{1} z_{n-1}, \quad n=0,1,\ldots, $$

(3)

be the standard linearization about the equilibrium $\overline{x}$ of (2) where $p_{i}={\partial f}/{\partial x_{n-i}}(\bar{x},\bar{x})\geq0$, $i=0,1$. Then the equilibrium $\overline{x}$ of (2) is one of the following:

• (a) locally asymptotically stable if $p_{0}+p_{1}<1$,

• (b) nonhyperbolic and locally stable if $p_{0}+p_{1}=1$,

• (c) unstable if $p_{0}+p_{1}>1$.

Let $f_{1}(x):=a_{n} x^{n}+a_{n-1}x^{n-1}+\cdots+a_{1}x+a_{0}$ and $g_{1}(x):=b_{m} x^{m}+b_{m-1}x^{m-1}+\cdots+b_{1}x+b_{0}$ be two polynomials of degrees n and m, respectively. Their resultant (see [14–16]) $\operatorname{Res}(f_{1},g_{1})$ is the determinant of the $(m+n)\times(m+n)$ Sylvester matrix given by

$$\operatorname{Syl}(f_{1},g_{1})= \begin{pmatrix} a_{n} &a_{n-1} & \cdots&a_{1} & a_{0} & 0 & \cdots&0 \\ 0&a_{n} & a_{n-1} & \cdots& a_{1} & a_{0} & \cdots& 0 \\ \vdots& & & & & & & \\ 0 & \cdots&a_{n} &a_{n-1} & a_{n-2} & \cdots& a_{1} & a_{0}\\ b_{m} &b_{m-1} & \cdots& b_{1} & b_{0} & 0 & \cdots&0 \\ 0& b_{m} & b_{n-1} & \cdots& b_{1} & b_{0} & \cdots& 0 \\ \vdots& & & & & & & \\ 0 & 0 & \cdots& b_{m} & b_{m-1} & b_{m-2} & \cdots& b_{0} \end{pmatrix} . $$

The following theorem is from [14].

Theorem 6

Assume that $f_{1},g_{1}\in\mathbb{R}[x]$ and $n,m\geq1$. Then the following hold:

• (i) The discriminant of $f_{1}$ is given by

  $$\operatorname{Dis}(f_{1})=(-1)^{n(n-1)/2}\frac{1}{a_{n}}\operatorname{Res} \bigl(f_{1},f_{1}'\bigr) $$

  and

  $$\operatorname{Dis}(f_{1}\cdot g_{1})= \operatorname{Dis}(f_{1})\operatorname{Dis}(g_{1})\operatorname{Res}(f_{1},g_{1})^{2}, $$

  where $f_{1}'$ is the derivative of $f_{1}$.

• (ii) $\operatorname{Dis}(f_{1}) = 0 \Leftrightarrow f_{1}$ has a double root in $\mathbb{C}$: Equivalently, $f_{1}$ has n distinct roots in $\mathbb{C}$ if and only if the discriminant is ≠0.

For two bivariate polynomials $f,g \in\mathbb{R}[x,y]$, Theorem 7 in [14, 15] holds.

Theorem 7

Let $f (x,y),g(x,y) \in\mathbb{R}[x,y]$ where

$$f(x,y)=\sum_{i=0}^{n} f_{i}(x)y^{i} \quad\textit{and}\quad g(x,y)=\sum_{i=0}^{m} g_{i}(x)y^{i}, $$

and let $r(x) = \operatorname{Res}_{y}( f ,g) \in\mathbb{R}[x]$ be the resultant of f and g with respect to the variable y. Then:

• (a) f and g have a nontrivial common factor if and only if r is identically zero.

• (b) If f and g are co-prime (do not have a common factor), the following conditions are equivalent:

  • $\alpha\in\mathbb{C}$ is a root of r.

  • $f_{n}(\alpha) = g_{m}(\alpha) = 0$ or there is $\beta\in\mathbb {C}$ with $f (\alpha,\beta) = 0 = g(\alpha,\beta) = 0$.

• (c) For all $(\alpha,\beta)\in\mathbb{C}\times\mathbb{C}$: if $f(\alpha,\beta) = 0 = g(\alpha,\beta)= 0$, then $r(\alpha) = 0$.

We list some results when solution $\{ x_{n} \}$ of (1) tends to the point at infinity in a monotonic way.

Theorem 8

If $H+I>1$, then every solution $\{ x_{n} \}$ of (1) satisfies $\lim_{n\rightarrow\infty}x_{n}=\infty$.

Proof

If $\{ x_{n} \}$ is a solution of (1), then $\{ x_{n} \}$ satisfies the inequality

$$ x_{n+1}\geq Hx_{n}+Ix_{n-1},\quad n=0,1,\ldots, $$

which in view of the result on difference inequalities, see Theorem 1.4.1 in [17], implies that $x_{n}\geq y_{n}$, $n\geq1$, where $\{ y_{n} \}$ is a solution of the initial value problem

$$ y_{n+1}=Hy_{n}+Iy_{n-1},\qquad y_{-1}=x_{-1}\quad\mbox{and}\quad y_{0}=x_{0} \quad n=0,1,\ldots. $$

Consequently,

$$ x_{n}\geq y_{n}=\frac{1}{\sqrt{H^{2}+4I}} \bigl[ ( Ix_{-1}+\lambda _{2}x_{0} ) \lambda_{2}^{n}+ ( \lambda_{1}x_{-1}+Ix_{0} ) \lambda_{1}^{n} \bigr] ,\quad n=1,2,\ldots, $$

where

$$ \lambda_{1,2}=\frac{H\pm\sqrt{H^{2}+4I}}{2}\quad\mbox{and}\quad\lambda _{1}< \lambda_{2}\quad\mbox{and}\quad\lambda_{2}>1, $$

which implies $\lim_{n\rightarrow\infty}x_{n}=\infty$. □

Theorem 9

If $J\geq1$, then every solution $\{ x_{n} \}$ of (1) satisfies $\lim_{n\rightarrow \infty}x_{n}=\infty$.

Proof

Let $J\geq1$ and set $A+B+C+D+E+F+G+H+I=\alpha>0$. If $\{ x_{n} \}$ is a solution of (1), then $x_{n}>J$ for $n\geq1$, and by using the fact that the function

$$ f ( x,y ) =Ax^{3}+Bx^{2}y+Cxy^{2}+Dy^{3}+Ex^{2}+Fxy+Gy^{2}+Hx+Iy+J $$

is increasing in both variables we obtain the following result:

$$\begin{aligned} f ( J,J ) =& ( A+B+C+D ) J^{3}+ ( E+F+G ) J^{2}+ ( H+I ) J+J\\ \geq& ( A+B+C+D ) J+ ( E+F+G ) J+ ( H+I ) J+J\\ =& ( A+B+C+D+E+F+G+H+I+1 ) J= ( 1+\alpha ) J \end{aligned}$$

and

$$\begin{aligned}& x_{3} =f ( x_{2},x_{1} ) >f ( J,J ) \geq ( 1+ \alpha ) J,\\& x_{4} =f ( x_{3},x_{2} ) >f ( J,J ) \geq ( 1+ \alpha ) J,\\& x_{5} =f ( x_{4},x_{3} ) >f \bigl( ( 1+\alpha ) J, ( 1+\alpha ) J \bigr) \geq ( 1+\alpha ) ^{2}J,\\& x_{6} =f ( x_{5},x_{4} ) >f \bigl( ( 1+\alpha ) J, ( 1+\alpha ) J \bigr) \geq ( 1+\alpha ) ^{2}J,\\& \vdots \end{aligned}$$

Continuing in this way we obtain

$$ x_{2n+1}> ( 1+\alpha ) ^{n}J\quad\mbox{and}\quad x_{2n+2}> ( 1+\alpha ) ^{n}J\quad\mbox{for all }n\geq0, $$

which implies $x_{2n+1}\rightarrow\infty$ and $x_{2n+2}\rightarrow \infty$. □

Theorem 10

If $A+B+C+D+E+F+G+H+I > 1$ or $J>0$ and $A+B+C+D+E+F+G+H+I\geq1$, then the box $[ 1,\infty ) ^{2}$ is a part of the basin of attraction of the point at infinity of (1).

Proof

Set

$$ A+B+C+D+E+F+G+H+I=k\geq1 $$

and

$$ f ( x,y ) =Ax^{3}+Bx^{2}y+Cxy^{2}+Dy^{3}+Ex^{2}+Fxy+Gy^{2}+Hx+Iy+J. $$

(4)

Clearly, the function $f ( x,y )$ is increasing in both variables. If $x,y\in [ 1,\infty )$, then

$$\begin{aligned}[b] f ( x,x ) &= ( A+B+C+D ) x^{3}+ ( E+F+G ) x^{2}+ ( H+I ) x+J\\ &\geq ( A+B+C+D ) x+ ( E+F+G ) x+ ( H+I ) x+J\\ &=kx+J. \end{aligned} $$

Since the initial conditions $x_{-1},x_{0}\in [ 1,\infty )$,

$$\begin{aligned}& x_{1} =f ( x_{0},x_{-1} ) \geq f ( 1,1 ) =k+J, \\& x_{2} =f ( x_{1},x_{0} ) \geq f ( 1,1 ) =k+J, \\& x_{3} =f ( x_{2},x_{1} ) \geq f ( k+J,k+J ) \geq k ( k+J ) +J=k^{2}+kJ+J, \\& x_{4} =f ( x_{3},x_{2} ) \geq f ( k+J,k+J ) >k^{2}+kJ+J, \\& x_{5} =f ( x_{4},x_{3} ) \geq f \bigl( k^{2}+kJ+J,k^{2}+kJ+J \bigr) \geq k \bigl( k^{2}+kJ+J \bigr) +J=k^{3}+k^{2}J+kJ+J, \\& x_{6} =f ( x_{5},x_{4} ) \geq f \bigl( k^{2}+kJ+J,k^{2}+kJ+J \bigr) \geq k \bigl( k^{2}+kJ+J \bigr) +J=k^{3}+k^{2}J+kJ+J. \end{aligned}$$

An immediate application of mathematical induction yields

$$\begin{aligned}& x_{2n-1} \geq k^{n}+ \bigl( k^{n-1}+k^{n-2}+ \cdots+1 \bigr) J , \\& x_{2n} >k^{n}+ \bigl( k^{n-1}+k^{n-2}+ \cdots+1 \bigr) J \end{aligned}$$

for all $n\geq1$. If $k>1$, then $k^{n}\rightarrow\infty$ and if $k=1$ then $k^{n-1}+k^{n-2}+\cdots+1\rightarrow\infty$. In both cases we get $x_{2n-1}\rightarrow\infty$ and $x_{2n}\rightarrow\infty$. □

Theorem 11

If $( H+I-1 ) ^{2}<4J ( E+F+G )$, then every solution $\{ x_{n} \}$ of (1) satisfies $\lim_{n\rightarrow\infty}x_{n}=\infty$.

Proof

If $\{ x_{n} \}$ is a solution of (1), then $\{ x_{n} \}$ satisfies the inequality

$$ x_{n+1}\geq Ex_{n}^{2}+Fx_{n}x_{n-1}+Gx_{n-1}^{2}+Hx_{n}+Ix_{n-1}+J, \quad n=0,1,\ldots $$

which in view of the result on difference inequalities, see Theorem 1.4.1 in [17], implies that $x_{n}\geq y_{n}$, $n\geq1$ where $\{ y_{n} \}$ is a solution of the initial value problem

$$ \begin{aligned} &y_{n+1}=Ey_{n}^{2}+Fy_{n}y_{n-1}+Gy_{n-1}^{2}+Hy_{n}+Iy_{n-1}+J,\\ &y_{-1}=x_{-1}\quad\mbox{and}\quad y_{0}=x_{0}, \quad n=0,1,\ldots. \end{aligned} $$

(5)

Equation (5) has no equilibrium solutions if and only if

$$ ( H+I-1 ) ^{2}< 4J ( E+F+G ) . $$

In view of the results for (5) in [1], see Lemma 2, there is no prime period-two solution when $( H+I-1 ) ^{2}\leq4J ( E+F+G )$ or $E\geq G$ or $H+I>1$. So if there is no equilibrium point and there is no prime period-two solution of (5), then every solution $\{ y_{n} \}$ of (5) satisfies $\lim_{n\rightarrow\infty}y_{n}=\infty$, which implies $\lim_{n\rightarrow\infty}x_{n}=\infty$. □

Remark 2

Theorems 8, 9, 10, and 11 describe the parametric region which is the part of the basin of attraction of the point at ∞ called the escape region. The remaining part of the escape region will be described more precisely in the next sections. We will precisely describe the boundary of the escape region in all situations when the equilibrium points and period-two points are hyperbolic. In particular, we will describe the manifold that solutions will be following on their way to ∞. In general, it is clear from the proof of Theorem 11 that the escape region of (5) is a subset of the escape region of (1). In the subsequent sections we will consider global dynamics of (5) in the complement of the parametric region described by Theorems 8, 9, 10, and 11.

2 Local stability analysis of equilibrium solutions

The equilibrium solutions of (1) are the nonnegative solutions of the equation

$$ ( A+B+C+D ) \overline{x}^{3}+ ( E+F+G ) \overline{x} ^{2}+ ( H+I-1 ) \overline{x}+J=0. $$

(6)

Define the function $g ( x )$ such that

$$ g ( x ) = ( A+B+C+D ) x^{3}+ ( E+F+G ) x^{2}+ ( H+I-1 ) x+J, $$

(7)

so the nonnegative solutions of $g ( x ) =0$ are the nonnegative equilibrium solutions of (1).

2.1 The case $J>0$

Since $g ( -\infty ) =-\infty$ and $g ( 0 ) =J>0$, we always have one negative root, which means that there are at most two nonnegonnegative equilibrium solutions. The first derivative of $g ( x )$ is

$$ g^{\prime} ( x ) =3 ( A+B+C+D ) x^{2}+2 ( E+F+G ) x+ ( H+I-1 ). $$

(8)

Denote by Δ the discriminant of quadratic equation $g^{\prime} ( x ) =0$, that is,

$$ \Delta=4 ( E+F+G ) ^{2}-12 ( A+B+C+D ) ( H+I-1 ) . $$

(9)

We have:

(i) If $\Delta<0$, then $g^{\prime} ( x ) >0$ for all x and the function $g ( x )$ is monotonically increasing, which implies that there is no nonnegative root of $g ( x ) =0$.

(ii) If $\Delta=0$, then

$$ H+I-1=\frac{ ( E+F+G ) ^{2}}{3 ( A+B+C+D ) }\geq0. $$

Thus, $g ( x ) >0$ for all $x\geq0$, which implies that (6) has no positive solutions.

(iii) If $\Delta>0$, then for the roots $x_{1,2}=\frac{ -2 ( E+F+G ) \pm\sqrt{\Delta}}{6 ( A+B+C+D ) }$ ($x_{1}< x_{2}$) of $g^{\prime} ( x ) =0$ we get the following:

$$ x_{1}+x_{2}=-\frac{2}{3}\frac{E+F+G}{A+B+C+D}\leq0, $$

which means that at least $x_{1}$ is negative, and

$$ x_{1}x_{2}=\frac{1}{3} \frac{H+I-1}{A+B+C+D}. $$

(10)

From (10) we see that the following statements hold:

• If $H+I>1$, then (6) has no positive solutions.

• If $H+I=1$, then $x_{2}=0$ and $g(x)\geq g ( x_{2} ) =J>0$ so there is no positive solution of $g ( x ) =0$.

• If $H+I<1$, then $x_{2}>0$, $g^{\prime} ( x ) <0$ for $x\in ( x_{1},x_{2} )$ so $g ( x_{1} ) >g ( 0 ) =J>g ( x_{2} )$ and:

(a) If $g ( x_{2} ) \in ( 0,J )$ there is no positive solution of $g ( x ) =0$.

(b) If $g ( x_{2} ) =0$ there is one positive solution $\overline{x} =x_{2}$ of $g ( x ) =0$. This case is possible, for example when the values of parameters are

$$ A+B+C+D=1,\qquad E+F+G=\frac{1}{3},\qquad H+I=\frac{4}{9},\qquad J=\frac{1}{ 9}. $$

(c) If $g ( x_{2} ) <0$ there are two positive solutions $\overline{x} _{1}\in ( 0,x_{2} )$ and $\overline{x}_{2}\in ( x_{2},+\infty )$ of $g ( x ) =0$. The calculation of $g ( x_{2} )$ gives

$$ g ( x ) =\frac{x}{3}g^{\prime} ( x ) +\frac {x}{3} \bigl[ ( E+F+G ) x+2 ( H+I-1 ) \bigr] +J $$

and

$$\begin{aligned} g ( x_{2} ) =&\frac{x_{2}}{3}g^{\prime} ( x_{2} ) +\frac{ x_{2}}{3} \bigl[ ( E+F+G ) x_{2}+2 ( H+I-1 ) \bigr] +J \\ =&\frac{1}{3} \bigl[ ( E+F+G ) x_{2}^{2}+ ( H+I-1 ) x_{2} \bigr] +J. \end{aligned}$$

After a straightforward calculation we get

$$ g ( x_{2} ) =\frac{S_{2} ( 2S_{2}^{2}-9S_{1}S_{3} ) -2 \sqrt{ ( S_{2}^{2}-3S_{1}S_{3} ) ^{3}}}{27S_{1}^{2}}+J, $$

where

$$ S_{1}= A+B+C+D>0, \qquad S_{2} = E+F+G\geq0,\qquad S_{3} = H+I-1< 0. $$

Also, a straightforward calculation yields

$$ S_{2} \bigl( 2S_{2}^{2}-9S_{1}S_{3} \bigr) -2\sqrt{ \bigl( S_{2}^{2}-3S_{1}S_{3} \bigr) ^{3}}< 0. $$

2.2 The case $J=0$

In this case (6) becomes

$$ \overline{x} \bigl[ ( A+B+C+D ) \overline{x}^{2}+ ( E+F+G ) \overline{x}+ ( H+I-1 ) \bigr] =0, $$

(11)

so we always have the zero equilibrium. Denote by $\Delta_{0}$ discriminant of quadratic equation

$$ ( A+B+C+D ) \overline{x}^{2}+ ( E+F+G ) \overline{x} + ( H+I-1 ) =0, $$

(12)

that is

$$ \Delta_{0}= ( E+F+G ) ^{2}-4 ( A+B+C+D ) ( H+I-1 ) . $$

(13)

The following statements are immediate:

(i) If $\Delta_{0}<0$, then the zero equilibrium is the only equilibrium.

(ii) If $\Delta_{0}=0$, then a solution of the quadratic equation (12) is given by

$$ \overline{x}=-\frac{E+F+G}{2 ( A+B+C+D ) }. $$

If $E+F+G>0$ there is no positive solution. $E+F+G=0$ implies $H+I=1$, and again the zero equilibrium is the only equilibrium.

(iii) If $\Delta_{0}>0$, then for equilibriums $\overline{x}_{1,2}=\frac{- ( E+F+G ) \pm\sqrt{\Delta_{0}}}{ 2 ( A+B+C+D ) }$ ($\overline{x}_{1}<\overline {x}_{2}$) we have $\overline{x}_{1}<0$ and

$$ \overline{x}_{1}\overline{x}_{2}=\frac{H+I-1}{A+B+C+D}. $$

Now, we see that the following statements hold:

• (a) If $H+I>1$, then (12) has no positive solutions.

• (b) If $H+I=1$, then $\overline{x}_{2}=0$, and there is no positive equilibrium solutions.

• (c) If $H+I<1$, then the only positive equilibrium solution is $\overline{x}_{2}$.

The equilibrium solutions $\bar{x}$ of (1) satisfy the cubic equation (6). This immediately shows that there exists a zero equilibrium $\bar{x}=0$ if and only if $J=0$. The linearized equation at the zero equilibrium is

$$ z_{n+1}=Hz_{n}+Iz_{n-1}. $$

(14)

In view of Theorem 5 and the local stability theorem from [7] we obtain the following result on the local stability of the zero equilibrium.

Proposition 1

The zero equilibrium of (1) is one of the following:

• (a) locally asymptotically stable if $H+I<1$,

• (b) nonhyperbolic and locally stable if $H+I=1$,

• (c) unstable if $H+I>1$,

• (d) a saddle point if $H>|I-1|$,

• (e) a repeller if $1-I< H< I-1$.

A necessary condition for the existence of the positive equilibrium(s) is $H+I<1$. The linearized equation at the positive equilibrium solutions $\overline{x}$ is

$$ z_{n+1}=pz_{n}+qz_{n-1}, $$

(15)

where

$$\begin{aligned}& p = ( 3A+2B+C ) \overline{x}^{2}+ ( 2E+F ) \overline{x}+H,\\& q = ( B+2C+3D ) \overline{x}^{2}+ ( F+2G ) \overline{x}+I, \end{aligned}$$

and

$$ p+q=3 ( A+B+C+D ) \overline{x}^{2}+2 ( E+F+G ) \overline{x} + ( H+I ) . $$

Now,

$$ p+q=g^{\prime} ( \overline{x} ) +1, $$

where $g ( x )$ is given by (7). Let $x_{1}$ and $x_{2}$ ($x_{1}< x_{2}$) be the roots of $g^{\prime} ( x ) =0$. By applying Theorem 5 and the local stability theorem from [7] we obtain the following result on the local stability of the positive equilibrium(s).

Proposition 2

The positive equilibrium solution of (1) is one of the following:

• (a) locally asymptotically stable if $p+q<1$,

• (b) nonhyperbolic and locally stable if $p+q=1$,

• (c) unstable if $p+q>1$,

• (d) a saddle point if $p>|q-1|$,

• (e) a repeller if $1-q< p< q-1$.

The next theorems will describe the local stability for the positive equilibrium(s) in more detail.

Theorem 12

Let $J\in ( 0,\infty )$ and $H+I<1$, then:

• (a) If $g ( x_{2} ) <0$, then there are two positive equilibrium solutions $\overline{x}_{1}\in ( 0,x_{2} )$ and $\overline{x}_{2}\in ( x_{2},+\infty )$ of (1). Furthermore, $\overline{x}_{1}$ is locally asymptotically stable and $\overline{x}_{2}$ is unstable.

• (b) If $g ( x_{2} ) =0$, then there is one positive equilibrium solution $\overline{x}=x_{2}$, which is nonhyperbolic and locally stable.

Proof

(a) When $x\in ( 0,x_{2} )$, then $g^{\prime} ( x ) <0$, and for $\overline{x}_{1}\in ( 0,x_{2} )$ we get

$$ p+q=g^{\prime} ( \overline{x}_{1} ) +1< 1, $$

so by applying Theorem 5 we see that $\overline{x}_{1}$ is locally asymptotically stable. When $x\in ( x_{2},+\infty )$ then $g^{\prime} ( x ) >0$, and for $\overline{x}_{2}\in ( x_{2},+\infty )$ we get

$$ p+q=g^{\prime} ( \overline{x}_{2} ) +1>1, $$

so by applying Theorem 5 we see that $\overline{x}_{2}$ is unstable. By using the Proposition 2 we obtain the following result:

• $\overline{x}_{2}$ is a saddle point if $q-p<1$,

• $\overline{x}_{2}$ is a repeller if $q-p>1$.

It remains to determine $q-p$ for equilibrium $\overline{x}_{2}$:

$$ q-p= \bigl[ 3 ( D-A ) + ( C-B ) \bigr] \overline{x} _{2}^{2}+2 ( G-E ) \overline{x}_{2}+I-H. $$

(b) In this case we have

$$ p+q=g^{\prime} ( \overline{x} ) +1=g^{\prime} ( x_{2} ) +1=1, $$

and Theorem 5 implies that $\overline{x_{2}}$ is nonhyperbolic and locally stable. □

Theorem 13

If $J=0$ and $H+I<1$ and $\Delta_{0}$ is given by (13), then the only positive equilibrium

$$ \overline{x}_{+}=\frac{- ( E+F+G ) +\sqrt{\Delta_{0}}}{2 ( A+B+C+D ) } $$

of (1) is unstable.

Proof

In this case (1) has three equilibrium solutions $\overline{x}_{-}\in ( -\infty,x_{1} )$, $\overline{x}=0$, and $\overline{x}_{+}\in ( x_{2},\infty )$, where $x_{1}$ and $x_{2}$ ($x_{1}< x_{2}$) are the roots of $g^{\prime} ( x ) =0$. Now we have

$$\begin{aligned} p+q =&3 ( A+B+C+D ) \overline{x}_{+}^{2}+2 ( E+F+G ) \overline{x}_{+}+ ( H+I ) \\ =&g^{\prime} ( \overline{x}_{+} ) +1>1, \end{aligned}$$

so by applying Theorem 5 we see that $\overline{x}_{+}$ is unstable. By using Proposition 2 we obtain the following result:

• $\overline{x}_{+}$ is saddle point if $q-p<1$;

• $\overline{x}_{+}$ is repeller if $q-p>1$.

It remains to determine $q-p$ for equilibrium $\overline{x}_{+}$:

$$ q-p= \bigl[ 3 ( D-A ) + ( C-B ) \bigr] \overline{x} _{+}^{2}+2 ( G-E ) \overline{x}_{+}+I-H. $$

 □

Example 1

It has been shown in [12] that the difference equation

$$ x_{n+1}=x_{n}^{3}+x_{n-1}^{3} $$

has one positive equilibrium $\overline{x}_{+}=\frac{\sqrt{2}}{2}$, which is a saddle point. The difference equation

$$ x_{n+1}=Ax_{n}^{3}+Bx_{n}^{2}x_{n-1}+Cx_{n}x_{n-1}^{2}+Dx_{n-1}^{3}+ \frac{1}{ 8}x_{n}+\frac{5}{8}x_{n-1} $$

such that $D>5A+3B+C$ has one positive equilibrium $\overline {x}_{+}=\frac{1}{ 2 ( A+B+C+D ) }$. As

$$ q-p=\frac{3 ( D-A ) + ( C-B ) }{4 ( A+B+C+D ) }+ \frac{1}{2}=\frac{D-5A-3B-C}{4 ( A+B+C+D ) }+1>1, $$

the positive equilibrium is a repeller.

3 Local stability of period-two solutions

Period-two solutions $\ldots,\Phi,\Psi,\Phi,\Psi,\ldots$ satisfy the system:

$$\begin{aligned}& \Phi=A\Psi^{3}+B\Phi\Psi^{2}+C\Phi^{2}\Psi+D \Phi^{3}+E\Psi ^{2}+F\Phi \Psi+G\Phi^{2}+H\Psi+I \Phi+J, \end{aligned}$$

(16)

$$\begin{aligned}& \Psi=A\Phi^{3}+B\Phi^{2}\Psi+C\Phi\Psi^{2}+D \Psi^{3}+E\Phi ^{2}+F\Phi \Psi+G\Psi^{2}+H\Phi+I \Psi+J. \end{aligned}$$

(17)

It is clear that if there is a period-two solution it has to be $I\leq1$. Also, for $H+I>1$ or $J\geq1$ or $( H+I-1 ) ^{2}<4J ( E+F+G )$ we have seen that $x_{n}\rightarrow\infty$ (see Theorems 8, 9, 10, and 11) so interesting dynamics for our system happens in the parametric region $H+I\leq1$, $J<1$ and $( H+I-1 ) ^{2}\geq4J ( E+F+G )$, which will be our main objective in the sequel. As the system (16), (17) is symmetric, which means that if $( \Phi ,\Psi )$ is a solution of the system (16), (17), then also $( \Psi,\Phi )$ is a solution of the system (16), (17); then we assume without a loss of generality that $0\leq\Phi<\Psi$. By subtracting (16) and (17), and using the fact that $\Psi-\Phi>0$, we get

$$ ( D-A ) ( \Phi+\Psi ) ^{2}+ ( A-B+C-D ) \Phi \Psi+ ( G-E ) ( \Phi+ \Psi ) +I-H-1=0. $$

(18)

By summing (16) and (17) we get

$$\begin{aligned} &( A+D ) ( { \Phi+\Psi} ) ^{{ 3}} { +} ( { B+C-3A-3D} ) { \Phi\Psi} ( { \Phi+\Psi} ) { +} ( { E+G} ) ( { \Phi +\Psi} ) ^{{ 2}} \\ &\quad{} +2 ( { F-E-G} ) { \Phi\Psi+} ( { H+I-1} ) ( { \Phi+\Psi } ) +2J=0. \end{aligned}$$

(19)

If we set $\Phi+\Psi=x>0$ and $\Phi\Psi=y\geq0$, then (18) and (19) yields another system

$$\begin{aligned}& ( D-A ) x^{2}+ ( G-E ) x+ ( A-B+C-D ) y+I-H=1, \end{aligned}$$

(20)

$$\begin{aligned}& \begin{aligned}[b] &( A+D ) x^{3}+ ( E+G ) x^{2}+ ( B+C-3A-3D ) xy+ ( H+I-1 ) x\\ &\quad{}+2 ( F-E-G ) y+2J=0. \end{aligned} \end{aligned}$$

(21)

One can see that the special cases when the period-two solutions do not exist are

$$ H+I>1 $$

or

$$ D\leq A\quad\mbox{and}\quad A+C\leq B+D \ (\mbox{or }C+D\leq A+B ) \quad\mbox{and}\quad G \leq E. $$

This immediately leads to the following result.

Theorem 14

Let $D\leq A$, $A+C\leq B+D$ (or $C+D\leq A+B$), $G\leq E$, $J\in ( 0,1 )$, and $H+I<1$. If $\Delta\leq0$ or $\Delta>0$ and $g ( x_{2} ) \in ( 0,J )$, then every solution $\{ x_{n} \}$ of (1) satisfies $\lim_{n\rightarrow\infty}x_{n}=\infty$. Here $g ( x )$ and Δ are defined by (7) and (9), respectively, and $x_{2}$ is the greater root of equation $g^{\prime} ( x ) =0$.

Proof

Since $H+I<1$, $D\leq A$, $A+C\leq B+D$ (or $C+D\leq A+B$), and $G\leq E$, we see that (18) implies that the period-two solutions do not exist. If $\Delta\leq0$ or $\Delta>0$ and $g ( x_{2} ) \in ( 0,J )$, then there is no positive solution of $g ( x ) =0$, so there are no positive equilibriums of (1). As a consequence of Theorem 2 every bounded solution of (1) approaches either an equilibrium solution or a period-two solution and every unbounded solution is asymptotic to the point at infinity in a monotonic way. Hence, every solution $\{ x_{n} \}$ of (1) satisfies $\lim_{n\rightarrow\infty} x_{n}=\infty$. □

Remark 3

If $D\leq A$, $A+C\leq B+D$ ($\Rightarrow C\leq B$), $G\leq E$, then a period-two solution does not exist, so if there exists an unstable positive equilibrium of (1) we obtain

$$ q-p-1= \bigl[ 3 ( D-A ) + ( C-B ) \bigr] \overline{x} _{+}^{2}+2 ( G-E ) \overline{x}_{+}+I-H-1< 0, $$

which implies that the positive equilibrium is a saddle point. In this case a global result follows from Theorem 3.

Set

$$ u_{n}=x_{n-1}\quad\mbox{and}\quad v_{n}=x_{n},\quad n=0,1,\ldots $$

and write (1) in the equivalent form:

$$\begin{aligned}& u_{n+1} =v_{n} ,\\& v_{n+1} =Av_{n}^{3}+Bv_{n}^{2}u_{n}+Cv_{n}u_{n}^{2} +Du_{n}^{3}+Ev_{n}^{2}+Fv_{n}u_{n}+Gu_{n}^{2}+Hv_{n}+Iu_{n}+J. \end{aligned}$$

Let the function $f ( u,v )$ be defined by (4) and let T be a map on $[ 0,\infty ) \times [ 0,\infty )$ defined by $T ( u,v ) = ( v,f ( v,u ) )$, that is,

$$ T ( u,v ) = \bigl( v,Av^{3}+Buv^{2}+Cu^{2}v+Du^{3}+Ev^{2}+Fuv+Gu^{2}+Hv+Iu+J \bigr) . $$

(22)

Then $( \Phi,\Psi )$ is a fixed point of $T^{2}$, the second iterate of T. It is clear that if $( \Phi,\Psi )$ is a fixed point of $T^{2}$, then $( \Psi,\Phi )$ is also a fixed point of $T^{2}$. Furthermore,

$$ T^{2} ( u,v ) = \bigl( g ( u,v ) ,h ( u,v ) \bigr), $$

(23)

where

$$\begin{aligned}& g ( u,v ) =f ( v,u ) ,\\& h ( u,v ) =f \bigl( f ( v,u ) ,v \bigr) =f \bigl( g ( u,v ) ,v \bigr) . \end{aligned}$$

The period-two solution is locally asymptotically stable if the eigenvalues of the Jacobian matrix $J_{T^{2}}$, evaluated at $( \Phi,\Psi )$, lie inside the unit disk. By definition

$$ J_{T^{2}} ( \Phi,\Psi ) = \begin{pmatrix} \frac{\partial g}{\partial u} ( \Phi,\Psi ) & \frac{\partial g}{\partial v} ( \Phi,\Psi )\\ \frac{\partial h}{\partial u} ( \Phi,\Psi ) & \frac{\partial h}{\partial v} ( \Phi,\Psi ) \end{pmatrix} . $$

By computing the partial derivatives of g and h and by using the fact that $g ( \Phi,\Psi ) =f ( \Psi,\Phi ) =\Phi$, we find the following identities:

$$\begin{aligned} &a=\frac{\partial g}{\partial u} ( \Phi,\Psi ) =\frac {\partial f}{\partial v} ( \Psi,\Phi ) =B\Psi^{2}+2C\Phi\Psi+3D\Phi ^{2}+F\Psi+2G\Phi+I, \\ &b=\frac{\partial g}{\partial v} ( \Phi,\Psi ) =\frac {\partial f}{\partial u} ( \Psi,\Phi ) =3A\Psi^{2}+2B\Phi\Psi+C\Phi ^{2}+2E\Psi+F\Phi+H, \end{aligned} $$

(24)

and

$$\begin{aligned}& \frac{\partial h}{\partial u} ( \Phi,\Psi ) =\frac {\partial f}{ \partial g}\frac{\partial g}{\partial u} ( \Phi, \Psi ) =\frac{ \partial f}{\partial u} \bigl( g ( \Phi,\Psi ) ,\Psi \bigr) \frac{\partial f}{\partial v} ( \Psi,\Phi ) =\frac {\partial f}{ \partial u} ( \Phi,\Psi ) \frac{\partial f}{\partial v} ( \Psi,\Phi ),\\& \begin{aligned}[b] \frac{\partial h}{\partial v} ( \Phi,\Psi ) &=\frac {\partial f}{ \partial g}\frac{\partial g}{\partial v} ( \Phi, \Psi ) +\frac{ \partial f}{\partial v} ( \Phi,\Psi ) =\frac{\partial f}{ \partial u} \bigl( g ( \Phi,\Psi ) ,\Psi \bigr) \frac {\partial f}{ \partial u} ( \Psi,\Phi ) +\frac{\partial f}{\partial v} ( \Phi,\Psi ) \\ &=\frac{\partial f}{\partial u} ( \Phi,\Psi ) \frac {\partial f}{ \partial u} ( \Psi,\Phi ) + \frac{\partial f}{\partial v} ( \Phi,\Psi ) , \end{aligned} \end{aligned}$$

where

$$\begin{aligned} &c=\frac{\partial f}{\partial u} ( \Phi,\Psi ) =3A\Phi ^{2}+2B\Phi\Psi+C\Psi^{2}+2E\Phi+F\Psi+H, \\ &d=\frac{\partial f}{\partial v} ( \Phi,\Psi ) =B\Phi ^{2}+2C\Phi \Psi+3D\Psi^{2}+F\Phi+2G\Psi+I. \end{aligned} $$

(25)

Set

$$\begin{aligned} \mathcal{S} =&\operatorname{tr}J_{T^{2}} ( \Phi,\Psi ) =\frac{\partial g}{ \partial u} ( \Phi,\Psi ) +\frac{\partial h}{\partial v} ( \Phi,\Psi )\\ =&a+bc+d\geq0 \end{aligned}$$

and

$$\begin{aligned} \mathcal{D} =&\det J_{T^{2}} ( \Phi,\Psi ) =\frac{\partial g}{ \partial u} ( \Phi, \Psi ) \frac{\partial h}{\partial v} ( \Phi,\Psi ) -\frac{\partial g}{\partial v} ( \Phi,\Psi ) \frac{\partial h}{\partial u} ( \Phi,\Psi )\\ =&a ( cb+d ) -b ( ca ) =ad\geq0. \end{aligned}$$

By applying the linearized stability theorem, the period-two solution $( \Phi,\Psi )$ is locally asymptotically stable if $\vert \mathcal{S}\vert <1+\mathcal{D}$ and $\mathcal{D}<1$, that is,

$$ \mathcal{S}< 1+\mathcal{D}\quad\Leftrightarrow\quad a+bc+d< 1+ad\quad\Leftrightarrow\quad bc< ( a-1 ) ( d-1 ) $$

(26)

and

$$ \mathcal{D}< 1\quad\Leftrightarrow\quad ad< 1, $$

(27)

which implies the following lemma.

Lemma 1

The eigenvalues of the Jacobian matrix of $J_{T^{2}}$ at a period-two solution are nonnegative numbers.

In order to solve system of equations (20) and (21), we consider two different cases.

(i) If $A+C=B+D$, then (20) becomes the quadratic equation

$$ ( D-A ) x^{2}+ ( G-E ) x+I-H-1=0. $$

(28)

If $D\leq A$ and $G\leq E$, then the period-two solutions do not exist, so at least one of the following inequalities is true:

$$ D>A\quad\mbox{or}\quad G>E. $$

Also, $H+I<1$ implies $I-H-1<-2H\leq0$. The solutions of the quadratic equation (28) are given by

$$ x_{1,2}=\frac{E-G\pm\sqrt{ ( E-G ) ^{2}-4 ( D-A ) ( I-H-1 ) }}{2 ( D-A ) } . $$

(29)

(a) If $D>A$, then $( D-A ) ( I-H-1 ) <0$ and

$$ x=\frac{E-G+\sqrt{ ( E-G ) ^{2}-4 ( D-A ) ( I-H-1 ) }}{2 ( D-A ) } $$

is the only positive solution of (28).

(b) If $D< A$, then (28) can be written as

$$ \underset{\mathrm{positive}}{\underbrace{ ( A-D ) }}x^{2}+ ( E-G ) x+ \underset{\mathrm{positive}}{\underbrace{H-I+1}}=0. $$

It is clear that $E< G$. As

$$ x_{1}x_{2}=\frac{G-E}{A-D}>0\quad\mbox{and}\quad x_{1}+x_{2}=\frac{H-I+1}{A-D}>0, $$

when $( E-G ) ^{2}>4 ( A-D ) ( H-I+1 )$ we have two positive solutions given by (29). If $( E-G ) ^{2}=4 ( A-D ) ( H-I+1 )$, then we have one positive solution

$$ x=\frac{G-E}{2 ( A-D ) }. $$

(c) If $D=A$, then $B=C$, and (28) becomes the linear equation

$$ ( G-E ) x+I-H-1=0. $$

If $G=E$, then $I=H+1\Rightarrow I\geq1$. Case $H=0$ and $I=1$ implies $J=0$, so (16) and (17) become the same equation,

$$ A \bigl( \Phi^{3}+\Psi^{3} \bigr) +B\Phi\Psi ( \Phi+\Psi ) +E \bigl( \Phi^{2}+\Psi^{2} \bigr) +F\Phi\Psi=0, $$

which is impossible for positive Φ, Ψ. For positive parameter H we have $H+I>2H+1>1$, so there is no minimal period-two solution. If $G\neq E$, then

$$ x=\frac{\overset{\mathrm{positive}}{\overbrace{H-I+1}}}{G-E}, $$

in which case we get $G>E$.

In all the above cases, when $2 ( E+G-F ) + ( 3A+3D-B-C ) x\neq0$, y is given by

$$ y=\frac{ ( A+D ) x^{3}+ ( E+G ) x^{2}+ ( H+I-1 ) x+2J}{2 ( E+G-F ) + ( 3A+3D-B-C ) x}. $$

(30)

(ii) If $A+C\neq B+D$, then (20) gives

$$ y=\frac{ ( D-A ) x^{2}+ ( G-E ) x+I-H-1}{B+D-A-C}, $$

and, finally, from (21) we get the following cubic equation:

$$ K_{1}x^{3}+K_{2}x^{2}+K_{3}x+K_{4}=0, $$

(31)

where

$$\begin{aligned}& K_{1} =A ( C-A ) +D ( D-B ), \\& K_{2} =CE-D ( E+F-2G ) -BG+A ( -2E+F+G ), \\& K_{3} =-D+ ( E-G ) ( E-F+G ) -C H+2DH+A ( 2+H-2I ) +B ( -1+I ) -D I, \\& K_{4} = ( F-E-G ) ( I-H-1 ) +J ( A-B+C-D ). \end{aligned}$$

Remark 4

Equation (31) shows that (1) can have at most three period-two solutions. Lemma 3 gives an upper bound of the number of period-two solutions of equation (1) in some special cases. Equation (31) can be solved but its solutions are very complicated and would depend on 10 parameters. In the remaining part of the paper we will work under the assumptions that (1) has between zero and three period-two solutions and we will present the global dynamics in all possible cases. In particular, Theorem 20 describes a global dynamics in the case when (1) has one or three period-two solutions, while Theorem 21 gives a global dynamics in the case when (1) has zero or two period-two solutions. The existence of at least one period-two solution is guaranteed by Theorem 19.

4 Global behavior

In this section we present the global dynamics of (1) in different parametric regions.

4.1 The case that there exists a minimal period-two solution on the coordinate axes ($\Phi=0$ and $\Psi>0$)

In this section we consider the special case when $\Phi=0$ and $\Psi >0$. Equation (16) implies

$$ A\Psi^{3}+E\Psi^{2}+H\Psi+J=0. $$

Since $\Psi>0$, we get $A=E=H=J=0$. Now, from (17) we obtain

$$ D\Psi^{2}+G\Psi+I-1=0. $$

(32)

If $I=1$, then we must have $D=G=0$ and $\Psi\in\mathbb{R}^{+}$. If $I<1$, then Ψ is a positive solution of (4), that is,

$$ \Psi=\frac{-G+\sqrt{G^{2}+4D ( 1-I ) }}{2D}. $$

Also, if $D=0$ and $I<1$, then

$$ \Psi=\frac{1-I}{G}. $$

4.1.1 The case: $A=D=E=G=H=J=0$, $I=1$

If $A=D=E=G=H=J=0$ and $I=1$, then $( \Phi ,\Psi ) = ( 0,t )$ where $t\in\mathbb{R}^{+}$ is a period-two solution of the difference equation

$$ x_{n+1}=Bx_{n}^{2}x_{n-1}+Cx_{n}x_{n-1}^{2}+Fx_{n}x_{n-1}+x_{n-1}. $$

(33)

All other minimal period-two solutions $( \Phi,\Psi )$ are solutions of the system

$$\begin{aligned}& \Phi= \bigl( B\Psi^{2}+C\Phi\Psi+F\Psi+1 \bigr) \Phi,\\& \Psi= \bigl( B\Phi^{2}+C\Phi\Psi+F\Phi+1 \bigr) \Psi. \end{aligned}$$

One can show that this system has no other positive solutions except $( \Phi ,\Psi ) = ( 0,t )$ where

$$ a = Bt^{2}+Ft+1, \qquad b = 0,\qquad c = Ct^{2}+Ft,\qquad d =1. $$

Since $b c= ( a-1 ) ( d-1 )$ or $\mathcal {S}=1+\mathcal{D}$, the period-two solution is nonhyperbolic. The global behavior of this equation is described in Section 8 and, in this case, every solution goes to infinity.

4.1.2 The case: $A=E=H=J=0$ and $I<1$

If $A=E=H=J=0$ and $I<1$, then $( \Phi,\Psi ) = ( 0,\frac{-G+\sqrt{G^{2}+4D ( 1-I ) }}{2D} )$ is the period-two solution of the difference equation

$$ x_{n+1}=Bx_{n}^{2}x_{n-1}+Cx_{n}x_{n-1}^{2}+Dx_{n-1}^{3}+ Fx_{n}x_{n-1}+Gx_{n-1}^{2}+Ix_{n-1}. $$

(34)

All other minimal period-two solutions $( \Phi,\Psi )$ are solutions of the system

$$\begin{aligned}& \Phi= \bigl( B\Psi^{2}+C\Phi\Psi+D\Phi^{2}+F\Psi+G\Phi+I \bigr) \Phi, \\& \Psi= \bigl( B\Phi^{2}+C\Phi\Psi+D\Psi^{2}+F\Phi+G\Psi+I \bigr) \Psi. \end{aligned}$$

Since $H+I<1$, there are locally stable zero equilibrium and positive unstable equilibrium. It is useful to note that if $B=D$ and $F=G$, then $( \Phi,\Psi ) = ( 0,\frac{-F+\sqrt{F^{2}+4B ( 1-I ) }}{2B} )$ is also a solution of the equation

$$ B \bigl( \Phi^{2}+\Psi^{2} \bigr) +C\Phi\Psi+F ( \Phi+\Psi ) +I-1=0. $$

In this case every point on the curve

$$ B \bigl( x^{2}+y^{2} \bigr) +Cxy+F ( x+y ) +I-1=0 $$

(35)

is a period-two solution of (34) except the equilibrium point $( \overline {x}_{+},\overline{x} _{+} )$.

One can see that for $( \Phi,\Psi ) = ( 0,\frac{-G+\sqrt {G^{2}+4D ( 1-I ) }}{2D} )$

$$ a =B\Psi^{2}+F\Psi+I,\qquad b =0, \qquad c= C\Psi^{2}+F \Psi, \qquad d = 3D\Psi^{2}+2G\Psi+I. $$

Now, we have

$$\begin{aligned}[b] ( a-1 ) ( d-1 ) &= \bigl( B\Psi^{2}+F\Psi +I-1 \bigr) \bigl( 3D \Psi^{2}+2G\Psi+I-1 \bigr) \\ &= \bigl( B\Psi^{2}+F\Psi+I-1 \bigr) \bigl( \underset{=0}{ \underbrace{ D\Psi^{2}+G\Psi+I-1}}+2D\Psi^{2}+G\Psi \bigr) \\ &=\Psi \bigl( B\Psi^{2}+F\Psi+I-1 \bigr) ( 2D\Psi+G ) \\ &=\Psi \bigl( B\Psi^{2}+F\Psi+I-1 \bigr) \sqrt{G^{2}+4D ( 1-I ) } \end{aligned} $$

and

$$\begin{aligned} ad =& \bigl( B\Psi^{2}+F\Psi+I \bigr) \bigl( 3D\Psi^{2}+2G \Psi +I \bigr) \\ =& \bigl( B\Psi^{2}+F\Psi+I \bigr) \bigl( \underset{=0}{\underbrace {D\Psi ^{2}+G\Psi+I-1}}+2D\Psi^{2}+G\Psi+1 \bigr) \\ =& \bigl( B\Psi^{2}+F\Psi+I \bigr) \bigl( \Psi ( 2D\Psi+G ) +1 \bigr) \\ =& \bigl( B\Psi^{2}+F\Psi+I \bigr) \bigl( \Psi\sqrt{G^{2}+4D ( 1-I ) }+1 \bigr) . \end{aligned}$$

Thus

$$\begin{aligned} \mathcal{S}-\mathcal{D}-1 =&- ( a-1 ) ( d-1 ) \\ =&-\Psi \bigl( B\Psi^{2}+F\Psi+I-1 \bigr) \sqrt{G^{2}+4D ( 1-I )} . \end{aligned}$$

After some calculation we find

$$ B\Psi^{2}+F\Psi+I-1=\frac{2D ( 1-I ) ( B-D ) + ( FD-BG ) ( -G+\sqrt{G^{2}+4D ( 1-I ) } ) }{2D^{2}}, $$

and for the positive equilibrium $\overline{x}_{+}$ we obtain

$$\begin{aligned} q-p-1 =& \bigl[ 3D+ ( C-B ) \bigr] \overline{x}_{+}^{2}+2G \overline{x}_{+}+I-1 \\ =&\underset{=0}{\underbrace{ ( B+C+D ) \overline {x}_{+}^{2}+ ( F+G ) \overline{x}_{+}+I-1}}+2 ( D-B ) \overline{x} _{+}^{2}+ ( G-F ) \overline{x}_{+} \\ =&-\overline{x}_{+} \bigl( 2 ( B-D ) \overline{x}_{+}+ ( F-G ) \bigr) . \end{aligned}$$

• It is clear that if $B\Psi^{2}+F\Psi+I>1$, then $( a-1 ) ( d-1 ) >0=bc$ and $ad>1$ or $\mathcal{S}<1+\mathcal{D}$ and $\mathcal{D}>1$, which implies $( \Phi,\Psi ) = ( 0,\frac{-G+ \sqrt{G^{2}+4D ( 1-I ) }}{2D} )$ is a repeller. For example, if $B=D$ and $F>G$ or $B>D$ and $F=G$, then $q-p-1<0$, which implies that the positive equilibrium $\overline{x}_{+}$ is a saddle point.

• In the case when $B\Psi^{2}+F\Psi+I=1$, then $( a-1 ) ( d-1 ) =0=bc$ or $\mathcal{S}=1+\mathcal{D}$, which implies that the period-two solution $( \Phi,\Psi ) = ( 0,\frac{-G+\sqrt{ G^{2}+4D ( 1-I ) }}{2D} )$ is a nonhyperbolic point. A special case when this holds is $B=D$ and $G=F$. In this case every point on the curve (35) is a period-two solution of (34), except the equilibrium point, Since $q-p-1=0$, the positive equilibrium $\overline{x}_{+}$ is a nonhyperbolic point. Also, the points $( \Phi,\Psi ) = \{ ( 0,\frac{-F+\sqrt{F^{2}+4B ( 1-I ) }}{2B} ) , ( \frac{-F+\sqrt{F^{2}+4B ( 1-I ) }}{ 2B},0 ) \}$ are the endpoints of the curve (35) in the first quadrant.

• If $B\Psi^{2}+F\Psi+I<1$, then $( a-1 ) ( d-1 ) <0=bc$ or $\mathcal{S}>1+\mathcal{D}$, which implies that the period-two solution $( \Phi ,\Psi ) = ( 0,\frac{-G+\sqrt{G^{2}+4D ( 1-I ) }}{2D} )$ is a saddle point. That is true for example if $B=D$ and $F< G$ or $B>D$ and $F=G$, then $q-p-1>0$, which implies that the positive equilibrium $\overline{x}_{+}$ is a repeller.

One can prove that if $B=D$, $F\neq G$ or $F=G$, $B\neq D$ the map T has no other minimal period-two solutions except $( \Phi,\Psi ) = ( 0,\frac{-G+\sqrt{G^{2}+4D ( 1-I ) }}{2D} )$. This leads to the following result.

Theorem 15

Let $A=E=H=J=0$ and $I<1$, $D>0$, then (34) has locally stable zero equilibrium and the positive equilibrium $\overline{x}_{+}$. The following statements hold:

(a) If $B=D$, $F< G$ or $B>D$, $F=G$, then the positive equilibrium $\overline{x}_{+}$ is a repeller and there exists the period-two solution which is a saddle point. In this case there exist four continuous curves $\mathcal{W}^{s}(P_{1})$, $\mathcal{W}^{s}(P_{2})$, $\mathcal{W}^{u}(P_{1})$, and $\mathcal{W}^{u}(P_{2})$, with $P_{1}=(\Phi ,\Psi)$ and $P_{2}=(\Psi,\Phi)$, where $\mathcal{W}^{s}(P_{1})$ and $\mathcal{W} ^{s}(P_{2})$ are passing through the point $E_{+}=(\bar{x}_{+},\bar{x}_{+})$ and they are graphs of decreasing functions. The curves $\mathcal{W} ^{u}(P_{1})$ and $\mathcal{W}^{u}(P_{2})$ are the graphs of increasing functions which are starting at $E_{0} ( 0,0 )$. Every solution $\{ x_{n} \}$ which starts below $\mathcal{W}^{s}(P_{1})\cup \mathcal{W}^{s}(P_{2})$ in the North-East ordering converges to $E_{0}$ and every solution $\{ x_{n} \}$ which starts above $\mathcal{W} ^{s}(P_{1})\cup\mathcal{W}^{s}(P_{2})$ in the North-East ordering satisfies $\lim_{n\rightarrow\infty} x_{n}=\infty$. (See Figure  1(a).)

Figure 1

Visual illustration of Theorems 15 and 16 .

(b) If $B=D$, $F>G$ or $B>D$, $F=G$, then the positive equilibrium $\overline{x}_{+}$ is a saddle point and there exists the period-two solution which is a repeller. Global behavior of (34) is described by Theorem  3 where we set $E_{1}(0,0)$ and $E_{2}(\overline{x}_{+},\overline {x}_{+})$. (See Figure  1(b).)

(c) If $B=D$ and $F=G$, then every point on the curve (35), which is passing through the equilibrium $E=(\bar{x}_{+},\bar{x}_{+})$, is a period-two solution of (34) except point E. Period-two solutions and the positive equilibrium $\overline{x}_{+}$ are nonhyperbolic points, while the zero equilibrium is locally stable. The curve (35) divides the first quadrant into two regions. The region below (35) in the first quadrant is a basin of attraction of the zero equilibrium and the region above the (35) in the first quadrant is a basin of attraction of point at infinity. (See Figure  1(c).)

Proof

(a) The existence of four curves $\mathcal{W} ^{s}(P_{1})$, $\mathcal{W}^{s}(P_{2})$, $\mathcal{W}^{u}(P_{1})$, and $\mathcal{W}^{u}(P_{2})$ with the described properties is guaranteed by Theorems 1 and 4 of [18] applied to the map $T^{2}$ given by (23). The global result follows from Theorem 3.

(b) All conditions of Theorem 3 are satisfied, which implies the proof.

(c) It was already noticed that every point on the curve (35) is a nonhyperbolic period-two solution of (34) and points

$$ \biggl\{ \biggl( 0,\frac{-F+\sqrt{F^{2}+4B ( 1-I ) }}{2B} \biggr) , \biggl( \frac{-F+\sqrt{F^{2}+4B ( 1-I ) }}{ 2B},0 \biggr) \biggr\} $$

are the endpoints of (35) in the first quadrant. It is also known that the positive equilibrium $\overline{x}_{+}$ is a nonhyperbolic point. This implies that the curve (35) which is passing through the point $E=(\bar {x}_{+},\bar{x}_{+})$ is not a global stable manifold. Since the curve (35) is the boundary of the basin of attraction of the points at infinity, it is a Julia set. Let $\mathcal{V}$ denote the matrix

$$ \mathcal{V}= \begin{pmatrix} B & \frac{C}{2} \\ \frac{C}{2} & B \end{pmatrix} . $$

Then $4\det\mathcal{V}=4B^{2}-C^{2}$ and by the well-known result as regards the identification of the quadratic curves, we obtain the following statements:

• if $\det\mathcal{V}>0\Leftrightarrow2B>C$, then $\mathcal{U}$ is either an ellipse or a circle if $C=0$,

• if $\det\mathcal{V}<0\Leftrightarrow2B<C$, then $\mathcal{U}$ is a hyperbole,

• if $\det\mathcal{V}=0 \Leftrightarrow2B=C$, then $\mathcal {U}$ is union of two parallel lines.

We avoid the cases when $\mathcal{U}$ is one point set or empty set. As a consequence of $I<1$ and the fact that the curve (35) has endpoints in the first quadrant we conclude that the point $( 0,0 )$ is always below the curve (35). It remains to describe global behavior of (34) when $B=D$ and $F=G$. If an initial point $( x_{-1},x_{0} )$ is below the curve (35), then

$$ B \bigl( x_{0}^{2}+x_{-1}^{2} \bigr) +Cx_{0}x_{-1}+F ( x_{0}+x_{-1} ) +I< 1. $$

As $x_{n+1}= ( B ( x_{n}^{2}+x_{n-1}^{2} ) +Cx_{n}x_{n-1}+F ( x_{n}+x_{n-1} ) +I ) x_{n-1}$, then

$$\begin{aligned}& x_{1} = \bigl( B \bigl( x_{0}^{2}+x_{-1}^{2} \bigr) +Cx_{0}x_{-1}+F ( x_{0}+x_{-1} ) +I \bigr) x_{-1}< x_{-1}, \\& \begin{aligned}[b] x_{2} &= \bigl( B \bigl( x_{1}^{2}+x_{0}^{2} \bigr) +Cx_{1}x_{0}+F ( x_{1}+x_{0} ) +I \bigr) x_{0} \\ &< \bigl( B \bigl( x_{-1}^{2}+x_{0}^{2} \bigr) +Cx_{-1}x_{0}+F ( x_{-1}+x_{0} ) +I \bigr) x_{0}< x_{0}. \end{aligned} \end{aligned}$$

Thus the point $( x_{2},x_{1} )$ is below the curve (35) and

$$ B \bigl( x_{2}^{2}+x_{1}^{2} \bigr) +Cx_{2}x_{1}+F ( x_{2}+x_{1} ) +I< 1. $$

Now

$$\begin{aligned}& x_{3} = \bigl( B \bigl( x_{2}^{2}+x_{1}^{2} \bigr) +Cx_{2}x_{1}+F ( x_{2}+x_{1} ) +I \bigr) x_{1}< x_{1}, \\& \begin{aligned}[b] x_{4} &= \bigl( B \bigl( x_{3}^{2}+x_{2}^{2} \bigr) +Cx_{3}x_{2}+F ( x_{3}+x_{2} ) +I \bigr) x_{2} \\ &< \bigl( B \bigl( x_{1}^{2}+x_{2}^{2} \bigr) +Cx_{1}x_{2}+F ( x_{1}+x_{2} ) +I \bigr) x_{2}< x_{2}. \end{aligned} \end{aligned}$$

Continuing in this way, we obtain $( 0,0 ) \preceq _{ne}\cdots\preceq _{ne} ( x_{3},x_{4} ) \preceq_{ne} ( x_{1},x_{2} ) \preceq_{ne} ( x_{-1},x_{0} )$. Hence, both of the subsequences $\{ x_{2n} \}$ and $\{x_{2n+1}\}$ are decreasing, which implies $x_{2n}\rightarrow0$ and $x_{2n+1}\rightarrow0$. If we start at the point $( x_{-1},x_{0} )$ above the curve (35), then

$$ B \bigl( x_{0}^{2}+x_{-1}^{2} \bigr) +Cx_{0}x_{-1}+F ( x_{0}+x_{-1} ) +I>1. $$

Now the proof is very similar and will be omitted. In this case both of the subsequences $\{ x_{2n} \}$ and $\{x_{2n+1}\}$ are increasing, which implies $x_{2n}\rightarrow\infty$ and $x_{2n+1}\rightarrow\infty$.

If $2B=C$, then

$$\begin{aligned}[b] &B \bigl( x^{2}+y^{2} \bigr) +2Bxy+F ( x+y ) +I =1 ,\\ &B ( x+y ) ^{2}+F ( x+y ) - ( 1-I ) =0. \end{aligned} $$

Since $F^{2}+4B ( 1-I ) >0$, $\mathcal{U}$ is the union of two parallel lines. If we want to obtain two intersecting lines it has to be $2B< C$ and a straightforward calculation gives another condition $F^{2}+ ( 2B+C ) ( 1-I ) =0$, which is not true. □

4.1.3 The case: $A=D=E=H=J=0$ and $I<1$

If $A=D=E=H=J=0$ and $I<1$, then $( \Phi ,\Psi ) = ( 0,\frac{1-I}{G} )$ is the period-two solution of the difference equation

$$ x_{n+1}=Bx_{n}^{2}x_{n-1}+Cx_{n}x_{n-1}^{2}+Fx_{n}x_{n-1}+Gx_{n-1}^{2}+Ix_{n-1}, $$

(36)

and the period-two solution $( \Phi,\Psi )$ satisfies the system

$$\begin{aligned}& \Phi= \bigl( B\Psi^{2}+C\Phi\Psi+F\Psi+G\Phi+I \bigr) \Phi, \\& \Psi= \bigl( B\Phi^{2}+C\Phi\Psi+F\Phi+G\Psi+I \bigr) \Psi. \end{aligned}$$

As $H+I<1$, there are locally stable zero equilibrium and positive unstable equilibrium. If $B=0$ and $F=G$ we obtain the special case where $( \Phi,\Psi )$ is the solution of the equation

$$ C\Phi\Psi+F ( \Phi+\Psi ) +I-1=0. $$

Thus every point on the curve $Cxy+F ( x+y ) +I-1=0$ is a period-two solution of (36) except the equilibrium point $( \overline{x}_{+},\overline{x}_{+} )$.

One can see that in this case we obtain

$$ a = B\Psi^{2}+F\Psi+I, \qquad b = 0, \qquad c = C\Psi^{2}+F \Psi, \qquad d = 2G\Psi+I. $$

Now,

$$\begin{aligned}& ( a-1 ) ( d-1 ) = \bigl( B\Psi^{2}+F\Psi +I-1 \bigr) ( 2G\Psi+I-1 ) = \bigl( B\Psi^{2}+F\Psi+I-1 \bigr) ( 1-I ) ,\\& ad = \bigl( B\Psi^{2}+F\Psi+I \bigr) ( 2G\Psi+I ) = \bigl( B\Psi^{2}+F\Psi+I \bigr) ( 2-I ), \end{aligned}$$

and

$$\begin{aligned} \mathcal{S}-\mathcal{D}-1 =- ( a-1 ) ( d-1 ) =- \bigl( B\Psi^{2}+F\Psi+I-1 \bigr) ( 1-I ) . \end{aligned}$$

Therefore,

$$ B\Psi^{2}+F\Psi+I-1=\frac{1-I}{G^{2}} \bigl( B ( 1-I ) +G ( F-G ) \bigr), $$

and for the positive equilibrium $\overline{x}_{+}$ we have

$$\begin{aligned}[b] q-p-1 &= ( C-B ) \overline{x}_{+}^{2}+2G \overline{x}_{+}+I-1 \\ &=\underset{=0}{\underbrace{ ( C+B ) \overline {x}_{+}^{2}+ ( F+G ) \overline{x}_{+}+I-1}}-2B\overline{x}_{+}^{2}+ ( G-F ) \overline{x}_{+} \\ &=-\overline{x}_{+} ( 2B\overline{x}_{+}+F-G ) . \end{aligned} $$

• If $B ( 1-I ) +G ( F-G ) >0$, then $\mathcal{S}- \mathcal{D}-1<0$, $\mathcal{D}>2-I>1$, and the period-two solution $( \Phi ,\Psi ) = ( 0,\frac{1-I}{G} )$ is a repeller. For example, if $F>G$, $B\geq0$ or $F=G$, $e B>0$, then $q-p-1<0$ and the positive equilibrium $\overline{x}_{+}$ is a saddle point.

• If $B ( 1-I ) +G ( F-G ) =0$, then $\mathcal{S}- \mathcal{D}-1=0$ and the period-two solution $( \Phi,\Psi ) = ( 0,\frac{1-I}{G} )$ is a nonhyperbolic point. For example if $F=G$ and $B=0$, then the positive equilibrium $\overline{x}_{+}$ is a nonhyperbolic point and every point on the curve $Cxy+F ( x+y ) +I-1=0$ is a period-two solution of (36), except $( \overline{x} _{+},\overline{x}_{+} )$. The points $( \Phi,\Psi ) = \{ ( 0,\frac{1-I}{G} ) , ( \frac{1-I}{G},0 ) \}$ are the endpoints of the curve $Cxy+F ( x+y ) +I-1=0$ in the first quadrant.

• If $B ( 1-I ) +G ( F-G ) <0$, then $\mathcal{S}- \mathcal{D}-1>0$ and the period-two solution $( \Phi,\Psi ) = ( 0,\frac{1-I}{G} )$ is a saddle point and $B ( 1-I ) +G ( F-G ) <0$. For example if $F< G$ and $B=0$, then $q-p-1>0$ and the positive equilibrium $\overline{x}_{+}$ is repeller.

One can prove that in this case the map T has no other minimal period-two solutions except $( \Phi,\Psi ) = ( 0,\frac{-G+\sqrt{G^{2}+4D ( 1-I ) }}{2D} )$. This leads to the following theorem.

Theorem 16

Let $A=D=E=H=J=0$ and $I<1$, then (36) has locally stable zero equilibrium and the positive equilibrium $\overline{x}_{+}$. The following statements hold:

(a) If $B=0$ and $F< G$, then the positive equilibrium $\overline{x}_{+}$ is repeller and there exists the period-two solution which is a saddle point. In this case there exist four continuous curves $\mathcal{W} ^{s}(P_{1})$, $\mathcal{W}^{s}(P_{2})$, $\mathcal{W}^{u}(P_{1})$, and $\mathcal{W}^{u}(P_{2})$. The curves $\mathcal{W}^{s}(P_{1})$ and $\mathcal{W}^{s}(P_{2})$ are passing through the point $E_{+}=(\bar{x}_{+},\bar{x}_{+})$ and they are graphs of decreasing functions. The curves $\mathcal{W}^{u}(P_{1})$ and $\mathcal{W} ^{u}(P_{2})$ are the graphs of increasing functions and are starting at $E_{0} ( 0,0 )$. Every solution $\{ x_{n} \}$ which starts below $\mathcal{W}^{s}(P_{1})\cup\mathcal{W}^{s}(P_{2})$ in the North-East ordering converges to $E_{0}$ and every solution $\{ x_{n} \}$ which starts above $\mathcal{W}^{s}(P_{1})\cup\mathcal{W} ^{s}(P_{2})$ in the North-East ordering satisfies $\lim_{n\rightarrow\infty}x_{n}=\infty$. (See Figure  1(a).)

(b) If $F>G$, $B\geq0$ or $F=G$, $B>0$, then the positive equilibrium $\overline{x}_{+}$ is a saddle point and there exists the period-two solution which is repeller. Global behavior of (34) is described by Theorem  3 where we set $E_{1}(0,0)$ and $E_{2}(\overline{x}_{+},\overline {x}_{+})$. (See Figure  1(b).)

(c) If $F=G$ and $B=0$, then every point on the curve $\mathcal{J}_{1}=Cxy+F ( x+y ) +I=1$, which is passing through the point $E=(\bar{x} _{+},\bar{x}_{+})$, is a period-two solution of (36) except E. All period-two solutions and the positive equilibrium $\overline{x}_{+}$ are nonhyperbolic points and the zero equilibrium is locally stable. The curve $\mathcal{J}_{1}$ divides the first quadrant into two regions. The region below $\mathcal{J}_{1}$ in the first quadrant is the basin of attraction of the zero equilibrium and the region above $\mathcal{J}_{1}$ in the first quadrant is the basin of attraction of point at infinity. (See Figure  1(c).)

Proof

(a) The existence of four curves $\mathcal{W} ^{s}(P_{1})$, $\mathcal{W}^{s}(P_{2})$, $\mathcal{W}^{u}(P_{1})$, and $\mathcal{W}^{u}(P_{2})$ with the described properties is guaranteed by Theorems 1 and 4 of [18] applied to the map $T^{2}$ given by (23). The global result follows from Theorem 3.

(b) All conditions of Theorem 3 are satisfied and the proof follows from Theorem 3.

(c) The facts that every point on the curve $\mathcal{J}_{1}$ is a nonhyperbolic period-two solution of (36), except $( \overline {x}_{+},\overline{x} _{+} )$, and that the points $( \Phi,\Psi ) = \{ ( 0, \frac{1-I}{G} ) , ( \frac{1-I}{G},0 ) \}$ are the endpoints of $\mathcal{J}_{1}$ in the first quadrant were proven earlier. This implies that the curve $\mathcal{J}_{1}$, which is passing through the point $E=(\bar{x}_{+},\bar {x}_{+})$ is not a global stable manifold of E. It remains to describe global behavior of (36) when $F=G$ and $B=0$. Since $I<1$ and the curve has endpoints in the first quadrant, the point $( 0,0 )$ is always below the curve. If the point $( x_{-1},x_{0} )$ is below the Julia set $\mathcal{J}_{1}$, then

$$ Cx_{-1}x_{0}+F ( x_{-1}+x_{0} ) +I< 1. $$

As $x_{n+1}= ( Cx_{n}x_{n-1}+F ( x_{n}+x_{n-1} ) +I ) x_{n-1}$, we have

$$\begin{aligned}& x_{1} = \bigl( Cx_{0}x_{-1}+F ( x_{0}+x_{-1} ) +I \bigr) x_{-1}< x_{-1},\\& x_{2} = \bigl( Cx_{1}x_{0}+F ( x_{1}+x_{0} ) +I \bigr) x_{0}< \bigl( Cx_{0}x_{-1}+F ( x_{0}+x_{-1} ) +I \bigr) x_{0}< x_{0}. \end{aligned}$$

Thus the point $( x_{1},x_{2} )$ is below $\mathcal{J}_{1}$ too and

$$ Cx_{1}x_{2}+F ( x_{1}+x_{2} ) +I< 1. $$

Now

$$\begin{aligned}& x_{3} = \bigl( Cx_{1}x_{2}+F ( x_{1}+x_{2} ) +I \bigr) x_{1}< x_{1},\\& x_{4} = \bigl( Cx_{3}x_{2}+F ( x_{3}+x_{2} ) +I \bigr) x_{2}< \bigl( Cx_{1}x_{2}+F ( x_{1}+x_{2} ) +I \bigr) x_{2}< x_{2}. \end{aligned}$$

Continuing in this way, we obtain $( 0,0 ) \preceq _{ne}\cdots\preceq _{ne} ( x_{3},x_{4} ) \preceq_{ne} ( x_{1},x_{2} ) \preceq_{ne} ( x_{-1},x_{0} )$. Hence, both subsequences $\{ x_{2n} \}$ and $\{x_{2n+1}\}$ are decreasing, which implies $x_{2n}\rightarrow0$ and $x_{2n+1}\rightarrow0$. If we start at the point $( x_{-1},x_{0} )$ above $\mathcal{J}_{1}$, then

$$ Cx_{-1}x_{0}+F ( x_{-1}+x_{0} ) +I>1, $$

which, in a similar way as above implies $( x_{-1},x_{0} ) \preceq_{ne} ( x_{1},x_{2} ) \preceq_{ne} \ldots$ . In this case both subsequences $\{ x_{2n} \}$ and $\{ x_{2n+1}\}$ are increasing, which implies that $x_{2n}\rightarrow\infty$ and $x_{2n+1}\rightarrow\infty$. □

The special cases studied in Theorem 16 leads to the following general result.

Theorem 17

Consider the difference equation

$$ x_{n+1}=P ( x_{n},x_{n-1} ) x_{n-1}, $$

(37)

where P is a symmetric polynomial function with nonnegative coefficients. If we assume that $P ( 0,0 ) <1$, then the curve $P ( x,y ) =1$ is the Julia set and separates the first quadrant into two regions: the region below the curve is the basin of attraction of $E_{0} ( 0,0 )$ and the region above the curve is the basin of attraction of a point at infinity.

Proof

Equilibrium points of (37) are solutions of the equation

$$\overline{x}=P ( \overline{x},\overline{x} ) \overline{x} \quad\Leftrightarrow\quad \overline{x} \bigl( P ( \overline{x},\overline {x} ) -1 \bigr) =0, $$

which implies that (37) has zero equilibrium. Let $h ( x ) =P ( x,x ) -1$. Then

$$h ( 0 ) < 0,\qquad h ( \infty ) =\infty\quad\mbox{and}\quad h^{\prime} ( x ) >0 \quad\mbox{for all }x>0, $$

and we see that the equation $P ( x,x ) =1$ has exactly one positive equilibrium $\overline{x}_{+}$. As a consequence of the symmetry we have $\frac{\partial P}{\partial x} ( x,y ) =\frac{\partial P}{ \partial y} ( y,x )$ and

$$\begin{aligned} p+q =&\overline{x} \biggl( \frac{\partial P}{\partial x} ( \overline{x}, \overline{x} ) + \frac{\partial P}{\partial y} ( \overline{x}, \overline{x} ) \biggr) +P ( \overline{x}, \overline{x} ) \\ =&2\overline{x}\frac{\partial P}{\partial x} ( \overline {x},\overline{x} ) +P ( \overline{x},\overline{x} ) , \end{aligned}$$

where p and q denote partial derivatives of function $g ( x,y ) =yP ( x,y )$ evaluated at the equilibrium $\overline{x}$. For the zero equilibrium we get

$$p+q=P ( 0,0 ) < 1, $$

and by applying Proposition 2, the zero equilibrium is locally asymptotically stable. Similarly, in the case of the positive equilibrium $\overline{x}_{+}$, we get

$$\begin{aligned} p+q =&2\overline{x}_{+}\frac{\partial P}{\partial x} ( \overline{x}_{+}, \overline{x}_{+} ) +P ( \overline{x}_{+}, \overline{x}_{+} ) \\ =&2\overline{x}_{+}\frac{\partial P}{\partial x} ( \overline{x}_{+}, \overline{x}_{+} ) +1\geq0 \end{aligned}$$

and

$$q-p=P ( \overline{x}_{+},\overline{x}_{+} ) =1. $$

By applying Proposition 2 the positive equilibrium $\overline{x} _{+}$ is unstable nonhyperbolic point. The period-two solution $\{ \Phi ,\Psi \}$, where $0\leq\Phi<\Psi$, satisfies the system

$$\begin{aligned}& \Phi=P ( \Psi,\Phi ) \Phi,\\& \Psi=P ( \Phi,\Psi ) \Psi. \end{aligned}$$

Since $\Psi>0$, we have $P ( \Phi,\Psi ) =1$ and $P ( \Psi,\Phi ) =P ( \Phi,\Psi ) =1$. Hence, every point of the set $\mathcal{J}_{2}= \{ ( x,y ) :P ( x,y ) =1 \}$ is a period-two solution of (37) except the equilibrium point $( \overline{x}_{+}, \overline{x}_{+} )$. If we start at the point $( x_{0},x_{-1} )$ below the curve $\mathcal{J}_{2}$, then

$$P ( x_{0},x_{-1} ) < 1 $$

and from the fact that $P ( x,y )$ is an increasing function in both variables we obtain the following:

$$\begin{aligned}& x_{1} =P ( x_{0},x_{-1} ) x_{-1}< x_{-1},\\& x_{2} =P ( x_{1},x_{0} ) x_{0}< P ( x_{-1},x_{0} ) x_{0}=P ( x_{0},x_{-1} ) x_{0}< x_{0}. \end{aligned}$$

Therefore the point $( x_{2},x_{1} )$ is also below $\mathcal {J}_{2}$ and

$$P ( x_{2},x_{1} ) < 1. $$

So

$$\begin{aligned}& x_{3} =P ( x_{2},x_{1} ) x_{1}< x_{1},\\& x_{4} =P ( x_{3},x_{2} ) x_{2}< P ( x_{1},x_{2} ) x_{2}=P ( x_{2},x_{1} ) x_{2}< x_{2}. \end{aligned}$$

Continuing in this way, we obtain $( 0,0 ) \preceq _{ne}\cdots\preceq _{ne} ( x_{3},x_{4} ) \preceq_{ne} ( x_{1},x_{2} ) \preceq_{ne} ( x_{-1},x_{0} )$. Hence, both subsequences $\{ x_{2n} \}$ and $\{x_{2n+1}\}$ are decreasing, which implies $x_{2n}\rightarrow0$ and $x_{2n+1}\rightarrow0$. If we start at the point $( x_{-1},x_{0} )$ above $\mathcal{J}_{2}$, then

$$P ( x_{0},x_{-1} ) >1. $$

The proof of the remaining case is similar and will be omitted. In this case both subsequences $\{ x_{2n} \}$ and $\{x_{2n+1}\}$ are increasing, which implies $x_{2n}\rightarrow\infty$ and $x_{2n+1}\rightarrow\infty$. □

4.1.4 The Case: $J=0$, $H+I\geq1$

Next we consider the case where $J=0$ and $H+I\geq1$. In this case (1) has exactly one equilibrium which is the zero equilibrium, which in view of Proposition 1 is unstable.

The following result describes a global dynamics of (1) in this case.

Theorem 18

Consider (1) under the conditions $J=0$ and $H+I\geq1$. The global behavior of solutions of (1) is as follows:

• (a) If $1-H< I\leq1$ and (1) does not possess a period-two solution, then every solution $\{x_{n}\}$ of (1) satisfies $\lim_{n\rightarrow\infty }x_{n}=\infty$;

• (b) If $1-H=I\leq1$, then every solution $\{x_{n}\}$ of (1) is either a period-two solution or it satisfies $\lim_{n\rightarrow\infty}x_{n}=\infty$;

• (c) If $1+H< I$ and (1) does not possess a period-two solution, then every solution $\{x_{n}\}$ of (1) satisfies $\lim_{n\rightarrow\infty }x_{n}=\infty$.

Proof

(a) In view of Theorem 3 there exist the global stable manifold $W^{s}(0,0)$ and the global unstable manifold $W^{u}(0,0)$, where $W^{s}(0,0)$ is a graph of continuous decreasing curve and $W^{u}(0,0)$ is a graph of continuous increasing curve and both manifolds are invariant sets. The only decreasing curve in the first quadrant $Q_{1}$ passing through $(0,0)$ is the union of the coordinate axes, but this set is clearly not an invariant set, which means that $W^{s}(0,0)$ is not a part of $Q_{1}$. On the other hand $W^{u}(0,0)$ exists and all solutions are asymptotic to $W^{u}(0,0)$. Thus if (1) does not possess a period-two solution, then every solution $\{x_{n}\}$ of (1) satisfies $\lim_{n\rightarrow\infty} x_{n}=\infty$.

(b) Assume that $\{x_{n}\}$ is not a minimal period-two solution of (1). Then $\{x_{n}\}$ is eventually monotone or the subsequences $\{x_{2n}\}$ and $\{x_{2n+1}\}$ are eventually monotone. If $\{x_{n}\}$ is eventually decreasing, then $x_{n}< x_{n-1}$ for all $n\geq K$, which implies $x_{n+1}\geq Hx_{n}+Ix_{n-1}>(H+I)x_{n}=x_{n}$, which is a contradiction. If the subsequences $\{x_{2n}\}$ and $\{x_{2n+1}\}$ are eventually monotone, then without loss of generality we can assume that $\{x_{2n}\}$ is eventually nondecreasing and $\{x_{2n+1}\}$ is eventually non-increasing. In this case $x_{2n}\rightarrow\infty$, which would imply that $x_{2n+1}\rightarrow\infty$, which is a contradiction. Thus the remaining possibility is that $\{x_{n}\}$ is eventually increasing, which implies that $x_{n}\rightarrow\infty$ as $n\rightarrow\infty$. Another way of proving the global behavior in the case when the equilibrium point E is nonhyperbolic was used in [18, 19]. To complete this we will find the image of $E+t\mathbf{v}$, where $t>0$ and v is the eigenvector that corresponds to the eigenvalue 1, under the map T. Since $E+t\mathbf{v}=(t,t)$, we have

$$ T(E+t\mathbf{v})-(E+t\mathbf{v} )=T\bigl((t,t)\bigr)-(t,t)=\bigl(0,(A+B+C+D)t^{3}+(E+F+G)t^{2}+(H+I-1)t \bigr). $$

By using the condition $H+I-1=0$, we have

$$ T(E+t\mathbf{v})-(E+t\mathbf{v})=\bigl(0,(A+B+C+D)t^{3}+(E+F+G)t^{2} \bigr), $$

which implies $E+t\mathbf{v}\preceq_{ne}T(E+t\mathbf{v})$ for every $t>0$. This shows that every point in $\mathbf{u} \in(\bar{x},\infty)^{2}$ is a supersolution for the map T, that is, $\mathbf{u} \preceq_{ne} T(\mathbf{u})$, see [18], and so every solution tends to ∞.

(c) In view of $I>1+H$ from (1) we have $x_{n+1}>Ix_{n-1}$, which implies $x_{n+1}>I^{k}x_{-1}$ or $x_{n+1}>I^{k}x_{0}$ for some k such that $k\rightarrow\infty$ as $n\rightarrow\infty$. Consequently every solution $\{x_{n}\}$ of (1) satisfies $\lim_{n\rightarrow\infty} x_{n}=\infty$. □

4.2 The case of two equilibrium points and a finite number of hyperbolic minimal period-two solutions

In this section we present the global dynamics of (1) in the parametric regions where there exist two distinct equilibrium points $E_{-}(\bar{x}_{-},\bar{x}_{-})$ and $E_{+}(\bar{x}_{+},\bar{x}_{+})$, such that $E_{-}\ll_{ne} E_{+}$, which holds if and only if

$$ \frac{S_{2} ( 2S_{2}^{2}-9S_{1}S_{3} ) -2\sqrt{ ( S_{2}^{2}-3S_{1}S_{3} ) ^{3}}}{27S_{1}^{2}}+J< 0, $$

and a finite number of period-two solutions which are hyperbolic. In this case, we prove that the Julia set is the union of the stable manifolds of some saddle period-two points and separates the first quadrant into two regions: the region below the curve is the basin of attraction of $E_{-}$ and the region above the curve is the basin of attraction of the point at infinity.

Let $T^{2}(x,y)=(g(x,y),h(x,y))$ where $g(x,y)=f(y,x)$ and $h(x,y)=f(f(y,x),y)$. Then the period-two curves, that is, the curves of which the intersection is a period-two solution, are given by

$$\begin{aligned}& \mathcal{C}_{\tilde{F}}:= \bigl\{ (x,y):f(y,x)=x\bigr\} =\bigl\{ (x,y):g(x,y)=x \bigr\} \quad\mbox{and}\\& \mathcal{C}_{\tilde{G}}:= \bigl\{ (x,y):f(x,y)=y\bigr\} =\bigl\{ (x,y):h(x,y)=y \bigr\} . \end{aligned}$$

Let

$$\begin{aligned}& \tilde{F}(x,y):=D x^{3}+(C y+G)x^{2} + \bigl(B y^{2}+F y+I-1 \bigr)x+A y^{3}+E y^{2}+H y+J, \\& \tilde{G}(x,y):=A x^{3}+(B y+E)x^{2} + \bigl(C y^{2}+F y+H \bigr)x +D y^{3}+G y^{2}+(I-1) y+J, \\& P_{1}(y):= ( A+B+C+D ) y^{3}+ ( E+F+G ) y^{2} + ( H+I-1 ) y+J. \end{aligned}$$

It is easy to see that $f(y,x)=x$ and $f(x,y)=y$ if and only if $\tilde {F}(x,y)=\tilde{G}(x,y)=0$. We can view $\tilde{F}$ and $\tilde{G}$ as polynomials in y with coefficients in $\mathbb{R}[x]$ and as polynomials in x with coefficients in $\mathbb{R}[y]$. Let $\operatorname{Res}_{x}(\tilde{F},\tilde{G})$ and $\operatorname{Res}_{y}(\tilde{F},\tilde{G})$ be resultants (see [14, 15]) of the polynomials $\tilde{F}(x,y)$ and $\tilde{G}(x,y)$ with respect to the variables x and y, respectively.

The following lemma gives some properties of $\tilde{F}(x,y)$, $\tilde {G}(x,y)$, $\operatorname{Res}_{x}(\tilde{F},\tilde{G})$, and $\operatorname{Res}_{y}(\tilde{F},\tilde{G})$.

Lemma 2

The following hold:

• (i) $\tilde{F}(x,y)=\tilde{G}(y,x)$ and $\operatorname{deg}_{x} (\tilde{F})=\operatorname{deg}_{y} (\tilde{G})\leq3$, and $\operatorname{deg}_{y} (\tilde{F})=\operatorname{deg}_{x} (\tilde{G})\leq3$, where the indices indicate in which variable we consider that polynomial.

• (ii) If $s(y)=\operatorname{Res}_{x}(\tilde{F},\tilde{G})$ and $r(x)=\operatorname{Res}_{y}(\tilde {F},\tilde{G})$, then $r(x)=(-1)^{\operatorname{deg}_{x}(\tilde{F})\cdot \operatorname{deg}_{x}(\tilde {G})} s(x)$.

Proof

• (i) The proof follows from the fact $\tilde{F}(x,y)=f(y,x)-x$ and $\tilde{G}(x,y)=f(x,y)-y$.

• (ii) Let $\tilde{F}(x,y):=\sum_{i=0}^{3} a_{i}(y)x^{i}$ and $\tilde{G}(x,y):=\sum_{i=0}^{3} b_{i}(y)x^{i}$. Since $\tilde{F}(x,y)=\tilde {G}(y,x)$, we obtain $\tilde{F}(x,y):=\sum_{i=0}^{3} b_{i}(x)y^{i}$ and $\tilde{G}(x,y):=\sum_{i=0}^{3} a_{i}(x)y^{i}$. From this and the definition of the Sylvester matrix we get $r(x)=(-1)^{\operatorname{deg}_{x}(\tilde{F})\cdot \operatorname{deg}_{x}(\tilde{G})} s(x)$.

 □

Let $P_{i}$, $i=2,3,4,5$ be the polynomials in the Appendix. The following lemma gives us information as regards the number of minimal period-two solutions.

Lemma 3

Assume that $\operatorname{Res}_{x}(\tilde{F},\tilde{G})=P_{1}(y) P(y)\not\equiv0$. Then there exist at most $\lfloor \operatorname{deg}(P)/2\rfloor$ isolated minimal period-two solutions. Let $\sharp PS$ denote the number of isolated minimal period-two solution. The following statements are true:

• (i) If $A>0$ and $D>0$, then $\operatorname{Res}_{x}(\tilde{F},\tilde{G})=-P_{1}(y) P_{2}(y)$. If $P_{2}\not\equiv0$, then $\sharp PS\leq3$.

• (ii) If $A=0$ and $D>0$, then $\operatorname{Res}_{x}(\tilde{F},\tilde {G})=-D^{3-\operatorname{deg}_{x} (\tilde{G})}P_{1}(y)P_{2}(y)$. If $P_{2}\not\equiv0$, then $\sharp PS\leq3$.

• (iii) If $D=0$ and $A>0$, then $\operatorname{Res}_{x}(\tilde{F},\tilde {G})=(-1)^{\operatorname{deg}_{x} (\tilde{F})}A^{3-\operatorname{deg}_{x} (\tilde{F})}P_{1}(y)P_{2}(y)$. If $P_{2}\not\equiv0$, then $\sharp PS\leq3$.

• (iv) If $A=D=0$, $C+B>0$, $C+G>0$, and $B+E>0$, then $\operatorname{Res}_{x}(\tilde{F},\tilde{G})=P_{1}(y)P_{3}(y)$. If $P_{3}\not\equiv0$, then $\sharp PS\leq2$.

• (v) If $A=D=0$, $B=E=0$, and $C>0$, then $\operatorname{Res}_{x}(\tilde{F},\tilde {G})=P_{1}(y)P_{4}(y)$. If $P_{4}\not\equiv0$, then $\sharp PS\leq1$.

• (vi) If $A=D=0$, $C=G=0$, and $B>0$, then $\operatorname{Res}_{x}(\tilde{F},\tilde {G})=P_{1}(y)P_{5}(y)$. If $P_{5}\not\equiv0$, then $\sharp PS\leq1$.

Proof

Suppose that $\{\Phi,\Psi\}$ is a minimal period-two solution ($\Phi\neq\Psi$). Then $\tilde{F}(\Phi,\Psi)=\tilde{G}(\Psi ,\Phi)=0$, Lemma 2, and Theorem 7 imply $\operatorname{Res}_{x}(\tilde{F},\tilde{G})(\Psi) =\operatorname{Res}_{y}(\tilde{F},\tilde{G})(\Phi)=0$, and we see that $P(\Phi)=P(\Psi)=0$. This implies that there exist at most $\lfloor \operatorname{deg}(P)/2\rfloor$ isolated minimal period-two solutions. By using the package Mathematica, one can see that the statements (i)-(vi) are true. □

The following lemma gives the necessary conditions under which an isolated period-two solution is nonhyperbolic.

Lemma 4

Assume that $\operatorname{Res}_{x}(\tilde{F},\tilde{G})=P_{1}(y)P(y)\not \equiv0$ (i.e. $P(y)\not\equiv0$). Then the following statements hold:

• (i) If (1) has a nonhyperbolic period-two solution, then $\operatorname{Dis}(P)=0$.

• (ii) If (1) has a nonhyperbolic equilibrium point, then

  $$\operatorname{Dis}(P_{1}\cdot P)=\operatorname{Dis}(P_{1}) \operatorname{Dis}(P)\operatorname{Res}(P_{1},P)^{2}=0, $$

  where $\operatorname{Dis}(P_{1})$ is given in the Appendix.

Proof

(i) Suppose that $(\Phi,\Psi)$ is a nonhyperbolic minimal period-two solution. This implies that $\tilde{F}(\Phi,\Psi)=0$, $\tilde {G}(\Phi,\Psi)=0$. By Theorem 7 we have $P_{1}(\Phi)P(\Phi )=0$. Since Φ is not an equilibrium point, we obtain $P_{1}(\Phi)\neq 0$ and $P(\Phi)=0$. Taking derivatives of $g(x, y) = x$ and $h(x, y) = y$ with respect to x we get

$$y'_{\tilde{F}}(x)=\frac{1-g'_{x}(x,y)}{g'_{y}(x,y)}\quad\mbox{and}\quad y'_{\tilde {G}}(x)=\frac{h'_{x}(x,y)}{1-h'_{y}(x,y)}. $$

One can see that

$$\begin{aligned} y'_{\tilde{F}}( \Phi)-y'_{\tilde{G}}(\Phi) =&\frac{1-g'_{x}(\Phi,\Psi )}{g'_{y}(\Phi,\Psi)}-\frac{h'_{x}(\Phi,\Psi)}{1-h'_{y}(\Phi,\Psi)} \\ =&\frac{1-e_{1}}{f_{1}}-\frac{g_{1}}{1-h_{1}}=\frac{1-(e_{1}+h_{1})+(e_{1} h_{1}-f_{1} g_{1})}{f_{1}(1-h_{1})} \\ =&\frac{p(1)}{f_{1}(1-h_{1})}=\frac{(1-\mu_{1})(1-\mu_{2})}{f(1-h)}, \end{aligned}$$

where $p(\mu)$ is the characteristic equation of the matrix

$$ J_{T^{2}}(\Phi,\Psi)= \begin{pmatrix} e_{1} & f_{1} \\ g_{1} & h_{1} \end{pmatrix}. $$

By Lemma 1 we have $\mu_{1},\mu_{2}\geq0$. Since $(\Phi ,\Psi)$ is a nonhyperbolic minimal period-two solution, we have $\mu _{1}=1$ or $\mu_{2}=1$. This implies $y'_{\tilde{F}}(\Phi)-y'_{\tilde {G}}(\Phi)=0$. Since $P\not\equiv0$, by Theorem 7 the curves $\mathcal{C}_{\tilde{F}}$ and $\mathcal{C}_{\tilde{G}}$ have no common components. In view of Lemmas 6 and 7 from [20], the curves $\mathcal{C}_{F}$ and $\mathcal{C}_{G}$ intersect tangentially at $(\Phi ,\Psi)$ (i.e. $y'_{\tilde{F}}(\Phi)-y'_{\tilde{G}}(\Phi)= 0$) if and only if Φ is zero of $P_{1}(y)P(y)$ of multiplicity greater than one. Since $P_{1}(\Phi)\neq0$, Φ is a zero of $P(y)$ of multiplicity greater than one. By Theorem 6, $P(y)$ has zero of multiplicity greater than one if and only if the discriminant $\operatorname{Dis}(P)$ is equal to zero, from which the proof follows.

(ii) Suppose that $\bar{x}$ is a nonhyperbolic equilibrium point. This implies that $\tilde{F}(\bar{x},\bar{x})=0$, $\tilde{G}(\bar {x},\bar{x})=0$. Similarly, as in (i), we have $P_{1}(\bar{x})P(\bar {x})=0$. Since $J_{T^{2}}(\bar{x},\bar{x})= (J_{T}(\bar{x},\bar {x}) )^{2}$, we see that all eigenvalues that correspond to $J_{T^{2}}(\bar{x},\bar{x})$ are nonnegative. Since $\bar{x}$ is a nonhyperbolic equilibrium point, we have $\mu_{1}=1$ or $\mu_{2}=1$. This implies $y'_{\tilde{F}}(\bar{x})-y'_{\tilde{G}}(\bar{x})=0$. Similarly, as in (i), we see that

$$\operatorname{Dis}(P_{1}\cdot P)=\operatorname{Dis}(P_{1}) \operatorname{Dis}(P)\operatorname{Res}(P_{1},P)^{2}=0. $$

 □

Let $\mathcal{B}(E_{-})$ be the basin of attraction of $E_{-}(\bar{x}_{-},\bar{x}_{-})$ and $\mathcal{B}(\infty,\infty)$ be the basin of attraction of $(\infty,\infty)$. The following lemma is true.

Lemma 5

If $E_{+}$ is nonhyperbolic, then the following statements hold:

• (i) If $Q_{1}(E_{+})=\{(x,y): x\geq\bar{x}_{+} \textit{ and } y\geq\bar{x}_{+}\}$, then $\operatorname{int}( Q_{1}(E_{+}))\subset\mathcal{B}(\infty,\infty)$.

• (ii) If $Q_{3}(E_{+})=\{(x,y): 0\leq x\leq\bar{x}_{+} \textit{ and } 0\leq y\leq\bar {x}_{+}\}$, then $\operatorname{int}(Q_{3}(E_{+}))\subset\mathcal{B}(\bar{x}_{-},\bar{x}_{-})$.

Proof

Assume that $(x_{0},y_{0})\in \operatorname{int}(Q_{3}(E_{+}))$. By Theorem 6 [18] there exists $(\tilde{x}_{0},\tilde{x}_{1})\in \operatorname{int}(Q_{3}(E_{+}))$ such that $(x_{0},y_{0})\preceq_{ne}(\tilde{x}_{0},\tilde{x}_{1})$ and $(\bar {x}_{-},\bar{x}_{-})\preceq_{ne}(\tilde{x}_{0},\tilde{x}_{1})$, and $T(\tilde {x}_{0},\tilde{x}_{1})\preceq_{ne}(\tilde{x}_{0},\tilde{x}_{1})$. By monotonicity of T we obtain $T^{i+1}(\tilde{x}_{0},\tilde {x}_{1})\preceq_{ne}T^{i}(\tilde{x}_{0},\tilde{x}_{1})\prec_{ne} E_{+}$, which implies $T^{n}(\tilde{x}_{0},\tilde{x}_{1})\to(\bar{x}_{-},\bar{x}_{-})$ as $n\to\infty$. Since $(0,0)\preceq_{ne}T(0,0)$ we have $T^{i}(0,0)\preceq _{ne}T^{i+1}(0,0)\prec_{ne} (\bar{x}_{-},\bar{x}_{-})$, which implies $T^{n}(0,0)\to(\bar{x}_{-},\bar{x}_{-})$ as $n\to\infty$. Similarly, one can prove $\operatorname{int}( Q_{1}(E_{+}))\subset\mathcal{B}(\infty,\infty)$. □

In view of Theorem 2 it is easy to see that $\{ T^{n}(x_{0},y_{0})\}$ is either asymptotic to $(\infty,\infty)$ or converges to a period-two solution, for all $(x_{0},y_{0})\in\mathcal{R}=[0,\infty)^{2}$. In view of Lemma 3 we can suppose that $\operatorname{Res}_{x}(\tilde{F},\tilde{G})=P_{1}(y) P(y)$, $P\in\mathbb{R} [y]$. If $P\not\equiv0$ and $\operatorname{\operatorname{Dis}}(P_{1}\cdot P)=\operatorname{Dis}(P_{1}) \operatorname{Dis}(P)\operatorname{Res}(P_{1},P)^{2}\neq0$, then by Theorem 12 and Lemma 4 we see that $E_{+}(\bar{x}_{+},\bar {x}_{+})$ is a repeller or a saddle point and all minimal period-two solutions are hyperbolic. By Lemma 5 we see that $\operatorname{int}(Q_{3}(E_{+}))\subset\mathcal {B}(\bar{x}_{-},\bar{x}_{-})$ and $\operatorname{int}( Q_{1}(E_{+}))\subset\mathcal{B}(\infty,\infty)$. Let $\mathcal{S}_{1}$ denote the boundary of $\mathcal{B}(\bar{x}_{-},\bar{x}_{-})$ considered as a subset of $Q_{2}(E)$ and $\mathcal{S}_{2}$ denote the boundary of $\mathcal{B}(\bar{x}_{-},\bar{x}_{-})$ considered as a subset of $Q_{4}(E_{+})$. It is easy to see that $E_{+}\in\mathcal{S}_{1}$, $E_{+}\in\mathcal{S}_{2}$ and $T(\mathcal{R})\subset \operatorname{int}(\mathcal{R})$.

The proof of the following lemma for a cooperative map is the same as the proof of Claims 1 and 2 [21] for a competitive map, so we skip it.

Lemma 6

Suppose that $P\not\equiv0$ and $\operatorname{Dis}(P_{1}\cdot P)\neq0$. Let $\mathcal {S}_{1}$ and $\mathcal{S}_{2}$ be the sets defined as above. Then

• (a) If $(x_{0},y_{0})\in\mathcal{B}(\bar{x}_{-},\bar{x}_{-})$, then $(x_{1},y_{1})\in\mathcal{B}(\bar{x}_{-},\bar{x}_{-})$ for all $(x_{1},y_{1})\preceq_{ne} (x_{0},y_{0})$.

• (b) If $(x_{0},y_{0})\in\mathcal{S}_{1}\cup\mathcal{S}_{2}$, then $(x_{1},y_{1})\in \operatorname{int}(\mathcal{B}(\bar{x}_{-},\bar{x}_{-}))$ for all $(x_{1},y_{1})\ll_{ne} (x_{0},y_{0})$.

• (c) $\mathcal{S}_{1} \cap \operatorname{int}(Q_{2}(E_{+}))\neq\emptyset$ and $\mathcal {S}_{2} \cap \operatorname{int}(Q_{4}(E_{+}))\neq\emptyset$.

• (d) $T(\mathcal{S}_{1}\cup\mathcal{S}_{2})\subseteq\mathcal {S}_{1}\cup\mathcal{S}_{2}$.

• (e) $(x_{0},y_{0}), (x_{1},y_{1})\in\mathcal{S}_{1}\cup\mathcal {S}_{2}\Rightarrow(x_{0},y_{0})\ll_{se} (x_{1},y_{1})\textit{ or } (x_{1},y_{1})\ll _{se} (x_{0},y_{0})$.

• (f) $\mathcal{S}_{1}\cup\mathcal{S}_{2}$ is the graph of continuous strictly decreasing function.

Theorem 19

Suppose that $P\not\equiv0$ and $\operatorname{Dis}(P_{1}\cdot P)\neq0$. If $E_{+}$ is a repeller, then there exists at least one minimal period-two solution $\{(\phi,\psi),(\psi,\phi)\}$, which is a saddle point, such that $(\phi,\psi)\ll_{se}E_{+}\ll_{se} (\psi,\phi)$.

Proof

In view of Lemma 6 we see that $(\mathcal{S}_{1}\cup \mathcal{S}_{2},\ll_{se})$ is a totally ordered set which is invariant under T. If $(x_{0},y_{0})\in(\mathcal{S}_{1}\cup\mathcal{S}_{2})\setminus \{E_{+}\}$, then $\{T^{(2n)}(x_{0},y_{0})\}$ is eventually componentwise monotone. Since $\mathcal{S}_{1}\cup\mathcal{S}_{2}$ is the graph of a continuous strictly decreasing function, there exists a minimal period-two solution $\{(\Phi,\Psi),(\Psi,\Phi)\} \in(\mathcal{S}_{1}\cup\mathcal {S}_{2})\setminus\{E_{+}\}$ such that $T^{(2n)}(x_{0},y_{0})\to(\Phi,\Psi)$ as $n\to\infty$. Since $\mathcal {S}_{1}\cup\mathcal{S}_{2}=\partial\mathcal{B}(\bar{x}_{-},\bar{x}_{-})$ is a closed set, we see that $\{(\Phi,\Psi),(\Psi,\Phi)\}$ belongs to $(\mathcal {S}_{1}\cup\mathcal{S}_{2})\setminus\{E_{+}\}$. By Lemma 4 all period-two solutions are hyperbolic. Since $\{(\Phi,\Psi),(\Psi,\Phi )\}\in\partial\mathcal{B}(\bar{x}_{-},\bar{x}_{-})$, it is not locally asymptotically stable. Thus $\{(\Phi,\Psi),(\Psi,\Phi)\}$ must be a saddle point. □

Corollary 1

Suppose that $P\not\equiv0$ and $\operatorname{Dis}(P_{1}\cdot P)\neq0$. If $E_{+}(\bar {x}_{+},\bar{x}_{+})$ is a repeller, then $\operatorname{int}(Q_{2}(E_{+})) \cup \operatorname{int}(Q_{4}(E_{+}))$ contains one or three distinct minimal period-two solutions. If T has one minimal period-two solution $\{ (\Phi_{1},\Psi_{1}),(\Psi_{1},\Phi_{1})\}$, then it is a saddle point and $(\Phi _{1},\Psi_{1})\ll_{se} E_{+} \ll_{se}(\Psi_{1},\Phi_{1})$. If T has three minimal period-two solutions $\{(\Phi_{i},\Psi_{i}), (\Psi_{i},\Phi_{i})\} _{i=1}^{3}$, then they are ordered in the South-East ordering. If $(\Phi _{3},\Psi_{3})\ll_{se}(\Phi_{2},\Psi_{2})\ll_{se}(\Phi_{1},\Psi_{1})\ll_{se} E_{+} \ll _{se}(\Psi_{1},\Phi_{1})\ll_{se}(\Psi_{2},\Phi_{2})\ll_{se}(\Psi_{3},\Phi_{3})$, then odd indexed period-two points are saddles and even indexed period-two points are repellers.

Proof

By Lemma 4 all equilibrium points and minimal period-two solutions are hyperbolic. In view of Theorem 7 we see that $\tilde{F}$ and $\tilde{G}$ have no common component. From Lemma 3 the number of minimal period-two solutions is at most three. In view of Theorem 19, T has at least one minimal period-two solution, which is a saddle point. Assume that T has two minimal period-two solutions $\{(\Phi_{1},\Psi_{1}),(\Psi_{1},\Phi_{1})\}$ and $\{(\Phi_{2},\Psi_{2}),(\Psi _{2},\Phi_{2})\}$. Assume that $\{(\Phi_{1},\Psi_{1}),(\Psi_{1},\Phi_{1})\}$ is a saddle point and $(\Phi_{1},\Psi_{1})\ll_{se} E_{+} \ll_{se}(\Psi_{1},\Phi_{1})$. Further, suppose that $(\Phi_{2},\Psi_{2})\ll_{se}(\Phi_{1},\Psi_{1})\ll_{se} E_{+} \ll_{se}(\Psi_{1},\Phi_{1})\ll_{se}(\Psi_{2},\Phi_{2})$. The map $T^{2}$ satisfies all conditions of Theorem 3, which yields the existence of the global stable manifolds $\mathcal{W}^{s}(\{(\Phi_{1},\Psi_{1}),(\Psi_{1},\Phi_{1})\} )$, the union of two curves $\mathcal{W}^{s}(\Phi_{1},\Psi_{1})$ and $\mathcal{W}^{s}((\Psi_{1},\Phi_{1}))$ that have a common endpoint $E_{+}$. Then $\mathcal{W}^{s}(\Phi_{1},\Psi_{1})$ has the second endpoint at $(\Phi _{2},\Psi_{2})$ and $\mathcal{W}^{s}(\Psi_{1},\Phi_{1})$ has the second endpoint at $(\Psi_{2},\Phi_{2})$. Furthermore, the minimal period-two solution $\{(\Phi_{2},\Psi_{2}),(\Psi _{2},\Phi_{2})\}$ is a repeller. Since the global stable manifold is unique, the set $(\mathcal{S}_{1}\cap Q_{2}(\Phi_{2},\Psi_{2}))\cup(\mathcal {S}_{2}\cap Q_{4}(\Psi_{2},\Phi_{2}))$ is invariant under T. Similarly, as in Theorem 19, one can prove that $\operatorname{int}(Q_{2}(\Phi_{2},\Psi_{2}))\cup \operatorname{int}(Q_{4}(\Psi_{2},\Phi_{2}))$ contains exactly one minimal period-two solution, which is a saddle point. Hence, if T has two minimal period-two solutions, then there exists a third minimal period-two solution. This proves the lemma. If $(\Phi_{1},\Psi_{1})\ll_{se}(\Phi_{2},\Psi _{2})\ll_{se} E_{+} \ll_{se}(\Psi_{2},\Phi_{2})\ll_{se}(\Psi_{1},\Phi_{1})$ the proof is similar and will be omitted. □

Corollary 2

Assume that $P\not\equiv0$ and $\operatorname{Dis}(P_{1}\cdot P)\neq0$. If $E_{+}(\bar {x}_{+},\bar{x}_{+})$ is a saddle point, then $\operatorname{int}(Q_{2}(E_{+})) \cup \operatorname{int}(Q_{4}(E_{+}))$ contains either zero or two minimal period-two solutions $\{(\Phi_{i},\Psi_{i}),(\Psi_{i},\Phi_{i})\}$, $i=1,2$, which are ordered to the South-East ordering. If there exist two minimal period-two solutions such that $(\Phi_{2},\Psi_{2})\ll_{se}(\Phi _{1},\Psi_{1})\ll_{se} E_{+} \ll_{se}(\Psi_{1},\Phi_{1})\ll_{se}(\Psi_{2},\Phi_{2})$, then an even indexed period-two point is a saddle and an odd indexed period-two point is a repeller.

Proof

The proof is similar to the proof of Corollary 1 and it will be omitted. □

Theorem 20

Suppose that $P\not\equiv0$ and $\operatorname{Dis}(P_{1}\cdot P)\neq0$. If $E_{+}(\bar {x}_{+},\bar{x}_{+})$ is a repeller, then $\operatorname{int}(Q_{2}(E_{+}))\cup \operatorname{int}(Q_{4}(E_{+}))$ contains one or three minimal period-two solutions $\{(\Phi_{i},\Psi_{i}), (\Psi_{i},\Phi_{i})\}_{i=1}^{2n+1}$, where $n=0$ or $n=1$, such that $(\Phi_{i+1},\Psi_{i+1})\ll_{se} (\Phi_{i},\Psi_{i})\ll_{se} E_{+}$ and $E_{+}\ll_{se}(\Psi_{i},\Phi_{i})\ll_{se} (\Psi_{i+1},\Phi_{i+1})$, and $(\Psi_{i},\Phi_{i})=T(\Phi_{i},\Psi_{i})$. Furthermore, the odd indexed period-two points are saddles and the even indexed period-two points are repellers and the following hold:

(i) If there exists one minimal period-two solution $\{(\Phi _{1},\Psi_{1}),(\Psi_{1},\Phi_{1})\}$, the global stable manifolds is given by

$$\mathcal{W}^{s} \bigl( \bigl\{ (\Phi_{1}, \Psi_{1}),(\Psi_{1},\Phi_{1}) \bigr\} \bigr)= \mathcal{W}^{s}(\Phi_{1},\Psi_{1})\cup \mathcal{W}^{s}(\Psi_{1},\Phi_{1}), $$

where $\mathcal{W}^{s}(\Phi_{1},\Psi_{1})$ and $\mathcal{W}^{s}(\Psi _{1},\Phi_{1})$ are the graphs of a continuous strictly decreasing functions with common endpoint at $E_{+}$ and $\mathcal{W}^{s}(\Psi_{1},\Phi _{1})=T (\mathcal{W}^{s}(\Phi_{1},\Psi_{1}) )$. The Julia set is the curve

$$\mathcal{C}=\mathcal{W}^{s}(\Phi_{1},\Psi_{1}) \cup\mathcal{W}^{s}(\Psi _{1},\Phi_{1})\cup \{E_{+} \} $$

that separates $\mathcal{R} =\{(x_{-1},x_{0}):x_{-1}\geq0,x_{0}\geq0\}$ into two components $\mathcal{W}^{-}$ and $\mathcal{W}^{+}$, which are basins of attraction of $E_{-}(\bar{x}_{-},\bar {x}_{-})$ and $(\infty,\infty)$, respectively, where

$$\begin{aligned}[b] &\mathcal{W}^{-}=\bigl\{ (x,y)|(x,y)\preceq_{ne}( \tilde{x}_{0},\tilde{y}_{0}) \textit{ for some } ( \tilde{x}_{0},\tilde{y}_{0})\in \mathcal{C}\bigr\} ; \\ &\mathcal{W}^{+}=\bigl\{ (x,y)|(\tilde{x}_{1},\tilde{y}_{1}) \preceq_{ne}(x,y) \textit{ for some } (\tilde{x}_{1}, \tilde{y}_{1})\in \mathcal{C}\bigr\} . \end{aligned} $$

The global unstable manifold is given by

$$\mathcal{W}^{u} \bigl( \bigl\{ (\Phi_{1}, \Psi_{1}),(\Psi_{1},\Phi_{1}) \bigr\} \bigr)= \mathcal{W}^{u}(\Phi_{1},\Psi_{1})\cup \mathcal{W}^{u}(\Psi _{1},\Phi_{1}), $$

where $\mathcal{W}^{u}(\Phi_{1},\Psi_{1})$ and $\mathcal{W}^{u}(\Psi_{1},\Phi _{1})$ are the graphs of continuous strictly increasing functions such that $\mathcal{W}^{u}(\Psi_{1},\Phi_{1})=T (\mathcal{W}^{u}(\Phi _{1},\Psi_{1}) )$, with common endpoint at $E_{-}$. See Figure  2(a) for visual illustration.

Figure 2

Visual illustration of Theorems 20 and 21 . (a) Visual illustration of Theorem 20, case (i); (b) visual illustration of Theorem 21, case (a).

(ii) If there exist three minimal period-two solutions, the global stable manifolds are given by

$$\mathcal{W}^{s} \bigl( \bigl\{ (\Phi_{2 k+1}, \Psi_{2 k+1}),(\Psi_{2 k+1},\Phi_{2 k+1}) \bigr\} \bigr)= \mathcal{W}^{s}(\Phi_{2 k+1},\Psi_{2 k+1})\cup \mathcal{W}^{s}(\Psi_{2 k+1},\Phi_{2 k+1}), \quad k=0,1, $$

where $\mathcal{W}^{s}(\Phi_{2 k+1},\Psi_{2 k+1})$ and $\mathcal{W}^{s}(\Psi_{2 k+1},\Phi_{2 k+1})$, $k=0,1$ are the graphs of a continuous strictly decreasing function such that $\mathcal{W}^{s}(\Psi_{2 k+1},\Phi_{2 k+1})=T (\mathcal{W}^{s}(\Phi_{2 k+1},\Psi_{2 k+1}) )$, with endpoints at  $E_{+}$, and $(\Phi_{2}, \Psi_{2})$, $(\Psi_{2},\Phi_{2})$. The Julia set is the curve

$$\mathcal{C}=\bigcup_{k=0}^{1} \bigl( \mathcal{W}^{s}(\Phi_{2 k+1},\Psi_{2 k+1})\cup \mathcal{W}^{s}(\Psi_{2 k+1},\Phi_{2 k+1})\cup \bigl\{ ( \Phi_{2 k},\Psi_{2 k}),(\Psi_{2k}, \Phi_{2k}) \bigr\} \bigr) $$

that separates $\mathcal{R} =\{(x_{-1},x_{0}):x_{-1}\geq0,x_{0}\geq0\}$ into two components $\mathcal{W}^{-}$ and $\mathcal{W}^{+}$, which are basins of attraction of $E_{-}$ and $(\infty,\infty)$, respectively, where

$$\begin{aligned}& \mathcal{W}^{-}=\bigl\{ (x,y)|(x,y)\preceq_{ne}( \tilde{x}_{0},\tilde{y}_{0}) \textit{ for some } ( \tilde{x}_{0},\tilde{y}_{0})\in \mathcal{C}\bigr\} ; \\& \mathcal{W}^{+}=\bigl\{ (x,y)|(\tilde{x}_{1},\tilde{y}_{1}) \preceq_{ne}(x,y) \textit{ for some } (\tilde{x}_{1}, \tilde{y}_{1})\in \mathcal{C}\bigr\} . \end{aligned}$$

The global unstable manifolds are given by

$$\begin{aligned} &\mathcal{W}^{u} \bigl( \bigl\{ (\Phi_{2 k+1}, \Psi_{2 k+1}),(\Psi_{2 k+1},\Phi _{2 k+1}) \bigr\} \bigr)\\ &\quad= \mathcal{W}^{u}(\Phi_{2 k+1},\Psi_{2 k+1})\cup \mathcal{W}^{u}(\Psi_{2 k+1},\Phi_{2 k+1}),\quad k=0,1, \end{aligned}$$

where $\mathcal{W}^{u}(\Phi_{2 k+1},\Psi_{2 k+1})$ and $\mathcal{W}^{u}(\Psi_{2 k+1},\Phi_{2 k+1})$ are the graphs of continuous strictly increasing functions such that $\mathcal{W}^{u}(\Psi_{2 k+1},\Phi_{2 k+1})=T (\mathcal{W}^{u}(\Phi_{2 k+1},\Psi_{2 k+1}) )$, with common endpoints at $E_{-}$.

Proof

In view of Corollary 1 the set $\operatorname{int}(Q_{2}(E_{+}))\cup \operatorname{int}(Q_{4}(E_{+}))$ contains an odd number of minimal period-two solutions $\{(\Phi_{i},\Psi_{i}),(\Psi_{i},\Phi_{i})\}$, $i=1,\ldots,2 n+1$ ($n=0$ or $n=1$), such that $(\Phi_{i+1},\Psi_{i+1})\ll_{ne} (\Phi_{i},\Psi_{i})\ll_{ne} E_{+}$ and $E_{+}\ll_{ne}(\Psi_{i},\Phi_{i})\ll_{ne} (\Psi_{i+1},\Phi_{i+1})$ and $(\Psi_{i},\Phi_{i})=T(\Phi_{i},\Psi_{i})$. Furthermore, the odd indexed period-two points are saddles and the even indexed period-two points are repellers. The map $T^{2}$ satisfies all conditions of Theorems 1, 2, and 4 [18], which yields the existence of the global stable and unstable manifolds with the above properties. In view of Theorems 1, 2, and 4 [18] for $(x_{0},y_{0})\in\mathcal{W^{-}}\cap \mathcal{R}$ there exists $n_{0}>0$ such that

$$\begin{aligned}[b] &T^{n}(x_{0},y_{0})\in \bigcup _{k=0}^{n} \bigl(\operatorname{int}\bigl(Q_{3}( \Phi_{2 k+1},\Psi_{2 k+1})\bigr)\cup \operatorname{int}\bigl(Q_{3}( \Psi_{2 k+1},\Phi_{2 k+1})\bigr) \bigr)\cap\mathcal{R}\subset \mathcal{B}(\bar{x}_{-},\bar {x}_{-}), \\ &\quad n>n_{0}, \end{aligned} $$

and for $(x_{0},y_{0})\in \mathcal{W^{+}}\cap\mathcal{R}$ there exists $n_{1}>0$ such that

$$\begin{aligned}[b] &T^{n}(x_{0},y_{0})\in\bigcup _{k=0}^{n} \bigl(\operatorname{int}\bigl(Q_{1}( \Phi_{2 k+1},\Psi_{2 k+1})\bigr)\cup \operatorname{int}\bigl(Q_{1}( \Psi_{2 k+1},\Phi_{2 k+1})\bigr) \bigr)\cap\mathcal{R}\subset \mathcal{B}(\infty ,\infty),\\ & \quad n>n_{1}, \end{aligned} $$

which completes the proof. □

Theorem 21

Suppose that $P\not\equiv0$ and $\operatorname{Dis}(P_{1}\cdot P)\neq0$. If $E_{+}(\bar {x}_{+},\bar{x}_{+})$ is a saddle point, then either T has no minimal period-two solution or $\operatorname{int}(Q_{2}(E_{+}))\cup \operatorname{int}(Q_{4}(E_{+}))$ contains two distinct minimal period-two solutions and the following hold:

(a) If T has no minimal period-two solution, then there exist two continuous curves $\mathcal{W}^{s}(E_{+})$ and $\mathcal{W}^{u}(E_{+})$, both passing through the point $E_{+}(\bar{x}_{+},\bar{x}_{+})$, such that $\mathcal{W}^{s}(E_{+})$ is a graph of a decreasing function and $\mathcal{W}^{u}(E_{+})$ is a graph of an increasing function. The first quadrant of the initial condition $Q_{1}= \{(x_{-1},x_{0}):x_{-1}\geq0,x_{0}\geq0\}$ is the union of three disjoint basins of attraction, namely

$$ Q_{1}=\mathcal{B}(\bar{x}_{-},\bar{x}_{-})\cup\mathcal{B}(E_{+})\cup \mathcal {B}(\infty,\infty), $$

where $\mathcal{B}(E_{+})=\mathcal{W}^{s}(E_{+})$, and

$$\begin{aligned}& \mathcal{B}(\bar{x}_{-},\bar{x}_{-})=\mathcal{W}^{-}=\bigl\{ (x,y)|(x,y)\preceq _{ne}(\tilde{x}_{0},\tilde{y}_{0}) \textit{ for some } (\tilde{x}_{0},\tilde{y}_{0})\in \mathcal{W}^{s}(E_{+})\bigr\} , \\& \mathcal{B}(\infty,\infty)=\mathcal{W}^{+}=\bigl\{ (x,y)|(\tilde{x}_{1}, \tilde {y}_{1})\preceq _{ne}(x,y) \textit{ for some } ( \tilde{x}_{1},\tilde{y}_{1})\in \mathcal{W}^{s}(E_{+}) \bigr\} . \end{aligned}$$

In addition, for every $(x_{-1},x_{0})\in Q_{1}\setminus W^{s}(E_{+})$ every solution is asymptotic to $W^{u}(E_{+})$. See Figure  2(b) for visual illustration.

(b) If $\operatorname{int}(Q_{2}(E_{+}))\cup \operatorname{int}(Q_{4}(E_{+}))$ contains two minimal period-two solutions $\{(\Phi_{1},\Psi_{1}), (\Psi_{1},\Phi_{1})\}$ and $\{(\Phi _{2},\Psi_{2}),(\Psi_{2},\Phi_{2})\}$ such that $(\Psi_{2},\Phi_{2})\ll_{se}(\Phi _{1},\Psi_{1})\ll_{se} E_{+} \ll_{se}(\Psi_{1},\Phi_{1})\ll_{se}(\Psi_{2},\Phi_{2})$ then $\{(\Phi_{1},\Psi_{1}),(\Psi_{1},\Phi_{1})\}$ is a repeller and $\{(\Phi _{2},\Psi_{2}),(\Psi_{2},\Phi_{2})\}$ is a saddle point.

The global stable manifold of $\{(\Phi_{2},\Psi_{2}),(\Psi_{2},\Phi_{2})\}$ is given by

$$\mathcal{W}^{s} \bigl( \bigl\{ (\Phi_{2}, \Psi_{2}),(\Psi_{2},\Phi_{2}) \bigr\} \bigr)= \mathcal{W}^{s}(\Phi_{2},\Psi_{2})\cup \mathcal{W}^{s}(\Psi_{2},\Phi_{2}), $$

where $\mathcal{W}^{s}(\Phi_{2},\Psi_{2})$ and $\mathcal{W}^{s}(\Psi _{2},\Phi_{2})$ are the graphs of a continuous strictly decreasing functions such that $\mathcal{W}^{s}(\Psi_{2},\Phi_{2})=T (\mathcal {W}^{s}(\Phi_{2},\Psi_{2}) )$, with the first endpoints at $(\Phi_{1},\Psi_{1})$ and $(\Psi_{1}, \Phi_{1})$, respectively. The second endpoints of $\mathcal{W}^{s}(\Phi _{2},\Psi_{2})$ and $\mathcal{W}^{s}(\Psi_{2},\Phi_{2})$ belong to the boundary of $\mathcal{R}$. The global stable manifold $\mathcal {W}^{s}(E_{+})$ of $E_{+}$ is a graph of a continuous strictly decreasing function with endpoint at $(\Phi_{1},\Psi_{1})$ and $(\Psi_{1},\Phi _{1})$ The Julia set is the curve

$$\mathcal{C}=\mathcal{W}^{s}(\Phi_{2},\Psi_{2}) \cup\mathcal{W}^{s}(\Psi _{2},\Phi_{2})\cup \bigl\{ (\Phi_{1},\Psi_{1}),(\Psi_{1}, \Phi_{1}) \bigr\} \cup\mathcal{W}^{s}(E_{+}) $$

that separates $\mathcal{R}$ into two components $\mathcal{W}^{-}$ and $\mathcal{W}^{+}$, which are the basins of attraction of $(\bar{x}_{-},\bar {x}_{-})$ and $(\infty,\infty)$, respectively, where

$$\begin{aligned}& \mathcal{W}^{-}=\bigl\{ (x,y)|(x,y)\preceq_{ne}( \tilde{x}_{0},\tilde{y}_{0}) \textit{ for some } ( \tilde{x}_{0},\tilde{y}_{0})\in \mathcal{C}\bigr\} ; \\& \mathcal{W}^{+}=\bigl\{ (x,y)|(\tilde{x}_{1},\tilde{y}_{1}) \preceq_{ne}(x,y) \textit{ for some } (\tilde{x}_{1}, \tilde{y}_{1})\in \mathcal{C}\bigr\} . \end{aligned}$$

Further, we have

$$\begin{aligned}& \mathcal{B} \bigl( \bigl\{ (\Phi_{2}, \Psi_{2 }),(\Psi_{2},\Phi_{2 }) \bigr\} \bigr)= \mathcal{W}^{s}(\Phi_{2},\Psi_{2})\cup \mathcal{W}^{s}(\Psi _{2},\Phi_{2});\\& \mathcal{B} (E )=\mathcal{W}^{s}(E_{+}). \end{aligned}$$

The global unstable manifold of $\{(\Phi_{2},\Psi_{2}),(\Psi_{2},\Phi_{2})\}$ is given by

$$\mathcal{W}^{u} \bigl( \bigl\{ (\Phi_{2}, \Psi_{2}),(\Psi_{2},\Phi_{2}) \bigr\} \bigr)= \mathcal{W}^{u}(\Phi_{2},\Psi_{2})\cup \mathcal{W}^{u}(\Psi _{2},\Phi_{2}), $$

where $\mathcal{W}^{u}(\Phi_{2},\Psi_{2})$ and $\mathcal{W}^{u}(\Psi_{2},\Phi_{2})$ are the graphs of continuous strictly increasing functions such that $\mathcal{W}^{u}(\Psi_{2},\Phi_{2})=T (\mathcal{W}^{u}(\Phi_{2},\Psi _{2}) )$, with endpoints at $(\bar{x}_{-},\bar{x}_{-})$ and $(\infty,\infty)$. The global unstable manifold $\mathcal{W}^{u}(E_{+})$ of $E_{+}$ is the graph of a continuous strictly increasing function such that $\mathcal{W}^{u}(\Phi_{2},\Psi_{2})$ and $\mathcal{W}^{u}(\Psi_{2},\Phi _{2})$, and $\mathcal{W}^{u}(E_{+})$ have a common endpoint at $(\bar {x}_{-},\bar{x}_{-})$.

Proof

The proof is similar to the proof of Theorem 20 and it will be omitted. □

All figures are generated by the software package Dynamics 3 [22].

Appendix:  Values of coefficients \(p_{i}\) for \(i=0,\ldots,6\), and \(\operatorname{Dis}(P_{1})\)

$$\begin{aligned} P_{2}(y)={}&p_{6}y^{6}+p_{5}y^{5}+ \cdots+p_{1}y+p_{0},\\ p_{0}={}&A^{3} J^{2}+A^{2} \bigl(-3 D J^{2}+J \bigl(G \bigl(-2 H-(3-3 I)\bigr)+(1-I) E\bigr)-2 I \bigl(H+(1-I)\bigr) \bigr)\\ &{}+A \bigl(3 D^{2} J^{2}+\bigl(H+(1-I) \bigr) \bigl(H \bigl(G^{2}+(2-2 I) D \bigr)+(1-I) E G \bigr)\\ &{}+J \bigl(G (E-G)^{2}+D \bigl(G \bigl(H+(3-3 I)\bigr)+E \bigl(3 H+(1-I) \bigr)\bigr) \bigr) \bigr)-D^{3} J^{2} \\ &{}+D \bigl(H+(1-I)\bigr) \bigl(D H^{2}-E G H-(1-I) E^{2} \bigr)-D J \bigl(E (E-G)^{2}\\ &{}+D \bigl(-G H+E \bigl(3 H+(2-2 I)\bigr) \bigr) \bigr), \\ p_{1}={}&A^{3} \bigl(2 H+(1-I)\bigr) J+A^{2} \bigl(J \bigl((1-I) B+C \bigl(-2 H-(3-3 I)\bigr)-3 D H+F G-E F\bigr)\\ &{}-\bigl(H+(1-I)\bigr) \bigl(2 \bigl(G H+(1-I) F\bigr)-(1-I) E\bigr)\bigr)\\ &{}+A \bigl(J \bigl(B \bigl(2 (E-G) G+D \bigl(3 H+(1-I)\bigr)\bigr)\\ &{}+C \bigl((E-3 G) (E-G) +D \bigl(H+(3-3 I)\bigr)\bigr)+D \bigl(2 F (E-G)-(3-3 I) D\bigr)\bigr)\\ &{}+ \bigl(H+(1-I)\bigr) \bigl(G \bigl((E-G) (-F+G+E)+(1-I) B\bigr) \\ &{}+C \bigl(2 G H+(1-I) E\bigr)+D \bigl(3 \bigl(E H-(1-I) G\bigr)-2 F \bigl(H-(1-I) \bigr)\bigr)\bigr)\bigr)\\ &{}+D \bigl(-B J \bigl(D \bigl(3 H+(2-2 I)\bigr)+G^{2}-4 E G+3 E^{2} \bigr) \\ &{}- B \bigl(H+(1-I)\bigr) \bigl(G H+(2-2 I) E\bigr)+D J (C H+F G-E F)\\ &{}-E \bigl(H+(1-I)\bigr) \bigl(C H+(E-G) (-F+G+E)\bigr)+ 2 E C (E-G) J \\ &{}+D^{2} \bigl(H+(2-2 I)\bigr) J+D \bigl(H+(1-I) \bigr) \bigl(2 F H-G H+(2-2 I) E\bigr) \bigr), \\ p_{2}={}&A^{3} \bigl(-J (F+G)+H^{2}+(1-I) H+2 E J-2 I \bigr)\\ &{}+A^{2} \bigl(-B J (F-3 G+E)-(1-I) B \bigl(H+(2-2 I)\bigr) \\ &{}+ C \bigl(J (F-2 G)-2H^{2}-(2-2 I) H-2 I \bigr)+3 D F J+(3-3 I) D H-3 E D J\\ &{}+F^{2} H-I F^{2}-I F G \\ &{}+ (E-F) (-F-G+E)-E F H+2 I E F+2 G^{2} H-2 E G H+I E G-I E^{2} \bigr) \\ &{}+A (B^{2} G J+B \bigl(C \bigl(2 (E-2 G) J+(1-I) H-2 I\bigr)-D J (-2 F+3 G+E)\\ &{}+D \bigl(H-(1-I)\bigr) H - G H (F+G-E)+(-1+I) F G\bigr)\\ &{}+C^{2} \bigl(3 G J+H^{2}+(1-I) H-2 E J \bigr)+C \bigl(D J (-2 F+G+3 E) \\ &{} +(1-I) D \bigl(H-(1-I)\bigr)-E \bigl(F \bigl(H+(1-I)\bigr)-(1-I) G\bigr)\\ &{}+G \bigl(2 F \bigl(H+(1-I)\bigr)-G \bigl(H+(2-2 I)\bigr)\bigr) \\ & {} +E^{2} \bigl(H+(1-I)\bigr) \bigr)+3 D^{2} J (G-F)-(3-3 I) D^{2} H-2 D F^{2} H-(2-2 I) D F^{2}\\ &{}+2D F G H+(2-2 I) E D F +(3-3 I) D G^{2}-3 E D G H-(3-3 I) E D G\\ & {}+3 E^{2} D H-F G^{3}+2 E F G^{2}-E^{2} F G+G^{4}-E G^{3}-E^{2} G^{2}+E^{3} G ) \\ &{}+D \bigl(B^{2} \bigl(2 G J+(-1+I) \bigl(H+(1-I)\bigr)-3 E J\bigr)\\ &{}+B \bigl(E \bigl(4 C J+2 F \bigl(H+(1-I)\bigr)+G H\bigr)-\bigl(H+(1-I)\bigr) \\ &{}\times \bigl(C H+G (F-G)\bigr)-2 C G J-D \bigl(F J+H^{2}-(2-2 I) H-2 E J+4 I \bigr)\\ &{}+E^{2} \bigl(-2 H-(1-I)\bigr) \bigr) \\ &{} -E \bigl(C^{2} J+C \bigl(-I (F-G+E)+F H+F-G+E \bigr)+(E-G)^{2} (-F+G+E) \bigr) \\ &{} +D \bigl(F \bigl(C J+G \bigl(-2 H-(1-I)\bigr)\bigr)+E \bigl(-3 C J+F H-G \bigl(H+(2-2 I)\bigr)\bigr)\\ &{}+H \bigl(2 C H+(1-I) C+G^{2} \bigr)+C G J +F^{2} \bigl(H+(1-I)\bigr)+(2-2 I) E^{2} \bigr)\\ &{}+D^{2} J (F-2 G+E)-D^{2} \bigl(H^{2}+(1-I) H-2 I \bigr) \bigr), \\ p_{3}={}&2 A^{4} J-A^{3} \bigl(J (B+C+4 D)+H (F+3 G)+(2-2 I) (F+G)-2 E \bigl(H+(1-I)\bigr)\bigr) \\ &{}-A^{2} \bigl(B^{2} J+B \bigl(-3 J (C+D)-H (F+2 G)+(-2+2 I) (F+G)\\ &{}+E \bigl(H+(1-I)\bigr)\bigr)+2 C^{2} J \\ &{} +C \bigl(-3 D J-I (F-G+E)+F-2 G H-G+2 E H+E\bigr)+I D (3 F-3 G+2 E)\\ &{}-3 D F+D G H+3 D G - 3 E D H-2 E D+F^{2} G-E F^{2}-2 F G^{2}+E F G\\ &{}+E^{2} F+G^{3}-E^{2} G \bigr)+A (-C \bigl(-B^{2} J+E \bigl(-B H-3 F G+G^{2} \bigr) \\ &{}+ B F H+(1-I) B F+2 B G H-(1-I) B G+3 D^{2} J+D \bigl(H (2 F+G)+(1-I) E\bigr)\\ &{}+2 F G^{2}+E^{2} (F+2 G)-2 G^{3}-E^{3} \bigr) -B^{2} \bigl(D J+(1-I) G\bigr)\\ &{}+C^{2} \bigl(J (D-2 B)+F \bigl(H+(1-I)\bigr)+G \bigl(H-(1-I)\bigr)+(1-I) E\bigr) \\ &{}- B \bigl(3 D^{2} J+D \bigl(2 F-I (2 F+E)+G H+E\bigr)-(E-G) G (-F+G+E) \bigr)\\ &{}+C^{3} J+D \bigl(4 D^{2} J+D \bigl(3 F H+G \bigl(2 H+(3-3 I)\bigr) \\ &{}\times E \bigl(-3 H-(1-I)\bigr)\bigr)-(E-G) (-2 F+G+E) (-F+G+E) \bigr) )\\ &{}+D \bigl(B^{3} (-J)+B^{2} \bigl(2 J (C+D)+H (F+G) \\ & {}+(1-I) F-E \bigl(H-(1-I)\bigr)\bigr)-B \bigl(C^{2} J+C \bigl(3 D J+F \bigl(H+(1-I)\bigr)\\ &{}+(-1+I) G-E \bigl(H-(2-2 I)\bigr)\bigr)-D^{2} J \\ &{}+ D \bigl(F H+G \bigl(H+(2-2 I)\bigr)-E \bigl(H+(2-2 I)\bigr) \bigr)\\ &{}+ \bigl(G^{2}-3 E G+2 E^{2} \bigr) (-F+G+E) \bigr) \\ &{} +C^{2} (D J-E H)+C \bigl(D^{2} J+D \bigl(F \bigl(2 H+(1-I)\bigr)\\ &{}+G \bigl(-H-(1-I)\bigr)+2 E \bigl(H+(1-I)\bigr)\bigr)+E (E-G) (-F+G+E) \bigr) \\ &{} +D \bigl(-2 D^{2} J-D \bigl(2 H (F-G+E)+F-2 G+3 E\bigr)\\ &{}+I D (F-2 G+3 E)+(E-F) (E-G) (-F+G+E) \bigr) \bigr), \\ p_{4}={}&A^{4} \bigl(2 H+(1-I)\bigr)+A^{3} \bigl(B \bigl(-H-(1-I)\bigr)+C \bigl(-3 H-(2-2 I)\bigr)-D (H+I)\\ &{}+D+(E-F)^{2}+G^{2}-E G \bigr) \\ &{}+A^{2} \bigl((1-I) B^{2}+B \bigl(2 C H+D \bigl(3 H+(2-2 I) \bigr)-(E-F) (E-3 G)\bigr)\\ &{}+C^{2}+D \bigl(2 C H-3 F^{2}+2 F G+4 E F-3 E G \bigr) \\ &{}- C \bigl(G (-3 F+G+2 E)+I C\bigr)+E C (E-F)-3 D^{2} \bigl(H+(1-I)\bigr) \bigr)\\ &{}+A \bigl(B^{2} \bigl(-(1-I) C+D \bigl(-H-(1-I) \bigr)+G^{2} \bigr) \\ &{}+ B \bigl(C^{2} \bigl(-\bigl(H-(1-I)\bigr)\bigr)+C \bigl(-2 D \bigl(H+(1-I)\bigr)+G (F-2 G)\\ &{}-E (F+G)+E^{2} \bigr)+D \bigl((2-2 I) D+2 G (F-G) +E (2 F+G)-3 E^{2} \bigr) \bigr)\\ &{}+C^{3} H+C^{2} \bigl(D \bigl(-H-(1-I)\bigr)+G (G-F)+E (F+G)-E^{2} \bigr)\\ &{}+C D \bigl(D \bigl(2 H+(3-3 I)\bigr) - (2 F+E) G+2 E (E-F)+3 G^{2} \bigr)\\ &{}+D^{2} \bigl(D \bigl(H-(1-I)\bigr)+3 F^{2}-4 F G-2 E F+3 E G \bigr) \bigr) \\ &{}+D \bigl((1-I) B^{3}+B^{2} \bigl(C \bigl(H-(1-I)\bigr)+D H-(F+E) G+E (F-E)+G^{2} \bigr)\\ &{}+B \bigl(C^{2} (-H)+C \bigl((2-2 I) D + (F+E) G+E (2 E-F)-G^{2} \bigr)\\ & {}+D \bigl(D \bigl(-2 H-(3-3 I)\bigr)+F G-3 E F-G^{2}+2 E G+E^{2} \bigr) \bigr)\\ &{}+C^{2} \bigl(D H-E^{2} \bigr) - C D \bigl((3 E-G) (G-F)+D \bigl(H+(1-I)\bigr)\bigr)\\ &{}+D^{2} \bigl(D \bigl(H+(2-2 I)\bigr)-(F-G)^{2}+E G-E^{2} \bigr) \bigr), \\ p_{5}={}&(A-B+C-D) \bigl(A^{2}-A C+D (B-D) \bigr) \bigl(A (-F-G+2 E)\\ &{}+B G-E C+D (F-2 G+E)\bigr) , \\ p_{6}={}&(A-B+C-D) \bigl(A^{2}-A C+D (B-D) \bigr)^{2},\\ \operatorname{Dis}(P_{1})={}&{-}27 A^{2} J^{2}-54 A B J^{2}-54 A C J^{2}-54 A D J^{2}+18 A F H J+18 A F J I-18 A F J\\ &{}+18 A G H J+18 A G J I-18 A G J-4 A H^{3} -12 A H^{2} I+12 A H^{2}+18 A H J E\\ &{}-12 A H I^{2}+24 A H I-12 A H+18 A J EI-18 A J E-4 A I^{3}+12 A I^{2} \\ &{}-12 A I+4 A-27 B^{2} J^{2}-54 B C J^{2}-54 B D J^{2}+18 B F H J+18 B F J I\\ &{}-18 B F J+18 B G H J+18 B G J I-18 B G J -4 B H^{3}-12 B H^{2} I+12 B H^{2}\\ &{}+18 B H J E-12 B HI^{2}+24 B H I-12 B H+18 B J E I-18 B J E-4 B I^{3}+12 B I^{2} \\ &{}-12 B I+4 B-27 C^{2} J^{2}-54 C D J^{2}+18 C F H J+18 C F J I-18 C F J\\ &{}+18 C G H J+18 C G J I-18 C G J-4 C H^{3} -12 C H^{2} I+12 CH^{2}+18 C H J E\\ &{}-12 C H I^{2}+24 C H I-12 C H+18 C J E I-18 C J E-4 C I^{3}+12 C I^{2}-12 C I+4 C \\ &{}-27 D^{2} J^{2}+18 D F H J+18 D F J I-18 D F J+18 D G H J+18 D G J I-18 D G J\\ &{}-4 D H^{3}-12 D H^{2}I+12 D H^{2} +18 D H J E-12 D H I^{2}+24 D H I-12 D H\\ &{}+18 D J E I-18 D J E-4 D I^{3}+12 D I^{2}-12 D I+4 D-4 F^{3} J-12 F^{2} G J \\ &{}+F^{2} H^{2}+2 F^{2} H I-2 F^{2} H-12 F^{2} J E+F^{2} I^{2}-2 F^{2} I+F^{2}-12 F G^{2}J+2 F G H^{2}\\ &{}+4 F G H I-4 F G H-24 F G J E +2 F G I^{2}-4 F G I+2 F G+2 F H^{2} E\\ &{}+4 F H E I-4 F H E-12 F J E^{2}+2 F E I^{2}-4 F E I+2 F E-4 G^{3} J+G^{2} H^{2} \\ &{}+2 G^{2} H I-2 G^{2} H-12 G^{2} J E+G^{2}I^{2}-2 G^{2} I+G^{2}+2 G H^{2} E+4 G H E I\\ &{}-4 G H E-12 G J E^{2}+2 G E I^{2}-4 G E I +2 G E+H^{2} E^{2}+2 H E^{2} I-2 H E^{2}\\ &{}-4 J E^{3}+E^{2} I^{2}-2 E^{2} I+E^{2},\\ P_{3}(y)={}&y^{4} (B-C) (B G-E C)+y^{3} (B-C) \bigl(B (I-1)+F G-F E-G^{2}-H C+E^{2} \bigr)\\ &{}+y^{2} \bigl(B^{2} J+E \bigl(B H+2 F G-G^{2} \bigr) + C (-2 B J+F H-F I+F+G I-G-E I+E)\\ &{}-B F H+B F I-B F-B G H-FG^{2}-E^{2} (F+G)+G^{3}+J C^{2}+E^{3} \bigr) \\ &{}+y \bigl(B \bigl(-2 G J-H I+H+2 J E+I^{2}-2 I+1 \bigr)+F G H-F G I+F G\\ &{}-E (F H-F I+F+2 J C) +G^{2} H +G^{2} I-G^{2}+2 G J C+H^{2} C\\ &{}+E^{2} (H-I+1)-H I C+H C \bigr) +G \bigl(G J+H^{2}-H I+H \bigr)\\ &{}+E \bigl(-2 G J-H I+H+I^{2}-2 I+1 \bigr)+J E^{2},\\ P_{4}(y)={}&y^{2} \bigl(C H-F G+G^{2} \bigr)+y \bigl(CJ+F (H-I+1)+G (-H+I-1)\bigr)\\ &{}+G J+H^{2}-H I+H, \\ P_{5}(y)={}&y^{2} \bigl(B (I-1)+E (E-F)\bigr)+y \bigl(B J+E (H-I+1)+F (-H+I-1)\bigr)\\ &{}+E J-H I+H+I^{2}-2 I+1. \end{aligned}$$