In this section, we prove the main results in this paper, i.e., the existence and nonexistence of entire k-convex radial solutions for problem (1.1).
Proof of Theorem 1.1
From the system
$$ \textstyle\begin{cases} C_{N-1}^{k-1}u^{\prime \prime }(r) (u^{\prime }(r) )^{k-1}r + (C_{N}^{k}\mu r+C_{N-1}^{k} ) (u^{\prime }(r) )^{k} =\frac{r^{k}p(r)}{(1+\mu r)^{k-1}} f_{1}(u(r))f_{2}(v(r)), \\ C_{N-1}^{l-1}v^{\prime \prime }(r) (v^{\prime }(r) )^{l-1}r + (C_{N}^{l}\nu r+C_{N-1}^{l} ) (v^{\prime }(r) )^{l} =\frac{r^{l}q(r)}{(1+\nu r)^{l-1}} g_{1}(u(r))g_{2}(v(r)), \end{cases} $$
we get
$$ \textstyle\begin{cases} u^{\prime }(r)= (\frac{k}{C_{N-1}^{k-1}}\mathrm{e}^{-\psi _{k,\mu }(r)} \int _{0}^{r} \mathrm{e}^{\psi _{k,\mu }(s)} \frac{s^{k-1}p(s)}{(1+\mu s)^{k-1}} f_{1}(u(s))f_{2}(v(s)) \,d s )^{ \frac{1}{k}}, \\ v^{\prime }(r)= (\frac{l}{C_{N-1}^{l-1}}\mathrm{e}^{-\psi _{l,\nu }(r)} \int _{0}^{r} \mathrm{e}^{\psi _{l,\nu }(s)} \frac{s^{l-1}q(s)}{(1+\nu s)^{l-1}} g_{1}(u(s))g_{2}(v(s)) \,d s )^{ \frac{1}{k}}, \end{cases} $$
furthermore we have
$$ \textstyle\begin{cases} u(r)=\int _{1}^{r} (\frac{k}{C_{N-1}^{k-1}}\mathrm{e}^{-\psi _{k, \mu }(t)} \int _{0}^{t} \mathrm{e}^{\psi _{k,\mu }(s)} \frac{s^{k-1}p(s)}{(1+\mu s)^{k-1}} f_{1}(u(s))f_{2}(v(s)) \,d s )^{ \frac{1}{k}}\,dt, \\ v(r)=\int _{1}^{r} (\frac{l}{C_{N-1}^{l-1}}\mathrm{e}^{-\psi _{l, \nu }(t)} \int _{0}^{t} \mathrm{e}^{\psi _{l,\nu }(s)} \frac{s^{l-1}q(s)}{(1+\nu s)^{l-1}} g_{1}(u(s))g_{2}(v(s)) \,d s )^{ \frac{1}{k}}\,dt. \end{cases} $$
Define
$$ \mathcal{L}(u,v) (r)= \begin{pmatrix} \int _{1}^{r} (\frac{k}{C_{N-1}^{k-1}}\mathrm{e}^{-\psi _{k,\mu }(t)} \int _{0}^{t} \mathrm{e}^{\psi _{k,\mu }(s)} \frac{s^{k-1}p(s)}{(1+\mu s)^{k-1}} f_{1}(u(s))f_{2}(v(s)) \,d s )^{ \frac{1}{k}}\,dt \\ \int _{1}^{r} (\frac{l}{C_{N-1}^{l-1}}\mathrm{e}^{-\psi _{l,\nu }(t)} \int _{0}^{t} \mathrm{e}^{\psi _{l,\nu }(s)} \frac{s^{l-1}q(s)}{(1+\nu s)^{l-1}} g_{1}(u(s))g_{2}(v(s)) \,d s )^{ \frac{1}{k}}\,dt \end{pmatrix}^{T}, $$
then we need only to find a fixed point of \(\mathcal{L}\). Here we use the monotone iterative method to find such a fixed point.
It is easy to show that \(\mathcal{L}\) is a mapping from \(C^{2}[0,1]\times C^{2}[0,1]\) to \(C^{2}[0,1]\times C^{2}[0,1]\), and it is continuous on \(C[0,1]\times C[0,1]\).
Let \(\{u_{n}\}\) and \(\{v_{n}\}\) be the sequence of continuous functions defined by
$$ \textstyle\begin{cases} u_{0}(r)=0, \\ v_{0}(r)=0, \\ u_{n}(r)=\int _{1}^{r} (\frac{k}{C_{N-1}^{k-1}}\mathrm{e}^{-\psi _{k, \mu }(t)} \int _{0}^{t} \mathrm{e}^{\psi _{k,\mu }(s)} \frac{s^{k-1}p(s)}{(1+\mu s)^{k-1}} f_{1}(u_{n-1}(s))f_{2}(v_{n-1}(s)) \,d s )^{\frac{1}{k}}\,dt, \\ v_{n}(r)=\int _{1}^{r} (\frac{l}{C_{N-1}^{l-1}}\mathrm{e}^{-\psi _{l, \nu }(t)} \int _{0}^{t} \mathrm{e}^{\psi _{l,\nu }(s)} \frac{s^{l-1}q(s)}{(1+\nu s)^{l-1}} g_{1}(u_{n-1}(s))g_{2}(v_{n-1}(s)) \,d s )^{\frac{1}{k}}\,dt. \end{cases} $$
It is easy to see that \(u_{n}\) and \(v_{n}\) are decreasing on \([0,1]\) for \(n>1\) and by induction \(\{u_{n}\}\) and \(\{v_{n}\}\) are decreasing as well, i.e., \(u_{n+1}(r)< u_{n}(r)\) and \(v_{n+1}(r)< v_{n}(r)\) for \(0\leq r<1\) and \(n\ge 1\).
By condition (H1), for each \(0< r<1\) and \(n>1\),
$$\begin{aligned} 0 < &u_{n}^{\prime }(r) \\ =& \biggl(\frac{k}{C_{N-1}^{k-1}}\mathrm{e}^{-\psi _{k,\mu }(r)} \int _{0}^{r} \mathrm{e}^{\psi _{k,\mu }(s)} \frac{s^{k-1}p(s)}{(1+\mu s)^{k-1}} f_{1} \bigl(u_{n-1}(s) \bigr)f_{2} \bigl(v_{n-1}(s) \bigr) \,d s \biggr)^{\frac{1}{k}} \\ \le &C(N,k,p) \bigl(f_{1} \bigl(u_{n}(r) \bigr)f_{2} \bigl(v_{n}(r) \bigr) \bigr)^{ \frac{1}{k}} \\ \le &C(N,k,p) \bigl(f_{1} \bigl(u_{n}(r)+v_{n}(r) \bigr)f_{2} \bigl(u_{n}(r)+v_{n}(r) \bigr) \,d s \bigr)^{\frac{1}{k}}, \end{aligned}$$
where \(C(N,k,p)\) is a constant dependent on N, k, and p.
Similarly,
$$ 0< v_{n}^{\prime }(r)\le C(N,l,q) \bigl(g_{1} \bigl(u_{n}(r)+v_{n}(r) \bigr)g_{2} \bigl(u_{n}(r)+v_{n}(r) \bigr) \bigr)^{\frac{1}{l}} $$
and further
$$\begin{aligned} \begin{aligned} 0&< \bigl(u_{n}(r)+v_{n}(r) \bigr)^{\prime } \\ &\le C(N,k,l,p,q) \bigl( \bigl(f_{1} \bigl(u_{n}(r)+v_{n}(r) \bigr)f_{2} \bigl(u_{n}(r)+v_{n}(r) \bigr) \bigr)^{\frac{1}{k}} \\ & \quad{} + \bigl(g_{1} \bigl(u_{n}(r)+v_{n}(r) \bigr)g_{2} \bigl(u_{n}(r)+v_{n}(r) \bigr) \bigr)^{\frac{1}{l}} \bigr), \end{aligned} \end{aligned}$$
(3.1)
i.e.,
$$\begin{aligned} 0 < &\frac{ (u_{n}(r)+v_{n}(r) )^{\prime }}{ (f_{1}(u_{n}(r)+v_{n}(r))f_{2}(u_{n}(r)+v_{n}(r)) )^{\frac{1}{k}} + (g_{1}(u_{n}(r)+v_{n}(r))g_{2}(u_{n}(r)+v_{n}(r)) )^{\frac{1}{l}}} \\ \le & C(N,k,l,p,q), \end{aligned}$$
where \(C(N,l,q)\) and \(C(N,k,l,p,q)\) are constants dependent on N, l, q and N, k, l, p, q, respectively.
Integrating from 1 to r, we have
$$ \int _{0}^{u_{n}(r)+v_{n}(r)} \frac{d\tau }{ (f_{1}(\tau )f_{2}(\tau ) )^{\frac{1}{k}} + (g_{1}(\tau )g_{2}(\tau ) )^{\frac{1}{l}}}\ge -C(N,k,l,p,q). $$
(3.2)
By condition (H2), denote
$$ F(w)= \int _{0}^{w} \frac{d\tau }{ (f_{1}(\tau )f_{2}(\tau ) )^{\frac{1}{k}} + (g_{1}(\tau )g_{2}(\tau ) )^{\frac{1}{l}}}, $$
then F is continuous and increasing on \((-\infty ,0]\), and it has an inverse function \(F^{-1}\). From (3.2), we have
$$ F^{-1} \bigl(-C(N,k,l,p,q) \bigr)\le u_{n}(r)+v_{(}r) \le 0 $$
for \(0\le r\le 1\) and \(n\ge 1\).
By condition (H1) and (3.1), we have for \(n\ge 1\)
$$\begin{aligned} 0 < & \bigl(u_{n}(r)+v_{n}(r) \bigr)^{\prime } \\ \le &C(N,k,l,p,q) \Bigl(\max_{F^{-1}(-C(N,k,l,p,q))\le w \le 0} \bigl( \bigl(f_{1}(w)f_{2}(w) \bigr)^{\frac{1}{k}} + \bigl(g_{1}(w)g_{2}(w) \bigr)^{\frac{1}{l}} \bigr) \Bigr) \\ =&C(N,k,l,p,q, f_{1},f_{2},g_{1},g_{2}), \end{aligned}$$
where \(C(N,k,l,p,q, f_{1},f_{2},g_{1},g_{2})\) is a constant dependent on N, k, l, p, q, \(f_{1}\), \(f_{2}\), \(g_{1}\), and \(g_{2}\). So \(\{u_{n}\}\) and \(\{v_{n}\}\) are bounded in \(C^{1}[0,1]\) and by Arzela–Ascoli theorem \(\{u_{n}\}\) and \(\{v_{n}\}\) have convergent subsequences (still denoted by \(\{u_{n}\}\) and \(\{v_{n}\}\)) in \(C[0,1]\). Denote
$$\begin{aligned}& u(r)=\lim_{n\rightarrow +\infty }u_{n}(r), \\& v(r)=\lim_{n\rightarrow +\infty }v_{n}(r). \end{aligned}$$
By the continuity of \(\mathcal{L}\) on \(C[0,1]\times C[0,1]\), from
$$ (u_{n},v_{n})=\mathcal{L}(u_{n-1},v_{n-1}), $$
we conclude that \((u,v)\) is a fixed point of \(\mathcal{L}\) after letting \(n\rightarrow +\infty \). □
Proof of Theorem 1.2
We prove by contradiction. Suppose that \((u,v)\) is a k-convex radial solution to problem (1.1). Then u and v are decreasing on \([0,1]\). For \(0< r< 1\), by Lemma 2.2 we can get
$$\begin{aligned}& 0< u^{\prime }(r)\le C(N,k,p) \bigl(f_{1} \bigl(u(r)+v(r) \bigr)g_{2} \bigl(u(r)+v(r) \bigr) \bigr)^{\frac{1}{k}}, \\& 0< v^{\prime }(r)\le C(N,l,q) \bigl(g_{1} \bigl(u(r)+v(r) \bigr)g_{2} \bigl(u(r)+v(r) \bigr) \bigr)^{\frac{1}{l}}. \end{aligned}$$
So
$$\begin{aligned} 0 < &\frac{ (u(r)+v(r) )^{\prime }}{ (f_{1}(u(r)+v(r))f_{2}(u(r)+v(r)) )^{\frac{1}{k}} + (g_{1}(u(r)+v(r))g_{2}(u(r)+v(r)) )^{\frac{1}{l}}} \\ \le & C(N,k,l,p,q). \end{aligned}$$
Integrating for 0 to 1, we have
$$ 0< \int ^{0}_{u(0)+v(0)} \frac{d\tau }{ (f_{1}(\tau )f_{2}(\tau ) )^{\frac{1}{k}} + (g_{1}(\tau )g_{2}(\tau ) )^{\frac{1}{l}}}\le C(N,k,l,p,q), $$
which contradicts condition (H3). Now we finish the proof. □
At the end of this section, we give some examples for the sake of clearly understanding the results in this paper.
Assume that α, β, \(\alpha _{1}\), \(\beta _{1}\), \(\alpha _{2}\), and \(\beta _{2}\) are positive.
Example 3.1
If \(\alpha _{1} +\beta _{1}\le k\) and \(\alpha _{2} +\beta _{2}\le l\), then the following problem admits an entire k-convex radial solution \((u,v) \in C^{2} (\overline{B_{1}(0)} )\times C^{2} ( \overline{B_{1}(0)} )\):
$$ \textstyle\begin{cases} \sigma _{k} (\lambda (D^{2} u+\mu \vert \nabla u \vert I ) )=(1+ \vert x \vert )^{\alpha } (1+ \vert u \vert )^{\alpha _{1}} \vert v \vert ^{\beta _{1}}, & x \in B_{1}(0), \\ \sigma _{l} (\lambda (D^{2} v+\nu \vert \nabla v \vert I ) )=(1+ \vert x \vert )^{\beta } (1+ \vert u \vert )^{\alpha _{2}}(1+ \vert v \vert )^{\beta _{2}}, & x \in B_{1}(0), \\ u=v=0,& x\in \partial B_{1}(0). \end{cases} $$
Example 3.2
If \(\alpha _{1} +\beta _{1}\ge k\) and \(\alpha _{2} +\beta _{2}\ge l\), then the following problem admits no entire k-convex radial solution \((u,v) \in C^{2} (\overline{B_{1}(0)} )\times C^{2} ( \overline{B_{1}(0)} )\):
$$ \textstyle\begin{cases} \sigma _{k} (\lambda (D^{2} u+\mu \vert \nabla u \vert I ) )=(1+ \vert x \vert )^{\alpha } \vert u \vert ^{\alpha _{1}} \vert v \vert ^{\beta _{1}}, & x \in B_{1}(0), \\ \sigma _{l} (\lambda (D^{2} v+\nu \vert \nabla v \vert I ) )=(1+ \vert x \vert )^{\beta } \vert u \vert ^{\alpha _{2}} \vert v \vert ^{\beta _{2}}, & x \in B_{1}(0), \\ u=v=0,& x\in \partial B_{1}(0). \end{cases} $$