Definition 1. (see [35, 36]). The fractional-order derivatives in the meaning of Caputo are defined as follows:

Some results for the fractional-order derivatives are found in References [21, 25, 37] that are needed throughout this paper.

Lemma 1. Assume that f(t) ∈ C[a, b].

• (1)

  $$

•  

  where a ≤ ζ ≤ sfor alls ∈ (a, b]

• (2)

  If $$, then we have:

  • (i)

    If $$for allt ∈ (a, b), thenf(t)is a nondecreasing function for all t ∈ [a, b].

  • (ii)

    If $$for allt ∈ (a, b) then f(t)is a nonincreasing function for allt ∈ [a, b].

Lemma 2. Assume the Cauchy problem.

In the autonomous case:

Then, the solution is h(t) = bE_q[λ(t − a)^q] and E_q is the Mittag-Leffler function.

Lemma 3 (see [21].)Let w(t) be a continuous function on [t₀, ∞) and satisfies

$$, then the form of the solution of this equation is given by:

The next lemma is found in Reference [38] which gives the uniqueness of the solution of fractional-order system.

Lemma 4. Let $$ be a system fractional-q-order derivatives with the initial condition $$. Then, the above system has a unique solution on [t₀, ∞) × Ω if f(t, u) satisfies the locally Lipchitz condition concerning u.

To examine the stability of the fractional-order system, we need the following.

Definition 2. (see [39]). Let f(x) = xⁿ + v_n−1xⁿ⁻¹ + v_n−2xⁿ⁻² + ……+v₀ be a polynomial of degree n, then the discriminant of f(x) denoted by D(f) is defined by D(f) = (−1)^{n(n − 1)/2}R(f, f^′), where f^′ is the derivative of f and g(x) = xⁿ + k_n−1x^l−1 + k_n−2x^l−2 + ……+k₀ and R(f, g) are (n + l) ⊗ (n + l) determinants.

Lemma 5 (see [39].)Consider a characteristic polynomial equation, f(λ) = λⁿ + v_n−1λⁿ⁻¹ + v_n−2λⁿ⁻² + …….. + v₀, then:

• (1)

  If n  = 2, then the conditions for |argλ_i| > qπ/2, for i = 1,2 q ∈ (0,1) are either Routh–Hurwitz conditions or $$ and

  $$ (7)

• (2)

  If n  = 3, then we have the following cases:

  • (i)

    $$

  • (ii)

    If D(f) < 0, v₂ ≥ 0, v₁ ≥ 0, v₀ > 0, then|argλ_i| > qπ/2 i = 1, 2, 3 for q < 2/3.

Theorem 1. For the fractional-order system (13), we have the following:

• (1)

  All solutions that start in F₊ = {(x₁, x₂, x₃) ∈ ℝ³|x₁ ≥ 0, x₂ ≥ 0, x₃ ≥ 0} are non-negative solutions.

• (2)

  If 1 + b > rd₁/d₂, then all solutions that start in F₊ are bounded.

Proof.

• (1)

  To prove the solution of the fractional-order system (13) is nonnegative, we start with x₁(0) > 0fort = 0 assuming that x₁(t) ≥ 0, fort ≥ 0 is not true; therefore, there exists t_∗ > 0 such that

  $$ (14)

•  

  So, from the first equation of the system (13), we obtain the following:

  $$ (15)

•  

  According to part (1) of Lemma 1, we get $$ and this is contradiction because $$.

•  

  Therefore, x₁(t) ≥ 0, for allt ≥ 0. In the same manner, we can prove that x₂(t) ≥ 0, x₃(t) ≥ 0, for allt ≥ 0.

• (2)

  Let F₊ = {(x₁, x₂, x₃) ∈ ℝ³|x₁ ≥ 0, x₂ ≥ 0, x₃ ≥ 0}, and x₁, x₂, and x₃ in ℝ³, define a function V_t = x₁(t) + r/d₂x₂(t) + ra₁/d₂a₂ra₁/d₂a₂x₃(t), it follows that

  $$ (16)

•  

  Now, for each k> 0, we have

  $$ (17)

•  

  If we choose k < min{1 + b − rd₁/d₂, h, d₃}, then we get D^qV(t) + kV(t) ≤ ε for some ε > 0.

Then, from Lemma 3, we obtain that V(t) ≤ (V(0) − (ε/k))E_q[−k(t − 0)^q] + v/K. Therefore, we have 0 ≤ V(t) ≤ ε/k as t⟶∞ and all solutions of system (13) are bounded.

The next theorem gives the existence and uniqueness of the solution system (13).

Theorem 2. The existence and uniqueness solution of system (4) are given for each non-negative initial condition:

Proof. Let the region be defined by F × (0, T], T < ∞.

Where $$.

We assume that $$ = $$ and B(X) = (B₁(X), B₂(X), B₃(X)) be a mapping, such that

For $$, one can get the following:

Since $$, we have.

Where L = max{1 + b + d₁, (r + d₂ + h + a₁ + a₂)(M(1 + M)), (d₃ + (a₁ + a₂)M(1 + M))}. Therefore, B(X) satisfies the Lipchitz condition, so that the system (4) has a unique solution.

Theorem 3 (see [41], [42].)Consider the following fractional-order differential system:

$$, q ∈ (0,1), and $$. The point $$ that satisfies $$ is called the equilibrium point of the system (20). It is called locally asymptotically stable if |arg(λ_i)| > qπ/2 for i = 1,2, …..n, where λ_i are the eigenvalues of the Jacobian matrix which are evaluated at $$. Otherwise, it is called an unstable point.

To find all possible equilibrium points of the system (4), we have to solve the following equations:

Therefore, the system (4) has three possible equilibrium points, namely:

• (1)

  The trivial equilibrium point e₀ = (0,0,0) always exists.

• (2)

  The free predator point $$ exists if

  $$ (22)

• (3)

  The interior $$ exists only if

  $$ (23)

•  

  where $$ = ($$ is the positive root of the following equation:

  $$ (24)

•  

  Here, a = a₁d₃ − a₁a₂ + a₂(d1r/1 + b–d₂ − h)/a₁d₃ and b₁ = a₂(d1r/1 + b − d₂ − h)/a₁d₃.

Since b₁ > 0ifa < 0 then, equation (24) may have two positive real roots.

The Jacobian matrix of fractional-order system (4) associated with arbitrary fixed point (x₁, x₂, x₃) is given by

So, the general characteristic equation of (25) is as follows:

To analyse the local stability of the fixed points e₀, e₁, e₂, we give the following theorem.

Theorem 4. For the system (4), we have the following:

• (1)

  If d₂ + h + bd₂ + bh > rd₁, then the point e₀ is a locally stable point.

• (2)

  The free predator point e₁ is always an unstable point.

• (3)

  The interior equilibrium pointe₂ is locally asymptotically stable if one of the following conditions hold.

  • (i)

    If D(f) > 0, p₂ > 0, p₀ > 0 and p₁p₂ > p₀for q ∈ (0,1).

  • (ii)

    If D(f) < 0, p₂ ≥ 0, p₁ ≥ 0, p₀ > 0 for q < 2/3, where p₂, p₁, and p₀ are defined in equation(29).

Proof.

• (1)

  For the point e₀, the Jacobian matrix J(e₀) is given as follows:

  $$ (29)

•  

  So, the characteristic equation is P(λ) = (−d₃ − λ)(λ² + u₁λ + u₀).

•  

  Where u₁ = (1 + b + d₂ + h) and u₀ = (d₂ + h + bd₂ + bh − rd₁). The roots of the above characteristic equation are λ₁ = (−d₃). It is clear that |argλ₁| = π > qπ/2.

•  

  Since u₁ > 0 and if d₂ + h + bd₂ + bh > rd₁ that means u₀ > 0, then the Routh–Hurwitz conditions are satisfied.

•  

  According to part 1 of Lemma 5, e₀ is stable.

• (2)

  It is clear that the Jacobian matrix J(e₁) is as follows:

  $$ (30)

•  

  And the corresponding characteristic equation is as follows:

  $$ (31)

•  

  Therefore, one of the eigenvalues is equal to zero. Thus, the free predator pointe₁ is unstable.

• (3)

  The Jacobian matrix at the interior equilibrium pointe₂ is given as follows:

  $$ (32)

•  

  So, the characteristic polynomial of J(e₂) can be obtained:

  $$ (33)

•  

  where

  $$ (34)

•  

  Then, the discriminant D(f) of the cubic polynomial f(λ) is the following:

  $$ (35)

According to Lemma 5 (2), if D(f) > 0, p₂ > 0, p₀ > 0 and p₁p₂ > p₀for q ∈ (0,1), or D(f) < 0, p₂ ≥ 0, p₁ ≥ 0, p₀ > 0 for q < 2/3.

Then, the interior fixed point e₂ is locally asymptotically stable.