Lemma 1. Let $$ and $$, and A_i is an N function; then, these assertions are equivalent as follows:

Lemma 2. If A_i is an N function satisfies (22), then we have

Lemma 3. Let $$ be the complement of A_i and $$, $$, t ≥ 0 where $$ and $$. If A_i is an N function and (22) holds, then $$ satisfies

Lemma 4. We have $$, i.e., $$.

Remark 1. By Lemma 1, Lemma 3, and (22), we show that $$.

We now look at the definition of the fractional Orlicz–Sobolev spaces $$ (see[1]), and we set some properties on these spaces. The fractional Orlicz–Sobolev spaces $$ defined as follows:

To deal with the problem under consideration, we choose

In these spaces, the generalized Poincaré inequality reads as follows (see [19]):

Theorem 1 (see [19].)Let A_i be an N function s ∈ (0,1), Ω ⊂ ℝ^N, and C^0,1− regularity with bounded boundary. If (H₀), (H_∞), and (2.5) hold, then the embedding $$ is continuous, and the embedding $$ is compact for any N function $$.

Remark 2. The assumption (ϕ₄) implies that there is η_i ∈ (1, max{r₁, r₂}] such that $$; then, by Lemma 4 and Theorem 1, the following embedding $$ are compact. i.e., there exists a constant $$ such that

Now, we defined our working space $$ under the norm

So, by the above arguments, (W, ‖(u, v)‖) is a separable and reflexive Banach space.

Remark 3. By (13) and the fact that

The fractional a_i(.)-Laplacian operator specified in (2) is indeed defined between $$ and its dual space $$. Indeed, in [1], Theorem 6.12, the following expression is found:

Lemma 5. The function ℐ is well defined, and it is the C¹(W, ℝ), and we have

Proof. First, we can see that

On the other hand, if u_n⟶u in $$, then $$ in $$, so by Dominated convergence theorem, there exists a subsequence $$ and a function h in $$ such that

And

Again by dominated convengeance theorem, we obtain that

Combining (43) in (40), we get $$ is continuous. Now, we turn to prove that

By (32) and (36), we have

Then, ℱ is well defined in W. Now, by (13) and (32) and the similar argument in [[20] Lemma 3.1,] we see that (44) is holds. This completes the proof.

To find the critical points of ℐ, we will minimize the energy functional ℐ on the constraint of Nehari manifold

The main result of this section is as follows:

Theorem 2. We assume that (ϕ₁)-(ϕ₄) and (F₁)-(F₃) hold true. Then, the system (1) has a nontrivial ground state solution (u, v) in the way that there is $$ such that

Lemma 6 (see [16] Lemma 4.1.)The following properties are true:

Lemma 7 (see [21].)We suppose that $$ is convex, u_k⇀u in $$ and $$. Then, $$. Similary, we can obtain that, v_k⟶v in $$. Therefore, (u_k, v_k)⟶(u, v) in W.

Lemma 8. We suppose that (ϕ₁)-(ϕ₄), (22), and (F₁) hold true. Then, we have

Proof. Let {(u_k, v_k)} be sequence in W such that (u_k, v_k)⇀(u, v). By (13) and Hölder inequality, we have

Since (u_k, v_k)⇀(u, v) in W, it is clear that u_k⇀u in $$ and v_k⇀v in $$, then $$ and $$ are bounded. By Remark 2, there exists G > 0 such that

Combining (50) and (51), then we have

Hence, the compact embedding $$ (see Remark 2) make sure that u_k⟶u in $$; then, for any positive ϵ, we can choose n₀ ∈ ℕ such that

Furthermore,

Lemma 9. We assume that (ϕ₁)-(ϕ₃), (F₁)-(F₃). Then, there exists a constant C > 0 such that ‖u, v‖ ≥ C for each $$.

Proof. The proof is augmented by contradiction. We suppose that there exists a subsequence denoted by $$ such that ‖(u_k, v_k)‖ ≤ 1/k, for each integer k ≥ 1. Using (ϕ₃)', the construction of $$ (see (46)), (13), the Hölder inequality, and (32), we have

According to Lemma 1, Remark 2, and for k large enough, last inequality rewrites

Dividing the last expression by $$, we get to

The fact that n_i < r₁, r₂, r₁(r₂ − 1)/r₂ + 1, r₂(r₁ − 1)/r₁ + 1, we get that for a large k,

Lemma 10. We assume that (ϕ₁)-(ϕ₃) and (F₁)-(F₃) hold true. Then, the function $$ given by

Proof. We set

So, by definition of a_i, we have h ∈ C¹ and

Furthermore, there is θ ∈ ℝ with 0 < θ < t ≤ 1 such that

Thus,

We recall that by (ϕ₃)':

Using (ϕ₁)-(ϕ₃) and 0 ≤ θ ≤ 1, we have

Indeed, using Young’s inequality, (15) and the fact $$, we infer that

Hence,

Now, let us see that $$ is continuous. Let $$ be a sequence such that u_k⟶u.

Then,

By arguments as above,

Applying the Lebesgue dominated convergence theorem,

Hence, we have $$ which completes the proof of Lemma 10.

At this step, in view of determining the behavior of ℐ on $$, we introduce the fibering map Υ_u,v : ℝ⁺⟶ℝ associated to the Nehari manifold given by

Using (38), then the first derivative of the map Υ_u,v is given by

Lemma 11. We suppose that (ϕ₁)-(ϕ₃) and (F₁)-(F₃) hold. Then,

Proof. By equation (36), there exists c₃, c₄ > 0 such that

Using Lemma 2.3 in [10] and (23), we infer that

With the above arguments, we have for all (u, v) ∈ W/{0,0},

We note that n_i=1,2 < min{r₁, r₂}, so the last inequality (79) rewrites as follows:

Hence,

Now, for item (ii), by using (23), we have

Moreover, we can see that

In fact from the condition (F₂) and (83), we infer that

Combining (84) in (82), we proved item (ii). Now, for (iii), we first show that, by equation (13), there exists c₁, c₂ > 0 such that

Now, using $$ and (85) in (78), we have

In fact from Lemma 2.3 in [10] and (23), we infer that

Then,

Since n_i=1,2 < min{r₁, r₂, 1 + r₁(r₂ − 1)/r₂, 1 + r₂(r₁ − 1)/r₁}, we conclude the result for item (iii). For the last item, we have

Using (35), we have

We can easily see that

As a consequence of (F₃), we infer that

It follows using (91) and (92) and inequalities (90) that

Using F₂, we conclude that

At this point, last limit just above and (89), we infer the item (iv).

Lemma 12. We assume that (ϕ₁)-(ϕ₃) and (F₁)-(F₃) hold true; then, for each (u, v) ∈ W/{(0,0)}, there is an only t^∗ = t(u, v) > 0 such that $$. Moreover, ℐ((u, v)) > 0 for each $$.

Proof. On the one hand, let (u, v) ∈ W/{(0,0)}. By Lemma 11, we have $$ for t small enough and $$ for t large enough. On the other hand, the map $$ is continuous, and there is at least one number t ∈ (0, ∞) such that $$, which means that $$. Now, let us see that there is an only t^∗ = t(u, v) > 0 such that $$. Using (78), we have

Indeed,

We remark that (ϕ₃) implies that

Using inequality (98) in (97), we infer that

Hence,

It can be seen that

According to (F₃) and (101), we get

Therefore by (102), we have that $$ is a decreasing function that vanishes once in (0, ∞). So, there exists a unique t = t(u, v) > 0 such that $$. Therefore, the function Υ_u,v has one critical point, namely t^∗ = t(u, v) > 0 and $$. Furthermore, by Lemma 11, it follows that t^∗ is a maximal point of Υ_u,v on (0, ∞) and, in fact Υ_u,v(t^∗) > 0, implying that ℐ(t^∗u, t^∗v) > 0. The arguments above also show that $$, for every (u, v) ∈ W/{(0,0)}. And finally, because $$ if and only if t^∗ = 1, we conclude that ℐ(u, v) > 0 for every $$. This completes the proof.

Proposition 1. We assume that (ϕ₁)-(ϕ₃) and (F₁)-(F₃) hold true. Then $$ is a C¹-submanifold of W, i.e., any critical point of $$ is a critical point of ℐ.

Proof. We consider the functional J_t : W⟶ℝ defined by

According to Lemma 10, we can see that ℐ_t ∈ C¹, and by using equation (59), we show that

We also show that

Now, we set

By using Lemma 10, it follows that Q ∈ C¹. Furthermore, from Lemma 11, we infer that t = 1 is the global maximum of Υ_u,v, and also in the proof of Lemma 12, we see that

Using the fact that $$ and 0 is a regular value for Q, the set $$ is a C¹-submanifold of W. Now, we assume that $$ is a critical point of $$. According to the theorem of Lagrange multiplier, there exists a real constant λ such that

Using (108), we infer that λ = 0. Therefore, ℐ(u, v) ≡ 0 so that (u, v) is a free critical point of ℐ. This completes the proof.

Lemma 13. We assume that (ϕ₁)-(ϕ₃), (F₁)-(F₃) hold. Let $$ be a minimizing sequence of ℐ over the Nehari manifold $$. Then, (u_k, v_k) is bounded in W.

Proof. Let $$ be a minimizing sequence that is $$ and $$. To prove the boundedness of (u_k, v_k), we argue by contradiction and consider that there exists a subsequence of (u_k, v_k), always denoted by (u_k, v_k) such that ‖(u_k, v_k)‖⟶+∞. We approach the problem in two cases.

Case 1. We suppose that $$ and also $$. Let $$ and $$. Then, $$ is bounded in separable and reflexive Banach space W. By Theorem 1 and Remark 2, there exists a point $$ such that

• (i)

  $$ in $$; and $$ in a.e in Ω

• (ii)

  $$ in $$ and $$ in a.e in Ω

We claim that $$ has nonzero Lebesgue measure. Firstly, we assume by the way of contradiction that both $$ and $$ have zero Lebesgue measure, that is $$ in $$ and $$ in $$. Since $$, then

Let G > 0 be a constant. Now, we observe that

Now using (a), (b), and the continuous of the function F, we infer that

Combining (111), (112), and using Lemma 6, we get

Passing to the limit in (113), we get

On the one hand, employing Lemma 1, we have

Passing to the limit above, we have

On the other hand, applying Fatou’s lemma and the fact that $$ and $$ has nonzero Lebesgue measure, we have

It is clear that $$ when k⟶+∞. Then, by using (F₂) and last inequalities just above, we conclude

Case 1. We suppose that $$ or $$ for some C > 0 and all k ∈ ℕ. Without loss of generality, we assume that $$ and $$, for some C > 0 and for all k ∈ ℕ. Let $$ and $$, then $$ and $$. By Theorem 1 and Remark 2, there exists a point $$ such that

• (i)

  $$ in $$ and $$ in a.e in Ω

• (ii)

  $$ in $$; and $$ in a.e in Ω

By similar argument in Case 1, we prove that $$ has nonzero Lebesgue measure. It is recalled that we are assuming that ‖(u_k, v_k)‖⟶+∞ and $$. Hence,

Applying Lemma 1 and the last equation just above, we get

Passing to the limit to the inequalities just above, we have

In other way, using Fatou’s lemma and the fact that $$, we have

It is clear to see that

According to (F₂), (123), and last inequalities just above, we conclude

Lemma 14. We assume that (ϕ₁)-(ϕ₃), (F₁)-(F₃) hold. Then, there exists $$ such that

Proof. Let $$ be a minimizing sequence of ℐ over the Nehari manifold $$. By Lemma 13, there exists (u, v)W such that

• (i)

  u_k⇀u in $$ and v_k⇀v in $$

• (ii)

  We claim that (u, v) ≠ (0,0). We assume on the contrary that (u, v) = (0,0). The fact that $$ and by using (ϕ₃), we have

  $$ (126)

Applying Lemma 8, we get

As a consequence of Lemma 6, ‖u_k, v_k‖⟶0 which is contradicting Lemma 9. That proves the claim.

Accordingly, from Lemma 10, we have that (u, v) ∈ W ↦ 〈ℐ^′(u, v), (u, v)〉 is weakly lower continuous. Hence,

We recall that $$. By Lemma 12, there is t ∈ [0,1] such that $$. Hence, $$.

• (i)

  We claim that t = 1 so that (u, v) is in $$.

Furthermore, by contrary, we assume that t ∈ (0,1). Then,

Again by using (F₃), we have

Then, we infer that

Moreover, the functions

Now, using the weak lower continuity of the functions