Mathematical Methods in the Applied Sciences
Volume 48, Issue 12 pp. 12309-12319
RESEARCH ARTICLE
Open Access

Existence of Normalized Solutions of a Hartree–Fock System With Mass Subcritical Growth

Hua Jin, 

Hua Jin

School of Mathematics, China University of Mining and Technology, Xuzhou, China

Contribution: Conceptualization, ​Investigation

Search for more papers by this author
Yanyun Chang, 

Yanyun Chang

School of Mathematics, China University of Mining and Technology, Xuzhou, China

Contribution: Conceptualization, ​Investigation

Search for more papers by this author
Marco Squassina, 

Corresponding Author

Marco Squassina

Dipartimento di Matematica e Fisica, Università Cattolica del Sacro Cuore, Brescia, Italy

Correspondence:

Marco Squassina ([email protected])

Contribution: Conceptualization, ​Investigation

Search for more papers by this author
Jianjun Zhang, 

Jianjun Zhang

College of Mathematica and Statistics, Chongqing Jiaotong University, Chongqing, China

Contribution: Conceptualization, ​Investigation

Search for more papers by this author
First published: 02 May 2025
https://doi.org/10.1002/mma.11028

Funding: H. Jin and Y. Chang were supported by the Fundamental Research Funds for the Central Universities (2019XKQYMS90). M. Squassina was supported by Gruppo Nazionale per l'Analisi Matematica, la Probabilità e loro Applicazioni. J.J. Zhang was supported by NSFC (Nos. 12371109 and 11871123).

ABSTRACT

In this paper, we are concerned with normalized solutions of a class of Hartree-Fock type systems. By seeking the constrained global minimizers of the corresponding functional, we prove that the existence and nonexistence of normalized solutions. Also, the orbital stability of standing waves is obtained under local well-posedness assumptions of the evolution problem.

1 Introduction

1.1 Background

The Hartree–Fock type system
− Δ x Ψ 1 + α x − 1 ∗ Ψ 1 2 + Ψ 2 2 Ψ 1 = i ∂ t Ψ 1 + Ψ 1 2 q − 2 Ψ 1 + β Ψ 2 q Ψ 1 q − 2 Ψ 1 , − Δ x Ψ 2 + α x − 1 ∗ Ψ 1 2 + Ψ 2 2 Ψ 2 = i ∂ t Ψ 2 + Ψ 2 2 q − 2 Ψ 2 + β Ψ 1 q Ψ 2 q − 2 Ψ 2 , Ψ j = Ψ j ( x , t ) ∈ ℂ , ( x , t ) ∈ ℝ 3 × ℝ , j = 1 , 2 , $$ \left\{\begin{array}{l}-{\Delta}_x{\Psi}_1+\alpha \left[{\left|x\right|}^{-1}\ast \left({\Psi}_1^2+{\Psi}_2^2\right)\right]{\Psi}_1=i{\partial}_t{\Psi}_1\\ {}+{\left|{\Psi}_1\right|}^{2q-2}{\Psi}_1+\beta {\left|{\Psi}_2\right|}^q{\left|{\Psi}_1\right|}^{q-2}{\Psi}_1,\\ {}-{\Delta}_x{\Psi}_2+\alpha \left[{\left|x\right|}^{-1}\ast \left({\Psi}_1^2+{\Psi}_2^2\right)\right]{\Psi}_2=i{\partial}_t{\Psi}_2\\ {}+{\left|{\Psi}_2\right|}^{2q-2}{\Psi}_2+\beta {\left|{\Psi}_1\right|}^q{\left|{\Psi}_2\right|}^{q-2}{\Psi}_2,\\ {}{\Psi}_j={\Psi}_j\left(x,t\right)\in \mathbb{C},\left(x,t\right)\in {\mathbb{R}}^3\times \mathbb{R},j=1,2,\end{array}\right. $$ (1.1)
has received a lot of attention in recent years. For instance, it appears in the basic quantum, chemistry model of the small number of electrons interacting with static nuclear, see [1-3]. and the references therein for details. This system consists of two Schrödinger equations, in which there are Coulomb interaction terms. The constant β ∈ ℝ $$ \beta \in \mathbb{R} $$ describes the interspecies scattering lengths. When β > 0 $$ \beta &gt;0 $$ , it indicates interspecies attraction and β < 0 $$ \beta &lt;0 $$ indicates interspecies repulsion.
Such problem was initially introduced by Hartree in [4] by employing a set of specialized test functions, without explicitly considering the Pauli exclusion principle. Subsequently, Fock in [5] and Slater in [6] addressed the Pauli exclusion principle by selecting a distinct class of test functions known as Slater determinants. By doing so, they derived a system of N $$ N $$ -coupled nonlinear Schrödinger equations:
− ℏ 2 2 m Δ ψ k + V ext ψ k + ∫ ℝ 3 | x − y | − 1 ∑ j = 1 N ψ j ( y ) 2 d y ψ k + V ex ψ k = E k ψ k , $$ {\displaystyle \begin{array}{cc}\hfill &amp; -\frac{\hslash&#x0005E;2}{2m}\Delta {\psi}_k&#x0002B;{V}_{\mathrm{ex}\mathrm{t}}{\psi}_k&#x0002B;\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;x-y\right&#x0007C;}&#x0005E;{-1}\sum \limits_{j&#x0003D;1}&#x0005E;N{\left&#x0007C;{\psi}_j(y)\right&#x0007C;}&#x0005E;2\mathrm{d}y\right){\psi}_k\hfill \\ {}\hfill &amp; &#x0002B;{\left({V}_{\mathrm{ex}}\psi \right)}_k&#x0003D;{E}_k{\psi}_k,\hfill \end{array}} $$ (1.2)
where ψ k : ℝ 3 → ℂ , k = 1 , … , N , V ext $$ {\psi}_k:{\mathbb{R}}&#x0005E;3\to \mathbb{C},k&#x0003D;1,\dots, N,{V}_{\mathrm{ext}} $$ is a given external potential, and
V ex ψ k : = − ∑ j = 1 N ψ j ∫ ℝ 3 ψ k ( y ) ψ ‾ j ( y ) | x − y | d y $$ {\left({V}_{\mathrm{ex}}\psi \right)}_k:&#x0003D; -\sum \limits_{j&#x0003D;1}&#x0005E;N{\psi}_j{\int}_{{\mathbb{R}}&#x0005E;3}\frac{\psi_k(y){\overline{\psi}}_j(y)}{\mid x-y\mid}\mathrm{d}y $$
is the k $$ k $$ -th component of the crucial exchange term and E k $$ {E}_k $$ is the k $$ k $$ -th eigenvalue. For more details about the Hartree–Fock method, we refer to [7-10] and references therein.
In this paper, our main interest is focused on the case of N = 2 $$ N&#x0003D;2 $$ and assume the external potential has the following form:
V ex ψ = − ψ 1 2 q − 2 ψ 1 + β ψ 1 q − 2 ψ 2 q ψ 1 ψ 2 2 q − 2 ψ 2 + β ψ 1 q ψ 2 q − 2 ψ 2 , $$ \left({V}_{\mathrm{ex}}\psi \right)&#x0003D;-\left(\begin{array}{l}{\left&#x0007C;{\psi}_1\right&#x0007C;}&#x0005E;{2q-2}{\psi}_1&#x0002B;\beta {\left&#x0007C;{\psi}_1\right&#x0007C;}&#x0005E;{q-2}{\left&#x0007C;{\psi}_2\right&#x0007C;}&#x0005E;q{\psi}_1\\ {}{\left&#x0007C;{\psi}_2\right&#x0007C;}&#x0005E;{2q-2}{\psi}_2&#x0002B;\beta {\left&#x0007C;{\psi}_1\right&#x0007C;}&#x0005E;q{\left&#x0007C;{\psi}_2\right&#x0007C;}&#x0005E;{q-2}{\psi}_2\end{array}\right), $$
which is consistent with the assumptions in [11]. It leads us to investigate the system (1.1). Since we are mainly interested in the existence of standing wave solutions to (1.1), namely, solutions having the form of
Ψ 1 ( x , t ) = e − i λ 1 t u ( x ) , Ψ 2 ( x , t ) = e − i λ 2 t v ( x ) , λ 1 , λ 2 ∈ ℝ $$ {\Psi}_1\left(x,t\right)&#x0003D;{e}&#x0005E;{-i{\lambda}_1t}u(x),{\Psi}_2\left(x,t\right)&#x0003D;{e}&#x0005E;{-i{\lambda}_2t}v(x),{\lambda}_1,{\lambda}_2\in \mathbb{R} $$ (1.3)
it suffices to consider the following coupled elliptic equations with nonlocal interaction:
− Δ u + α ϕ u , v u = λ 1 u + u 2 q − 2 u + β v q u q − 2 u in ℝ 3 , − Δ v + α ϕ u , v v = λ 2 v + v 2 q − 2 v + β u q v q − 2 v in ℝ 3 , $$ \left\{\begin{array}{ll}-\Delta u&#x0002B;\alpha {\phi}_{u,v}u&#x0003D;{\lambda}_1u&#x0002B;{\left&#x0007C;u\right&#x0007C;}&#x0005E;{2q-2}u&#x0002B;\beta {\left&#x0007C;v\right&#x0007C;}&#x0005E;q{\left&#x0007C;u\right&#x0007C;}&#x0005E;{q-2}u&amp; \mathrm{in}\kern1em {\mathbb{R}}&#x0005E;3,\\ {}-\Delta v&#x0002B;\alpha {\phi}_{u,v}v&#x0003D;{\lambda}_2\kern0.3em v&#x0002B;{\left&#x0007C;v\right&#x0007C;}&#x0005E;{2q-2}v&#x0002B;\beta {\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;{q-2}v&amp; \mathrm{in}\kern1em {\mathbb{R}}&#x0005E;3,\end{array}\right. $$ (1.4)
where
ϕ u , v ( x ) : = ∫ ℝ 3 u 2 ( y ) + v 2 ( y ) | x − y | d y ∈ D 1 , 2 ℝ 3 $$ {\phi}_{u,v}(x):&#x0003D; {\int}_{{\mathbb{R}}&#x0005E;3}\frac{u&#x0005E;2(y)&#x0002B;{v}&#x0005E;2(y)}{\mid x-y\mid}\mathrm{d}y\in {D}&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right) $$
is the unique solution in D 1 , 2 ℝ 3 $$ {D}&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right) $$ of
− Δ ϕ u , v = 4 π u 2 + v 2 in ℝ 3 . $$ -\Delta {\phi}_{u,v}&#x0003D;4\pi \left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\kern2.00em \mathrm{in}\kern1em {\mathbb{R}}&#x0005E;3. $$
System (1.4) is called a Schrödinger–Poisson type system, see [12].
In [11], the authors first studied the system (1.4), where λ 1 , λ 2 ∈ ℝ $$ {\lambda}_1,{\lambda}_2\in \mathbb{R} $$ are fixed parameter. They dealt with the functional
ℐ ( u , v ) = 1 2 | | ∇ u | | 2 2 + 1 2 | | ∇ v | | 2 2 + α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x − 1 2 λ 1 u 2 2 + λ 2 v 2 2 − 1 2 q | | u | | 2 q 2 q + | | v | | 2 q 2 q − β q ∫ ℝ 3 | u | q | v | q d x $$ {\displaystyle \begin{array}{cc}\hfill \mathcal{I}\left(u,v\right)&amp; &#x0003D;\frac{1}{2}{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;\frac{1}{2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\\ {}\hfill &amp; \kern1em -\frac{1}{2}\left({\lambda}_1{\left\Vert u\right\Vert}_2&#x0005E;2&#x0002B;{\lambda}_2{\left\Vert v\right\Vert}_2&#x0005E;2\right)-\frac{1}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; \kern1em -\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x\end{array}} $$
and looked for its critical points in H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ . In that direction, mainly by variational methods, they showed the existence of semitrivial and vectorial ground states solutions depending on the parameters involved. In addition, the authors in [13] considered the least energy solutions of Hartree–Fock systems when the nonlinearities are subcritical. However, nothing can be said a priori on the L 2 $$ {L}&#x0005E;2 $$ -norm of solutions.
In recent years, the study of normalized solutions has attracted considerable attentions; that is, the desired solutions have a priori prescribed L 2 $$ {L}&#x0005E;2 $$ - norm. Let us introduce some related results about the Schrödinger–Poisson equations:
− Δ u + ϕ u u − | u | p − 2 u = ω u in ℝ 3 , $$ -\Delta u&#x0002B;{\phi}_uu-{\left&#x0007C;u\right&#x0007C;}&#x0005E;{p-2}u&#x0003D;\omega u\kern1em \mathrm{in}\kern0.3em {\mathbb{R}}&#x0005E;3, $$
where ϕ u ( x ) = ∫ ℝ 3 | u ( y ) | 2 | x − y | d y $$ {\phi}_u(x)&#x0003D;{\int}_{{\mathbb{R}}&#x0005E;3}\frac{{\left&#x0007C;u(y)\right&#x0007C;}&#x0005E;2}{\mid x-y\mid}\mathrm{d}y $$ satisfies − Δ ϕ u = 4 π u 2 $$ -\Delta {\phi}_u&#x0003D;4\pi {u}&#x0005E;2 $$ . In the last decades, the existence and stability of normalized solutions have been studied by many authors. We refer the reader to [14-18] and the references therein. The usual way in studying such problem is to look for the constrained critical points of the functional:
𝒥 ( u ) = 1 2 ∫ ℝ 3 | ∇ u | 2 d x + 1 4 ∫ ℝ 3 ∫ ℝ 3 | u ( x ) | 2 | u ( y ) | 2 | x − y | d x d y − 1 p ∫ ℝ 3 | u | p d x
on the constraint
S ( c ) = u ∈ H 1 ℝ 3 : ∫ ℝ 3 | u | 2 d x = c . $$ S(c)&#x0003D;\left\{u\in {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right):{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;2\kern0.3em \mathrm{d}x&#x0003D;c\right\}. $$
In [17], the authors proved the existence of minimizers when p = 8 3 $$ p&#x0003D;\frac{8}{3} $$ , and c ∈ 0 , c 0 $$ c\in \left(0,{c}_0\right) $$ for a suitable c 0 > 0 $$ {c}_0&gt;0 $$ . When p ∈ ( 2 , 3 ) $$ p\in \left(2,3\right) $$ , it was shown in [18] that a minimizer exists if c > 0 $$ c&gt;0 $$ is small enough. In [14], J. Bellazzini and G. Siciliano obtained the existence and stability only for sufficiently large L 2 $$ {L}&#x0005E;2 $$ -norm in case 3 < p < 10 3 $$ 3&lt;p&lt;\frac{10}{3} $$ , in case p = 8 3 $$ p&#x0003D;\frac{8}{3} $$ for sufficiently small charges. In [16], L. Jeanjean and T. Luo gave a threshold value of c 1 > 0 $$ {c}_1&gt;0 $$ for existence and nonexistence by a detailed study of the function c → m ( c ) : = inf u ∈ S c 𝒥 ( u ) in the range p ∈ 3 , 10 3 $$ p\in \left[3,\frac{10}{3}\right] $$ . Also, they gave a nonexistence result of normalized solutions when p = 3 $$ p&#x0003D;3 $$ for all c > 0 $$ c&gt;0 $$ and when p = 10 3 $$ p&#x0003D;\frac{10}{3} $$ for c > 0 $$ c&gt;0 $$ is small enough. In addition, when p ∈ 10 3 , 6 , m ( c ) = − ∞ $$ p\in \left(\frac{10}{3},6\right),\kern0.3em m(c)&#x0003D;-\infty $$ for all c > 0 $$ c&gt;0 $$ . In [15], the authors considered the mass supercritical case p ∈ 10 3 , 6 $$ p\in \left(\frac{10}{3},6\right) $$ . By virtue of a mountain-pass argument developed on S ( c ) $$ S(c) $$ , they showed that for c > 0 $$ c&gt;0 $$ small enough, 𝒥 admits a critical point constrained on S ( c ) $$ S(c) $$ at a strictly positive energy level, and it is orbitally unstable.
As for the existence of normalized solutions to nonlinear Schrödinger system
− Δ u + λ 1 u = μ 1 | u | p − 2 u + α | u | α − 2 | v | β u in ℝ N , − Δ v + λ 2 v = μ 2 | v | q − 2   v +   β | u | α | v | β − 2 v in ℝ N , ∫ ℝ N u 2 = a 2 , ∫ ℝ N v 2 = b 2 , $$ \left\{\begin{array}{l}-\Delta u&#x0002B;{\lambda}_1u&#x0003D;{\mu}_1{\left&#x0007C;u\right&#x0007C;}&#x0005E;{p-2}u&#x0002B;\alpha {\left&#x0007C;u\right&#x0007C;}&#x0005E;{\alpha -2}{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\beta }u\kern1em \mathrm{in}\kern0.3em {\mathbb{R}}&#x0005E;N,\\ {}-\Delta v&#x0002B;{\lambda}_2v&#x0003D;{\mu}_2{\left&#x0007C;v\right&#x0007C;}&#x0005E;{q-2}\;v&#x0002B;\beta {\left&#x0007C;u\right&#x0007C;}&#x0005E;{\alpha }{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\beta -2}v\kern1em \mathrm{in}\kern0.3em {\mathbb{R}}&#x0005E;N,\\ {}{\int}_{{\mathbb{R}}&#x0005E;N}{u}&#x0005E;2&#x0003D;{a}&#x0005E;2,\kern1em {\int}_{{\mathbb{R}}&#x0005E;N}{v}&#x0005E;2&#x0003D;{b}&#x0005E;2,\end{array}\right. $$
we refer to [19-26] and point out that no nonlocal terms are involved. In [27], J. Wang and W. Yang studied the coupled nonlinear Hartree equations with nonlocal interaction:
− Δ u + V 1 ( x ) u = λ 1 u + μ 1 ∫ ℝ N u 2 ( y ) | x − y | 2 d y u + β ∫ ℝ N v 2 ( y ) | x − y | 2 d y u , x ∈ ℝ N , − Δ v + V 2 ( x ) v = λ 2 v + μ 2 ∫ ℝ N v 2 ( y ) | x − y | 2 d y v + β ∫ ℝ N u 2 ( y ) | x − y | 2 d y v , x ∈ ℝ N . $$ \left\{\begin{array}{ll}-\Delta u&#x0002B;{V}_1(x)u&#x0003D;{\lambda}_1u&#x0002B;{\mu}_1\left({\int}_{{\mathbb{R}}&#x0005E;N}\frac{u&#x0005E;2(y)}{{\left&#x0007C;x-y\right&#x0007C;}&#x0005E;2}\mathrm{d}y\right)&amp; \\ {}u&#x0002B;\beta \left({\int}_{{\mathbb{R}}&#x0005E;N}\frac{v&#x0005E;2(y)}{{\left&#x0007C;x-y\right&#x0007C;}&#x0005E;2}\mathrm{d}y\right)u,&amp; x\in {\mathbb{R}}&#x0005E;N,\\ {}-\Delta v&#x0002B;{V}_2(x)v&#x0003D;{\lambda}_2v&#x0002B;{\mu}_2\left({\int}_{{\mathbb{R}}&#x0005E;N}\frac{v&#x0005E;2(y)}{{\left&#x0007C;x-y\right&#x0007C;}&#x0005E;2}\mathrm{d}y\right)&amp; \\ {}v&#x0002B;\beta \left({\int}_{{\mathbb{R}}&#x0005E;N}\frac{u&#x0005E;2(y)}{{\left&#x0007C;x-y\right&#x0007C;}&#x0005E;2}\mathrm{d}y\right)v,&amp; x\in {\mathbb{R}}&#x0005E;N.\end{array}\right. $$
In addition to proving the existence and nonexistence of normalized solutions, they also obtained a precise description of the concentration behavior of solutions to the system under certain type trapping potentials by proving some delicate energy estimates. Due to the influence of nonlocal terms, we should emphasize that it is more difficult to estimate the energy and obtain the compactness of the Palais–Smale sequence, which also leads to fewer research on such problems.

1.2 Main Results

Motivated by these recent works above, we consider the existence of solutions to (1.4) satisfying the conditions:
∫ ℝ 3 u 2 d x = a 1 > 0 and ∫ ℝ 3 v 2 d x = a 2 > 0 $$ {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;2\mathrm{d}x&#x0003D;{a}_1&gt;0\kern1em \mathrm{and}\kern1em {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;v\right&#x0007C;}&#x0005E;2\mathrm{d}x&#x0003D;{a}_2&gt;0 $$ (1.5)
Define
S a 1 , a 2 : = ( u , v ) ∈ H r 1 ℝ 3 × H r 1 ℝ 3 : u 2 2 = a 1 , v 2 2 = a 2 $$ {\displaystyle \begin{array}{cc}\hfill &amp; S\left({a}_1,{a}_2\right)\hfill \\ {}\hfill &amp; :&#x0003D; \left\{\left(u,v\right)\in {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right):{\left\Vert u\right\Vert}_2&#x0005E;2&#x0003D;{a}_1,{\left\Vert v\right\Vert}_2&#x0005E;2&#x0003D;{a}_2\right\}\hfill \end{array}} $$
and a solution ( u , v ) ∈ H r 1 ℝ 3 × H r 1 ℝ 3 $$ \left(u,v\right)\in {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ of (1.4)–(1.5) can be obtained by seeking a critical point of the functional
I ( u , v ) = 1 2 | | ∇ u | | 2 2 + 1 2 | | ∇ v | | 2 2 + α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x − 1 2 q | | u | | 2 q 2 q + | | v | | 2 q 2 q − β q ∫ ℝ 3 | u | q | v | q d x $$ {\displaystyle \begin{array}{cc}\hfill I\left(u,v\right)&amp; &#x0003D;\frac{1}{2}{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;\frac{1}{2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\hfill \\ {}\hfill &amp; \kern1em -\frac{1}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)-\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x\hfill \end{array}} $$
constrained on S a 1 , a 2 $$ S\left({a}_1,{a}_2\right) $$ . The parameters λ 1 , λ 2 ∈ ℝ $$ {\lambda}_1,{\lambda}_2\in \mathbb{R} $$ are no longer fixed but appear as Lagrange multipliers. I $$ I $$ is a functional of C 1 $$ {C}&#x0005E;1 $$ -class and bounded from below when 1 < q < 5 3 $$ 1&lt;q&lt;\frac{5}{3} $$ . Let
m a 1 , a 2 : = inf ( u , v ) ∈ S a 1 , a 2 I ( u , v ) $$ m\left({a}_1,{a}_2\right):&#x0003D; \underset{\left(u,v\right)\in S\left({a}_1,{a}_2\right)}{\operatorname{inf}}I\left(u,v\right) $$ (1.6)
In the present paper, by analyzing the compactness of the minimizing sequence of the related constraint problem, we obtain the existence of the normalized solutions of system (1.4). The orbital stability and some nonexistence results are also considered.

We state the main results as follows.

Theorem 1.1.Assume α , β > 0 $$ \alpha, \beta &gt;0 $$ and 1 < q < 5 3 $$ 1&lt;q&lt;\frac{5}{3} $$ .

  • i.

    When 1 < q < 4 3 $$ 1&lt;q&lt;\frac{4}{3} $$ , problem (1.4)–(1.5) admits a normalized solution for any a 1 , a 2 > 0 $$ {a}_1,{a}_2&gt;0 $$ .

  • ii.

    When 4 3 ≤ q < 3 2 $$ \frac{4}{3}\le q&lt;\frac{3}{2} $$ , problem (1.4)–(1.5) admits a normalized solution for a 1 , a 2 > 0 $$ {a}_1,{a}_2&gt;0 $$ small.

  • iii.

    When 3 2 < q < 5 3 $$ \frac{3}{2}&lt;q&lt;\frac{5}{3} $$ , problem (1.4)–(1.5) admits no normalized solution for a 1 , a 2 > 0 $$ {a}_1,{a}_2&gt;0 $$ small.

  • iv.

    When q = 3 2 , 1 ≤ α < 8 π $$ q&#x0003D;\frac{3}{2},\kern0.3em 1\le \alpha &lt;8\pi $$ and 0 < β < α $$ 0&lt;\beta &lt;\alpha $$ , problem (1.4)–(1.5) admits no normalized solution for any a 1 , a 2 > 0 $$ {a}_1,{a}_2&gt;0 $$ .

Next, we consider the orbital stability of minimizers.

Definition 1.2.Let

G a 1 , a 2 = ( u , v ) ∈ S ( a 1 , a 2 ) : I ( u , v ) = m a 1 , a 2 . $$ G\left({a}_1,{a}_2\right)&#x0003D;\left\{\left(u,v\right)\in S\Big({a}_1,{a}_2\left):I\right(u,v\Big)&#x0003D;m\left({a}_1,{a}_2\right)\right\}. $$
G a 1 , a 2 $$ G\left({a}_1,{a}_2\right) $$ is called orbitally stable, if for every ε > 0 $$ \varepsilon &gt;0 $$ , there exists δ > 0 $$ \delta &gt;0 $$ so that if the initial datum Ψ 1 · , 0 , Ψ 2 · , 0 $$ \left({\Psi}_1\left(\cdotp, 0\right),{\Psi}_2\left(\cdotp, 0\right)\right) $$ in the system (1.1) satisfies
inf ( u , v ) ∈ G a 1 , a 2 ( Ψ 1 · , 0 , Ψ 2 · , 0 ) − ( u , v ) H 1 < δ , $$ \underset{\left(u,v\right)\in G\left({a}_1,{a}_2\right)}{\operatorname{inf}}{\left\Vert \left({\Psi}_1\left(\cdotp, 0\right),{\Psi}_2\left(\cdotp, 0\right)\right)-\Big(u,v\Big)\right\Vert}_{H&#x0005E;1}&lt;\delta, $$
and there holds that
inf ( u , v ) ∈ G a 1 , a 2 Ψ 1 · , t , Ψ 2 · , t − ( u , v ) H 1 < ε , ∀ t > 0 , $$ \underset{\left(u,v\right)\in G\left({a}_1,{a}_2\right)}{\operatorname{inf}}{\left\Vert \left({\Psi}_1\left(\cdotp, t\right),{\Psi}_2\left(\cdotp, t\right)\right)-\left(u,v\right)\right\Vert}_{H&#x0005E;1}&lt;\varepsilon, \kern1em \forall t&gt;0, $$
where Ψ i · , t i = 1 , 2 $$ {\Psi}_i\left(\cdotp, t\right)\kern0.3em i&#x0003D;1,2 $$ is the solution of (1.1) with initial datum Ψ 1 · , 0 , Ψ 2 · , 0 $$ \left({\Psi}_1\left(\cdotp, 0\right),{\Psi}_2\left(\cdotp, 0\right)\right) $$ .

Theorem 1.3.Let q ∈ 1 , 3 2 $$ q\in \left(1,\frac{3}{2}\right) $$ . Then the set G a 1 , a 2 $$ G\left({a}_1,{a}_2\right) $$ is orbitally stable.

1.3 Main Difficulties and Ideas

The main difficulty of the problem is the compactness of the minimizing sequence with respect to m a 1 , a 2 $$ m\left({a}_1,{a}_2\right) $$ . In order to overcome this difficulty, the method in [19] is adopted. We consider the problem in H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ . By establishing a weak subadditive inequality, the strong convergence of the minimizing sequence is obtained. For the non-existence results, we mainly obtain it by a delicate estimate of the nonlocal term and applying the fact that any critical point of I ( u , v ) $$ I\left(u,v\right) $$ on S ( a 1 , a 2 ) $$ S\left({a}_1,{a}_2\right) $$ satisfies the identity Q ( u , v ) = 0 $$ Q\left(u,v\right)&#x0003D;0 $$ , where Q ( u , v ) $$ Q\left(u,v\right) $$ is defined in (3.9). In addition, through the scaling transformation u θ ( x ) = θ 2 u θ x , v θ ( x ) = θ 2 v θ x $$ {u}_{\theta }(x)&#x0003D;{\theta}&#x0005E;2u\left(\theta x\right),\kern0.3em {v}_{\theta }(x)&#x0003D;{\theta}&#x0005E;2v\left(\theta x\right) $$ , compared with the case of a single Schrödinger–Poisson equation, a new similar L 2 $$ {L}&#x0005E;2 $$ -critical index q = 4 3 $$ q&#x0003D;\frac{4}{3} $$ appears in our study. That is, when 1 < q < 4 3 , m a 1 , a 2 < 0 $$ 1&lt;q&lt;\frac{4}{3},\kern0.3em m\left({a}_1,{a}_2\right)&lt;0 $$ for any a 1 , a 2 > 0 $$ {a}_1,{a}_2&gt;0 $$ , but when 4 3 ≤ q < 3 2 , m a 1 , a 2 < 0 $$ \frac{4}{3}\le q&lt;\frac{3}{2},\kern0.3em m\left({a}_1,{a}_2\right)&lt;0 $$ only for sufficiently small a 1 , a 2 $$ {a}_1,{a}_2 $$ , when 3 2 < q < 5 3 , m a 1 , a 2 < 0 $$ \frac{3}{2}&lt;q&lt;\frac{5}{3},\kern0.3em m\left({a}_1,{a}_2\right)&lt;0 $$ only for sufficiently large a 1 , a 2 $$ {a}_1,{a}_2 $$ .

1.4 Notation

  • Denote the norm of L p ℝ 3 $$ {L}&#x0005E;p\left({\mathbb{R}}&#x0005E;3\right) $$ by
    ‖ u ‖ p : = ∫ ℝ 3 | u | p d x 1 p ,     1 ≤ p < ∞ . $$ {\left\Vert u\right\Vert}_p:&#x0003D; {\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;p\mathrm{d}x\right)}&#x0005E;{\frac{1}{p}},1\le p&lt;\infty . $$
  • H 1 ℝ 3 $$ {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ is the usual Sobolev space endowed with the norm
    ‖ u ‖ H 1 : = ∫ ℝ 3 ∇ u 2 d x + ∫ ℝ 3 u 2 d x 1 2 $$ {\left\Vert u\right\Vert}_{H&#x0005E;1}:&#x0003D; {\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla u\right&#x0007C;}&#x0005E;2\mathrm{d}x&#x0002B;{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;2\mathrm{d}x\right)}&#x0005E;{\frac{1}{2}} $$
    and
    H r 1 ℝ 3 : = u ∈ H 1 ℝ 3 : u ( x ) = u | x | . $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right):&#x0003D; \left\{u\in {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right):u(x)&#x0003D;u\left(&#x0007C;x|\right)\right\}. $$
  • D 1 , 2 ℝ 3 : = u ∈ L 2 ∗ ( ℝ 3 ) : ∇ u ∈ L 2 ℝ 3 $$ {D}&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right):&#x0003D; \left\{u\in {L}&#x0005E;{2&#x0005E;{\ast }}\left({\mathbb{R}}&#x0005E;3\right):\nabla u\in {L}&#x0005E;2\left({\mathbb{R}}&#x0005E;3\right)\right\} $$ with the norm ∫ ℝ 3 ∇ u 2 d x 1 2 $$ {\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla u\right&#x0007C;}&#x0005E;2\mathrm{d}x\right)}&#x0005E;{\frac{1}{2}} $$ and
    D r 1 , 2 ℝ 3 : = u ∈ D 1 , 2 ℝ 3 : u ( x ) = u | x | . $$ {D}_r&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right):&#x0003D; \left\{u\in {D}&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right):u(x)&#x0003D;u\left(&#x0007C;x|\right)\right\}. $$
  • Denote by “ ⇀ $$ \rightharpoonup $$ ” and “ → $$ \to $$ ” weak convergence and strong convergence, respectively.
  • C $$ C $$ represents various positive constants which may be different from line to line.
  • The symbol o n ( 1 ) $$ {o}_n(1) $$ is used to denote a quantity that goes to zero as n → + ∞ $$ n\to &#x0002B;\infty $$ .

This paper is organized as follows. In Section 2, some preliminaries are introduced. Particularly, some results in [11] are recalled that will be used to get compactness. We also give the variational setting for our problem. Section 3 is devoted to the proof of Theorem 1.1, which is about the existence and nonexistence of normalized solutions of (1.4). In Section 4, the orbital stability of the set of minimizers is established.

2 Preliminary Results

First, let us observe that the C 1 $$ {C}&#x0005E;1 $$ functional I ( u , v ) $$ I\left(u,v\right) $$ is well-defined in H 1 ℝ 3 × H 1 ℝ 3 $$ {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ . For 1 < q < 5 3 $$ 1&lt;q&lt;\frac{5}{3} $$ , thanks to the Hölder inequality, there is p > 1 $$ p&gt;1 $$ with 2 < q p , q p ′ ≤ 6 , p ′ : = p p − 1 $$ 2&lt; qp,q{p}&#x0005E;{\prime}\le 6,{p}&#x0005E;{\prime}:&#x0003D; \frac{p}{p-1} $$ ; hence,
∫ ℝ 3 u q v q d x ≤ u q p q v q p ′ q < ∞ f o r u , v ∈ H 1 ℝ 3 . $$ {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x\le {\left\Vert u\right\Vert}_{qp}&#x0005E;q{\left\Vert v\right\Vert}_{q{p}&#x0005E;{\prime}}&#x0005E;q&lt;\infty \kern1em for\kern3.0235pt u,v\in {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right). $$
We now give an upper bound estimate for the nonlocal term.

Lemma 2.1.There exist constants C 1 , C 2 , C 3 > 0 $$ {C}_1,{C}_2,{C}_3&gt;0 $$ independent of u and v, such that for all u , v ∈ H 1 ℝ 3 $$ u,v\in {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ ,

∫ ℝ 3 u 2 + v 2 ϕ u , v d x ≤ C 1 u 2 4 3 u 8 3 8 3 + C 2 v 2 4 3 v 8 3 8 3 + C 3 u 2 2 3 u 8 3 4 3 v 2 2 3 v 8 3 4 3 . $$ {\displaystyle \begin{array}{cc}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x&amp; \le {C}_1{\left\Vert u\right\Vert}_2&#x0005E;{\frac{4}{3}}{\left\Vert u\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{8}{3}}&#x0002B;{C}_2{\left\Vert v\right\Vert}_2&#x0005E;{\frac{4}{3}}{\left\Vert v\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{8}{3}}\hfill \\ {}\hfill &amp; \kern1em &#x0002B;{C}_3{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2}{3}}{\left\Vert u\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{4}{3}}{\left\Vert v\right\Vert}_2&#x0005E;{\frac{2}{3}}{\left\Vert v\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{4}{3}}.\hfill \end{array}} $$

Proof.Since ϕ u ( x ) : = ∫ ℝ 3 u ( y ) 2 | x − y | d y ∈ D 1 , 2 ℝ 3 $$ {\phi}_u(x):&#x0003D; {\int}_{{\mathbb{R}}&#x0005E;3}\frac{{\left&#x0007C;u(y)\right&#x0007C;}&#x0005E;2}{\mid x-y\mid}\mathrm{d}y\in {D}&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right) $$ solves the equation

− Δ ϕ u = 4 π u 2 in ℝ 3 $$ -\Delta {\phi}_u&#x0003D;4\pi {u}&#x0005E;2\kern1em \mathrm{in}\kern1em {\mathbb{R}}&#x0005E;3 $$ (2.1)
multiplying (2.1) by ϕ u ( x ) $$ {\phi}_u(x) $$ and integrating, we obtain
4 π ∫ ℝ 3 u 2 ϕ u d x = ∫ ℝ 3 ∇ ϕ u 2 d x . $$ 4\pi {\int}_{{\mathbb{R}}&#x0005E;3}{u}&#x0005E;2{\phi}_u\mathrm{d}x&#x0003D;{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla {\phi}_u\right&#x0007C;}&#x0005E;2\mathrm{d}x. $$
Recall the following inequality:
∫ ℝ 3 u 2 ϕ u d x ≤ C u 2 4 3 u 8 3 8 3 , $$ {\int}_{{\mathbb{R}}&#x0005E;3}{u}&#x0005E;2{\phi}_u\mathrm{d}x\le C{\left\Vert u\right\Vert}_2&#x0005E;{\frac{4}{3}}{\left\Vert u\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{8}{3}}, $$
then we have
∫ ℝ 3 v 2 ϕ u d x ≤ ∫ ℝ 3 ϕ u 6 d x 1 6 ∫ ℝ 3 v 12 5 d x 5 6 ≤ C ∫ ℝ 3 ∇ ϕ u 2 d x 1 2 ∫ ℝ 3 v 12 5 d x 5 6 = 2 π C ∫ ℝ 3 u 2 ϕ u d x 1 2 ∫ ℝ 3 v 12 5 d x 5 6 ≤ C ˜ u 2 2 3 u 8 3 4 3 v 2 2 3 v 8 3 4 3 . $$ {\displaystyle \begin{array}{cc}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}{v}&#x0005E;2{\phi}_u\mathrm{d}x&amp; \le {\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{\phi}_u\right&#x0007C;}&#x0005E;6\mathrm{d}x\right)}&#x0005E;{\frac{1}{6}}{\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\frac{12}{5}}\mathrm{d}x\right)}&#x0005E;{\frac{5}{6}}\\ {}\hfill &amp; \le C{\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla {\phi}_u\right&#x0007C;}&#x0005E;2\mathrm{d}x\right)}&#x0005E;{\frac{1}{2}}{\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\frac{12}{5}}\mathrm{d}x\right)}&#x0005E;{\frac{5}{6}}\\ {}\hfill &amp; &#x0003D;2\sqrt{\pi }C{\left({\int}_{{\mathbb{R}}&#x0005E;3}{u}&#x0005E;2{\phi}_u\mathrm{d}x\right)}&#x0005E;{\frac{1}{2}}{\left({\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\frac{12}{5}}\mathrm{d}x\right)}&#x0005E;{\frac{5}{6}}\\ {}\hfill &amp; \le \tilde{C}{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2}{3}}{\left\Vert u\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{4}{3}}{\left\Vert v\right\Vert}_2&#x0005E;{\frac{2}{3}}{\left\Vert v\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{4}{3}}.\end{array}} $$

Thus,

∫ ℝ 3 u 2 + v 2 ϕ u , v d x = ∫ ℝ 3 u 2 ϕ u d x + ∫ ℝ 3 v 2 ϕ v d x + ∫ ℝ 3 u 2 ϕ v d x + ∫ ℝ 3 v 2 ϕ u d x ≤ C 1 u 2 4 3 u 8 3 8 3 + C 2 v 2 4 3 v 8 3 8 3 + C 3 u 2 2 3 u 8 3 4 3 v 2 2 3 v 8 3 4 3 . $$ {\displaystyle \begin{array}{cc}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x&amp; &#x0003D;{\int}_{{\mathbb{R}}&#x0005E;3}{u}&#x0005E;2{\phi}_u\mathrm{d}x&#x0002B;{\int}_{{\mathbb{R}}&#x0005E;3}{v}&#x0005E;2{\phi}_v\mathrm{d}x\\ {}\hfill &amp; \kern1em &#x0002B;{\int}_{{\mathbb{R}}&#x0005E;3}{u}&#x0005E;2{\phi}_v\mathrm{d}x&#x0002B;{\int}_{{\mathbb{R}}&#x0005E;3}{v}&#x0005E;2{\phi}_u\mathrm{d}x\\ {}\hfill &amp; \le {C}_1{\left\Vert u\right\Vert}_2&#x0005E;{\frac{4}{3}}{\left\Vert u\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{8}{3}}&#x0002B;{C}_2{\left\Vert v\right\Vert}_2&#x0005E;{\frac{4}{3}}{\left\Vert v\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{8}{3}}\\ {}\hfill &amp; \kern1em &#x0002B;{C}_3{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2}{3}}{\left\Vert u\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{4}{3}}{\left\Vert v\right\Vert}_2&#x0005E;{\frac{2}{3}}{\left\Vert v\right\Vert}_{\frac{8}{3}}&#x0005E;{\frac{4}{3}}.\end{array}} $$

Next, we begin to show that the following properties hold, which are important for proving the convergence of the minimizing sequence u n , v n $$ \left({u}_n,{v}_n\right) $$ with respect to m ( a 1 , a 2 ) $$ m\left({a}_1,{a}_2\right) $$ .

Lemma 2.2. (see [[11]], Lemma 3.2)Let q ∈ ( 1 , 3 ) $$ q\in \left(1,3\right) $$ and u n , v n ⊂ H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ \left\{\left({u}_n,{v}_n\right)\right\}\subset {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ be such that u n , v n ⇀ ( u , v ) $$ \left({u}_n,{v}_n\right)\rightharpoonup \left(u,v\right) $$ in H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ as n → + ∞ $$ n\to &#x0002B;\infty $$ . We have, as n → + ∞ $$ n\to &#x0002B;\infty $$ ,

ϕ u n , v n → ϕ u , v in D r 1 , 2 ℝ 3 , ∫ ℝ 3 u n 2 + v n 2 ϕ u n , v n d x → ∫ ℝ 3 u 2 + v 2 ϕ u , v d x , ∫ ℝ 3 u n q v n q d x → ∫ ℝ 3 | u | q | v | q d x . $$ {\displaystyle \begin{array}{cc}\hfill {\phi}_{u_n,{v}_n}&amp; \to {\phi}_{u,v}\kern0.3em \mathrm{in}\kern0.3em {D}_r&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right),\\ {}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}_n&#x0005E;2&#x0002B;{v}_n&#x0005E;2\right){\phi}_{u_n,{v}_n}\mathrm{d}x&amp; \to {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x,\\ {}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{u}_n\right&#x0007C;}&#x0005E;q{\left&#x0007C;{v}_n\right&#x0007C;}&#x0005E;q\mathrm{d}x&amp; \to {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x.\end{array}} $$

As it is usual for elliptic equations, the solutions of (1.4) satisfy a suitable identity called Pohozaev identity, which can be found in [11], Lemma 3.1]. Benefiting from this Pohozaev identity, our nonexistence results are obtained.

Lemma 2.3.If ( u , v ) $$ \left(u,v\right) $$ is a solution of (1.4), then it satisfies the Pohozaev identity:

P λ 1 , λ 2 ( u , v ) = 1 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 − 3 2 λ 1 u 2 2 + λ 2 v 2 2 + 5 α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x − 3 2 q | | u | | 2 q 2 q + | | v | | 2 q 2 q − 3 β q ∫ ℝ 3 | u | q | v | q d x = 0 . $$ {\displaystyle \begin{array}{cc}\hfill {P}_{\lambda_1,{\lambda}_2}\left(u,v\right)&amp; &#x0003D;\frac{1}{2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)-\frac{3}{2}\left({\lambda}_1{\left\Vert u\right\Vert}_2&#x0005E;2&#x0002B;{\lambda}_2{\left\Vert v\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; \kern1em &#x0002B;\frac{5\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\\ {}\hfill &amp; \kern1em -\frac{3}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)-\frac{3\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x\\ {}\hfill &amp; &#x0003D;0.\end{array}} $$

3 Proof of Theorem 1.1

Before proving the main theorem, some lemmas are in order. The next lemma shows that the functional I ( u , v ) $$ I\left(u,v\right) $$ is bounded from below on S ( a 1 , a 2 ) $$ S\left({a}_1,{a}_2\right) $$ when 1 < q < 5 3 $$ 1&lt;q&lt;\frac{5}{3} $$ .

Lemma 3.1.If 1 < q < 5 3 $$ 1&lt;q&lt;\frac{5}{3} $$ , then for every a 1 , a 2 > 0 $$ {a}_1,{a}_2&gt;0 $$ , the functional I ( u , v ) $$ I\left(u,v\right) $$ is bounded from below and coercive on S a 1 , a 2 $$ S\left({a}_1,{a}_2\right) $$ .

Proof.The Gagliardo–Nirenberg inequality

| | u | | p ≤ C N , p | | ∇ u | | 2 N p − 2 2 p | | u | | 2 1 − N p − 2 2 p f o r u ∈ H 1 ℝ N , $$ {\left\Vert u\right\Vert}_p\le {C}_{N,p}{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{N\left(p-2\right)}{2p}}{\left\Vert u\right\Vert}_2&#x0005E;{1-\frac{N\left(p-2\right)}{2p}}\kern1em for\kern3.0235pt u\in {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;N\right), $$
which holds for 2 ≤ p ≤ 2 ∗ $$ 2\le p\le {2}&#x0005E;{\ast } $$ when N ≥ 3 $$ N\ge 3 $$ , implies for ( u , v ) ∈ S a 1 , a 2 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right) $$ ,
∫ ℝ 3 | u | q | v | q d x ≤ | | u | | q p q | | v | | q p ′ q ≤ C | | ∇ u | | 2 3 ( p q − 2 ) 2 p | | ∇ v | | 2 3 ( p ′ q − 2 ) 2 p ′ , $$ {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x\le {\left\Vert u\right\Vert}_{qp}&#x0005E;q{\left\Vert v\right\Vert}_{q{p}&#x0005E;{\prime}}&#x0005E;q\le C{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left( pq-2\right)}{2p}}{\left\Vert \nabla v\right\Vert}_2&#x0005E;{\frac{3\left({p}&#x0005E;{\prime }q-2\right)}{2{p}&#x0005E;{\prime }}}, $$
where p > 1 , 1 p ′ + 1 p = 1 , 2 ≤ q p , q p ′ ≤ 6 $$ p&gt;1,\kern0.3em \frac{1}{p&#x0005E;{\prime }}&#x0002B;\frac{1}{p}&#x0003D;1,\kern0.3em 2\le qp,q{p}&#x0005E;{\prime}\le 6 $$ .

So, we obtain

I ( u , v ) = 1 2 | | ∇ u | | 2 2 + 1 2 | | ∇ v | | 2 2 + α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x − 1 2 q | | u | | 2 q 2 q + | | v | | 2 q 2 q − β q ∫ ℝ 3 | u | q | v | q d x ≥ 1 2 | | ∇ u | | 2 2 + 1 2 | | ∇ v | | 2 2 − 1 2 q | | u | | 2 q 2 q + | | v | | 2 q 2 q − β q ∫ ℝ 3 | u | q | v | q d x ≥ 1 2 | | ∇ u | | 2 2 + 1 2 | | ∇ v | | 2 2 − 1 2 q C q , a 1 | | ∇ u | | 2 3 q − 1 + C q , a 2 | | ∇ v | | 2 3 q − 1 − β C q | | ∇ u | | 2 3 ( p q − 2 ) 2 p | | ∇ v | | 2 3 ( p ′ q − 2 ) 2 p ′ = 1 2 | | ∇ u | | 2 2 + 1 2 | | ∇ v | | 2 2 − C 1 | | ∇ u | | 2 3 q − 1 − C 2 | | ∇ v | | 2 3 q − 1 − C 3 | | ∇ u | | 2 3 ( p q − 2 ) 2 p | | ∇ v | | 2 3 ( p ′ q − 2 ) 2 p ′ . $$ {\displaystyle \begin{array}{cc}\hfill I\left(u,v\right)&amp; &#x0003D;\frac{1}{2}{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;\frac{1}{2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\\ {}\hfill &amp; \kern1em -\frac{1}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)-\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x\\ {}\hfill &amp; \ge \frac{1}{2}{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;\frac{1}{2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;2-\frac{1}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; \kern1em -\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x\\ {}\hfill &amp; \ge \frac{1}{2}{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;\frac{1}{2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\\ {}\hfill &amp; \kern1em -\frac{1}{2q}\left({C}_{q,{a}_1}{\left\Vert \nabla u\right\Vert}_2&#x0005E;{3\left(q-1\right)}&#x0002B;{C}_{q,{a}_2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;{3\left(q-1\right)}\right)\\ {}\hfill &amp; \kern1em -\frac{\beta C}{q}{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left( pq-2\right)}{2p}}{\left\Vert \nabla v\right\Vert}_2&#x0005E;{\frac{3\left({p}&#x0005E;{\prime }q-2\right)}{2{p}&#x0005E;{\prime }}}\\ {}\hfill &amp; &#x0003D;\frac{1}{2}{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;\frac{1}{2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;2-{C}_1{\left\Vert \nabla u\right\Vert}_2&#x0005E;{3\left(q-1\right)}\\ {}\hfill &amp; \kern1em -{C}_2{\left\Vert \nabla v\right\Vert}_2&#x0005E;{3\left(q-1\right)}-{C}_3{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left( pq-2\right)}{2p}}{\left\Vert \nabla v\right\Vert}_2&#x0005E;{\frac{3\left({p}&#x0005E;{\prime }q-2\right)}{2{p}&#x0005E;{\prime }}}.\end{array}} $$

As 1 < q < 5 3 $$ 1&lt;q&lt;\frac{5}{3} $$ , it follows that 0 < 3 q − 1 < 2 , 0 < 3 p q − 2 2 p + 3 p ′ q − 2 2 p ′ < 2 $$ 0&lt;3\left(q-1\right)&lt;2,\kern0.3em 0&lt;\frac{3\left( pq-2\right)}{2p}&#x0002B;\frac{3\left({p}&#x0005E;{\prime }q-2\right)}{2{p}&#x0005E;{\prime }}&lt;2 $$ , which ensures the boundedness of I ( u , v ) $$ I\left(u,v\right) $$ from below and the coerciveness on S a 1 , a 2 $$ S\left({a}_1,{a}_2\right) $$ .

Hereafter, we use the same notation m a 1 , a 2 $$ m\left({a}_1,{a}_2\right) $$ for a 1 , a 2 ≥ 0 $$ {a}_1,{a}_2\ge 0 $$ with either a 1 > 0 $$ {a}_1&gt;0 $$ or a 2 > 0 $$ {a}_2&gt;0 $$ , namely, one component of a 1 , a 2 $$ \left({a}_1,{a}_2\right) $$ maybe zero.

In what follows, we collect some basic properties of m a 1 , a 2 $$ m\left({a}_1,{a}_2\right) $$ .

Lemma 3.2.

  • 1.

    Let 1 < q < 4 3 $$ 1&lt;q&lt;\frac{4}{3} $$ , for any a 1 , a 2 ≥ 0 $$ {a}_1,{a}_2\ge 0 $$ with either a 1 > 0 $$ {a}_1&gt;0 $$ or a 2 > 0 $$ {a}_2&gt;0 $$ ,

    − ∞ < m a 1 , a 2 < 0 . $$ -\infty &lt;m\left({a}_1,{a}_2\right)&lt;0. $$

  • 2.

    If 4 3 ≤ q < 3 2 $$ \frac{4}{3}\le q&lt;\frac{3}{2} $$ , there exist ρ 1 , ρ 2 > 0 $$ {\rho}_1,{\rho}_2&gt;0 $$ such that − ∞ < m a 1 , a 2 < 0 $$ -\infty &lt;m\left({a}_1,{a}_2\right)&lt;0 $$ for all a 1 ∈ 0 , ρ 1 , a 2 ∈ 0 , ρ 2 $$ {a}_1\in \left(0,{\rho}_1\right),{a}_2\in \left(0,{\rho}_2\right) $$ . If 3 2 < q < 5 3 $$ \frac{3}{2}&lt;q&lt;\frac{5}{3} $$ , then there exist ρ 3 , ρ 4 > 0 $$ {\rho}_3,{\rho}_4&gt;0 $$ such that − ∞ < m a 1 , a 2 < 0 $$ -\infty &lt;m\left({a}_1,{a}_2\right)&lt;0 $$ for all a 1 ∈ ρ 3 , + ∞ , a 2 ∈ ρ 4 , + ∞ $$ {a}_1\in \left({\rho}_3,&#x0002B;\infty \right),{a}_2\in \left({\rho}_4,&#x0002B;\infty \right) $$ .

  • 3.

    m a 1 , a 2 $$ m\left({a}_1,{a}_2\right) $$ is continuous with respect to a 1 , a 2 $$ {a}_1,{a}_2 $$ .

  • 4.

    For any a 1 ≥ b 1 ≥ 0 , a 2 ≥ b 2 ≥ 0 $$ {a}_1\ge {b}_1\ge 0,{a}_2\ge {b}_2\ge 0 $$ ,

    m a 1 , a 2 ≤ m b 1 , b 2 + m a 1 − b 1 , a 2 − b 2 . $$ m\left({a}_1,{a}_2\right)\le m\left({b}_1,{b}_2\right)&#x0002B;m\left({a}_1-{b}_1,{a}_2-{b}_2\right). $$

Proof.

  1. It follows from Lemma 3.1 that I ( u , v ) $$ I\left(u,v\right) $$ is coercive and in particular m a 1 , a 2 > − ∞ $$ m\left({a}_1,{a}_2\right)&gt;-\infty $$ . We define u s ( x ) = e 3 s 2 u e s x , v s ( x ) = e 3 s 2 v e s x $$ {u}_s(x)&#x0003D;{e}&#x0005E;{\frac{3s}{2}}u\left({e}&#x0005E;sx\right),\kern0.3em {v}_s(x)&#x0003D;{e}&#x0005E;{\frac{3s}{2}}v\left({e}&#x0005E;sx\right) $$ , so that u s ( x ) 2 2 = u ( x ) 2 2 , v s ( x ) 2 2 = v ( x ) 2 2 $$ {\left\Vert {u}_s(x)\right\Vert}_2&#x0005E;2&#x0003D;{\left\Vert u(x)\right\Vert}_2&#x0005E;2,\kern0.3em {\left\Vert {v}_s(x)\right\Vert}_2&#x0005E;2&#x0003D;{\left\Vert v(x)\right\Vert}_2&#x0005E;2 $$ , then we have the following scaling laws,

    ∇ u s ( x ) 2 2 = e 2 s ∇ u ( x ) 2 2 , $$ {\left\Vert \nabla {u}_s(x)\right\Vert}_2&#x0005E;2&#x0003D;{e}&#x0005E;{2s}{\left\Vert \nabla u(x)\right\Vert}_2&#x0005E;2, $$
    u s ( x ) 2 q 2 q = e 3 s q − 1 u ( x ) 2 q 2 q , $$ {\left\Vert {u}_s(x)\right\Vert}_{2q}&#x0005E;{2q}&#x0003D;{e}&#x0005E;{3s\left(q-1\right)}{\left\Vert u(x)\right\Vert}_{2q}&#x0005E;{2q}, $$
    ∫ ℝ 3 u s ( x ) q v s ( x ) q d x = e 3 s q − 1 ∫ ℝ 3 u ( x ) q v ( x ) q d x , $$ {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{u}_s(x)\right&#x0007C;}&#x0005E;q{\left&#x0007C;{v}_s(x)\right&#x0007C;}&#x0005E;q\mathrm{d}x&#x0003D;{e}&#x0005E;{3s\left(q-1\right)}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u(x)\right&#x0007C;}&#x0005E;q{\left&#x0007C;v(x)\right&#x0007C;}&#x0005E;q\mathrm{d}x, $$
    ∫ ℝ 3 u s ( x ) 2 + v s ( x ) 2 ϕ u s , v s ( x ) d x = e s ∫ ℝ 3 u ( x ) 2 + v ( x ) 2 ϕ u , v ( x ) d x . $$ {\displaystyle \begin{array}{cc}\hfill &amp; {\int}_{{\mathbb{R}}&#x0005E;3}\left[{\left({u}_s(x)\right)}&#x0005E;2&#x0002B;{\left({v}_s(x)\right)}&#x0005E;2\right]{\phi}_{u_s,{v}_s}(x)\mathrm{d}x&#x0003D;{e}&#x0005E;s\hfill \\ {}\hfill &amp; {\int}_{{\mathbb{R}}&#x0005E;3}\left[u{(x)}&#x0005E;2&#x0002B;v{(x)}&#x0005E;2\right]{\phi}_{u,v}(x)\mathrm{d}x.\hfill \end{array}} $$
    Thus,
    I u s , v s = e 2 s 2 ∇ u 2 2 + ∇ v 2 2 + α e s 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v ( x ) d x − e 3 s q − 1 2 q u 2 q 2 q + v 2 q 2 q − β e 3 s q − 1 q ∫ ℝ 3 u q v q d x . $$ {\displaystyle \begin{array}{cc}\hfill I\left({u}_s,{v}_s\right)&#x0003D;&amp; \frac{e&#x0005E;{2s}}{2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; &#x0002B;\frac{\alpha {e}&#x0005E;s}{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}(x)\mathrm{d}x\\ {}\hfill &amp; -\frac{e&#x0005E;{3s\left(q-1\right)}}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; -\frac{\beta {e}&#x0005E;{3s\left(q-1\right)}}{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x.\end{array}} $$

    We notice that 0 < 3 s q − 1 < s $$ 0&lt;3s\left(q-1\right)&lt;s $$ for 1 < q < 4 3 $$ 1&lt;q&lt;\frac{4}{3} $$ ; thus, for s → − ∞ $$ s\to -\infty $$ , we have I u s , v s → 0 − $$ I\left({u}_s,{v}_s\right)\to {0}&#x0005E;{-} $$ , which prove the first claim.

  2. When 4 3 ≤ q < 3 2 $$ \frac{4}{3}\le q&lt;\frac{3}{2} $$ , we set u θ ( x ) = θ 1 2 − 3 2 r u x θ r , v θ ( x ) = θ 1 2 − 3 2 r v x θ r $$ {u}_{\theta }(x)&#x0003D;{\theta}&#x0005E;{\frac{1}{2}-\frac{3}{2}r}u\left(\frac{x}{\theta&#x0005E;r}\right),\kern0.3em {v}_{\theta }(x)&#x0003D;{\theta}&#x0005E;{\frac{1}{2}-\frac{3}{2}r}v\left(\frac{x}{\theta&#x0005E;r}\right) $$ , so that u θ ( x ) 2 2 = θ u ( x ) 2 2 , v θ ( x ) 2 2 = θ v ( x ) 2 2 $$ {\left\Vert {u}_{\theta }(x)\right\Vert}_2&#x0005E;2&#x0003D;\theta {\left\Vert u(x)\right\Vert}_2&#x0005E;2,\kern0.3em {\left\Vert {v}_{\theta }(x)\right\Vert}_2&#x0005E;2&#x0003D;\theta {\left\Vert v(x)\right\Vert}_2&#x0005E;2 $$ , then the following scaling laws can be obtained:

    ∇ u θ ( x ) 2 2 = θ 1 − 2 r ∇ u ( x ) 2 2 , $$ {\left\Vert \nabla {u}_{\theta }(x)\right\Vert}_2&#x0005E;2&#x0003D;{\theta}&#x0005E;{1-2r}{\left\Vert \nabla u(x)\right\Vert}_2&#x0005E;2, $$
    u θ ( x ) 2 q 2 q = θ 1 − 3 r q + 3 r u ( x ) 2 q 2 q , $$ {\left\Vert {u}_{\theta }(x)\right\Vert}_{2q}&#x0005E;{2q}&#x0003D;{\theta}&#x0005E;{\left(1-3r\right)q&#x0002B;3r}{\left\Vert u(x)\right\Vert}_{2q}&#x0005E;{2q}, $$
    ∫ ℝ 3 u θ ( x ) q v θ ( x ) q d x = θ 1 − 3 r q + 3 r ∫ ℝ 3 u ( x ) q v ( x ) q d x , $$ {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{u}_{\theta }(x)\right&#x0007C;}&#x0005E;q{\left&#x0007C;{v}_{\theta }(x)\right&#x0007C;}&#x0005E;q\mathrm{d}x&#x0003D;{\theta}&#x0005E;{\left(1-3r\right)q&#x0002B;3r}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u(x)\right&#x0007C;}&#x0005E;q{\left&#x0007C;v(x)\right&#x0007C;}&#x0005E;q\mathrm{d}x, $$
    ∫ ℝ 3 u θ ( x ) 2 + v θ ( x ) 2 ϕ u θ , v θ ( x ) d x = θ 2 − r ∫ ℝ 3 u ( x ) 2 + v ( x ) 2 ϕ u , v ( x ) d x . $$ {\displaystyle \begin{array}{cc}\hfill &amp; {\int}_{{\mathbb{R}}&#x0005E;3}\left[{\left({u}_{\theta }(x)\right)}&#x0005E;2&#x0002B;{\left({v}_{\theta }(x)\right)}&#x0005E;2\right]{\phi}_{u_{\theta },{v}_{\theta }}(x)\mathrm{d}x&#x0003D;{\theta}&#x0005E;{2-r}\hfill \\ {}\hfill &amp; {\int}_{{\mathbb{R}}&#x0005E;3}\left[u{(x)}&#x0005E;2&#x0002B;v{(x)}&#x0005E;2\right]{\phi}_{u,v}(x)\mathrm{d}x.\hfill \end{array}} $$
    Therefore,
    I u θ , v θ = 1 2 θ 1 − 2 r ∇ u 2 2 + ∇ v 2 2 + α 4 θ 2 − r ∫ ℝ 3 u 2 + v 2 ϕ u , v ( x ) d x − 1 2 q θ ( 1 − 3 r ) q + 3 r u 2 q 2 q + v 2 q 2 q − β q θ ( 1 − 3 r ) q + 3 r ∫ ℝ 3 u q v q d x . $$ {\displaystyle \begin{array}{cc}\hfill I\left({u}_{\theta },{v}_{\theta}\right)&#x0003D;&amp; \frac{1}{2}{\theta}&#x0005E;{1-2r}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; &#x0002B;\frac{\alpha }{4}{\theta}&#x0005E;{2-r}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}(x)\mathrm{d}x\\ {}\hfill &amp; -\frac{1}{2q}{\theta}&#x0005E;{\left(1-3r\right)q&#x0002B;3r}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; -\frac{\beta }{q}{\theta}&#x0005E;{\left(1-3r\right)q&#x0002B;3r}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x.\end{array}} $$

    Note that for r = − 1 $$ r&#x0003D;-1 $$ , we get

    I u θ , v θ = 1 2 θ 3 ∇ u 2 2 + ∇ v 2 2 + α 4 θ 3 ∫ ℝ 3 u 2 + v 2 ϕ u , v ( x ) d x − 1 2 q θ 4 q − 3 u 2 q 2 q + v 2 q 2 q − β q θ 4 q − 3 ∫ ℝ 3 u q v q d x . $$ {\displaystyle \begin{array}{cc}\hfill I\left({u}_{\theta },{v}_{\theta}\right)&#x0003D;&amp; \frac{1}{2}{\theta}&#x0005E;3\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; &#x0002B;\frac{\alpha }{4}{\theta}&#x0005E;3{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}(x)\mathrm{d}x\\ {}\hfill &amp; -\frac{1}{2q}{\theta}&#x0005E;{4q-3}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; -\frac{\beta }{q}{\theta}&#x0005E;{4q-3}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x.\end{array}} $$

    Since 4 q − 3 < 3 $$ 4q-3&lt;3 $$ for 4 3 ≤ q < 3 2 $$ \frac{4}{3}\le q&lt;\frac{3}{2} $$ , there holds for θ → 0 , I u θ , v θ → 0 − $$ \theta \to 0,\kern0.3em I\left({u}_{\theta },{v}_{\theta}\right)\to {0}&#x0005E;{-} $$ . Thus, there exist ρ 1 , ρ 2 > 0 $$ {\rho}_1,{\rho}_2&gt;0 $$ such that − ∞ < m a 1 , a 2 < 0 $$ -\infty &lt;m\left({a}_1,{a}_2\right)&lt;0 $$ for all a 1 ∈ 0 , ρ 1 , a 2 ∈ 0 , ρ 2 $$ {a}_1\in \left(0,{\rho}_1\right),{a}_2\in \left(0,{\rho}_2\right) $$ . If 3 2 < q < 5 3 $$ \frac{3}{2}&lt;q&lt;\frac{5}{3} $$ , we have 4 q − 3 > 3 $$ 4q-3&gt;3 $$ , then for θ → + ∞ , I u θ , v θ → 0 − $$ \theta \to &#x0002B;\infty, \kern0.3em I\left({u}_{\theta },{v}_{\theta}\right)\to {0}&#x0005E;{-} $$ . Thus, there exist ρ 3 , ρ 4 > 0 $$ {\rho}_3,{\rho}_4&gt;0 $$ such that − ∞ < m a 1 , a 2 < 0 $$ -\infty &lt;m\left({a}_1,{a}_2\right)&lt;0 $$ for all a 1 ∈ ρ 3 , + ∞ , a 2 ∈ ρ 4 , + ∞ $$ {a}_1\in \left({\rho}_3,&#x0002B;\infty \right),{a}_2\in \left({\rho}_4,&#x0002B;\infty \right) $$ . The second claim is completed.

  3. We assume a 1 n , a 2 n = a 1 , a 2 + o n ( 1 ) $$ \left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right)&#x0003D;\left({a}_1,{a}_2\right)&#x0002B;{o}_n(1) $$ . From the definition of m a 1 n , a 2 n $$ m\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right) $$ , for any ε > 0 $$ \varepsilon &gt;0 $$ , there exists u n , v n ∈ S a 1 n , a 2 n $$ \left({u}_n,{v}_n\right)\in S\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right) $$ such that
    I u n , v n ≤ m a 1 n , a 2 n + ε $$ I\left({u}_n,{v}_n\right)\le m\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right)&#x0002B;\varepsilon $$ (3.1)
    Setting ū n : = u n u n 2 a 1 1 2 , v ‾ n : = v n v n 2 a 2 1 2 $$ {\bar{u}}_n:&#x0003D; \frac{u_n}{{\left\Vert {u}_n\right\Vert}_2}{a}_1&#x0005E;{\frac{1}{2}},\kern0.3em {\overline{v}}_n:&#x0003D; \frac{v_n}{{\left\Vert {v}_n\right\Vert}_2}{a}_2&#x0005E;{\frac{1}{2}} $$ , we have that ū n , v ‾ n ∈ S a 1 , a 2 $$ \left({\bar{u}}_n,{\overline{v}}_n\right)\in S\left({a}_1,{a}_2\right) $$ and
    m a 1 , a 2 ≤ I ū n , v ‾ n = I u n , v n + o n ( 1 ) $$ m\left({a}_1,{a}_2\right)\le I\left({\bar{u}}_n,{\overline{v}}_n\right)&#x0003D;I\left({u}_n,{v}_n\right)&#x0002B;{o}_n(1) $$ (3.2)
    Combining (3.1) and (3.2), we obtain
    m a 1 , a 2 ≤ m a 1 n , a 2 n + ε + o n ( 1 ) $$ m\left({a}_1,{a}_2\right)\le m\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right)&#x0002B;\varepsilon &#x0002B;{o}_n(1) $$ (3.3)
    Similarly, from the definition of m a 1 , a 2 $$ m\left({a}_1,{a}_2\right) $$ , for any ε > 0 $$ \varepsilon &gt;0 $$ , there exists ( u , v ) ∈ S a 1 , a 2 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right) $$ such that
    I ( u , v ) ≤ m a 1 , a 2 + ε $$ I\left(u,v\right)\le m\left({a}_1,{a}_2\right)&#x0002B;\varepsilon $$ (3.4)
    Let ū : = u u 2 a 1 n 1 2 , v ‾ : = v v 2 a 2 n 1 2 $$ \bar{u}:&#x0003D; \frac{u}{{\left\Vert u\right\Vert}_2}{\left({a}_1&#x0005E;n\right)}&#x0005E;{\frac{1}{2}},\kern0.3em \overline{v}:&#x0003D; \frac{v}{{\left\Vert v\right\Vert}_2}{\left({a}_2&#x0005E;n\right)}&#x0005E;{\frac{1}{2}} $$ , then ū , v ‾ ∈ S a 1 n , a 2 n $$ \left(\bar{u},\overline{v}\right)\in S\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right) $$ and
    m a 1 n , a 2 n ≤ I ū , v ‾ = I ( u , v ) + o n ( 1 ) $$ m\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right)\le I\left(\bar{u},\overline{v}\right)&#x0003D;I\left(u,v\right)&#x0002B;{o}_n(1) $$ (3.5)
    Combining (3.4) and (3.5), we deduce that
    m a 1 n , a 2 n ≤ m a 1 , a 2 + ε + o n ( 1 ) $$ m\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right)\le m\left({a}_1,{a}_2\right)&#x0002B;\varepsilon &#x0002B;{o}_n(1) $$ (3.6)
    Therefore, since ε > 0 $$ \varepsilon &gt;0 $$ is arbitrary, according to (3.3) and (3.6), we deduce that
    m a 1 n , a 2 n = m a 1 , a 2 + ε + o n ( 1 ) . $$ m\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right)&#x0003D;m\left({a}_1,{a}_2\right)&#x0002B;\varepsilon &#x0002B;{o}_n(1). $$
    The third claim is obtained.
  4. By density of C 0 ∞ ℝ ℕ $$ {C}_0&#x0005E;{\infty}\left({\mathbb{R}}&#x0005E;{\mathbb{N}}\right) $$ into H 1 ℝ ℕ $$ {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;{\mathbb{N}}\right) $$ , for any ε > 0 $$ \varepsilon &gt;0 $$ , there exist ξ ‾ 1 , ξ ‾ 2 , ξ ^ 1 , ξ ^ 2 ∈ C 0 ∞ ℝ ℕ × C 0 ∞ ℝ ℕ $$ \left({\overline{\xi}}_1,{\overline{\xi}}_2\right),\left({\hat{\xi}}_1,{\hat{\xi}}_2\right)\in {C}_0&#x0005E;{\infty}\left({\mathbb{R}}&#x0005E;{\mathbb{N}}\right)\times {C}_0&#x0005E;{\infty}\left({\mathbb{R}}&#x0005E;{\mathbb{N}}\right) $$ with ξ i ‾ 2 2 = b i , ξ i ^ 2 2 = a i − b i $$ {\left\Vert \overline{\xi_i}\right\Vert}_2&#x0005E;2&#x0003D;{b}_i,{\left\Vert \hat{\xi_i}\right\Vert}_2&#x0005E;2&#x0003D;{a}_i-{b}_i $$ for i = 1 , 2 $$ i&#x0003D;1,2 $$ such that
    I ξ ‾ 1 , ξ ‾ 2 ≤ m b 1 , b 2 + ε 2 $$ I\left({\overline{\xi}}_1,{\overline{\xi}}_2\right)\le m\left({b}_1,{b}_2\right)&#x0002B;\frac{\varepsilon }{2} $$ (3.7)
    I ξ ^ 1 , ξ ^ 2 ≤ m a 1 − b 1 , a 2 − b 2 + ε 2 $$ I\left({\hat{\xi}}_1,{\hat{\xi}}_2\right)\le m\left({a}_1-{b}_1,{a}_2-{b}_2\right)&#x0002B;\frac{\varepsilon }{2} $$ (3.8)
    We may assume that
    s u p p ξ ‾ 1 ∪ s u p p ξ ‾ 2 ∩ s u p p ξ ^ 1 ∪ s u p p ξ ^ 2 = ∅ , $$ \left(\mathit{\operatorname{supp}}{\overline{\xi}}_1\cup \mathit{\operatorname{supp}}{\overline{\xi}}_2\right)\cap \left(\mathit{\operatorname{supp}}{\hat{\xi}}_1\cup \mathit{\operatorname{supp}}{\hat{\xi}}_2\right)&#x0003D;\varnothing, $$
    and
    ∫ ℝ 3 ξ 1 ‾ 2 + ξ 2 ‾ 2 ϕ ξ ^ 1 , ξ ^ 2 d x = ∫ ℝ 3 ∫ ℝ 3 ξ 1 ‾ 2 ( x ) + ξ 2 ‾ 2 ( x ) ξ 1 ^ 2 ( y ) + ξ 2 ^ 2 ( y ) x − y < 2 ε α , $$ {\displaystyle \begin{array}{cc}\hfill &amp; {\int}_{{\mathbb{R}}&#x0005E;3}\left({\overline{\xi_1}}&#x0005E;2&#x0002B;{\overline{\xi_2}}&#x0005E;2\right){\phi}_{{\hat{\xi}}_1,{\hat{\xi}}_2}\mathrm{d}x&#x0003D;{\int}_{{\mathbb{R}}&#x0005E;3}\hfill \\ {}\hfill &amp; {\int}_{{\mathbb{R}}&#x0005E;3}\frac{\left({\overline{\xi_1}}&#x0005E;2(x)&#x0002B;{\overline{\xi_2}}&#x0005E;2(x)\right)\left({\hat{\xi_1}}&#x0005E;2(y)&#x0002B;{\hat{\xi_2}}&#x0005E;2(y)\right)}{\left&#x0007C;x-y\right&#x0007C;}&lt;\frac{2\varepsilon }{\alpha },\hfill \end{array}} $$
    then for i = 1 , 2 $$ i&#x0003D;1,2 $$ ,
    ξ ‾ i + ξ ^ i 2 2 = ξ ‾ i 2 2 + ξ ^ i 2 2 = b i + a i − b i = a i . $$ {\left\Vert {\overline{\xi}}_i&#x0002B;{\hat{\xi}}_i\right\Vert}_2&#x0005E;2&#x0003D;{\left\Vert {\overline{\xi}}_i\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert {\hat{\xi}}_i\right\Vert}_2&#x0005E;2&#x0003D;{b}_i&#x0002B;\left({a}_i-{b}_i\right)&#x0003D;{a}_i. $$
    It follows that m a 1 , a 2 ≤ I ξ ‾ 1 + ξ ^ 1 , ξ ‾ 2 + ξ ^ 2 $$ m\left({a}_1,{a}_2\right)\le I\left({\overline{\xi}}_1&#x0002B;{\hat{\xi}}_1,{\overline{\xi}}_2&#x0002B;{\hat{\xi}}_2\right) $$ . Set ξ i = ξ ‾ i + ξ ^ i $$ {\xi}_i&#x0003D;{\overline{\xi}}_i&#x0002B;{\hat{\xi}}_i $$ , we have ξ i 2 2 = a i $$ {\left\Vert {\xi}_i\right\Vert}_2&#x0005E;2&#x0003D;{a}_i $$ for i = 1 , 2 $$ i&#x0003D;1,2 $$ , and
    I ξ 1 , ξ 2 = 1 2 ∇ ξ 1 2 2 + ∇ ξ 2 2 2 + α 4 ∫ ℝ 3 ξ 1 2 + ξ 2 2 ϕ ξ 1 , ξ 2 d x − 1 2 q ξ 1 2 q 2 q + ξ 2 2 q 2 q − β q ∫ ℝ 3 ξ 1 q ξ 2 q d x = 1 2 ∇ ξ ‾ 1 2 2 + ∇ ξ 2 ‾ 2 2 + α 4 ∫ ℝ 3 ξ 1 ‾ 2 + ξ 2 ‾ 2 ϕ ξ 1 ‾ , ξ 2 ‾ d x − 1 2 q ξ 1 ‾ 2 q 2 q + ξ 2 ‾ 2 q 2 q − β q ∫ ℝ 3 ξ 1 ‾ q ξ 2 ‾ q d x + 1 2 ∇ ξ ^ 1 2 2 + ∇ ξ ^ 2 2 2 + α 4 ∫ ℝ 3 ξ ^ 1 2 + ξ ^ 2 2 ϕ ξ ^ 1 , ξ ^ 2 d x − 1 2 q ξ ^ 1 2 q 2 q + ξ ^ 2 2 q 2 q − β q ∫ ℝ 3 ξ ^ 1 q ξ ^ 2 q d x + α 4 ∫ ℝ 3 ξ 1 ‾ 2 + ξ 2 ‾ 2 ϕ ξ ^ 1 , ξ ^ 2 d x + ∫ ℝ 3 ξ ^ 1 2 + ξ ^ 2 2 ϕ ξ 1 ‾ , ξ 2 ‾ d x ≤ I ξ 1 ‾ , ξ 2 ‾ + I ξ ^ 1 , ξ ^ 2 + ε . $$ {\displaystyle \begin{array}{ll}\hfill I\left({\xi}_1,{\xi}_2\right)&amp; &#x0003D;\frac{1}{2}\left({\left\Vert \nabla {\xi}_1\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla {\xi}_2\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; \kern1em &#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({\xi}_1&#x0005E;2&#x0002B;{\xi}_2&#x0005E;2\right){\phi}_{\xi_1,{\xi}_2}\mathrm{d}x\\ {}\hfill &amp; \kern0.90em -\frac{1}{2q}\left({\left\Vert {\xi}_1\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert {\xi}_2\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; \kern1em -\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{\xi}_1\right&#x0007C;}&#x0005E;q{\left&#x0007C;{\xi}_2\right&#x0007C;}&#x0005E;q\mathrm{d}x\\ {}\hfill &amp; &#x0003D;\frac{1}{2}\left({\left\Vert \nabla {\overline{\xi}}_1\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla \overline{\xi_2}\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; \kern1em &#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({\overline{\xi_1}}&#x0005E;2&#x0002B;{\overline{\xi_2}}&#x0005E;2\right){\phi}_{\overline{\xi_1},\overline{\xi_2}}\mathrm{d}x\\ {}\hfill &amp; \kern0.90em -\frac{1}{2q}\left({\left\Vert \overline{\xi_1}\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert \overline{\xi_2}\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; \kern1em -\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\overline{\xi_1}\right&#x0007C;}&#x0005E;q{\left&#x0007C;\overline{\xi_2}\right&#x0007C;}&#x0005E;q\mathrm{d}x\\ {}\hfill &amp; \kern1.20em &#x0002B;\frac{1}{2}\left({\left\Vert \nabla {\hat{\xi}}_1\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla {\hat{\xi}}_2\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; \kern1em &#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({\hat{\xi}}_1&#x0005E;2&#x0002B;{\hat{\xi}}_2&#x0005E;2\right){\phi}_{{\hat{\xi}}_1,{\hat{\xi}}_2}\mathrm{d}x\\ {}\hfill &amp; \kern0.90em -\frac{1}{2q}\left({\left\Vert {\hat{\xi}}_1\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert {\hat{\xi}}_2\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; \kern1em -\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{\hat{\xi}}_1\right&#x0007C;}&#x0005E;q{\left&#x0007C;{\hat{\xi}}_2\right&#x0007C;}&#x0005E;q\mathrm{d}x\\ {}\hfill &amp; \kern0.60em &#x0002B;\frac{\alpha }{4}\left({\int}_{{\mathbb{R}}&#x0005E;3}\left({\overline{\xi_1}}&#x0005E;2&#x0002B;{\overline{\xi_2}}&#x0005E;2\right){\phi}_{{\hat{\xi}}_1,{\hat{\xi}}_2}\mathrm{d}x\right.\\ {}\hfill &amp; \kern1em \left.&#x0002B;{\int}_{{\mathbb{R}}&#x0005E;3}\left({\hat{\xi}}_1&#x0005E;2&#x0002B;{\hat{\xi}}_2&#x0005E;2\right){\phi}_{\overline{\xi_1},\overline{\xi_2}}\mathrm{d}x\right)\\ {}\hfill &amp; \le I\left(\overline{\xi_1},\overline{\xi_2}\right)&#x0002B;I\left({\hat{\xi}}_1,{\hat{\xi}}_2\right)&#x0002B;\varepsilon .\end{array}} $$

Combining (3.7) and (3.8), we obtain

m a 1 , a 2 ≤ I ξ 1 , ξ 2 ≤ m b 1 , b 2 + m a 1 − b 1 , a 2 − b 2 + 2 ε ; $$ m\left({a}_1,{a}_2\right)\le I\left({\xi}_1,{\xi}_2\right)\le m\left({b}_1,{b}_2\right)&#x0002B;m\left({a}_1-{b}_1,{a}_2-{b}_2\right)&#x0002B;2\varepsilon; $$
thus,
m a 1 , a 2 ≤ m b 1 , b 2 + m a 1 − b 1 , a 2 − b 2 . $$ m\left({a}_1,{a}_2\right)\le m\left({b}_1,{b}_2\right)&#x0002B;m\left({a}_1-{b}_1,{a}_2-{b}_2\right). $$
This completes the proof of the lemma.

Remark 3.3.Note that if we set u s ( x ) = e 3 s 2 u e s x , s ∈ ℝ $$ {u}_s(x)&#x0003D;{e}&#x0005E;{\frac{3s}{2}}u\left({e}&#x0005E;sx\right),\kern0.3em s\in \mathbb{R} $$ , then

ϕ u s ( x ) = ∫ ℝ 3 e 3 s u e s y 2 x − y d y = e s ϕ u e s x . $$ {\phi}_{u_s}(x)&#x0003D;{\int}_{{\mathbb{R}}&#x0005E;3}\frac{e&#x0005E;{3s}{\left&#x0007C;u\left({e}&#x0005E;sy\right)\right&#x0007C;}&#x0005E;2}{\left&#x0007C;x-y\right&#x0007C;} dy&#x0003D;{e}&#x0005E;s{\phi}_u\left({e}&#x0005E;sx\right). $$

To obtain our nonexistence results, we use the fact that any critical point of I ( u , v ) $$ I\left(u,v\right) $$ restricted to S a 1 , a 2 $$ S\left({a}_1,{a}_2\right) $$ satisfies Q ( u , v ) = 0 $$ Q\left(u,v\right)&#x0003D;0 $$ , where
Q ( u , v ) : = | | ∇ u | | 2 2 + | | ∇ v | | 2 2 + α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x − 3 q − 1 2 q | | u | | 2 q 2 q + | | v | | 2 q 2 q − 3 β q − 1 q ∫ ℝ 3 | u | q | v | q d x . $$ {\displaystyle \begin{array}{cc}\hfill Q\left(u,v\right):&#x0003D; &amp; {\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\hfill \\ {}\hfill &amp; -\frac{3\left(q-1\right)}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)\hfill \\ {}\hfill &amp; -\frac{3\beta \left(q-1\right)}{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x.\hfill \end{array}} $$ (3.9)

Indeed, we have the following lemmas.

Lemma 3.4.If u 0 , v 0 $$ \left({u}_0,{v}_0\right) $$ is a critical point of I ( u , v ) $$ I\left(u,v\right) $$ on S a 1 , a 2 $$ S\left({a}_1,{a}_2\right) $$ , then Q u 0 , v 0 = 0 $$ Q\left({u}_0,{v}_0\right)&#x0003D;0 $$ .

Proof.First, we denote

I λ 1 , λ 2 ( u , v ) = 1 2 | | ∇ u | | 2 2 + 1 2 | | ∇ v | | 2 2 − λ 1 2 u 2 2 − λ 2 2 v 2 2 + α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x − 1 2 q ( | | u | | 2 q 2 q + | | v | | 2 q 2 q ) − β q ∫ ℝ 3 | u | q | v | q d x ; $$ {\displaystyle \begin{array}{cc}\hfill {I}_{\lambda_1,{\lambda}_2}\left(u,v\right)&#x0003D;&amp; \frac{1}{2}{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;\frac{1}{2}{\left\Vert \nabla v\right\Vert}_2&#x0005E;2-\frac{\lambda_1}{2}{\left\Vert u\right\Vert}_2&#x0005E;2\\ {}\hfill &amp; -\frac{\lambda_2}{2}{\left\Vert v\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\\ {}\hfill &amp; -\frac{1}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right)-\frac{\beta }{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x;\end{array}} $$
here, λ 1 , λ 2 ∈ R $$ {\lambda}_1,{\lambda}_2\in R $$ , and I λ 1 , λ 2 ( u , v ) $$ {I}_{\lambda_1,{\lambda}_2}\left(u,v\right) $$ is the energy functional corresponding to the equation (1.4).

Clearly,

I λ 1 , λ 2 ( u , v ) = I ( u , v ) − λ 1 2 u 2 2 − λ 2 2 v 2 2 , $$ {I}_{\lambda_1,{\lambda}_2}\left(u,v\right)&#x0003D;I\left(u,v\right)-\frac{\lambda_1}{2}{\left\Vert u\right\Vert}_2&#x0005E;2-\frac{\lambda_2}{2}{\left\Vert v\right\Vert}_2&#x0005E;2, $$
and simple calculations imply that
Q ( u , v ) = 3 2 I λ 1 , λ 2 ′ ( u , v ) , ( u , v ) − P λ 1 , λ 2 ( u , v ) . $$ Q\left(u,v\right)&#x0003D;\frac{3}{2}\left\langle {I}_{\lambda_1,{\lambda}_2}&#x0005E;{\prime}\left(u,v\right),\Big(u,v\Big)\right\rangle -{P}_{\lambda_1,{\lambda}_2}\left(u,v\right). $$
Now, from Lemma 3.1 of [11], we know that P λ 1 , λ 2 ( u , v ) = 0 $$ {P}_{\lambda_1,{\lambda}_2}\left(u,v\right)&#x0003D;0 $$ is a Pohozaev identity for the Hartree–Fock equation (1.4). In particular, any critical point ( u , v ) $$ \left(u,v\right) $$ of I λ 1 , λ 2 ( u , v ) $$ {I}_{\lambda_1,{\lambda}_2}\left(u,v\right) $$ satisfies P λ 1 , λ 2 ( u , v ) = 0 $$ {P}_{\lambda_1,{\lambda}_2}\left(u,v\right)&#x0003D;0 $$ . On the other hand, since u 0 , v 0 $$ \left({u}_0,{v}_0\right) $$ is a critical point of I ( u , v ) $$ I\left(u,v\right) $$ on S a 1 , a 2 $$ S\left({a}_1,{a}_2\right) $$ , there exists a Lagrange multiplier λ 1 , λ 2 ∈ R $$ {\lambda}_1,{\lambda}_2\in R $$ , such that I ′ u 0 , v 0 = λ 1 u 0 , 0 + λ 2 0 , v 0 $$ {I}&#x0005E;{\prime}\left({u}_0,{v}_0\right)&#x0003D;{\lambda}_1\left({u}_0,0\right)&#x0002B;{\lambda}_2\left(0,{v}_0\right) $$ . Thus, for any φ 1 , φ 2 ∈ H 1 ℝ 3 × H 1 ℝ 3 $$ \left({\varphi}_1,{\varphi}_2\right)\in {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ , we have
I λ 1 , λ 2 ′ u 0 , v 0 , φ 1 , φ 2 = I ′ u 0 , v 0 − λ 1 u 0 , 0 − λ 2 0 , v 0 , φ 1 , φ 2 = 0 , $$ {\displaystyle \begin{array}{cc}\hfill &amp; \left\langle {I}_{\lambda_1,{\lambda}_2}&#x0005E;{\prime}\left({u}_0,{v}_0\right),\left({\varphi}_1,{\varphi}_2\right)\right\rangle \hfill \\ {}\hfill &amp; &#x0003D;\left\langle {I}&#x0005E;{\prime}\left({u}_0,{v}_0\right)-{\lambda}_1\left({u}_0,0\right)-{\lambda}_2\left(0,{v}_0\right),\left({\varphi}_1,{\varphi}_2\right)\right\rangle &#x0003D;0,\hfill \end{array}} $$
which shows that u 0 , v 0 $$ \left({u}_0,{v}_0\right) $$ is also a critical point of I λ 1 , λ 2 ( u , v ) $$ {I}_{\lambda_1,{\lambda}_2}\left(u,v\right) $$ . Hence, P λ 1 , λ 2 u 0 , v 0 = 0 $$ {P}_{\lambda_1,{\lambda}_2}\left({u}_0,{v}_0\right)&#x0003D;0 $$ and
I λ 1 , λ 2 ′ ( u 0 , v 0 ) , u 0 , v 0 = 0 , $$ \left\langle {I}_{\lambda_1,{\lambda}_2}&#x0005E;{\prime}\left({u}_0,{v}_0\right),\left({u}_0,{v}_0\right)\right\rangle &#x0003D;0, $$
Q u 0 , v 0 = 0 $$ Q\left({u}_0,{v}_0\right)&#x0003D;0 $$ follows then.

Now, a delicate estimate of the nonlocal term is given, which is available to control the functional I ( u , v ) $$ I\left(u,v\right) $$ and Q ( u , v ) $$ Q\left(u,v\right) $$ .

Lemma 3.5.When 3 2 ≤ q ≤ 2 $$ \frac{3}{2}\le q\le 2 $$ , for any ε > 0 $$ \varepsilon &gt;0 $$ , there are constants C 1 , C 2 > 0 $$ {C}_1,\kern0.3em {C}_2&gt;0 $$ depending on q , ε $$ q,\kern0.3em \varepsilon $$ , such that for any ( u , v ) ∈ S a 1 , a 2 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right) $$ ,

∫ ℝ 3 u 2 + v 2 ϕ u , v d x ≥ − 1 8 π ε 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 + C 1 u 2 q 2 q 4 − 2 q ∇ u 2 3 2 q − 3 4 − 2 q u 2 2 q − 3 4 − 2 q + C 2 v 2 q 2 q 4 − 2 q ∇ v 2 3 2 q − 3 4 − 2 q v 2 2 q − 3 4 − 2 q . $$ {\displaystyle \begin{array}{cc}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x&amp; \ge -\frac{1}{8\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\hfill \\ {}\hfill &amp; \kern1em &#x0002B;{C}_1\frac{{\left\Vert u\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}\hfill \\ {}\hfill &amp; \kern1em &#x0002B;{C}_2\frac{{\left\Vert v\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{{\left\Vert \nabla v\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert v\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}.\hfill \end{array}} $$

Proof.When 3 2 ≤ q ≤ 2 $$ \frac{3}{2}\le q\le 2 $$ , by interpolation, we have

u 2 q 2 q ≤ u 3 3 4 − 2 q u 4 4 2 q − 3 $$ {\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}\le {\left\Vert u\right\Vert}_3&#x0005E;{3\left(4-2q\right)}{\left\Vert u\right\Vert}_4&#x0005E;{4\left(2q-3\right)} $$ (3.10)
Since the ϕ u , v ( x ) ∈ D 1 , 2 ℝ 3 $$ {\phi}_{u,v}(x)\in {D}&#x0005E;{1,2}\left({\mathbb{R}}&#x0005E;3\right) $$ solves the equation
− Δ ϕ u , v = 4 π u 2 + v 2 in ℝ 3 $$ -\Delta {\phi}_{u,v}&#x0003D;4\pi \left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\kern2.00em \mathrm{in}\kern1em {\mathbb{R}}&#x0005E;3 $$ (3.11)
on one hand, multiplying (3.11) by ϕ u , v ( x ) $$ {\phi}_{u,v}(x) $$ and integrating, we obtain
4 π ∫ ℝ 3 u 2 + v 2 ϕ u , v d x = ∫ ℝ 3 ∇ ϕ u , v ( x ) 2 d x $$ 4\pi {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x&#x0003D;{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla {\phi}_{u,v}(x)\right&#x0007C;}&#x0005E;2\mathrm{d}x $$ (3.12)
On the other hand, multiplying (3.11) by u + v $$ \left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C; $$ and integrating, we get for any η > 0 $$ \eta &gt;0 $$ ,
4 π η ∫ ℝ 3 u 2 + v 2 u + v d x = − η ∫ ℝ 3 Δ ϕ u , v ( x ) u + v d x = η ∫ ℝ 3 ∇ ϕ u , v ( x ) ∇ ( u + v ) d x . $$ {\displaystyle \begin{array}{cc}\hfill 4\pi \eta {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x&amp; &#x0003D;-\eta {\int}_{{\mathbb{R}}&#x0005E;3}\Delta {\phi}_{u,v}(x)\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x\hfill \\ {}\hfill &amp; &#x0003D;\eta {\int}_{{\mathbb{R}}&#x0005E;3}\nabla {\phi}_{u,v}(x)\nabla \left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x.\hfill \end{array}} $$ (3.13)

It follows from Young inequality that for any ε > 0 $$ \varepsilon &gt;0 $$ ,

4 π η ∫ ℝ 3 u 2 + v 2 u + v d x ≤ ε ∫ ℝ 3 ∇ ϕ u , v ( x ) 2 d x + η 2 4 ε ∫ ℝ 3 ∇ ( u + v ) 2 d x . $$ {\displaystyle \begin{array}{cc}\hfill &amp; 4\pi \eta {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x\le \varepsilon {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla {\phi}_{u,v}(x)\right&#x0007C;}&#x0005E;2\mathrm{d}x\hfill \\ {}\hfill &amp; &#x0002B;\frac{\eta&#x0005E;2}{4\varepsilon }{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla \left(u&#x0002B;v\right)\right&#x0007C;}&#x0005E;2\mathrm{d}x.\hfill \end{array}} $$ (3.14)

Thus, taking η = 1 $$ \eta &#x0003D;1 $$ in (3.14), combining (3.12) and (3.14), we obtain

4 π ∫ ℝ 3 u 2 + v 2 u + v d x ≤ 4 π ε ∫ ℝ 3 u 2 + v 2 ϕ u , v d x + 1 4 ε ∫ ℝ 3 ∇ ( u + v ) 2 d x . $$ {\displaystyle \begin{array}{cc}\hfill &amp; 4\pi {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x\le 4\pi \varepsilon {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\hfill \\ {}\hfill &amp; &#x0002B;\frac{1}{4\varepsilon }{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla \left(u&#x0002B;v\right)\right&#x0007C;}&#x0005E;2\mathrm{d}x.\hfill \end{array}} $$ (3.15)
Clearly, we observe that
∫ ℝ 3 u 2 + v 2 u + v d x ≥ ∫ ℝ 3 u 3 + v 3 d x $$ {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x\ge {\int}_{{\mathbb{R}}&#x0005E;3}\left({\left&#x0007C;u\right&#x0007C;}&#x0005E;3&#x0002B;{\left&#x0007C;v\right&#x0007C;}&#x0005E;3\right)\mathrm{d}x $$ (3.16)
Then, from (3.15) and (3.16),
u 3 3 + v 3 3 ≤ ε ∫ ℝ 3 u 2 + v 2 ϕ u , v d x + 1 16 π ε ∫ ℝ 3 ∇ u + ∇ v 2 d x $$ {\left\Vert u\right\Vert}_3&#x0005E;3&#x0002B;{\left\Vert v\right\Vert}_3&#x0005E;3\le \varepsilon {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x&#x0002B;\frac{1}{16\pi \varepsilon}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla u&#x0002B;\nabla v\right&#x0007C;}&#x0005E;2\mathrm{d}x $$ (3.17)
is obtained. By (3.17),
∫ ℝ 3 u 2 + v 2 ϕ u , v d x ≥ 1 ε u 3 3 + v 3 3 − 1 16 π ε 2 ∫ ℝ 3 ∇ u + ∇ v 2 d x ≥ 1 ε u 3 3 + v 3 3 − 1 8 π ε 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 . $$ {\displaystyle \begin{array}{cc}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x&amp; \ge \frac{1}{\varepsilon}\left({\left\Vert u\right\Vert}_3&#x0005E;3&#x0002B;{\left\Vert v\right\Vert}_3&#x0005E;3\right)\hfill \\ {}\hfill &amp; \kern1em -\frac{1}{16\pi {\varepsilon}&#x0005E;2}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;\nabla u&#x0002B;\nabla v\right&#x0007C;}&#x0005E;2\mathrm{d}x\hfill \\ {}\hfill &amp; \ge \frac{1}{\varepsilon}\left({\left\Vert u\right\Vert}_3&#x0005E;3&#x0002B;{\left\Vert v\right\Vert}_3&#x0005E;3\right)\hfill \\ {}\hfill &amp; \kern1em -\frac{1}{8\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right).\hfill \end{array}} $$ (3.18)

Now, using Gagliardo-Nirenberg's inequality, there exists a constant C q > 0 $$ {C}_q&gt;0 $$ , such that

u 4 4 ( 2 q − 3 ) ≤ C q ∇ u 2 3 ( 2 q − 3 ) u 2 2 q − 3 $$ {\left\Vert u\right\Vert}_4&#x0005E;{4\left(2q-3\right)}\le {C}_q{\left\Vert \nabla u\right\Vert}_2&#x0005E;{3\left(2q-3\right)}{\left\Vert u\right\Vert}_2&#x0005E;{2q-3} $$ (3.19)
Taking (3.19) into (3.10), we obtain
u 2 q 2 q ≤ C q u 3 3 4 − 2 q ∇ u 2 3 2 q − 3 u 2 2 q − 3 $$ {\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}\le {C}_q{\left\Vert u\right\Vert}_3&#x0005E;{3\left(4-2q\right)}{\left\Vert \nabla u\right\Vert}_2&#x0005E;{3\left(2q-3\right)}{\left\Vert u\right\Vert}_2&#x0005E;{2q-3} $$ (3.20)
Thus,
u 3 3 ≥ C q ˜ u 2 q 2 q 4 − 2 q ∇ u 2 3 2 q − 3 4 − 2 q u 2 2 q − 3 4 − 2 q $$ {\left\Vert u\right\Vert}_3&#x0005E;3\ge \frac{\tilde{C_q}{\left\Vert u\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}} $$ (3.21)
It follows from (3.21) and (3.18) that
∫ ℝ 3 u 2 + v 2 ϕ u , v d x ≥ C q ˜ u 2 q 2 q 4 − 2 q ε ∇ u 2 3 2 q − 3 4 − 2 q u 2 2 q − 3 4 − 2 q + C q ′ ˜ v 2 q 2 q 4 − 2 q ε ∇ v 2 3 2 q − 3 4 − 2 q v 2 2 q − 3 4 − 2 q − 1 8 π ε 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 = C 1 u 2 q 2 q 4 − 2 q ∇ u 2 3 2 q − 3 4 − 2 q u 2 2 q − 3 4 − 2 q + C 2 v 2 q 2 q 4 − 2 q ∇ v 2 3 2 q − 3 4 − 2 q v 2 2 q − 3 4 − 2 q − 1 8 π ε 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 $$ {\displaystyle \begin{array}{cc}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\ge &amp; \frac{\tilde{C_q}{\left\Vert u\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{\varepsilon {\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}\\ {}\hfill &amp; &#x0002B;\frac{\tilde{C_q&#x0005E;{\prime }}{\left\Vert v\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{\varepsilon {\left\Vert \nabla v\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert v\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}\\ {}\hfill &amp; -\frac{1}{8\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &#x0003D;&amp; \frac{C_1{\left\Vert u\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}&#x0002B;\frac{C_2{\left\Vert v\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{{\left\Vert \nabla v\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert v\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}\\ {}\hfill &amp; -\frac{1}{8\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\end{array}} $$

Then, the proof is completed.

The estimate on the nonlocal term leads to a lower bound on Q ( u , v ) $$ Q\left(u,v\right) $$ .

Lemma 3.6.When 3 2 < q < 5 3 $$ \frac{3}{2}&lt;q&lt;\frac{5}{3} $$ and α , β > 0 $$ \alpha, \beta &gt;0 $$ , for any ε > 0 $$ \varepsilon &gt;0 $$ , there are constants C 3 ε , q , α , β , C 4 ε , q , α , β > 0 $$ {C}_3\left(\varepsilon, q,\alpha, \beta \right),\kern0.3em {C}_4\left(\varepsilon, q,\alpha, \beta \right)&gt;0 $$ , such that for any ( u , v ) ∈ S a 1 , a 2 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right) $$ ,

Q ( u , v ) ≥ 32 π ε 2 − α 32 π ε 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 − C 3 ∇ u 2 3 a 1 1 2 − C 4 ∇ v 2 3 a 2 1 2 . $$ {\displaystyle \begin{array}{cc}\hfill Q\left(u,v\right)&amp; \ge \frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\hfill \\ {}\hfill &amp; \kern1em -{C}_3{\left\Vert \nabla u\right\Vert}_2&#x0005E;3{a}_1&#x0005E;{\frac{1}{2}}-{C}_4{\left\Vert \nabla v\right\Vert}_2&#x0005E;3{a}_2&#x0005E;{\frac{1}{2}}.\hfill \end{array}} $$ (3.22)

Proof.By Lemma 3.5, for any ε > 0 $$ \varepsilon &gt;0 $$ , there are constants C 1 > 0 , C 2 > 0 $$ {C}_1&gt;0,\kern0.3em {C}_2&gt;0 $$ depending on ε , q $$ \varepsilon, q $$ , such that, for any ( u , v ) ∈ S a 1 , a 2 , α , β > 0 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right),\kern0.3em \alpha, \beta &gt;0 $$ , there holds

Q ( u , v ) ≥ 32 π ε 2 − α 32 π ε 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 + α C 1 u 2 q 2 q 4 − 2 q ∇ u 2 3 2 q − 3 4 − 2 q u 2 2 q − 3 4 − 2 q + α C 2 v 2 q 2 q 4 − 2 q ∇ v 2 3 2 q − 3 4 − 2 q v 2 2 q − 3 4 − 2 q − 3 q − 1 β + 1 2 q | | u | | 2 q 2 q + | | v | | 2 q 2 q . $$ {\displaystyle \begin{array}{cc}\hfill Q\left(u,v\right)&amp; \ge \frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\hfill \\ {}\hfill &amp; \kern1em &#x0002B;\alpha {C}_1\frac{{\left\Vert u\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{{\left\Vert \nabla u\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert u\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}\hfill \\ {}\hfill &amp; \kern1em &#x0002B;\alpha {C}_2\frac{{\left\Vert v\right\Vert}_{2q}&#x0005E;{\frac{2q}{4-2q}}}{{\left\Vert \nabla v\right\Vert}_2&#x0005E;{\frac{3\left(2q-3\right)}{4-2q}}{\left\Vert v\right\Vert}_2&#x0005E;{\frac{2q-3}{4-2q}}}\hfill \\ {}\hfill &amp; \kern1em -\frac{3\left(q-1\right)\left(\beta &#x0002B;1\right)}{2q}\left({\left\Vert u\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert v\right\Vert}_{2q}&#x0005E;{2q}\right).\hfill \end{array}} $$ (3.23)

To obtain (3.22) from (3.23), we introduce the auxiliary function

f k 1 , k 2 x 1 , x 2 = 32 π ε 2 − α 32 π ε 2 k 1 + k 2 + α D 1 x 1 1 4 − 2 q + α D 2 x 2 1 4 − 2 q − 3 q − 1 β + 1 2 q x 1 + x 2 , x 1 , x 2 > 0 $$ {\displaystyle \begin{array}{cc}\hfill {f}_{k_1,{k}_2}\left({x}_1,{x}_2\right)&amp; &#x0003D;\frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({k}_1&#x0002B;{k}_2\right)&#x0002B;\alpha {D}_1{x}_1&#x0005E;{\frac{1}{4-2q}}&#x0002B;\alpha {D}_2{x}_2&#x0005E;{\frac{1}{4-2q}}\hfill \\ {}\hfill &amp; \kern1em -\frac{3\left(q-1\right)\left(\beta &#x0002B;1\right)}{2q}\left({x}_1&#x0002B;{x}_2\right),\kern1em {x}_1,{x}_2&gt;0\hfill \end{array}} $$
with D 1 = C 1 k 1 3 ( 2 q − 3 ) 2 4 − 2 q · a 1 2 q − 3 2 4 − 2 q − 1 $$ {D}_1&#x0003D;{C}_1{\left({k}_1&#x0005E;{\frac{3\left(2q-3\right)}{2\left(4-2q\right)}}\cdotp {a}_1&#x0005E;{\frac{2q-3}{2\left(4-2q\right)}}\right)}&#x0005E;{-1} $$ , and D 2 = C 2 k 2 3 ( 2 q − 3 ) 2 4 − 2 q · a 2 2 q − 3 2 4 − 2 q − 1 $$ {D}_2&#x0003D;{C}_2{\left({k}_2&#x0005E;{\frac{3\left(2q-3\right)}{2\left(4-2q\right)}}\cdotp {a}_2&#x0005E;{\frac{2q-3}{2\left(4-2q\right)}}\right)}&#x0005E;{-1} $$ . The study of the auxiliary function will provide us with an estimate independent of | | u | | 2 q 2 q , | | v | | 2 q 2 q $$ {\left\Vert u\right\Vert}_{2q}&#x0005E;{2q},\kern0.3em {\left\Vert v\right\Vert}_{2q}&#x0005E;{2q} $$ . Clearly,
∂ f k 1 , k 2 x 1 , x 2 ∂ x i = α 4 − 2 q · D i · x i 2 q − 3 4 − 2 q − 3 q − 1 β + 1 2 q , $$ \frac{\partial {f}_{k_1,{k}_2}\left({x}_1,{x}_2\right)}{\partial {x}_i}&#x0003D;\frac{\alpha }{4-2q}\cdotp {D}_i\cdotp {x}_i&#x0005E;{\frac{2q-3}{4-2q}}-\frac{3\left(q-1\right)\left(\beta &#x0002B;1\right)}{2q}, $$
∂ 2 ∂ x i 2 f k 1 , k 2 x 1 , x 2 = α 4 − 2 q · 2 q − 3 4 − 2 q · D i · x i 4 q − 7 4 − 2 q > 0 , f o r x 1 , x 2 > 0 , i = 1 , 2 . $$ {\displaystyle \begin{array}{cc}\hfill &amp; \frac{\partial&#x0005E;2}{\partial {x}_i&#x0005E;2}{f}_{k_1,{k}_2}\left({x}_1,{x}_2\right)&#x0003D;\frac{\alpha }{4-2q}\cdotp \frac{2q-3}{4-2q}\cdotp {D}_i\cdotp {x}_i&#x0005E;{\frac{4q-7}{4-2q}}&gt;0,\hfill \\ {}\hfill &amp; \kern1em for\kern3.0235pt {x}_1,{x}_2&gt;0,i&#x0003D;1,2.\hfill \end{array}} $$
For convenience, we set M : = 3 q − 1 β + 1 4 − 2 q 2 q $$ M:&#x0003D; \frac{3\left(q-1\right)\left(\beta &#x0002B;1\right)\left(4-2q\right)}{2q} $$ . Therefore, f k 1 , k 2 x 1 , x 2 $$ {f}_{k_1,{k}_2}\left({x}_1,{x}_2\right) $$ has the unique global minimum at
x 1 ‾ , x 2 ‾ = M α D 1 4 − 2 q 2 q − 3 , M α D 2 4 − 2 q 2 q − 3 , $$ \left[\overline{x_1},\overline{x_2}\right]&#x0003D;\left[{\left(\frac{M}{\alpha {D}_1}\right)}&#x0005E;{\frac{4-2q}{2q-3}},{\left(\frac{M}{\alpha {D}_2}\right)}&#x0005E;{\frac{4-2q}{2q-3}}\right], $$
and
f k 1 , k 2 x 1 ‾ , x 2 ‾ = 32 π ε 2 − α 32 π ε 2 k 1 + k 2 + α D 1 M α D 1 1 2 q − 3 + α D 2 M α D 2 1 2 q − 3 − 3 q − 1 β + 1 2 q M α D 1 4 − 2 q 2 q − 3 + M α D 2 4 − 2 q 2 q − 3 = 32 π ε 2 − α 32 π ε 2 k 1 + k 2 + α D 1 2 q − 4 2 q − 3 · M 1 2 q − 3 · 1 − 1 4 − 2 q + α D 2 2 q − 4 2 q − 3 · M 1 2 q − 3 · 1 − 1 4 − 2 q = 32 π ε 2 − α 32 π ε 2 k 1 + k 2 − α 2 q − 4 2 q − 3 C 1 ˜ · M 1 2 q − 3 · 2 q − 3 4 − 2 q · k 1 3 2 · a 1 1 2 − α 2 q − 4 2 q − 3 C 2 ˜ · M 1 2 q − 3 · 2 q − 3 4 − 2 q · k 2 3 2 · a 2 1 2 = 32 π ε 2 − α 32 π ε 2 | | ∇ u | | 2 2 + | | ∇ v | | 2 2 − C 3 ∇ u 2 3 a 1 1 2 − C 4 ∇ v 2 3 a 2 1 2 . $$ {\displaystyle \begin{array}{cc}\hfill {f}_{k_1,{k}_2}\left(\overline{x_1},\overline{x_2}\right)&amp; &#x0003D;\frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({k}_1&#x0002B;{k}_2\right)\\ {}\hfill &amp; \kern1em &#x0002B;\alpha {D}_1{\left(\frac{M}{\alpha {D}_1}\right)}&#x0005E;{\frac{1}{2q-3}}&#x0002B;\alpha {D}_2{\left(\frac{M}{\alpha {D}_2}\right)}&#x0005E;{\frac{1}{2q-3}}\\ {}\hfill &amp; \kern0.90em -\frac{3\left(q-1\right)\left(\beta &#x0002B;1\right)}{2q}\left[{\left(\frac{M}{\alpha {D}_1}\right)}&#x0005E;{\frac{4-2q}{2q-3}}&#x0002B;{\left(\frac{M}{\alpha {D}_2}\right)}&#x0005E;{\frac{4-2q}{2q-3}}\right]\\ {}\hfill &amp; &#x0003D;\frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({k}_1&#x0002B;{k}_2\right)\\ {}\hfill &amp; \kern1em &#x0002B;{\left(\alpha {D}_1\right)}&#x0005E;{\frac{2q-4}{2q-3}}\cdotp {M}&#x0005E;{\frac{1}{2q-3}}\cdotp \left(1-\frac{1}{4-2q}\right)\\ {}\hfill &amp; \kern0.60em &#x0002B;{\left(\alpha {D}_2\right)}&#x0005E;{\frac{2q-4}{2q-3}}\cdotp {M}&#x0005E;{\frac{1}{2q-3}}\cdotp \left(1-\frac{1}{4-2q}\right)\\ {}\hfill &amp; &#x0003D;\frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({k}_1&#x0002B;{k}_2\right)\\ {}\hfill &amp; \kern1em -{\alpha}&#x0005E;{\frac{2q-4}{2q-3}}\tilde{C_1}\cdotp {M}&#x0005E;{\frac{1}{2q-3}}\cdotp \frac{2q-3}{4-2q}\cdotp {k}_1&#x0005E;{\frac{3}{2}}\cdotp {a}_1&#x0005E;{\frac{1}{2}}\\ {}\hfill &amp; \kern0.90em -{\alpha}&#x0005E;{\frac{2q-4}{2q-3}}\tilde{C_2}\cdotp {M}&#x0005E;{\frac{1}{2q-3}}\cdotp \frac{2q-3}{4-2q}\cdotp {k}_2&#x0005E;{\frac{3}{2}}\cdotp {a}_2&#x0005E;{\frac{1}{2}}\\ {}\hfill &amp; &#x0003D;\frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)\\ {}\hfill &amp; \kern1em -{C}_3{\left\Vert \nabla u\right\Vert}_2&#x0005E;3{a}_1&#x0005E;{\frac{1}{2}}-{C}_4{\left\Vert \nabla v\right\Vert}_2&#x0005E;3{a}_2&#x0005E;{\frac{1}{2}}.\end{array}} $$

Because of f k 1 , k 2 x 1 , x 2 ≥ f k 1 , k 2 x 1 ‾ , x 2 ‾ $$ {f}_{k_1,{k}_2}\left({x}_1,{x}_2\right)\ge {f}_{k_1,{k}_2}\left(\overline{x_1},\overline{x_2}\right) $$ for all x 1 , x 2 > 0 $$ {x}_1,{x}_2&gt;0 $$ , we get (3.22).

Next, we are ready to give the proof of Theorem 1.1.

Proof of Theorem 1.1.Assume that u n , v n $$ \left({u}_n,{v}_n\right) $$ is a minimizing sequence with respect to m ( a 1 , a 2 ) $$ m\left({a}_1,{a}_2\right) $$ , then I u n , v n = m a 1 , a 2 + o n ( 1 ) $$ I\left({u}_n,{v}_n\right)&#x0003D;m\left({a}_1,{a}_2\right)&#x0002B;{o}_n(1) $$ . By the coerciveness of I ( u , v ) $$ I\left(u,v\right) $$ on S a 1 , a 2 $$ S\left({a}_1,{a}_2\right) $$ , the sequence u n , v n $$ \left({u}_n,{v}_n\right) $$ is bounded, and so, u n , v n ⇀ ( u , v ) $$ \left({u}_n,{v}_n\right)\rightharpoonup \left(u,v\right) $$ in H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ . By the compactness of the embedding H r 1 ℝ N ⊂ L p ℝ N $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;N\right)\subset {L}&#x0005E;p\left({\mathbb{R}}&#x0005E;N\right) $$ for 2 < p < 6 $$ 2&lt;p&lt;6 $$ , Lemma 2.2, and the weak convergence, the following formulas hold

u n → u in L 2 q ℝ 3 , v n → v in L 2 q ℝ 3 , ∫ ℝ 3 u n 2 + v n 2 ϕ u n , v n d x → ∫ ℝ 3 u 2 + v 2 ϕ u , v d x , ∫ ℝ 3 u n q v n q d x → ∫ ℝ 3 | u | q | v | q d x ; $$ {\displaystyle \begin{array}{cc}\hfill {u}_n&amp; \to u\kern1em \mathrm{in}\kern1em {L}&#x0005E;{2q}\left({\mathbb{R}}&#x0005E;3\right),\\ {}\hfill {v}_n&amp; \to v\kern1em \mathrm{in}\kern1em {L}&#x0005E;{2q}\left({\mathbb{R}}&#x0005E;3\right),\\ {}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}_n&#x0005E;2&#x0002B;{v}_n&#x0005E;2\right){\phi}_{u_n,{v}_n}\mathrm{d}x&amp; \to {\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x,\\ {}\hfill {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{u}_n\right&#x0007C;}&#x0005E;q{\left&#x0007C;{v}_n\right&#x0007C;}&#x0005E;q\mathrm{d}x&amp; \to {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;q{\left&#x0007C;v\right&#x0007C;}&#x0005E;q\mathrm{d}x;\end{array}} $$
thus, we have
m a 1 , a 2 = lim n → ∞ I u n , v n ≥ I ( u , v ) $$ m\left({a}_1,{a}_2\right)&#x0003D;\underset{n\to \infty }{\lim }I\left({u}_n,{v}_n\right)\ge I\left(u,v\right) $$ (3.24)
Assume that u n , v n ⇀ ( u , v ) = ( 0 , 0 ) $$ \left({u}_n,{v}_n\right)\rightharpoonup \left(u,v\right)&#x0003D;\left(0,0\right) $$ in H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ , it follows that m a 1 , a 2 ≥ 0 $$ m\left({a}_1,{a}_2\right)\ge 0 $$ , which contradicts with m a 1 , a 2 < 0 $$ m\left({a}_1,{a}_2\right)&lt;0 $$ . Note that if u 2 2 = a 1 $$ {\left\Vert u\right\Vert}_2&#x0005E;2&#x0003D;{a}_1 $$ and v 2 2 = a 2 $$ {\left\Vert v\right\Vert}_2&#x0005E;2&#x0003D;{a}_2 $$ , we are done. Indeed, from the definition of m a 1 , a 2 $$ m\left({a}_1,{a}_2\right) $$ , we deduce I ( u , v ) ≥ m a 1 , a 2 $$ I\left(u,v\right)\ge m\left({a}_1,{a}_2\right) $$ this moment, this together with (3.24) leads to
m a 1 , a 2 = I ( u , v ) $$ m\left({a}_1,{a}_2\right)&#x0003D;I\left(u,v\right) $$ (3.25)
Therefore, combined with I u n , v n = m a 1 , a 2 + o n ( 1 ) $$ I\left({u}_n,{v}_n\right)&#x0003D;m\left({a}_1,{a}_2\right)&#x0002B;{o}_n(1) $$ , the strong convergence of u n , v n $$ \left({u}_n,{v}_n\right) $$ in H r 1 ℝ 3 × H r 1 ℝ 3 $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ then directly follows. Otherwise, we assume by contradiction that u 2 2 : = b 1 < a 1 $$ {\left\Vert u\right\Vert}_2&#x0005E;2:&#x0003D; {b}_1&lt;{a}_1 $$ or v 2 2 : = b 2 < a 2 $$ {\left\Vert v\right\Vert}_2&#x0005E;2:&#x0003D; {b}_2&lt;{a}_2 $$ . By definition, I ( u , v ) ≥ m b 1 , b 2 $$ I\left(u,v\right)\ge m\left({b}_1,{b}_2\right) $$ , and thus, it results from (3.24) that
m b 1 , b 2 ≤ m a 1 , a 2 $$ m\left({b}_1,{b}_2\right)\le m\left({a}_1,{a}_2\right) $$ (3.26)
At this point, by Lemma 3.2, in case 1 < q < 4 3 $$ 1&lt;q&lt;\frac{4}{3} $$ , m a 1 − b 1 , a 2 − b 2 < 0 $$ m\left({a}_1-{b}_1,{a}_2-{b}_2\right)&lt;0 $$ . In case 4 3 ≤ q < 3 2 $$ \frac{4}{3}\le q&lt;\frac{3}{2} $$ , then there are ρ 1 , ρ 2 > 0 $$ {\rho}_1,{\rho}_2&gt;0 $$ such that m a 1 − b 1 , a 2 − b 2 < 0 $$ m\left({a}_1-{b}_1,{a}_2-{b}_2\right)&lt;0 $$ for all a 1 ∈ 0 , ρ 1 , a 2 ∈ 0 , ρ 2 $$ {a}_1\in \left(0,{\rho}_1\right),{a}_2\in \left(0,{\rho}_2\right) $$ . So we get
m a 1 , a 2 > m b 1 , b 2 + m a 1 − b 1 , a 2 − b 2 , $$ m\left({a}_1,{a}_2\right)&gt;m\left({b}_1,{b}_2\right)&#x0002B;m\left({a}_1-{b}_1,{a}_2-{b}_2\right), $$
which is a contradiction to Lemma 3.2(4) and Theorem 1.1(1) and (2) is proved.

Since there is ( u , v ) ∈ S a 1 , a 2 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right) $$ with m a 1 , a 2 = I ( u , v ) $$ m\left({a}_1,{a}_2\right)&#x0003D;I\left(u,v\right) $$ . By the Lagrange multiplier, there exist λ 1 , λ 2 ∈ ℝ $$ {\lambda}_1,{\lambda}_2\in \mathbb{R} $$ such that

I ′ ( u , v ) = λ 1 ( u , 0 ) + λ 2 ( 0 , v ) . $$ {I}&#x0005E;{\prime}\left(u,v\right)&#x0003D;{\lambda}_1\left(u,0\right)&#x0002B;{\lambda}_2\left(0,v\right). $$
Therefore, we obtain the normalized solution λ 1 , λ 2 , u , v $$ \left({\lambda}_1,{\lambda}_2,u,v\right) $$ of (1.4)–(1.5) in ℝ 2 × H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ {\mathbb{R}}&#x0005E;2\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ for the above several cases.

We consider the non-existence for 3 2 < q < 5 3 $$ \frac{3}{2}&lt;q&lt;\frac{5}{3} $$ . By contradiction, assuming that there are sequence a 1 n ⊂ R + , a 2 n ⊂ R + $$ {a}_1&#x0005E;n\subset {R}&#x0005E;{&#x0002B;},\kern0.3em {a}_2&#x0005E;n\subset {R}&#x0005E;{&#x0002B;} $$ , with a 1 n → 0 , a 2 n → 0 $$ {a}_1&#x0005E;n\to 0,\kern0.3em {a}_2&#x0005E;n\to 0 $$ , as n → ∞ $$ n\to \infty $$ , and u n , v n ⊂ S a 1 n , a 2 n $$ \left\{\left({u}_n,{v}_n\right)\right\}\subset S\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right) $$ such that u n , v n ⊂ S a 1 n , a 2 n $$ \left({u}_n,{v}_n\right)\subset S\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right) $$ is a critical point of I ( u , v ) $$ I\left(u,v\right) $$ restricted to S a 1 n , a 2 n $$ S\left({a}_1&#x0005E;n,{a}_2&#x0005E;n\right) $$ . Then, on the one hand, from Lemma 3.4,

Q u n , v n = | | ∇ u n | | 2 2 + | | ∇ v n | | 2 2 + α 4 ∫ ℝ 3 u n 2 + v n 2 ϕ u n , v n d x − 3 q − 1 2 q | | u n | | 2 q 2 q + | | v n | | 2 q 2 q − 3 β q − 1 q ∫ ℝ 3 | u n | q | v n | q d x = 0 . $$ {\displaystyle \begin{array}{cc}\hfill Q\left({u}_n,{v}_n\right)&amp; &#x0003D;{\left\Vert \nabla {u}_n\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla {v}_n\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}_n&#x0005E;2&#x0002B;{v}_n&#x0005E;2\right){\phi}_{u_n,{v}_n}\mathrm{d}x\\ {}\hfill &amp; \kern1em -\frac{3\left(q-1\right)}{2q}\left({\left\Vert {u}_n\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert {v}_n\right\Vert}_{2q}&#x0005E;{2q}\right)\\ {}\hfill &amp; \kern1em -\frac{3\beta \left(q-1\right)}{q}{\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;{u}_n\right&#x0007C;}&#x0005E;q{\left&#x0007C;{v}_n\right&#x0007C;}&#x0005E;q\mathrm{d}x\\ {}\hfill &amp; &#x0003D;0.\end{array}} $$

Since α > 0 , β > 0 $$ \alpha &gt;0,\kern0.3em \beta &gt;0 $$ and 3 2 < q < 5 3 $$ \frac{3}{2}&lt;q&lt;\frac{5}{3} $$ , naturally, we deduce

| | ∇ u n | | 2 2 + | | ∇ v n | | 2 2 ≤ 3 q − 1 β + 1 2 q | | u n | | 2 q 2 q + | | v n | | 2 q 2 q $$ {\left\Vert \nabla {u}_n\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla {v}_n\right\Vert}_2&#x0005E;2\le \frac{3\left(q-1\right)\left(\beta &#x0002B;1\right)}{2q}\left({\left\Vert {u}_n\right\Vert}_{2q}&#x0005E;{2q}&#x0002B;{\left\Vert {v}_n\right\Vert}_{2q}&#x0005E;{2q}\right) $$ (3.27)
We have, from Gagliardo–Nirenberg's inequality, that for some C 1 > 0 $$ {C}_1&gt;0 $$ and C 2 > 0 $$ {C}_2&gt;0 $$ ,
| | ∇ u n | | 2 2 + | | ∇ v n | | 2 2 ≤ C 1 ( a 1 n ) 3 − q 2 ∇ u n 2 3 q − 1 + C 2 ( a 2 n ) 3 − q 2 ∇ v n 2 3 q − 1 . $$ {\displaystyle \begin{array}{cc}\hfill {\left\Vert \nabla {u}_n\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla {v}_n\right\Vert}_2&#x0005E;2&amp; \le {C}_1{\left({a}_1&#x0005E;n\right)}&#x0005E;{\frac{3-q}{2}}{\left\Vert \nabla {u}_n\right\Vert}_2&#x0005E;{3\left(q-1\right)}\hfill \\ {}\hfill &amp; \kern1em &#x0002B;{C}_2{\left({a}_2&#x0005E;n\right)}&#x0005E;{\frac{3-q}{2}}{\left\Vert \nabla {v}_n\right\Vert}_2&#x0005E;{3\left(q-1\right)}.\hfill \end{array}} $$ (3.28)
Because of 3 q − 1 < 2 $$ 3\left(q-1\right)&lt;2 $$ , we obtain that
| | ∇ u n | | 2 → 0 a n d | | ∇ v n | | 2 → 0 a s n → ∞ $$ {\left\Vert \nabla {u}_n\right\Vert}_2\to 0\kern1em and\kern1em {\left\Vert \nabla {v}_n\right\Vert}_2\to 0\kern2em as\kern2em n\to \infty $$ (3.29)
On the other hand, by Lemma 3.6, it follows that there are constants C 3 ε , q , α , β , C 4 ε , q , α , β > 0 $$ {C}_3\left(\varepsilon, q,\alpha, \beta \right),\kern0.3em {C}_4\left(\varepsilon, q,\alpha, \beta \right)&gt;0 $$ such that
32 π ε 2 − α 32 π ε 2 | | ∇ u n | | 2 2 + | | ∇ v n | | 2 2 ≤ C 3 ∇ u n 2 3 a 1 n 1 2 + C 4 ∇ v n 2 3 a 2 n 1 2 . $$ {\displaystyle \begin{array}{cc}\hfill &amp; \frac{32\pi {\varepsilon}&#x0005E;2-\alpha }{32\pi {\varepsilon}&#x0005E;2}\left({\left\Vert \nabla {u}_n\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla {v}_n\right\Vert}_2&#x0005E;2\right)\hfill \\ {}\hfill &amp; \kern1em \le {C}_3{\left\Vert \nabla {u}_n\right\Vert}_2&#x0005E;3{\left({a}_1&#x0005E;n\right)}&#x0005E;{\frac{1}{2}}&#x0002B;{C}_4{\left\Vert \nabla {v}_n\right\Vert}_2&#x0005E;3{\left({a}_2&#x0005E;n\right)}&#x0005E;{\frac{1}{2}}.\hfill \end{array}} $$ (3.30)
According to the arbitrariness of ε $$ \varepsilon $$ , we can take ε > α 32 π $$ \varepsilon &gt;\sqrt{\frac{\alpha }{32\pi }} $$ , then (3.30) implies that
| | ∇ u n | | 2 → ∞ o r | | ∇ v n | | 2 → ∞ a s n → ∞ , $$ {\left\Vert \nabla {u}_n\right\Vert}_2\to \infty \kern1em or\kern1em {\left\Vert \nabla {v}_n\right\Vert}_2\to \infty \kern2em as\kern2em n\to \infty, $$
which are contradictory to (3.29). Thus, we finish the proof of Theorem 1.1(3).

Now, when q = 3 2 $$ q&#x0003D;\frac{3}{2} $$ , it is enough to prove that, for any a 1 . a 2 > 0 $$ {a}_1.{a}_2&gt;0 $$ , there holds Q ( u , v ) > 0 $$ Q\left(u,v\right)&gt;0 $$ for all ( u , v ) ∈ S ( a 1 , a 2 ) $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right) $$ . Indeed, if Q ( u , v ) > 0 $$ Q\left(u,v\right)&gt;0 $$ holds true, we can conclude the nonexistence of minimizers directly from Lemma 3.4.

To check Q ( u , v ) > 0 $$ Q\left(u,v\right)&gt;0 $$ for all ( u , v ) ∈ S a 1 , a 2 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right) $$ , let η = 2 $$ \eta &#x0003D;2 $$ in (3.13) and ε = 1 $$ \varepsilon &#x0003D;1 $$ in (3.14), then from (3.12) and (3.14), we get

α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x ≥ α 2 ∫ ℝ 3 u 2 + v 2 u + v d x − α 16 π ∫ ℝ 3 ∇ ( u + v ) 2 d x . $$ {\displaystyle \begin{array}{cc}\hfill \frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x&amp; \ge \frac{\alpha }{2}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x\hfill \\ {}\hfill &amp; \kern1em -\frac{\alpha }{16\pi }{\int}_{{\mathbb{R}}&#x0005E;3}\left(\nabla {\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)}&#x0005E;2\mathrm{d}x\right).\hfill \end{array}} $$ (3.31)
Thus, for any ( u , v ) ∈ S a 1 , a 2 , q = 3 2 $$ \left(u,v\right)\in S\left({a}_1,{a}_2\right),\kern0.3em q&#x0003D;\frac{3}{2} $$ ,
Q ( u , v ) = | | ∇ u | | 2 2 + | | ∇ v | | 2 2 + α 4 ∫ ℝ 3 u 2 + v 2 ϕ u , v d x − 1 2 | | u | | 3 3 + | | v | | 3 3 − β ∫ ℝ 3 | u | 3 2 | v | 3 2 d x ≥ | | ∇ u | | 2 2 + | | ∇ v | | 2 2 + α 2 ∫ ℝ 3 u 2 + v 2 u + v d x − α 16 π ∫ ℝ 3 ∇ ( u + v ) 2 d x − 1 2 | | u | | 3 3 + | | v | | 3 3 − β ∫ ℝ 3 | u | 3 2 | v | 3 2 d x ≥ 1 − α 8 π | | ∇ u | | 2 2 + | | ∇ v | | 2 2 + α − 1 2 | | u | | 3 3 + | | v | | 3 3 + α − β ∫ ℝ 3 | u | 3 2 | v | 3 2 d x . $$ {\displaystyle \begin{array}{cc}\hfill Q\left(u,v\right)&amp; &#x0003D;{\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{4}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right){\phi}_{u,v}\mathrm{d}x\\ {}\hfill &amp; \kern1em -\frac{1}{2}\left({\left\Vert u\right\Vert}_3&#x0005E;3&#x0002B;{\left\Vert v\right\Vert}_3&#x0005E;3\right)-\beta {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;{\frac{3}{2}}{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\frac{3}{2}}\mathrm{d}x\\ {}\hfill &amp; \ge {\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2&#x0002B;\frac{\alpha }{2}{\int}_{{\mathbb{R}}&#x0005E;3}\left({u}&#x0005E;2&#x0002B;{v}&#x0005E;2\right)\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)\mathrm{d}x\\ {}\hfill &amp; \kern0.90em -\frac{\alpha }{16\pi }{\int}_{{\mathbb{R}}&#x0005E;3}\left(\nabla {\left(\left&#x0007C;u\right&#x0007C;&#x0002B;\left&#x0007C;v\right&#x0007C;\right)}&#x0005E;2\right)\mathrm{d}x-\frac{1}{2}\left({\left\Vert u\right\Vert}_3&#x0005E;3&#x0002B;{\left\Vert v\right\Vert}_3&#x0005E;3\right)\\ {}\hfill &amp; \kern1em -\beta {\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;{\frac{3}{2}}{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\frac{3}{2}}\mathrm{d}x\\ {}\hfill &amp; \ge \left(1-\frac{\alpha }{8\pi}\right)\left({\left\Vert \nabla u\right\Vert}_2&#x0005E;2&#x0002B;{\left\Vert \nabla v\right\Vert}_2&#x0005E;2\right)&#x0002B;\frac{\alpha -1}{2}\left({\left\Vert u\right\Vert}_3&#x0005E;3&#x0002B;{\left\Vert v\right\Vert}_3&#x0005E;3\right)\\ {}\hfill &amp; \kern1em &#x0002B;\left(\alpha -\beta \right){\int}_{{\mathbb{R}}&#x0005E;3}{\left&#x0007C;u\right&#x0007C;}&#x0005E;{\frac{3}{2}}{\left&#x0007C;v\right&#x0007C;}&#x0005E;{\frac{3}{2}}\mathrm{d}x.\end{array}} $$

Since 1 ≤ α < 8 π $$ 1\le \alpha &lt;8\pi $$ , and 0 < β < α $$ 0&lt;\beta &lt;\alpha $$ , we can get Q ( u , v ) > 0 $$ Q\left(u,v\right)&gt;0 $$ . At this point, the proof is complete.

4 Proof of Theorem 1.3

In this section, we prove Theorem 1.3 following the classical arguments of [28, 29].

Proof of Theorem 1.3.First of all, we notice explicitly that G a 1 , a 2 $$ G\left({a}_1,{a}_2\right) $$ is invariant by translation; that is, if ( u , v ) ∈ G a 1 , a 2 $$ \left(u,v\right)\in G\left({a}_1,{a}_2\right) $$ , then also u ( · − y ) , v ( · − y ) ∈ G a 1 , a 2 $$ \left(u\left(\cdotp -y\right),v\left(\cdotp -y\right)\right)\in G\left({a}_1,{a}_2\right) $$ for any y ∈ ℝ 3 $$ y\in {\mathbb{R}}&#x0005E;3 $$ . We argue by contradiction, assuming that there exist a 1 $$ {a}_1 $$ and a 2 > 0 $$ {a}_2&gt;0 $$ such that G a 1 , a 2 $$ G\left({a}_1,{a}_2\right) $$ is not orbitally stable. This means that there is a ϵ 0 > 0 $$ {\epsilon}_0&gt;0 $$ , and a sequence of initial Ψ 1 n ( 0 ) , Ψ 2 n ( 0 ) ⊂ H r 1 ℝ 3 × H r 1 ℝ 3 $$ \left({\Psi}_1&#x0005E;n(0),{\Psi}_2&#x0005E;n(0)\right)\subset {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ and t n ⊂ ℝ + $$ \left\{{t}_n\right\}\subset {\mathbb{R}}&#x0005E;{&#x0002B;} $$ such that

inf ( u , v ) ∈ G a 1 , a 2 ( Ψ 1 n ( 0 ) , Ψ 2 n ( 0 ) ) − ( u , v ) H 1 → 0 $$ \underset{\left(u,v\right)\in G\left({a}_1,{a}_2\right)}{\operatorname{inf}}{\left\Vert \Big({\Psi}_1&#x0005E;n(0),{\Psi}_2&#x0005E;n(0)\left)-\right(u,v\Big)\right\Vert}_{H&#x0005E;1}\to 0 $$
and
inf ( u , v ) ∈ G a 1 , a 2 Ψ 1 n · , t n , Ψ 2 n · , t n − ( u , v ) H 1 ≥ ε 0 $$ \underset{\left(u,v\right)\in G\left({a}_1,{a}_2\right)}{\operatorname{inf}}{\left\Vert \left({\Psi}_1&#x0005E;n\left(\cdotp, {t}_n\right),{\Psi}_2&#x0005E;n\left(\cdotp, {t}_n\right)\right)-\left(u,v\right)\right\Vert}_{H&#x0005E;1}\ge {\varepsilon}_0 $$ (4.1)
Since by the conservation laws, the energy and the charge associated with Ψ i · , t $$ {\Psi}_i\left(\cdotp, t\right) $$ i = 1 , 2 $$ i&#x0003D;1,2 $$ . satisfies I Ψ 1 n ( · , t n ) , Ψ 2 n ( · , t n ) = I Ψ 1 n · , 0 , Ψ 2 n · , 0 $$ I\left({\Psi}_1&#x0005E;n\left(\cdotp, {t}_n\right),{\Psi}_2&#x0005E;n\Big(\cdotp, {t}_n\Big)\right)&#x0003D;I\left({\Psi}_1&#x0005E;n\left(\cdotp, 0\right),{\Psi}_2&#x0005E;n\left(\cdotp, 0\right)\right) $$ , and Ψ i n ( · , t n ) 2 2 = Ψ i n ( 0 ) 2 2 $$ {\left\Vert {\Psi}_i&#x0005E;n\left(\cdotp, {t}_n\right)\right\Vert}_2&#x0005E;2&#x0003D;{\left\Vert {\Psi}_i&#x0005E;n(0)\right\Vert}_2&#x0005E;2 $$ , for i = 1 , 2 $$ i&#x0003D;1,2 $$ . Define
Ψ i n ˜ ( · , t n ) = Ψ i n ( · , t n ) Ψ i n ( · , t n ) 2 a i 1 2 , for i = 1 , 2 , $$ \tilde{\Psi_i&#x0005E;n}\left(\cdotp, {t}_n\right)&#x0003D;\frac{\Psi_i&#x0005E;n\left(\cdotp, {t}_n\right)}{{\left\Vert {\Psi}_i&#x0005E;n\left(\cdotp, {t}_n\right)\right\Vert}_2}{a}_i&#x0005E;{\frac{1}{2}},\mathrm{for}\kern1em i&#x0003D;1,2, $$
we have
Ψ ˜ i n ( · , t n ) 2 2 = a i , for i = 1 , 2 , and I Ψ 1 n ˜ , Ψ 2 n ˜ = m a 1 , a 2 + o n ( 1 ) . $$ {\displaystyle \begin{array}{cc}\hfill &amp; {\left\Vert {\tilde{\Psi}}_i&#x0005E;n\left(\cdotp, {t}_n\right)\right\Vert}_2&#x0005E;2&#x0003D;{a}_i,\mathrm{for}\kern1em i&#x0003D;1,2,\mathrm{and}\hfill \\ {}\hfill &amp; I\left(\tilde{\Psi_1&#x0005E;n},\tilde{\Psi_2&#x0005E;n}\right)&#x0003D;m\left({a}_1,{a}_2\right)&#x0002B;{o}_n(1).\hfill \end{array}} $$
So we find a minimizing sequence ( Ψ 1 n ˜ , Ψ 2 n ˜ ) $$ \left(\tilde{\Psi_1&#x0005E;n},\tilde{\Psi_2&#x0005E;n}\right) $$ with respect to m a 1 , a 2 $$ m\left({a}_1,{a}_2\right) $$ . However, according to Theorem 1.1 (1) and (2), the minimizing sequence is precompact (up to translation) in H r 1 ( ℝ 3 ) × H r 1 ( ℝ 3 ) $$ {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right)\times {H}_r&#x0005E;1\left({\mathbb{R}}&#x0005E;3\right) $$ , which contradicts (4.1). The proof is completed.

Author Contributions

Hua Jin: conceptualization, investigation. Yanyun Chang: conceptualization, investigation. Marco Squassina: conceptualization, investigation. Jianjun Zhang: conceptualization, investigation.

Acknowledgments

H. Jin and Y. Chang were supported by the Fundamental Research Funds for the Central Universities (2019XKQYMS90). M. Squassina was supported by Gruppo Nazionale per l'Analisi Matematica, la Probabilità e loro Applicazioni. J.J. Zhang was supported by NSFC (Nos. 12371109 and 11871123). Open access publishing facilitated by Universita Cattolica del Sacro Cuore, as part of the Wiley - CRUI-CARE agreement.

    Conflicts of Interest

    The authors declare no conflicts of interest.

    Data Availability Statement

    The authors have nothing to report.

      The full text of this article hosted at iucr.org is unavailable due to technical difficulties.