Ground State Solutions for Fractional Choquard Equations with Potential Vanishing at Infinity

Abstract

In this paper, we study a class of nonlinear Choquard equation driven by the fractional Laplacian. When the potential function vanishes at infinity, we obtain the existence of a ground state solution for the fractional Choquard equation by using a non-Nehari manifold method. Moreover, in the zero mass case, we obtain a nontrivial solution by using a perturbation method. The results improve upon those in Alves, Figueiredo, and Yang (2015) and Shen, Gao, and Yang (2016).

1. Introduction

In this paper, we deal with the following nonlocal equation:

$$\left\{\begin{array}{c}{(-\mathsf{\Delta })}^{s}u+V\left(x\right)u=\left({\int }_{{\mathbb{R}}^N}\frac{Q\left(y\right)F\left(u\right(y\left)\right)}{{|x-y|}^{\mu }}dy\right)Q\left(x\right)f\left(u\right),\mathrm{in}{\mathbb{R}}^N,\hfill \\ u\in D^{s,2}\left({\mathbb{R}}^N\right),\hfill \end{array}\right.$$

(1)

where $N\ge 3$, $0<s<1$, $0<\mu <N$, $V\in C({\mathbb{R}}^N,[0,\infty )),Q\in C({\mathbb{R}}^N,(0,\infty ))$, $f\in C(\mathbb{R},\mathbb{R})$ and $F\left(t\right)={\int }_0^{t}f\left(s\right)ds$. The fractional Laplacian ${(-\mathsf{\Delta })}^s$ is defined as

$${(-\mathsf{\Delta })}^{s}u\left(x\right)=C_{N,s}P.V.{\int }_{{\mathbb{R}}^N}\frac{u\left(x\right)-u\left(y\right)}{{|x-y|}^{N+2s}}dy,u\in \mathcal{S}\left({\mathbb{R}}^N\right),$$

where P.V. denotes the $principalvalue$ of the singular integral, $\mathcal{S}\left({\mathbb{R}}^N\right)$ is the Schwartz space of rapidly decaying $C^{\infty }$ functions in ${\mathbb{R}}^N$, and

$$C_{N,s}=\frac{2^{2s}s\mathsf{\Gamma }(N+s)}{{\pi }^{N/2}\mathsf{\Gamma }(1-s)}.$$

${(-\mathsf{\Delta })}^s$ is a pseudo-differential operator, and can be equivalently defined via Fourier transform as

$$\mathcal{F}\left[{(-\mathsf{\Delta })}^{s}u\right]\left(\xi \right)={\left|\xi \right|}^{2s}\mathcal{F}\left[u\right]\left(\xi \right),u\in \mathcal{S}\left({\mathbb{R}}^N\right),$$

where $\mathcal{F}$ is the Fourier transform, that is,

$$\mathcal{F}\left[u\right]\left(\xi \right)=\frac{1}{{\left(2\pi \right)}^{\frac{N}{2}}}{\int }_{{\mathbb{R}}^N}{e}^{-i\xi ·x}u\left(x\right)dx,u\in \mathcal{S}\left({\mathbb{R}}^N\right).$$

The fractional Laplace operator ${(-\mathsf{\Delta })}^s$ is the infinitesimal generator of Lévy stable diffusion processes, and appears in several areas such as the thin obstacle problem, anomalous diffusion, optimization, finance, phase transitions, crystal dislocation, multiple scattering, and materials science, see [1,2,3,4,5] and their references.

Recently, a great deal of work has been devoted to the study of the Choquard equations, see [6,7,8,9,10,11,12,13,14] and their references. For instance, Alves, Cassani, Tarsi, and Yang [7] studied the following singularly perturbed nonlocal Schrödinger equation:

$$-{\epsilon }^2\mathsf{\Delta }u+V\left(x\right)u={\epsilon }^{\mu -2}\left[\frac{1}{{\left|x\right|}^{\mu }}\ast F\left(u\right)\right]f\left(u\right),\mathrm{in}{\mathbb{R}}^2,$$

where $0<\mu <2$ and $\epsilon$ is a positive parameter, the nonlinearity f has critical exponential growth in the sense of Trudinger–Moser. By using variational methods, the authors established the existence and concentration of solutions for the above equation.

In [6], Alves, Figueiredo and Yang studied the following Choquard equation:

$$\left\{\begin{array}{c}-\mathsf{\Delta }u+V\left(x\right)u=(\frac{1}{{\left|x\right|}^{\mu }}\ast F\left(u\right))f\left(u\right),\mathrm{in}{\mathbb{R}}^N.\hfill \\ u\in H^1\left({\mathbb{R}}^N\right).\hfill \end{array}\right.$$

(2)

Under the assumption $V\left(x\right)\to 0$ as $\left|x\right|\to \infty$, the authors obtained a nontrivial solution for (2) by using a penalization method.

In the physical case $N=3,\mu =1$, $V\left(x\right)=1$ and $F\left(t\right)=\frac{t^2}{2}$, (2) is also known as the stationary Hartree equation [15]. It dates back to the description of the quantum mechanics of a polaron at rest by Pekar in 1954 [16]. In 1976, Choquard used (2) to describe an electron trapped in its own hole, in a certain approximation to the Hartree–Fock theory of one-component plasma [11]. In 1996, Penrose proposed (2) as a model of self-gravitating matter, in a programme in which quantum state reduction is understood as a gravitational phenomenon [15].

In addition, there is little literature on the fractional Choquard equations. Frank and Lenzmann [17] established the uniqueness and radial symmetry of ground state solutions for the following equation:

$${(-\mathsf{\Delta })}^{\frac{1}{2}}u+u={\left(\right|x|}^{-1}\ast {\left|u\right|}^2)u,\mathrm{in}{\mathbb{R}}^N.$$

D’Avenia, Siciliano, and Squassina [18] obtained the existence, regularity, symmetry, and asymptotic of the solutions for the nonlocal problem

$${(-\mathsf{\Delta })}^{s}u+\omega u={\left(\right|x|}^{-\mu }\ast {\left|u\right|}^p{\left)\right|u|}^{p-2}u,\mathrm{in}{\mathbb{R}}^N.$$

In [19], Shen, Gao, and Yang studied the following fractional Choquard equation:

$${(-\mathsf{\Delta })}^{s}u+u={\left(\right|x|}^{-\mu }\ast F\left(u\right))f\left(u\right),\mathrm{in}{\mathbb{R}}^N,$$

(3)

where $N\ge 3,s\in (0,1)$, and $\mu \in (0,N)$. Under the general Berestycki–Lions-type conditions [20], the authors obtained the existence and regularity of ground states for (3). The authors also established the Pohožaev identity for (3):

$$\frac{N-2s}{2}{\int }_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u|}^{2}dx+\frac{N}{2}{\int }_{{\mathbb{R}}^N}{u}^{2}dx=\frac{2N-\mu }{2}{\int }_{{\mathbb{R}}^N}{\left(\right|x|}^{-\mu }\ast F\left(u\right))F\left(u\right)dx.$$

Motivated by the above works, in the first part of this article, we study the ground state solution for (1). We assume

(I)

$V\left(x\right),Q\left(x\right)>0$ for all $x\in {\mathbb{R}}^N,V\in C({\mathbb{R}}^N,\mathbb{R})$ and $Q\in C({\mathbb{R}}^N,\mathbb{R})\cap L^{\infty }({\mathbb{R}}^N,\mathbb{R})$;

(II)

if $\left\{A_n\right\}\subset {\mathbb{R}}^N$ is a sequence of Borel sets such that $\mathrm{meas}\left\{A_n\right\}\le \delta$ for all n and some $\delta >0$, then

$$\underset{r\to \infty }{lim}{\int }_{A_n\cap B_r^c\left(0\right)}{\left[Q\left(x\right)\right]}^{\frac{2N}{2N-\mu }}dx=0\mathrm{uniformly}\mathrm{in}n\in \mathbb{N};$$

(III)

one of the below conditions occurs:

$$\frac{Q}{V}\in L^{\infty }\left({\mathbb{R}}^N\right),$$

(4)

or there exists $p\in (2,2_s^{\ast })$ such that

$$\frac{{\left[Q\left(x\right)\right]}^{\frac{2N}{2N-\mu }}}{{\left[V\left(x\right)\right]}^{\frac{2_s^{\ast }-p}{2_s^{\ast }-2}}}\to 0,\left|x\right|\to \infty ,$$

(5)

where $2_s^{\ast }=\frac{2N}{N-2s}$ is the fractional critical exponent;

($F1$)

$F\left(t\right)=o\left(\right|t{|}^{\frac{2N-\mu }{N}})$ as $t\to 0$ if $\left(4\right)$ holds; or $F\left(t\right)=o\left(\right|t{|}^{\frac{p(2N-\mu )}{2N}})$ as $t\to 0$ if $\left(5\right)$ holds;

($F2$)

$F\left(t\right)=o\left(\right|t{|}^{\frac{2N-\mu }{N-2s}})$ as $t\to \infty$;

($F3$)

$f\left(t\right)$ is nondecreasing on $\mathbb{R}$;

($F4$)

$\underset{\left|t\right|\to +\infty }{lim}\frac{F\left(t\right)}{\left|t\right|}=+\infty$.

It is necessary for us to point out that the original of assumptions (I)–(III) come from [21,22,23]. The assumptions can be used to prove that the work space E is compactly embedded into the weighted Lebesgue space $L_K^q\left({\mathbb{R}}^N\right)$, see Section 2 and Lemma 1.

Now, we can state the first result of this article.

Theorem 1.

Suppose that $\left(I\right),\left(II\right),\left(III\right)$ and (F1)–(F4) hold. Then (1) has a ground state solution.

Remark 1.

Since the Nehari-type monotonicity condition for f is not satisfied, the Nehari manifold method used in [24] no longer works in our setting. To prove Theorem 2, we use the non-Nehari manifold method developed by Tang [25], which relies on finding a minimizing sequence outside the Nehari manifold by using the diagonal method (see Lemma 8).

In the second part of this article, we consider the following fractional Choquard equation with zero mass case:

$$\left\{\begin{array}{c}{(-\mathsf{\Delta })}^{s}u=\left(\frac{1}{{\left|x\right|}^{\mu }}\ast F\left(u\right)\right)f\left(u\right),\mathrm{in}{\mathbb{R}}^N,\hfill \\ u\in D^{s,2}\left({\mathbb{R}}^N\right),\hfill \end{array}\right.$$

(6)

where $N\ge 3$, $0<s<1$, $0<\mu <min\{N,4s\}$. The homogeneous fractional Sobolev space $D^{s,2}\left({\mathbb{R}}^N\right)$, also denoted by ${\dot{H}}^s\left({\mathbb{R}}^N\right)$, can be characterized as the space

$$D^{s,2}\left({\mathbb{R}}^N\right)=\left\{u\in L^{2_s^{\ast }}\left({\mathbb{R}}^N\right):{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{{|u\left(x\right)-u\left(y\right)|}^2}{{|x-y|}^{N+2s}}dxdy<+\infty \right\}.$$

$f\in C(\mathbb{R},\mathbb{R})$ satisfy the following Berestycki–Lions-type condition [19,20]:

($F5$)

F is not trivial, that is, $F\equiv 0$;

($F6$)

there exists $C>0$ such that for every $t\in \mathbb{R}$,

$$\left|tf\left(t\right)\right|\le {C\left|t\right|}^{\frac{2N-\mu }{N-2s}};$$

($F7$)

$$\underset{t\to 0}{lim}\frac{F\left(t\right)}{{\left|t\right|}^2}=\underset{t\to \infty }{lim}\frac{F\left(t\right)}{{\left|t\right|}^{\frac{2N-\mu }{N-2s}}}=0.$$

The second result of this paper is as follows.

Theorem 2.

Suppose that f satisfies (F5)–(F7). Then (6) has a nontrivial solution.

Remark 2.

Notice that the method used in [13] is no longer applicable for (6), because it relies heavily on the constant potentials. In the zero mass case, we use the perturbation method and the Pohožaev identity established in [19] to overcome this difficulty.

In this article, we make use of the following notation:

• ${\parallel ·\parallel }_p$ denotes the usual norm of $L^p\left({\mathbb{R}}^3\right)$;
• $C,C_i,i=1,2,\dots ,$ denote various positive constants whose exact values are irrelevant;
• $o\left(1\right)$ denotes the infinitesimal as $n\to +\infty$.

2. Ground State Solutions for (1)

Set

$$D^{s,2}\left({\mathbb{R}}^N\right):=\left\{u\in L^{2_s^{\ast }}\left({\mathbb{R}}^N\right):{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{{|u\left(x\right)-u\left(y\right)|}^2}{{|x-y|}^{N+2s}}dxdy<+\infty \right\},$$

endowed with the Gagliardo (semi)norm

$$\left[u\right]:={\left({\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{{|u\left(x\right)-u\left(y\right)|}^2}{{|x-y|}^{N+2s}}dxdy\right)}^{1/2}.$$

From [5], we have the following identity:

$${\left[u\right]}^2={\int }_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u|}^{2}dx={\int }_{{\mathbb{R}}^N}{\left|\xi \right|}^{2s}{\left|\mathcal{F}\left[u\right]\left(\xi \right)\right|}^{2}d\xi .$$

From [26], $D^{s,2}\left({\mathbb{R}}^N\right)$ is continuously embedded into $L^{2_s^{\ast }}\left({\mathbb{R}}^N\right)$. Then, we can define the best constant $S>0$ as

$$S:=\underset{u\in D^{s,2}\left({\mathbb{R}}^N\right)}{sup}\frac{{\left({\int }_{{\mathbb{R}}^N}{\left|u\right|}^{2_s^{\ast }}dx\right)}^{\frac{2}{2_s^{\ast }}}}{{\int }_{{\mathbb{R}}^N}{\left|{(-\mathsf{\Delta })}^{\frac{s}{2}}u\right|}^{2}dx}.$$

Let

$$E:=\left\{u\in D^{s,2}\left({\mathbb{R}}^N\right):{\int }_{{\mathbb{R}}^N}V\left(x\right)u^{2}dx<+\infty \right\}.$$

Under the assumptions (I)–(III), following the idea of ([21], Proposition 2.1) or ([22], Proposition 2.2), we can prove that the Hilbert space E endowed with scalar product and norm

$$(u,v)={\int }_{{\mathbb{R}}^N}[{(-\mathsf{\Delta })}^{\frac{s}{2}}u{(-\mathsf{\Delta })}^{\frac{s}{2}}v+V\left(x\right)uv]dx,\parallel u\parallel ={\left({\int }_{{\mathbb{R}}^N}{\left[\right|(-\mathsf{\Delta })}^{\frac{s}{2}}u{|}^2+V\left(x\right)u^2]dx\right)}^{\frac{1}{2}}$$

is compactly embedded into the weighted space $L_K^q\left({\mathbb{R}}^N\right)$ for every $q\in (2,2_s^{\ast })$, where $K\left(x\right):={\left[Q\left(x\right)\right]}^{2N/(2N-\mu )}$ and

$$L_K^q\left({\mathbb{R}}^N\right):=\left\{u:\mathrm{meas}\left\{u\right\}<\infty \mathrm{and}{\int }_{{\mathbb{R}}^N}K\left(x\right){\left|u\right|}^{q}dx<\infty \right\},\forall q\ge 2.$$

Lemma 1.

Assume that (I)–(III) hold. If $\left(K1\right)$ holds, E is compactly embedded in $L_K^q\left({\mathbb{R}}^N\right)$ for all $q\in (2,2_s^{\ast })$. If $\left(K2\right)$ holds, E is compactly embedded in $L_K^p\left({\mathbb{R}}^N\right)$.

Proof. 

If $\left(K1\right)$ holds, then

$$\frac{K\left(x\right)}{V\left(x\right)}=\frac{Q\left(x\right)}{V\left(x\right)}{\left[Q\left(x\right)\right]}^{\frac{\mu }{2N-\mu }}\in L^{\infty }\left({\mathbb{R}}^N\right).$$

Given $\epsilon >0$ and fixed $q\in (2,2_s^{\ast })$, there exist $0<t_0<t_1$ and $C>0$ such that

$${K\left(x\right)\left|t\right|}^q\le {\epsilon C\left(V\left(x\right)\right|t|}^2+{\left|t\right|}^{2_s^{\ast }})+CK\left(x\right){\chi }_{[t_0,t_1]}{\left(\right|t\left|\right)\left|t\right|}^{2_s^{\ast }}\forall t\in \mathbb{R}.$$

Hence,

$${\int }_{B_r^c\left(0\right)}K\left(x\right){\left|u\right|}^{q}dx\le \epsilon CW\left(u\right)+CK\left(x\right){\int }_{A\cap B_r^c\left(0\right)}K\left(x\right)dx\forall u\in E,$$

(7)

where

$$W\left(u\right)={\int }_{{\mathbb{R}}^N}{V\left(x\right)\left|u\right|}^{2}dx+{\int }_{{\mathbb{R}}^N}{\left|u\right|}^{2_s^{\ast }}dx$$

and

$$A=\{x\in {\mathbb{R}}^N:s_0\le |u\left(x\right)|\le s_1\}.$$

Let $\left\{v_n\right\}$ be a sequence such that $v_n\rightharpoonup v$ in E, then there exists a constant $M_1>0$ such that

$${\int }_{{\mathbb{R}}^N}{\left[\right|(-\mathsf{\Delta })}^{\frac{s}{2}}{v}_n{|}^2+V\left(x\right)|v_n{|}^2]dx\le M_1\mathrm{and}{\int }_{{\mathbb{R}}^N}{\left|v_n\right|}^{2_s^{\ast }}dx\le M_1\forall n\in \mathbb{N},$$

which implies that $\left\{W\left(v_n\right)\right\}$ is bounded. On the other hand, setting

$$A_n=\{x\in {\mathbb{R}}^N:s_0\le |v_n\left(x\right)|\le s_1\},$$

we have

$$s_0^{2_s^{\ast }}|A_n|\le {\int }_{A_n}{\left|v_n\right|}^{2_s^{\ast }}dx\le M_1\forall n\in \mathbb{N}$$

and so $\underset{n\in \mathbb{N}}{sup}\left|A_n\right|<+\infty$. Therefore, from $\left(II\right)$, there is $r>0$ such that

$${\int }_{A_n\cap B_r^c\left(0\right)}K\left(x\right)dx<\frac{\epsilon }{s_1^{2_s^{\ast }}}\forall n\in \mathbb{N}.$$

(8)

Combining (7) and (8), we have

$${\int }_{B_r^c\left(0\right)}K\left(x\right){\left|v_n\right|}^{q}dx<\epsilon C{M}_1+s_1^{2_s^{\ast }}{\int }_{F_n\cap B_r^c\left(0\right)}K\left(x\right)dx<(C{M}_1+1)\epsilon \forall n\in \mathbb{N}.$$

(9)

By $q\in (2,2_s^{\ast })$, we have from Sobolev embeddings that

$$\underset{n\to +\infty }{lim}{\int }_{B_r\left(0\right)}K\left(x\right)|v_n{|}^{q}dx={\int }_{B_r\left(0\right)}K\left(x\right){\left|v\right|}^{q}dx.$$

(10)

Combining (9) and (10), we have

$$\underset{n\to +\infty }{lim}{\int }_{{\mathbb{R}}^N}K\left(x\right)|v_n{|}^{q}dx={\int }_{{\mathbb{R}}^N}K\left(x\right){\left|v\right|}^{q}dx,$$

which yields

$$v_n\to v\mathrm{in}{L}_K^q\left({\mathbb{R}}^N\right)\forall q\in (2,2_s^{\ast }).$$

Next, we suppose that $\left(K2\right)$ holds. For each $x\in {\mathbb{R}}^N$ fixed, we observe that the function

$$g\left(t\right)=V\left(x\right)t^{2-p}+t^{2_s^{\ast }-p}\forall t>0$$

has $C_{p}V{\left(x\right)}^{\frac{2_s^{\ast }-p}{2_s^{\ast }-2}}$ as its minimum value, where

$$C_p={\left(\frac{p-2}{2_s^{\ast }-p}\right)}^{\frac{2-p}{2_s^{\ast }-2}}+{\left(\frac{p-2}{2_s^{\ast }-p}\right)}^{\frac{2_s^{\ast }-p}{2_s^{\ast }-2}}.$$

Hence

$$C_{p}V{\left(x\right)}^{\frac{2_s^{\ast }-p}{2_s^{\ast }-2}}\le V\left(x\right)t^{2-p}+t^{2_s^{\ast }-p}\forall x\in {\mathbb{R}}^N\mathrm{and}t>0.$$

Combining this inequality with $\left(K2\right)$, given $\epsilon \in (0,C_p)$, there exists $r>0$ large enough such that

$${K\left(x\right)\left|t\right|}^p\le {\epsilon \left(V\left(x\right)\right|t|}^2+{\left|t\right|}^{2_s^{\ast }})\forall t\in \mathbb{R}\mathrm{and}|x|\ge r,$$

leading to

$${\int }_{B_r^c\left(0\right)}{K\left(x\right)\left|u\right|}^{p}dx\le \epsilon {\int }_{B_r^c\left(0\right)}{\left(V\left(x\right)\right|u|}^2+{\left|u\right|}^{2_s^{\ast }})dx\forall u\in E.$$

Let $\left\{v_n\right\}$ be a sequence such that $v_n\rightharpoonup v$ in E, then there exists a constant $M_2>0$ such that

$${\int }_{{\mathbb{R}}^N}V\left(x\right)|v_n{|}^{2}dx\le M_2\mathrm{and}{\int }_{{\mathbb{R}}^N}{\left|v_n\right|}^{2_s^{\ast }}dx\le M_2\forall n\in \mathbb{N},$$

and so,

$${\int }_{B_r^c\left(0\right)}K\left(x\right){|v_n|}^{p}dx\le 2\epsilon M_2\forall n\in \mathbb{N}.$$

(11)

Since $p\in (2,2_s^{\ast })$ and K is a continuous function, we have

$$\underset{n\to +\infty }{lim}{\int }_{B_r^c\left(0\right)}K\left(x\right)|v_n{|}^{p}dx={\int }_{B_r^c\left(0\right)}K\left(x\right){\left|v\right|}^{p}dx.$$

(12)

From (11) and (12), we have

$$\underset{n\to +\infty }{lim}{\int }_{{\mathbb{R}}^N}K\left(x\right)|v_n{|}^{p}dx={\int }_{{\mathbb{R}}^N}K\left(x\right){\left|v\right|}^{p}dx.$$

Therefore

$$v_n\to v\mathrm{in}{L}_K^p\left({\mathbb{R}}^N\right).$$

 □

Lemma 2.

(Hardy–Littlewood–Sobolev inequality, see [26]). Let $1<r,t<\infty$, and $\mu \in (0,N)$ with $\frac{1}{r}+\frac{1}{t}=2-\frac{\mu }{N}$. If $\varphi \in L^r\left({\mathbb{R}}^N\right)$ and $\psi \in L^t\left({\mathbb{R}}^N\right)$, then there exists a constant $C(N,\mu ,r,t)>0$, such that

$${\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{\varphi \left(x\right)\psi \left(y\right)}{{|x-y|}^{\mu }}dxdy\le {C(N,\mu ,r,t)\parallel \varphi \parallel }_r{\parallel \psi \parallel }_t.$$

Lemma 3.

Assume that (I)–(III) and $\left(F1\right)$–$\left(F3\right)$ hold. Then for $u\in E$

$$\left|{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u\right(x\left)\right)F\left(u\right(y\left)\right)}{{|x-y|}^{\mu }}dxdy\right|<+\infty ,$$

(13)

and there exists a constant $C_1>0$ such that

$$\left|{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u\right(x\left)\right)f\left(u\right(y\left)\right)v\left(y\right)}{{|x-y|}^{\mu }}dxdy\right|<C_1\parallel v\parallel ,\forall v\in E.$$

(14)

Furthermore, let $\left\{u_n\right\}\subset E$ be a sequence such that $u_n\rightharpoonup u$ in E, then

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)[F\left(u_n\left(x\right)\right)F\left(u_n\left(y\right)\right)-F\left(u\left(x\right)\right)F\left(u\left(y\right)\right)]}{{|x-y|}^{\mu }}dxdy=0$$

(15)

and

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u_n\left(x\right)\right)f\left(u_n\left(y\right)\right)[u_n\left(y\right)-u\left(y\right)]}{{|x-y|}^{\mu }}dxdy=0.$$

(16)

Proof. 

Set

$$\beta =\left\{\begin{array}{c}2,\mathrm{if}\left(K1\right)\mathrm{holds},\hfill \\ p,\mathrm{if}\left(K2\right)\mathrm{holds}.\hfill \end{array}\right.$$

By $\left(F1\right),\left(F2\right)$, Lemma 2, Hölder inequality and Sobolev inequality, we have

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{R}}^N}{K\left(x\right)\left|F\left(u\right)\right|}^{\frac{2N}{2N-\mu }}dx& \le C_1{\int }_{{\mathbb{R}}^N}K\left(x\right){\left[{\left|u\right|}^{\frac{\beta (2N-\mu )}{2N}}+{\left|u\right|}^{\frac{2N-\mu }{N-2s}}\right]}^{\frac{2N}{2N-\mu }}dx\hfill \\ & \le C_2{\int }_{{\mathbb{R}}^N}{K\left(x\right)\left|u\right|}^{\beta }dx+C_2{\int }_{{\mathbb{R}}^N}{\left|u\right|}^{2_s^{\ast }}dx\hfill \\ & \le C_3{(\parallel u\parallel }^{\beta }+{\left[u\right]}^{2_s^{\ast }}),\forall u\in E\hfill \end{array}$$

(17)

and

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{R}}^N}{K\left(x\right)\left|f\left(u\right)v\right|}^{\frac{2N}{2N-\mu }}dx& \le C_1{\int }_{{\mathbb{R}}^N}K\left(x\right){\left[{\left|u\right|}^{\frac{\beta (2N-\mu )-2N}{2N}}+{\left|u\right|}^{\frac{N-\mu +2s}{N-2s}}\right]}^{\frac{2N}{2N-\mu }}{\left|v\right|}^{\frac{2N}{2N-\mu }}dx\hfill \\ & \le C_4{\int }_{{\mathbb{R}}^N}{\left[K\left(x\right)\right]}^{\frac{\beta (2N-\mu )-2N}{\beta (2N-\mu )}}{\left|u\right|}^{\frac{\beta (2N-\mu )-2N}{2N-\mu }}{\left[K\left(x\right)\right]}^{\frac{2N}{\beta (2N-\mu )}}{\left|v\right|}^{\frac{2N}{2N-\mu }}dx\hfill \\ & +C_5{\int }_{{\mathbb{R}}^N}{\left|u\right|}^{\frac{2N(N+2s-\mu )}{(N-2s)(2N-\mu )}}{\left|v\right|}^{\frac{2N}{2N-\mu }}dx\hfill \\ & \le C_6\left[{\parallel u\parallel }^{\frac{\beta (2N-\mu )-2N}{2N-\mu }}+{\parallel u\parallel }^{\frac{2N(N+2s-\mu )}{(N-2s)(2N-\mu )}}\right]{\parallel v\parallel }^{\frac{2N}{2N-\mu }},\forall u,v\in E.\hfill \end{array}$$

(18)

Applying Lemma 2 and (17), we have

$$\begin{array}{l}\left|{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u\right(x\left)\right)F\left(u\right(y\left)\right)}{{|x-y|}^{\mu }}dxdy\right|\\ \le C_7{\left[{\int }_{{\mathbb{R}}^N}K\left(x\right){\left|F\left(u\right)\right|}^{\frac{2N}{2N-\mu }}dx\right]}^{\frac{2N-\mu }{N}}\\ \le C_8\left[{\parallel u\parallel }^{\frac{\beta (2N-\mu )}{N}}+{\parallel u\parallel }^{\frac{2(2N-\mu )}{N-2s}}\right],\forall u\in E,\end{array}$$

(19)

which yields (13) holds. Similarly, we have

$$\begin{array}{l}\left|{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u\right(x\left)\right)f\left(u\right(y\left)\right)v\left(y\right)}{{|x-y|}^{\mu }}dxdy\right|\\ \le C_9{\left[{\int }_{{\mathbb{R}}^N}K\left(x\right){\left|F\left(u\right)\right|}^{\frac{2N}{2N-\mu }}dx\right]}^{\frac{2N-\mu }{2N}}{\left[{\int }_{{\mathbb{R}}^N}K\left(x\right){\left|f\left(u\right)v\right|}^{\frac{2N}{2N-\mu }}dx\right]}^{\frac{2N-\mu }{2N}},\forall u,v\in E,\end{array}$$

(20)

which, together with (17) and (18), implies that (14) holds.

Similar to ([21], Lemma 2), by $\left(F_2\right),\left(F_3\right)$, and Lemma 2, we have

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}K\left(x\right)|F\left(u_n\right)-F\left(u\right){|}^{\frac{2N}{2N-\mu }}dx=0,\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}K\left(x\right)|f\left(u_n\right){|}^{\frac{2N}{2N-\mu }}{|u_n-u|}^{\frac{2N}{2N-\mu }}dx=0.$$

(21)

Combining (18), (20), and (21), we deduce that (15) and (16) hold. □

The energy functional $\mathsf{\Phi }:E\mapsto \mathbb{R}$ given by

$$\mathsf{\Phi }\left(u\right):=\frac{1}{2}{\int }_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u|}^{2}dx+\frac{1}{2}{\int }_{{\mathbb{R}}^N}V\left(x\right){\left|u\right|}^{2}dx-\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u\right(x\left)\right)F\left(u\right(y\left)\right)}{{|x-y|}^{\mu }}dxdy.$$

(22)

By Lemmas 2 and 3, $\mathsf{\Phi }$ is well-defined and belongs to $C^1$-class. Moreover, we have

$$\begin{array}{cc}\hfill \langle {\mathsf{\Phi }}^{\prime }\left(u\right),v\rangle =& {\int }_{{\mathbb{R}}^N}{(-\mathsf{\Delta })}^{\frac{s}{2}}u{(-\mathsf{\Delta })}^{\frac{s}{2}}vdx+{\int }_{{\mathbb{R}}^N}V\left(x\right)uvdx\hfill \\ & -{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u\right(x\left)\right)f\left(u\right(y\left)\right)v\left(y\right)}{{|x-y|}^{\mu }}dxdy,\forall u,v\in E.\hfill \end{array}$$

(23)

Lemma 4.

Assume that $\left(F1\right)$–$\left(F3\right)$ hold. Then, for all $t\ge 0$ and ${\tau }_1,{\tau }_2\in \mathbb{R}$,

$$l(t,{\tau }_1,{\tau }_2):=F\left(t{\tau }_1\right)F\left(t{\tau }_2\right)-F\left({\tau }_1\right)F\left({\tau }_2\right)+\frac{1-t^2}{2}[F\left({\tau }_1\right)f\left({\tau }_2\right){\tau }_2+F\left({\tau }_2\right)f\left({\tau }_1\right){\tau }_1]\ge 0.$$

(24)

Proof. 

Firstly, it follows from $\left(F1\right)$ that $f\left(0\right)=0$. By $\left(F3\right)$, we have

$$f\left(\tau \right)\ge 0,\forall \tau \ge 0;f\left(\tau \right)\le 0,\forall \tau \le 0;F\left(\tau \right)\ge 0,\forall \tau \in \mathbb{R}$$

and

$$f\left(\tau \right)\tau \ge {\int }_0^{\tau }f\left(t\right)dt=F\left(\tau \right),\forall \tau \in \mathbb{R}.$$

(25)

It is easy to verify that (24) holds for $t=0$. For $\tau \ne 0$, we have from (25) that

$${\left[\frac{F\left(\tau \right)}{\tau }\right]}^{\prime }=\frac{f\left(\tau \right)\tau -F\left(\tau \right)}{{\tau }^2}\ge 0.$$

(26)

For every ${\tau }_1,{\tau }_2\in \mathbb{R}$, we deduce from $\left(F3\right)$ and (26) that

$$\begin{array}{l}\frac{d}{dt}l(t,{\tau }_1,{\tau }_2)\\ ={\tau }_1{\tau }_{2}t\left[\frac{F\left(t{\tau }_1\right)}{t{\tau }_1}f\left(t{\tau }_2\right)+\frac{F\left(t{\tau }_2\right)}{t{\tau }_2}f\left(t{\tau }_1\right)-\frac{F\left({\tau }_1\right)}{{\tau }_1}f\left({\tau }_2\right)-\frac{F\left({\tau }_2\right)}{{\tau }_2}f\left({\tau }_1\right)\right]\\ \left\{\begin{array}{c}\ge 0,t\ge 1,\hfill \\ \le 0,0<t<1,\hfill \end{array}\right.\end{array}$$

which implies that $l(t,{\tau }_1,{\tau }_2)\ge l(1,{\tau }_1,{\tau }_2)=0$ for all $t>0$ and ${\tau }_1,{\tau }_2\in \mathbb{R}$. □

Lemma 5.

Assume that $\left(I\right)$–$\left(III\right)$ and $\left(F1\right)$–$\left(F4\right)$ hold. Then

$$\mathsf{\Phi }\left(u\right)\ge \mathsf{\Phi }\left(tu\right)+\frac{1-t^2}{2}\langle {\mathsf{\Phi }}^{\prime }\left(u\right),u\rangle ,\forall u\in E,t\ge 0.$$

(27)

Proof. 

By (22), (23), and (24), we have

$$\begin{array}{l}\mathsf{\Phi }\left(u\right)-\mathsf{\Phi }\left(tu\right)-\frac{1-t^2}{2}\langle {\mathsf{\Phi }}^{\prime }\left(u\right),u\rangle \\ =\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{1}{{|x-y|}^{\mu }}\left[F\left(tu\right(x\left)\right)F\left(tu\right(y\left)\right)-F\left(u\right(x\left)\right)F\left(u\right(y\left)\right)\right.\\ \hspace{1em}\left.+\frac{1-t^2}{2}\left(F\left(u\right(x\left)\right)f\left(u\right(y\left)\right)u\left(y\right)+F\left(u\right(y\left)\right)f\left(u\right(x\left)\right)u\left(x\right)\right)\right]dxdy\\ =\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{l(t,u(x),u(y\left)\right)}{{|x-y|}^{\mu }}dxdy\\ \ge 0,\forall u\in E,t\ge 0.\end{array}$$

 □

Corollary 1.

Assume that $\left(I\right)$–$\left(III\right)$ and $\left(F1\right)$–$\left(F4\right)$ hold. Let

$$\mathcal{N}:=\{u\in E\setminus \left\{0\right\}:\langle {\mathsf{\Phi }}^{\prime }\left(u\right),u\rangle =0\}.$$

Then

$$\mathsf{\Phi }\left(u\right)=\underset{t\ge 0}{max}\mathsf{\Phi }\left(tu\right),\forall u\in \mathcal{N}.$$

Lemma 6.

Assume that $\left(I\right)$–$\left(III\right)$ and $\left(F1\right)$–$\left(F4\right)$ hold. Then, for any $u\in E\setminus \left\{0\right\}$, there exists $t_u>0$ such that $t_{u}u\in \mathcal{N}$.

Proof. 

Let $u\in E\setminus \left\{0\right\}$ be fixed. Define a function $\zeta \left(t\right):=\mathsf{\Phi }\left(tu\right)$ on $(0,\infty )$. By (22) and (23), we have

$$\begin{array}{cc}\hfill {\zeta }^{\prime }\left(t\right)=0& ⟺t{\parallel u\parallel }^2-{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(tu\right(x\left)\right)F\left(tu\right(y\left)\right)\left)f\right(tu\left(y\right)\left)u\right(y)}{{|x-y|}^{\mu }}dxdy=0\hfill \\ & ⟺tu\in \mathcal{N}.\hfill \end{array}$$

By (19), we have for $u\in E$

$$\mathsf{\Phi }\left(u\right)\ge \left\{\begin{array}{c}\frac{1}{2}\parallel u\parallel -C_8\left[{\parallel u\parallel }^{\frac{4N-2\mu }{N}}+{\parallel u\parallel }^{\frac{4N-2\mu }{N-2s}}\right],\mathrm{if}\left(K1\right)\mathrm{holds},\hfill \\ \frac{1}{2}\parallel u\parallel -C_8\left[{\parallel u\parallel }^{\frac{2pN-p\mu }{N}}+{\parallel u\parallel }^{\frac{4N-2\mu }{N-2s}}\right],\mathrm{if}\left(K2\right)\mathrm{holds},\hfill \end{array}\right.$$

(28)

which implies that there exists ${\rho }_0>0$ such that

$${\delta }_0:=\underset{\parallel u\parallel ={\rho }_0}{inf}\mathsf{\Phi }\left(u\right)>0.$$

(29)

Therefore, $\underset{t\to 0}{lim}\zeta \left(t\right)=0$ and $\zeta \left(t\right)>0$ for small $t>0$. By $\left(F4\right)$, for t large, we have

$$\zeta \left(t\right)=\frac{t^2}{2}\left[{\parallel u\parallel }^2-\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)F\left(tu\right(x\left)\right)}{\left|tu\right(x\left)\right|}\frac{Q\left(y\right)F\left(tu\right(y\left)\right)}{\left|tu\right(y\left)\right|}\frac{\left|u\right(x\left)u\right(y\left)\right|}{{|x-y|}^{\mu }}dxdy\right]<0.$$

(30)

Therefore $\underset{t\in [0,\infty )}{max}\zeta \left(t\right)$ is achieved at some $t_u>0$ so that ${\zeta }^{\prime }\left(t_u\right)=0$ and $t_{u}u\in \mathcal{N}$. □

Lemma 7.

Assume that $\left(I\right)$–$\left(III\right)$ and $\left(F1\right)$–$\left(F4\right)$ hold. Then

$$\underset{u\in \mathcal{N}}{inf}\mathsf{\Phi }\left(u\right):=c=\underset{u\in E\setminus \left\{0\right\}}{inf}\underset{t\ge 0}{max}\mathsf{\Phi }\left(tu\right)>0.$$

Proof. 

Corollary 1 and Lemma 6 imply that

$$c=\underset{u\in E\setminus \left\{0\right\}}{inf}\underset{t\ge 0}{max}\mathsf{\Phi }\left(tu\right).$$

By (22) and (29),

$$c\ge \underset{u\in E\setminus \left\{0\right\}}{inf}\mathsf{\Phi }\left(\frac{{\rho }_0}{\parallel u\parallel }u\right)=\underset{\parallel u\parallel ={\rho }_0}{inf}\mathsf{\Phi }\left(u\right)>0.$$

 □

Next, we will seek a Cerami sequence for $\mathsf{\Phi }$ outside $\mathcal{N}$ by using the diagonal method, which is used in [25,27,28].

Lemma 8.

Assume that $\left(I\right)$–$\left(III\right)$ and $\left(F1\right)$–$\left(F4\right)$ hold. Then there exist $\left\{u_n\right\}\subset E$ and $c^{\ast }\in (0,c]$ such that

$$\mathsf{\Phi }\left(u_n\right)\to c^{\ast },(1+\parallel u_n\parallel )\parallel {\mathsf{\Phi }}^{\prime }\left(u_n\right)\parallel \to 0,$$

(31)

as $n\to \infty$.

Proof. 

For $c=\underset{\mathcal{N}}{inf}\mathsf{\Phi }$, we can choose a sequence $\left\{v_k\right\}\subset \mathcal{N}$ such that

$$c\le \mathsf{\Phi }\left(v_k\right)<c+\frac{1}{k},k\in \mathbb{N}.$$

(32)

By (29) and (30), it is easy to verify that $\mathsf{\Phi }\left(0\right)=0$, $\mathsf{\Phi }\left(T{v}_k\right)<0$ when T is large enough, and $\mathsf{\Phi }\left(u\right)\ge {\delta }_0>0$ when $\parallel u\parallel ={\rho }_0$. Therefore, from Mountain Pass Lemma ([29]), there is a sequence $\left\{u_{n,k}\right\}$ such that

$$\mathsf{\Phi }\left(u_{k,n}\right)\to c_k\in [{\delta }_0,\underset{t\in [0,1]}{sup}\mathsf{\Phi }\left(t{v}_k\right)],(1+\parallel u_{k,n}\parallel )\parallel {\mathsf{\Phi }}^{\prime }\left(u_{k,n}\right)\parallel \to 0,k\in \mathbb{N}.$$

(33)

By Corollary 1 and $\left\{v_k\right\}\subset \mathcal{N}$, we have

$$\mathsf{\Phi }\left(t{v}_k\right)\le \mathsf{\Phi }\left(v_k\right),\forall t\ge 0.$$

(34)

It follows from (34) that $\mathsf{\Phi }\left(v_k\right)=\underset{t\in [0,1]}{sup}\mathsf{\Phi }\left(t{v}_k\right)$. Hence, by (32)–(34), we have

$$\mathsf{\Phi }\left(w_{k,n}\right)\to c_k\in \left[{\delta }_0,c+\frac{1}{k}\right),(1+\parallel u_{k,n}\parallel )\parallel {\mathsf{\Phi }}^{\prime }\left(u_{k,n}\right)\parallel \to 0,k\in \mathbb{N}.$$

Then, we can choose $\left\{n_k\right\}\subset \mathbb{N}$ such that

$$\mathsf{\Phi }\left(u_{k,n_k}\right)\in \left[{\delta }_0,c+\frac{1}{k}\right),(1+\parallel u_{k,n_k}\parallel )\parallel {\mathsf{\Phi }}^{\prime }\left(u_{k,n_k}\right)\parallel <\frac{1}{k},k\in \mathbb{N}.$$

Let $u_k=u_{k,n_k}$, $k\in \mathbb{N}$. Therefore, up to a subsequence, we have

$$\mathsf{\Phi }\left(u_n\right)\to c^{\ast }\in [{\delta }_0,c],(1+\parallel u_n\parallel )\parallel {\mathsf{\Phi }}^{\prime }\left(u_n\right)\parallel \to 0.$$

 □

Lemma 9.

Assume that $\left(I\right)$–$\left(III\right)$ and $\left(F1\right)$–$\left(F4\right)$ hold. Then, the sequence $\left\{u_n\right\}$ satisfying (31) is bounded in E.

Proof. 

Arguing by contradiction, suppose that $\parallel u_n\parallel \to \infty$. Let $v_n=\frac{u_n}{\parallel u_n\parallel }$, then $\parallel v_n\parallel =1$. Passing to a subsequence, we have $v_n\rightharpoonup v$ in E. There are two possible cases: (i). $v=0$; (ii) $v\ne 0$.

Case (i) $v=0$. In this case

$$\begin{array}{l}\left|{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(2\sqrt{c^{\ast }+1}{v}_n\left(x\right)\right)F\left(2\sqrt{c^{\ast }+1}{v}_n\left(y\right)\right)}{{|x-y|}^{\mu }}dxdy\right|\\ \le C_1{\left[{\int }_{{\mathbb{R}}^N}K\left(x\right){\left|F\left(2\sqrt{c^{\ast }+1}{v}_n\left(x\right)\right)\right|}^{\frac{2N}{2N-\mu }}dx\right]}^{\frac{2N-\mu }{N}}\\ =o\left(1\right).\end{array}$$

(35)

Combining (27), (31), and (35), we have

$$\begin{array}{cc}\hfill c^{\ast }+o\left(1\right)& =\mathsf{\Phi }\left(u_n\right)\hfill \\ & \ge \mathsf{\Phi }\left(\frac{2\sqrt{c^{\ast }+1}}{\parallel u_n\parallel }{u}_n\right)+\frac{1-{\left(\frac{2\sqrt{c^{\ast }+1}}{\parallel u_n\parallel }\right)}^2}{2}\langle {\mathsf{\Phi }}^{\prime }\left(u_n\right),u_n\rangle \hfill \\ & =\mathsf{\Phi }\left(2\sqrt{c^{\ast }+1}{v}_n\right)+o\left(1\right)\hfill \\ & =2(c^{\ast }+1)+o\left(1\right),\hfill \end{array}$$

which is a contradiction.

Case (ii) $v\ne 0$. In this case, since $|u_n|=|v_n|\parallel u_n\parallel$ and $u_n/\parallel u_n\parallel \to v$ a.e. in ${\mathbb{R}}^N$, we have $\underset{n\to \infty }{lim}\left|u_n\left(x\right)\right|=\infty$ for $x\in \{y\in {\mathbb{R}}^N:v\left(x\right)\ne 0\}$. Hence, it follows from (22), (31), $\left(F4\right)$, and Fatou’s lemma that

$$\begin{array}{cc}\hfill 0& =\underset{n\to \infty }{lim}\frac{c^{\ast }+o\left(1\right)}{\parallel u_n{\parallel }^2}=\underset{n\to \infty }{lim}\frac{\mathsf{\Phi }\left(u_n\right)}{\parallel u_n{\parallel }^2}\hfill \\ & =\frac{1}{2}-\frac{1}{2}\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)F\left(u_n\left(x\right)\right)}{|u_n\left(x\right)|}\frac{Q\left(y\right)F\left(u_n\left(y\right)\right)}{|u_n\left(y\right)|}\frac{|v_n\left(x\right)v_n\left(y\right)|}{{|x-y|}^{\mu }}dxdy\hfill \\ & \le \frac{1}{2}-\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\underset{n\to \infty }{lim\; inf}\frac{Q(x+k_n)F\left(u_n\left(x\right)\right)}{|u_n\left(x\right)|}\frac{Q(y+k_n)F\left(u_n\left(y\right)\right)}{|u_n\left(y\right)|}\frac{|v_n\left(x\right)v_n\left(y\right)|}{{|x-y|}^{\mu }}dxdy\hfill \\ & =-\infty .\hfill \end{array}$$

This contradiction shows that $\left\{u_n\right\}$ is bounded in E. □

Proof of Theorem 1.

In view of Lemmas 8 and 9, there exists a bounded sequence $\left\{u_n\right\}\subset E$ such that (31) holds. Passing to a subsequence, we have $u_n\rightharpoonup u$ in E. Thus, it follows from (22), (23), (31), and Lemma 3 that

$$\parallel u_n{-u\parallel }^2=\langle {\mathsf{\Phi }}^{\prime }\left(u_n\right),u_n-u\rangle +{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{Q\left(x\right)Q\left(y\right)F\left(u_n\left(x\right)\right)f\left(u_n\left(y\right)\right)[u_n\left(y\right)-u\left(y\right)]}{{|x-y|}^{\mu }}=o\left(1\right),$$

which implies that ${\mathsf{\Phi }}^{\prime }\left(u\right)=0$ and $\mathsf{\Phi }\left(u\right)=c^{\ast }\in (0,c]$. Moreover, since $u\in \mathcal{N}$, we have $\mathsf{\Phi }\left(u\right)\ge c$. Hence, $u\in E$ is a ground state solution for (1) with $\mathsf{\Phi }\left(u\right)=c>0$. □

3. Zero Mass Case

In this section, we consider the zero mass case, and give the proof of Theorem 2. In the following, we suppose that $\left(F5\right)$–$\left(F7\right)$ and $\mu <4s$ hold. Fix $q\in (2,\frac{2N-\mu }{N-2s})$, by $\left(F7\right)$, for every $ϵ>0$ there is $C_{ϵ}>0$ such that

$$\left|f\left(t\right)t\right|\le {ϵ\left(\right|t|}^2+{\left|t\right|}^{\frac{2N-\mu }{N-2s}})+C_{ϵ}{\left|t\right|}^q,\left|F\left(t\right)\right|\le {ϵ\left(\right|t|}^2+{\left|t\right|}^{\frac{2N-\mu }{N-2s}})+C_{ϵ}{\left|t\right|}^q,\forall t\in \mathbb{R}.$$

(36)

To find nontrivial solutions for (6), we study the approximating problem

$$\left\{\begin{array}{c}{(-\mathsf{\Delta })}^{s}u+\epsilon u=\left(\frac{1}{{\left|x\right|}^{\mu }}\ast F\left(u\right)\right)f\left(u\right),\mathrm{in}{\mathbb{R}}^N,\hfill \\ u\in H^s\left({\mathbb{R}}^N\right),\hfill \end{array}\right.$$

(37)

where $\epsilon \ge 0$ is a small parameter. The energy functional associated to (37) is

$${\mathsf{\Phi }}_{\epsilon }\left(u\right)=\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\left[\right|(-\mathsf{\Delta })}^{\frac{s}{2}}u{|}^2+\epsilon u^2]dx-\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{F\left(u\right(x\left)\right)F\left(u\right(y\left)\right)}{{|x-y|}^{\mu }}dxdy.$$

(38)

By using $\left(F5\right)$–$\left(F7\right)$ and Lemma 2, it is easy to check that ${\mathsf{\Phi }}_0\in C^1(D^{s,2}\left({\mathbb{R}}^N\right),\mathbb{R})$ and ${\mathsf{\Phi }}_{\epsilon }\in C_1(H^s\left({\mathbb{R}}^N\right),\mathbb{R})$ for every $\epsilon >0$. Moreover, for every $\epsilon \ge 0$,

$$\langle {\mathsf{\Phi }}_{\epsilon }^{\prime }\left(u\right),v\rangle ={\int }_{{\mathbb{R}}^N}[{(-\mathsf{\Delta })}^{\frac{s}{2}}u{(-\mathsf{\Delta })}^{\frac{s}{2}}v+\epsilon uv]dx-\frac{1}{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{F\left(u\right(x\left)\right)f\left(u\right(y\left)\right)v\left(y\right)}{{|x-y|}^{\mu }}dxdy.$$

(39)

In view of ([19], Proposition 2), for every $\epsilon >0$, any critical point u of ${\mathsf{\Phi }}_{\epsilon }$ in $H^s\left({\mathbb{R}}^N\right)$ satisfies the following Pohožaev identity

$$\begin{array}{cc}\hfill {\mathcal{P}}_{\epsilon }\left(u\right):& =\frac{N-2s}{2}{\int }_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u|}^{2}dx+\frac{N}{2}\epsilon {\int }_{{\mathbb{R}}^N}{\left|u\right|}^{2}dx-\frac{2N-\mu }{2}{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{F\left(u\right(x\left)\right)F\left(u\right(y\left)\right)}{{|x-y|}^{\mu }}dxdy\hfill \\ & =0.\hfill \end{array}$$

(40)

For every $\epsilon >0$, let

$$\begin{array}{ccc}\hfill {\mathcal{M}}_{\epsilon }:& =& \{u\in H^s\left({\mathbb{R}}^N\right)\setminus \left\{0\right\}:{\mathsf{\Phi }}_{\epsilon }^{\prime }\left(u\right)=0\},\hfill \\ \hfill {\mathsf{\Gamma }}_{\epsilon }:& =& \{\gamma \in C([0,1],H^s\left({\mathbb{R}}^N\right)):\gamma \left(0\right)=0,{\mathsf{\Phi }}_{\epsilon }\left(\gamma \left(1\right)\right)<0\},\hfill \\ \hfill c_{\epsilon }:& =& \underset{\gamma \in {\mathsf{\Gamma }}_{\epsilon }}{inf}\underset{t\in [0,1]}{max}{\mathsf{\Phi }}_{\epsilon }\left(\gamma \left(t\right)\right).\hfill \end{array}$$

Lemma 10.

For every $\epsilon >0$, (37) has a ground state solution $u_{\epsilon }\in H^s\left({\mathbb{R}}^N\right)$ such that $0<{\mathsf{\Phi }}_{\epsilon }\left(u_{\epsilon }\right)=\underset{{\mathcal{M}}_{\epsilon }}{inf}{\mathsf{\Phi }}_{\epsilon }=c_{\epsilon }$. Moreover, there exists a constant $K_0>0$ independent of ε such that $c_{\epsilon }\le K_0$ for all $\epsilon \in (0,1]$.

Proof. 

In view of ([19], Theorem 1.3), under the assumption $\left(F5\right)$–$\left(F7\right)$, for every $\epsilon >0$, (37) has a ground state solution $u_{\epsilon }\in H^s\left({\mathbb{R}}^N\right)$ such that $0<{\mathsf{\Phi }}_{\epsilon }\left(u_{\epsilon }\right)=\underset{{\mathcal{M}}_{\epsilon }}{inf}{\mathsf{\Phi }}_{\epsilon }=c_{\epsilon }$. Let $\gamma \in {\mathsf{\Gamma }}_1$, since ${\mathsf{\Phi }}_{\epsilon }\left(u\right)\le {\mathsf{\Phi }}_1\left(u\right)$ for $u\in H^s\left({\mathbb{R}}^N\right)$ and $\epsilon \in (0,1]$, we have $\gamma \in {\mathsf{\Gamma }}_{\epsilon }$ for $\epsilon \in (0,1]$, and so

$$c_{\epsilon }\le \underset{t\in [0,1]}{max}{\mathsf{\Phi }}_{\epsilon }\left(\gamma \left(t\right)\right)={\mathsf{\Phi }}_{\epsilon }\left(\gamma \left(t_{\epsilon }\right)\right)\le {\mathsf{\Phi }}_1\left(\gamma \left(t_{\epsilon }\right)\right)\le \underset{t\in [0,1]}{max}{\mathsf{\Phi }}_1\left(\gamma \left(t\right)\right):=K_0,\forall \epsilon \in (0,1],$$

where $t_{\epsilon }\in (0,1)$. □

Lemma 11.

There exists a constant $K_1>0$ independent of ε such that

$$\left[u_{\epsilon }\right]\ge K_1,\forall u_{\epsilon }\in {\mathcal{M}}_{\epsilon }.$$

(41)

Proof. 

Since $\langle {\mathsf{\Phi }}_{\epsilon }^{\prime }\left(u_{\epsilon }\right),u_{\epsilon }\rangle =0$ for $u_{\epsilon }\in {\mathcal{M}}_{\epsilon }$, from $\left(F6\right)$, (39), and Sobolev inequality, we have

$$\begin{array}{cc}\hfill {\left[u_{\epsilon }\right]}^2& ={\int }_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{\frac{s}{2}}{u}_{\epsilon }{|}^{2}dx\le {\int }_{{\mathbb{R}}^N}{\left[\right|(-\mathsf{\Delta })}^{\frac{s}{2}}{u}_{\epsilon }{|}^2+\epsilon u_{\epsilon }^2]dx\hfill \\ & ={\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{F\left(u_{\epsilon }\left(x\right)\right)f\left(u_{\epsilon }\left(y\right)\right)u_{\epsilon }\left(y\right)}{{|x-y|}^{\mu }}dxdy\hfill \\ & \le C_1{\left({\int }_{{\mathbb{R}}^N}{\left|F\left(u_{\epsilon }\right)\right|}^{\frac{2N}{2N-\mu }}dx\right)}^{\frac{2N-\mu }{2N}}{\left({\int }_{{\mathbb{R}}^N}{\left|f\left(u_{\epsilon }\right)u_{\epsilon }\right|}^{\frac{2N}{2N-\mu }}dx\right)}^{\frac{2N-\mu }{2N}}\hfill \\ & \le C_2{\left({\int }_{{\mathbb{R}}^N}{\left|u_{\epsilon }\right|}^{\frac{2N}{N-2s}}dx\right)}^{\frac{2N-\mu }{N}}\hfill \\ & \le C_2{S}^{\frac{2N-\mu }{N-2s}}{\left[u_{\epsilon }\right]}^{\frac{2(2N-\mu )}{N-2s}},\forall u_{\epsilon }\in {\mathcal{M}}_{\epsilon },\hfill \end{array}$$

which, together with $(2N-\mu )/(N-2s)>1$, implies that (41) holds. □

The following lemma is a version of Lions’ concentration-compactness Lemma for fractional Laplacian.

Lemma 12.

([18]) Assume $\left\{u_n\right\}$ is a bounded sequence in $H^s\left({\mathbb{R}}^N\right)$, which satisfies

$$\underset{n\to +\infty }{lim}\underset{y\in {\mathbb{R}}^N}{sup}{\int }_{B_1\left(y\right)}{\left|u_n\left(x\right)\right|}^{2}dx=0.$$

Then $u_n\to 0$ in $L^q\left({\mathbb{R}}^N\right)$ for $q\in (2,2_s^{\ast })$.

Proof of Theorem 2.

We choose a sequence $\left\{{\epsilon }_n\right\}\subset (0,1]$ such that ${\epsilon }_n\searrow 0$. In view of Lemma 10, there exists a sequence $\left\{u_{{\epsilon }_n}\right\}\subset {\mathcal{M}}_{{\epsilon }_n}$ such that $0<{\mathsf{\Phi }}_{{\epsilon }_n}\left(u_{{\epsilon }_n}\right)=\underset{{\mathcal{M}}_{{\epsilon }_n}}{inf}{\mathsf{\Phi }}_{{\epsilon }_n}=c_{{\epsilon }_n}\le K_0$. For simplicity, we use $u_n$ instead of $u_{{\epsilon }_n}$. Now, we prove that $\left\{u_n\right\}$ is bounded in $D^{s,2}\left({\mathbb{R}}^N\right)$. Since ${\mathcal{P}}_{{\epsilon }_n}\left(u_n\right)=0$ for $u_n\in {\mathcal{M}}_{{\epsilon }_n}$, it follows from (38) and (40) that

$$\begin{array}{cc}\hfill K_0& \ge c_{{\epsilon }_n}={\mathsf{\Phi }}_{{\epsilon }_n}\left(u_n\right)-\frac{1}{2N-\mu }{\mathcal{P}}_{{\epsilon }_n}\left(u_n\right)\hfill \\ & =\left[\frac{1}{2}-\frac{N-2s}{2(2N-\mu )}\right]{\left[u_n\right]}^2+\left[\frac{1}{2}-\frac{N}{2(2N-\mu )}\right]{\epsilon }_n{\parallel u_n\parallel }_2^2.\hfill \end{array}$$

(42)

Thus, $\left\{u_n\right\}$ is bounded in $D^{s,2}\left({\mathbb{R}}^N\right)$ and $L^2\left({\mathbb{R}}^N\right)$. If

$$\delta :=\underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^N}{sup}{\int }_{B_1\left(y\right)}{\left|u_n\right|}^{2}dx=0.$$

Then, by Lemma 12, for $q\in (2,\frac{2N-\mu }{N-2s})$, we have

$${\int }_{{\mathbb{R}}^N}|u_n{|}^{\frac{4N}{2N-\mu }}dx\to 0,{\int }_{{\mathbb{R}}^N}{\left|u_n\right|}^{\frac{2Nq}{2N-\mu }}dx\to 0.$$

Therefore, by (36) and Sobolev embedding for $D^{s,2}\left({\mathbb{R}}^N\right)$, for every $ϵ>0$ there exists $C_{ϵ}>0$ such that

$$\begin{array}{cc}\hfill \left|{\int }_{{\mathbb{R}}^N}{\left|F\left(u_n\right)\right|}^{\frac{2N}{2N-\mu }}dx\right|& \le ϵ\left[{\int }_{{\mathbb{R}}^N}\left(|u_n{|}^{\frac{4N}{2N-\mu }}+{\left|u_n\right|}^{2_s^{\ast }}\right)dx\right]+C_{ϵ}{\int }_{{\mathbb{R}}^N}{\left|u_n\right|}^{\frac{2Nq}{2N-\mu }}dx\hfill \\ & \le ϵC+o\left(1\right).\hfill \end{array}$$

By the arbitrariness of $ϵ$, we get

$${\int }_{{\mathbb{R}}^N}{\left|F\left(u_n\right)\right|}^{\frac{2N}{2N-\mu }}dx\to 0.$$

(43)

Combining (36), (43), and Lemma 2, we have

$$\begin{array}{l}\left|{\int }_{{\mathbb{R}}^N}{\int }_{{\mathbb{R}}^N}\frac{F\left(u_n\left(x\right)\right)f\left(u_n\left(y\right)\right)u_n\left(y\right)}{{|x-y|}^{\mu }}dxdy\right|\\ \le C_1{\left({\int }_{{\mathbb{R}}^N}{\left|F\left(u_n\right)\right|}^{\frac{2N}{2N-\mu }}dx\right)}^{\frac{2N-\mu }{2N}}{\left({\int }_{{\mathbb{R}}^N}{\left|f\left(u_n\right)u_n\right|}^{\frac{2N}{2N-\mu }}dx\right)}^{\frac{2N-\mu }{2N}}\\ =o\left(1\right).\end{array}$$

(44)

Notice that $\left\{u_n\right\}$ is bounded in $L^2\left({\mathbb{R}}^N\right)$, we have from (44) and $u_n\in {\mathcal{M}}_{{\epsilon }_n}$ that ${\left[u_n\right]}^2=o\left(1\right)$. This contradicts (41). Thus, we get $\delta >0$. Passing to a subsequence, there exists a sequence $\left\{y_n\right\}\subset {\mathbb{R}}^N$ such that

$${\int }_{B_{1+\sqrt{N}}\left(y_n\right)}{\left|u_n\right|}^{2}dx>\frac{\delta }{2}.$$

Let ${\tilde{u}}_n\left(x\right)=u_n(x+y_n)$. Then

$${\mathsf{\Phi }}_{{\epsilon }_n}^{\prime }\left({\tilde{u}}_n\right)=0,{\mathsf{\Phi }}_{{\epsilon }_n}\left({\tilde{u}}_n\right)=c_{{\epsilon }_n}$$

and

$${\int }_{B_{1+\sqrt{N}}\left(0\right)}{\left|{\tilde{u}}_n\right|}^{2}dx>\frac{\delta }{2}.$$

(45)

Passing to a subsequence, we have ${\tilde{u}}_n\rightharpoonup u_0$ in $D^{s,2}\left({\mathbb{R}}^N\right)$. Clearly, (45) implies that $u_0\ne 0$. By the standard argument, $u_0\in D^{s,2}\left({\mathbb{R}}^N\right)$ is a nontrivial solution for (6). □

4. Conclusions

In this work, we study a class of nonlinear Choquard equation driven by the fractional Laplacian. When potential function vanishes at infinity and the Nehari-type monotonicity condition for the nonlinearity is not satisfied, we prove that the fractional Choquard equation has a ground state solution by using the non-Nehari manifold method. Unlike the Nehari manifold method, the main idea of our approach lies in finding a minimizing sequence for the energy functional outside the Nehari manifold by using the diagonal method. Moreover, by using a perturbation method, we obtain a nontrivial solution in the zero mass case.