Stability and instability of solitary waves in fractional generalized KdV equation in all dimensions

We study stability of solitary wave solutions for the fractional generalized Korteweg-de Vries equation $${\mathrm{\partial }}_{t}u-{\mathrm{\partial }}_{x_1}{D}^{\alpha }u+\frac{1}{m}{\mathrm{\partial }}_{x_1}(u^m)=0,(x_1,\dots ,x_d)\in {\mathbb{R}}^d,t\in \mathbb{R},0<\alpha <2,$$ in any spatial dimension $d\ge 1$ and nonlinearity $m>1$. The arguments developed here are independent of the spatial dimension and rely on the new estimates for spatial decay of ground states and their regularity. In the $L^2$-subcritical case, we prove the orbital stability of solitary waves using the concentration-compactness argument, the commutator estimates and expansions of nonlocal operator $D^{\alpha }$ in several variables. In the $L^2$-supercritical case, we show that solitary waves are unstable. More precisely, the instability is obtained by constructing an explicit sequence of initial conditions that move away from a soliton orbit in finite time, this is shown in conjunction with the modulation and truncation arguments, and incorporating the decay and regularity of the ground states. As a consequence, in 1D we show the instability of solitary waves of the supercritical generalized Benjamin-Ono equation ($\alpha =1$) and the dispersion-generalized Benjamin-Ono equation ($1<\alpha <2$); furthermore, new results on the instability are obtained in the weaker dispersion regime when $\frac{1}{2}<\alpha <1$. This work should be of interest in studying stability of solitary waves and other coherent structures in a variety of dispersive equations that involve nonlocal operators.