Magnetic phase diagram slightly below the saturation field in the stacked $J_1$-$J_2$ model in the square lattice with the $J_{\text{C}}$ interlayer coupling

We study the effect of adding interlayer coupling to the square lattice, $J_1$-$J_2$ Heisenberg model in high external magnetic field. In particular, we consider a cubic lattice formed from stacked $J_1$-$J_2$ layers, with interlayer exchange coupling $J_{\text{C}}$. For the 2-dimensional model ($J_{\text{C}}=0$) it has been shown that a spin-nematic phase appears close to the saturation magnetic field for the parameter range $-0.4\lesssim J_2\mathrm{/}{J}_1$ and $J_2>0$. We determine the phase diagram for 3-dimensional model at high magnetic field by representing spin flips out of the saturated state as bosons, considering the dilute boson limit and using the Bethe-Salpeter equation to determine the first instability of the saturated paramagnet. Close to the highly frustrated point $J_2\mathrm{/}{J}_1\sim 0.5$, we find that the spin-nematic state is stable even for $\mathrm{\mid }{J}_{\text{C}}\mathrm{/}{J}_1\mathrm{\mid }\sim 1$. For larger values of $J_2\mathrm{/}{J}_1$, interlayer coupling favors a broad, phase-separated region. Further increase of $\mathrm{\mid }{J}_{\text{C}}\mathrm{\mid }$ stabilizes a collinear antiferromagnet, which is selected via the order-by-disorder mechanism.