Mechanism of Incommensurate Magnetic Order in $\mathrm{B}\mathrm{a}\mathrm{C}{\mathrm{o}}_2(\mathrm{A}\mathrm{s}{\mathrm{O}}_4{)}_2$: Interplay of Frustrated Further-Neighbor Exchanges and Bond-Directional Anisotropy

The microscopic mechanism governing the zero-field incommensurate magnetic order in the honeycomb cobaltate ${\mathrm{B}\mathrm{a}\mathrm{C}\mathrm{o}}_2({\mathrm{A}\mathrm{s}\mathrm{O}}_4{)}_2$ remains a significant unresolved problem, particularly concerning the relative contributions of bond-directional Kitaev-type interactions and exchange frustration. This study investigates an extended $J_1\text{–}K\text{–}\mathrm{\Gamma }\text{–}{\mathrm{\Gamma }}^{\mathrm{\prime }}\text{–}{J}_2\text{–}{J}_3$ model, which incorporates XXZ-type exchange anisotropies on the honeycomb lattice. Nearest-neighbor parameters are constrained by ab initio electronic structure calculations. By integrating the analytical Luttinger-Tisza approach with exact diagonalization calculations under twisted boundary conditions, we delineate the classical and quantum phase diagrams across the $(J_2,J_3)$ parameter space. We demonstrate that the stabilization of the experimentally observed incommensurate spiral phase, propagating along the $\mathrm{\Gamma }\to M$ direction, does not necessitate an anomalously dominant Kitaev coupling. Instead, this phase arises naturally from the synergistic interplay between exchange frustration, driven by further-neighbor couplings ($J_2,J_3$), and intermediate off-diagonal bond anisotropies ($\mathrm{\Gamma },{\mathrm{\Gamma }}^{\mathrm{\prime }}$). In the quantum regime, we elucidate the competition between this incommensurate manifold and an out-of-plane ferromagnetic (${\mathrm{F}\mathrm{M}}_z$) phase, which is selectively stabilized via a quantum order-by-disorder mechanism. Our findings reconcile conflicting interpretations of the magnetic interactions in ${\mathrm{B}\mathrm{a}\mathrm{C}\mathrm{o}}_2({\mathrm{A}\mathrm{s}\mathrm{O}}_4{)}_2$ and establish the microscopic origin and stability range of its incommensurate ground state.