Noncollinear Magnetic Order Stabilized by Entangled Spin-Orbital Fluctuations

Quantum phase transitions in the two-dimensional Kugel-Khomski model on a square lattice are studied using the plaquette mean field theory and the entanglement renormalization ansatz. When $3z^2-r^2$ orbitals are favored by the crystal field and Hund’s exchange is finite, both methods give a noncollinear exotic magnetic order which consists of four sublattices with mutually orthogonal nearest neighbor and antiferromagnetic second neighbor spins. We derive effective frustrated spin model with second and third neighbor spin interactions which stabilize this phase and follow from spin-orbital quantum fluctuations involving spin singlets entangled with orbital excitations.