Quantum and Thermal Phase Transitions in a Bosonic Atom-Molecule Mixture in a Two-dimensional Optical Lattice

We study the ground state and the thermal phase diagram of a two-species Bose-Hubbard model, with $U(1)\times {\mathbb{Z}}_2$ symmetry, describing atoms and molecules on a 2D optical lattice interacting via a Feshbach resonance. Using quantum Monte Carlo simulations and mean field theory, we show that the conversion between the two-species, coherently coupling the atomic and molecular states, has a crucial impact on the Mott-Superfluid transition and stabilizes an insulating phase with a gap controlled by the conversion term – $\mathit{\text{the}}\mathit{\text{Feshbach}}\mathit{\text{insulator}}$ – instead of a standard Mott insulating phase. Depending on the detuning between atoms and molecules, this model exhibits three phases: the Feshbach insulator, a molecular condensate coexisting with non condensed atoms and a mixed atomic-molecular condensate. Employing a finite-size scaling method, we observe 3D XY (3D Ising) transition when $U(1)$ (${\mathbb{Z}}_2$) is broken whereas the transition is first-order when both $U(1)$ and ${\mathbb{Z}}_2$ symmetries are spontaneously broken. The finite temperature phase diagram is also discussed. The thermal disappearance of the molecular superfluid leads to a Berezinskii-Kosterlitz-Thouless transition with unusual universal jump in the superfluid density. The loss of the quasi-long-range coherence of the mixed atomic and molecular superfluid is more subtle since only atoms exhibit conventional Berezinskii-Kosterlitz-Thouless criticality. We also observe a classical first-order transition between the mixed superfluid and the normal Bose liquid at low temperature.