Quotient sets in nonabelian groups

We show that for a finite, nonempty subset $A$ of a group, the quotient set $A^{-1}A:=\{a_1^{-1}{a}_2\text{\hspace{0.17em}⁣}:a_1,a_2\in A\}$ has size $\mathrm{\mid }{A}^{-1}A\mathrm{\mid }\ge \frac{5}{3}\mathrm{\mid }A\mathrm{\mid }$, unless $A$ is densely contained in a coset, or in a union of two cosets of a finite subgroup.