Fusion 2-categories with no line operators are grouplike

We show that if $\mathcal{C}$ is a fusion $2$-category in which the endomorphism category of the unit object is $\mathrm{V}\mathrm{e}\mathrm{c}$ or $\mathrm{S}\mathrm{V}\mathrm{e}\mathrm{c}$, then the indecomposable objects of $\mathcal{C}$ form a finite group.