Ground state of indefinite coupled nonlinear Schrödinger systems

In this paper, we study the ground state solutions of the following coupled nonlinear Schrödinger system (P) $-\mathrm{\Delta }{u}_1-{\tau }_1{u}_1={\mu }_1{u}_1^3+\beta u_1{u}_2^2$, $-\mathrm{\Delta }{u}_2-{\tau }_2{u}_2={\mu }_2{u}_2^3+\beta u_1^2{u}_2$ in $\mathrm{\Omega }$, $u_1=u_2=0$ on $\mathrm{\partial }\mathrm{\Omega }$, where ${\mu }_1,{\mu }_2>0$, $\beta >0$ and $\mathrm{\Omega }\subset {\mathbb{R}}^N(N\le 3)$ is a bounded domain with smooth boundary. We are concerned with the indefinite case, i.e., ${\tau }_1,{\tau }_2$ are greater than or equal to the principal eigenvalue of $-\mathrm{\Delta }$ with the Dirichlet boundary datum. By delicate variational arguments, we obtain the existence of ground state solution to $(P)$, and also provide information on critical energy levels for coupling parameter $\beta$ in some ranges.