On the Schr\"odinger-Poisson system with steep potential well and indefinite potential

In this paper, we study the following Schr"odinger-Poisson system: \left\{\aligned&-\Delta u+V_\lambda(x)u+K(x)\phi u=f(x,u)&\quad\text{in }\bbr^3,\\ &-\Delta\phi=K(x)u^2&\quad\text{in }\bbr^3,\\ &(u,\phi)\in\h\times\D,\endaligned\right.\eqno{(\mathcal{SP}_{\lambda})} where $V_{\lambda }(x)=\lambda a(x)+b(x)$ with a positive parameter $\lambda$, $K(x)\ge 0$ and $f(x,t)$ is continuous including the power-type nonlinearity $\mathrm{\mid }u{\mathrm{\mid }}^{p-2}u$. By applying the method of penalized functions, the existence of one nontrivial solution for such system in the less-studied case $3<p\le 4$ is obtained for $\lambda$ sufficiently large. The concentration behavior of this nontrivial solution for $\lambda \to +\mathrm{\infty }$ are also observed. It is worth to point out that some new conditions on the potentials are introduced to obtain this nontrivial solution and the Schr"odinger operator $-\mathrm{\Delta }+V_{\lambda }(x)$ may be strong indefinite in this paper.