Asymptotic Joint Distribution of Extreme Eigenvalues of the Sample Covariance Matrix in the Spiked Population Model

In this paper, we consider a data matrix $X\in {\mathbb{C}}^{N\times M}$ where all the columns are i.i.d. samples being $N$ dimensional complex Gaussian of mean zero and covariance $\mathrm{\Sigma }\in {\mathbb{C}}^{N\times N}$. Here the population matrix $\mathrm{\Sigma }$ is of finite rank perturbation of the identity matrix. This is the “spiked population model” first proposed by Johnstone in \cite{21}. As $M, N\to\infty$but$M/N = \gamma\in(1, \infty)$, we first establish in this paper the asymptotic distribution of the smallest eigenvalue of the sample covariance matrix $S:=X{X}^{\ast }\mathrm{/}M$. It also exhibits a phase transition phenomenon proposed in \cite{1} — the local fluctuation will be the generalized Tracy-Widom or the generalized Gaussian to be defined in the paper. Moreover we prove that the largest and the smallest eigenvalue are asymptotically independent as $M, N\to\infty$.