Penalized versus constrained generalized eigenvalue problems

We investigate the difference between using an ${\mathrm{\ell }}_1$ penalty versus an ${\mathrm{\ell }}_1$ constraint in generalized eigenvalue problems, such as principal component analysis and discriminant analysis. Our main finding is that an ${\mathrm{\ell }}_1$ penalty may fail to provide very sparse solutions; a severe disadvantage for variable selection that can be remedied by using an ${\mathrm{\ell }}_1$ constraint. Our claims are supported both by empirical evidence and theoretical analysis. Finally, we illustrate the advantages of an ${\mathrm{\ell }}_1$ constraint in the context of discriminant analysis and principal component analysis.