On the spectral theory of groups of affine transformations of compact nilmanifolds

Let $N$ be a connected and simply connected nilpotent Lie group, $\mathrm{\Lambda }$ a lattice in $N$, and $X=N\mathrm{/}\mathrm{\Lambda }$ the corresponding nilmanifold. Let $Aff(X)$ be the group of affine transformations of $X$. We characterize the countable subgroups $H$ of $Aff(X)$ for which the action of $H$ on $X$ has a spectral gap, that is, such that the associated unitary representation $U$ of $H$ on the space of functions from $L^2(X)$ with zero mean does not weakly contain the trivial representation. Denote by $T$ the maximal torus factor associated to $X$. We show that the action of $H$ on $X$ has a spectral gap if and only if there exists no proper $H$-invariant subtorus $S$ of $T$ such that the projection of $H$ on $Aut(T\mathrm{/}S)$ has an abelian subgroup of finite index. We first establish the result in the case where $X$ is a torus. In the case of a general nilmanifold, we study the asymptotic behaviour of matrix coefficients of $U$ using decay properties of metaplectic representations of symplectic groups. The result shows that the existence of a spectral gap for subgroups of $Aff(X)$ is equivalent to strong ergodicity in the sense of K.Schmidt. Moreover, we show that the action of $H$ on $X$ is ergodic (or strongly mixing) if and only if the corresponding action of $H$ on $T$ is ergodic (or strongly mixing).