Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Head

Costly, noisy, and over-specialized, labels are to be set aside in favor of unsupervised learning if we hope to learn cheap, reliable, and transferable models. To that end, spectral embedding, self-supervised learning, or generative modeling have offered competitive solutions. Those methods however come with numerous challenges $\mathit{\text{e.g.}}$ estimating geodesic distances, specifying projector architectures and anti-collapse losses, or specifying decoder architectures and reconstruction losses. In contrast, we introduce a simple explainable alternative – coined $\mathbf{\text{DIET}}$ – to learn representations from unlabeled data, free of those challenges. $\mathbf{\text{DIET}}$ is blatantly simple: take one’s favorite classification setup and use the $\mathbf{\text{D}}$atum $\mathbf{\text{I}}$nd$\mathbf{\text{E}}$x as its $\mathbf{\text{T}}$arget class, $\mathit{\text{i.e.}}\mathit{\text{each}}\mathit{\text{sample}}\mathit{\text{is}}\mathit{\text{its}}\mathit{\text{own}}\mathit{\text{class}}$, no further changes needed. $\mathbf{\text{DIET}}$ works without a decoder/projector network, is not based on positive pairs nor reconstruction, introduces no hyper-parameters, and works out-of-the-box across datasets and architectures. Despite $\mathbf{\text{DIET}}$'s simplicity, the learned representations are of high-quality and often on-par with the state-of-the-art $\mathit{\text{e.g.}}$ using a linear classifier on top of DIET’s learned representation reaches $71.4\mathrm{\%}$ on CIFAR100 with a Resnet101, $52.5\mathrm{\%}$ on TinyImagenet with a Resnext50.