Generative Distribution Distillation

In this paper, we formulate the knowledge distillation (KD) as a conditional generative problem and propose the $\mathit{\text{Generative}}\mathit{\text{Distribution}}\mathit{\text{Distillation}}\mathit{\text{(GenDD)}}$ framework. A naive $\mathit{\text{GenDD}}$ baseline encounters two major challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels. To address these issues, we introduce a $\mathit{\text{Split}}\mathit{\text{Tokenization}}$ strategy, achieving stable and effective unsupervised KD. Additionally, we develop the $\mathit{\text{Distribution}}\mathit{\text{Contraction}}$ technique to integrate label supervision into the reconstruction objective. Our theoretical proof demonstrates that $\mathit{\text{GenDD}}$ with $\mathit{\text{Distribution}}\mathit{\text{Contraction}}$ serves as a gradient-level surrogate for multi-task learning, realizing efficient supervised training without explicit classification loss on multi-step sampling image representations. To evaluate the effectiveness of our method, we conduct experiments on balanced, imbalanced, and unlabeled data. Experimental results show that $\mathit{\text{GenDD}}$ performs competitively in the unsupervised setting, significantly surpassing KL baseline by $\mathbf{\text{16.29\%}}$ on ImageNet validation set. With label supervision, our ResNet-50 achieves $\mathbf{\text{82.28\%}}$ top-1 accuracy on ImageNet in 600 epochs training, establishing a new state-of-the-art.