Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better

Knowledge distillation field delicately designs various types of knowledge to shrink the performance gap between compact student and large-scale teacher. These existing distillation approaches simply focus on the improvement of $\mathit{\text{knowledge}}\mathit{\text{quality}}$, but ignore the significant influence of $\mathit{\text{knowledge}}\mathit{\text{quantity}}$ on the distillation procedure. Opposed to the conventional distillation approaches, which extract knowledge from a fixed teacher computation graph, this paper explores a non-negligible research direction from a novel perspective of $\mathit{\text{knowledge}}\mathit{\text{quantity}}$ to further improve the efficacy of knowledge distillation. We introduce a new concept of knowledge decomposition, and further put forward the $\mathbf{\text{P}}$artial to $\mathbf{\text{W}}$hole $\mathbf{\text{K}}$nowledge $\mathbf{\text{D}}$istillation~($\mathbf{\text{PWKD}}$) paradigm. Specifically, we reconstruct teacher into weight-sharing sub-networks with same depth but increasing channel width, and train sub-networks jointly to obtain decomposed knowledge~(sub-networks with more channels represent more knowledge). Then, student extract partial to whole knowledge from the pre-trained teacher within multiple training stages where cyclic learning rate is leveraged to accelerate convergence. Generally, $\mathbf{\text{PWKD}}$ can be regarded as a plugin to be compatible with existing offline knowledge distillation approaches. To verify the effectiveness of $\mathbf{\text{PWKD}}$, we conduct experiments on two benchmark datasets:~CIFAR-100 and ImageNet, and comprehensive evaluation results reveal that $\mathbf{\text{PWKD}}$ consistently improve existing knowledge distillation approaches without bells and whistles.