Kavli Affiliate: Zhiting Tian | First 5 Authors: Huiyuan Tian, Huiyuan Tian, , , | Summary: While feature-based knowledge distillation has proven highly effective for compressing CNNs, these techniques unexpectedly fail when applied to Vision Transformers (ViTs), often performing worse than simple logit-based distillation. We provide the first comprehensive analysis of this phenomenon through a […]
Continue.. Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers