Figure 1. From noisy supervision to meaningful transfer. SiNGER filters out artifacts in teacher representations and guides the student toward clean, interpretable, and transferable features.
Contributions
Propose a novel distillation framework (SiNGER) that refines teacher signals via the LoRA-based adapter with nullspace initialization to guide effective perturbations.
Analyze a fundamental limitation of naïve ViT distillation, showing degraded transfer on downstream benchmarks along with qualitative evidence.
Provide extensive ablation studies to analyze the contribution of each component in SiNGER and validate the robustness of our framework.
Demonstrate through extensive experiments that our method exceeds baseline performance across tasks and produces more interpretable feature maps.
Abstract
Vision Transformers are widely adopted as the backbone of vision foundation models, but they are known to produce high-norm artifacts that degrade representation quality.
When knowledge distillation transfers these features to students, high-norm artifacts dominate the objective, so students overfit to artifacts and underweight informative signals, diminishing the gains from larger models.
Prior work attempted to remove artifacts but encountered an inherent trade-off between artifact suppression and preserving informative signals from teachers.
To address this, we introduce Singular Nullspace-Guided Energy Reallocation (SiNGER), a novel distillation framework that suppresses artifacts while preserving informative signals.
The key idea is principled teacher feature refinement: during refinement, we leverage the nullspace-guided perturbation to preserve information while suppressing artifacts.
Then, the refined teacher's features are distilled to a student.
We implement this perturbation efficiently with a LoRA-based adapter that requires minimal structural modification.
Extensive experiments show that SiNGER consistently improves student models, achieving state-of-the-art performance in multiple downstream tasks and producing clearer and more interpretable representations.
SiNGER
Figure 3. The overall pipeline of knowledge distillation with the SiNGER adapter at l-th layer.
Figure 4a. The cosine similarity between x-marked patch and every path of another feature map is visualized.
In each pair, the left side is the original feature maps with artifacts, and the right side is the refined feature map.
In the refined feature map, the cosine similarity between x-marked patch and all other patches are visualized.
Figure 4b. The cosine similarity between x-marked patch and every patch of another feature map is visualized.
The original teacher's representation and the SiNGER-refined representation after the next layer are compared.
SiNGER preserves information after refinement with respect to the next layer.
Figure 2. Input image, two pretrained ViTs, and three Large → Tiny distilled variants.
Each panel shows similarity from the x-marked patch.
SiNGER most closely preserves teacher semantics, showing the most coherent teacher-consistent similarity patterns.
Experimental Results
Downstream Task Evaluation
To evaluate SiNGER-distilled ViT as a VFM, we adopt the student network to a diverse set of downstream tasks.
Specifically, we consider six representative benchmarks: ImageNet-1K validation set for large-scale classification, ADE-20K for semantic segmentation, NYUd-v2 for depth estimation, iNaturalist-2019 for long-tail classification, ImageNet-R and ImageNet-v2 for domain shift robustness, and four fine-grained classification datasets.
We evaluate SiNGER on multiple teacher-student configurations spanning both the canonical ViT and the modern DeiT-III, covering a range of model scales.
SiNGER demonstrates consistent improvements over FitNet and ViTKD on most benchmarks.
On IN-val, ADE-20K, NYUd-v2, DS, and FG, SiNGER yields large gains, approaching teacher performance despite the smaller capacity.
Table 1. Multi-task linear evaluation results. ImageNet-1K validation (IN-val) for large-scale classification, ADE-20K for semantic segmentation, NYUd-v2 for monocular depth estimation, iNaturalist2019 (iNat2019) for long-tail learning, ImageNet-R and ImageNet-v2 for domain shift (DS), and four fine-grained classification (FG) benchmarks: FGVC-Aircraft, Oxford-IIIT Pet, Food-101, and Flowers-102 were tested. Δ rows indicate the performance gains of SiNGER, computed against the best-performing baseline among the distilled students (underlined).
Adapter Operation
We empirically analyze how the optimized adapter operates on ImageNet-1K. To probe the coupling
with the next layer, we evaluate at the 17th intermediate layer.
The teacher produces high-norm artifacts that are distinctly gathered as a group. We observed that SiNGER effectively draws such artifacts into the normal-patch range while preserving informative features. This results in stabilized gradient flow through the normal patches (Figure 6).
The 17th and 18th layers yield cosine similarity of 0.9566 and 0.9731 with negligible variance, respectively, which is clearly considered similar (Table 3).
Figure 6. Patch-norm distributions of the original and refined features.
Artifacts are scale-colored separately with inferno.
Table 3. Patch-wise cosine similarity (perimage mean over patches) on ImageNet-1K.
Component Ablation
We report four ablations, focusing on initialization, losses, hyperparameters, and distillation layers.
Table 4. Full ablation study on the nullspace initialization and loss terms.
In Table 5a, initialization markedly increases alignment to N_l+1: E_safe reaches 0.83/0.76 for φup,l at l=17, 23, and 0.55/0.58 for φdown,l.
Both are under 0.27 for random initialization.
This provides strong evidence that the initialization guides optimization into the null space, yielding substantially higher Esafe across layers and for both φup,l and φdown,l, which indicates successful guidance toward the null space directions.
Meanwhile, Eprob remains lower or comparable under SiNGER, but our objective is not to minimize Eprob per se; rather, to ensure that the learned parameters predominantly occupy N_l+1 so as to suppress high-norm amplification while preserving useful directions.
Although nullspace initialization stabilizes the refinement direction and avoids perturbations that conflict with the subsequent block, it does not lead to measurable performance improvements (Table 4).
Table 5a. Ablation studies on the initialization method.
In Table 5b, lower DG indicates better preservation of pairwise feature relations.
Compared to outlier suppression loss alone, adding information preservation loss nearly halves the DG distance (14.22 → 7.25) and substantially improves teacher-student alignment (72.36 → 41.71).
Thus, the information preservation term prevents degenerate updates and maintains the relational geometry that is crucial for effective transfer.
Table 4 shows that adding Loutlier yields the largest improvement on both ImageNet-1K and ADE-20K, as it directly mitigates the dominant artifact tokens that bias distillation.
Information preservation loss provides additional gains by enforcing information consistency between the teacher and student.
When all components are combined, the student reaches its best performance across all tasks, demonstrating that SiNGER functions most effectively as an integrated framework rather than as a set of independent mechanisms.
These results highlight the individual contribution of each component.
Table 5b. Ablation studies on the information preservation loss.
BibTeX
@inproceedings{yu2026singer,
title={SiNGER: A Clearer Voice Distills Vision Transformers Further},
author={Yu, Geunhyeok and Jeong, Sunjae and Choi, Yoonyoung and Kim, Jaeseung and Hwang, Hyoseok},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=exjrxVc2yT}
}
