Attention Transfer Is Not Universally Effective for Vision Transformers

Abstract

A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard student ViT, is sufficient to recover the full benefit of the teacher's pre-trained weights. We revisit this finding on a comprehensive benchmark of 20 teachers from 11 well-known ViT families and reveal that Attention Transfer is not universally effective. While 7 families transfer successfully, 4 consistently fail, falling up to 5.1% below the from-scratch no-transfer baseline. Further results demonstrate that this failure is family-consistent across model sizes, and persists under extended training durations, different transfer datasets, and out-of-distribution evaluations. Controlled analyses then consistently localize the problem to the attention-routing channel, indicating that the key issue is not whether the student can match the teacher's attention patterns, but whether the matched patterns remain functional for the student. Crucially, we identify architectural mismatch between the pre-trained teacher and the standard student as the primary mechanism. By adding only the teacher's native architectural components to the student in a randomly initialized state, we completely reverse the failure for all 4 families. Notably, these components alone do not improve from-scratch training, confirming that they specifically unlock the usability of the teacher's attention. We further show that this failure is not explained by the inadequate choice of transfer loss, differences in pre-training recipes, or the intrinsic quality of attention maps. Our findings refine the prevailing understanding of attention in ViT representations: attention is sufficient only when the student architecture matches the teacher.

What is Attention Transfer?

Following Li et al. (2024), we study Attention Transfer, where a randomly initialized student Vision Transformer (ViT) is trained to inherit the full benefit of a pre-trained teacher without loading the teacher’s full pre-trained weights. We study two forms of Attention Transfer from a pre-trained teacher ViT to a standard student of the same size: Attention Copy, which directly copies the teacher’s attention projection weights and keeps them frozen; and Attention Distillation, which trains the student to match the teacher’s attention patterns through an auxiliary transfer objective.

Model Zoo. We revisit Attention Transfer on a comprehensive model zoo which consists of 20 pre-trained weights from 11 well-known ViT families, including DeiT, DINO, MoCov3, MAE, iBOT, SigLIP2, SAM, DINOv2, DINOv2-wr, CLIP, and BEiTv2, spanning three model sizes: Tiny, Small, and Base. All Attention Transfer experiments are conducted on ImageNet-1K classification with the default protocol in Li et al. (2024) using the standard teacher-student architecture.

No-Transfer Baseline. As a primary reference, we define the No-Transfer (NoT) baseline, where the same standard ViT is randomly initialized and trained from scratch for 20 epochs. Attention Transfer from one given pre-trained weights is considered a “failure” when the final accuracy falls below this baseline, indicating that imposing attention patterns from the teacher is less effective than learning from scratch under the same student architecture.

Attention Transfer Is Not Universally Effective

The results reveal a clear family-dependent split, and this split boundary is consistent across model sizes rather than driven by isolated pre-trained weights. Specifically, 7 families, including DeiT, DINO, MoCov3, MAE, iBOT, SigLIP2, and SAM, improve over the No-Transfer baseline, with gains reaching 13.9% for DeiT-T under Attention Copy. In contrast, 4 families: DINOv2, DINOv2-wr, CLIP, and BEiTv2, consistently fall into the failure region. The worst case is DINOv2-wr-B, which underperforms the No-Transfer baseline by 5.1% under Attention Copy. Notably, all teachers evaluated in Li et al. (2024) belong exclusively to our success group, while the failure families are absent from their evaluations.

Extended Training Robustness. A natural counter-hypothesis is that the failure families might suffer from slower convergence under the transferred attention patterns. We extend the training schedule from 20 to 100 epochs for both the success and failure families. The performance gap persists throughout the entire extended learning process. While successful families maintain strong positive gains over the No-Transfer baseline, the failure families remain consistently negative even with substantially longer training.

Localizing the Failure

Having established the robust failure of Attention Transfer for specific teacher families, we now investigate exactly where the transferred pre-trained weights harm the student. We diagnose this problem systematically through the three-level analysis below.

Layer-wise Decomposition. We transfer attention from the top or bottom k layers of one representative failure teacher, DINOv2-S, under both Attention Copy and Attention Distillation. No layer subset recovers the positive gains observed for success families. Instead, performance remains below the No-Transfer baseline across all tested transfer depths. This rules out a single problematic depth as the source of failure and suggests that the failure is distributed broadly across the transferred attention patterns.

Architectural Compatibility Is the Key

Motivation. This leads us to hypothesize that the missing factor is architectural compatibility in the default protocol: transferred attention remains usable only when the student preserves the architectural components under which the teacher originally learned its attention patterns. Concretely, each failure family contains one or more such components that are absent from the standard student architecture: LayerScale in DINOv2 and DINOv2-wr, PreLayerNorm in CLIP, and LayerScale together with RelativePositionBias in BEiTv2. These native components define the scaling, normalization, and positional context under which the teacher’s attention was learned.

Native-architecture Rescue. We augment the student architecture by adding only the teacher-native missing components. The inserted components are initialized randomly and trained jointly with the student, with no pre-trained weights loaded.

Why Architectural Mismatch Causes Failure. To explain why this rescue is effective, we further test the architectural compatibility hypothesis directly at the attention level. We measure this via per-layer KL divergence between the transferred student’s attention and the teacher’s native attention, for both standard-arch and native-arch students under Attention Distillation. The standard-arch student matches the early layers almost perfectly with near-zero KL across all three failure families, but exhibits a sharp divergence spike in the last three layers. By contrast, the native-architecture student remains consistently low and smooth across all layers, without any collapse for transferring attention.

Ruling Out Alternative Explanations

We now ask whether simpler explanations could account for the observed failure. We therefore investigate three natural alternative explanations below.

Not the Transfer Loss. MSE exhibits the same family-level trends as CE when the two losses’ magnitudes are matched by adjusting λ. For success teachers (DeiT-S, DINO-S), both losses produce monotonic improvements as λ increases. For failure teachers (DINOv2-S, DINOv2-wr-S), both losses instead produce monotonic degradation as λ increases. These results rule out the inadequate choice of transfer loss as the primary explanation for the observed failure.

Not the Pre-training Recipe. As shown in Table 6, for each candidate explanation from the failure families, we can identify at least one counter-example with that property from the success families, or one teacher that lacks it but still fails. Overall, none of these single-factor recipe properties predicts the observed family-level failure boundary. This rules out pre-training recipe differences as the primary explanation.

BibTeX

@inproceedings{qin2026attention,
  title     = {Attention Transfer Is Not Universally Effective for Vision Transformers},
  author    = {Qin, Huaiyuan and Yang, Muli and Goenawan, Gabriel James and
               Hu, Peng and Gong, Chen and Peng, Xi and Zhu, Hongyuan},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

Acknowledgements

This research is supported by the NSF of China (Grant No. 62336003, 12371510) and the Shanghai Municipal Special Program for Basic Research on General AI Foundation Models (Grant No. 2025SHZDZX025G12), and is conducted in collaboration with ST Engineering Geo-Insights Pte Ltd. We thank the anonymous reviewers and area chairs for their valuable feedback and constructive comments. This page is adapted from the ECHO project page, itself built on the Nerfies template.

References

1. Alexander C. Li, Yuandong Tian, Beidi Chen, Deepak Pathak, and Xinlei Chen. On the surprising effectiveness of attention transfer for vision transformers. Advances in Neural Information Processing Systems (NeurIPS), 2024.