DiD It in 87 Minutes:
A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detection

Abstract

While linear attention is a compelling mechanism for high-resolution object detection due to its reduced cost for global token mixing, converting the Softmax-attention ViT backbone of a trained detector into a linear-attention one is not a trivial drop-in replacement. Directly swapping the attention operator leads to severe performance degradation, and generic label-free distillation, though effective for classification, often fails on detection tasks. We argue that the central challenge is detector-interface preservation: the converted backbone must reproduce the exact feature tensors expected by the fixed downstream detector, rather than merely imitating internal Softmax hidden states. To address this, we introduce Detector-Interface Distillation (DiD), a label-free conversion method that exclusively trains the linear-attention backbone by aligning detector-facing interface tensors with those of a frozen Softmax teacher. On DOTA-v1.5, DiD substantially outperforms established baselines and matches supervised, fully trained linear models. Adaptation completes in roughly 87 minutes on 4 GPUs, and the linearized backbone cuts inference latency by ∼62% and peak memory by ∼49%. We hope our findings offer the community a simple, label-free route to reusing trained Softmax detectors as efficient linear ones, and encourage interface-aware objectives in future architecture-conversion work.

What is DiD?

Label-Free Softmax-to-Linear Conversion. Rather than retraining the full detector, we replace the quadratic attention modules in the backbone of a trained ViT detector with linear attention and perform a short label-free adaptation stage that trains only the backbone, as illustrated in Figure 1. The difficulty is that the converted backbone must remain compatible with a detector pipeline that was trained around the original Softmax feature distribution.

Overview. Detector-Interface Distillation (DiD) adapts a linear-attention student backbone to a detector that was originally trained with a Softmax-attention backbone. DiD aligns the multi-scale tensors produced at the output of the neck/FPN, while updating only the student backbone. Softmax and linear attention implement different token-mixing rules, so their intermediate hidden states need not follow identical trajectories. DiD instead preserves the feature boundary on which the fixed detector depends. Algorithm 1 summarizes this computation in pseudocode.

# B_t: frozen Softmax teacher backbone# B_s: trainable linear-attention student# N_t, N_s: identical frozen neck/FPN copies# H: frozen detector heads (unused in loss)# beta: weights, beta[i] for interface m = i+1B_s.params = B_t.params # copy compatible weightsfor x in unlabeled_images: # no detection labels    R_t = B_t.readout_features(x) # frozen path    R_s = B_s.readout_features(x) # trainable    Z_t = N_t(R_t) # tensors z_t^(1), ..., z_t^(M)    Z_s = N_s(R_s) # tensors z_s^(1), ..., z_s^(M)    # weighted detector-interface reconstruction    loss = 0    for i in range(M):        loss += beta[i]*mse(Z_s[i], Z_t[i].detach())    loss /= sum(beta)    loss.backward() # update only student backbone    update(B_s) # AdamW step# deploy: drop teacher, reuse the frozen detectorconverted_detector = H o N_s o B_s

Weighted Interface Objective. DiD aligns corresponding teacher and student interface tensors using the weighted reconstruction objective

(9)

where β_m > 0 controls the contribution of interface output m.

Conversion and Deployment. After conversion, the teacher branch is discarded. The converted detector reuses the original neck/FPN and prediction modules without additional fine-tuning.

Why Is Generic Transfer Insufficient?

To test whether a classification-oriented transfer objective is sufficient for object detection, we compare direct copy and generic LightlyTrain-style global-feature distillation on ImageNet and CIFAR-100 classification, and on DOTA-v1.5 detection with RetinaNet.

Direct copy collapses across all three tasks (Table 1), confirming that the attention operator cannot be replaced without adaptation. Generic distillation, however, exhibits a clear task gap: it retains 97.5% and 97.2% of the Softmax-reference accuracy on ImageNet and CIFAR-100, respectively, but only 54.0% of the reference mAP on DOTA-v1.5.

Where Should Conversion Be Supervised?

Main Results

Experimental Setup. We evaluate label-free detector conversion on DOTA-v1.5, a high-resolution aerial-detection benchmark where global token mixing is costly and small objects are common. We use RetinaNet and Faster R-CNN with a ViT-S backbone. We evaluate two linear-attention feature maps: φ(x) = ReLU(x) following Han et al. (2024), and φ(x) = ELU(x) + 1 following Katharopoulos et al. (2020).

Baselines. We compare DiD with four label-free alternatives and one supervised reference. Direct copy replaces the attention operator and evaluates the copied parameters without adaptation. Generic distillation uses LightlyTrain to align global teacher-student representations. Softmax-generated pseudo-label distillation uses AutoDistill to train the student from the frozen teacher’s predicted detections. Hidden-state distillation follows RADLADS-style intermediate representation matching across transformer layers. Full training trains the corresponding linear-attention detector using detection labels and is included only as a supervised reference.

Table 3 establishes three consistent trends: Firstly, direct copy produces near-zero mAP for both attention kernels and detector families. Secondly, generic representation matching and pseudo-label supervision recover only part of the lost detector performance. Thirdly, DiD is the strongest label-free method. Its performance is also comparable to the supervised linear-attention references.

Interface Alignment and Conversion Dynamics

To understand why DiD outperforms hidden-state distillation, we compare how the two objectives affect teacher-student alignment inside the backbone and at the detector-facing interface. Figure 2(a) compares the normalized feature error at internal backbone states and detector-facing tensors.

(a)

(b)

All-layer hidden-state distillation achieves lower mismatch within the transformer, whereas DiD achieves lower mismatch at the detector-facing interface and higher detector accuracy.

Figure 2(b) shows that DiD maintains an advantage over hidden-state distillation throughout conversion and reaches the supervised linear-training regime after approximately 20k-30k steps.

Inference Efficiency

Figure 3 compares backbone-only inference for Softmax attention and the ReLU-linearized ViT. The linearized backbone reduces average latency by approximately 62% and peak memory by approximately 49%.

BibTeX

@article{qin2026did,
  title   = {DiD It in 87 Minutes: A Label-Free Softmax-to-Linear
             Adaptation of Vision Transformers for Object Detection},
  author  = {Qin, Huaiyuan and Goenawan, Gabriel James and Lin, Zihang and
             Yang, Muli and Zhu, Hongyuan},
  journal = {arXiv preprint arXiv:2608.22368},
  year    = {2026}
}

Acknowledgements

This page is adapted from the ECHO project page, itself built on the Nerfies template.

References

1. Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang. Bridging the Divide: Reconsidering Softmax and Linear Attention. Advances in Neural Information Processing Systems (NeurIPS), 2024.
2. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. International Conference on Machine Learning (ICML), 2020.
3. Daniel Goldstein, Eric Alcaide, Janna Lu, and Eugene Cheah. RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale. Conference on Language Modeling (COLM), 2025.