Beyond Instance Consistency:
Investigating View Diversity in Self-supervised Learning

Abstract

Self-supervised learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However, this assumption breaks down for non-iconic data, where different views may contain distinct objects or semantic information. In this paper, we investigate the effectiveness of SSL when instance consistency is not guaranteed. Through extensive ablation studies, we demonstrate that SSL can still learn meaningful representations even when positive pairs lack strict instance consistency. Furthermore, our analysis further reveals that increasing view diversity, by enforcing zero overlapping or using smaller crop scales, can enhance downstream performance on classification and dense prediction tasks. However, excessive diversity is found to reduce effectiveness, suggesting an optimal range for view diversity. To quantify this, we adopt the Earth Mover’s Distance (EMD) as an estimator to measure mutual information between views, finding that moderate EMD values correlate with improved SSL learning, providing insights for future SSL framework design. We validate our findings across a range of settings, highlighting their robustness and applicability on diverse data sources.

What is Instance Consistency?

SSL aligns different views of the same image, relying on the instance consistency paradigm, which considers each image as a separate class: different augmentations of an image are treated as positive pairs, while augmentations from other images serve as negatives.

This paradigm works remarkably well for iconic datasets, where images typically feature a single, dominant object. In contrast, non-iconic datasets, such as COCO and OpenImages, often feature complex scenes with multiple objects and diverse backgrounds, so two augmented views from the same image may not contain the same object or share consistent semantic information (Figure 1). Surprisingly, despite the breakdown of instance consistency on non-iconic data, SSL methods can still achieve competitive performance, raising a critical research question: Is instance consistency a strict requirement for self-supervised learning?

How Does SSL React to Different Levels of Instance Consistency?

We conduct ablation experiments with MoCo-v2 and DINO, systematically varying the shared instance semantics between the positive pair (v₁, v₂). Given target instances with ground-truth bounding boxes, we introduce five configurations (Figure 2):

1. Completely Random Crop (Baseline). Two views are randomly cropped from the same image without any spatial constraint, the default setup in MoCo-v2 and DINO.
2. Zero Spatial Overlap. Views are sampled with no spatial overlap.
3. Instance vs Bg. v₁ is sampled around a foreground instance, while v₂ contains only background information.
4. Only Bg. Two views are sampled purely from background regions.
5. Lower Bound. Each view is sampled from entirely different images.

Results. As expected, the Lower Bound yields the lowest performance, highlighting the importance of shared information between positive pairs. Surprisingly, all other configurations (Zero Spatial Overlap, Instance vs Bg, and Only Bg) outperform the baseline in classification (Table 1) and object detection (Table 2). This suggests that SSL is capable of leveraging broader contextual cues beyond instance consistency, including shared background patterns, consistent camera viewpoints, and general color style, as long as both views are sampled from the same image.

How Much View Diversity is Beneficial?

While these configurations reduce the instance consistency, they simultaneously increase diversity between positive pairs, yet still serve as a valid, and even better supervision signal. We regulate this diversity by varying crop scales:

1. Smaller Crop. A smaller crop scale minimizes shared information to increase diversity between positive pairs.
2. Larger Crop. Larger crop scales preserve more shared information, thereby reducing the diversity.
3. Smaller Crop with Zero Spatial Overlap (Smaller Crop^† in Tables 1 and 2). Combining both maximizes the diversity.

Results. Smaller Crop consistently outperforms the baseline, whereas Larger Crop leads to a significant performance drop, suggesting that excessive redundancy between views can hinder SSL effectiveness. Interestingly, while Smaller Crop and Zero Spatial Overlap individually boost performance, their combination does not yield additional gains and instead results in a slight performance drop.

Earth Mover’s Distance as Diversity Estimator

To quantify this diversity, we adopt Earth Mover’s Distance (EMD) as an estimator to measure the shared information between positive pairs, which provides an estimation of augmentation quality before model pre-training.

Implementations. Following Zhang et al. (2020), we extract the features of patches of the two views with a pre-trained ResNet-50, using two strategies (Figure 3): grid-based, where each view is divided into uniform grids with grid factors of 2 and 3, and sampling-based, where each view is randomly sampled into 9 patches of varying sizes and aspect ratios.

Results. Figure 4 reveals a clear reverse-U relationship between the EMD score and downstream accuracy. Configurations with moderate EMD scores (ranging from 3 to 4) consistently yield the highest performance, while very high (above 5) or very low EMD scores (below 2) exhibit a drop in downstream accuracy, indicating that extreme overlap or excessive diversity between views can hinder SSL’s ability to learn meaningful feature representations.

(a) Detection, MoCo-v2

(b) Classification, MoCo-v2

(c) Detection, DINO

(d) Classification, DINO

EMD scores in Figure 4 are scaled by 10 for better visualization.

Suggestions for Positive Pair Selection. Our results suggest pre-calculating EMD scores before pre-training: the optimal EMD range lies between the baseline score (upper bound) and that of Smaller Crop^† (lower bound).

BibTeX

@article{qin2025beyond,
  title   = {Beyond Instance Consistency: Investigating View
             Diversity in Self-supervised Learning},
  author  = {Qin, Huaiyuan and Yang, Muli and Hu, Siyuan and Hu, Peng and
             Zhang, Yu and Gong, Chen and Zhu, Hongyuan},
  journal = {Transactions on Machine Learning Research},
  issn    = {2835-8856},
  year    = {2025},
  url     = {https://openreview.net/forum?id=urWCU3YMA0}
}

Acknowledgements

This research is supported by the EDB Space Technology Development Programme under Project S22-19016-STDP and the National Natural Science Foundation of China under Grant 62576089. The authors sincerely thank the Reviewers and Action Editor for their valuable and constructive feedback.

This page is adapted from the ECHO project page, itself built on the Nerfies template.

References

1. Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. DeepEMD: Few-Shot Image Classification with Differentiable Earth Mover’s Distance and Structured Classifiers. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.