Contrastive & Self-Supervised Learning

What is Self-Supervised Contrastive Learning?

Supervised learning requires millions of human labeled samples $(x, y)$.

Self-Supervised Contrastive Learning trains deep neural networks on unlabelled data by learning to distinguish similar inputs from dissimilar inputs.

The core intuition:

Pull representations of Positive Pairs (similar samples) close together in embedding space, while pushing Negative Pairs (different samples) far apart:

  UNLABELLED DATA ──► Augmentation View 1 (x_i) ──► [ ENCODER f(x) ] ──► Embedding z_i ──┐ (PULLED TOGETHER!)
                   ──► Augmentation View 2 (x_j) ──► [ ENCODER f(x) ] ──► Embedding z_j ──┘
  
  DIFFERENT IMAGE ──► Augmentation View (x_k)   ──► [ ENCODER f(x) ] ──► Embedding z_k ─── (PUSHED AWAY!)

InfoNCE Loss Function

Contrastive learning is driven by the InfoNCE (Information Noise Contrastive Estimation) Loss:

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp\left(\text{sim}(z_i, z_j) / \tau\right)}{\sum_{k=1}^K \exp\left(\text{sim}(z_i, z_k) / \tau\right)}$$

• $\text{sim}(z_i, z_j) = \frac{z_i^T z_j}{\|z_i\| \|z_j\|}$: Cosine similarity between normalized embeddings.
• $\tau$ (tau): Temperature hyperparameter (typically $\tau = 0.07$) scaling prediction sharpness.
• Numerator: Similarity of Positive Pair $(z_i, z_j)$.
• Denominator: Sum of similarities across all $K$ Negative Pairs in the batch.

InfoNCE is mathematically equivalent to multi-class Cross-Entropy loss where the target is identifying the single positive pair out of $K$ candidate choices.

Foundational Architectures

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. SimCLR (Google 2020)  │ 2. MoCo (Meta 2020)      │ 3. CLIP (OpenAI 2021)    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Requires HUGE mini batch │ Uses dynamic Memory Queue│ Contrastive vision-text  │
│ sizes (4096) to collect  │ and Momentum Encoder to  │ pretraining across 400M  │
│ enough negative samples. │ decouple queue from batch│ Image Text web pairs.    │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. SimCLR (Chen et al., 2020)

• Applies two data augmentations (Random Crop + Color Jitter + Blur) to an image to form positive pair $(x_i, x_j)$.
• All other $2(N-1)$ augmented images in the mini-batch serve as negative samples.
• Key Insight: Proves that Projection Heads (MLP layers placed after encoder representations during training and discarded for downstream tasks) boost feature quality by $10\%$.

2. MoCo (Momentum Contrast - He et al., 2020)

SimCLR requires massive batch sizes ($4096$) to get enough negative samples.

MoCo maintains a dynamic Memory Queue of 65,536 negative keys:

• Encoders keys using a slow Momentum-Updated Network ($\theta_k \leftarrow m \theta_k + (1-m) \theta_q$).
• Decouples negative sample pool size from GPU mini-batch size!

3. CLIP (Contrastive Language-Image Pretraining - Radford et al., 2021)

Applies contrastive learning across two modalities (Image + Text):

• Positive Pair: (Image $i$, Matching Caption $i$).
• Negative Pairs: All non-matching (Image $i$, Caption $j$) pairs in the batch.

Enables zero-shot image classification and powers text-to-image models (Stable Diffusion, DALL-E).

Say this out loud

Contrastive Learning is a self-supervised paradigm that pulls positive pair embeddings together while pushing negative pair embeddings apart using InfoNCE loss. SimCLR uses augmented views of unlabelled images with high batch sizes. MoCo uses a momentum encoder and negative sample queue, while CLIP applies contrastive learning to align image and text representations.

Followups to expect

1. What is Collapse in Self-Supervised Learning? A failure mode where the encoder outputs a constant vector $z = c$ for all inputs, minimizing distance for positive pairs trivially. Avoided in BYOL (Bootstrap Your Own Latent) using asymmetric predictor networks and EMA updates.
2. What is Supervised Contrastive Learning (SupCon)? Extending InfoNCE loss to labeled data by defining all samples sharing the same class label $y$ as positive pairs, outperforming standard Cross-Entropy classification.