Self-Supervised Learning

Self-supervised learning creates a training signal from unlabeled data: predict masked content, match two augmented views, order frames, or reconstruct missing features. The goal is usually a reusable representation that can be adapted by transfer learning, not the pretext task itself. Autoencoders and contrastive learning are two major families.

Making labels from the data

Self-supervision invents a target from the input itself. The common recipes differ in what they hide and ask the model to recover:

Pretext task	Signal derived from $x$	Example
Masked prediction	reconstruct hidden tokens or patches	BERT, masked autoencoders
Contrastive views	two augmentations should embed alike	SimCLR
Ordering	recover the correct sequence order	frame-order prediction
Inpainting / colorization	recover removed content	image colorization

Formally, let $t$ be a transformation that hides or augments part of $x$. A masked-prediction objective can be written as

$$\underset{\theta }{\min }{\mathbb{E}}_x\left[L(g_{\theta }(t(x)),s(x))\right],$$

where $s(x)$ is the hidden target derived from the same example. Contrastive self-supervision instead forms two views $v_1=t_1(x)$ and $v_2=t_2(x)$ and pulls their embeddings together while pushing apart other examples.

Worked example

This snippet trains a small predictor to reconstruct masked features from visible features and reports the loss change plus learned weights.

import torch
import torch.nn.functional as F
 
torch.manual_seed(9)
X = torch.randn(100, 3)
y = X[:, 2:3]
visible = X[:, :2]
pred = torch.nn.Linear(2, 1)
opt = torch.optim.SGD(pred.parameters(), lr=0.2)
start = F.mse_loss(pred(visible), y).item()
for _ in range(80):
    opt.zero_grad()
    loss = F.mse_loss(pred(visible), y)
    loss.backward()
    opt.step()
print("masked_feature_loss_before", round(start, 4), "after", round(loss.item(), 4))
print("learned_weights", torch.round(pred.weight.detach(), decimals=3).tolist())

Observed output:

masked_feature_loss_before 1.4685 after 0.8666
learned_weights [[0.014000000432133675, -0.19900000095367432]]

The target is not externally labeled; it is the hidden third feature. Loss falls because the model extracts whatever correlation exists in the visible features, but the remaining error shows that the hidden feature is not fully determined.

Caveats

The pretext task must require information that transfers. A model can solve a bad pretext task through shortcuts, such as color artifacts or augmentation fingerprints. In video, leakage through adjacent frames is especially easy, which is why self-supervised video work needs careful sampling and masking.

References

• Bengio, Courville, and Vincent, 2012, Representation Learning
• Chen et al., 2020, A Simple Framework for Contrastive Learning of Visual Representations

Nav

Section — Deep Learning

← Autoencoders Contrastive Learning →