Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

August 31, 2026

Training Objective: predict the representations of various target blocks in an image from a context block

-it is crucial to

1.sample target blocks with a sufficiently large scale

2.use sufficiently informative context block

-training recipe, not a model architecture

self-supervised methods

-invariance-based methods

-different views/augmentations of the same image should produce similar representations

-example: take two (hand crafted) augmentations

x_1 = t_1(x)

and

x_2 = t_2(x)

of image

x

-train the encoder

f_\theta

such that

z_1 = f_\theta(x_1) \approx z_2 = f_\theta(x_2)

-become invariant to transformations that don’t change semantic context

-generative methods

-hide/corrupt some visual information and learn a representation that can be used to train the model to reconstruct it

joint-embedding architecture (invariance-based) + generative architecture (generative) = joint embedding predictive architecture (JEPA)

-use abstract prediction targets to learn more semantic features instead of unnecessary pixel-level details

-I-JEPA is 2.5x faster than training ViT-H/14 with iBOT and 10x more efficient than training ViT-H/14 with MAE

energy-based models (EBMs) - $E(x, y)$ is high when $x$ and $y$ are incompatible/dissimilar, and low otherwise