Vision Transformers Need Registers (Darcet et al.)

October 6, 2026

-in large, well-trained ViTs (e.g. DINO and DINOv2), some patch tokens in low-information regions develop abnormally high norms

-probing shows these high-norm tokens contain much less information about their original patch, including its position and pixel content

-this suggests the network has repurposed them for internal, global computation rather than local visual representation

-this is undesirable because a patch token that becomes scratch space no longer provides a clean spatial representation of its corresponding region