VGGT: Visual Geometry Grounded Transformer

September 21, 2026

-infer key 3D attributes of a scene including camera parameters, point maps, depth maps, and 3D point tracks from 1-100s of views

-predicts all 3D quantities of interest together, in contrast to models that specialize in one thing (i.e. DepthAnything for monocular depth, NVS, etc.)

bundle adjustment - an optimization technique to jointly refine camera params and 3D scene geometry

-DUSt3R and MASt3R can only process two images at once, relying on pairwise reconstructions/optimizations in postprocessing

structure from motion (Sfm) - estimate camera params/sparse point cloud from images of a static scene from different views

multi-view stereo (MVS) - densely reconstruct the geometry of a scene from multiple overlapping images, often using camera params calculated from Sfm

depth map - $D \in \mathbb{R}^{H \times W}$ where each pixel has a scalar $D(u, v) = z$ corresponding to depth from the camera frame

-not necessarily the distance to the point

point map - $P \in \mathbb{R}^{H \times W \times 3}$ where each pixel stores $P(u, v) = (X, Y, Z)$ coordinate of the corresponding point in the camera coordinate frame

Formulation

-input sequence

(I_i)_{i=1}^N

of N RGB images

I_i \in \mathbb{R}^{3 \times H \times W}

observing the same 3D scene

-VGGT is a function

f

that maps the input sequence to a set of 3D annotations

f((I_i)_{i=1}^N) = (g_i, D_i, P_i, T_i)_{i=1}^N

g_i \in \mathbb{R}^9

- camera intrinsics/extrinsics

g = [q, t, f]

where

q

is the rotation quaternion,

t

is the translation vector,

f

is the field of view

D_i \in \mathbb{R}^{H \times W}

- depth map

P_i \in \mathbb{R}^{3 \times H \times W}

- point map

-viewpoint invariant, meaning that the 3D points are defined in the coordinate system of the first camera, taken as the world frame

T_i \in \mathbb{R}^{C \times H \times W}

of C-dim features for point tracking

equivariance: $f(Tx) = T'f(x)$

invariance: $f(Tx) = f(x)$

-image order is permutation equivariant except for the first frame (used for world grounding)

-uses DINO feature patches

-alternating frame-wise and global self-attention

-frame-wise self-attention - attend tokens within each frame separately

-global self-attention - attend to tokens across all frames jointly

-train to predict output tokens

(t_i^I, t_i^g, (t_{i, j}^R)_{j=1}^4)_{i=1}^N

where

t_i^I

are the image features (DINO)

-used to predict the depth map, point map, tracking features

t_i^g

is the camera token

-camera parameters

g_i

are predicted using a camera head on the camera token outputs

(t_{i, j}^R)_{j=1}^4

are 4 register tokens for extra learnable attributes

-trained to optimize multi-task loss:

\mathcal{L} = \mathcal{L}_{\text{camera}} + \mathcal{L}_{\text{depth}} + \mathcal{L}_{\text{pmap}} + \lambda \mathcal{L}_{\text{track}}

-normalize ground truth depth:

-express all quantities in the coordinate frame of the first camera

-calculate average Euclidean distance of all 3D points in the point map and scale to normalize camera translations, point map, and depth map