Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-training
October 14, 2026
-traditional ViT architectures encode separate views in isolation and then fuse them
-Muskie jointly attends to and fuses viewpoints, letting different scene views inform each other
-achieves higher multi-view correspondence accuracy than DINO
-trained to reconstruct heavily masked content in one view by finding and using geometric correspondences from other views
Muskie vs. VGGT
-VGGT is explicitly supervised for 3D reconstruction and optimized for geometry
-extracts DINO features for each image separately before fusing them
-treats the first image view as the global coordinate system, so it lacks permutation equivariance
-Muskie learns multi-view features through masked image modeling
-jointly processes multiple images during feature extraction
-treats all views equally, creating permutation equivariance
-uses very aggressive masking and alternating attention ratios to prevent the model from reconstructing masked regions using only intra-view cues
Methods
-uses alternating frame-wise and global attention, similar to VGGT
-effectively associates spatially corresponding regions between viewpoints
-masking ratio is significantly higher than the 75% used in MAE
-spatially concentrated, contiguous mask blocks make intra-view reconstruction unviable
Objective: minimize pixel-space reconstruction error.
Training data: DL3DV, ScanNet++, Waymo, RealEstate10K, HyperSim, BlendedMVG, and ARKitScenes.