Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

September 17, 2026

Video LDMs - first pre-train LDM on images, then introduce a temporal dimension

-a relatively small set of captioned videos can be used to temporally align different text-to-image LDMs (i.e. Stable Diffusion) for high-res, long video generation

score-matching - st(x)=∇xlog⁡pt(x)s_t(x) = \nabla_x \log p_t(x)

-learn a vector field that informs the direction which increases the probability density
-different from flow matching, where a velocity field to transform noise to the original image is learned
-add temporal attention to the image LDM to learn important aspects of video