High-Resolution Image Synthesis with Latent Diffusion Models

September 14, 2026

-to save on computational cost of training diffusion models in pixel state, train them in the latent space of pretrained autoencoders
x→Encoder Ez→Diffusion Modelz^→Decoder Dx^x\xrightarrow{\text{Encoder } E}z\xrightarrow{\text{Diffusion Model}}\hat{z}\xrightarrow{\text{Decoder } D}\hat{x}

perceptual compression - using an autoencoder remove high-frequency detail, more computationally suitable for diffusion

-only needs to be trained once (i.e. “universal representation”)
-experiment with KL-reg and VQ-req

semantic compression - generative DM learns semantic/conceptual data composition

vector-quantized VAE (VQ-VAE) - map each latent zez_e onto a learned codebook vector C={e1,e2,…,eK}\mathcal{C}= \{ e_1, e_2, \dots, e_K\}

-image→encodez∈Rx×x×d→quantize[e1,…,ed]→decodeimage\text{image} \xrightarrow{\text{encode}} z \in \mathbb{R}^{x \times x \times d} \xrightarrow{\text{quantize}} [e_1, \dots , e_d] \xrightarrow{\text{decode}} \text{image}
-sample from a discrete space for generation instead of a continuous latent
-to generate coherent images, you need to learn priors over codebook vectors that should be sampled together