Latent space: Where generation actually happens

Introduction

A generation model does not paint pixels. It works on a compressed representation of an image, shapes that, and only converts the result into a picture at the very end. That representation is a latent, and the space it lives in is where every technique on this site actually operates.

This is the single fact that makes the rest of the pipeline make sense. Once you know generation happens in a smaller space, it becomes obvious why dimensions come in multiples, why a decoder is a separate swappable part, and why starting from an image is the same operation as starting from noise.

Why not pixels

A 1024 x 1024 image is a little over a million pixels, three numbers each. Denoising that directly, dozens of times per generation, is possible and was how the first diffusion models worked. It is also slow enough to be impractical at the resolutions people want.

Latent diffusion changes the arithmetic. A VAE compresses that image into roughly 128 x 128 values, each carrying far more meaning than a pixel does. The model then works on about one sixtieth of the data, and the decoder restores the detail at the end.

What the space is

A latent is a compressed picture, not a map of meaning. It keeps the layout of the image: the grid of latent cells lines up with the grid of pixels, so the top left of one is the top left of the other, and a cell carries what that patch of the image looks like. What the encoder discarded is detail a viewer would not miss, not the arrangement.

Meaning lives somewhere else. Your prompt becomes an embedding in a space that is organized by meaning, and cross-attention is what joins the two: at every denoising step the model reads that embedding and decides what each region of the latent should turn into.

That split is the useful part. The latent carries where things are, the conditioning carries what they should be, and almost every technique on this site is one or the other.

Everything is a position in that space

Once you see generation as movement through this space, the techniques stop looking like separate features.

Text to image starts from a random position, which is what a seed picks, and moves toward the region your prompt describes.

Image to image starts from your input image instead of from noise. strength decides how much noise is added to it first, and so how much of the original survives being rebuilt.

Inpainting adds noise everywhere but re-imposes the unmasked part at every step, so only the region the mask marks is actually rebuilt.

Image to video works on a latent volume, the same idea with time as an extra dimension, and a supplied frame fixes one slice of it.

None of these are different pipelines. They are the same movement with different constraints on where it starts and what is allowed to change.

What this explains

• Why a decoder is swappable. The generator produces a latent, and turning that into pixels is a separate job. Changing the VAE changes the final interpretation without touching what was generated.
• Why small details are the first thing to suffer. Detail that survived compression can be restored, detail that did not was never in the latent to begin with.
• Why resolution has a sweet spot. A model learned what a plausible latent looks like at the size it was trained on. Ask for a much larger one and it fills the extra room with more of what it knows, which is how you get a second head or a repeated subject.
• Why conditioning works at all. A reference image, a depth map and a prompt are different ways of saying the same thing: constrain the region of the space the result is allowed to land in.

Where this stops being true

This is the account of models that shape a continuous latent, which covers diffusion and flow matching and so most of what you will meet.

An autoregressive image model works on discrete tokens rather than a continuous latent: it picks each piece from a vocabulary, one after another, instead of shaping the whole thing at once. There is still a compressed representation, but the ideas on this page about moving through a space do not describe how it gets there.

The same caution applies to a model served behind a single endpoint that takes a prompt and returns a picture. The compression is still happening somewhere, and nothing about it is yours to address.