Text encoders: How a prompt becomes conditioning

Introduction

A generation model never sees your words. It sees numbers produced by a separate network that read them, and most of what feels arbitrary about prompting is a property of that reader rather than of the generator attached to it.

Knowing which reader an architecture uses explains more about how to prompt it than any list of tips.

The prompt is cut up first

Before anything reads your prompt it is cut into tokens. Tokens are not words. A common word is usually one token, while a rare, compound or misspelled one becomes several pieces that the encoder then has to make sense of together.

Two things follow. A model handles a familiar term better than an invented one, because the familiar term arrives whole while the invented one arrives as pieces the encoder has to reassemble. And every encoder has a token limit: some pipelines work around it by encoding the prompt in chunks, and where they do not, everything past the limit is dropped, which is why a prompt can stop responding to more words.

What the encoder produces

The text encoder turns that token sequence into an embedding: not one vector for the prompt, but one per token, each shaped by the others around it.

Which encoder an architecture was trained with decides how your prompt is read:

• CLIP was trained on images and their captions, so it is strong at matching a short descriptive phrase to a visual style. It is weak at relations between things, at counting, and at negation, because a caption rarely needed any of them.
• T5 is a general language model. It holds long grammatical sentences together and keeps the relation between clauses, which is why architectures built on it respond to prose where CLIP-based ones respond to tags.

Some architectures use both and feed each a version of the prompt, taking the style matching from one and the sentence structure from the other.

How the embedding reaches the image

During denoising, every region of the latent looks at those token embeddings and weighs them, at every step. That is what cross-attention does, and it is why a single word can change the whole frame rather than one corner of it: nothing in the mechanism assigns a token to a place.

It is also where a negative prompt fits. Guidance works by running the step twice, once with your conditioning and once without, and a negative prompt is that second pass given real content instead of nothing. Where guidance has been distilled into the weights there is no second pass to fill, which is why a negative prompt can be accepted and then do nothing you can see.

What this explains

• Why clipSkip exists on some models and not others. It takes the embedding from a layer before the last one, so it only means something on an architecture whose reader is CLIP.
• Why embeddings are tied to an architecture. A trained embedding is a vector in one encoder's space. Another encoder's space has no matching position, so the file is not portable.
• Why prompt weighting syntax is not universal. Weighting edits the token embeddings before they reach the generator, so it depends on which encoder produced them.
• Why translating a prompt between architectures rarely works. The words that carried the result were chosen for one reader.

Where this stops being true

Some recent models have no separate text encoder at all. The prompt goes to a multimodal model that reads text and images in the same pass, which is why prompting those feels like giving instructions rather than writing a description, and why clipSkip and trained embeddings have nothing to attach to.