Part 2: Image Generation Guesses the "Noise" or the "Token" — TXT2IMG Training Data and Captions

Image Generation Is a “Text-to-Image” Model

Last time’s LLM was a model where “text goes in and text comes out” — that is, TXT2TXT. This time’s protagonist, image generation, is a model where text (a prompt) goes in and an image comes out. Since the input is text and the output is an image, it’s called “TXT2IMG (text-to-image).” Stable Diffusion, DALL·E, and Midjourney are representative examples: type “a red sports car in front of a building at dusk” and it draws an image of that scene.

Even within the same “generative AI,” once the output changes from letters to images, how you train it changes completely. In this part we look first at “how to prepare the pairs” of training data, then at “what you make the model learn from those pairs.”

The Difference from LLMs: Pairs That Are Hard to Align

Supervised learning needs input-answer pairs. This holds for both LLMs and image generation: an LLM, too, learns from an input (the preceding token sequence) paired with an answer (the next token).

What differs is how easily that answer can be gathered. The LLM could extract the answer from within a single stream of text — tokenize it, shift by one, and input and answer come from the same material, with no need to prepare the answer separately.

In image generation (TXT2IMG), the text that serves as input (the prompt/caption) and the image that serves as the answer are simply in different data formats. You can’t shift text by one to produce an image, so you have to prepare, in advance, pairs of “an image” and “text describing that image.”

image: cat.jpg
caption: a cat sitting on a table

And most images in the world don’t come with clean descriptions. Assembling these different-format pairs — in large numbers, with contents that actually match — was a major hurdle in early image generation. Below, we look separately at “how to prepare the pairs” and “what you learn from those pairs.”

Where Do Captions Come From?

There are roughly three ways to attach a caption to an image.

1. Use data that already has captions — caption datasets, or product-image + product-description, where the correspondence exists from the start.
2. Infer from surrounding text. For web images, the alt attribute and nearby text are hints.

<img src="cat.jpg" alt="a sleeping cat">
<p>This is my cat sleeping near the window.</p>

From this you extract a caption like a cat sleeping near the window.

3. Auto-generate with an image-captioning model. Pass the image to a model like BLIP, Florence, Qwen-VL, or Gemini and have it write a description.

from transformers import pipeline

captioner = pipeline("image-to-text", model="Salesforce/blip-image-captioning-base")
caption = captioner("cat.jpg")[0]["generated_text"]
# → "a cat sitting on a table"

Recently the third option — having AI write the captions — has surged. The “AI makes the training data” trend from Part 1 shows up clearly in images too.

Sifting Out the Dirty Pairs — Quality Filtering

Image-caption pairs scraped from the web aren’t usable as-is. Broken images, meaningless alt text, low resolution, duplicates, watermarks everywhere — it’s a pile of noise. So you apply at least basic filters.

from PIL import Image

def is_valid_caption(caption: str) -> bool:
    if len(caption) < 5 or len(caption) > 300:
        return False
    if caption.lower() in ["image", "photo", "img", "untitled"]:
        return False
    return True

def is_valid_image(path: str) -> bool:
    try:
        w, h = Image.open(path).size
        return w >= 256 and h >= 256
    except Exception:
        return False

In earnest, this stacks up further: deduplication, NSFW removal, excluding text-heavy images via OCR, filtering by aesthetic-quality score, copyright/trademark policy checks.

And crucially, there’s the step of scoring, with AI, whether the image and caption actually match. This is where CLIP comes in.

# Measure image-text agreement with CLIP (conceptual)
score = clip_similarity(image, caption)

if score >= 0.30:
    keep(image, caption)     # high agreement → accept
else:
    discard(image, caption)  # low agreement → reject

Accept at 0.93, reject at 0.22 — in other words, AI is selecting the training data. Only after passing this multi-stage pipeline — gather, clean, score with AI — do usable pairs remain.

There’s More Than One Way — Three Lineages of Image Generation

The mechanism for generating images is not just one thing. Historically, the main lineages are these three:

• GAN (generative adversarial network): train by pitting a “generator” that makes images against a “discriminator” that tells real from fake. The star of roughly 2015–2020, famous for StyleGAN and others. Its weaknesses were unstable training and a poor fit for flexible text-driven generation.
• Autoregressive (token) type: convert an image into a sequence of “image tokens” and predict the next token one at a time, like an LLM. The original DALL·E and Parti are representative (we cover this in detail later).
• Diffusion models (+ flow matching): learn the procedure to restore an image from noise. Stable Diffusion, Imagen, DALL·E 3, and others fall here, and this is the current mainstream of text-to-image. The flow matching used by SD3 and Flux is an evolution of the same lineage.

These three lineages are not a mutually exclusive taxonomy. There are approaches that don’t fit neatly, like masked (parallel token-filling) models such as Muse, and there are models that combine autoregression and diffusion in stages. In practice, real products are often blends of several approaches.

Since diffusion is by far the most widely used today, we look at it in detail first. Then we’ll touch on the autoregressive type, which is close to an LLM.

What the Mainstream Diffusion Model Learns as the “Answer” — Denoising

From here we look at the mainstream diffusion model. What’s interesting is that diffusion-type TXT2IMG does not make the image itself the answer.

The basis of diffusion models (the Stable Diffusion family) is this:

1. Mix random noise into the original image.
2. Give the model how much you mixed in (the timestep) and the caption.
3. Have the model guess the noise you just mixed in.

Simplified quite a bit, the picture is:

import torch

image_latent = torch.randn(1, 4, 64, 64)  # a compressed representation of the image (latent)
noise = torch.randn_like(image_latent)     # the noise about to be mixed in

t = torch.randint(0, 1000, (1,))           # noise strength (the timestep)
noisy_latent = image_latent + 0.1 * noise  # the noised image

What you give the model is “noised image, timestep, text embedding”; the answer is “the noise actually mixed in.”

predicted_noise = model(
    noisy_latent,
    timestep=t,
    text_embedding=text_embedding,
)

loss = ((predicted_noise - noise) ** 2).mean()

So the training data comes together like this:

input:
- the noised image
- the noise strength (timestep)
- the text description

answer:
- the mixed-in noise

Being able to correctly guess “the added noise” means, conversely, being able to remove the noise and restore the original image. So after training, if you start from pure noise and peel the noise away step by step, you can generate an image that follows the text. Learning the denoising procedure rather than drawing the image directly — that’s the heart of diffusion models.

A Minimal Training Loop

Compress the image to a latent with a VAE, make embeddings with a text encoder, predict noise with a U-Net. Roughly:

import torch
import torch.nn.functional as F

def train_step(unet, vae, text_encoder, batch, optimizer):
    images, captions = batch["image"], batch["caption"]

    with torch.no_grad():
        latents = vae.encode(images).latent_dist.sample()
        text_emb = text_encoder(captions)

    noise = torch.randn_like(latents)
    timesteps = torch.randint(0, 1000, (latents.size(0),), device=latents.device)
    noisy_latents = add_noise(latents, noise, timesteps)

    pred_noise = unet(
        noisy_latents,
        timesteps,
        encoder_hidden_states=text_emb,
    ).sample

    loss = F.mse_loss(pred_noise, noise)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss.item()

As a diagram, the whole thing is:

caption ─▶ Text Encoder ─▶ text embedding ┐
                                           ├▶ U-Net ─▶ predicted noise
image ─▶ VAE Encoder ─▶ latent ─▶ +noise ──┘                │
                                                             ▼
                                        predicted noise vs actual noise → loss

Another Stream — Turn the Image into “Tokens” and Make It Like an LLM

Diffusion may be mainstream, but a contrasting approach is gaining presence: the autoregressive (token) type mentioned at the top.

The idea is simple. Using a mechanism like VQ-VAE or VQGAN, you convert an image into a sequence of discrete “image tokens.” It’s the same idea as tokenizing audio with Encodec.

image_tokens = vqgan.encode(image)
# e.g. [913, 22, 480, 17, ...]

After that, it’s exactly like an LLM. Conditioned on text, you guess the image tokens one at a time by “next-token prediction.”

text_ids = tokenizer(caption)
target = vqgan.encode(image)

logits = model(text_ids)          # predict image tokens from text
loss = F.cross_entropy(
    logits.reshape(-1, vocab_size),
    target.reshape(-1),
)

The answer is not “the added noise” but “the next image token.” In other words, in this approach image generation takes on literally the same shape as an LLM. The original DALL·E and Parti are on this line, and it’s drawing renewed attention lately in multimodal models that handle both text and images in a single model.

To sum up: there are broadly two ways to construct the “answer” for image generation — the diffusion type that guesses the noise and the autoregressive type that guesses the next token.

Line It Up with the LLM and the Structure Shows

Side by side with Part 1’s LLM, the difference in how training data is built stands out.

The LLM likewise learns from input-answer pairs, but it could extract both automatically from a single stream of text. TXT2IMG, regardless of approach, needs pairs of two different forms — “an image” and “its description” — which you gather, clean, and score with AI. That’s the biggest difference between them.

The commonality is just as clear: neither has humans labeling every item; both mechanically construct the learning problem from existing data. The LLM extracts the answer by “shifting”; the image does so with diffusion by “mixing in noise,” and with the autoregressive type by “tokenizing and guessing the next” — again, mechanically. The autoregressive type in particular shows image generation stepping onto the same ground as the LLM, and this “each field converging to the same shape” trend becomes clearly visible in next part’s audio too.

Summary

• Image generation isn’t made just one way. There are lineages — GAN, autoregressive (token) type, and diffusion — and the current mainstream for text-to-image is diffusion.
• Whatever the approach, the training data is “image + caption” pairs. Captions come from existing data, surrounding text, and AI-based auto-generation; the pairs are dirty, so you filter by resolution, duplication, and policy, then use CLIP to score “image-text agreement” and select.
• The answer isn’t the image guessed directly in pixels but, for the diffusion type, “the added noise,” and for the autoregressive type, “the next image token.” The latter shows image generation taking on the same shape as an LLM.

Next time we take on “audio,” as familiar as text. The story is that TTS and Speech2Speech training data have recently been converging toward the LLM’s format.