Part 4: What's the Difference Between Video Generation and World Models? — "Plausible Footage" vs. "State Transitions"

Video Generation and World Models: Alike but Not the Same

This time we contrast two kinds of models. One is video generation: a model where text goes in and a short video comes out, so it can be called “TXT2Video.” Sora, Veo, and Runway are representative examples — it helps to think of it as last part’s image generation (TXT2IMG) with “time” added.

The other is the world model. Its goal is not “make plausible footage” but to predict how the world will change next when you take an action in some situation. It’s used as the “simulator inside the head” of robots, games, and self-driving systems.

Because both deal with “moving footage,” they’re easily confused, but they differ completely in what they learn for and in their training data. Let’s start with the more approachable one: video generation.

Video Generation Is “Image Generation with More Frames”

Start with video generation (Sora, Veo, Runway, and so on). The training data is pairs of video and its caption. Captions are made from subtitles, surrounding text, video descriptions, ASR (speech recognition), and image/video captioning models — the same idea as image generation in Part 2.

The generation approach also carries straight over from image generation. As we saw in Part 2, generation comes in a diffusion type and an autoregressive (token) type, and both exist for video too (the autoregressive type includes VideoPoet; the diffusion type includes Sora and Veo). But, as in Part 2, the approaches aren’t mutually exclusive — Sora, for example, is a Diffusion Transformer combining diffusion with a Transformer, and in practice combinations are the norm. Here we take up the mainstream diffusion type.

Its training mechanism is an extension of image generation. You turn the video into a sequence of frames, compress them to latents with a VAE, mix in noise, and have the model guess “the added noise.”

video_latents = vae.encode(video_frames)

noise = torch.randn_like(video_latents)
t = torch.randint(0, 1000, (video_latents.size(0),))

noisy_video = scheduler.add_noise(video_latents, noise, t)

pred_noise = video_model(noisy_video, t, text_embedding)
loss = mse(pred_noise, noise)

Where image generation was “denoising a single image,” video generation is “denoising a latent that carries a time axis.” In other words:

image generation: 1 frame
video generation:  100 frames (latents connected over time)

Of course, having time means it also learns temporal consistency like “frame 5 → frame 6” — that “the dog doesn’t teleport,” for example. Even so, its essence is making plausible footage.

What Does a World Model Learn?

Here’s the main topic. A world model’s goal is not to make footage that looks right. It learns how the world changes when you act — that is, state transitions.

World models, too, come in several flavors: predicting the next frame in pixel space, predicting in a latent space (like the RSSM in the Dreamer family), predicting only features without generating footage (the JEPA family), or tokenizing states and actions and predicting them autoregressively (the Genie family). Here we take up, as representative, the formulation common to all of these: state S_t + action A_t → next state S_{t+1}.

With that way of learning, video alone isn’t enough as training data. For a robot, for example, you record the following synchronized at each timestep:

Camera / Depth / joint angles / motor commands / force sensors / IMU / gripper state

And at each time you associate the action taken then.

time t:   image, hand position, hand angle, applied force
action:   move 3 cm to the right
time t+1: image, hand position, ... (the result of the action)

So the unit of training data becomes:

input:  state S_t + action A_t
answer: next state S_{t+1}

Training is remarkably plain.

pred_next_state = world_model(state_t, action_t)
loss = mse(pred_next_state, state_t1)

When images or video are treated as the state itself, you drop them into a state representation with an encoder and then learn the transition.

state_t  = encoder(frame_t,  depth_t,  sensor_t)
state_t1 = encoder(frame_t1, depth_t1, sensor_t1)

pred = world_model(state_t, action_t)
loss = mse(pred, state_t1)

In equation form, what the world model approximates is this function:

f(S, A) = S'

“Put in the current state and action, get the next state back.” It’s learning this function with a giant Transformer.

The Difference Between Video Generation and World Models

Side by side, the difference in aim is clear.

video generation:
  a dog runs ─▶ plausible video

world model:
  a dog runs + action ─▶ three seconds later it has moved to the right (future prediction)

Video generation only needs to be “natural as a picture.” A world model must be “natural as the result of an operation.” So the biggest difference is that its training data always associates not just video but “what was done then (the Action).”

• Robot: camera footage, joint angles, motor commands, force sensors, IMU saved time-synchronized
• Game: screen synchronized with gamepad input
• Self-driving: camera, LiDAR, GPS, vehicle speed, steering angle, throttle, brake synchronized

Only with this data can you learn not “generate the next footage” but the causal relationship of “how does the world change if I do this operation.” This is the decisive point separating video-generation models from world models.

Action Tokens — Making Actions into Tokens Too

Recently the trend of treating the action itself as a token has grown stronger.

Observation ─▶ Action ─▶ Observation

By making actions discrete tokens, learning state transitions fits neatly into the “LLM that predicts a token sequence” framework we’ve seen in earlier parts.

Joystick ─▶ [31, 52, 14]
Move Arm ─▶ Token
Jump     ─▶ Token

“Observation token + action token → next observation token.” Part 1’s LLM, Part 3’s audio tokens — the same structure surfaces here too.

Real Examples — NVIDIA Cosmos and Google Genie

Two concrete models to note.

NVIDIA Cosmos is positioned as a World Foundation Model for physical AI. It converts camera footage into video tokens and moves toward generating “future footage” conditioned on text and actions. Its training data is combinations of video, text, and actions.

Google Genie learns controllable environments from game screens alone. What’s distinctive is that even from video with no explicit action labels, it automatically discovers pseudo-actions (latent actions) from “changes in the screen.”

Genie:
  screen ─▶ (auto-discovered) controller input ─▶ next screen

The training data is “observation, action, next observation.” Even the action labels get inferred by AI from video — you can see that the “AI makes the training data” trend running since Part 1 reaches its peak in world models.

A Bird’s-Eye View of Generative AI

Lining up the five modalities so far by the structure of their training data, it organizes cleanly.

From LLM to video generation, these were models that make “plausible outputs.” Only the world model learns something one level different — how the world moves.

Summary

• Video generation is the temporal extension of image generation: from “video + caption,” guess “the added noise.” The goal is “plausible footage.”
• A world model learns S_t + A_t → S_{t+1}. Its training data must associate not just footage but “the action taken then,” time-synchronized.
• Cosmos uses video, text, and actions; Genie infers pseudo-actions from unlabeled video — world-model training data is the domain that depends most deeply on AI.

But video generation and world models are not opposites. In the finale, we’ll draw how to combine the two to make physically correct video, and the view beyond it — building training data wholesale with AI and simulation.