Part 5: Toward Physically Correct Video — World-Model Integration and the Era of AI-Made Training Data

What Makes a Video “More Real”?

At a dining table, an actor is eating spaghetti. The camera pushes in on his hands. The fork twirls up the pasta and lifts it toward his mouth — and then, in the next instant, the fork and the heaping tangle of spaghetti pass straight through his lips and cheeks and are drawn into the back of his throat. The texture of his skin, the rising steam, the glint of the light are all unmistakably real; only what is happening is decisively wrong.

Today’s video generation has moments like this: indistinguishable from live action to look at, yet broken as physics. So, once more — what exactly does a “more real” video mean?

Break the question down and “realism” has two layers: natural as a picture, and correct as the behavior of the world. Today’s video generation is good only at the former; the latter — physical consistency — needs a different mechanism.

That mechanism is the combination of the two models from last time. Video generation (text → video) is good at making “plausible footage,” but it doesn’t guarantee physical consistency. The world model (state + action → next state), on the other hand, can predict how the world moves, but is not itself a mechanism for producing clean footage.

The theme of this finale is combining the two to make “physically correct video.” And finally, we’ll look at where the “AI makes the training data” trend seen throughout the series ultimately leads — synthetic data from simulation.

The Cause: The “World State” Is Missing

These breakdowns are no accident. Video generation guesses P(video | text) — “a plausible pixel sequence from text” — directly, with no state in between that holds “how the world is right now.” An object vanishing in the next frame (broken object permanence), speed jumping discontinuously, the result of opening a drawer not staying consistent — they all trace back to this.

To fix it, you need to embed state, action, geometry/physics, and future prediction into the video-generation model.

The Basic Idea — Put a “World State” in the Middle

Plain video generation is:

text + noisy video latent ─▶ video diffusion model ─▶ video

The world-model-integrated type inserts a world state in the middle.

state S_t + action A_t + text + physics/geometry representation
   ─▶ world model ─▶ future states S_{t+1}, S_{t+2}, ...
   ─▶ video decoder / diffusion renderer ─▶ physically consistent video

The point is to not aim directly at video pixels from text, but to go in the order:

Text ─▶ world state ─▶ future state ─▶ video

You learn Part 4’s P(S_{t+1} | S_t, A_t) (state transition) and P(video | S) (state to footage) separately. Use the video-generation model as “a model that draws pictures,” and the world model as “a model that manages real transitions.” This division of labor is the core.

There is, of course, more than one road toward physically consistent video. You can just keep scaling data and compute and let a standalone diffusion model learn it; embed a physics engine or simulator into the generation loop; correct the output with reinforcement-learning rewards; or route through an explicit 3D representation or neural rendering. This article takes up, among these, the representative construction that connects naturally to Part 4’s world model: “state transition + state-conditioned generation.”

The Skeleton of Training — Add Two Losses

The required training data is more than plain video. For a robot you hold camera / depth / joint angles / motor commands / force / gripper state / future frames; for self-driving, camera / lidar / depth / GPS / IMU / steering / throttle / brake / future frames, all time-synchronized (see Part 4).

First compress the video into latents, then build a world state from them.

state = state_encoder(
    video_latents=latents,
    depth=batch["depth"],
    camera_pose=batch["camera_pose"],
)
# state.shape = [B, T, D]

Next, predict future states conditioned on the action (the world-model-side loss).

pred_next_state   = world_model(state[:, :-1], actions=batch["action"][:, :-1])
target_next_state = state[:, 1:]

loss_state = F.mse_loss(pred_next_state, target_next_state)

Then restore the video conditioned on the predicted future state (the video-generation-side loss). With a diffusion model, it’s the usual noise prediction.

target_latents = latents[:, 1:]
noise = torch.randn_like(target_latents)
t = torch.randint(0, 1000, (target_latents.size(0),))

noisy = scheduler.add_noise(target_latents, noise, t)

pred_noise = video_diffusion(noisy, t, condition=pred_next_state)
loss_video = F.mse_loss(pred_noise, noise)

The minimal training loop just adds these two.

for batch in dataloader:
    latents = vae.encode(batch["frames"]).latent_dist.sample()
    state = state_encoder(latents, batch["depth"], batch["camera_pose"])

    pred_state = world_model(state[:, :-1], batch["action"][:, :-1])

    target_latents = latents[:, 1:]
    noise = torch.randn_like(target_latents)
    t = torch.randint(0, 1000, (target_latents.size(0),), device=latents.device)
    noisy = scheduler.add_noise(target_latents, noise, t)
    pred_noise = video_diffusion(noisy, t, condition=pred_state)

    loss = F.mse_loss(pred_noise, noise) + 0.5 * F.mse_loss(pred_state, state[:, 1:])

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Rather than guessing “text to video” in one shot, inserting the state transition makes the video not only “natural as a picture” but “natural as a state transition.”

Put Physical Constraints into the Loss

To push one more step toward “not contradicting reality,” add the physical constraints themselves to the loss. For example —

Continuity of velocity/acceleration (no sudden jumps):

def velocity_loss(pos):
    v = pos[:, 1:] - pos[:, :-1]
    a = v[:, 1:] - v[:, :-1]
    return (a ** 2).mean()

Objects don’t suddenly vanish (object permanence):

def object_permanence_loss(mask_t, mask_next):
    return torch.abs(mask_t - mask_next).mean()

Objects don’t pass through the ground (contact constraint):

def ground_penetration_loss(object_z, ground_z=0.0):
    return torch.relu(ground_z - object_z).mean()

You sum these with weights.

loss = (
    loss_diffusion
    + 0.5 * loss_state
    + 0.1 * velocity_loss(object_positions)
    + 0.1 * ground_penetration_loss(object_z)
    + 0.1 * object_permanence_loss(mask_t, mask_next)
)

This pushes the footage not just toward “looks natural” but toward “objects don’t vanish, speed doesn’t jump, it doesn’t contradict gravity, action results are consistent, the viewpoint doesn’t break.” The finished structure is:

Text ─▶ Scene / State Plan ─▶ World Model ─▶ Future State Rollout
     ─▶ Video Diffusion Renderer ─▶ Physically Consistent Video

Here’s Where “AI-Made Training Data” Pays Off

The pipeline above demands large amounts of depth / camera pose / object position / action / mask. Attaching these to video one by one by hand is unrealistic. So current research combines multiple AI models for preprocessing.

video (the raw data is only this)
  ├─ Depth Anything     ─▶ depth
  ├─ Grounding DINO     ─▶ object detection
  ├─ SAM2               ─▶ segmentation / tracking
  ├─ COLMAP / DUSt3R    ─▶ camera pose, 3D
  └─ VLM                ─▶ action labels (Open Drawer, Pick Cup, ...)

So even if the raw data is only “video,” AI can produce all of Depth / Pose / Object / Mask / Tracking / Camera / action. The “AI makes the training data” trend recurring since Part 1 becomes, in world models, the very core of the preprocessing pipeline. Research is also progressing on having AI estimate physical quantities like velocity, acceleration, contact, and gravity direction from video.

The Ultimate Form — Simulation Produces Complete Training Data

And this is where the biggest change is happening now: the approach of generating training data wholesale in simulation.

When you run a scene in an environment like NVIDIA Omniverse, Isaac Sim, MuJoCo, Unity, or Unreal, Object Pose / Camera Pose / Depth / Normal / Segmentation / Velocity / Force / Action are all known. You don’t even need to annotate real video with AI — you can save complete training data directly.

until now:  reality ─▶ AI (annotation) ─▶ training data
from now:   simulation ─▶ complete training data

World foundation models for Physical AI are increasingly designed on the premise of such simulation-derived synthetic data — an extension of this trend. Simulation leaps straight over the scarcity of real data and the cost of annotation.

Where This Leads — Physical AI

In the end, LLMs, images, audio, and video differ only in “how they represent the training data.” Yet the ingenuity each representation demands was completely different. And it’s along the line of that ingenuity that Physical AI lies.

The fact that a world model can predict “how the world changes when you act” means, conversely, that AI is gaining a foothold to act in the real world. Build complete training data in simulation, learn “how to move” there, and transfer it to a physical robot — the making of study material for generative AI becomes the very foundation of Physical AI, where robots take on a body and act on the world.

What that looks like in concrete terms — how to unify “seeing, understanding language, and physically acting” in a robot, how to stack VLA, diffusion, and world models — is the subject of Physical AI itself. A separate series digs into it in detail; let these be your entry point.

• Part 1: Where Physical AI Stands Now — An Extension of LLMs, or Something Else?
• Part 2: Give AI a “Cerebellum” and “Autonomic Nerves” — A Blueprint for Embodied Intelligence
• Part 3: Don’t Do Everything with One Giant Model — An Architecture of Hierarchy and Separated Responsibilities