World Models

World models learn a latent state and transition model that supports prediction, planning, or control. In video understanding, this means representing how scenes evolve, not only recognizing a label. The topic overlaps with video representation, self-supervised video representation learning, and V-JEPA 2.

Encoder, dynamics, decoder

A compact world model has an encoder, dynamics model, and decoder or cost head:

$$z_t=E(x_t),{\hat{z}}_{t+1}=F(z_t,a_t),{\hat{x}}_{t+1}=D({\hat{z}}_{t+1}).$$

Planning can happen in latent space by rolling out candidate actions and minimizing a cost. World models and JEPA focuses on the variant where prediction happens in representation space rather than pixel space.

flowchart LR
  Obs[Observation x_t] --> Encoder[Encoder E]
  Encoder --> Latent[Latent state z_t]
  Action[Action a_t] --> Dynamics[Dynamics F]
  Latent --> Dynamics
  Dynamics --> Next[Predicted next latent]
  Next --> Decoder[Decoder or cost head]

Worked rollout

With a simple latent transition

$$A=\left[\begin{array}{cc}1.0& 0.2\\ 0.0& 0.9\end{array}\right],z_0=(1.0,0.5),$$

the rollout is:

step	latent state
0	$(1.000,0.500)$
1	$(1.100,0.450)$
2	$(1.190,0.405)$
3	$(1.271,0.364)$

The first coordinate grows because it receives $0.2$ times the second coordinate at each step; the second coordinate decays by a factor of $0.9$. Real learned world models replace this hand-coded matrix with neural dynamics trained from video or interaction.

Caveats

A useful latent rollout is not automatically a faithful simulator. Evaluation must test counterfactuals, long-horizon error, uncertainty, and planning performance, not only next-step prediction. Video-only data may not identify action effects without interaction data.

References

• Ha and Schmidhuber, 2018, World Models
• Assran et al., 2025, V-JEPA 2

Nav

Section — Video Understanding

← V-JEPA 2 versus Vision-Language Models World Models and JEPA →