Video Representation

A video representation is the form in which a system stores evidence from frames over time: raw clips, optical-flow fields, tubelet tokens, per-frame embeddings, object tracks, or pooled clip vectors. It determines what information is easy to use later. A representation built from frame means may support retrieval, while a track-level representation is better for person tracking and track aggregation.

From frames to a clip vector

For frame embeddings $e_1,\dots ,e_T$, a simple clip representation can concatenate appearance and motion summaries:

$$r=\left[\frac{1}{T}\sum _{t=1}^T{e}_t,e_T-e_1\right].$$

This is not a universal best representation; it shows the contract. The first term carries average visual content, and the second carries coarse temporal change. Video transformers keep many tokens instead of reducing early, while self-supervised video representation learning trains the encoder that produces the embeddings.

flowchart LR
  Frames[Video frames] --> Encoder[Per-frame encoder]
  Encoder --> Emb[Frame embeddings e1 to eT]
  Emb --> Mean[Mean: average appearance]
  Emb --> Delta[Endpoint delta: temporal change]
  Mean --> Rep[Clip representation r]
  Delta --> Rep

Worked representation example

For four frame embeddings

frame	embedding
1	$(1.0,0.0)$
2	$(0.8,0.2)$
3	$(0.2,0.9)$
4	$(-0.1,1.0)$

the mean embedding is

$$\frac{(1.0,0.0)+(0.8,0.2)+(0.2,0.9)+(-0.1,1.0)}{4}=(0.475,0.525),$$

and the endpoint delta is

$$e_4-e_1=(-0.1,1.0)-(1.0,0.0)=(-1.1,1.0).$$

The concatenated representation is therefore $(0.475,0.525,-1.1,1.0)$. The pooled part says the clip contains both embedding directions; the delta says it moved from the first direction toward the second.

Caveats

Pooling too early erases order and boundaries, which hurts temporal localization. Dense tokens preserve detail but increase memory. Track and object representations can be robust to camera motion, but they depend on detector and association quality.

References

• Carreira and Zisserman, 2017, Quo Vadis, Action Recognition?
• Tong et al., 2022, VideoMAE

Nav

Section — Video Understanding

← Spatial and Temporal Modelling Optical Flow →