Video Transformers

Video transformers tokenize a clip into frame patches or tubelets and let attention route information across space and time. Compared with 3D convolutional networks, they are less tied to local kernels; compared with two-stream models, they learn motion interactions inside the same token system that carries appearance. Their core operation is the same scaled dot-product attention used in language models.

Attention over video tokens

For $N$ video tokens with dimension $d_k$,

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V.$$

Full space-time attention forms an $N\times N$ score matrix, where $N=(T/\tau )(H/P)(W/P)$ for tubelet length $\tau$ and patch size $P$. Factorized variants attend spatially and temporally in separate steps to reduce cost and stabilize learning.

The important implementation detail is that video tokens have geometry. A token is not just sequence element $i$; it corresponds to an original address such as $(t,h,w)$ or to a tubelet covering a small block of time and space. Positional encodings must agree with that address. If a system physically keeps only person- or hand-region tokens for gesture recognition, the kept tokens should retain their original positions rather than being renumbered as a dense sequence.

From clip to tokens

A video transformer turns pixels into a token sequence before any attention happens:

1. Split. Divide the clip into non-overlapping patches per frame, or into tubelets that each span a small block of time and space.
2. Project. Apply one shared linear map to flatten each patch or tubelet into a token embedding of dimension $d_k$.
3. Position. Add positional encodings that carry the original $(t,h,w)$ address, so attention can tell apart tokens that share appearance but differ in space or time.
4. Attend. Run stacked attention blocks over the tokens, either full space-time or factorized into separate spatial and temporal steps.
5. Read out. Pool the tokens (or a class token) into a clip vector for the task head.

flowchart LR
  Clip[Input clip] --> Split[Split into tubelets or patches]
  Split --> Proj[Linear projection to tokens]
  Proj --> Pos[Add space-time positions]
  Pos --> Attn[Space-time attention blocks]
  Attn --> Head[Pool and task head]

Worked token budget

The useful implementation question is usually not a random attention weight; it is how many tokens and pairwise scores the design creates. For $8$ frames of size $16\times 16$, tubelet length $\tau =2$, and patch size $P=8$:

$$N=(8/2)(16/8)(16/8)=16,N^2=256.$$

Changing only the patch size changes the cost quickly:

patch size	token count $N$	full-attention score count $N^2$	interpretation
$8\times 8$	16	256	cheap, coarse hand/object detail
$4\times 4$	64	4096	four times more tokens, sixteen times more scores
$2\times 2$	256	65536	fine detail, expensive full attention

This quadratic score growth is why real clips quickly make token count the dominant memory and latency constraint, especially for real-time video understanding.

Patch Size And RoI Tradeoffs

Patch size is a budget knob. Smaller patches preserve small objects such as hands, fingers, tools, and signs, but increase token count and attention cost. Larger patches reduce compute but can erase the motion detail needed for fine-grained gestures.

design choice	effect
Smaller spatial patch	More hand detail, more tokens, higher attention cost.
Larger spatial patch	Cheaper inference, coarser hand and object geometry.
Tubelets	Fewer temporal tokens, but each token already mixes adjacent frames.
Person RoI crop before the backbone	Keeps the actor and nearby context while removing other people and background.
Hand RoI crop before the backbone	More pixels allocated to the hand, less body and object context.
RoI token keep inside the backbone	Lower token budget while preserving the original frame, if positions are handled correctly.

This is why comparing full-frame, static-crop, tracking-crop, and token-keep variants can reveal more about input-domain alignment than about the model family alone.

History and adoption

Video transformers followed directly from the image vision transformer, which showed that patch tokens plus attention could match convolutional networks when pretrained at scale. Extending this to time raised an immediate cost problem, because full space-time attention is quadratic in the token count. TimeSformer (2021) addressed it by factorizing attention into separate temporal and spatial steps, and ViViT (2021) explored tubelet embeddings and factorized encoders for the same reason. The line then merged with video-language models, where a video-transformer encoder feeds a language model for captioning, retrieval, and question answering. Throughout, the recurring design tension is the token budget from the worked token budget above: richer spatial or temporal detail means more tokens and quadratically more attention cost.

Caveats

Attention can model long-range dependencies, but it does not automatically solve sampling, supervision, or temporal boundary errors. Long videos require sparse, factorized, streaming, or hierarchical designs. Fine-tuning image-pretrained transformers can help, but video-specific motion cues still need adequate temporal coverage. RoI cropping and token keeping can improve signal-to-noise, but they can also remove context or corrupt geometry if token positions no longer match the physical video grid.

References

• Bertasius et al., 2021, Is Space-Time Attention All You Need for Video Understanding?
• Arnab et al., 2021, ViViT: A Video Vision Transformer
• Vaswani et al., 2017, Attention Is All You Need

Nav

Section — Video Understanding

← Two-Stream Models Temporal Action Recognition →