Person Tracking and Track Aggregation

Person tracking links detections of the same person across frames. Track aggregation then turns frame-level detections, poses, logits, or embeddings into person-level evidence. This is essential when gesture recognition or temporal action recognition must answer “which person did it?” rather than only “did it happen?”

Tracking by detection

Tracking-by-detection builds an association score between existing tracks $i$ and detections $j$, often using box IoU, motion prediction, and appearance distance:

$$\mathrm{score}(i,j)=\lambda \mathrm{IoU}(b_i,b_j)-(1-\lambda )d(e_i,e_j).$$

Assignments are solved per frame, then each track aggregates features over time with pooling, smoothing, or a temporal model. Optical flow can help predict short-term motion, but identity association is the central problem.

Worked example

This snippet computes IoU between existing tracks and new detections, then uses Hungarian assignment to choose the best track-detection matches.

import numpy as np
from scipy.optimize import linear_sum_assignment
 
tracks = np.array([[0,0,2,2],[5,5,7,7]], float)
dets = np.array([[0.2,0.1,2.2,2.1],[5.4,5.2,7.4,7.2]], float)
def iou(a, b):
    ix1, iy1 = max(a[0],b[0]), max(a[1],b[1])
    ix2, iy2 = min(a[2],b[2]), min(a[3],b[3])
    inter = max(0, ix2-ix1) * max(0, iy2-iy1)
    return inter / ((a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - inter)
M = np.array([[iou(t, d) for d in dets] for t in tracks])
r, c = linear_sum_assignment(-M)
print("iou_matrix", np.round(M, 3).tolist())
print("matches", list(zip(r.tolist(), c.tolist())), "mean_iou", round(float(M[r,c].mean()), 3))

Observed output:

iou_matrix [[0.747, 0.0], [0.0, 0.562]]
matches [(0, 0), (1, 1)] mean_iou 0.655

The IoU matrix is nearly diagonal: track 0 overlaps detection 0 at 0.747 and track 1 overlaps detection 1 at 0.562, while the cross-pairs are 0.0. The Hungarian assignment therefore links each track to its compatible detection, yielding a mean matched IoU of 0.655.

This code earns its place because it demonstrates the programming API used for the assignment step. linear_sum_assignment minimizes a cost matrix, so the example passes -M to maximize IoU. In a real tracker, that matrix would usually combine IoU, motion prediction, appearance embeddings, and gating rules before assignment.

Caveats

Occlusion and crossing paths cause identity switches. Aggregating over a broken track can dilute the event or assign it to the wrong person. Real-time deployments must bound track memory and handle detector dropouts explicitly.

References

• Bewley et al., 2016, Simple Online and Realtime Tracking
• Wojke et al., 2017, Simple Online and Realtime Tracking with a Deep Association Metric

Nav

Section — Video Understanding

← Trigger Point Prediction Real-Time Video Understanding →