Model Benchmarking

Computer-vision benchmarking compares models under the same data, preprocessing, metrics, and deployment constraints. It should connect task scores from detection and segmentation metrics to practical constraints such as latency, memory, calibration, and domain shift.

What a benchmark specifies

A benchmark should specify a tuple

$$B=(D_{\mathrm{test}},M,S,C),$$

where $D_{\mathrm{test}}$ is a frozen dataset, $M$ is the metric set, $S$ is the slice taxonomy, and $C$ is the compute environment. Reporting only $\frac{1}{n}{\sum }_{i}1\{{\hat{y}}_i=y_i\}$ misses false-positive cost, recall requirements, and runtime.

For detector benchmarks, $M$ should include both ranking metrics and operating-point metrics. Detection and segmentation metrics defines AP50, AP75, COCO-style mAP, AR@K, and error buckets such as miss, duplicate, localization, and background rates. Reporting only one mAP number can hide whether the model fails by missing objects, drawing loose boxes, duplicating detections, or ranking background boxes too high.

Worked benchmark comparison

On a 10-image binary benchmark with 5 positives, compare two candidate models:

model	correct predictions	true positives found	false positives	latency
fast	7 of 10	3 of 5	1	12 ms
slow	9 of 10	4 of 5	0	47 ms

The resulting metrics are

model	accuracy	recall	false positives	latency
fast	$7/10=0.700$	$3/5=0.600$	1	12 ms
slow	$9/10=0.900$	$4/5=0.800$	0	47 ms

The slow model is better on accuracy, recall, and false positives, but it is almost four times slower. If the product needs sub-20 ms inference, the fast model may still be the deployable choice even though its task metrics are worse.

Caveats

Benchmark leakage is common: duplicate images, slices from the same patient, near-identical video frames, or training-set augmentations in the test set. For object detection, match image size and NMS settings; for image classification, report class imbalance and calibration.

References

• Speed/accuracy trade-offs for modern convolutional object detectors
• Microsoft COCO: Common Objects in Context

Nav

Section — Computer Vision

← Document Image Analysis and Field Extraction Domain Shift →