Evaluation Datasets
Evaluation datasets are curated examples used to measure model behavior before release and during regression checks. They are broader than golden datasets: an evaluation set may be large and statistical, while a golden set is usually small, trusted, and inspectable.
What an evaluation set specifies
An evaluation dataset needs an owner, sampling policy, label policy, slice definitions, refresh cadence, allowed uses, and known limitations. It should be versioned through dataset versioning and kept separate from training data. Offline evaluation informs release decisions, while A-B testing measures live impact.
Artifact: Evaluation Card
evaluation_dataset:
name: fraud_release_eval
version: v12
owner: risk-ml
source_period: "2026-04-01..2026-06-30"
label_policy: "chargeback confirmed within 45 days"
slices:
- payment_type
- country
- new_account
gates:
global_auc: ">= 0.89"
new_account_recall: ">= 0.72"
p95_latency_ms: "<= 120"
refresh_policy: "new major version quarterly; patch version for label corrections"The gate should match failure cost. A fraud system may block release on a low-recall slice even when global AUC improves. The experimentation section covers offline evaluation more generally.
Slice gates
Release gates should be defined per slice, not only globally, because an averaged metric hides localized regressions. In the card above, new_account_recall >= 0.72 can block a release whose global_auc improved: the model got better on the easy majority while getting worse on the rare, high-cost new-account cases. Choose slices by cost and by who is harmed when the model is wrong — payment type, geography, protected group, new-versus-returning — then set each gate to the level the team can defend to that population, not just the level that maximizes the headline number.
Failure Modes
Evaluation sets go stale when product flows, abuse patterns, or label rules change. They can also leak into model development through repeated tuning. Track every release decision made against a version so model degradation can compare production labels against the same assumptions.
References
Nav