Experiment Tracking
Experiment tracking records the evidence behind training and evaluation runs: code commit, dataset versioning, parameters, metrics, artifacts, environment, notes, and promotion status. It prevents “best model” from meaning “the notebook output someone remembers.”
The run record
Each serious run should create an immutable run record. The record should link input datasets, feature pipeline version, random seed, hyperparameters, metrics by slice, produced artifact, and reviewer notes. A promoted model-versioning entry should point back to the run that created it.
A run is only evidence if it is reproducible: the same code commit, dataset version, configuration, and seed must reproduce the same metrics. Tracking then makes runs comparable — you can rank candidates, see which change moved which slice metric, and diff two runs’ configurations. That comparability is what turns a pile of notebooks into a defensible decision record.
Artifact: Run Record
run:
tracking_uri: "mlflow://experiments/fraud-scorer"
run_id: "6f4a9d2"
code_commit: "9b51c0e"
dataset: "fraud_training:2026-07-11.v3"
params:
model: xgboost
max_depth: 6
learning_rate: 0.04
metrics:
validation_auc: 0.913
new_account_recall: 0.742
p95_latency_ms: 84
artifacts:
model_uri: "registry://fraud-scorer/candidate-6f4a9d2"
eval_report: "s3://ml-reports/fraud/6f4a9d2.html"Training pipelines should log this automatically, and ci-cd-for-ml should consume it for promotion gates.
Failure Modes
Tracking fails when it stores metrics without data versions, when failed runs are discarded, or when manual notebook steps are not captured. For generative systems, prompts, retrieval indexes, judge versions, and sampled outputs are part of the experiment.
References
Nav