Offline Evaluation

Offline evaluation measures a system on fixed historical or curated data before it reaches live users. It is a deployment gate, not a product-impact estimate. A recommender, classifier, RAG system, and extraction model can all pass offline checks and still need online experiments because real users change exposure, feedback, and cost.

Task-matched metrics

For ranked retrieval or recommendation, NDCG rewards placing highly relevant items near the top:

$$DCG@k=\sum _{i=1}^k\frac{2^{re{l}_i}-1}{{\log }_2(i+1)},NDCG@k=\frac{DCG@k}{IDCG@k}.$$

The metric is only meaningful if relevance labels, candidate generation, and time splits match the intended decision. That is why golden datasets and coverage sit upstream of the metric.

Worked calculation

Compare two rankings with graded gains:

system	top-five gains	NDCG@3	relevant items in top 3
A	3, 0, 2, 1, 0	0.905	2 of 3
B	2, 3, 1, 0, 0	0.843	3 of 3

System B retrieves more relevant items in the first three slots, but system A puts the highest-gain item first and therefore wins NDCG@3. The decision depends on whether the product values the first slot, any relevant result, or a downstream action.

Caveats

Offline logs can encode the old system’s exposure bias. Random splits can leak future information when examples are time-dependent. Reusing the same public evaluation set for every prompt or model iteration overfits the gate, so pair it with repeated sampling, blind holdouts, and incident-derived examples.

References

• scikit-learn documentation: ndcg_score
• scikit-learn documentation: Metrics and scoring

Nav

Section — Experimentation and Evaluation

Golden Datasets →