Ranking and Retrieval Metrics

Ranking metrics measure whether useful results appear early enough. They are the feedback loop for BM25, hybrid search, reranking, and RAG retrieval. The right metric depends on the task: one good result, many relevant results, or graded evidence quality.

In retrieval, relevance means how useful a document or chunk is for the query. Some tasks use binary relevance, where a result is either relevant or not; others use graded relevance, where a result can be perfect, useful, marginal, or irrelevant.

For the worked definitions and examples, see Precision, Recall, MAP, MRR, and NDCG.

Precision, recall, and ranked metrics

For binary relevance in the top $k$:

$$Precision@k=\frac{\#\text{ relevant in top }k}{k},Recall@k=\frac{\#\text{ relevant in top }k}{\#\text{ relevant in corpus}}.$$

Average precision for one query averages precision at ranks where a relevant item appears:

$$\mathrm{AP}=\frac{1}{R}\sum _{i=1}^{n}Precision@i1\{{\mathrm{rel}}_i=1\}.$$

MRR uses the first relevant rank. NDCG handles graded labels:

$$DCG@k=\sum _{i=1}^k\frac{{\mathrm{rel}}_i}{{\log }_2(i+1)},NDCG@k=\frac{DCG@k}{IDCG@k}.$$

Worked example

For ranked graded relevance labels $[3,0,2,1,0]$, binary relevance appears at ranks 1, 3, and 4. The table calculates several metrics on the same ranking so the differences are visible: some metrics care only about the first relevant hit, while others reward multiple relevant or highly graded results.

Metric	Calculation	Value
$P@3$	two relevant results in the top three	0.667
$R@3$	two of three known relevant results found	0.667
$AP$	$(1/1+2/3+3/4)/3$	0.806
$RR$	first relevant result at rank 1	1.000
$NDCG@5$	graded gain discounted by rank and normalized by ideal order	0.930

The first result is relevant, so reciprocal rank is perfect. $R@3$ is not perfect because one relevant item is still below rank 3.

Choosing metrics

Use MRR when one answer is enough, precision@k when the visible page must be clean, recall@k when missing evidence is costly, MAP when many relevant documents should be found, and NDCG when labels are graded. For RAG, pair these with source coverage and answer-level RAG evaluation.

References

• Manning, Raghavan, and Schuetze, Introduction to Information Retrieval: Evaluation
• scikit-learn API: ndcg_score

Nav

Section — Information Retrieval and Search

← Search Evaluation Precision, Recall, MAP, MRR, and NDCG →