RAG Evaluation

RAG evaluation separates retrieval quality from generation quality. A good final answer can hide weak retrieval, and a bad answer can occur despite perfect retrieved evidence. Evaluate retrieval pipelines, grounding, citations, and abstention separately.

Retrieval and generation metrics

Useful retrieval metrics include context recall, precision@k, nDCG, and filter correctness. Useful generation metrics include answer support, citation precision, citation coverage, abstention quality, and task success. LLM-as-judge can grade semantic support, but source IDs, retrieved chunk membership, and citation presence should be deterministic checks.

Metric	Definition
Context recall	Fraction of labelled evidence sources or chunks recovered into the retrieval context.
Citation precision	Fraction of cited sources that actually support the cited claim or belong to the expected evidence set.
Citation coverage	Fraction of answer claims or required facts that carry a citation.
Answer support	Whether generated claims are entailed by, or at least directly backed by, retrieved evidence.
Claim support rate	Fraction of checked claims that are supported by the retrieved evidence.
Abstention quality	Whether the system refuses answerless or unsafe cases while still answering supported cases.
Task success	Whether the final answer satisfies the benchmark’s task-specific acceptance criteria.

With expected evidence $E$, retrieved context $C$, cited claims $Q_c$, and all checked claims $Q$, common deterministic summaries are:

$$context\; recall=\frac{|E\cap C|}{|E|},citation\; precision=\frac{\#\text{ supported citations}}{\#\text{ citations}},$$ $$citation\; coverage=\frac{|Q_c|}{|Q|},claim\; support\; rate=\frac{\#\text{ supported claims}}{|Q|}.$$

Abstention quality and task success are usually binary or rubric scores defined per benchmark item, then averaged across items.

The evaluation set should include answerable questions, unanswerable questions, stale-source cases, conflicting-source cases, and adversarial retrieved text. Without those slices, a RAG system can look strong by answering easy questions while failing the exact cases that retrieval was meant to solve.

Worked evaluation table

Suppose the gold evidence set is {leave-eligibility, manager-approval}. The retriever returns leave-eligibility, parking, and manager-approval, and the answer cites only leave-eligibility.

Check	Calculation	Result
Context recall	$2/2$ gold chunks retrieved	1.0
Citation precision	$1/1$ cited sources are gold evidence	1.0
Claim support rate	$2/3$ answer claims supported	0.667

The retriever found both gold chunks and the citation points to a gold source, but only two of three answer claims were supported.

Caveats

A single aggregate score hides the failing stage. Keep per-stage traces from retrieval pipelines, and report metrics by query type, source freshness, permission scope, and answerability.

Framework traces help only when the evaluation harness reads them. For LangChain RAG agents, record retriever inputs, selected chunks, tool calls, and final citations. For LangGraph workflows, score the node trace and checkpointed state so retrieval, routing, drafting, and validation failures remain separable.

References

• Lewis et al., 2020, Retrieval-Augmented Generation
• OpenAI API documentation: Evals
• Anthropic Claude docs: Reduce hallucinations

Nav

Section — Generative AI and Agentic Systems

← Hallucination Mitigation RAG Architecture Comparison →

Learning path — Generative AI systems

← Tool Use and Function Calling Responsible AI, Safety, and Governance →