Abstention

Abstention is the decision not to answer, classify, retrieve, or take an action when evidence or confidence is insufficient. It is a first-class outcome, not a fallback message. In generative systems it overlaps with refusals and guardrails; in classifiers it is selective prediction based on score thresholds.

Coverage and selective risk

For threshold $\tau$, answer only when confidence $c_i\ge \tau$. The two basic evaluation quantities are

$$\text{coverage}(\tau )=\frac{1}{n}\sum _{i}1(c_i\ge \tau ),$$

and answered error rate,

$$\text{error}(\tau )=\frac{{\sum }_{i}1(c_i\ge \tau )1({\hat{y}}_i\ne y_i)}{{\sum }_{i}1(c_i\ge \tau )}.$$

The threshold should be chosen with calibration, coverage, and risk-weighted error taxonomies, not accuracy alone.

Worked calculation

For ten examples with confidences 0.98, 0.91, 0.84, 0.79, 0.73, 0.68, 0.61, 0.55, 0.49, 0.42, suppose the correct answered examples are the first, second, third, fifth, and eighth cases. The threshold trade-off is:

threshold	answered cases	coverage	answered errors	answered error rate	abstained
0.5	8	0.80	3	0.38	2
0.7	5	0.50	1	0.20	5
0.8	3	0.30	0	0.00	7
0.9	2	0.20	0	0.00	8

Raising the threshold removes errors here because the wrong cases sit below 0.8, but it answers far fewer cases. If the unanswered cases are support tickets, the cost is human queue load; if they are medical questions, the cost may be preferable to unsupported advice.

Caveats

Abstention can hide poor performance if reports only show answered accuracy. Refusal quality must be evaluated too: the system should explain limits, route to a safer workflow, or ask for missing evidence. For LLM systems, an LLM-as-judge rubric should score both unsafe answers and unnecessary refusals.

References

• scikit-learn documentation: Metrics and scoring
• NIST AI Risk Management Framework

Nav

Section — Experimentation and Evaluation

← Coverage Risk-Weighted Error Taxonomies →