Off-Policy Evaluation

Off-policy evaluation (OPE) estimates the value of a target policy ${\pi }_e$ using data collected by a different behavior policy ${\pi }_b$, without deploying ${\pi }_e$. It is essential wherever running the new policy live is costly or risky: recommenders, healthcare, ads, autonomous systems, and any offline RL pipeline that must decide whether a learned policy is safe to ship.

The Problem

We want the expected return of the target policy,

$$V({\pi }_e)={\mathbb{E}}_{\tau \sim {\pi }_e}\left[\sum _t{\gamma }^t{r}_t\right],$$

but every logged trajectory $\tau$ was generated by ${\pi }_b$. The data distribution is therefore wrong for the quantity we care about, and correcting for that mismatch is the whole task.

Importance Sampling

Inverse propensity scoring reweights logged rewards by how much more likely the target policy was to take the observed actions. For a single decision with logged reward $r$, action $a$, and context $x$:

$${\hat{V}}_{\text{IPS}}=\frac{1}{n}\sum _{i=1}^n\frac{{\pi }_e(a_i\mid x_i)}{{\pi }_b(a_i\mid x_i)}{r}_i.$$

This is unbiased when ${\pi }_b$ has support everywhere ${\pi }_e$ does and the logged propensities ${\pi }_b(a\mid x)$ are known. Its weakness is variance: when the two policies disagree, weights blow up. Weighted (self-normalized) IPS divides by the sum of weights, trading a little bias for much lower variance. In the sequential case, per-decision importance sampling multiplies the per-step ratios along the trajectory, and variance compounds with horizon.

Direct Method

The direct method fits a model of outcomes, such as a reward model or a fitted-Q evaluation (FQE) of the value function, then evaluates the target policy against the model:

$${\hat{V}}_{\text{DM}}=\frac{1}{n}\sum _i\hat{Q}(x_i,{\pi }_e(x_i)).$$

It has low variance but is biased whenever the model is wrong, especially for actions the target policy favors but the data rarely contains.

Doubly Robust Estimation

The doubly robust estimator combines the two, using the model as a baseline and importance sampling to correct its residual:

$${\hat{V}}_{\text{DR}}=\frac{1}{n}\sum _i\left[\hat{Q}(x_i,{\pi }_e)+\frac{{\pi }_e(a_i\mid x_i)}{{\pi }_b(a_i\mid x_i)}(r_i-\hat{Q}(x_i,a_i))\right].$$

It is unbiased if either the propensities or the outcome model is correct, and it typically has lower variance than plain IPS because the model absorbs most of the signal and importance weights only correct the residual.

Estimator Trade-offs

flowchart TD
  Logged[Logged data from the behavior policy] --> IPS[Importance sampling: reweight rewards by the policy ratio]
  Logged --> Direct[Direct method: fit an outcome or value model]
  IPS --> DR[Doubly robust: model baseline plus reweighted residual]
  Direct --> DR
  IPS --> Value[Estimated value of the target policy]
  Direct --> Value
  DR --> Value

Estimator	Bias	Variance	Key requirement
IPS	unbiased (with full support)	high	known behavior propensities
Weighted IPS	slightly biased	lower	known propensities
Direct method	biased if model wrong	low	accurate outcome/value model
Doubly robust	unbiased if model or propensity right	medium	one of the two components correct

Worked IPS Calculation

Suppose a logged action had behavior probability ${\pi }_b=0.25$, target probability ${\pi }_e=0.5$, and reward $r=4$. Its importance weight is $0.5/0.25=2$, so this sample contributes $2\cdot 4=8$ to the IPS average. An action the target policy avoids (${\pi }_e$ small) is down-weighted toward zero.

Caveats

OPE fails silently when the behavior policy is deterministic or unknown (no valid propensities), when support is missing (the target does things never logged), and when effective sample size collapses because a few trajectories carry almost all the weight. Report effective sample size and confidence intervals, and prefer staged online tests for final validation.

Connections

• Offline and Model-Based RL needs OPE to select and gate policies before deployment.
• Offline Evaluation covers the broader practice of estimating system quality without live traffic.
• Contextual Bandits are the one-step special case where IPS and doubly robust estimators originate.

References

• Dudík, Langford, and Li, 2011, Doubly Robust Policy Evaluation and Learning
• Jiang and Li, 2016, Doubly Robust Off-Policy Value Evaluation for Reinforcement Learning
• Thomas and Brunskill, 2016, Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning

Nav

Section — Reinforcement Learning

← Offline and Model-Based Reinforcement Learning Reinforcement Learning from Human Feedback →