Diversity, Novelty, Coverage, and Serendipity

These metrics measure qualities that relevance alone misses in evaluation of recommenders. Diversity asks whether a list contains varied items; novelty asks whether items are not already obvious; coverage asks how much catalog or user space the system reaches; serendipity asks whether recommendations are both unexpected and useful.

Beyond-accuracy measures

One intra-list diversity score is

$$\mathrm{ILD}(L)=1-\frac{2}{k(k-1)}\sum _{i<j,i,j\in L}\mathrm{sim}(i,j).$$

A simple novelty score uses item popularity $p(i)$:

$$\mathrm{Novelty}(L)=\frac{1}{k}\sum _{i\in L}-{\log }_{2}p(i).$$

Catalog coverage over many recommendation lists $\mathcal{L}$ is

$$\mathrm{Coverage}(\mathcal{L})=\frac{|{\bigcup }_{L\in \mathcal{L}}L|}{|\mathcal{I}|},$$

where $\mathcal{I}$ is the eligible item catalog. A simple serendipity score averages relevance times unexpectedness:

$$\mathrm{Serendipity}(L)=\frac{1}{k}\sum _{i\in L}\mathrm{rel}(i)\mathrm{unexpected}(i).$$

These complement ranking metrics such as NDCG.

Worked example

This snippet computes intra-list diversity, mean novelty, and catalog coverage for a recommended item list from item features and popularity.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
features = np.array([[1,0], [.8,.2], [0,1], [.2,.9]])
rec = [0, 2, 3]
catalog_pop = np.array([100, 80, 10, 5])
sim = cosine_similarity(features[rec])
div = 1 - sim[np.triu_indices(len(rec), 1)].mean()
novelty = np.mean(-np.log2(catalog_pop[rec] / catalog_pop.sum()))
print("intra_list_diversity", round(float(div), 3))
print("mean_novelty_bits", round(float(novelty), 3))
print("coverage", f"{len(set(rec))}/{len(features)}")

Observed output:

intra_list_diversity 0.602
mean_novelty_bits 3.511
coverage 3/4

The list covers three of four items and mixes two feature regions. A pure relevance ranker might reduce these values by showing near-duplicates, while narrow candidate generation can cap coverage before the ranker even runs.

Caveats

Diversity is not randomization; irrelevant variety is bad. Novelty can overpromote obscure items unless relevance is preserved. Coverage metrics should be segmented by item type and user group, because aggregate coverage can hide cold-start failures.

References

• Herlocker et al., 2004, Evaluating Collaborative Filtering Recommender Systems
• scikit-learn documentation: cosine_similarity

Nav

Section — Recommendation Systems and Personalization

← Offline Versus Online Evaluation Feedback Loops →