Sequence Labelling

Sequence labelling assigns an output tag to each token. Named entity recognition, part-of-speech tagging, slot filling, and some information extraction pipelines are sequence labelling tasks. The labels depend on tokenization, so the model predicts over model tokens even when humans annotate words or spans.

Per-token classification

A token classifier estimates

$$P(y_i=k\mid x_{1:n})=\mathrm{softmax}(W{h}_i+b{)}_k,$$

where $h_i$ is the contextual representation from an encoder such as a BERT-style encoder. A linear-chain CRF adds transition scores between adjacent labels:

$$P(y\mid x)=\frac{\exp {\sum }_i(s_i(y_i)+T_{y_{i-1},y_i})}{Z(x)}.$$

BIO tags encode span boundaries with labels like B-LOC, I-LOC, and O.

Worked example

This snippet evaluates token-level sequence labels with accuracy, macro-F1 excluding O, and a label-ordered confusion matrix.

import numpy as np
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score
 
np.random.seed(7)
gold = [["O", "B-LOC", "O", "B-DATE"], ["B-PER", "O", "B-ORG", "I-ORG"]]
pred = [["O", "B-LOC", "O", "O"], ["B-PER", "O", "B-ORG", "I-ORG"]]
labels = ["B-DATE", "B-LOC", "B-ORG", "B-PER", "I-ORG", "O"]
y_true, y_pred = sum(gold, []), sum(pred, [])
print("token_accuracy", round(accuracy_score(y_true, y_pred), 3))
print("macro_f1_no_O", round(f1_score(y_true, y_pred, labels=labels[:-1], average="macro", zero_division=0), 3))
print("confusion_labels", labels)
print(confusion_matrix(y_true, y_pred, labels=labels))

Observed output:

token_accuracy 0.875
macro_f1_no_O 0.8
confusion_labels ['B-DATE', 'B-LOC', 'B-ORG', 'B-PER', 'I-ORG', 'O']
[[0 0 0 0 0 1]
 [0 1 0 0 0 0]
 [0 0 1 0 0 0]
 [0 0 0 1 0 0]
 [0 0 0 0 1 0]
 [0 0 0 0 0 3]]

Token accuracy looks high because O is common, but the date span was missed entirely. Span-level review is therefore mandatory for extraction tasks.

Caveats

BIO legality matters: an I-ORG after O is ambiguous unless the decoder fixes or rejects it. Subword tokenization forces a policy for projecting word labels to pieces. Report token-level and span-level metrics separately in evaluation of NLP systems.

References

• Jurafsky and Martin, Speech and Language Processing, 3rd ed. draft
• scikit-learn API: f1_score

Nav

Section — Natural Language Processing

← Urgency Classification Named Entity Recognition →