Document Understanding

Document understanding extracts meaning from pages where text alone is incomplete: forms, receipts, contracts, tables, scans, handwriting, stamps, and layout. It often starts with OCR and handwritten text recognition, then applies information extraction, named entity recognition, table parsing, and validation rules.

Layout-aware representations

A layout-aware token representation combines text and geometry:

$$h_i=f_{\theta }({\mathrm{text}}_i,x_i^{\min },y_i^{\min },x_i^{\max },y_i^{\max },p_i),$$

where $p_i$ is the page id. The model may classify tokens, link key-value pairs, or predict document type. The key difference from plain NLP is that “Total” above $42.10 and “Total” in a footer can be distinguished by coordinates.

Worked example

This snippet trains a simple token classifier using layout-style features and reports predicted token roles, accuracy, and the probability for an amount token.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
 
np.random.seed(7)
tokens = ["Invoice", "Total", "$42.10", "Due", "2026-08-01", "Thanks"]
y = np.array(["header", "key", "value", "key", "value", "other"])
X = np.array([[0, 1], [1, 1], [1, 0], [2, 1], [2, 0], [3, 0]], dtype=float)
clf = LogisticRegression(max_iter=1000, random_state=7).fit(X, y)
pred = [str(x) for x in clf.predict(X)]
print("predicted", list(zip(tokens, pred)))
print("training_accuracy", round(accuracy_score(y, pred), 3))
print("value_prob_for_amount", {str(clf.classes_[i]): round(float(clf.predict_proba([[1, 0]])[0, i]), 3) for i in range(len(clf.classes_))})

Observed output:

predicted [('Invoice', 'header'), ('Total', 'key'), ('$42.10', 'value'), ('Due', 'key'), ('2026-08-01', 'value'), ('Thanks', 'other')]
training_accuracy 1.0
value_prob_for_amount {'header': 0.169, 'key': 0.224, 'other': 0.098, 'value': 0.508}

This toy model uses only vertical position and a lexical flag, but it shows the core idea: layout features help classify tokens as keys, values, headers, or other content.

Caveats

Document systems fail when scan quality, templates, language, or page order shifts. OCR confidence should propagate into extraction confidence. Evaluation should include field exact match, source-span correctness, page-level failures, and human-review outcomes, not only token accuracy.

References

• Xu et al., LayoutLM: Pre-training of Text and Layout for Document Image Understanding
• Jurafsky and Martin, Speech and Language Processing, 3rd ed. draft

Nav

Section — Natural Language Processing

← OCR and Handwritten Text Recognition Evaluation of NLP Systems →