Information Retrieval and Search

Information retrieval is the engineering and measurement discipline behind finding useful material in a collection. This section moves from lexical indexes and ranking formulas to vector retrieval, hybrid systems, graph retrieval, the systems that host them, and evaluation.

Knowledge map

Lexical and dense retrieval are the two families; they combine in hybrid search, run on inverted and vector indexes, feed reranking and systems, and are judged by retrieval metrics.

flowchart TD
  Lexical[TF-IDF and BM25] --> Index[Inverted Indexes]
  Lexical --> Sparse[Sparse Retrieval]
  Dense[Dense Retrieval] --> Hybrid[Hybrid Search]
  Sparse --> Hybrid
  Dense --> ANN[Vector Indexes and ANN Search]
  Hybrid --> Rerank[Reranking]
  Rerank --> Systems[Search Systems and Graph Retrieval]
  Systems --> Eval[Search Evaluation and Metrics]

Reading path

Read lexical retrieval first, then dense and hybrid, indexes and systems, and finally evaluation.

1. TF-IDF: the classic term-weighting baseline.
2. BM25: the strong lexical ranking function with saturation and length normalization.
3. Inverted Indexes: the postings-list structure that makes lexical search fast.
4. Sparse Retrieval: the broader term-feature retrieval family.
5. Dense Retrieval: embedding-based semantic retrieval.
6. Hybrid Search: fusing lexical and dense results.
7. Reranking: second-stage ordering with richer features or cross-encoders.
8. Vector Indexes: exact, compressed, and approximate vector search structures.
9. Approximate Nearest Neighbour Search: trading exactness for speed.
10. Elasticsearch: a widely used Lucene-backed search engine.
11. ELK Stack: Elasticsearch, Logstash, and Kibana together for operational search.
12. Graph Based Retrieval: retrieval through links, citations, and typed neighborhoods.
13. Knowledge Graphs: structured entity-relationship stores.
14. Literature Management Search Systems: scholarly search as a worked system.
15. Search Evaluation: labelled query sets, slices, and online checks.
16. Ranking and Retrieval Metrics: the metric families for ranked lists.
17. Precision, Recall, MAP, MRR, and NDCG: the canonical ranking metrics in detail.

Connections

• Natural Language Processing supplies the embeddings and tokenization retrieval relies on.
• Recommendation Systems shares candidate generation and ranking, and Generative AI uses this as the retrieval half of RAG.

Nav

Learning path — Information retrieval and search

Inverted Indexes →