Data Engineering

Data engineering builds reliable paths from source events and operational records to analytical tables, features, and governed datasets. The useful through-line is contract first: what is the grain, who owns it, how is it tested, and can it be rebuilt?

Knowledge map

Data modelling and SQL underpin warehouses; vaults, distributed layout, and vendor platforms turn warehouse models into scalable, auditable query surfaces; storage and processing choices feed pipelines and orchestration; governance and feature pipelines sit on top.

flowchart TD
  Model[Relational and Dimensional Modelling] --> SQL[SQL and Warehouses]
  SQL --> Vault[Data Vault]
  Vault --> Distributed[Distributed Warehouse Modelling]
  Distributed --> Vendors[Vendor Solutions]
  Vendors --> BigQuery[BigQuery]
  BigQuery --> Storage[Cloud Storage and Batch vs Streaming]
  Storage[Cloud Storage and Batch vs Streaming] --> Pipelines[ETL/ELT and Data Pipelines]
  Distributed --> Pipelines
  Pipelines --> Orchestration[Airflow and dbt]
  Pipelines --> Quality[Data Quality, Contracts, Lineage]
  Quality --> Features[Feature Pipelines]

Reading path

Read modelling and SQL, then storage and pipelines, orchestration, governance, and feature pipelines.

1. Relational Modelling: keys, constraints, and table relationships that protect integrity.
2. SQL: joins, aggregation, and window functions as the core transformation language.
3. Dimensional Modelling: facts, dimensions, grain, and slowly changing context.
4. Data Warehouses: curated analytical stores for shared metrics.
5. Data Vault: hubs, links, satellites, and auditable enterprise integration history.
6. Distributed Warehouse Modelling: star schemas, normalized cores, and physical layout for large analytical systems.
7. Vendor Solutions: Snowflake, Databricks, BigQuery, Redshift, Fabric, and adjacent platform tools.
8. BigQuery: managed warehouse design with partitioning and clustering.
9. Cloud Storage: object layout for raw, staged, curated, and versioned assets.
10. Batch Versus Streaming: bounded and unbounded processing trade-offs.
11. ETL and ELT: where extraction, loading, and transformation happen.
12. Data Pipelines: production dataflows with sources, transforms, targets, and watermarks.
13. Airflow: orchestration for scheduled, observable task graphs.
14. dbt: versioned SQL models, tests, and warehouse dependency graphs.
15. Data Quality: executable checks that block or warn on invalid data.
16. Data Contracts: producer-consumer agreements for schema, semantics, and ownership.
17. Data Lineage: job and dataset graph metadata for impact analysis.
18. Reproducibility: pinned inputs, code, parameters, and output snapshots.
19. Feature Pipelines: point-in-time model features for training and serving.

Connections

• Software Engineering supplies the testing and design discipline pipelines need.
• ML Engineering and MLOps consumes feature pipelines, and Cloud and Distributed Systems runs the processing.