CI/CD for ML
CI/CD for ML extends ordinary software delivery with checks for data contracts, training reproducibility, model quality, serving compatibility, and rollback metadata. A green unit-test suite is not enough to promote a model that was trained on the wrong snapshot.
What the pipeline tests
The pipeline should separately test code, data, model artifact, and deployment contract. Continuous integration blocks broken changes; continuous delivery promotes an approved model-versioning record through staging and production using docker images and rollout controls.
flowchart TD Change[Code, data, or model change] --> TestCode[Test code and contracts] TestCode --> TestData[Validate dataset contract] TestData --> EvalModel[Evaluate candidate against gates] EvalModel --> Build[Build serving image] Build --> Staging[Promote to staging] Staging --> Production[Promote to production with a rollback target]
Artifact: Promotion Workflow
name: ml-release
on:
pull_request:
workflow_dispatch:
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: python -m pytest tests/unit tests/contracts
- run: python scripts/check_dataset_contract.py --dataset churn_eval:v7
- run: python scripts/evaluate_candidate.py --min-auc 0.84 --max-p95-ms 120
- run: docker build -t registry.example.com/churn-scorer:${{ github.sha }} .This workflow references evaluation datasets and produces evidence that experiment tracking should store with the candidate run. Promotion should fail closed when dataset versions, thresholds, or serving schemas are missing.
Failure Modes
CI/CD becomes theater when it retrains on mutable tables, treats aggregate accuracy as the only gate, or deploys without a rollbacks target. Human approval is still useful for risky changes, but it should review concrete evidence, not notebook screenshots.
References
Nav