Retraining with a promotion gate on Azure Machine Learning
A small side project: does the pipeline stop a worse model from reaching production?
Setup
- Data: Amazon MASSIVE sv-SE (CC BY 4.0). 11,514 train / 2,033 dev / 2,974 test Swedish utterances, 60 intents. Registered as versioned Azure ML data assets.
- Model: TF-IDF + logistic regression (scikit-learn), deliberately simple. The subject is the pipeline.
- Pipeline: two Azure ML components,
train→gate, on serverless compute. Params, metrics and models are tracked with MLflow in the workspace. - Gate: candidate vs the registered production model on the same test set. Per-sentence paired bootstrap (2,000 resamples). Promote only if the 95% CI of the accuracy difference is above zero. A promoted candidate is registered as a new version in the model registry.
Three rounds
| Round | Candidate | Candidate | Production | Difference (95% CI) | Gate |
|---|---|---|---|---|---|
| 1 | word n-grams | 80.8% | 7.0% (majority class) | +73.7 (+72.1 to +75.3) | promoted → v1 |
| 2 | character n-grams | 83.9% | 80.8% (v1) | +3.2 (+2.1 to +4.3) | promoted → v2 |
| 3 | character n-grams, 30% labels flipped | 75.3% | 83.9% (v2) | −8.6 (−9.9 to −7.4) | blocked |
Test accuracy on 2,974 utterances. Round 3 simulates a bad data delivery. Its dev accuracy (75%) looks plausible on its own, but the comparison against production is what catches it.
Serving
Blue/green on a managed online endpoint: v1 (blue) takes 100% of traffic. v2 (green) is deployed alongside it at 0%, tested directly with the azureml-model-deployment header, then traffic moves 90/10 → 0/100 and blue is removed. Serving uses a scoring script (init loads the model once, run maps {"utterances": [...]} to {"intents": [...]}) and an environment pinned to the training versions.
| Step | Accuracy (200 calls) | p50 | p95 |
|---|---|---|---|
| blue = v1, 100% traffic | 83.5% | 95 ms | 184 ms |
| green = v2, 0% traffic, called directly | 86.0% | 92 ms | 146 ms |
| endpoint after 90/10 → 0/100 switch | 86.0% | 92 ms | 154 ms |
One Standard_E2s_v3 instance per deployment, one utterance per request, called from Gothenburg. The endpoint was deleted after the test.
What went wrong
- The no-code MLflow deployment crashed on start (a version conflict in the generated image). I replaced it with an explicit scoring script and pinned environment.
- The per-subscription vCPU quota for the first VM size was too small for blue and green side by side. I switched to a different VM family.
Limits
One public dataset, a simple model and a gate on a single metric. Registration runs from a script after the gate rather than as a pipeline step. Retraining is triggered by script, not by a schedule or CI. No Databricks and no endpoint monitoring. Built with coding agents.
Ulf Fäldt Reifenberg · November 2026