Retraining with a promotion gate on Azure Machine Learning

A small side project: does the pipeline stop a worse model from reaching production?

Setup

Three rounds

RoundCandidateCandidateProductionDifference (95% CI)Gate
1word n-grams80.8%7.0% (majority class)+73.7 (+72.1 to +75.3)promoted → v1
2character n-grams83.9%80.8% (v1)+3.2 (+2.1 to +4.3)promoted → v2
3character n-grams, 30% labels flipped75.3%83.9% (v2)−8.6 (−9.9 to −7.4)blocked

Test accuracy on 2,974 utterances. Round 3 simulates a bad data delivery. Its dev accuracy (75%) looks plausible on its own, but the comparison against production is what catches it.

Serving

Blue/green on a managed online endpoint: v1 (blue) takes 100% of traffic. v2 (green) is deployed alongside it at 0%, tested directly with the azureml-model-deployment header, then traffic moves 90/10 → 0/100 and blue is removed. Serving uses a scoring script (init loads the model once, run maps {"utterances": [...]} to {"intents": [...]}) and an environment pinned to the training versions.

StepAccuracy (200 calls)p50p95
blue = v1, 100% traffic83.5%95 ms184 ms
green = v2, 0% traffic, called directly86.0%92 ms146 ms
endpoint after 90/10 → 0/100 switch86.0%92 ms154 ms

One Standard_E2s_v3 instance per deployment, one utterance per request, called from Gothenburg. The endpoint was deleted after the test.

What went wrong

Limits

One public dataset, a simple model and a gate on a single metric. Registration runs from a script after the gate rather than as a pipeline step. Retraining is triggered by script, not by a schedule or CI. No Databricks and no endpoint monitoring. Built with coding agents.

Ulf Fäldt Reifenberg · November 2026