Your ML Model Is Accurate in Testing and Useless in Production: Run This MLOps Audit

An MLOps production readiness checklist covers the checks separating a model that scores well on a held-out set from a system that still scores well next quarter: training/serving parity, drift detection, prediction monitoring, rollback, reproducibility, retraining triggers, a named owner. Accuracy is one line item on it.

Here's the position I'll defend. The most dangerous model you run isn't the one that crashes. A crashed inference service pages someone at 3am and gets fixed by breakfast. The model that drifted quietly keeps answering with the same confident float attached, the API keeps returning 200, the dashboard stays green, and nobody finds out until finance asks why approval rates look strange four months later.

The gap between a notebook and a production system is ownership, not accuracy.

Why does accuracy in testing predict almost nothing about production?

Test accuracy measures one model against a frozen sample of the past, scored under the conditions that produced the training data. Production measures a whole system against a moving present: live feature pipelines, retries, stale caches, schema changes, users behaving differently than last quarter. The score holds while the value collapses.

Your held-out set samples the same cleaned dataset, not the future. I've watched a team push AUC from 0.86 to 0.89 while the serving path fed that model a feature arriving empty 11% of the time. The gain was real and irrelevant, the same pattern behind why an AI demo works and the rollout still stalls. Offline accuracy is a gate, never a guarantee.

What is training/serving skew and why does it kill good models quietly?

Training/serving skew is the gap that appears when the same feature is computed one way for training and another way at inference. Same feature name, two codebases, two results. The model receives inputs it never saw during training and its predictions degrade, with no error appearing in any log.

Null handling is where it hides. Training dropped rows where avg_basket_value was missing; serving fills the gap with 0, so the model reads "we don't know" as "spends nothing". One code path called by both sides fixes this: a feature store like Feast, or a shared library plus a CI test that runs 10,000 rows through both transforms.

How do data drift and concept drift differ, and how do you detect each?

Data drift means the input distribution moved while the relationship between inputs and labels still holds. Concept drift means that relationship itself changed, so the same inputs now have a different correct answer. Data drift shows in your features today; concept drift surfaces only when ground truth arrives.

Aggregates hide segment collapse, so slice by cohort. Delayed labels are the harder constraint: predicting chargebacks, ground truth lands 60 to 90 days later, so your quality signal is a quarter stale. Common practice treats a PSI above 0.2 as material, though that threshold is credit-scoring convention, not a universal result. Backtest the level that preceded real drops in your own metrics, and use that instead.

What does monitoring that actually catches silent failure look like?

Uptime monitoring proves the service answered. It says nothing about whether it was right. Production ML monitoring needs four layers: input distributions per feature, prediction distributions, correlation with the business metric the model exists to move, and delayed accuracy once ground truth lands.

Signals that should wake somebody: mean predicted probability drifting from 0.21 to 0.34 over three weeks with no release in between, or a feature's null rate jumping from 0.2% to 11% the morning after an upstream team shipped. Neither is an error. Both say something changed.

Business-metric correlation is the layer teams skip. If predicted relevance climbed for six weeks while click-through stayed flat, one of those numbers is lying. Evidently, WhyLabs and Arize all handle this. A dashboard nobody opens isn't monitoring.

How do you keep latency and cost sane at the serving layer?

Serving latency and cost are product decisions, not infrastructure leftovers. Measure p50, p95 and p99 at the API boundary including feature fetch and post-processing, rather than timing model.predict() alone. Agree what the business will pay per thousand predictions, then design backward from both numbers.

Feature retrieval usually dominates: the forward pass might take 4ms while three warehouse lookups take 90ms. Ask whether the prediction must be fresh at request time at all: a churn score recomputed nightly into a column serves most surfaces for a fraction of the cost. When it genuinely does, ONNX Runtime and int8 quantization beat more autoscaling tuning. Route through a hosted LLM and per-token pricing dominates, the trade-off behind our guide to choosing an LLM in 2026.

Can you roll back to the previous model in minutes, and does anyone know how?

Rollback readiness means three things at once: the previous version is still deployable, switching to it is a configuration change rather than a rebuild, and someone other than the model's author can perform that switch under pressure. If reverting requires re-running a training job, you don't have a rollback.

Keep every artifact in a registry as an immutable version. Serving reads a pointer; deployment changes it, rollback changes it back. MLflow, SageMaker and Vertex AI all do this.

Then comes the part everybody skips: write the runbook, then have an engineer who didn't build the model run it as a drill on a quiet Thursday. An untested rollback is a plan. A rehearsed one is a capability, the same argument for why more generated code demands a stronger QA layer.

What does reproducibility mean for an ML system?

Reproducibility means rebuilding a specific model artifact from record: the exact training data snapshot, the feature definitions as they existed then, the hyperparameters and seeds, the library versions, and the container image that ran the job. Miss one coordinate and a regression becomes an argument instead of an investigation.

Data version is the one most teams lack. "We trained on the orders table" isn't a version, because today's orders table isn't March's. DVC or a dated snapshot path fixes that cheaply, and pinning the image by digest handles the rest, since python:3.11 resolves to different bytes in June than in February. The test: pick a model serving traffic and rebuild it from record.

Who owns the model on a Tuesday afternoon when it misbehaves?

Every production model needs a named owner, a named backup and a written escalation path, agreed before launch. Ownership covers responding to alerts, making the retraining call, making the rollback call, and answering the business when predictions look wrong. Shared ownership between data science and platform reliably means nobody answers.

The failure is organisational. A data scientist builds it, a platform team deploys it, and the drift alert routes to platform, who can't judge whether a PSI of 0.24 on region_code matters. The alert gets acknowledged, then muted two months later. Route by type instead: latency to the platform on-call, drift to the model owner by name.

At Shanti Infosoft the CMMI Level 5 process work governing our engineering comes down to named accountability and auditable evidence. Applied to ML, the deliverable was never the model file. It's the model plus the person who answers for it.

How often should you retrain, and what fires the trigger?

Two retraining strategies work and most systems need both. Scheduled retraining runs on a cadence set by how fast your data moves. Triggered retraining fires when a drift or performance threshold breaches. Either way, promotion passes an evaluation gate, because an auto-deploying retrain can quietly push a worse model live.

Cadence follows volatility, not habit: a pricing model in a weekly-repricing market needs a weekly refresh, a classifier over stable contract templates might sit happily for a year.

One trap deserves its own paragraph. If your predictions influence the labels you later train on, you have a feedback loop. A loan model that declines an applicant never learns whether they'd have repaid, so training data becomes a record of your past decisions. Hold out a randomised slice where the model doesn't decide, if your regulator allows it.

Where does human-in-the-loop belong, and what confidence threshold?

Human review belongs wherever the cost of a wrong automated decision exceeds the cost of a person checking it. Set a confidence threshold that routes low-confidence predictions to a review queue, measure what reviewers change, and feed those corrections back as labels. Calibrate first, because raw model scores are not probabilities.

A classifier that outputs 0.95 should be right about 95 times in 100 at that score, and an uncalibrated gradient-boosted model often isn't close. Run a reliability curve; if it bends, apply Platt scaling first. The queue then doubles as your best drift sensor: when override rate climbs from 4% to 12% in a fortnight, something moved before any test reached significance.

The MLOps production-readiness checklist

Run this before launch and again every quarter. Anything you can't answer with a dashboard, a runbook or a test is a finding.

1. Training/serving parity test in CI, failing the build on divergence.
2. One feature definition, one code path for training and serving.
3. Schema and range validation at ingestion, failing loudly on upstream changes.
4. Per-feature drift monitoring against a fixed reference window.
5. Prediction distribution monitoring: mean, variance, confidence-decile share.
6. Business-metric correlation, model output charted beside the KPI.
7. Segment-sliced performance plus a delayed ground-truth pipeline.
8. Latency budget and cost per thousand predictions, tracked monthly.
9. Immutable model registry: metrics, data snapshot and owner per artifact.
10. Rollback as a config change, rehearsed, with shadow and canary first.
11. Data, feature, code and environment versions recorded per artifact.
12. A named owner and backup, with alerts routed by type.
13. A retraining policy: cadence, triggers, promotion gate.
14. Calibration, a confidence threshold from error costs, and a review queue.
15. An incident runbook for drift, skew, upstream breakage and bad deploys.
16. A quarterly re-audit on the calendar with a name against it.

If you'd rather someone ran this against a model you already have live, our machine learning development team does exactly this audit, and you can book a slot to walk through your setup. There's a 7-day free trial if you want to see the work first.

Frequently Asked Questions

How is MLOps different from DevOps?

DevOps versions code and infrastructure. MLOps versions code, infrastructure, data and models together, because an ML system can break when nothing in the repository changed. Rollback has to cover data too.

How often should we retrain our model?

There's no universal answer, and anyone offering one without seeing your drift metrics is guessing. Measure how far your feature distributions move over four weeks, then set a cadence that keeps degradation acceptable.

What's the minimum viable monitoring for a first production model?

Null and range checks on every input feature, prediction distribution over a rolling window, one business metric beside it, and a log of every prediction with its inputs and version.

How do we monitor accuracy when labels arrive months later?

Watch proxies meanwhile and compute true accuracy retrospectively when labels land. Drift metrics, prediction distributions and override rates all move earlier than delayed accuracy.

Our model has been live a year with no monitoring. Where do we start?

Log every prediction with its inputs and model version, since nothing is diagnosable without that. Then compare current input distributions against whatever training data you can recover.