Why models fail after the notebook

Training a good model is often the easiest part of a machine learning project. Industry surveys and practitioner reports have described for years how many models never reach production, or quietly degrade after they do. The reasons are rarely about the algorithm.

Common failure points

• Irreproducible training: nobody can recreate the model that is running, because the notebook was edited, the data was overwritten or a library version changed.
• Training serving skew: a feature is computed one way in the training notebook and another way in the production service, so the model sees inputs it was never trained on.
• Silent data changes: an upstream team renames a category or changes units, and predictions become nonsense without any error.
• No monitoring: the model's accuracy drops for months before someone notices in a business report.
• Risky releases: a new model replaces the old one for everyone at once, with no way to compare or roll back quickly.

What MLOps means here

MLOps borrows from software engineering and DevOps: version control, automated tests, continuous integration, repeatable deployments and monitoring. It adds what is specific to machine learning: data and model versioning, experiment tracking, evaluation gates and drift detection.

You do not need a large platform to start. A small team can get most of the value from a few habits and widely used open source tools.

Our case study

Crumb, a chain of twelve bakeries, wants to forecast tomorrow's sales per product per store, so each store bakes enough without waste. A data scientist built a promising gradient boosting model in a notebook. Our job is to turn it into a reliable system:

1. A training pipeline that anyone can rerun with one command.
2. Tracked experiments and a registry of approved models.
3. A nightly batch job that writes forecasts for every store.
4. An API for the store app to request updated forecasts during the day.
5. Monitoring that alerts us when data or accuracy drifts.
6. A safe process for retraining and releasing new versions.

Maturity in steps

Think of MLOps as a ladder, not a checklist you must complete before shipping. Step one is reproducible training and a manual release. Each later step reduces risk or manual work. Climb only as high as your problem needs. A model that runs once a month for one team needs less machinery than one serving millions of requests.