AI Engineering · Infrastructure

ML model deployment explained: taking AI from notebook to production

Informatics HubAugust 20267 min read

Training a machine learning model is one thing. Getting it to actually serve predictions to real users reliably, at scale, without falling over is a completely different challenge. Most ML courses stop at the notebook. This post covers what comes after.

Model deployment is the process of taking a trained model and making it available to the outside world so applications can send it data and receive predictions back. It sounds straightforward. In practice it involves a surprising number of decisions that significantly affect how well your system holds up under real conditions.

Why deployment is harder than training

Training happens in a controlled environment. You have your dataset, your compute, and you iterate until the model performs well on your validation set. Deployment puts that model in front of unpredictable real-world inputs, variable traffic volumes, and infrastructure that can fail in ways your training loop never anticipated.

A model that performs brilliantly in a Jupyter notebook and breaks in production within a week is not a useful model. Deployment is where the real engineering begins.

Production ML systems require monitoring, versioning, and graceful failure handling that notebooks never need

The main deployment approaches

REST API serving

The most common approach. You wrap your model in a web server that accepts HTTP requests with input data and returns predictions as JSON. FastAPI in Python makes this straightforward to build. The model loads once when the server starts and serves predictions on demand. Simple, flexible, and works for most use cases.

Batch inference

Instead of serving predictions in real time, you run the model periodically over a large dataset and store the results. This is the right approach when predictions do not need to be instant. Processing a million records overnight to generate recommendations for the next day is batch inference. Cheaper and simpler than real-time serving for many use cases.

Edge deployment

Running the model directly on the device rather than sending data to a server. Used in mobile apps, IoT devices, and any context where latency or connectivity is a constraint. Requires quantizing or compressing the model to fit within device memory and compute limits. Tools like TensorFlow Lite and ONNX Runtime handle this well.

Managed ML platforms

Services like AWS SageMaker, Google Vertex AI, and Azure ML handle the infrastructure for you. You provide the model and they handle scaling, monitoring, and uptime. Higher cost but dramatically lower operational overhead. Reasonable choice for teams without dedicated ML infrastructure engineers.

The tools worth knowing

FastAPI

Building prediction APIs in Python

Docker

Packaging model and dependencies together

MLflow

Tracking experiments and model versions

BentoML

Framework specifically built for ML serving

Prometheus

Monitoring prediction latency and errors

Grafana

Visualizing model performance over time

Things that go wrong in production

Knowing the common failure modes saves you from discovering them at the worst possible time.

• Data drift. The distribution of real-world inputs gradually shifts away from what the model was trained on. Predictions quietly become less accurate over months. Monitoring input distributions catches this before it becomes a problem.
• Dependency conflicts. The model was trained with library version X. The production server runs version Y. The predictions are subtly different. Docker containers solve this by locking the exact environment.
• Memory issues. Large models consume significant memory. Under heavy traffic, serving too many requests simultaneously can cause the server to run out of memory. Load testing before launch reveals this early.
• Missing input handling. Real users send incomplete, malformed, or unexpected inputs. A robust prediction API validates inputs before passing them to the model and returns meaningful error messages when something is wrong.

The simplest production setup that works

For most projects, wrapping your model in a FastAPI endpoint, containerizing it with Docker, and deploying it to a small cloud server is everything you need to get started. Add MLflow to track your model versions so you can roll back if a new version performs worse than expected. This setup is minimal, understandable, and handles a surprising amount of real-world load before you need anything more sophisticated.

Key takeaways

• Deployment is where the real engineering challenge begins, not where it ends
• REST API serving, batch inference, and edge deployment each suit different use cases
• Docker solves the dependency conflict problem by locking the entire environment
• Monitor input data distributions over time to catch model degradation before users notice it