Deploying Machine Learning Models to Production

Deploying machine learning models to production involves far more than wrapping a model in a Flask app. Reliable ML systems require thoughtful API design, containerization, monitoring, and scaling strategies that account for the unique characteristics of inference workloads.

Serving Patterns

Three primary serving patterns exist: online (real-time) inference for low-latency predictions, batch inference for periodic processing of large datasets, and streaming inference for event-driven architectures. The choice depends on latency requirements, throughput needs, and infrastructure constraints.

Online Inference with FastAPI

FastAPI provides an excellent foundation for ML inference services with automatic OpenAPI documentation, async support, and Pydantic validation. The model is loaded at startup and reused across requests to avoid reloading overhead.

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI()
model = joblib.load("model.pkl")

class InputData(BaseModel):
    features: list[float]

class Prediction(BaseModel):
    prediction: float
    confidence: float

@app.post("/predict", response_model=Prediction)
async def predict(data: InputData):
    X = np.array(data.features).reshape(1, -1)
    pred = model.predict(X)[0]
    proba = model.predict_proba(X).max()
    return Prediction(prediction=float(pred), confidence=float(proba))

Containerization

Docker ensures consistency across environments. Multi-stage builds keep images lean by separating build dependencies from runtime dependencies. For Python ML images, using slim base images and avoiding unnecessary packages can reduce image size from 2GB to under 500MB.

FROM python:3.11-slim as builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dirs -r requirements.txt

FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY model.pkl app.py ./
ENV PATH=/root/.local/bin:$PATH
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Scaling Strategies

ML inference introduces unique scaling challenges. Models are often memory-heavy and compute-intensive. Horizontal scaling with load balancers works well for stateless models. For large models, model parallelism or model sharding spreads the computation across multiple workers.

Monitoring & Observability

Beyond standard metrics (latency, throughput, error rate), ML systems require model-specific monitoring. Prediction drift, feature drift, and data quality checks alert you when the model receives inputs outside its training distribution. Tools like Prometheus + Grafana provide dashboards, while Evidently AI handles ML-specific monitoring.