Manyakin
РУ

IV / NOTES

Models live in production, not on a laptop

A laptop with 97% accuracy and a service running the same checkpoint are two different models. What changes when inference moves into Kubernetes.

04 JUN 2026 · 1 MIN READ

We had a laptop where everything worked. 97% accuracy, clean dataset, beautiful chart. A week after the checkpoint moved to production, the same model scored 84%. It had not degraded — the world around it had changed.

What actually changes on the move

On a laptop, the model gets data "as is". In production there is infrastructure between the model and the world, and infrastructure has habits:

  • Parsing. In the dataset a date was a datetime; in production it is a string in three formats, because three source systems write differently.
  • Packages. Train and serve live in different images, and the versions must match. pip freeze is not documentation, it is evidence.
  • Time. Inference on a laptop happens whenever you ran the notebook. In production it is a p99 at a specific second under a specific SLA.

The minimum inference container

FROM python:3.12-slim
COPY requirements.lock /tmp/
RUN pip install --no-cache-dir -r /tmp/requirements.lock
COPY app/ /app/
HEALTHCHECK --interval=15s CMD python -c "import urllib.request as u; u.urlopen('http://127.0.0.1:8000/healthz')"
CMD ["uvicorn", "app.serve:app", "--host", "0.0.0.0", "--port", "8000"]

The /healthz endpoint must answer "not alive" when the model failed to load or did not pass a canary data packet. Otherwise Kubernetes happily sends traffic to a pod whose only skill is failing gracefully.

Metrics that must exist on day one

  1. Share of features outside the training distribution.
  2. The hourly prediction distribution — its own baselines, not "cluster average".
  3. Inference time as a histogram, never as a mean.

Drift does not arrive with headlines; it arrives as a quiet histogram shift in week two. Whoever watches the mean finds out last.

COPIED