The first version of our pipeline could do exactly one thing: get an image into the cluster. It worked flawlessly until the first Friday-evening incident, when "just put it back" turned out to be a forty-minute manual ritual with kubectl and a search through build history for the previous tag.
Since then my rule for deploys is simple: fix the rollback path first, optimise rollout speed second.
What every service declaration must contain
| Field | Why |
|---|---|
previous |
the exact image that ran before this one |
rollback hook |
what to do with state on rollback (migrations, cache) |
smoke |
two or three checks that the service is alive |
owner |
the person we wake up, not the general channel |
Migrations are the one thing that does not roll back
An image rolls back in seconds. A database schema does not. So migrations live a separate life from the release: first the column, one release later the code that reads it, and only then the removal of the old path. Backward-compatible migrations are boring right up until the day they save your evening.
A deploy is not the moment the image lands in a pod. A deploy is the moment any human can put everything back in two minutes and prove it with two smoke tests.
This is tested automatically, by the way: CI has a job that deploys, pushes a deliberately broken image, rolls back, and runs the smoke checks. A failure of that job counts as an incident even though nothing user-facing went down.