- 05 · Reliability
- docs/runbooks
Runbooks
Written for whoever is on call, at the hour they are likely to be reading. Each starts from what the reader can see, and every command in them has been run against these manifests.
Runbooks #
Start here
Incident triage
something is wrong and you do not yet know what. A page, a user report, a graph
that looks wrong. The other runbooks start from a diagnosis; this one gets you to one.
docs/runbooks/incident-triage.md · 5 min
One section per alert
Alerts
One section per alert, in the order you are likely to meet them. Every alert's runbook_url
annotation links to its section here.
docs/runbooks/alerts.md · 15 min
A release is live and wrong
Roll back a bad deploy
a release is live and wrong. Errors are up, latency is up, or a feature is
behaving badly enough that waiting for a fix is worse than going back.
docs/runbooks/rollback.md · 3 min
The data is wrong or gone
Restore the database
the data is wrong or gone. A migration or a bug destroyed rows, someone ran a
DELETE without a WHERE, or the volume itself is lost.
docs/runbooks/db-restore.md · 6 min
TLS is failing
A certificate is not renewing, or has expired
CertificateExpiringSoon raised a ticket, or
clients report a TLS error.
docs/runbooks/cert-expiry.md · 2 min
Before writing a migration
Change the schema without downtime
writing a Flyway migration. Read it before the migration is written, not after it
has failed.
docs/runbooks/zero-downtime-migration.md · 3 min
How they are written #
- Symptom first. Each runbook opens with what the reader is seeing, what they get back, and how long it takes — before any command.
- “Is this the right runbook?” comes before the procedure. A rollback does not undo a migration; a restore is the only procedure that throws data away.
- Cluster commands, with the local equivalent underneath where they differ, so the same page works against the Compose stack.
- Every alert’s
runbook_urlpoints at its own section of the alerts runbook; adding an alert means adding its section in the same commit. - They end with what stops it, and what to record afterwards — a missed page or a useless one is a defect in the rules, and gets a test.