Alert catalog — Production Infrastructure Blueprint

Two severities #

Severity	Bar	Channel
page	User-visible impact now, or a hole in monitoring that would hide it. Someone is woken up, so it has to be actionable at 03:00.	PagerDuty
ticket	Something is wrong that nobody outside will notice yet. Looked at in working hours, before it becomes a page.	Slack

What is deliberately absent: CPU, memory or disk above 80%. Those are causes, and a cause that is not hurting anyone is not worth waking someone for. The error budget pages when users are hurt, whatever the cause turns out to be (Service level objectives). Locally, the env="local" label routes every alert to a receiver with no integrations, so they fire and are visible and nobody's phone rings.

The catalog #

Alert	Severity	For	What it says
blueprint.slo.alerts
ErrorBudgetBurnFast	page	2m	The ‹slo› error budget is burning fast ‹value› of events are missing the ‹slo› objective. At this rate the 30-day error budget is gone within five days.
ErrorBudgetBurnSlow	ticket	15m	The ‹slo› error budget is burning faster than it refills ‹value› of events are missing the ‹slo› objective. Not urgent today; the budget runs out before the 30-day window does.
blueprint.coverage
GatewayDown gateway	page	2m	The gateway is down or unscrapeable, so the availability SLO is blind No healthy gateway target for 2 minutes. Requests may all be failing, and the error-budget alerts cannot fire without gateway metrics.
WorkerDown worker	page	5m	No worker is consuming, so the inventory read model is going stale The only consumer of item-events has been unreachable for 5 minutes. Writes still succeed, the inventory summary stops reflecting them, and consumer lag is not reported while nothing is consuming.
TargetDown ‹job›	ticket	5m	Prometheus cannot scrape ‹job› at ‹instance› The target has failed every scrape for 5 minutes. Either the process is down, or it returned more samples than its sample_limit and the whole scrape was rejected.
blueprint.async
OutboxRelayStalled api	page	5m	The outbox relay has stopped publishing and events are piling up ‹value› or more events have been waiting for at least 15 minutes with nothing published. The API is serving normally; the inventory read model is not being updated.
OutboxEventsStuck api	ticket	5m	‹value› outbox events exhausted their publish attempts These events will never be published without intervention. The reconciler keeps the read model's totals correct, but no consumer ever sees these events.
ConsumerLagHigh worker	ticket	10m	The worker is ‹value› records behind on ‹topic› Lag has stayed above 1000 records for 10 minutes. The worker is alive and consuming, but slower than events arrive.
DeadLetterTopicReceiving worker	ticket	—	Events are being routed to the dead-letter topic Records dead-lettered in the last 15 minutes: ‹value›. Each one exhausted its retries or could not be parsed, and is an update the read model never applied.
ProjectionDriftCorrected api	ticket	—	The reconciler corrected drift between the read model and Postgres The inventory projection disagreed with the system of record and was rewritten. Something upstream dropped or repeated an event.
ReconciliationStarved api	ticket	—	The projection has not been reconciled on ‹instance› for two hours Every reconciliation run in the last two hours found events in flight and stood down. Drift, if there is any, is not being repaired.
blueprint.dependencies
DatabasePoolExhausted api	ticket	5m	Requests on ‹instance› are queueing for database connections Threads have been waiting for a pooled connection for 5 minutes. Latency will climb before errors do; the latency SLO pages if it gets that far.
CacheUnavailable api	ticket	10m	The API cannot reach Redis and is serving reads from Postgres Cache lookups have been failing for 10 minutes. Responses are still correct, but database load is multiplied and the worker cannot update the projection either.
JvmHeapPressure ‹job›	ticket	15m	‹instance› has used over 90% of its heap for 15 minutes Sustained, not a spike before a collection. The heap is derived from the container memory limit, so the limit is what to revisit, against the measured RSS.
CertificateExpiringSoon gateway	ticket	1h	The certificate ‹name› expires in under 14 days cert-manager renews at 30 days remaining, so a certificate this close to expiry has already failed to renew at least once.
blueprint.backups
PostgresWalArchivingFailing postgres	ticket	10m	Postgres cannot archive WAL, so recent writes are in no backup The archive command has been failing for 20 minutes without shipping a segment. Nothing is lost yet; writes since the failure cannot be restored until it recovers, and pg_wal grows until it does.
PostgresBackupMissing postgres	ticket	15m	Postgres has not had a successful base backup in over 30 hours The postgres-backup CronJob has not completed a run in more than a day. A restore now replays more WAL, and once the newest base backup is 14 days old the bucket's lifecycle rule deletes it, and with it the ability to restore at all.
blueprint.monitoring
Watchdog monitoring	none	—	The alerting pipeline is working. This alert is supposed to be firing. Routed to the dead man's switch. If it ever stops, Prometheus, Alertmanager, or the route between them has failed.
LogPipelineDropping monitoring	ticket	10m	Alloy is dropping log lines instead of delivering them to Loki Lines rejected by Loki are discarded after retries. The usual cause is a new label exceeding the label budget, which Loki rejects outright.
AlertmanagerNotificationsFailing monitoring	ticket	15m	Alertmanager cannot deliver to ‹integration› Alerts are firing and nobody is being told. The dead man's switch still works if the webhook integration is healthy; check which one is failing.
PrometheusRuleFailures monitoring	ticket	—	Rules in ‹rule_group› are failing to evaluate A rule that fails to evaluate never fires. Every alert in the group is silently disabled until this is fixed.

Read from rules/alerts.yml and rules/slo.yml. The burn-rate alerts are each written once per window pair and listed once. The alert name opens its runbook section.

Adding an alert #

Three changes, in one commit:

The rule, in observability/prometheus/rules/, with a service label and a runbook_url.
A test in observability/prometheus/tests/ that shows when it fires — and, just as important, when it must stay quiet. Sample series at one minute, or the five-minute lookback makes them stale between samples and for never completes.
A section in the alerts runbook: what it means, what users see, where to look first, and what makes it stop.

make obs-validate runs every configuration through its own binary, the rule tests and the routing tree — exactly the CI job, with the same pinned images. An alert without a runbook wakes someone up to guess.

01Two severities #

02The catalog #

03Adding an alert #

Two severities #

The catalog #

Adding an alert #