Observability

Metrics, logs, traces, alerts and dashboards — one set of files, run by both environments.

prometheus/
  prometheus.yml        scrape config for the Compose stack (static targets, env="local")
  kubernetes.yml        scrape config for the cluster (Kubernetes discovery, env="prod")
  rules/                recording rules, alerts, SLOs — loaded unchanged by both
  tests/                promtool unit tests for every alert and SLO rule
alertmanager/           routing: page, ticket, dead man's switch; env="local" goes nowhere
loki/loki.yml           the Compose Loki; the cluster's is the HelmRelease values
alloy/
  pipeline.alloy        what a log line becomes — imported by both configs below
  config.alloy          Compose: containers found through the runtime socket
  kubernetes.alloy      cluster: pods found through the Kubernetes API
tempo/tempo.yml         the Compose Tempo
grafana/
  provisioning/         datasources (URLs from the environment) and the dashboard provider
  dashboards/           four dashboards, as JSON
kustomization.yaml      turns the files above into ConfigMaps for the cluster

01How the two environments share it#

Compose bind-mounts these files into the containers (deploy/compose/docker-compose.observability.yml). The cluster gets them as ConfigMaps generated by kustomization.yaml, which the HelmReleases in deploy/k8s/infrastructure/observability mount instead of rendering their own. Where the two genuinely differ — how targets and logs are found, which storage Loki uses — there are two files, and the differing part is the only thing in them. Everything a reader might edit to change behaviour (a rule, a route, a panel, a log stage) exists once.

The env label is the switch. Prometheus stamps it as an external label, and Alertmanager's first route sends env="local" to a receiver with no integrations: alerts fire and are visible locally, and nobody's phone rings because a laptop went to sleep.

02Budgets, enforced rather than documented#

Log streams carry exactly env, service_name, level and, in the cluster, namespace. Anything with a value per request — trace_id, request_id — is structured metadata: queryable, but it does not create a stream per value. The measurements behind each budget are in docs/cost-analysis.md.

03Changing something#

make obs-validate   # every config through its own binary, the rule unit tests, the routing tree
make obs-reload     # apply edited rules and routes to the running local stack
make alerts         # what is firing right now

make obs-validate is exactly what the observability job in CI runs, with the same pinned images the stack runs.

Adding an alert is three changes in one commit: the rule in prometheus/rules/, a test in prometheus/tests/ that shows when it fires and when it must not, and a section in docs/runbooks/alerts.md that its runbook_url points to. An alert without a runbook wakes someone up to guess.

Adding an objective is three rules in prometheus/rules/slo.yml — the target, the bad events, the total events — and a test. The windows, the budget and both burn-rate alerts are written over every objective at once. Why the existing numbers are what they are is in docs/slo.md.

Changing a dashboard means editing its JSON. Dashboards are provisioned read-only; a change made by clicking has no diff and no review, and is gone the next time the container is.

04Locally#

All of them bind to 127.0.0.1 only.