Operations

Running the system from day to day: what it is sized for, how it changes, what runs on a schedule, and what has actually been exercised. Incidents have their own runbooks in docs/runbooks/; start at incident-triage.md.

01What fits on the node#

One CAX21: four Ampere vCPUs and 8 GB. Every request and limit in deploy/k8s cites the measurement it came from; this table adds them up.

k3s and Flux's controllers are the rest of the node's fixed overhead and are not in the table.

Two things follow:

• Requests are what the scheduler guarantees, and 3.7 Gi of them fit with room. At the HPA ceiling the API adds two more replicas and requests reach 4.7 Gi.
• Limits add up to more than the node, which is deliberate: they are ceilings a container is killed at, not reservations, and they never all peak at once. What would make them peak together is a load spike, which is what the HPA and the objectives exist to catch.

The dev overlay is not on the node. Nothing in clusters/prod reconciles it; the zero-downtime drill deploys it into kind. Putting it on this node would add 2.1 Gi of requests, half of that a second Kafka, and scaling it to zero outside working hours would be the first thing to do.

02Connection budget#

Postgres allows 50 connections (deploy/k8s/base/postgres/config/postgresql.conf), spent as:

The base backup streams over a replication connection, counted against max_wal_senders. Raising DB_POOL_MAX without raising max_connections moves the queue from the pool into Postgres, where it is harder to see. Pool exhaustion is visible instead: DatabasePoolExhausted raises a ticket, and a request that waits longer than three seconds is answered 503 with Retry-After.

03Changing things#

Everything below is a commit. Flux applies it; nothing is changed with kubectl edit.

Application code. Merge to main. CI builds, scans and signs the image, image automation commits the new tag within fifteen minutes, and the rollout replaces pods one at a time with the drain. Schema changes follow zero-downtime-migration.md.

Postgres configuration. Edit deploy/k8s/base/postgres/config/postgresql.conf. The ConfigMap is updated in place — it deliberately has no content hash, so an edit does not restart the database — and the kubelet refreshes the mounted file within about a minute. Then:

kubectl -n blueprint exec postgres-0 -c postgres -- psql -U blueprint -d blueprint -c 'SELECT pg_reload_conf()'
kubectl -n blueprint exec postgres-0 -c postgres -- psql -U blueprint -d blueprint \
  -c "SELECT name, setting FROM pg_settings WHERE pending_restart"

A setting listed as pending_restart — shared_buffers, max_connections, archive_mode, wal_level — needs kubectl -n blueprint rollout restart statefulset/postgres. That is a short outage: readiness fails and the gateway returns 503 until the database is back. Do it at a quiet hour. The archive and backup scripts need nothing: they are read on every run.

A secret. sops deploy/k8s/config/prod/<file>.enc.yaml, commit. Pods read secrets as files, so most pick up a rotated value within a minute or two; the database password also has to be changed in Postgres itself (ALTER ROLE) in the same window.

A chart or an image version. Renovate opens the pull request. The HelmRelease upgrades with automatic rollback on failure; check flux get helmreleases -A after it merges.

k3s. Change k3s_version in infra/ansible/roles/k3s/defaults/main.yml and run make provision. The node restarts its control plane and every pod, so this is planned downtime of a few minutes, within the budget the availability objective allows for exactly this (docs/slo.md).

The node itself. terraform plan shows the replacement; the volume and the primary IP survive it. After make tf-apply and make provision, Flux reconciles everything from git. The data comes back from the volume, or from object storage if the volume is lost too (db-restore.md).

04Scaling#

05What runs on a schedule#

Retention: 30 days of metrics (the error-budget window), 30 days of logs in Loki and in the bucket, 14 days of base backups and WAL, 24 hours of Kafka log.

06What has been run, and where#

The difference between a configuration that validates and one that works was found, repeatedly, by running it. So this is stated plainly.

Not yet exercised against a live Hetzner project: terraform apply, the Ansible playbook on a real node, Flux bootstrapping from this repository, certificate issuance from Let's Encrypt, and Loki and the WAL archive against Hetzner's object storage endpoint rather than an S3 stand-in. Each of those is a first-deploy risk, and the first deploy should be watched with that list in hand.