Production Infrastructure BlueprintBlueprint Rev. 2026-08-09
  • Reference architecture
  • Rev. 2026-08-09
  • Hetzner CAX21 · k3s v1.33
  • 10 decision records · 6 runbooks

Production Infrastructure Blueprint

A four-component distributed system taken from git push to a monitored, alerting, zero-downtime production deployment — provisioned, built, deployed, observed, backed up and operated from one repository, with every footprint and cost figure measured rather than assumed.

Read the architecture → Run it locally What running it found Source ↗
Monthly bill
€17.12node, IP, volume, storage
API image
170 MBfrom 523 MB naive
Code-only deploy
33 KBthe application layer
Rollout failures
0of 6,001 requests under load
Alerts
21each with a promtool test
Point-in-time restore
12 sstop to promoted, in kind

What this is #

The point is not that it uses Kubernetes. It is that every layer is provisioned, built, deployed, observed, backed up and operated from code in one repository, that the resulting monthly bill is measured rather than assumed, and that the failure modes which only appear once services talk to each other are handled rather than hoped away.

The system is deliberately held at four components. A fifth would add surface without adding a new operational problem to demonstrate; each of these earns its place by exposing one that the others structurally cannot.

ComponentOwnsExists to demonstrate
Traefikrouting, TLS, the SLO measurement pointService discovery that works the same way in Compose (labels) and Kubernetes (Ingress) — one component, not two answers.
apithe schema, the synchronous path, the outbox and its relayCache-aside reads, transactional writes, graceful drain, and readiness that means something.
workerthe inventory read modelAt-least-once delivery, idempotency, bounded retry and a dead-letter topic that is actually inspected.
Postgres · Redis · KafkastateThree failure domains with three different consequences: one gates readiness, one degrades, one must never gate readiness at all.

The load-bearing constraint. Java and Kafka are the expensive option in their categories. Rather than hide that, the repository measures and pays it down in public: the image went from 523 MB to 170 MB, every memory limit cites the resident size it was derived from, and Kafka's 1.4 GB — a third of the node's budget — is recorded as the most expensive decision here (ADR 0002, ADR 0004).

The system #

A write commits the item and its event in one Postgres transaction. A relay publishes committed events to Kafka; the worker consumes them idempotently and maintains a read model in Redis that the API serves back. Nothing writes to two systems at once, so there is no state the two services can disagree about permanently.

The system: request path and asynchronous path A client reaches the api through the Traefik gateway. The api reads through Redis and writes an item and its event to Postgres in one transaction. An outbox relay in every api replica claims committed events with FOR UPDATE SKIP LOCKED and publishes them to Kafka. The worker consumes them at least once, applies each event once to the read model in Redis, and routes records that exhaust their retries to a dead-letter topic. Postgres ships WAL and nightly base backups to object storage. client Traefik gateway · :443 api :8080 · mgmt :9090 virtual threads Redis cache + read model noeviction HTTPS /api cache-aside read reconcile from Postgres, only when quiet PostgreSQL 16 items outbox_events item + event, one transaction object storage WAL archive nightly base backup ≤ 5 min outbox relay in every api replica claim · FOR UPDATE SKIP LOCKED Kafka item-events · KRaft acks=all worker idempotent consumer 3 partitions · 3 threads at-least-once applied once per event id item-events.DLT retries exhausted request path asynchronous path background work
Fig. 1 — Blue is the asynchronous path, the part the design turns on. A request finishes when the transaction commits; nothing below Postgres is on its path, which is why a broker outage costs freshness and nothing else.
  • No dual write. The item and the event describing it commit together; there is no moment where Postgres has the write and Kafka does not know (ADR 0005).
  • A broker outage costs freshness and nothing else. Writes keep committing, the relay loses no attempts while the broker is unreachable, and readiness excludes Kafka for exactly that reason (ADR 0007).
  • Delivery is at least once, and that is fine. A republished event is a no-op at the worker's idempotency guard; a record that keeps failing goes to the dead-letter topic instead of blocking its partition.

The write path end to end, the failure domains, and where it runs →

Claims, and what tests each one #

Every layer passed its linters and validators before it first ran, and most of the defects later found were invisible to all of them. So each claim this repository makes is held up by something that causes the failure and checks the outcome — most of them on a schedule, because a claim nobody has re-tested recently is a hope.

ClaimTested byWhereHow often
A request crosses the whole system scripts/smoke-test.sh: 27 assertions, gateway to Postgres to Kafka to the worker, and back out of Redis Compose, Docker and rootless podman every make bootstrap
A rollout fails no request make drill: every API pod replaced under constant load, half of it writes — 0 failed of 6,001 kind, the dev overlay weekly in CI
The database restores to any second make restore-drill: back up, destroy, restore; fails on one row lost or one kept that should not be containers, production's config and scripts weekly in CI
A broker outage costs events nothing Kafka stopped for 150 s under load: 15 of 15 writes succeeded, the backlog drained 12 s after it returned Compose; the relay's unit tests hold the semantics every change to the relay
Kafka never gates readiness an integration test on the readiness group, and a semgrep rule on the configuration CI every change
Nothing unsigned of ours runs admission test: a signed image admitted; unsigned, and signed by another workflow, refused kind, production chart values when the policy changes
Every alert fires when it should, and only then promtool unit tests for all 21 alerts and the objectives CI observability job every change under observability/
Every manifest is valid for its cluster every kustomization rendered and checked strictly against the Kubernetes 1.33 schemas CI manifests job every change under deploy/k8s/

Not yet exercised against a live Hetzner project: terraform apply, the Ansible playbook on a real node, Flux bootstrapping from this repository, certificate issuance from Let's Encrypt, and Loki and the WAL archive against Hetzner's object storage endpoint. Each is a first-deploy risk, stated in Operations rather than left to be found.

The bill #

One €10.49 ARM node and €6.63 of IP, volume and object storage run the application, its database, broker and cache, metrics, logs, traces and backups. Prices are list prices with sources; footprints are measured, with the method.

This system, Hetzner
€17.12per month, excluding VAT
Same architecture, AWS
$167.62EKS, one node, self-hosted stores
Managed + Datadog, AWS
$650.74RDS, ElastiCache, MSK, Datadog

Where the managed build's $650.74 goes

USD per month, us-east-1 on-demand list prices · docs/cost-analysis.md §7

  • Kafka (MSK, two brokers minimum) $301.84 (46%)
  • Observability (Datadog) $167.53 (26%)
  • EKS control plane $73.00 (11%)
  • Node, load balancer, IPs, RDS, ElastiCache $108.37 (17%)

Against €17.12, the same architecture on AWS costs roughly ten times as much and the managed build nearly forty — 7 to 12 and 29 to 47 times for any exchange rate between 0.8 and 1.3 dollars to the euro, so the conclusion does not depend on it. The comparison does not buy the same availability, and it ignores labour; the cost analysis says what it leaves out as plainly as what it includes.

Where to start #

Reviewing the design Architecture and the close calls How the pieces fit and fail, then the decisions that cost something: Kafka on one broker, the outbox, readiness. Architecture · ADR 0004 · ADR 0005 · ADR 0007 Judging operability What on-call actually gets Triage from the first page to a diagnosis, three objectives with burn-rate alerts, and a runbook section per alert. Incident triage · SLOs · Alert catalog Checking the numbers Footprint and the monthly bill Every size, memory and log-byte figure with its method, priced line by line against two AWS builds. Cost analysis · Measurements Auditing security Controls, and the known gaps The edge, the node, the cluster, the supply chain from pinned input to admission — and what is not protected. Security · Delivery pipeline · Invariants Seeing what broke Defects found by running it A false zero-downtime claim, an outbox that stranded events, an admission policy that did not exist. What running it found · Build record Running it One command, then the drills The whole stack with dashboards on a laptop, and the same drills CI runs every week. Quickstart · Make targets

Stack #

LayerTechnologyDecision
ServicesJava 21, Spring Boot 3.5 on virtual threads, Gradle multi-projectADR 0002
GatewayTraefik v3Architecture
System of recordPostgreSQL 16, Spring Data JPA, Flyway expand/contractSchema changes
Cache and read modelRedis 7ADR 0006
MessagingApache Kafka, KRaft, one broker; transactional outbox with a polling relayADR 0004 · 0005
BackupsContinuous WAL archiving and nightly base backups to object storageADR 0009
ContainerLayered JAR, jlink runtime, distroless, non-root, read-onlyMeasurements
Orchestrationk3s v1.33 on one ARM node, Kustomize base and overlaysADR 0001
ProvisioningTerraform (Hetzner Cloud, Cloudflare DNS), AnsibleTerraform · Ansible
DeliveryGitHub Actions builds, scans and signs; Flux v2 reconcilesADR 0003
Supply chainSHA-pinned actions, cosign keyless, SPDX SBOM, SLSA provenance, policy-controller at admissionADR 0008
SecretsSOPS and age in git, mounted as files, never environment variablesSecurity
ObservabilityPrometheus 3.13, Alertmanager 0.33, Loki 3.7 with Alloy, Tempo 2.10, Grafana 13.1ADR 0010
esc
↑ ↓ to move↵ to open/ or ⌘K to search