- 01 · Overview
Build record
39 commits across five weeks, in seven phases. Each phase ends with something that runs; the fixes that running it forced are commits of their own.
Commits by day #
Commits per day
4 July – 9 August 2026 · peak of 3 a day · hover a column for its date
Commit messages are one-line conventional commits. The history is linear and coarse on purpose: a commit is a unit of work finished in a sitting — “the whole log pipeline” — not a file.
Phases #
Foundation
4 – 7 July · 6 commitsThe API service, its schema, RFC 7807 errors, structured logs and metrics, and one Makefile entrypoint for everything.
- chore(repo): scaffold repository with license, editorconfig and ignore rules
- build(app): bootstrap spring boot service with gradle version catalog
- feat(db): add item domain model with jpa persistence and flyway baseline
- feat(api): expose item crud endpoints with rfc 7807 problem responses
- feat(obs): add correlation ids, structured json logs and prometheus metrics
- chore(make): add makefile entrypoints and environment template
Containers and the local stack
11 – 13 July · 8 commitsThe distroless jlink image and its measurements, the Compose stack, tests against real backends, and the first defects found by running it.
- feat(docker): add hardened multi-stage image with jlink runtime on distroless
- feat(compose): add local stack with healthchecked postgres and hardened api
- fix(api): map unmapped paths and constraint violations to real status codes
- fix(obs): rename item gauge so micrometer stops stripping the total suffix
- test(app): add unit, web-slice and testcontainers integration suites
- build(app): enforce java formatting with spotless and add pre-commit hooks
- feat(scripts): add bootstrap, teardown and smoke test entrypoints
- docs(cost): record measured image size, startup and memory figures
Four components
15 – 17 July · 5 commitsThe split into api, worker and a shared platform module; Kafka, Redis and Traefik; the outbox, the idempotent consumer and the decisions behind them.
- refactor(services): split into a gradle multi-project with a shared platform module
- feat(worker): add kafka consumer with idempotency, bounded retry and dead-letter topic
- feat(compose): add gateway, kafka and redis with health-gated startup ordering
- test(services): cover outbox, projection and dead-letter paths against real brokers
- docs(adr): record the kafka, outbox, cache and readiness decisions
CI and the supply chain
20 – 22 July · 4 commitsGates on formatting, tests and coverage; secret, SAST and configuration scanning; signed multi-arch images with SBOM and provenance; every action pinned.
- ci: gate merges on formatting, tests and per-service coverage thresholds
- ci(sec): add secret, sast and misconfiguration scanning with repo-specific rules
- ci(cd): publish signed multi-arch images to ghcr with sbom and provenance
- chore(ci): enforce action pinning and wire renovate to keep the pins current
Infrastructure and delivery
24 – 26 July · 4 commitsTerraform for the node, network, firewall and storage; Ansible hardening and k3s; Kustomize with SOPS; Flux with image automation.
- feat(tf): provision the hetzner node, network, firewall and object storage
- feat(ansible): harden the node and bootstrap k3s with audit logging and secrets encryption
- feat(k8s): add the kustomize base and overlays with sops-encrypted secrets
- feat(cd): reconcile the cluster from git with flux and automated image updates
Observability, and the manifests in a real cluster
29 July – 1 August · 7 commitsMetrics, logs, traces, burn-rate objectives and dashboards — which promptly found the outbox and reconciler defects — then the manifests in kind and the zero-downtime drill.
- feat(prom): scrape services and gateway, record red metrics and route alerts
- feat(loki): ship logs through alloy and traces to tempo within a label budget
- fix(api): keep broker outages and in-flight events from corrupting async state
- feat(grafana): add burn-rate slos and provision four dashboards as code
- feat(cd): reconcile the observability stack in-cluster from the same files
- fix(k8s): make a fresh cluster come up with a data stage and job egress
- test(load): prove zero-downtime rollouts with k6 and drain keep-alive first
Recovery, admission and the documents
6 – 9 August · 5 commitsPoint-in-time recovery and its drill, 503 on pool exhaustion, both drills weekly in CI, signature admission, and the architecture, security, operations and cost documents.
- feat(db): archive wal to object storage and prove point-in-time restores
- fix(api): answer pool exhaustion with 503 and run both drills weekly in ci
- feat(sec): enforce signed images at admission and scan rendered manifests
- docs: add architecture, security and operations guides, adrs and runbooks
- docs(cost): price the stack against managed equivalents and finish the readme
Not yet done #
Recorded where they belong, and gathered here:
- The first live deploy.
terraform apply, the playbook on a real node, Flux bootstrapping, Let’s Encrypt issuance, and object storage against Hetzner’s own endpoint — Operations. - Least-privilege database roles: an owner for Flyway, a DML-only role for the API,
pg_monitorfor the exporter — Security, known gaps. - Client-side backup encryption with a key stored off the cluster — ADR 0009.
- The k3s audit log shipped through Alloy, and the admission webhook’s own metrics scraped.
- A policy-controller release that reads bundle signatures, and then dropping the legacy signing flags from the build workflow.
- Error-and-slow-only gateway access logs, about three quarters less log volume, once traffic makes it worth it — Cost analysis.
- Scale-to-zero for a dev environment, if one is ever reconciled onto the node.