- 06 · Operations & cost
Measurements
Every footprint and result in one place, with where it was measured and how. The documents each figure comes from have the full method; this page is for reading them side by side.
Method #
Two kinds of number appear in this repository and are kept apart. Footprints — image size, memory, start-up time, series, log bytes, backup sizes, failed requests — were measured on this system, and the method is given so each can be reproduced or disputed. Prices are published list prices with sources, in the cost analysis. Kubernetes requests and limits cite the measurement they were derived from, never a round number.
Container image #
| Approach | Base image | Size | vs. naive |
|---|---|---|---|
| Fat JAR on a JDK base | eclipse-temurin:21-jdk-noble | 523 MB | — |
| Fat JAR on a JRE base | eclipse-temurin:21-jre-alpine | 279 MB | −47% |
| Layered JAR, jlink runtime — shipped | distroless/java-base-debian12:nonroot | 170 MB | −67% |
Where the shipped image's 170 MB goes
MB per layer, API image · the application layer is too small to see, which is the point
- dependencies, /app/lib 69 MB
- jlink runtime 65 MB
- distroless base 36 MB
- application, /app/app.jar 33 KB
A typical deploy changes application code and nothing else, so it pushes and pulls 33 KB, not 170 MB. The worker image, from the same Dockerfile, is 164 MB; the API grew to 200 MB once the Redis, Kafka and JPA dependencies arrived. The jlink runtime is 63 MB against the 345 MB JDK it was cut from, with a module list curated by hand and certified by the smoke test, because jdeps cannot see reflection or driver discovery.
Startup and CDS #
| Spring context refresh | Run 1 | Run 2 | Run 3 | Mean |
|---|---|---|---|---|
| Without CDS | 4.78 s | 4.93 s | 4.99 s | 4.90 s |
| With a CDS archive | 3.55 s | 3.69 s | 3.73 s | 3.66 s |
−25% start-up for +88 MB of image. CDS is a separate target, make image-cds, rather than
the default: for pods that live for days, a second per deploy does not pay for a 52% larger image. It flips when
start-up is on the critical path — aggressive autoscaling, or scale-to-zero.
Memory against limits #
Application containers: resident memory against the limit
podman stats after the smoke-test workload · the bar is utilisation of the limit
- api 433 MB / 640 MiB · 68%
- worker 311 MB / 448 MiB · 66%
- kafka 326 MB / 1400 MiB · 22%
- postgres 29 MB / 512 MiB · 5%
- gateway 29 MB / 128 MiB · 22%
- redis 7 MB / 128 MiB · 5%
- The API limit went from 512 to 640 MiB because adding the Redis and Kafka clients put the old limit at 81% — no headroom for a GC pause or a burst. The number said so.
- Kafka's limit is generous against its 22%, deliberately: its real consumer is the page cache holding recent segments, which never shows in RSS.
- No
-Xmxanywhere. Heap isMaxRAMPercentage=70of the cgroup limit, so the limit is the one place memory is tuned; the rest is metaspace, thread stacks, code cache and direct buffers.
Observability stack: resident memory against the limit
about 590 MB for all six — roughly one API replica
- grafana 178 MB / 256 MiB · 66%
- alloy 115 MB / 192 MiB · 57%
- tempo 110 MB / 256 MiB · 41%
- loki 86 MB / 256 MiB · 32%
- prometheus 77 MB / 384 MiB · 19%
- alertmanager 21 MB / 64 MiB · 31%
What fits on the node #
Memory requested on the CAX21
GiB of requests against 8 GB of RAM · k3s and Flux's controllers are the unshaded remainder's first tenants
- application 2.7 Gi
- observability 0.7 Gi
- Traefik, cert-manager, policy-controller 0.27 Gi
- not requested 3.8 Gi
3.7 Gi requested, 4.7 Gi with the API at its HPA ceiling of four replicas. Limits add up to 6.1 Gi — more than is comfortable, deliberately: they are ceilings a container is killed at, not reservations, and the thing that would make them peak together is a load spike, which the HPA and the objectives exist to catch. The full table, and the Postgres connection budget, are in Operations.
What observing it costs #
| Scrape target | Samples per scrape | Budget, sample_limit |
|---|---|---|
| api | 412 (was 1,025) | 1,500 |
| worker | 543 | 1,200 |
| gateway | 344 | 1,000 |
| postgres exporter, in the cluster | 224 (default 651) | 400 |
| grafana, after dropping its own series | 8 of ~3,700 | — |
| Logs, per request, 9,051 requests of steady load | Bytes | Share |
|---|---|---|
| All services — 1.48 lines | 1,101 | |
| gateway access log | 809 | 76% |
| api | 157 | 15% |
| worker | 94 | 9% |
Loki's chunks compressed 4.4 to 1: thirty days at a million requests a day is about 7.5 GB, inside the 1 TB the object storage price already includes. Before the span filter, 176 of 198 stored traces were scheduled tasks and probes.
Load, rollouts and restores #
| Run | Where | Requests | Failed | Notes |
|---|---|---|---|---|
make load, two minutes | Compose | 7,251 | 0 | p99 37 ms through the gateway |
| Steady k6 profile, 50 req/s, three minutes | Compose | 9,051 | 0 | p99 38 ms; the log measurement above |
Rollouts, preStop sleep | kind | 11,972 | 4 | all POSTs, keep-alive closes crossing requests |
| Rollouts, drain before SIGTERM | kind | 11,970 | 0 | identical load: 40 req/s, half writes, three rollouts |
make drill, weekly | kind, CI | 6,001 | 0 | every API pod replaced under load |
| Kafka stopped for 150 s under load | Compose | 15 writes | 0 | no attempts spent; drained 12 s after restart |
| Point-in-time restore | 30,000 rows, 27 MB | 300,000 rows, 205 MB |
|---|---|---|
| Base backup, taken and uploaded | 2.0 s | 4.0 s |
| Base backup in object storage | 5.3 MB | 15.2 MB |
| WAL archived by the drill | 11 MB | 80 MB |
| Fetch and verify the base backup | 1.6 s | 2.6 s |
| Until promoted and writable | 5.6 s | 10.6 s |
| A timed-out WAL segment, stored | 16 KB of 16 MB | 16 KB of 16 MB |
Restore time grows with the WAL to replay far more than with the base backup — which is the argument for a daily base backup rather than a weekly one. The runbook itself, run command by command against the production manifests in kind, took 12 seconds from stop to promoted for a 27 MB database (Restore the database).
Still to be measured #
- An invoice. Every price is a list price; the first month's bill replaces them.
- Loki and the WAL archive against Hetzner's own object storage endpoint, including request latency from the node.
- Kafka's page-cache working set under sustained produce load — whether 1.4 GB is generous or merely adequate.
- Registry storage and egress under the real deploy cadence, where the 33 KB layer should show.
- Backup size and restore time on data that compresses like real data rather than repeated strings.