Production Infrastructure BlueprintBlueprint Rev. 2026-08-09
  • 03 · Delivery & security

Delivery pipeline

CI never touches the cluster. Its last act is publishing a signed image; Flux, running inside the cluster, pulls the change from git and reconciles it in stages that each wait for the last to be healthy.

From commit to cluster #

The conventional answer to “how does a merge become a deploy” is a final CI step with a kubeconfig in a repository secret. That makes CI — which runs third-party actions and every transitive build dependency — the most privileged thing that can touch production. Here the cluster pulls instead, and the admission webhook refuses any image of ours that this repository’s build workflow did not sign (ADR 0003, ADR 0008).

How a change reaches the cluster A push runs the required checks and build.yml, which builds amd64 and arm64 natively, pushes both by digest to GHCR, scans the joined manifest and only then signs it and attaches an SBOM and provenance. Below a boundary CI never crosses, Flux image automation notices the new tag, commits it to git, and Flux reconciles the cluster in health-gated stages. The admission webhook admits the image only if build.yml in this repository signed it. GitHub Actions — publishes, then stops git push merge to main build.yml amd64 + arm64, native GHCR pushed by digest trivy the joined manifest cosign sign SBOM · SLSA provenance pass ci · security · drills — required before merge no kubeconfig · no cloud credential · no age key The cluster — pulls image automation polls GHCR · 5 min main tag committed Flux reconciles six stages, health-gated policy-controller signed by build.yml? pods new signed tag unsigned, or another workflow's: refused
Fig. 2 — An image that fails the scan is never signed, and an unsigned image is inert: admission requires a signature made by this repository's build.yml, on main or a release tag. It fails closed.

Four workflows #

Each workflow ends in a single job named after the workflow that fails if any upstream job did. Branch protection requires that one check, so adding a job never silently makes it optional.

WorkflowRuns onWhat it gates
ci.yml pull request, main Formatting, workflow and shell lint, tests against real Postgres, Redis and Kafka, per-service coverage thresholds, Terraform and Ansible validation, every manifest against the Kubernetes schemas, the alert unit tests.
security.yml pull request, main, Monday 05:00 UTC Secrets in the full history, semgrep with community rules and five of this repository’s own, hadolint, trivy config, checkov on the rendered manifests — and a weekly re-scan of the images already published.
build.yml pull request, main, tags Pull request: one architecture built, scanned and measured against a size budget, reported on the PR. Main: amd64 and arm64 built on native runners, joined, scanned, then signed and attested.
drills.yml on relevant change, main, Tuesday 04:00 UTC A point-in-time restore that must lose no row and keep none it should not, and a rollout of every API pod under load in a real cluster that must fail no request.

Every gate is an exit code. No scanner uploads SARIF into a dashboard: a finding that lands in a security tab and merges anyway is reporting, not a control.

Reconciliation order #

Ordering is data, not a deploy script. Two Flux Kustomizations must never own the same object, so each stage owns a disjoint set and expresses its place with dependsOn, wait: true and health checks. The split is what makes an expand/contract schema change safe rather than optimistic: the migration Job runs to completion before a single new pod starts.

  1. 01
    infrastructure-controllers

    Traefik, cert-manager, policy-controller

    HelmReleases whose CRDs everything after them needs.

  2. 02
    infrastructure-configs

    ACME ClusterIssuers, the image signature policy

    Cannot be applied until their controllers’ CRDs exist.

  3. 03
    apps-config

    Namespace, SOPS-decrypted Secrets, NetworkPolicies

    Separate, so a failed deploy cannot prune the credentials from under the database, and no pod runs before the policy that restricts it.

  4. 04
    apps-data prune: false

    Postgres, Redis, Kafka and its topics

    Healthy before the migration needs them. The one stage with prune: false — a path renamed by mistake must not become a deleted database.

  5. 05
    apps-migrations wait: true

    The Flyway Job, and nothing else

    Alone so wait: true can block on it. The rollout cannot start until the schema change has finished.

  6. 06
    apps

    api, worker — the rollout

    maxUnavailable 0, maxSurge 1, drain before SIGTERM.

observability — Prometheus, Alertmanager, Loki, Tempo, Alloy, Grafana. Nothing depends on it: an application deploy never waits for Grafana, and a broken monitoring upgrade can never block a fix.

Fig. 3 — Each stage waits for the previous one to be healthy, not merely applied. clusters/prod

Image tags are rewritten by Flux’s image automation and committed to main, so git log deploy/k8s/overlays/prod is the deploy history and a rollback is a revert (Roll back a bad deploy).

Properties worth knowing #

  • A change to one service does not rebuild the other. A path filter composes the test matrix; platform is shared, so touching it fans back out to both, and a worker-only change never starts the API’s Testcontainers stack.
  • Nothing is signed until it passes, and nothing unsigned runs. The image is pushed before it is scanned, because a multi-arch manifest cannot be assembled otherwise. The gate is the signature: an image that fails the scan is never signed, and admission requires one made by this repository’s workflow on main or a release tag.
  • arm64 is built natively. Under QEMU the same image takes over thirty minutes, because Gradle and jlink are CPU-bound; on a native runner it is the difference between a pipeline that runs on every push and one that gets disabled.
  • The drills run on a schedule, not only on change. Base images, the object store’s API and the kind node image all move without a commit here, and none of them needs one to break a drill.
  • Every input is pinned, and the pins move. Actions by 40-character SHA with a version comment, enforced by a script in CI and as a pre-commit hook; Renovate proposes upgrades, because a pin nobody moves becomes a known vulnerability.

Verifying it yourself #

shell
make verify-pins                                          # every action pinned to a commit SHA
make verify-image IMAGE=ghcr.io/tahirmohammedaman/blueprint-api:latest
make flux-status                                          # what the cluster thinks it is running

scripts/verify-image.sh checks the signature, the SBOM attestation and the build provenance against this repository’s workflow identity specifically. cosign verify without --certificate-identity passes for anything signed by any GitHub Actions workflow anywhere, which reads like a control and is not one; the cluster’s admission policy asserts the same identity string.

What CI never holds #

No workflow holds a kubeconfig, a cloud token, or the age key that decrypts the repository’s secrets. Workflows declare contents: read and the few jobs that write widen it for themselves; registry pushes use the job’s own token and signing uses its OIDC identity. A compromised CI run can publish an image. It cannot deploy one this workflow did not sign, and it cannot touch anything else.

The cost is stated in ADR 0003: a merge takes up to fifteen minutes to arrive, and a failed reconciliation is quieter than a failed pipeline — a condition on a resource, not a red build — which is why make flux-status is step two of incident triage. Infrastructure changes are applied by a person from a workstation, after reading the plan (Terraform).

The workflows, on GitHub

esc
↑ ↓ to move↵ to open/ or ⌘K to search