Declarative delivery

The test of a delivery setup is not how fast it deploys. It is whether you can delete the cluster and get it back. If the answer involves anyone remembering anything, the answer is no.

Push and pull are different trust models

A push pipeline holds cluster credentials in CI and runs kubectl apply. A pull reconciler runs inside the cluster, watches Git, and applies what it finds. The second is usually better, for a reason that has nothing to do with features: CI never needs cluster credentials, so a compromised CI runner cannot reach production.

Pull also fixes drift. Something edited by hand at 3am gets reverted on the next reconcile, and you find out, instead of discovering months later that prod does not match the manifests.

Kustomize: patch, do not template

Kustomize keeps one real set of manifests and expresses each environment as a patch over it. The base is valid YAML you can apply directly, which makes it reviewable.

# base/kustomization.yaml
resources:
  - deployment.yaml
  - service.yaml

# overlays/prod/kustomization.yaml
resources:
  - ../../base
replicas:
  - name: my-app
    count: 6
patches:
  - path: resources.yaml       # prod gets bigger requests
    target: { kind: Deployment, name: my-app }
images:
  - name: my-app
    newTag: v1.4.2

# always read the output before trusting it
kubectl kustomize overlays/prod | less
kubectl diff -k overlays/prod                  # what would change, against the live cluster

Helm or Kustomize: pick by failure mode

• Helm when you are consuming somebody else’s software. You want their upgrade path, their values schema, their defaults.
• Kustomize when the manifests are yours. Patches stay readable; templated YAML stops being YAML and starts being a string-concatenation program.
• Both is normal and fine: render the upstream chart, then patch the result.

App-of-apps, and why bootstrap order matters

Put one root application in Git that points at everything else. Recovering the cluster becomes: install the reconciler, apply the root, wait.

# the only thing you ever apply by hand
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: root, namespace: argocd }
spec:
  source:
    repoURL: https://github.com/you/platform
    path: apps                  # a directory of Application manifests
  destination: { server: https://kubernetes.default.svc }
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Order is the other bootstrap trap: CRDs must exist before the resources that use them, and namespaces before the things inside them. Sync waves — or Flux dependencies — exist for exactly this, and you will need them the first time you rebuild from scratch.

What to actually do with this

• Run kubectl diff -k against production once. Whatever it prints is your current drift.
• Move one service to an overlay structure and delete its duplicated YAML.
• Write down the bootstrap order for your cluster. If you cannot, that is the finding.