Workloads and configuration

Config, secrets and the twelve-factor edge cases

ConfigMap reload behaviour, projected volumes, and why your Secret is base64 and not encrypted.

CKADSecurity 8 min read

Configuration is the part of Kubernetes that appears finished after an hour and then produces a confusing incident six months later. Two behaviours cause most of them.

Your ConfigMap did not reload, and that is by design

How a ConfigMap reaches your container determines whether it can ever change:

  • As environment variables — resolved once, at container start. Updating the ConfigMap changes nothing until the pod is replaced. Ever.
  • As a mounted volume — the kubelet refreshes the files, eventually. Up to about a minute by default, and your process still has to notice and re-read them.
  • With subPath — never updates. This is the one that catches people.
A subPath mount is a one-time copy. It is extremely common, because it is how you put a single config file into a directory that already has other things in it — and it silently opts you out of all updates. If a file must be reloadable, mount the whole volume into its own directory and symlink, or accept that the pod has to restart.
# force a restart on config change: hash the config into the pod template
kubectl rollout restart deploy/my-app

# or, the declarative version - annotate with a checksum so the template itself changes
# annotations:
#   config/checksum: "{{ sha256sum of the ConfigMap }}"

That annotation trick is what Helm charts do, and it is worth copying even if you do not use Helm. It converts "config changed but nothing happened" into an ordinary rollout.

A Secret is not encrypted

Secret data is base64-encoded, which is an encoding, not a protection. Anyone who can read the Secret can read the value:

kubectl get secret db -o jsonpath='{.data.password}' | base64 -d

Two things follow. First, the real control is RBAC — who can get Secrets in that namespace, and who can create a pod that mounts them. Second, by default they are stored in etcd as plaintext, so an etcd backup is a credential dump. Encryption at rest is a cluster-level setting you have to turn on deliberately.

  • Prefer mounted files over environment variables: env vars leak into crash dumps, child processes and /proc/<pid>/environ.
  • Never kubectl describe a pod into a shared channel without checking what is in it.
  • For anything serious, keep the source of truth outside the cluster and sync it in — External Secrets Operator or the CSI secrets driver — so rotation happens in one place.

Projected volumes are underused

You can assemble one directory out of several sources, which keeps application code from caring where anything came from:

volumes:
  - name: config
    projected:
      sources:
        - configMap: { name: app-config }
        - secret:    { name: app-secrets }
        - serviceAccountToken:          # short-lived, audience-scoped
            path: token
            expirationSeconds: 3600
            audience: vault

That last source is the one worth knowing. A projected ServiceAccount token is time-limited and bound to an audience, so a leaked token expires and is only accepted by the service it was minted for. It is the basis of every sane workload-identity setup, and it replaces long-lived mounted tokens.

What to actually do with this

  • Find every subPath mount you own and decide whether it needs to be reloadable.
  • Move secrets from env to mounted files on your most sensitive service.
  • Check whether encryption at rest is enabled on your cluster. Assume it is not until you have looked.
This article covers one checkpoint on the roadmap. Open Config, secrets and the twelve-factor edge cases on the roadmap → — it lists what this depends on and everything else written about it.

Something wrong or out of date? Open an issue — corrections are welcome and get credited.