Runtime detection and response

Everything up to this point is prevention. Runtime detection assumes prevention failed, which over a long enough period it will. The question is whether you would notice.

Where the signal comes from

• Syscalls — eBPF or a kernel module watching what processes actually do. Falco and Tetragon live here. Highest fidelity, highest volume.
• The audit log — every request to the API server. The only record of who did what to the cluster.
• Network flows — who talked to whom. Hubble, or any flow log.
• Image and process inventory — what is running that was not running yesterday.

Rules worth having on day one

Falco ships hundreds of rules and the default set is noisy enough that people turn it off. Start with a handful that are nearly always true positives:

• A shell spawned inside a container that has no business running one.
• Writes below /etc or /usr/bin in a running container.
• A process reading /var/run/secrets/kubernetes.io/serviceaccount/token that is not your application.
• An outbound connection to an address outside your egress allowlist.
• Any attempt to mount /var/run/docker.sock or the container runtime socket.

- rule: Shell in a production container
  desc: interactive shell started where none should exist
  condition: >
    spawned_process and container
    and proc.name in (bash, sh, zsh, ash)
    and k8s.ns.name startswith "prod-"
  output: >
    shell in container (user=%user.name container=%container.name
    ns=%k8s.ns.name pod=%k8s.pod.name cmd=%proc.cmdline)
  priority: WARNING

Your audit log is your only witness

Syscall detection tells you what a container did. Only the API server audit log tells you what was done to the cluster — which token created that privileged pod, which identity read the Secret, what else that identity touched. It is off by default on many managed clusters and cannot be reconstructed afterwards.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse        # full bodies for the dangerous things
    resources:
      - group: ""
        resources: ["secrets", "serviceaccounts/token"]
      - group: "rbac.authorization.k8s.io"
        resources: ["*"]
  - level: Metadata               # who-did-what for everything else
    omitStages: ["RequestReceived"]

The next ten minutes, written down in advance

1. Contain, do not kill. Deleting the pod destroys the evidence and the Deployment recreates it anyway. Instead, cut it off.
2. Isolate with a policy. Label the pod and apply a NetworkPolicy that selects that label and allows nothing.
3. Detach it from its Service by changing a label the selector depends on — traffic stops, the pod keeps running, state is preserved.
4. Capture process list, open sockets, environment and the audit trail for its ServiceAccount.
5. Rotate every credential that pod could reach, which is every Secret in its namespace.
6. Then delete it.

# isolate, keeping the process alive for inspection
kubectl label pod suspect-xyz quarantine=true
kubectl label pod suspect-xyz app-                 # drop it out of the Service

# capture before anything else changes
kubectl exec suspect-xyz -- ps auxf > /tmp/ps.txt
kubectl exec suspect-xyz -- ss -tunap > /tmp/sockets.txt

That label-removal trick is the most useful one in the list. It takes the pod out of rotation instantly, the ReplicaSet notices the shortfall and starts a healthy replacement, and the suspect pod is left running and idle for you to look at.

What to actually do with this

• Deploy Falco with five rules rather than five hundred.
• Confirm today whether your audit log is enabled and where it lands.
• Write the six-step containment list somewhere findable, and rehearse the labelling once against a test pod.