failurePolicy: Ignore Silently Skips Admission Checks

The Trap

failurePolicy: Ignore on a ValidatingWebhookConfiguration that enforces security policy, typically an OPA Gatekeeper or Kyverno deployment standing in for the retired PodSecurityPolicy admission chain.

The Default State

Helm charts for Gatekeeper and Kyverno, and most kubebuilder-scaffolded webhook operators, ship with failurePolicy set to Ignore out of the box. The rationale is bootstrap safety: if the webhook pod hasn’t started yet, cluster operators don’t want the API server refusing every kubectl apply. Teams install the chart, confirm policies block a test pod, and never revisit the field once the demo passes.

The Blast Radius

Ignore means any failure to reach the webhook, whether from a rolling update, a node drain, HPA scaling the deployment to zero replicas, a TLS certificate expiring on the webhook’s serving cert, or a CNI hiccup between the API server and the webhook’s ClusterIP, results in silent admission. The API server treats a connection timeout, a 5xx, or a refused connection identically: it lets the request through as though no ValidatingWebhookConfiguration existed. There is no admission-denied event, no annotation on the created object, nothing but a line in the audit log that most teams don’t query in real time. A privileged pod with hostPath /var/run/docker.sock, a container running as UID 0, or a Deployment missing resource limits can be created during a two-minute webhook restart window and will persist indefinitely once etcd has it, because nothing re-validates existing objects retroactively. On multi-tenant clusters this converts a transient outage into a permanent policy bypass for whatever landed during the gap.

The Lead Mechanic Fix

Set failurePolicy to Fail on every webhook that enforces a security boundary, then make the webhook itself resilient enough to justify that setting: kubectl patch validatingwebhookconfiguration gatekeeper-validating-webhook-configuration –type=’json’ -p='[{“op”:”replace”,”path”:”/webhooks/0/failurePolicy”,”value”:”Fail”}]’. Pair it with timeoutSeconds: 5, a namespaceSelector excluding kube-system so control-plane bootstrap can’t deadlock, at least two webhook replicas behind a PodDisruptionBudget with minAvailable: 1, and a readinessProbe that only reports ready once the policy cache is loaded, not just when the process starts. Run Gatekeeper’s audit controller in parallel as a compensating detective control, since it catches anything that slips through during a legitimate Fail-mode outage.