failurePolicy: Ignore Leaves Kubernetes Admission Webhooks Fail-Open

Symptom

A ValidatingWebhookConfiguration that is supposed to block privileged containers, disallowed image registries or missing security context settings appears correctly configured, yet non-compliant workloads are admitted with no denial event ever logged. The webhook object still shows the expected rules, namespaceSelector and service reference when inspected with kubectl, and the webhook’s own pod may even be running, but the specific request that should have been rejected simply goes through as if the webhook had approved it.

False Assumption

Platform teams commonly assume that the presence of a well-formed ValidatingWebhookConfiguration is proof that admission control is active: if the object exists, targets the correct apiGroups and resources, and the backing service resolves, the policy must be enforced. This assumption ignores the one field that decides what happens when the webhook cannot be reached: failurePolicy.

Root Cause

The webhook’s failurePolicy is set to Ignore, either explicitly or inherited from a Helm chart or manifest written against an older admission API default, or set to Ignore deliberately during initial rollout to avoid blocking cluster bootstrap and never revisited. With failurePolicy: Ignore, whenever the webhook backend cannot be reached, times out, or returns an error, the API server admits the request as though the webhook had approved it. Typical triggers include a crashed or scaled-down webhook pod, an expired or rotated TLS certificate that invalidates the caBundle, or an unrelated NetworkPolicy blocking the path from kube-apiserver to the webhook service. None of these failures produce a policy denial log, because the webhook never receives the request; the ValidatingWebhookConfiguration object itself continues to report as present and correctly scoped, which is exactly why the failure is misleading.

Impact

The practical effect is a silent, intermittent policy bypass rather than a hard outage. Workloads that violate the intended policy – privileged containers, unapproved base images, missing resource limits – can be admitted into namespaces the organisation believes are protected, and the bypass window is tied to webhook availability rather than to any configuration change. It can pass validation testing performed while the webhook was healthy and then regress the next time the webhook pod restarts, its certificate expires, or a network change interrupts connectivity. Because no error surfaces to the person applying the manifest, the gap is typically discovered during an audit, an incident review or a security scan rather than at deploy time.

Diagnosis

Confirm the failurePolicy value and webhook backend health before concluding this is the cause, using read-only checks only.

• Inspect the configured failurePolicy for every webhook entry.
• Confirm the webhook backend pod is running and has not restarted recently.
• Check webhook logs for TLS handshake or certificate errors.
• Confirm the caBundle field is non-empty and matches the current serving certificate.

kubectl get validatingwebhookconfiguration <name> -o jsonpath='{.webhooks[*].failurePolicy}'
kubectl get pods -n <webhook-namespace> -l app=<webhook-app-label> -o wide
kubectl logs -n <webhook-namespace> deploy/<webhook-deployment> --tail=100
kubectl get validatingwebhookconfiguration <name> -o jsonpath='{.webhooks[*].clientConfig.caBundle}' | head -c 40

If failurePolicy reports Ignore and the webhook pod shows recent restarts, certificate errors, or zero ready replicas at any point in its history, the fail-open condition described above is present.

Correction

Switch the affected webhook to failurePolicy: Fail once the underlying backend availability and certificate issues have been remediated, so an unreachable webhook blocks the request instead of silently admitting it.

kubectl patch validatingwebhookconfiguration <name> 
  --type='json' 
  -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Fail"}]'

Before applying this in a production cluster, confirm the webhook has adequate replica count, a PodDisruptionBudget, and a namespaceSelector that excludes only the specific control-plane or bootstrap namespaces the webhook genuinely cannot police, rather than a broad exemption pattern. Fail-closed enforcement without sufficient webhook capacity converts an invisible security gap into a visible availability risk, which is why this change is state-changing and requires the validation and rollback steps below.

Validation

Confirm the corrected failurePolicy actually blocks a known-bad request before treating the change as complete.

• Apply a known-noncompliant test manifest, such as a privileged container, into a disposable test namespace after the patch. Expected evidence: kubectl apply returns an admission denial identifying the webhook. Pass condition: the object is rejected and no pod is created.
• In a non-production cluster, scale the webhook Deployment to zero replicas and repeat the same test apply. Expected evidence: the apply fails with a webhook connection or timeout error rather than succeeding. Pass condition: the request is blocked while the webhook is unreachable, confirming fail-closed behaviour is active.
• Monitor kube-apiserver logs for webhook call latency and error rate for at least 24 hours after the change. Expected evidence: no sustained increase in apiserver latency attributable to the webhook. Pass condition: latency stays within prior bounds and no legitimate workloads are unexpectedly denied.

Rollback

Revert failurePolicy to its previous value immediately if the Fail setting blocks legitimate deployments that cannot wait for a scoping fix.

kubectl patch validatingwebhookconfiguration <name> 
  --type='json' 
  -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

Stop condition: if switching to Fail blocks deployments in kube-system or any namespace required to keep the cluster operable, roll back without waiting for further diagnosis. Rolling back restores the original fail-open exposure described in Root Cause, so treat the rollback as temporary: open a tracked remediation item to fix webhook capacity or narrow the namespaceSelector, then reapply Fail once the blocking condition is resolved.

Prevention

Set failurePolicy: Fail as the default for any webhook enforcing a security-material policy, and use a narrowly scoped namespaceSelector limited to the specific namespaces the webhook cannot safely police, instead of a broad Ignore fallback. Run the webhook backend with multiple replicas and a PodDisruptionBudget so a single pod restart cannot create a fail-open window, and add alerting on webhook pod readiness and TLS certificate expiry so the availability gap is caught before it becomes a silent policy bypass. When reviewing Helm chart upgrades or third-party admission controllers, explicitly check the shipped failurePolicy default rather than trusting the chart’s historical behaviour, since defaults have changed across admission API versions and vendors do not always update it on upgrade. Include the known-bad test manifest from the Validation section in a recurring non-production check so a regression is caught on a schedule rather than during an incident review.