TOCTOU Race Conditions in Admission Webhooks

Admission control

This article dissects where the race sits inside the admission chain, how rules scoping and reinvocationPolicy interact to widen or close it, and what a production-grade mitigation strategy looks like when you can’t simply intercept every operation without destroying API server latency budgets.

#Where the TOCTOU Race Condition Actually Lives

The Kubernetes API server admission chain runs in a fixed order: authentication, authorisation, mutating webhooks, object schema validation, validating webhooks, and finally an optimistic-concurrency-controlled write to etcd using the object’s resourceVersion. Each webhook call is synchronous and stateless with respect to the rest of the chain — the webhook server receives a single AdmissionReview payload, returns an allow/deny decision, and has no further visibility into what happens to that object after the response leaves its process.

The TOCTOU race condition emerges from three structural properties of this design:

• Snapshot isolation per call. The webhook only ever sees the object as submitted in that specific request. If a validating webhook is scoped to CREATE only, an immediately following UPDATE or PATCH against the same object is invisible to it.
• Subresource exclusion. Many ValidatingWebhookConfiguration rule sets match on pods but omit pods/status, pods/ephemeralcontainers, or pods/resize. Fields writable through those subresources bypass the main webhook entirely, even though they mutate the live object.
• Non-transactional side effects. Policy engines that call out to external systems — an image signature verifier, a CMDB lookup, a rate limiter — perform that check against external state at time T0. If the external state changes before the object is actually scheduled or reconciled at T1, the decision is stale, and no re-validation occurs unless something explicitly triggers it.

#

Why reinvocationPolicy Doesn’t Close the Gap

Kubernetes offers reinvocationPolicy: IfNeeded to re-run mutating webhooks when a later mutating webhook changes the object again in the same request cycle. This only applies within a single admission chain execution for one request. It does nothing for a second, independent HTTP request arriving milliseconds later with a PATCH that targets fields your rules didn’t scope for. The TOCTOU race condition here is inter-request, not intra-request, and reinvocationPolicy was never designed to address it.

#Architectural Breakdown of the Attack Surface

Consider a cluster enforcing a Kyverno policy that blocks privileged: true containers on CREATE, matched via rules scoped strictly to CREATE operations on pods. This is an extremely common configuration pattern because scoping to CREATE alone reduces webhook call volume and latency overhead.

1apiVersion: admissionregistration.k8s.io/v1
2kind: ValidatingWebhookConfiguration
3metadata:
4  name: block-privileged-pods
5webhooks:
6  - name: block-privileged.kyverno.io
7    rules:
8      - apiGroups: [""]
9        apiVersions: ["v1"]
10        operations: ["CREATE"]
11        resources: ["pods"]
12        scope: "Namespaced"
13    failurePolicy: Fail
14    matchPolicy: Equivalent
15    sideEffects: None
16    timeoutSeconds: 5

An attacker with namespace-scoped create and patch permissions on pods — a common grant in shared-tenant clusters — submits a compliant, non-privileged pod. The CREATE request passes the webhook, and the pod object is persisted with a fresh resourceVersion. Because the webhook rules never include UPDATE, a follow-up PATCH targeting a mutable field in the container spec (mutable prior to pod scheduling, or via pods/ephemeralcontainers after scheduling) sails straight through with no admission review at all. The escalation window is the interval between the initial webhook decision and the point at which the kubelet actually acts on the container spec — typically tens to hundreds of milliseconds, but reliably exploitable with a scripted retry loop.

#

Step-by-Step Exploitation Logic

1. Submit a benign pod manifest that satisfies the CREATE-scoped policy.
2. Poll the API server for the assigned resourceVersion using kubectl get pod -o jsonpath='{.metadata.resourceVersion}'.
3. Immediately issue a strategic merge PATCH against a field outside the webhook’s operation scope — for example, adding an ephemeral debug container with elevated securityContext via the ephemeralcontainers subresource.
4. Confirm via kubectl exec that the container namespace has the escalated capability set applied, proving the check-time state diverged from use-time state.

1kubectl run race-poc --image=alpine --restart=Never -- sleep 3600
2RV=$(kubectl get pod race-poc -o jsonpath='{.metadata.resourceVersion}')
3kubectl patch pod race-poc --subresource=ephemeralcontainers --type=merge -p '
4{
5  "spec": {
6    "ephemeralContainers": [
7      {
8        "name": "debug-shell",
9        "image": "alpine",
10        "securityContext": {"privileged": true},
11        "command": ["sleep", "3600"]
12      }
13    ]
14  }
15}'

The ephemeralcontainers subresource frequently sits outside the scope of policies written against the parent pods resource, and unless explicitly listed under resources, this PATCH will not trigger the validating webhook at all — a direct, structural manifestation of the TOCTOU race condition.

#Closing the Race: Rule Scoping and Immutable Field Enforcement

The fix is not a single control but a layered set of admission configuration changes, applied consistently across every webhook that guards a security-relevant field. This is the same discipline you’d apply to any of the broader architectural patterns used for defence-in-depth: no single checkpoint is trusted to be sufficient on its own.

1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4  name: immutable-security-context
5spec:
6  validationFailureAction: Enforce
7  rules:
8    - name: block-privileged-any-operation
9      match:
10        any:
11          - resources:
12              kinds: ["Pod", "Pod/ephemeralcontainers", "Pod/status"]
13      validate:
14        message: "privileged containers and status mutations are forbidden"
15        pattern:
16          spec:
17            =(ephemeralContainers):
18              - =(securityContext):
19                  =(privileged): "false"

Three concrete measures close the majority of the exposure:

• Expand operations to include CREATE, UPDATE, and where relevant CONNECT, rather than optimising for call volume alone.
• Explicitly enumerate subresources (pods/status, pods/ephemeralcontainers, pods/resize) in the resources list rather than assuming the parent resource rule inherits coverage.
• Set matchPolicy: Equivalent so that requests hitting older or newer API group versions are still normalised into the version your webhook understands, closing a secondary version-skew race.

#Sequencing the Check-and-Commit Window

The following sequence illustrates the exploitable window between the validating webhook’s decision and the point where the attacker’s follow-up PATCH lands, when webhook rules are scoped only to CREATE.

#

Rule Scope Exposure Matrix

#Failure Modes and Edge Cases

Widening rule scope introduces its own failure modes. Setting failurePolicy: Fail on a webhook now covering UPDATE for a high-churn resource like pods/status means every kubelet status heartbeat is subject to webhook latency and availability. A webhook outage under failurePolicy: Fail will stall status propagation cluster-wide, which is a substantially worse operational outcome than the privilege-escalation window it was closing. Conversely, failurePolicy: Ignore reopens the exact same TOCTOU race condition during any webhook timeout or crash — an attacker who can induce backpressure

A second edge case involves namespaceSelector and objectSelector. If a policy is scoped via label selector to only apply to namespaces labelled tier=restricted, a request that concurrently removes that label via a separate RBAC-permitted call, then submits the sensitive object, may land outside the selector’s evaluation window depending on API server informer cache staleness — typically bounded by the default --watch-cache resync interval. This is a narrower but structurally identical race to the resourceVersion problem, and it’s rarely tested for because label-selector races require watching two independent controllers’ timing rather than a single object’s lifecycle.

Third, policy engines that call external services (SPIFFE

#Scaling and Security Trade-offs

Closing every instance of a TOCTOU race condition against admission webhooks has direct, measurable cost implications, and the trade-offs need to be made explicit to whoever owns the API server SLOs:

• Latency vs coverage — expanding rule scope to include UPDATE and subresources roughly doubles webhook call volume on high-churn resources; budget for this in timeoutSeconds and webhook replica count before rollout, not after.
• failurePolicy: Fail vs Ignore — Fail closes the race but converts webhook downtime into a cluster-wide outage risk; Ignore preserves availability but reopens the exact window this article describes during any webhook degradation.
• matchPolicy: Equivalent overhead — normalising across API versions adds CPU cost to the API server’s conversion webhook path; measure this against your actual API version skew before enabling it cluster-wide.
• External policy engine caching — caching signature or attestation lookups reduces external call latency but reintroduces a bounded TOCTOU race condition proportional to the cache TTL; a 30-second TTL trades a known, small exposure window for a large reduction in external dependency load.
• Reinvocation cost — setting reinvocationPolicy: IfNeeded on mutating webhooks adds a second execution pass per matching mutation, which is necessary for intra-request correctness but does not substitute for inter-request rule scoping fixes.

None of these controls eliminate the underlying property that admission control operates on snapshots rather than continuous state. The realistic engineering goal is shrinking the exploitable window to a size that matches your threat model — narrow enough that automated exploitation within a shared-tenant cluster becomes impractical, while keeping webhook latency and availability within the bounds your platform SLOs can actually sustain.