kubelet Anonymous Auth Left On Grants Unauthenticated Root-Level Node API Access

Symptom

A platform team notices that pods on certain nodes can be inspected, and in some cases have commands executed inside them, by clients that hold no Kubernetes credentials at all. Requests sent directly to a node’s kubelet HTTPS port (typically 10250) succeed without any bearer token, client certificate, or service account presented. Cluster RBAC audit logs show nothing, because the request never reaches the API server; kube-apiserver is not in the request path when a client talks to the kubelet directly.

False Assumption

The operating assumption was that Kubernetes RBAC, applied at the API server, was the single authorisation boundary for the cluster. Because namespace-scoped Roles and ClusterRoles had been carefully reviewed, the team assumed every path to workload data and control was mediated by that layer. This assumption is incorrect: the kubelet exposes its own HTTPS API directly on each node, and that API has independent authentication and authorisation settings that are not derived from, or enforced by, API server RBAC.

Root Cause

The root cause is that kubelet authentication was left at a permissive default: --anonymous-auth=true (or the equivalent authentication.anonymous.enabled: true in the kubelet configuration file), combined with an authorisation mode that is not Webhook. Per the Kubernetes documentation on security concepts, control plane and workload security depend on authentication and authorisation controls being deliberately configured at each control point, not assumed from a single central policy. When anonymous authentication is enabled on the kubelet and authorisation defers to AlwaysAllow or is otherwise not delegated back to the API server via the webhook authoriser, any client that can reach the kubelet’s port over the network is treated as an authenticated, authorised caller for that node’s kubelet API. This includes endpoints such as /pods, /logs, /exec and /run, which can expose workload metadata, container logs and, depending on kubelet version and endpoint, remote command execution inside containers scheduled on that node.

Impact

The practical impact is that node-level compromise or unauthenticated data exposure becomes possible for anyone with network reachability to the kubelet port, bypassing every RBAC Role and ClusterRoleBinding the team believes is protecting workloads. In a flat or under-segmented network (common in on-premises clusters, misconfigured cloud VPC rules, or clusters where node ports are reachable from a wider CI/build network), this converts a single missed kubelet flag into a cluster-wide confidentiality and integrity exposure: pod specs, environment-derived secrets exposed via logs, and potentially command execution in running containers.

Diagnosis

Diagnosis should be performed read-only and against a non-production or isolated cluster first. Confirm the kubelet’s effective authentication and authorisation configuration on a representative node, and confirm whether the anonymous endpoint actually responds without credentials.

1. Retrieve the running kubelet configuration to inspect authentication and authorisation settings for the node in question.
2. Check whether the API server’s authorisation mode includes Node and RBAC, and separately confirm the kubelet authorisation mode is Webhook rather than AlwaysAllow.
3. From a test host with only network reachability (no kubeconfig, no token) to the kubelet port, attempt an unauthenticated read against a non-sensitive endpoint to confirm whether the anonymous path is actually open, rather than assuming it from configuration alone.

Correction

The correction is to disable anonymous authentication on the kubelet and ensure the kubelet defers authorisation decisions to the API server via the webhook authoriser, so that the same RBAC boundary the team already reviews governs kubelet API access too.

• Set authentication.anonymous.enabled: false in the kubelet configuration (or --anonymous-auth=false for command-line-configured kubelets).
• Set authorization.mode: Webhook in the kubelet configuration so that authorisation decisions are delegated to the API server’s RBAC rules, keeping a single reviewed authorisation boundary.
• Restart the kubelet service on the affected node(s) after the configuration change, one node at a time, to observe workload impact before proceeding to the next node.

Validation

Validation must confirm both that legitimate, properly authenticated traffic still works and that the previously open anonymous path is now closed.

1. Repeat the unauthenticated request used during diagnosis against the corrected node’s kubelet port and confirm it now receives an authentication/authorisation rejection rather than a successful response.
2. Confirm that API-server-mediated operations that rely on the kubelet, such as kubectl logs and kubectl exec for a workload on that node, continue to succeed for a user who holds the appropriate RBAC grant.
3. Confirm the node reports Ready status and that existing pods on the node remain in their expected running state after the kubelet restart.

Rollback

If the corrected node fails validation, for example if legitimate kubelet-mediated operations stop working because the webhook authoriser is misconfigured or the API server’s Node authorisation mode is not correctly enabled, the rollback path is to restore the node’s prior kubelet configuration file (or command-line flags) from the pre-change backup taken before applying the correction, then restart the kubelet on that node only. Do not restore anonymous access as a permanent fix; treat any rollback as a temporary return to a known state while the webhook authoriser configuration on the API server side is investigated and corrected before reapplying the kubelet change.

Prevention

Treat kubelet authentication and authorisation settings as part of the same reviewed security boundary as API server RBAC, not a separate, lower-priority default. Include kubelet configuration (anonymous-auth and authorization-mode) in the same configuration review and drift-detection process used for RBAC bindings, and verify the setting on new nodes as they join the cluster rather than assuming a golden image remains correct indefinitely. Restrict network reachability to kubelet ports (10250 and related) to only the API server and authorised monitoring endpoints, as a defence-in-depth control that limits impact even if the authentication setting drifts again.