Etcd Client Port Without Certificate Authentication Exposes Every Kubernetes Secret

Symptom

An internal audit of a self-managed Kubernetes cluster finds that raw Secret values were readable by a party who held no Kubernetes API credentials, no kubeconfig and no RBAC role bindings, yet who had network reachability to the control-plane’s etcd client port. Kubernetes RBAC, admission control and audit logging showed nothing unusual, because none of those systems ever saw the request.

False Assumption

The platform team assumed that hardening the Kubernetes API server — strong authentication, tight RBAC, restrictive admission policies — was sufficient to protect every object stored in the cluster, including Secrets. This assumption treats the API server as the only path into cluster state.

Root Cause

etcd is the canonical datastore behind the Kubernetes API server: every object, including Secret contents (stored base64-encoded rather than encrypted, unless encryption-at-rest is separately configured), is persisted there. etcd enforces its own authentication and authorisation model, entirely independent of Kubernetes RBAC. Kubernetes’ own security documentation describes the platform’s security model as spanning control plane, workloads, authentication, authorisation and policy controls as distinct layers, not a single unified boundary. Where the etcd static pod manifest sets --client-cert-auth=false, omits it, or the client URLs are bound to an interface reachable beyond the control-plane nodes (for example 0.0.0.0 rather than a loopback or control-plane-only address), any network client that can reach the client port can read and write directly against the datastore’s key-value API. RBAC role bindings, network policies scoped to the API server, and admission webhooks never observe this traffic because it bypasses the API server completely.

Impact

The exposure allows full read and write access to every object the cluster holds, including all Secrets across all namespaces, without generating any Kubernetes API audit event. An attacker or a misconfigured internal client with etcd reachability can exfiltrate credentials, tokens and TLS material, or silently modify controller state, while the API server’s own audit trail records nothing. Because the write path bypasses admission control, mutations made this way also skip validating and mutating webhooks that operators rely on for policy enforcement.

Diagnosis

Confirm the exposure before changing anything. On a control-plane node, inspect the etcd static pod manifest (commonly /etc/kubernetes/manifests/etcd.yaml on kubeadm-style clusters) for the --client-cert-auth, --listen-client-urls and --advertise-client-urls flags. Separately confirm which network segments can actually reach the etcd client port (2379 by default), since the flag alone does not establish exploitability — reachability does. Where the cluster is managed by a provider that abstracts etcd entirely, this specific trap does not apply in the same form, and the diagnosis should instead confirm that assumption against the provider’s documented control-plane boundary before proceeding.

• Verify etcd is running as a static pod or systemd-managed process on identified control-plane nodes.
• Read the current values of --client-cert-auth and the bound client URL address.
• Test, from an isolated non-production replica of the topology, whether a TLS handshake without a client certificate succeeds or is rejected.

Correction

The correction is to require mutual TLS on the etcd client port and bind it only to addresses reachable from trusted control-plane components. Set --client-cert-auth=true, ensure --listen-client-urls is restricted to control-plane-reachable addresses rather than a globally reachable interface, and confirm the certificate authority trusted by etcd matches the one used to issue the API server’s etcd client certificate. Apply this by editing the static pod manifest directly on each control-plane node; kubelet detects the change and recreates the etcd pod automatically. Take a timestamped backup of the manifest immediately before editing, in the same session, so the previous working configuration is available without reconstruction.

Validation

Validation confirms the fix rejects unauthenticated access while preserving normal cluster operation. After the etcd pod restarts, attempt a connection to the client port without presenting a client certificate and confirm the TLS handshake is refused. Separately confirm the API server itself still reaches etcd successfully — kubectl get nodes and kubectl get pods -A should return normal results with no elevated latency or errors. Review etcd’s own logs for authentication-related log lines consistent with the new enforcement, rather than unexpected client rejections that would indicate the API server’s own certificate is no longer trusted.

Rollback

Rollback is bounded to restoring the previous static pod manifest, because this change alters only how etcd authenticates client connections and does not touch the etcd data directory. If the API server loses connectivity to etcd after the change — the most likely failure — copy the pre-change backup manifest back into place immediately; kubelet will detect the restored file and recreate the etcd pod with the previous flags. Confirm recovery by repeating the same kubectl get nodes check used in validation. Do not delete or modify the etcd data directory at any point in this procedure; the described rollback only ever touches the manifest file, not stored cluster state.

Prevention

Treat etcd’s client and peer authentication settings as a control-plane hardening item distinct from Kubernetes RBAC, and verify them explicitly during cluster build-out and in periodic audits. CIS Kubernetes Benchmark checks and tools such as kube-bench cover this area and are a reasonable starting point for a documented review, though their specific check identifiers and current recommendations should be confirmed against the version in use rather than assumed. Restrict etcd client-port reachability at the network layer (firewall, security group or network policy scoped to control-plane nodes) in addition to certificate enforcement, so a single misconfigured flag does not become a network-reachable exposure on its own. Where a managed Kubernetes offering is in use, confirm in the provider’s documentation exactly where the etcd security boundary sits, since responsibility for this control may rest entirely with the provider rather than the cluster operator.