etcd's Backup Proxy Port Bypasses apiserver ACLs
A backup sidecar's etcd grpc-proxy listener on a non-default TCP port forwards writes straight into the datastore using its own embedded certificate, so nothing at the kube-apiserver layer ever sees, authenticates, or logs the request before it lands in etcd's key space.
At a glance
- Unsafe setting
- A backup or monitoring sidecar runs etcd's grpc-proxy on a non-default port without its own client certificate authentication.
- Failure trigger
- Any host reaching the proxy port issues raw etcd reads or writes that never pass through kube-apiserver.
- Blast radius
- Unauthenticated access to the full etcd key space bypasses RBAC, admission control and audit logging entirely.
- Recommended control
- Require client-cert-auth on every proxy listener using the control-plane CA and restrict the port to the backup workload's IP via NetworkPolicy.
The Trap
Running etcd’s built-in grpc-proxy as a sidecar for backup or monitoring tooling, bound to a non-default TCP port, without enforcing client certificate authentication on the proxy’s own listener.
The Default State
A kubeadm-provisioned control plane firewalls 2379 correctly and runs etcd with --client-cert-auth=true against the cluster CA. Operations teams then add a backup agent, snapshot exporter, or Velero-style etcd plugin that shells out to etcd grpc-proxy start --endpoints=https://127.0.0.1:2379 --listen-addr=0.0.0.0:12379, authenticating to the real etcd endpoint with its own stored client certificate. The proxy’s own listener is left with no --cert-file, --key-file, or --client-cert-auth flags, because the team assumes TLS between the proxy and etcd covers the whole chain. It does not. Authentication happens only between the proxy and the backend; anything that can reach port 12379 talks to the proxy as an already-authenticated client.
The Blast Radius
Any host reaching the proxy port gets full read/write access to the etcd key space with no certificate, no kube-apiserver RBAC check, no admission webhook, and no PodSecurity enforcement, because none of that logic lives in etcd. An attacker can write a Pod object directly under /registry/pods with a hostPath mount or privileged securityContext, and the kubelet will happily run it once kube-apiserver’s watch loop picks up the change on its next resync. There is no audit log entry for the write, because the request never passed through the API server’s audit chain. Secrets stored without an EncryptionConfiguration provider come back as plain base64, readable in a single etcdctl get call routed through the exposed port.
The Lead Mechanic Fix
Terminate any grpc-proxy or backup sidecar listener with the same certificate chain as the control plane: etcd grpc-proxy start --endpoints=https://127.0.0.1:2379 --listen-addr=127.0.0.1:12379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --client-cert-auth, and bind it to loopback or a dedicated CNI network only reachable by the backup Pod’s own IP via NetworkPolicy. Run ss -tlnp | grep etcd across every control-plane node on a schedule and alert on any listening port outside the documented set for 2379 and 2380.
Apply the safer control
Before you change production
Confirm the affected scope, export the current configuration, and test the replacement control in a non-production environment first.
Fix commands and configuration
etcd grpc-proxy start --endpoints=https://127.0.0.1:2379 --listen-addr=127.0.0.1:12379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --client-cert-authss -tlnp | grep etcdVerify, roll back or escalate
Verify
Re-run the detection test and a controlled negative-path test. Confirm the unsafe behaviour is blocked while approved traffic still succeeds.
Rollback
Restore the exported configuration if the new control blocks required production traffic, then narrow the policy before redeployment.
Escalate
Escalate when the blast radius is uncertain, the control cannot be tested safely, or remediation requires an outage or security exception.