Kubernetes Secrets Look Encrypted But etcd Stores Them in Plain Base64

Symptom

An operator restores an etcd snapshot backup to a test box for a disaster-recovery drill and, while inspecting the restored data files with a hex viewer, finds full plaintext-recoverable Kubernetes Secret values, including database passwords, sitting inside the etcd data directory. No cluster compromise occurred; RBAC on Secrets was configured correctly and no one had `kubectl get secret` access outside the intended namespace owners. Yet the raw storage layer exposed everything.

False Assumption

The team assumed that because Kubernetes stores Secret values as base64-encoded strings in the API and requires RBAC permissions to read Secret objects through kubectl or the API server, the underlying etcd storage was also protected. Base64 is an encoding, not encryption, and by itself provides no confidentiality. The team had never checked whether an EncryptionConfiguration resource was enabled on the API server, because the cluster had never needed one during normal operation — RBAC-gated API access felt sufficient.

Root Cause

By default, the Kubernetes API server does not encrypt Secret data before persisting it to etcd. Unless an administrator explicitly creates an EncryptionConfiguration file, references it with the --encryption-provider-config flag on kube-apiserver, and confirms every existing Secret has been rewritten under that provider, Secret objects are stored in etcd with only base64 encoding applied to their values. Base64 is trivially reversible with no key. Anyone who can read the etcd data directory, an etcd snapshot, or a volume/backup containing that data — independent of Kubernetes RBAC — can recover every Secret in the cluster. This is documented Kubernetes behaviour, not a defect: the platform explicitly requires opt-in configuration of encryption at rest for Secrets, per the Kubernetes security documentation.

Impact

Every Secret ever written to this cluster, including those in namespaces that were never directly accessed by the affected operator, is recoverable from any copy of the etcd data: live data directory, etcd snapshots, disk-level backups, or cloned persistent volumes backing etcd nodes. This includes Secrets created before the drill, meaning historical credentials that may still be in production use elsewhere are exposed through a channel that RBAC auditing on the Kubernetes API will never show as an access event, because the read happened outside the API server entirely.

Diagnosis

Confirm whether encryption at rest is active before assuming any exposure scope. These are read-only checks against the control plane and etcd; none of them modify cluster state.

Evidence to collect

• Whether an EncryptionConfiguration resource is referenced on every kube-apiserver instance.
• Which resources (if any) are covered by that configuration, and whether Secrets are included.
• Whether existing Secrets were rewritten under the encryption provider after it was enabled (enabling it only affects newly written objects, not historical ones, until a re-encryption pass is run).

Correction

Enable encryption at rest for Secrets using a supported provider (KMS-based providers are preferred over local secretbox/aescbc keys stored on disk, since a locally stored key sitting beside the same control plane recreates a similar exposure). The corrective sequence is: author an EncryptionConfiguration resource naming secrets as a covered resource, distribute it to every API server instance, add the --encryption-provider-config flag, restart each kube-apiserver one at a time behind a health check, then force re-encryption of all existing Secret objects — enabling the provider alone does not retroactively re-encrypt Secrets already in etcd.

Validation

Confirm encryption is active and covers historical data before considering the trap closed. Validation requires proving both that new writes are encrypted and that old Secrets have been rewritten, since the two are independently controlled.

Rollback

If a faulty or lost encryption key blocks the API server from starting, or if enabling encryption breaks Secret reads immediately after rollout, keep the previous EncryptionConfiguration (or its absence) available as a named, version-controlled rollback artefact before making any change, and roll back one API server instance at a time behind the same health check used for rollout.

Prevention

Treat encryption at rest for Secrets as a day-one control-plane requirement rather than an optional hardening step, verify it explicitly during any cluster build or migration checklist, and re-run the re-encryption pass after any encryption-provider or key rotation. Restrict and audit access to etcd data directories, snapshots and their backups with the same rigour applied to Kubernetes RBAC, since this trap demonstrates that RBAC alone does not bound the actual blast radius of a Secret.