Missing etcd Encryption Configuration Leaves Kubernetes Secrets Recoverable in Backups

Symptom

A security audit or a routine etcd snapshot restore test reveals that every Secret’s data field decodes cleanly to its original plaintext credential. The cluster passed every kubectl-based access check, RBAC correctly restricted who could read Secrets through the API, and the team believed that was sufficient. The snapshot file itself, however, is effectively a plaintext credential store, and anyone who can read that file – a backup operator, a storage administrator, or an attacker with volume access – can recover every token, password and certificate key it contains without touching the Kubernetes API at all.

False Assumption

The working assumption was that Kubernetes Secrets are encrypted at rest by default, or that base64 encoding combined with RBAC restrictions on the Secrets API provides an equivalent guarantee. Those are two different controls operating at two different layers. RBAC governs who can call the API and receive a Secret object back; it says nothing about the form in which that object is stored on disk inside etcd, or in any backup, snapshot or replica taken from etcd’s data directory.

Root Cause

By default, kube-apiserver writes Secret objects into etcd without any encryption provider applied. Base64 is a reversible encoding, not a cipher, so the stored bytes decode directly back to the original value with a single command. Real encryption at rest only exists once an EncryptionConfiguration resource defines an actual provider – for example aescbc or a KMS-backed provider – for the secrets resource, and kube-apiserver is started with --encryption-provider-config pointing at that file. Without that flag, every Secret written to etcd, and every etcd snapshot, volume backup or peer replication stream taken afterwards, carries the credential in a form that is trivially reversed by anyone who can read the storage layer, independent of API-level RBAC.

Impact

Any actor who obtains etcd data – through a stale backup, a compromised storage volume, an unsecured etcd peer connection, or direct filesystem access to a control-plane node – can recover every Secret in the cluster regardless of the RBAC model layered on top. This includes service account tokens, TLS private keys and database credentials, so the exposure extends to every workload that consumes a Secret, not only the namespace where the gap was first noticed, and it persists retroactively inside any backup taken before the gap is closed.

Diagnosis

Confirm the exposure before changing anything.

1. Check whether the kube-apiserver static pod manifest or process arguments include --encryption-provider-config. Its absence means no encryption provider is active for any resource.
2. If the flag is present, open the referenced EncryptionConfiguration file and confirm the first provider listed for the secrets resource is not identity, which is an explicit no-op pass-through rather than encryption.
3. Using etcdctl with the cluster’s CA, certificate and key, read one known Secret key directly from etcd and base64-decode the stored value to confirm whether it matches the plaintext credential.

Correction

Enabling a real encryption provider closes the gap only for newly written objects, so existing Secrets must also be re-written to inherit the new protection.

1. Take a verified etcd snapshot backup before making any change and store it outside the cluster.
2. Create an EncryptionConfiguration that lists a real provider, such as aescbc, first for the secrets resource, keeping identity as a fallback entry further down the provider list during rollout.
3. Back up the current kube-apiserver static pod manifest, then add --encryption-provider-config pointing at the new configuration file.
4. Once the apiserver is confirmed Ready, force every existing Secret to be re-written under the new provider by reading and replacing each object across every namespace.

Validation

Validation must prove both that the apiserver stayed healthy and that stored Secrets actually changed form inside etcd.

• Confirm the kube-apiserver Pod remains Running and Ready, with no recent restarts, after the manifest change.
• Re-read the same Secret key directly from etcd with etcdctl and confirm the decoded value is no longer the recognisable plaintext credential.
• Spot-check Secrets in at least two namespaces that were not part of the initial test to confirm the re-write pass actually reached them.

Rollback

Rollback must remain possible at every stage of this change, not only at the end.

• If the apiserver fails to reach Ready after the manifest edit, restore the backed-up manifest immediately; kubelet redeploys the previous static pod automatically.
• If the bulk re-write pass causes workload-facing errors, restore the affected namespaces from the pre-change etcd snapshot rather than attempting to patch individual objects.
• Leave identity present in the provider list until the re-write pass and validation are both confirmed complete, so any object the pass missed stays readable instead of silently failing.

Prevention

Treat encryption-at-rest configuration as a mandatory provisioning check rather than an assumed default, and treat it as a separate control from RBAC that must be verified on its own.

• Add a provisioning-time check that fails cluster build if --encryption-provider-config is absent or resolves to an identity-only configuration.
• Repeat the etcdctl decode check from the Diagnosis section during periodic security review, not only at initial build time, so drift is caught after upgrades or manifest changes.
• Document explicitly, for anyone auditing the cluster, that RBAC restrictions on the Secrets API and etcd encryption at rest are independent controls, and that passing one says nothing about the other.