How a Kubernetes RBAC Hardening Change Can Leave Broad Access Untouched

Symptom

A Kubernetes RBAC “hardening” change appears to restrict a CI service account, but a direct capability check shows the account can still read Secrets far beyond the namespace the change was meant to confine it to.

In the reported scenario, the platform team replaced a broad ClusterRoleBinding for the service account ci-deployer (namespace ci) with a new, narrowly scoped Role and RoleBinding granting only get, list and watch on ConfigMaps in that namespace. The change was reviewed, applied, and the deployment was updated to use the “hardened” identity. A post-change audit using kubectl auth can-i still reported that ci-deployer could list Secrets across every namespace in the cluster.

False Assumption

The change relied on the assumption that binding a subject to a new, narrower Role replaces or overrides any broader permissions the same subject held through an older binding.

Kubernetes RBAC has no concept of override, priority or explicit deny. Authorization is decided by taking the union of every rule in every Role or ClusterRole referenced by any RoleBinding or ClusterRoleBinding whose subjects match the requester — including group subjects the requester belongs to automatically, such as system:serviceaccounts:<namespace>. Adding a tighter binding never removes permissions granted elsewhere; it only adds to them.

Root Cause

An earlier, still-active ClusterRoleBinding granted broad Secrets access by binding a secrets-reader ClusterRole to the group system:serviceaccounts:ci rather than to a specific service account.

Every service account created in the ci namespace, including ci-deployer, is automatically a member of that group. The hardening change added a new, minimal Role/RoleBinding pair for ci-deployer but never inspected or removed the older group-scoped ClusterRoleBinding. The service account’s effective permissions after the change were therefore the union of both bindings: the intended ConfigMap access plus the pre-existing, unreviewed cluster-wide Secrets access.

Impact

Any workload running as ci-deployer retained the ability to read Secrets in namespaces it should never reach, so the hardening change produced no real reduction in blast radius despite passing review.

The risk is compounded because the change looked complete: a new Role existed, a new RoleBinding existed, and the deployment manifest referenced the new identity. Nothing in the change itself signalled that an older, broader grant was still live. A reviewer checking only the newly added objects would sign off on a change that left the original exposure untouched.

Diagnosis

Confirm the account’s actual effective permissions rather than the permissions implied by the newest manifest.

kubectl auth can-i list secrets --all-namespaces --as=system:serviceaccount:ci:ci-deployer

A “yes” result here, immediately after a change intended to remove Secrets access, is the direct evidence of the trap. Next, enumerate every binding that references the subject or a group it belongs to:

kubectl get clusterrolebindings,rolebindings --all-namespaces -o json | jq '.items[] | select(.subjects[]? | (.name=="ci-deployer") or (.name=="system:serviceaccounts:ci"))'

Inspect each matched binding’s referenced Role or ClusterRole with kubectl describe clusterrole secrets-reader and kubectl describe role ci-deployer-scoped -n ci to see exactly which rules each binding contributes to the union.

Correction

Remove the over-broad group-scoped binding rather than layering further restrictions on top of it, then re-grant Secrets access only to the specific identities that genuinely require it.

Take an evidence snapshot before changing anything:

kubectl get clusterrolebinding ci-secrets-reader-binding -o yaml > ci-secrets-reader-binding.backup.yaml

Confirm which workloads, if any, in the ci namespace legitimately depend on the broad grant before deleting it; where a genuine need exists, replace it with a per-service-account RoleBinding scoped to the specific namespace and resource, never with a namespace-wide group subject for a sensitive resource. Once no legitimate dependency remains unaddressed:

kubectl delete clusterrolebinding ci-secrets-reader-binding

This is a state-changing, cluster-scoped removal: apply it only in an isolated or non-production validation environment first, with the backup file retained and the affected namespace’s workloads monitored immediately afterward.

Validation

Re-run the same capability check that first exposed the trap and confirm it now returns the expected denial.

kubectl auth can-i list secrets --all-namespaces --as=system:serviceaccount:ci:ci-deployer

Expect “no”. Then re-enumerate bindings for the subject and its groups to confirm only the intended, minimal bindings remain, and check the logs of every workload that previously depended on the removed binding for authorization errors during a full deployment cycle in the validation environment before considering the change safe to promote.

Rollback

If a legitimate workload loses required Secrets access after the broad binding is removed, restore the backed-up object immediately rather than attempting a partial fix under pressure.

kubectl apply -f ci-secrets-reader-binding.backup.yaml

Treat the restore as a stop condition, not a resolution: once access is restored, identify exactly which workload needed it, grant that workload a dedicated, minimally scoped RoleBinding, and only then re-attempt removal of the group-scoped binding. Do not leave the broad binding in place indefinitely as a substitute for a scoped one.

Prevention

Before treating any RBAC change as a restriction, enumerate every binding — direct and group-based — that already applies to the subject, and require the change to remove or narrow existing over-broad grants rather than only adding new ones.

• Avoid binding sensitive ClusterRoles to namespace-wide groups such as system:serviceaccounts:<namespace>; bind to specific service accounts instead.
• Add a pre- and post-change capability check (kubectl auth can-i) for the affected subject as a mandatory step in the change record, not an optional audit.
• Periodically run an automated binding audit across the cluster to surface subjects whose effective permissions come from more than one binding, so hardening work targets every contributing grant.