Part 1: Kubernetes Boundaries and Failure Domains

#Part Purpose

This is Part 1 of a six-part Series on Kubernetes

The reader outcome for this part is narrow and deliberate: leave with a documented boundary map, a dependency and trust inventory, and failure-domain acceptance criteria for a Kubernetes cluster, before implementation decisions begin in Part 2 onward.

#Necessary Recap

As the opening part of this Series, there is no prior part to recap. This section instead fixes the running example and terminology used throughout the Series so later parts can refer back to it without repetition.

The running example is a single Kubernetes cluster whose control plane serves multiple namespaces, each mapped to an independent application team. The control plane comprises the API server, etcd, the scheduler and the controller-manager, as described in Kubernetes documentation. The node-side data plane comprises the kubelet and kube-proxy on each node, plus the container runtime. This terminology and example will be used unchanged in Parts 2 through 6.

#New Material

#

System Boundary and Responsibility Map

A boundary map answers one question for every component: who is accountable when it fails, and who is accountable for changing it. For the running example, documented Kubernetes architecture supports the following split of responsibility.

• Cluster-admin scope: the API server, etcd, scheduler, controller-manager, and node lifecycle (provisioning, kubelet configuration, container runtime). Kubernetes documentation describes these as control-plane components that coordinate overall cluster state.
• Namespace-owner scope: workloads (Deployments, StatefulSets, Jobs), namespace-scoped RBAC, and any resources created within their namespace boundary.
• Shared but asymmetric scope: NetworkPolicy and ResourceQuota objects are namespace-scoped by design, but their cluster-wide enforcement behaviour depends on cluster-admin-configured components such as the CNI plugin, per Kubernetes networking documentation.

Inference, not documented fact: treating node lifecycle as strictly cluster-admin scope is an operational convention rather than a Kubernetes-enforced rule; some organisations delegate node pool scaling to namespace owners via managed node groups. This part assumes the stricter split because it produces a cleaner failure-domain boundary for the remainder of the Series; teams with a different delegation model should adjust the acceptance criteria below accordingly.

#

Dependency and Trust Inventory

Kubernetes documentation states that etcd is the canonical store of cluster state and that the API server is the only component that reads and writes to it directly. This makes etcd the highest-trust dependency in the system: its availability and data integrity bound the availability and integrity of everything else. The dependency chain for the running example is:

• etcd — canonical state store; single point of cluster-wide trust.
• API server — sole intermediary to etcd; all other components, including kubectl clients, controllers and the scheduler, depend on it rather than on etcd directly.
• Scheduler and controller-manager — depend on the API server for reads and writes; they hold no independent state.
• kubelet and kube-proxy — depend on the API server for desired state, and on the local container runtime and node OS for execution; they cache last-known state locally, so brief API server unavailability does not immediately stop running workloads, per Kubernetes documentation on node components.

This chain matters because it defines where trust cannot be delegated without changing the security model: a component with etcd write access has cluster-wide control-plane-equivalent trust, so etcd access must remain restricted to the API server, consistent with the least-privilege posture expected of control-plane components.

#

Failure-Domain Assumptions

For this Series, a failure domain is the blast radius of a single fault: the set of things that stop working together when one component fails. Based on the dependency inventory above, this part fixes three failure domains for the running example.

• Control-plane domain: etcd quorum loss or API server unavailability. Documented Kubernetes behaviour is that existing pods keep running on healthy nodes during this outage, but no new scheduling, scaling or reconciliation occurs until the control plane recovers.
• Node domain: loss of a kubelet, container runtime or the node itself. This is contained to the workloads scheduled on that node; the control plane detects and reschedules affected workloads onto healthy nodes, subject to available capacity.
• Workload domain: a single application failing (crash loop, bad rollout). This is contained to the namespace and workload in question and should never cross into the node or control-plane domain if RBAC and resource limits are correctly scoped.

Assumption made visible: this three-domain model assumes a single-cluster, single-region deployment, as stated in the running example. Multi-cluster or multi-region designs introduce an additional cross-cluster failure domain that is out of scope for this part.

#

Failure-Domain Acceptance Criteria

Before any component in this Series is designed or recovered, it must satisfy three acceptance criteria fixed here: it must be assignable to exactly one of the three failure domains above; it must have a single named accountable owner drawn from the responsibility map; and its documented blast radius must not silently cross a domain boundary. Any component that fails these criteria should be treated as a boundary gap requiring explicit resolution before Part 2 proceeds, not as an implementation detail to be resolved later.

#Continuity Promises

Every later part in this Series will use the terminology, running example and three failure domains fixed here without redefining them. Part 2 will use this boundary map to scope which control-plane components it designs in detail; it will not repeat the responsibility split, dependency chain or failure-domain definitions established here. Any change to these assumptions in a later part will be stated explicitly as a revision, not applied silently.

#Validating the Boundary Map Against a Live Cluster

Before this boundary map is treated as accurate, it should be checked against a real cluster’s RBAC state rather than accepted as a paper exercise. A namespace-owner service account should never be able to act outside its declared domain. The following read-only checks confirm this without changing any cluster state.

1kubectl auth can-i list nodes --as=system:serviceaccount:checkout:default
2kubectl auth can-i get secrets -n kube-system --as=system:serviceaccount:checkout:default

Both commands should return no. A yes result on either is evidence of RBAC drift relative to the responsibility map above and should be logged as a deviation for the accountable owner to review before any further workload changes proceed, rather than remediated silently or treated as urgent enough to justify an unreviewed change.

One caveat when interpreting these checks: kubectl auth can-i ... --as impersonation results depend on the impersonating identity itself holding the impersonate verb on service accounts. If it does not, the command can return a misleadingly clean no without having validated anything. Confirm the impersonating identity’s own permissions first, or run the check as a genuinely privileged auditor account rather than relying on impersonation alone.

#Recording the Boundary Map for Future Parts

The responsibility map and dependency inventory should not remain a narrative document only; later parts in this Series, and any reviewer checking RBAC changes, need a form that can be diffed. A practical minimum is a version-controlled manifest listing each component, its accountable owner and its failure domain, reviewed under the same pull-request process as workload manifests:

1components:
2  - name: etcd
3    owner: cluster-admin
4    domain: control-plane
5  - name: api-server
6    owner: cluster-admin
7    domain: control-plane
8  - name: kubelet
9    owner: cluster-admin
10    domain: node
11  - name: team-checkout-deployment
12    owner: namespace-owner
13    domain: workload

Where a proposed RoleBinding grants a namespace owner permissions that reach into the node or control-plane domain, treat that as a finding requiring explicit sign-off rather than routine approval, since it crosses the boundary fixed in this part.

#Escalation and Rollback for Boundary Deviations

Not every RBAC deviation carries equal weight. A deviation confined to namespace-scoped resources within the workload domain warrants a standard review at the next scheduled RBAC audit. A deviation granting a namespace-owner identity any verb against nodes, secrets in kube-system, or any cluster-scoped resource should be escalated to the accountable cluster-admin owner within one working day, since it crosses directly into the control-plane or node domain fixed above. Where a RoleBinding or ClusterRoleBinding is found to cross a fixed boundary, the safe response is to revert the binding to its last known-good version-controlled state through the same pull-request process used to introduce it, rather than deleting it directly from a live cluster, which removes the audit trail. After reverting, re-run the two impersonation checks above to confirm the identity returns to no before closing the finding, and annotate the original change record with the reversal date and reviewer.

#Verified Outcomes and the Dependency Handed to Part 2

This part produced three durable artefacts for the Series: a cluster-admin/namespace-owner responsibility map, a dependency and trust inventory rooted in etcd as the highest-trust component, and a three-domain failure model (control-plane, node, workload) tied to documented Kubernetes behaviour. Acceptance criteria carried forward: before any component is redesigned or any recovery procedure is written later in this Series, it must be possible to state which of the three failure domains it belongs to and which responsibility owner is accountable for its failure and its repair. The exact dependency handed to Part 2 is this responsibility map and the etcd-centred dependency chain defined above, which Part 2 will use to scope its control-plane component design decisions without re-deriving them.