STONITH (Shoot The Other Node In The Head)

STONITH is a hard fencing mechanism in high-availability clusters that forcibly powers off, reboots, or isolates a node presumed to be failed or partitioned, guaranteeing it cannot access shared resources before a peer resumes its role. It exists to convert an ambiguous 'suspected dead' state into a deterministic 'confirmed dead' state, which is the only safe precondition for automated failover onto shared storage or exclusive resources.

STONITH is the resolution strategy for the fundamental problem that a cluster manager cannot reliably distinguish a crashed node from a network-partitioned but still-running node. Soft fencing approaches (heartbeat timeouts, quorum votes) only tell you a node is unreachable from the perspective of the observer; they say nothing about whether that node has stopped writing to disk or holding a lock. STONITH removes the ambiguity by acting on the node itself via an out-of-band control plane, typically IPMI/BMC, an intelligent PDU, a hypervisor API (vCenter, libvirt), or a cloud provider API (EC2 StopInstances, force-detach volume). Because the fencing action bypasses the node’s own OS and network stack, it works even when the node is unresponsive precisely because that stack has failed.

The critical invariant STONITH enforces is at-most-one active writer to shared state (SAN LUNs, replicated block devices like DRBD, or exclusive leases). A cluster resource manager (Pacemaker, Corosync-based stacks, older RHCS) will not promote a resource on a surviving node until it receives positive confirmation from the fencing agent that the suspect node has been terminated or isolated. This confirmation is itself a distributed systems hazard: fencing agents can silently fail (BMC unreachable, IPMI credentials rotated, cloud API rate-limited), and a naive implementation that assumes success without an ack will produce split-brain — the exact failure STONITH was meant to prevent.

Self-fencing / suicide fencing (e.g. watchdog-based SBD — Storage-Based Death) has the suspect node fence itself after losing quorum or storage heartbeat, avoiding dependency on a remote fencing device but requiring a hardware or software watchdog timer as backstop.
Fencing races occur in symmetric partitions where both sides simultaneously attempt to fence each other; resolved via fencing delay asymmetry (pcmk_delay_base) or quorum-based tiebreaking so only the quorate side issues the shot.
Cloud-native equivalents replace physical power control with API-level isolation: revoking IAM instance-profile credentials, detaching an EBS/EFS mount, or cordoning a Kubernetes node combined with force-deleting stuck pods (though pod force-deletion without device fencing is unsafe for anything touching a ReadWriteOnce volume).

Architecturally, STONITH pushes a hard dependency into the failover path: no fencing confirmation, no promotion. This is why cluster designs budget explicit fencing timeout SLAs separate from failure-detection timeouts, and why runbooks for HA storage clusters treat an unreachable fencing device as a Sev-1 — the cluster is effectively unable to fail over safely, even if it appears healthy. Systems that skip STONITH in favor of pure quorum (e.g., relying solely on odd-node majority) still eventually need an equivalent fencing primitive whenever a resource is exclusive rather than quorum-replicated, since majority consensus alone doesn’t stop a stale minority node from writing to a shared, non-versioned device.