Skip to main content
Lexicon
Lexicon

STONITH (Shoot The Other Node In The Head)

Difficulty: Advanced
3 min read

In plain English

Plain definition

STONITH means Shoot The Other Node In The Head. A cluster uses this fencing mechanism to power off or isolate an unhealthy node before another node takes over, preventing both sides from writing to shared state during a split-brain failure.

STONITH stands for Shoot The Other Node In The Head

STONITH (an acronym for Shoot The Other Node In The Head) is a fencing technique used in high-availability clusters to prevent split-brain scenarios. When cluster nodes lose communication, STONITH actively powers off or isolates the unresponsive node to ensure data integrity and guarantee that only one primary node processes requests.

STONITH exists to solve one problem: a cluster manager cannot reliably distinguish a crashed node from a network-partitioned but still-running node. Soft fencing approaches (heartbeat timeouts, quorum votes) only tell the observer that a node is unreachable from where it stands; they say nothing about whether that node has stopped writing to disk or holding a lock. STONITH removes the ambiguity by acting on the node itself via an out-of-band control plane, usually IPMI/BMC, an intelligent PDU, a hypervisor API (vCenter, libvirt), or a cloud provider API (EC2 StopInstances, force-detach volume). Because the fencing action bypasses the node’s own OS and network stack, it works even when the node is unresponsive precisely because that stack has failed.

The rule STONITH enforces is at-most-one active writer to shared state (SAN LUNs, replicated block devices like DRBD, or exclusive leases). A cluster resource manager (Pacemaker, Corosync-based stacks, older RHCS) will not promote a resource on a surviving node until it receives positive confirmation from the fencing agent that the suspect node has been terminated or isolated. That confirmation step is a hazard in its own right: fencing agents can fail without any warning (BMC unreachable, IPMI credentials rotated, cloud API rate-limited), and a naive implementation that assumes success without an ack will produce split-brain, the exact failure STONITH was meant to prevent.

  • Self-fencing / suicide fencing (watchdog-based SBD, or Storage-Based Death, for example) has the suspect node fence itself after losing quorum or storage heartbeat, avoiding dependency on a remote fencing device but requiring a hardware or software watchdog timer as backstop.
  • Fencing races occur in symmetric partitions where both sides simultaneously attempt to fence each other; resolved via fencing delay asymmetry (pcmk_delay_base) or quorum-based tiebreaking so only the quorate side issues the shot.
  • Cloud-native equivalents replace physical power control with API-level isolation: revoking IAM instance-profile credentials, detaching an EBS/EFS mount, or cordoning a Kubernetes node combined with force-deleting stuck pods (though pod force-deletion without device fencing is unsafe for anything touching a ReadWriteOnce volume).

STONITH puts a hard dependency in the failover path: no fencing confirmation, no promotion. This is why cluster designs budget explicit fencing timeout SLAs separate from failure-detection timeouts, and why runbooks for HA storage clusters treat an unreachable fencing device as a Sev-1: the cluster is effectively unable to fail over safely, even if it appears healthy. Systems that skip STONITH in favour of pure quorum (relying solely on odd-node majority, for instance) still eventually need an equivalent fencing primitive whenever a resource is exclusive rather than quorum-replicated, since majority consensus alone doesn’t stop a stale minority node from writing to a shared, non-versioned device.

About this page

Credited to

Dates

Published

Corrections

No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

STONITH Fencing in HA Clusters: Preventing Split-Brain