Zero-Downtime Node Draining in EKS Spot Fleets

A c5.4xlarge spot node hosting 40 pods gets a 120-second interruption notice at 03:14 UTC. The default Kubernetes graceful termination window is 30 seconds. If your control plane relies on the standard kubectl drain workflow triggered manually, or worse, waits for the node to simply vanish from the API server, you are looking at abrupt pod eviction, connection resets on in-flight gRPC streams, and a cascading retry storm on downstream services. This is the core failure mode that proper spot interruption handling is designed to eliminate. At scale — clusters running 200+ spot nodes across mixed instance pools — the probability of a multi-node simultaneous reclaim event on any given day is non-trivial, and the blast radius of an unhandled interruption grows linearly with pod density per node.

This article details the architecture for automating spot interruption handling in Amazon EKS using the AWS Node Termination Handler (NTH), EventBridge-driven interruption notices, and lifecycle hooks on the underlying Auto Scaling Group, with the explicit goal of achieving zero-downtime node draining rather than best-effort cleanup.

#The Problem Statement: Why kubelet Alone Is Insufficient

EC2 spot capacity is reclaimed with a two-minute interruption notice delivered via the Instance Metadata Service (IMDS) at /latest/meta-data/spot/instance-action, and separately broadcast through EventBridge as a EC2 Spot Interruption Warning event. The kubelet has no native awareness of either signal. Left unmanaged, the sequence of events is:

• AWS reclaims the underlying host at T+120s with no further warning.
• The node transitions to NotReady only after the kubelet’s node-lease heartbeat expires, which is governed by node-monitor-grace-period (default 40 seconds).
• The Kubernetes scheduler only reschedules pods once the node is marked NotReady and the pod eviction timeout (default 5 minutes) elapses, or the node is force-deleted.

The arithmetic does not work. A 120-second warning followed by a multi-minute detection-and-eviction cycle guarantees that workloads are hard-killed mid-request rather than gracefully rescheduled. Zero-downtime draining requires intercepting the interruption signal before kubelet-level detection kicks in, and driving the drain from outside the standard node lifecycle loop.

#Architectural Breakdown

The correct architecture decouples signal detection from drain execution, using two complementary paths so that spot interruption handling functions even if one signal source is delayed:

• IMDS polling path (in-cluster, per-node): The AWS Node Termination Handler runs as a DaemonSet, polling http://169.254.169.254/latest/meta-data/spot/instance-action from within each node at a configurable interval (default 2s). On receiving a 200 response, NTH immediately cordons and drains the local node.
• EventBridge/SQS path (cluster-wide, queue-processor mode): A separate NTH deployment runs in Queue Processor mode, subscribed to an SQS queue fed by EventBridge rules matching EC2 Spot Interruption Warning, EC2 Instance Rebalance Recommendation, and Auto Scaling lifecycle termination events. This path handles interruptions even for nodes where local IMDS polling might be delayed by CPU starvation, and it also catches rebalance recommendations — a proactive signal issued before the two-minute notice, giving substantially more lead time.

Both paths converge on the same drain logic: cordon the node, evict pods respecting PodDisruptionBudgets, and signal the ASG lifecycle hook to complete only once draining is verified. This last step is what separates genuine zero-downtime handling from a fire-and-forget SIGTERM broadcast — the instance is not permitted to actually terminate until Kubernetes has finished migrating workloads.

#Why Rebalance Recommendations Matter More Than the Interruption Notice

The two-minute interruption notice is the last signal AWS gives you — by the time it fires, capacity reclaim is already committed. The rebalance recommendation, issued when AWS’s internal capacity models predict elevated interruption risk for a specific instance pool, arrives earlier and is non-committal — the instance may not actually be reclaimed. Treating rebalance recommendations as a proactive drain trigger, rather than waiting for the hard interruption notice, is what pushes your spot interruption handling pipeline from reactive to predictive.

#Implementation Logic

The rollout sequence for production-grade spot interruption handling on EKS follows this order:

1. Provision an SQS queue and EventBridge rules targeting Spot Interruption Warning, Rebalance Recommendation, and ASG Instance-Terminate Lifecycle Action events.
2. Attach an ASG lifecycle hook on autoscaling:EC2_INSTANCE_TERMINATING with a heartbeat timeout that exceeds your longest pod termination grace period (recommend 300–600s for stateful workloads).
3. Deploy NTH in IMDS mode as a DaemonSet with host network access, and NTH in Queue Processor mode as a Deployment with an IAM role scoped to the SQS queue and ASG APIs.
4. Define PodDisruptionBudgets on every Deployment/StatefulSet that must survive a drain without dropping below minimum available replicas.
5. Configure terminationGracePeriodSeconds per workload to match realistic connection-draining time — not the Kubernetes default of 30s for anything behind a load balancer with long-lived connections.
6. Validate with a chaos test that manually publishes a synthetic interruption event to the SQS queue and confirms pod rescheduling completes before the lifecycle hook heartbeat expires.

#Code and Configurations

#1. EventBridge Rule and SQS Queue (Terraform)

1resource "aws_sqs_queue" "nth_interruption_queue" {
2  name                      = "eks-spot-interruption-queue"
3  message_retention_seconds = 300
4  sqs_managed_sse_enabled   = true
5}
6
7resource "aws_cloudwatch_event_rule" "spot_interruption" {
8  name = "capture-spot-interruption"
9  event_pattern = jsonencode({
10    source      = ["aws.ec2"]
11    detail-type = [
12      "EC2 Spot Instance Interruption Warning",
13      "EC2 Instance Rebalance Recommendation",
14      "EC2 Instance State-change Notification"
15    ]
16  })
17}
18
19resource "aws_cloudwatch_event_target" "to_sqs" {
20  rule      = aws_cloudwatch_event_rule.spot_interruption.name
21  target_id = "send-to-sqs"
22  arn       = aws_sqs_queue.nth_interruption_queue.arn
23}
24
25resource "aws_autoscaling_lifecycle_hook" "terminating" {
26  name                   = "nth-terminating-hook"
27  autoscaling_group_name = aws_autoscaling_group.spot_pool.name
28  lifecycle_transition   = "autoscaling:EC2_INSTANCE_TERMINATING"
29  heartbeat_timeout      = 300
30  default_result         = "CONTINUE"
31}

#2. NTH Queue Processor Deployment (Kubernetes manifest)

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: node-termination-handler-queue
5  namespace: kube-system
6spec:
7  replicas: 1
8  selector:
9    matchLabels:
10      app: nth-queue-processor
11  template:
12    metadata:
13      labels:
14        app: nth-queue-processor
15    spec:
16      serviceAccountName: nth-irsa
17      containers:
18        - name: nth
19          image: public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.22.0
20          env:
21            - name: ENABLE_SQS_TERMINATION_DRAINING
22              value: "true"
23            - name: QUEUE_URL
24              value: "https://sqs.eu-west-1.amazonaws.com/111111111111/eks-spot-interruption-queue"
25            - name: NODE_TERMINATION_GRACE_PERIOD
26              value: "120"
27            - name: WEBHOOK_URL
28              value: "https://hooks.slack.com/services/T000/B000/XXXX"
29            - name: POD_TERMINATION_GRACE_PERIOD
30              value: "-1"

#3. PodDisruptionBudget for Drain-Safe Rescheduling

1apiVersion: policy/v1
2kind: PodDisruptionBudget
3metadata:
4  name: checkout-service-pdb
5  namespace: payments
6spec:
7  minAvailable: 2
8  selector:
9    matchLabels:
10      app: checkout-service

Without an explicit PodDisruptionBudget, NTH’s eviction call is unconstrained and will happily drop a StatefulSet to zero replicas during a coincident multi-node interruption — the API server enforces PDBs at eviction time, not at drain-initiation time, so their absence is silent until it isn’t.

#Failure Modes and Edge Cases

Several failure modes are specific to spot interruption handling pipelines and do not surface in standard node-drain testing:

• Correlated multi-AZ interruption: Spot reclaim events for a given instance pool are often correlated across an Availability Zone during capacity rebalancing. If your PDBs and topology spread constraints assume independent node failure, a correlated event can breach minAvailable across multiple services simultaneously. Diversify instance types and AZs in the same node group to decorrelate reclaim probability.
• Lifecycle hook heartbeat expiry: If pod eviction takes longer than the configured heartbeat timeout, the ASG defaults to ABANDON or CONTINUE depending on configuration, and the instance terminates regardless of drain state. Always set default_result = CONTINUE only after confirming your heartbeat window comfortably exceeds P99 drain duration under load.
• IMDS path starvation: Under severe CPU pressure (e.g. a noisy-neighbour batch job saturating the host), the local NTH DaemonSet’s IMDS poll loop can be delayed enough to miss the effective window. This is precisely why the SQS/EventBridge path must run independently — it is unaffected by conditions on the node being reclaimed.
• Karpenter interaction: If Karpenter is also managing node lifecycle, ensure NTH and Karpenter’s own interruption handling (consolidation and disruption budgets) are not both attempting to cordon/drain the same node — this produces duplicate eviction API calls and occasionally races on the finalizer removal.
• Stateful workloads with local PVs: EBS-backed StatefulSets require the CSI driver to detach and reattach volumes as part of the reschedule. If drain proceeds faster than volume detachment, pods enter a prolonged ContainerCreating state on the new node — factor CSI detach latency (typically 6–20s, occasionally much higher under EBS API throttling) into your grace period budget.

#Scaling and Security Trade-offs

Operationalising spot interruption handling across large fleets introduces trade-offs that need explicit sign-off from platform and security teams, not just the team implementing architectural patterns for resilience:

• IAM scope for Queue Processor mode requires autoscaling:CompleteLifecycleAction, ec2:DescribeInstances, and sqs:DeleteMessage — scope these to specific ASG ARNs via IRSA rather than granting cluster-wide Auto Scaling permissions.
• IMDSv2 enforcement: NTH’s IMDS polling path requires token-based IMDSv2 calls; if HttpTokens is set to required at the launch template level (recommended for SSRF mitigation), confirm the NTH container has the correct hop-limit and token-refresh logic — older NTH versions default to IMDSv1 semantics and will silently fail to detect interruptions.
• Blast radius vs. cost efficiency: Diversifying across more instance types and AZs reduces correlated interruption risk but reduces bin-packing efficiency and can increase per-node overhead costs by 5–15% depending on instance pool depth.
• Drain concurrency limits: Set --max-unavailable equivalents (via PDBs) conservatively for latency-sensitive services; overly permissive budgets accelerate drain throughput but increase P99 latency spikes during concurrent multi-node reclaim events.
• Observability overhead: Emitting NTH webhook events to Slack or PagerDuty for every interruption is useful for audit trails but generates significant noise at fleet scale (200+ nodes); route these instead to a structured log sink and alert only on drain-failure or heartbeat-timeout conditions.

Treat spot interruption handling as a first-class reliability control rather than a bolt-on DaemonSet install — the difference between a fire-and-forget NTH deployment and a properly tuned lifecycle-hook-backed pipeline is the difference between a genuinely zero-downtime drain and a five-nines SLA breach logged at 03:14 UTC.