KBY Technologies Logo
runbooks/kubernetes-container-orchestration/kubernetes-deployment-rollout-stalled-unavailable-pod.md
HIGH SEVERITYKubernetes Container OrchestrationKubernetes

Kubernetes Deployment Rollout Stalled by an Unavailable Pod

Severity
HIGH
Target Time
30m
DomainKubernetes Container Orchestration
Verified
Overview

Diagnose and safely recover a Kubernetes Deployment rollout that cannot reach its required available replica count.

Share

Destructive Operation

This intervention contains destructive operations. Proceed with extreme caution and ensure database backups exist before execution.

01 // Diagnose

Symptom

Incident signalWhat responders observe

A Deployment rollout remains incomplete and the available replica count stays below the desired count.

Detection Signature

Detection evidenceMetrics, logs, and confirmation commands

  1. Run kubectl rollout status deployment/api -n production --timeout=60s and record the timeout.

  2. Run kubectl get deployment api -n production -o wide and compare desired, updated, ready, and available replicas.

  3. Run kubectl get pods -n production -l app=api -o wide to identify the new unavailable Pod.

  4. Run kubectl describe pod <POD_NAME> -n production and inspect readiness probe failures and recent events.

  5. Run kubectl logs <POD_NAME> -n production --all-containers --tail=200 and record application errors without changing the Pod.

Root Cause Analysis

Causal chainWhy the incident occurred

  1. A new Pod is failing its readiness probe, so the Pod does not become Ready and the Deployment cannot reach its required available replica count.

  2. During the rolling update, the Deployment cannot complete scaling the new ReplicaSet while the new Pod remains unavailable.

02 // Contain & Prevent

Blast Radius

  • New application instances are unavailable.

  • The previous ReplicaSet may carry the remaining production load.

  • Undoing the rollout changes the active Pod template and can cause a temporary availability dip if the previous revision is unhealthy or lacks capacity.

Prevention Measures

Prevent recurrenceControls and architectural guardrails

  • Test readiness probes against the release candidate before deployment.

  • Alert on Deployment unavailable replicas and rollout duration.

  • Keep revision history sufficient for a controlled rollback.

03 // Fix & Intervention

Pre-Flight Checks

Change gateChecks required before intervention

  1. Confirm the current namespace and cluster context with kubectl config current-context.

  2. Confirm the operator is authorised with kubectl auth can-i patch deployments.apps -n production; stop if the result is not yes.

  3. Capture the Deployment and ReplicaSet state with kubectl get deployment,rs -n production -l app=api -o yaml.

  4. Run kubectl rollout history deployment/api -n production to identify the immediately previous revision. Cross-check that revision's image digests and release identifier against the approved deployment pipeline record; do not infer approval from rollout history alone.

  5. Confirm the previous ReplicaSet still exists, its image digest is available, and the cluster has enough schedulable capacity for the required replicas.

  6. Preview the API-server mutation with kubectl rollout undo deployment/api -n production --dry-run=server -o yaml and verify the resulting Pod template matches the approved previous revision.

  7. Confirm availability and error-rate monitoring is live, record abort thresholds, and obtain incident commander approval for the production state change.

Execution CommandsCOMMANDS

kubectl rollout undo deployment/api -n production
kubectl rollout status deployment/api -n production --timeout=5m

04 // Verify & Recover

Verification Steps

Recovery proofEvidence required before closure

  1. Verify kubectl rollout status deployment/api -n production --timeout=5m succeeds.

  2. Confirm desired and available replicas match with kubectl get deployment api -n production.

  3. Observe application availability and error rate for at least 15 minutes and confirm both remain at the healthy baseline.

Rollback Protocol

Safe reversal path

  1. If this runbook's rollout-undo action worsens service health: 1.

  2. If the undo rollout is still progressing, pause further progress with kubectl rollout pause deployment/api -n production

  3. if it has completed, do not rely on pause as a recovery action.

  4. Restore the captured pre-flight Deployment specification through the approved deployment pipeline.

  5. Resume only through the deployment pipeline after the incident commander confirms the restored specification.

  6. Re-run rollout status and availability checks against the restored revision.

Escalation

Conditions requiring additional ownership

  • No known-good ReplicaSet exists.

  • Available replicas remain below desired after rollback.

  • Error rate remains above the established healthy baseline for 15 minutes.

Authoritative Sources

Authoritative · Kubernetes

Deployments

Deployment rollout, availability, and revision behaviour.

Authoritative · Kubernetes

kubectl rollout undo

The supported command for rolling back a resource rollout.

Authoritative · Kubernetes

Configure Liveness, Readiness and Startup Probes

Readiness probe failures remove a Pod from matching Service endpoints and keep the container running for recovery.