Kubernetes Deployment Rollout Stalled by an Unavailable Pod
Overview
Diagnose and safely recover a Kubernetes Deployment rollout that cannot reach its required available replica count.
Destructive operation
This intervention contains destructive operations. Proceed with extreme caution and ensure database backups exist before execution.
Diagnose
Symptom
What responders observe.
A Deployment rollout remains incomplete and the available replica count stays below the desired count.
Detection
Metrics, logs and confirmation commands.
-
Run
kubectl rollout status deployment/api -n production --timeout=60sand record the timeout. -
Run
kubectl get deployment api -n production -o wideand compare desired, updated, ready, and available replicas. -
Run
kubectl get pods -n production -l app=api -o wideto identify the new unavailable Pod. -
Run
kubectl describe pod <POD_NAME> -n productionand inspect readiness probe failures and recent events. -
Run
kubectl logs <POD_NAME> -n production --all-containers --tail=200and record application errors without changing the Pod.
Commands in these steps
kubectl rollout status deployment/api -n production --timeout=60skubectl get deployment api -n production -o widekubectl get pods -n production -l app=api -o widekubectl describe pod <POD_NAME> -n productionkubectl logs <POD_NAME> -n production --all-containers --tail=200
Root cause
Why the incident occurred.
-
A new Pod is failing its readiness probe, so the Pod does not become Ready and the Deployment cannot reach its required available replica count.
-
During the rolling update, the Deployment cannot complete scaling the new ReplicaSet while the new Pod remains unavailable.
Contain and prevent
Blast radius
What else this can affect.
New application instances are unavailable.
The previous ReplicaSet may carry the remaining production load.
Undoing the rollout changes the active Pod template and can cause a temporary availability dip if the previous revision is unhealthy or lacks capacity.
Prevention
Controls and guardrails that stop it recurring.
-
Test readiness probes against the release candidate before deployment.
-
Alert on Deployment unavailable replicas and rollout duration.
-
Keep revision history sufficient for a controlled rollback.
Fix
Pre-flight checks
Check these before you change anything.
-
Confirm the current namespace and cluster context with
kubectl config current-context. -
Confirm the operator is authorised with
kubectl auth can-i patch deployments.apps -n production; stop if the result is not yes. -
Capture the Deployment and ReplicaSet state with
kubectl get deployment,rs -n production -l app=api -o yaml. -
Run
kubectl rollout history deployment/api -n productionto identify the immediately previous revision. Cross-check that revision's image digests and release identifier against the approved deployment pipeline record; do not infer approval from rollout history alone. -
Confirm the previous ReplicaSet still exists, its image digest is available, and the cluster has enough schedulable capacity for the required replicas.
-
Preview the API-server mutation with
kubectl rollout undo deployment/api -n production --dry-run=server -o yamland verify the resulting Pod template matches the approved previous revision. -
Confirm availability and error-rate monitoring is live, record abort thresholds, and obtain incident commander approval for the production state change.
Commands in these steps
kubectl config current-contextkubectl auth can-i patch deployments.apps -n productionkubectl get deployment,rs -n production -l app=api -o yamlkubectl rollout history deployment/api -n productionkubectl rollout undo deployment/api -n production --dry-run=server -o yaml
Commands
kubectl rollout undo deployment/api -n production
kubectl rollout status deployment/api -n production --timeout=5m
Verify and recover
Verification
Evidence required before you close the incident.
-
Verify
kubectl rollout status deployment/api -n production --timeout=5msucceeds. -
Confirm desired and available replicas match with
kubectl get deployment api -n production. -
Observe application availability and error rate for at least 15 minutes and confirm both remain at the healthy baseline.
Commands in these steps
kubectl rollout status deployment/api -n production --timeout=5mkubectl get deployment api -n production
Rollback
The safe way back.
-
If this runbook's rollout-undo action worsens service health: 1.
-
If the undo rollout is still progressing, pause further progress with
kubectl rollout pause deployment/api -n production -
if it has completed, do not rely on pause as a recovery action.
-
Restore the captured pre-flight Deployment specification through the approved deployment pipeline.
-
Resume only through the deployment pipeline after the incident commander confirms the restored specification.
-
Re-run rollout status and availability checks against the restored revision.
Commands in this rollback
kubectl rollout pause deployment/api -n production
Escalate when
Conditions that need another owner.
-
No known-good ReplicaSet exists.
-
Available replicas remain below desired after rollback.
-
Error rate remains above the established healthy baseline for 15 minutes.
Fixed? Get the next one before it hits you: The KBY Brief
Practical fixes for the systems you run, by email about twice a week. Unsubscribe at any time.
About this page
Credited to
Dates
Published
Last updated
Corrections
- : Corrected during a review of published articles. Details
Was this useful?
Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.
Sources
Based on the vendor sources listed below. Check each step against your environment before acting.
First published 20 August 2026 · Last updated 10 November 2026
Deployments
Cited for: Deployment rollout, availability, and revision behaviour.
kubectl rollout undo
Cited for: The supported command for rolling back a resource rollout.
Configure Liveness, Readiness and Startup Probes
Cited for: Readiness probe failures remove a Pod from matching Service endpoints and keep the container running for recovery.