Skip to main content

Kubernetes Deployment Rollout Stalled by an Unavailable Pod

High severity Target recovery 30mKubernetes

Overview

Diagnose and safely recover a Kubernetes Deployment rollout that cannot reach its required available replica count.

Markdown, one checkbox per step, for pasting into a ticket.

Destructive operation

This intervention contains destructive operations. Proceed with extreme caution and ensure database backups exist before execution.

Diagnose

Symptom

What responders observe.

A Deployment rollout remains incomplete and the available replica count stays below the desired count.

Detection

Metrics, logs and confirmation commands.

  1. Run kubectl rollout status deployment/api -n production --timeout=60s and record the timeout.

  2. Run kubectl get deployment api -n production -o wide and compare desired, updated, ready, and available replicas.

  3. Run kubectl get pods -n production -l app=api -o wide to identify the new unavailable Pod.

  4. Run kubectl describe pod <POD_NAME> -n production and inspect readiness probe failures and recent events.

  5. Run kubectl logs <POD_NAME> -n production --all-containers --tail=200 and record application errors without changing the Pod.

Commands in these steps

  • kubectl rollout status deployment/api -n production --timeout=60s
  • kubectl get deployment api -n production -o wide
  • kubectl get pods -n production -l app=api -o wide
  • kubectl describe pod <POD_NAME> -n production
  • kubectl logs <POD_NAME> -n production --all-containers --tail=200

Root cause

Why the incident occurred.

  1. A new Pod is failing its readiness probe, so the Pod does not become Ready and the Deployment cannot reach its required available replica count.

  2. During the rolling update, the Deployment cannot complete scaling the new ReplicaSet while the new Pod remains unavailable.

Contain and prevent

Blast radius

What else this can affect.

  • New application instances are unavailable.

  • The previous ReplicaSet may carry the remaining production load.

  • Undoing the rollout changes the active Pod template and can cause a temporary availability dip if the previous revision is unhealthy or lacks capacity.

Prevention

Controls and guardrails that stop it recurring.

  • Test readiness probes against the release candidate before deployment.

  • Alert on Deployment unavailable replicas and rollout duration.

  • Keep revision history sufficient for a controlled rollback.

Fix

Pre-flight checks

Check these before you change anything.

  1. Confirm the current namespace and cluster context with kubectl config current-context.

  2. Confirm the operator is authorised with kubectl auth can-i patch deployments.apps -n production; stop if the result is not yes.

  3. Capture the Deployment and ReplicaSet state with kubectl get deployment,rs -n production -l app=api -o yaml.

  4. Run kubectl rollout history deployment/api -n production to identify the immediately previous revision. Cross-check that revision's image digests and release identifier against the approved deployment pipeline record; do not infer approval from rollout history alone.

  5. Confirm the previous ReplicaSet still exists, its image digest is available, and the cluster has enough schedulable capacity for the required replicas.

  6. Preview the API-server mutation with kubectl rollout undo deployment/api -n production --dry-run=server -o yaml and verify the resulting Pod template matches the approved previous revision.

  7. Confirm availability and error-rate monitoring is live, record abort thresholds, and obtain incident commander approval for the production state change.

Commands in these steps

  • kubectl config current-context
  • kubectl auth can-i patch deployments.apps -n production
  • kubectl get deployment,rs -n production -l app=api -o yaml
  • kubectl rollout history deployment/api -n production
  • kubectl rollout undo deployment/api -n production --dry-run=server -o yaml

Commands

kubectl rollout undo deployment/api -n production
kubectl rollout status deployment/api -n production --timeout=5m

Verify and recover

Verification

Evidence required before you close the incident.

  1. Verify kubectl rollout status deployment/api -n production --timeout=5m succeeds.

  2. Confirm desired and available replicas match with kubectl get deployment api -n production.

  3. Observe application availability and error rate for at least 15 minutes and confirm both remain at the healthy baseline.

Commands in these steps

  • kubectl rollout status deployment/api -n production --timeout=5m
  • kubectl get deployment api -n production

Rollback

The safe way back.

  1. If this runbook's rollout-undo action worsens service health: 1.

  2. If the undo rollout is still progressing, pause further progress with kubectl rollout pause deployment/api -n production

  3. if it has completed, do not rely on pause as a recovery action.

  4. Restore the captured pre-flight Deployment specification through the approved deployment pipeline.

  5. Resume only through the deployment pipeline after the incident commander confirms the restored specification.

  6. Re-run rollout status and availability checks against the restored revision.

Commands in this rollback

  • kubectl rollout pause deployment/api -n production

Escalate when

Conditions that need another owner.

  • No known-good ReplicaSet exists.

  • Available replicas remain below desired after rollback.

  • Error rate remains above the established healthy baseline for 15 minutes.

The KBY Brief

Fixed? Get the next one before it hits you: The KBY Brief

Practical fixes for the systems you run, by email about twice a week. Unsubscribe at any time.

About this page

Credited to

Dates

Published

Last updated

Corrections

  • : Corrected during a review of published articles. Details

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

Sources

Based on the vendor sources listed below. Check each step against your environment before acting.

First published 20 August 2026 · Last updated 10 November 2026

Share
Kubernetes Deployment Rollout Stalled by an Unavailable Pod