Incident Overview
At 09:12 on a Thursday, the on-call platform engineer for the fictional retailer Northfell Retail receives an alert: the checkout-api Deployment in the checkout namespace has three of four pods in CrashLoopBackOff. The rollout of image tag checkout-api:2.14.0 completed twelve minutes earlier.
The release notes for 2.14.0 describe only a logging library bump. The on-call engineer's first assumption is that the new image is broken.The deployment uses a rolling update strategy with maxUnavailable: 1, so one old pod is still healthy and serving traffic.
Error budgets are close to being consumed if the remaining old pod is replaced or fails.
Investigation Options
Review the available operational moves and select the best immediate action.
Roll back the Deployment to image 2.13.4 immediately
Inspect the checkout-config ConfigMap and application config schema to confirm whether the new pods are failing to parse a field the application does not yet expect, correlating the ConfigMap update timestamp against the failing pods' start times before changing anything further
Scale the Deployment down to zero replicas
Restart the kubelet on the affected node