Readiness Probe Misconfiguration Leaves Healthy Pods Out of Service

Impact Summary

A fictional Kubernetes exercise shows how a readiness probe pointing at the wrong path can remove healthy pods from a Service while logs look normal. Diagnose using read-only evidence first, and change nothing without review.

SeveritySEV-3

StatusResolved simulation

PlatformKubernetes & Containers

Incident TypeKubernetes & Containers

Published6 Nov 2026

Technologies Involved

Kubernetes

Simulation Label

Fictional simulation. This exercise is a bounded, invented training scenario for practising operational reasoning. No real organisation, incident, telemetry, credentials or production identifiers are involved or implied.

Operational Summary

Root Cause

A fictional Deployment edit set the readiness probe to a path the application does not serve, removing pods from Service endpoints.

Detection Method

Evidence-led guided investigation

Current Status

Resolved simulation

Affected Services

Kubernetes

Incident Overview

In a fictional non-production cluster, a web Deployment has four replicas. Users of the Service report intermittent errors.

A dashboard shows all pods Running, yet traffic is failing. Assume the team has read-only access.

Investigation Options

Review the available operational moves and select the best immediate action.

Inspect the pod description and probe definition against the application routes.

Restart all pods at once.

Scale the Deployment up to compensate.

Drain the nodes to rule out a network fault.

Tags:Kubernetes

Related Incident Reports

Kubernetes & Containers