Incident Overview
Meridian Foods, a fictional grocery retailer, runs its checkout-service deployment on a fictional Kubernetes cluster. The scenario assumes Kubernetes 1.28 with the kubelet cgroup driver set to systemd and cgroup v2 enabled on all nodes u2014 an environmental assumption stated explicitly because the reveal depends on it.
During a routine evening peak, on-call engineer Priya notices two alerts firing within the same three-minute window: one reporting the checkout-service pods as OOMKilled and restarting repeatedly, and a second reporting that node-level memory pressure is normal. The alerts appear to contradict each other, and Priya must decide whether to raise container memory limits, restart infrastructure, or investigate further before changing anything.
Investigation Options
Review the available operational moves and select the best immediate action.
Increase the checkout-service container memory limit until OOMKilled events stop.
Inspect cgroup memory.stat and container_memory_working_set_bytes versus container_memory_rss to confirm whether sidecar page cache is being counted against the memory limit before changing any limits.
Restart the node's kubelet service to clear the alert state.
Disable the log-tailing sidecar entirely in production without further investigation.