HTTP 500 Spike Masks Log-Pipeline Backpressure

Simulation Label

Fictional simulation. This exercise is a bounded, invented training scenario for practising Incident Command reasoning. No real organisation, incident, telemetry, credentials or production identifiers are involved or implied.

Scenario

At "Northwind Cloud Services", a fictional organisation created solely for this exercise, an on-call platform engineer receives an alert at 02:14 UTC: the internal Orders API is returning a sustained spike of HTTP 500 responses. A deployment merged an hour earlier is the obvious suspect, and the temptation is to roll it back immediately. Before doing so, the engineer opens the Incident Command channel and begins a structured triage using the organisation's Logging platform, which aggregates application, infrastructure and log-shipping metrics into one queryable index. The exercise practises separating the observed symptom, the 500 responses, from whatever is actually causing them, using only read-only evidence collection appropriate to an isolated lab environment.

Evidence

The engineer runs two read-only commands against the lab environment to gather evidence before touching anything.

journalctl -u log-shipper --since "15 min ago" --no-pager

Fictional output shows repeated lines: queue full, dropping writes from the log-shipper unit, timestamped across the alert window.

df -h /var/log

Fictional output shows the log volume at 98% utilisation, far above its normal operating band. Application-tier deployment logs, checked separately, show no errors or exceptions correlating with the alert window.

Hypotheses and Actions

• Hypothesis 1: deployment regression. Action: review application logs for exceptions in the alert window. Result: no correlating errors found.
• Hypothesis 2: downstream dependency outage. Action: check dependency health dashboards for the same window. Result: dependencies report normal latency and error rates.
• Hypothesis 3: log-pipeline backpressure disrupting health checks. Action: inspect log-shipper status and disk utilisation on the logging volume. Result: log-shipper is dropping writes and the volume is near capacity, both starting minutes before the alert.

Reveal

The 500 responses were a downstream symptom, not a direct application fault. The logging volume filled to near capacity, causing the log-shipper service to enter a backpressure state and drop writes, including the writes used by the orchestrator's readiness probe to confirm pod health. With those writes missing, the orchestrator marked the affected pods unready and the load balancer failed requests over, producing the observed 500 spike. The deployment an hour earlier was coincidental in timing and not causal.

Learning Outcome

The key lesson is to check log-pipeline and infrastructure health as an early triage step, before assuming an application deployment is the cause of an error spike. In this exercise, a rollback of the deployment would not have resolved the incident and would have introduced unnecessary risk and rework. The safe next decision here is to escalate to the team responsible for the logging platform to remediate disk pressure and restore write throughput, then re-validate application health once the log-shipper queue has cleared. No state-changing or destructive action is required or was taken during this triage; the rollback boundary is that no application change should be reverted until log-pipeline health is confirmed as the resolved root cause.