Simulation Label
This is a fictional training simulation. No real incident, customer, credential or production system is described. All names, identifiers and log excerpts are invented for this exercise.
Scenario
At 09:40 on a Wednesday, the overnight on-call engineer hands off an ongoing minor alert to the day-shift Incident Commander (IC). The alert originated from a checkout service that intermittently returns HTTP 502 errors. The overnight engineer reports, verbally, that "logs look clean, error rate is dropping." The day IC accepts the handoff and downgrades the incident from SEV-3 to a monitoring ticket.
At 10:25, a customer support escalation reports a sustained spike in checkout failures that began roughly forty minutes earlier. The IC reopens the incident, expecting the centralised Logging platform to show a clear timeline of the 502 spike. Instead, the log explorer shows a flat, unremarkable graph with almost no entries between 09:35 and 10:20 for the checkout service, despite application dashboards showing elevated error rates throughout that window.
Evidence
- The centralised Logging platform's ingestion dashboard shows a sharp drop in indexed log volume for the checkout namespace starting at 09:34, falling from a typical 1,200 events/minute to fewer than 40 events/minute.
- The application performance monitoring (APM) tool, which is a separate pipeline from Logging, shows a genuine and sustained increase in HTTP 502 responses from 09:36 onward, rising from a baseline of 0.3% to a peak of 11% of checkout requests.
- The Logging agent's own health check endpoint on the affected hosts returns HTTP 200 throughout the window, giving no indication of an agent-level failure.
- The Logging platform's ingestion pipeline dashboard shows a queue depth metric for the checkout log forwarder climbing steadily from 09:34, reaching a sustained backlog of over 500,000 unprocessed events by 10:20.
- A configuration change log entry shows that a log parsing rule for the checkout service was modified at 09:31, nine minutes before the queue depth began climbing.
- The overnight engineer's handoff notes state that the log explorer "looked normal" at 09:40, six minutes after the parsing rule change and four minutes after the queue depth began rising, but before the backlog had grown large enough to be visually obvious on the default one-hour dashboard view.
Hypotheses and Actions
Four plausible explanations were raised during the simulated bridge call. Each is evaluated against the evidence above.
- Hypothesis A: the checkout service itself stopped emitting logs due to an application crash. This is inconsistent with the evidence, because the APM pipeline, which instruments the same application, continued to record elevated error rates throughout the window. If the application had crashed or stopped logging entirely, APM would also show a gap or a drop in request volume, not a sustained rise in observed 502s.
- Hypothesis B: a change to the log parsing rule at 09:31 caused the forwarder to fail silently on a growing proportion of events, creating a backlog rather than a clean stop, which matches the queue depth climbing steadily rather than dropping to zero immediately. This aligns with the timing evidence (parsing rule change at 09:31, queue depth rise beginning at 09:34) and explains why the agent health check still returned HTTP 200: the agent process was alive and accepting events, but downstream processing was degraded, not absent.
- Hypothesis C: a network partition between the checkout hosts and the Logging ingestion endpoint caused events to be dropped. This is not well supported, because a network partition would typically produce agent-side connection errors or a health check failure, neither of which is present in the evidence.
- Hypothesis D: the overnight engineer misread the dashboard and the apparent "clean" logs were simply a display caching artefact with no underlying pipeline issue. This is plausible as a contributing factor to the delayed detection, but it does not account for the measured queue depth backlog, which is an independent, non-visual metric showing a real processing delay rather than a display problem.
Reveal
The root cause was a parsing rule change applied to the checkout service's log forwarder configuration at 09:31. The new rule introduced a regular expression that failed to match a subset of log lines containing an updated field format deployed in the same release window. Non-matching lines were queued for a fallback parser rather than being dropped outright, which produced a growing backlog instead of an immediate, visible failure. Because the backlog built gradually and the default dashboard view aggregated data hourly, the overnight engineer's 09:40 glance did not reveal the emerging delay. The underlying application was genuinely degraded the entire time, as shown consistently by APM, but the Logging platform could not surface that degradation to the IC when it mattered because ingested events were stuck in an unprocessed queue.
Learning Outcome
This simulation demonstrates that a healthy-looking log explorer view does not confirm a healthy logging pipeline. Ingestion queue depth, forwarder health, and parsing error rates are distinct signals from raw dashboard appearance, and each must be checked independently before a log absence is treated as evidence of an absence of the underlying event. It also demonstrates that verbal handoffs summarising monitoring state ("logs look clean") should be supported by a specific, timestamped check of the relevant pipeline health metrics, not a glance at a default-range visualisation. An Incident Command workflow that treats "no errors visible in Logging" as equivalent to "no errors occurred" is vulnerable to exactly this failure mode whenever a parsing or ingestion change is made close to a handoff.
Bounded Recovery and Verification Checks for This Simulation
Within this fictional scenario, the recovery path the IC should validate before closing the incident is: confirm the parsing rule change is identified and reverted or corrected in a non-production environment first; confirm the ingestion queue depth returns to baseline after the fix is applied; confirm that a sample of the previously queued events, once reprocessed, is retrievable and correctly indexed; and confirm that APM-reported error rates and Logging-reported error rates converge for the same time window once the backlog has drained. The incident should not be downgraded again until the queue depth metric has been below its normal threshold for a full observation window, and a post-incident review should add an explicit check of log pipeline health metrics to the standard handoff checklist, separate from a visual dashboard glance.