Fictional Simulation Notice
This is a fictional Daily Triage exercise. All hostnames, dashboards, team names and timestamps below are invented for training purposes. No real incident, live telemetry, credentials or production system is described or implied.
Scenario
At 09:42 on a Wednesday, an on-call platform engineer opens the primary error-rate dashboard for the fictional "Checkout" service and sees a sharp spike: reported errors have jumped from a baseline of roughly 12 per minute to over 240 per minute. She pages the duty Incident Commander (IC) and opens an incident channel. Two minutes later, a second engineer, working from a different dashboard built on the same centralised Logging platform but pointed at a separate index, reports that the same service shows a completely normal error rate. The IC now has two Logging-backed dashboards, built from the same underlying platform, giving contradictory pictures of the same service at the same moment.
The IC must decide, within the next few minutes, whether this is a genuine customer-impacting spike requiring a higher severity declaration, a false alarm from a stale dashboard, or something else entirely, before either escalating unnecessarily or missing a real outage.
Evidence Gathered So Far
- Dashboard A (index "checkout-live") shows error count rising sharply starting at 09:40, sourced from the fleet's primary log shippers.
- Dashboard B (index "checkout-secondary") shows a flat, normal error rate for the same window, sourced from a secondary fleet of forwarders behind a different network path.
- The incident channel contains a message from a networking engineer noting a brief, unconfirmed network blip affecting one rack roughly ten minutes earlier.
- Neither dashboard's query panel currently displays the data freshness or last-received-event timestamp for its underlying index.
- No alerting rule has fired on ingestion lag or forwarder health for either fleet.
Hypotheses and Actions
Three plausible hypotheses fit the available evidence:
- Hypothesis 1 — Genuine outage: Checkout is truly failing, and Dashboard B is simply missing the affected traffic because of routing differences.
- Hypothesis 2 — Stale or lagging ingestion: One or both forwarder fleets are buffering events locally after the reported network blip, so one index has not yet received recent log entries.
- Hypothesis 3 — Query misconfiguration: One dashboard is querying the wrong time range, timezone or index alias, producing an artificially flat or spiked view.
Before choosing, the responder gathers direct evidence rather than trusting either dashboard's headline number, following the interventions evaluated below.
Reveal: What Actually Happened
Direct inspection of the forwarder fleet behind Dashboard B showed a rising buffer queue length and a forwarding lag climbing past four minutes, correlating exactly with the network blip mentioned in the incident channel. The forwarders on that fleet had queued events locally rather than dropping them, and delivery resumed once the network path recovered, but for the several minutes the IC was making a severity decision, Dashboard B's "normal" reading reflected missing data, not a healthy service. Dashboard A's spike was the accurate, real-time signal. The underlying Logging platform itself behaved correctly; the divergence was produced by uneven ingestion lag across two forwarder fleets during a brief network disruption, not by a fault in the platform or by falling error counts.
Learning Outcome
When two dashboards built on the same logging platform disagree during an incident, absence of recent errors is not equivalent to evidence of normal operation; it may simply mean the data has not arrived yet. Incident commanders should treat ingestion lag and forwarder buffer health as first-class evidence, check it explicitly before trusting a reassuring dashboard, and only downgrade severity once freshness has been confirmed on every source being compared.