Simulation Label
This is a fictional simulation. No real company, telemetry, credentials or production identifiers are involved. All metric values, alert names and log lines below are constructed for this exercise only and must not be treated as a genuine incident record.
Scenario
Aurelia Freight, a fictional logistics platform team, instruments its checkout-service fleet with an OpenTelemetry Collector (contrib distribution) that receives traces and metrics from application SDKs and forwards them to a backend observability platform. At 09:41 local time, two alerts fire within four minutes of each other and appear to contradict one another.
Alert A, generated from a trace-based SLO dashboard, reports that checkout-service error rate has climbed to 22% over a five-minute window. Alert B, generated from Collector health metrics, reports that the Collector pipeline is nominal with zero reported dropped spans. The on-call engineer is left with two signals that cannot both be fully true if interpreted at face value: either checkout-service is genuinely failing, or the Collector reporting the "healthy" status is correct and the trace alert is a false positive.
Evidence
- Alert A (trace-based SLO): checkout-service error rate at 22% for the 09:36–09:41 window, driven by 5xx responses on the payment-confirmation span.
- Alert B (Collector health check): reports zero dropped spans and a healthy exporter queue for the traces pipeline specifically.
- Collector self-telemetry endpoint: the counter
otelcol_exporter_send_failed_metric_pointsfor the metrics pipeline has been climbing steadily since 09:33, though no alert is configured on this counter. - Dashboard observation: the metrics-derived error-rate panel for checkout-service is flat and shows no anomaly, in direct contrast to the trace-derived panel showing the spike described in Alert A.
Hypotheses and Actions
Three hypotheses are on the table. First, the trace-based alert is correct and checkout-service is genuinely degraded, with the metrics view simply lagging. Second, the metrics pipeline itself is failing to deliver data, producing a falsely calm dashboard while the real state is reflected only in traces. Third, both alerts are noisy and neither reflects a real production condition.
Given the evidence, the climbing otelcol_exporter_send_failed_metric_points counter is the single strongest lead: it directly measures whether metric data points are being lost before reaching the dashboard, independent of what either alert claims. Four immediate actions are considered.
- Fail traffic over to a secondary region immediately without reviewing collector telemetry.
- Query the Collector’s own self-telemetry endpoint for exporter failure and queue metrics before taking any other action.
- Increase the batch processor queue size and restart the Collector fleet immediately on the assumption of queue exhaustion.
- Silence both alerts as contradictory noise and close the incident.
Only one of these actions gathers evidence without introducing risk or foreclosing the investigation; the others either act before confirmation or discard a real signal.
Reveal
The self-telemetry check confirms the second hypothesis. The metrics pipeline’s batch processor has a small queue and no retry policy configured. During the incident window, the metrics exporter began receiving intermittent 429 rate-limit responses from the backend. Because the queue could not absorb the backpressure, a subset of metric points was silently dropped rather than retried, so the metrics-derived error-rate panel simply stopped receiving fresh data and froze at its last healthy value — it never actually measured the real error spike. The traces pipeline used a separate, correctly sized queue and was unaffected, so it faithfully reported the genuine error-rate increase caused by a slow downstream dependency timeout. The two alerts were never actually contradictory: one was measuring reality, and the other had gone silent without announcing that it had done so.
Learning Outcome
The core lesson is that an apparently healthy telemetry signal is not evidence of a healthy system unless the pipeline producing that signal is itself verified to be delivering data. Before trusting or dismissing either alert, the safe first step is to inspect the Collector’s own self-telemetry counters for export failures and queue saturation on the specific pipeline in question.
Verified operational checks for this exercise: confirm that otelcol_exporter_send_failed_metric_points and queue-depth counters are reviewed for the affected pipeline before either alert is actioned; confirm that the timestamp of any climbing failure counter is correlated against the alert firing time before concluding a shared root cause. Because only read-only diagnostics were used in the recommended action, there is no state change to roll back; the rollback boundary is simply that no queue-size change, restart or failover should be applied until the self-telemetry evidence is reviewed. The next safe decision is to escalate to the platform on-call with the correlated evidence and agree remediation (for example, exporter retry configuration or backend rate-limit review) only after the mechanism is confirmed, not before.