Skip to main content
Daily Triage Briefing

OpenTelemetry Collector Drops Spans With memory_limiter Refusing Data

Impact Summary

When an OpenTelemetry Collector's memory_limiter processor refuses data, traces go missing. Confirm it from the Collector's own logs and internal metrics, then review processor order and limits, changing nothing until you have evidence.

SeveritySEV-3
StatusResolved simulation
PlatformObservability & Reliability
Incident TypeObservability & Reliability
Published9 Nov 2026
Technologies Involved
OpenTelemetry

Simulation Label

Fictional simulation. This exercise is a bounded, invented training scenario for practising operational reasoning. No real organisation, incident, telemetry, credentials or production identifiers are involved or implied.

Operational Summary

Root Cause

memory_limiter was placed after batch and configured with too little headroom under the container ceiling, so it refused data under load and spans were dropped.

Detection Method

Evidence-led guided investigation

Current Status

Resolved simulation

Affected Services

OpenTelemetry

Incident Overview

A fictional retail platform sends traces from its services to a single OpenTelemetry Collector gateway, which forwards them to a tracing backend. Since mid-morning, engineers report that traces for the checkout service are incomplete.

Some requests have no trace at all. The checkout team insists the application is healthy and that its error rate has not changed.

The Collector process is running and its container has not restarted.

Investigation Options

Review the available operational moves and select the best immediate action.

A

Collect the Collector's logs and internal refusal and export metrics read-only, and compare the limit with the container ceiling

B

Restart the backend tracing service to clear a suspected slow ingest

C

Remove memory_limiter from the pipeline in production so data is never refused

D

Ask the checkout team to disable tracing until the cause is known

Tags:OpenTelemetry