Tail-Based Sampling
In plain English
Plain definition
A trace sampling approach that waits until a full trace finishes before deciding whether to keep it, based on outcomes like errors or high latency, rather than deciding blindly at the start of the request.
Tail-based sampling requires buffering all spans belonging to a trace until a completion signal fires — typically a configurable idle timeout (e.g., 30s with no new spans) or an explicit root span termination. Only then does the sampling processor evaluate policies against the assembled trace: status_code == ERROR, duration > p99_threshold, specific attribute matches, or probabilistic fallback for the remainder. This is fundamentally different from head-based sampling, where the decision is stamped into the trace context at the root span and propagated via tracestate before any span data exists to judge.
- Consistent span routing: because spans for a single trace can be emitted by dozens of services and land on different collector replicas, every span must be routed by
trace_idto the same sampling processor instance — typically via consistent hashing at a load balancer or through a Kafka topic partitioned on trace ID. Without this, no single processor ever sees the complete trace and sampling decisions become non-deterministic per-span rather than per-trace. - Memory pressure: buffering full spans for every in-flight trace (before knowing whether they’ll be kept) is the dominant cost driver. Collector fleets sizing for tail sampling must provision for peak concurrent trace cardinality, not just throughput, and typically cap buffer windows aggressively to bound worst-case memory.
- Broken/partial traces: async workflows, long-lived sagas, or fire-and-forget spans that exceed the buffer timeout get flushed prematurely — the tail decision is made on an incomplete trace, silently defeating the strategy’s premise.
Architecturally, tail-based sampling pushes sampling logic out of the application SDK entirely and into a dedicated collector tier (e.g., otelcol‘s tailsamplingprocessor), which must sit behind a routing layer that guarantees trace affinity. This adds a hop and a stateful buffering stage into what is otherwise a stateless pipeline, and it means the collector tier itself becomes a scaling and failure-mode concern — a collector crash before flush loses buffered traces, including the error traces you specifically deployed this to keep.
The operational payoff is retaining nearly 100% of error and outlier-latency traces while sampling routine successful requests at 0.1-1%, which is the only economically viable way to run full-fidelity tracing at high request volumes without paying storage costs proportional to raw traffic. Teams adopt it specifically because head-based sampling, applied uniformly at ingress, statistically discards most of the traces an incident responder would actually want.
Tail-based sampling trades ingestion-time simplicity for a stateful, trace-affinity-aware collector architecture, and that tradeoff only pays off when the volume and cost pressure justify the added buffering, routing, and partial-trace failure modes it introduces.