Head-based vs. Tail-based Sampling

Recording every trace for every request is prohibitively expensive at any meaningful scale, so tracing systems sample: they keep some fraction of traces and discard the rest. How that decision gets made — and, critically, when — splits into two fundamentally different strategies with very different cost and accuracy trade-offs.

Head-based sampling decides whether to record a trace the moment it starts, before anything about the request’s actual behavior is known. Tail-based sampling waits until the entire trace has finished — every span from every service has come in — and only then decides whether to keep it.

Why it matters

• Head-based sampling is cheap but blind. The decision is typically a simple probabilistic roll (say, keep 1% of traces) made at the very first span, propagated to every downstream service via the trace context so the decision stays consistent. It costs almost nothing in memory or compute, but it has no way of knowing in advance that a given request is about to error out or take ten seconds — it might sample the boring requests and discard the exact ones an engineer would want to investigate.
• Tail-based sampling can prioritize the traces that actually matter. Because the decision happens after the full trace is assembled, policies can keep every trace that errored, every trace over a latency threshold, or every trace matching a particular attribute, while still discarding most routine traffic — this is the difference between sampling “randomly” and sampling “intelligently.”
• Tail-based sampling requires buffering, which has real cost. Every span for every trace has to be held in memory (or a similar buffer) until that trace is judged complete, since the sampling decision can’t be made until the last span arrives. This makes tail sampling meaningfully more expensive to operate than head-based sampling, and it introduces a delay between when a trace happens and when the keep/discard decision is finalized.
• The two aren’t mutually exclusive. Many production setups run a coarse head-based sample rate to cap raw ingestion volume, then apply tail-based rules on top of what makes it through, balancing cost control against not losing the traces that matter.

How it works

• Head-based decisions travel with the trace context. Because the decision is made once, at the root, and then encoded into the propagated trace flags, every downstream service can honor the same decision without needing to communicate — this is part of why head-based sampling is so cheap operationally.
• Tail-based sampling needs a buffering layer, typically the collector. The OpenTelemetry Collector’s tail sampling processor holds spans in memory, groups them by trace ID, and waits for a configurable decision window before evaluating policies and either forwarding or dropping the whole trace.
• Policies can combine multiple signals. Common tail sampling policies include latency (keep traces whose total duration crosses a threshold), status/error-based rules (always keep traces containing an error span), attribute matching (keep traces touching a specific customer or endpoint), and probabilistic sampling as a baseline on top of those rules — the tail sampling processor supports composing several such policies together.
• Tail sampling only works correctly when all spans of a trace reach the same collector instance (or a coordinated group of them). In a horizontally scaled collector deployment, this typically requires routing all traffic for a given trace ID to the same collector pod, since the sampling decision needs the complete picture.
• Tail sampling trades timeliness for completeness. Traces aren’t forwarded to the backend until the buffering window closes, so there’s an inherent lag between a request happening and its trace becoming visible, compared to head-based sampling where a kept trace is available almost immediately.

The practical choice usually comes down to how much of the “interesting” traffic is predictable versus rare. If failures and slow requests are common enough that a flat percentage sample would catch a reasonable number of them, head-based sampling alone is simpler and cheaper to run. If the whole point is catching the rare error buried in mostly-successful traffic, tail-based sampling — even with its buffering overhead — is usually the only strategy that reliably surfaces it.