Profiling Overhead and Sampling Rates

Continuous profiling only works as a production-wide, always-on practice because it’s built on statistical sampling rather than exhaustive instrumentation. Understanding how that sampling works — and the knobs it exposes — explains why profilers can run fleet-wide with minimal impact, and why occasionally increasing the sample rate for a short investigation is safe while leaving it high permanently is not.

Why it matters

• Instrumenting every call would be prohibitively expensive. Recording a timestamped event on every single function call and return can slow a program down by an order of magnitude or more, which is why traditional deep instrumentation-based profilers were only ever run in short, deliberate local sessions — never left running in production.
• Sampling trades exhaustiveness for negligible cost. Instead of recording every call, a sampling profiler interrupts execution at a fixed frequency — commonly around 100 times per second per CPU — and records just the current call stack at that instant. Over enough samples, a function that consumes 40% of CPU time will appear in roughly 40% of samples, which is accurate enough to find real hot paths without the cost of tracking everything.
• Low overhead is what makes “continuous” and “everywhere” possible. Because sampling touches the program only briefly and at a fixed rate, well-implemented continuous profilers are commonly cited as adding well under 1-2% overhead — eBPF-based agents in particular are often measured below 1% — which is low enough to run permanently across an entire fleet rather than being switched on only when something is already suspected to be wrong.
• Sampling rate is a tunable trade-off, not a fixed constant. Raising the sampling frequency increases resolution and shortens how long you must collect data to get a stable picture, but it also raises overhead and the data volume the backend has to store and query — most tools default to a frequency (often around 100Hz) chosen specifically to balance those two costs.

How it works

• Lower frequency means more statistical noise for short-lived or rare code paths. A function that only runs for a few milliseconds might be missed entirely between samples, or captured only a handful of times — enough to see a rough proportion for something that dominates a profile, but not enough to draw firm conclusions about a rarely-hit code path. Longer collection windows compensate for this by accumulating more samples over time.
• Higher frequency narrows that noise but isn’t free. Doubling the sample rate roughly doubles both the interrupt overhead the target process pays and the raw volume of stack data the profiler has to symbolicate, transmit, and store — useful for a focused, time-boxed investigation, but rarely left on permanently at that rate across a whole fleet.
• Overhead depends heavily on implementation, not just frequency. An eBPF-based profiler that walks stacks in the kernel and does symbolication out-of-band imposes a different (often lower) cost than a profiler that must pause or signal the target process (as some JVM- or interpreter-level profilers do), so quoted overhead numbers are tool- and language-specific rather than a single universal figure.
• Vendors and projects generally describe overhead qualitatively rather than promising an exact number. Because actual overhead depends on workload, language runtime, and configuration, tools tend to describe it as “typically under 1-2%” or similar ranges rather than a single guaranteed percentage — always validate against your own workload before trusting a vendor’s headline number.
• Some profilers support adaptive or on-demand rate changes. Rather than running one fixed rate everywhere, a profiling setup can run a low background rate continuously and temporarily increase resolution (or duration) when investigating a specific incident, capturing more detail exactly when it’s needed instead of paying that cost all the time.

The net effect is that continuous profiling earns its “always on” label specifically because sampling frequency is chosen conservatively enough to keep overhead near the noise floor of normal production variance — the same design principle that lets metrics collection run everywhere without becoming the bottleneck it’s trying to measure.