Observability Wiki

Profiling

Reading a Flame Graph

How to read the width, depth, and layout of a flame graph to find the actual bottleneck instead of chasing visual noise.

Last updated

A flame graph is a visualization of stacked call stacks collected by a sampling profiler. Every profiler that produces one — Pyroscope, Parca, async-profiler, pprof, Brendan Gregg’s original FlameGraph scripts — renders the same underlying structure, so learning to read one correctly transfers across every tool you’ll encounter.

The graph is built from thousands of stack samples merged together. Each rectangle is a stack frame (a function call); frames that share the same ancestry are stacked and merged into wider bars. Reading it well comes down to understanding what each axis actually encodes, because both axes are frequently misread by beginners.

Why it matters

  • The x-axis is not time. Frames are sorted alphabetically (or by merge order) to maximize how much identical stacks combine into single bars. A function’s horizontal position tells you nothing about when it ran — only its width matters. Assuming left-to-right means “over time,” as with a trace waterfall, is the single most common misreading.
  • Width is the entire signal. The width of a frame is proportional to how often it appeared in the collected samples — its share of total CPU time, memory allocated, or whatever resource was sampled. A wide bar means the code spent a lot of the profiled resource there, whether that’s one slow call or thousands of fast ones.
  • The y-axis is call stack depth. Each row up (or down, in an icicle graph) represents one level deeper in the call stack. The bottom of a traditional flame graph is the root (e.g., main or a runtime entry point); the top edge of the tallest stack is where the CPU was actually executing at the moment of that sample.
  • It compresses thousands of stack traces into one picture. Without this visualization, understanding “what is my program spending time on” would mean scrolling through raw stack dumps one at a time. The flame graph merges identical ancestries so patterns pop out visually.

How it works

  • Finding the bottleneck means looking for wide, flat plateaus, not tall towers. A tall stack just means deep recursion or many nested calls; it says nothing about resource consumption on its own. A wide plateau near the top of the graph — a leaf function that’s wide relative to its siblings — is where the profiled resource is actually concentrated.
  • Flame graph vs. icicle graph. A flame graph has its root at the bottom and grows upward, mimicking rising flames, with the actively-executing frame at the top edge. An icicle graph inverts this: the root sits at the top and stacks grow downward, with the actively-executing frame at the bottom edge. Both encode identical data — Parca and several web-based UIs default to icicle graphs because they read top-to-bottom like a document, but the interpretation rules (width = share of samples, x-axis = not time) are unchanged.
  • Color is usually meaningless. Most tools color frames by package, language runtime, or simply at random to make adjacent frames visually distinguishable — color is not a legend for a numeric value unless the tool explicitly says so (some tools, like differential flame graphs, do use color to encode a delta between two profiles — check the specific tool).
  • Differential flame graphs highlight change, not magnitude. When comparing two profiles (before/after a deploy, or two time ranges), a differential flame graph colors frames red or blue based on whether they grew or shrank between the two profiles, which is far more useful for regression-hunting than eyeballing two separate graphs side by side.
  • Common misreading: assuming the widest top-level bar is the culprit. The widest frame near the root is often just a framework entry point or dispatcher that everything passes through. The actionable frame is usually further up the stack — the widest leaf-level (topmost) frame that isn’t itself calling out to other wide children.

Once the axis conventions are internalized, reading a flame graph becomes fast: scan for the widest plateaus near the top of the stack, ignore horizontal position entirely, and treat stack height as structural context rather than a signal of cost. That’s the core skill that makes continuous profiling actionable rather than just another dashboard to stare at.

Related tools