Reading a Trace Waterfall for Latency

A trace waterfall — sometimes called a Gantt view — lists every span in a trace as a horizontal bar on a shared timeline, indented to show the parent-child hierarchy. The root span spans the full width of the request; each child span is drawn nested inside its parent, positioned at the offset where it actually started and sized to its actual duration. Reading one well is the single most useful skill for turning “checkout is slow” into “this specific downstream call is slow.”

Why it matters

• It separates real bottlenecks from spans that are just waiting. A parent span’s duration includes all the time its children took, so a wide bar doesn’t necessarily mean that operation itself was slow — it might just be waiting on a child. The bottleneck is the span whose own duration (its width minus the combined width of its children) is large, not merely the widest bar on the screen.
• It exposes serialization that should be parallel. When independent child spans that don’t depend on each other are drawn one after another instead of stacked at the same start time, that’s a direct visual signal that calls which could run concurrently are instead running sequentially, adding their durations together unnecessarily.
• It makes N+1 query patterns immediately visible. A long, repetitive series of near-identical short spans (the same query or the same downstream call, repeated dozens of times) is the classic waterfall signature of code fetching one row and then issuing a separate follow-up call per item, instead of batching.
• It shows gaps, not just spans. Empty space between a parent span starting and its first child starting represents time the service itself spent doing something that wasn’t captured as a span — often overlooked application code, serialization, or queueing that isn’t instrumented.

How to read it

• Start from the root and follow the critical path. The critical path is the chain of spans that, end to end, actually determines the total request duration — at each level, that’s the child span that finishes last (or takes the longest), since only that chain, not every sibling, is what the parent is waiting on.
• Look for the deepest span with a duration close to its own width, not its subtree’s. Most trace UIs show self time versus total time per span; the span with a large self time is where the request is actually spending time doing work, as opposed to a parent span that just looks big because of what’s nested under it.
• Compare sibling spans for repetition and sequencing. Rows at the same indentation level that repeat, or that run one after another with no overlap despite having no data dependency, are the pattern to look for when hunting for avoidable serial calls or N+1 issues.
• Check for large unexplained gaps. A visible blank stretch between spans, especially just before an outbound call starts, often points to something outside of what’s instrumented — connection pool exhaustion, garbage collection pauses, or thread scheduling delays — that’s worth investigating even though the trace itself can’t explain it directly.

Waterfalls vs. flame graphs

It’s worth being precise about the distinction, since both are drawn as stacked horizontal bars and get confused often: a trace waterfall shows spans across a distributed request — network calls between separate services, each span representing a discrete operation with its own start and end time as observed at the tracing layer. A flame graph, by contrast, is a profiling visualization of a single process’s call stack, built from repeated stack samples that show which functions were executing and for how much CPU time, aggregated within one process rather than across a network. A trace waterfall answers “which service call was slow and why did it happen in this order”; a flame graph answers “which function inside that one process actually burned the CPU.” When a waterfall points to a specific span as the bottleneck and that span turns out to be CPU-bound rather than waiting on the network, a flame graph from continuous profiling is the natural next step to find the exact line of code responsible.