Reading a flame graph: width is samples, the x-axis is not time

Goal

Find where a running program spends CPU time by reading a flame graph correctly, instead of scanning a flat function list and guessing at callers.

Prerequisites

A sampling profiler that records stack traces (Linux perf, py-spy for Python, a runtime's own sampler), the FlameGraph scripts or a profiler that writes the SVG directly, symbols and unwinding information so that frames are not [unknown], and representative load while sampling.

Steps

1. Capture under load. Gregg's example is perf record -F 99 -p PID -g -- sleep 30, then perf script | stackcollapse-perf.pl > out.folded and flamegraph.pl out.folded > out.svg; odd rates such as 99 Hz avoid sampling in lockstep with other activity. For Python, py-spy record -o profile.svg --pid 12345 writes the graph directly.
2. Read the axes as the flame graph page defines them: the x-axis is the stack profile population sorted alphabetically, not the passage of time; the y-axis is stack depth from zero at the bottom; each rectangle is a frame; the wider it is, the more often it appeared in samples; the top edge is what was on-CPU and everything beneath is its ancestry.
3. Look for wide plateaus along the top edge: those functions were executing themselves and are candidates for faster implementations.
4. Look for wide frames lower down whose children are many thin towers: time is spread across callees, so the caller is the place to change (call less often, batch, cache).
5. Use the SVG's search to sum a function's width across every stack; a function called from many places shows as thin towers that add up.
6. Compare two profiles (before and after, or a slow and a healthy host) with a differential flame graph, which colours each frame by the change between the two profiles.
7. Check the sample count before believing small widths: thirty seconds at 99 Hz is about 2,970 samples per CPU (arithmetic), so a frame at 1% width rests on roughly 30 samples and fractions of a percent are noise.
8. If the program is slow but the CPU is idle, a CPU flame graph shows nothing useful; capture an off-CPU profile of the waiting instead.

Expected result

A named function or call path with its share of samples, the profile saved next to the change that addresses it, and a second profile showing the width reduced.

Limits and test basis

Missing frame pointers or aggressive inlining truncate or merge frames. Sampling attributes time to the instruction that was running, not to its cause; a cache miss appears as the instruction that stalled. Interpreted and JIT-compiled languages need their own stack walkers. Based on the cited pages; no measurements claimed.