CPU profiling with perf: perf top, perf record -g, perf report, and perf_event_paranoid

Goal

Find which function is consuming CPU on a currently slow host, first with a live view and then with a saved, re-readable profile.

Prerequisites

The perf tool matching the running kernel; either root, or a kernel.perf_event_paranoid setting permissive enough for an unprivileged user; debug symbols for the binaries of interest, for readable output.

Steps

1. Check the current permission barrier: sysctl kernel.perf_event_paranoid. The kernel's admin documentation defines the scale precisely: -1 imposes no scope or access restrictions on perf_events; values >=0 allow per-process and system-wide monitoring but exclude raw tracepoints; >=1 allows per-process monitoring only; >=2 (the upstream default) additionally limits it to user-space events. Debian and Ubuntu kernels add stricter levels above 2 that block unprivileged use entirely. The setting governs unprivileged users only; root and, since Linux 5.8, processes with CAP_PERFMON are not limited by it. Record the value before changing it, so it can be restored: sudo sysctl -w kernel.perf_event_paranoid=<original>. A sysctl -w change is runtime-only and reverts on reboot unless also written under /etc/sysctl.d/.
2. Get an immediate live view: sudo perf top, described in its manual as generating and displaying a performance counter profile in real time. Read the top lines by overhead percentage per symbol.
3. If the symbol column shows raw addresses or [unknown] instead of names, the relevant binary or shared library lacks debug symbols or was stripped (or the code is JIT-compiled, which needs the runtime's own perf-map support); install the distribution's debug-info package for that binary rather than trusting the address list.
4. For a profile that can be saved, re-examined, or handed to someone else: sudo perf record -g -p <pid> -- sleep 30 (or -a for the whole system). perf record's manual describes -g as enabling call-graph (stack chain/backtrace) recording for kernel and user space, with fp (frame pointer) as the default unwind mode for user space and dwarf or lbr selectable via --call-graph when frame pointers are unreliable — commonly the case for optimised binaries built without frame pointers preserved. The output goes to ./perf.data (change with -o) and grows quickly with -a, dwarf unwinding or long durations; check free space first.
5. Read the saved data: sudo perf report (a file recorded as root is readable only by root), described in its manual as displaying the performance counter profile information recorded via perf record (defaulting to ./perf.data). Sort by overhead to find the hottest function, then expand to see its callers.

Expected result

A ranked list of functions, and with -g their call paths, by CPU time share, either live or from a file that can be reopened later without re-running the workload.

Limits and test basis

Sampling profilers miss code that never happens to be running at a sample tick; short spikes need a higher sample rate (-F) to catch. Many virtual machines expose no hardware performance counters; perf then falls back to a software CPU-clock event, which still profiles CPU time but cannot report cycle or cache counters. Restore perf_event_paranoid to its prior value once done if it was lowered for the session. Symbol resolution needs matching debug information for every binary in the call stack, not only the top frame.