Summarizing time in system calls with strace -c, its overhead, and perf trace as an alternative

Goal

Get a ranked summary of which system calls a process spends time in, rather than a scrolling live trace, and know when the tracing itself becomes the bottleneck.

Prerequisites

The strace package. Tracing a command you start (strace -c <command>) needs no extra privilege; attaching to a running process with -p needs root or CAP_SYS_PTRACE unless the Yama kernel.yama.ptrace_scope setting allows it (at the common value 1, an unprivileged user may attach only to its own descendants). perf if step 4's alternative is needed.

Steps

1. Attach for a bounded time, then detach, for example timeout 20 strace -c -f -p <pid>; without -f, -p attaches only to the one thread whose ID is <pid>, missing a multi-threaded server's worker threads. strace's manual describes -c/--summary-only as counting time, calls and errors for each system call and reporting a summary on exit, suppressing the regular per-call output — the aggregate view a "which syscall dominates" question needs — and notes this shows system time (kernel CPU time), independent of wall-clock time.
2. Read the summary sorted by time percentage first. Because the default measure is kernel CPU time, calls that mostly block (futex, epoll_wait, a socket read) look cheap in it; add -w/--summary-wall-clock to rank calls by wall-clock time from entry to exit when the question is where the process waits. Heavy futex or epoll_wait time then points at contention or idle waiting, read/write at I/O.
3. Know the cost before trusting the numbers: strace's manual states plainly that a traced process runs more slowly than a non-traced one, and that the performance impact can be mitigated with the --seccomp-bpf option. That option only takes effect together with -f, is not applicable to processes attached with -p, and only saves stops when a subset of calls is traced (-e trace=...), so it does not help the attach-and-count case above. For a latency-sensitive process, this overhead can itself change the behaviour being observed.
4. Where that overhead matters, use perf trace instead. Its manual introduces it as a strace-inspired tool that reports the same kind of syscall and system-event activity, live or from a previously recorded perf record session, built on the kernel's tracepoint/perf_events infrastructure rather than ptrace. Try perf trace -p <pid> for a live view, or perf trace -s -p <pid> for a per-thread summary with minimum, average and maximum times; it normally needs root.
5. Do not assume a fixed percentage difference between the two without measuring on the actual workload; neither manual states a numeric overhead figure, and the relative cost depends on how syscall-heavy the workload is.

Expected result

A sorted list of which system calls consumed the most time and how often they were called, obtained without a full line-by-line trace, plus an informed choice between strace -c's simplicity and perf trace's different overhead profile.

Limits and test basis

Both tools only see activity after attaching; an earlier spike is missed. -f/--follow-forks is needed to also count children's calls and, with -p, the other threads. Detach promptly once the summary is captured, since the overhead applies for the whole attached duration, not only while output is being read.