Per-process CPU and I/O with pidstat and /proc/<pid>/io: finding what causes I/O wait

Goal

Identify the single process responsible for CPU use or disk I/O that a system-wide tool (top, iostat) can only show in aggregate.

Prerequisites

The sysstat package for pidstat; read access to /proc/<pid>/io for the target process, which the kernel governs with a ptrace read-access check (in practice the process's own user, or root for another user's process); pidstat -d likewise reports other users' tasks only when run as root.

Steps

1. Start with CPU by process: pidstat 1 5 lists per-task CPU percentages. Sysstat's manual describes pidstat as reporting statistics for individual tasks currently being managed by the kernel, avoiding the need to filter a name out of top.
2. Add memory and context switches in the same pass: pidstat -r -w 1 5 (-r memory, -w context switching), useful when CPU alone does not explain the load.
3. Move to disk I/O per process: pidstat -d 1 5 reports kilobytes read and written per second, and I/O delay (iodelay), per task. iodelay depends on kernel delay accounting, which is off by default since Linux 5.14 (enable with sysctl kernel.task_delayacct=1 or the delayacct boot parameter); without it the column stays at zero.
4. For a specific suspect PID, read the raw counters directly: cat /proc/<pid>/io. proc_pid_io(5) documents rchar/wchar as the bytes returned by successful read(2)/write(2) and similar calls — which includes page-cache hits, pipes, sockets and terminals — and read_bytes/write_bytes as bytes really fetched from or sent to the storage layer, closer to what iostat sees at the device. write_bytes is charged when the process dirties page-cache pages, so the physical write may reach the device later, during writeback.
5. Sample /proc/<pid>/io twice with a known interval and subtract to get a rate, the way pidstat -d derives its own numbers; this also works for processes started before you began watching.
6. Correlate: a process whose read_bytes/write_bytes grow during exactly the window iostat -xz showed a saturated device is the one to investigate next, for example via its open files (ls -l /proc/<pid>/fd).

Expected result

A specific PID and command line responsible for the load seen system-wide, in the same units (KB/s) the device-level tools already reported.

Limits and test basis

rchar/wchar count logical I/O including cache hits and non-file I/O, so they can be far larger than physical device traffic; use read_bytes/write_bytes when the question is how much the process made the disk do. Both pidstat and reading /proc/<pid>/io are read-only observations with nothing to undo; a process that exits between two samples simply disappears from the next reading.