Disk I/O latency with iostat -x: r_await, w_await, aqu-sz, and why %util misleads on SSD and RAID

What it is

iostat -x (sysstat) adds extended per-device columns to the plain read/write throughput view. Its manual defines the fields most useful for latency work: r_await and w_await are the average time in milliseconds for read and write requests to be served, including time spent queued and time spent being serviced; aqu-sz is the average queue length of requests issued to the device; %util is the percentage of elapsed time during which I/O requests were issued to the device.

Column names depend on the sysstat version. Older releases (for example sysstat 10.x on RHEL 7) print avgqu-sz, avgrq-sz (in sectors) and svctm, a field their own manual already said not to trust; current releases call these aqu-sz and areq-sz/rareq-sz/wareq-sz (in kilobytes) and no longer document svctm. Whether a combined await column appears beside r_await/w_await in -x output also varies by version, so scripts should match fields by header name, not column position.

Why it matters

Splitting await into read and write matters because the two can differ enormously on the same device — for example sequential writes but random reads on a spinning disk — and a single blended average hides that. aqu-sz counts requests outstanding at the device, including those being serviced. On a device that serves one request at a time, a value persistently above roughly one means requests are queuing; SSDs, NVMe and RAID sets serve many requests in parallel, so a higher aqu-sz there is normal and only matters when it rises together with await. %util was a reliable saturation signal on a device that can serve only one request at a time; the manual states plainly that device saturation occurs when it is close to 100% for devices serving requests serially, but for devices serving requests in parallel — RAID arrays and modern SSDs are its own examples — this number does not reflect their performance limits, since such a device can sit at 100% (never idle) while nowhere near its real throughput or latency ceiling.

How to apply

• Read r_await/w_await first for user-visible latency, judged against the storage's own expected range rather than one fixed number across dissimilar devices.
• Watch aqu-sz trend upward across consecutive samples alongside rising await, judged against what is normal for that device.
• Treat %util as an activity indicator, not a saturation percentage, on SSD, NVMe and RAID; a fast device parked at 100% with low await is not the bottleneck.
• Sample repeatedly (iostat -xz 1 10) rather than once; the first report covers the time since boot, not the sampling interval (-y omits it).
• Narrow to one device (iostat -x <device> 1) once a candidate emerges from the full list.

Pitfalls

• Alerting on %util alone on modern storage and missing real queuing that aqu-sz would have shown.
• Comparing await across dissimilar devices (network storage, local NVMe, a RAID mid-rebuild) as if one threshold applied to all of them.
• Forgetting that the first row after starting iostat is a lifetime average, not the current state.