Diagnosing CPU steal time on virtualized Linux hosts with mpstat and vmstat

Goal

Tell real CPU exhaustion apart from time a virtual machine's vCPU spent waiting for its host to schedule it, before chasing an application-level fix that cannot exist.

Prerequisites

A Linux guest running under a hypervisor (KVM, Xen, VMware, or a cloud VM) that exposes a steal-time clock to the guest (KVM and Xen do; not every hypervisor or guest kernel reports it); the sysstat package for mpstat, and procps for vmstat, both normally preinstalled or installable non-interactively (DEBIAN_FRONTEND=noninteractive apt-get install -y sysstat / dnf install -y sysstat).

Steps

1. Get a per-CPU breakdown over a short window: mpstat -P ALL 1 5. Sysstat's manual defines %steal as the percentage of time spent in involuntary wait by the virtual CPU while the hypervisor was servicing another virtual processor — time the guest's kernel wanted to run but could not, not time it spent doing anything.
2. Cross-check with a second tool: vmstat 1 5. The st column is documented as time stolen from a virtual machine; it should roughly track the all row of mpstat's %steal. Ignore vmstat's first line, which is an average since boot, not the current interval.
3. Read the pattern, not just the number: steal spread evenly across every CPU during the interval suggests host-wide scheduling contention; steal that only appears while your own workload is CPU-bound suggests the host is oversubscribed for that shape of load. Steal accrues only while a vCPU has work to run, so a mostly idle guest shows little steal even on a busy host.
4. Treat it as a placement or capacity question, not a code bug: nothing on the guest can show what sibling virtual machines were doing at the time. Sustained, material steal (well above brief single-digit jitter) is worth raising with whoever controls the hypervisor. On burstable cloud instance types, exhausted CPU credits can also appear as steal.
5. On bare metal, %steal and st are structurally zero, since no hypervisor exists to withhold CPU; a non-zero reading suggests the system is in fact a guest, contrary to any assumption otherwise.
6. If you control the hypervisor too, correlate with its own per-VM CPU-ready or scheduler-wait metrics, which confirm from the other side what the guest can only infer.

Expected result

A clear split between "the application used all the CPU it was given" (low steal, high us/sy) and "the application was denied CPU it asked for" (material, sustained steal), which changes the next step from profiling code to a capacity conversation.

Limits and test basis

Both commands are read-only; there is nothing to back up or undo. Neither tool can name the tenant responsible for contention, and some hypervisor configurations (CPU pinning, dedicated cores) never produce steal even under load, and some hypervisors do not report steal to the guest at all, so zero steal does not by itself prove there is no host-side contention. The definitions cited are the tools' own, not a universal severity threshold.