Reading the load average and understanding the OOM killer

What it is

The first three fields of /proc/loadavg are, per proc_loadavg(5), the number of jobs in the run queue (state R) or waiting for disk I/O (state D), averaged over 1, 5 and 15 minutes. The number is not a percentage: it is meaningful only relative to the count of CPUs (nproc). Because D-state tasks count, a stalled NFS mount or a saturated disk raises the load while CPUs sit idle.

The kernel's memory documentation describes the other half. Linux overcommits memory by default (vm.overcommit_memory 0: obvious overcommits are refused, everything else is granted). When allocation fails and reclaim cannot free enough, the OOM killer scans the task list and kills the task with the highest badness score, normally the largest memory consumer, unless oom_kill_allocating_task is set, in which case the task that triggered the condition is killed. proc_pid_oom_score_adj(5) explains that /proc/<pid>/oom_score_adj shifts the score in the range -1000 (OOM_SCORE_ADJ_MIN, never killed) to +1000.

Why it matters

Both numbers are misread routinely. Load 8 on an 8-CPU host is full utilisation, on a 2-CPU host it is severe queueing, and on either it may be disk, not CPU. An OOM kill removes the biggest process, which is usually the main service, not the leaking helper, and leaves only a kernel log line as evidence.

How to apply

• Compare load with nproc, then split the cause: vmstat 1 shows the r and b columns and CPU idle; b (blocked) with idle CPU means I/O; top shows D-state processes.
• After a suspected OOM kill, read dmesg -T or journalctl -k for "Out of memory: Killed process"; with oom_dump_tasks at its default of 1, the log also lists every task with its oom_score_adj and memory sizes at that moment.
• Protect the primary service with OOMScoreAdjust=-500 or similar in its systemd unit and give batch jobs a positive value; better, bound the batch job with MemoryMax= so it is killed inside its own cgroup first.
• Watch MemAvailable in /proc/meminfo, not "free", because page cache is reclaimable.

Pitfalls

Setting -1000 on everything important just moves the kill to whatever is left, possibly sshd. overcommit_memory=2 (the "never overcommit" policy) moves the failure to allocation time, where many programs handle an error return badly, and refuses the large just-in-case requests that the kernel documentation says many programs make. A load average dominated by D-state tasks does not respond to adding CPUs.

When the primary service is the one that grows

On a single-purpose host the largest process is usually the one that leaks, and a strongly negative oom_score_adj on it makes the kernel kill everything else first while the leak continues. Bound the primary service as well: MemoryMax= in its unit, set below physical memory with room for the system and helpers, so an out-of-memory condition is resolved inside the service's own cgroup and ends in a restart (Restart=on-failure) rather than in the loss of sshd or the monitoring agent. MemoryHigh= slightly below the maximum throttles first and shows up in metrics, which turns a surprise kill into a warning. Keep the negative adjustment small, and reserve -1000 for nothing that can grow.