Triaging a full Linux disk: df, du, deleted-but-open files and the journal

Goal

Find why a filesystem reports full and reclaim space without guessing or deleting the wrong thing.

Prerequisites

Root or sudo access; the mount point that is reporting full (from an application error or a monitoring alert).

Steps

1. Confirm which filesystem and which resource is exhausted: df -h shows space per mounted filesystem; df -i shows inode usage separately. A filesystem can be at 100% inode use with space free, in which case deleting large files will not help.
2. Find where the space went, without crossing into other mounted filesystems: du -x -h --max-depth=2 /var 2>/dev/null | sort -h. The -x/--one-file-system flag keeps du from descending into a different mounted filesystem under /var and inflating the total.
3. Check for space held by deleted files that a process still has open — these never show up in du because the directory entry is gone, only the inode remains until the last file descriptor closes. List them with lsof +L1, which reports open files whose link count has dropped to zero (deleted). The kernel documents the same state as the (deleted) suffix appended to the target when a mapped or open file is unlinked while still referenced.
4. For each entry from lsof +L1, note the PID and command, then restart or gracefully reload that process (for a systemd service: systemctl restart <unit>). The space is only returned to the filesystem once the last file descriptor closes; deleting the path again does nothing, since the directory entry is already gone. If the process must not be restarted, the open file can be truncated through its descriptor, : > /proc/<pid>/fd/<fd> (the FD number is in the lsof output), which frees the blocks at once; a process that keeps writing at its old offset then produces a sparse file, so a restart in the next maintenance window is still the clean fix.
5. If the journal is the largest consumer, check with journalctl --disk-usage, then bound it with journalctl --vacuum-size=500M or journalctl --vacuum-time=2weeks.
6. On ext2/3/4, check the reserved-block percentage kept aside from ordinary users (tune2fs -l /dev/sdX1 | grep -i reserved); temporarily lowering it with tune2fs -m 1 /dev/sdX1 frees space for non-root writers, at the cost of the safety margin it existed to provide.

Expected result

df -h and df -i both show headroom after step 6; lsof +L1 returns no more entries tied to services that should have released their files.

Limits and test basis

Based on df(1), du(1), lsof(8) and proc_pid_maps(5). Restarting a process to release deleted file handles causes a short outage of that process; back up or note its PID and command before restarting. Lowering ext4 reserved blocks is reversible with the same tune2fs -m command and the original percentage.