Kernel panics and crash data on Linux: the previous boot's journal, pstore and netconsole

What it is

When a Linux system panics and reboots, or is power-cycled after hanging, the evidence lives in a few places with different survival properties:

• The previous boot's journal: if journald's storage is persistent (Storage= in journald.conf), messages written up to the crash survive on disk. journalctl -b -1 -k selects the previous boot (-b -1) and kernel-only messages (-k/--dmesg: "Show only kernel messages"). Check this first; it needs no special configuration beyond a persistent journal.
• pstore: the kernel's pstore subsystem writes an oops or panic record to a small persistent backend before the machine goes down — ramoops (a RAM region reserved via kernel parameters or device tree, which survives a warm reset but not a power loss), UEFI variables, or platform firmware storage. It only works if such a backend is present and enabled. After the next boot the records appear as files under /sys/fs/pstore/; where systemd-pstore.service is enabled, it then writes them to the journal and, by default, moves them to /var/lib/systemd/pstore/.
• netconsole: a kernel module (or built-in driver) that sends kernel log messages over UDP to a listening machine in real time, configured with a netconsole= parameter of the form [src-port]@[src-ip]/[dev],[tgt-port]@<tgt-ip>/[tgt-mac]. Because it transmits as messages are generated, it can capture a panic that never reaches local storage at all — at the cost of needing a network driver that still works at panic time, a network path, and a listener already running; UDP delivery is not guaranteed.
• kdump, covered elsewhere in this series, is different again: it kexec's into a second, reserved kernel after a crash and writes a full memory dump (vmcore) for later analysis, rather than a log excerpt.

Why it matters

Each source answers a different question. The journal shows what was logged up to the crash, in a familiar format. pstore survives crashes so early or severe that nothing was flushed to disk, but holds only a small, bounded record. netconsole survives an unresponsive local disk but needs the network interface and a receiver. kdump gives the most complete picture but needs prior setup and is comparatively heavyweight for rare panics.

How to apply

• Start with journalctl -b -1 -k --no-pager after any unexplained reboot; an empty result or a short boot list means the journal was not persistent for that boot.
• After a crash, check /sys/fs/pstore/ and /var/lib/systemd/pstore/ (reading them needs root) even if the journal has nothing; a record there is direct evidence pstore caught the panic.
• Set up netconsole in advance on systems where panics are otherwise hard to diagnose, pointed at a machine already running a listener.
• Reserve kdump for systems where a full crash dump justifies the memory and setup cost, not just because pstore or netconsole are also configured.

Pitfalls

• Assuming the journal kept anything: with Storage=auto and no /var/log/journal/ directory it is volatile, and -b -1 simply shows nothing. mkdir -p /var/log/journal followed by systemctl restart systemd-journald makes it persistent from then on.
• Leaving old records in /sys/fs/pstore/ where nothing archives them: backends are small (UEFI variable space in particular), ramoops overwrites older records, so copy and remove records after each incident.