Serial and out-of-band consoles for servers that keep access when the network is down

Goal

Keep a way into a server's console when its normal network path (SSH over the primary interface) is exactly what is broken — a misconfigured firewall, a dead network driver load, or a kernel that panics before bringing up networking.

Prerequisites

For a physical or virtualized serial line: a null-modem cable or hypervisor-provided serial device, and a terminal emulator on the other end (baud rate must match on both sides). For a cloud instance: the provider's serial/system console feature enabled ahead of time. For a physical server with a BMC: ipmitool and network or dedicated-port access to the BMC, configured in advance.

Steps

1. Add a serial console to the kernel command line so boot messages and emergency shells appear on it: console=tty0 console=ttyS0,115200 (device, baud rate; the kernel documentation names ttySx for "a serial port"). Output goes to every listed console, but the last one listed becomes /dev/console, where rescue and emergency shells prompt — so put the serial one last. Persist it with grubby --update-kernel=ALL --args="console=tty0 console=ttyS0,115200" (RHEL family) or in GRUB_CMDLINE_LINUX plus update-grub (Debian/Ubuntu); it takes effect at the next boot.
2. For the GRUB menu itself to appear on the line (needed to pick a fallback kernel remotely), set GRUB_TERMINAL="console serial" and GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0" in /etc/default/grub and regenerate grub.cfg.
3. Login prompt: systemd-getty-generator starts serial-getty@ttyS0.service automatically for a serial kernel console; for other lines enable it explicitly (systemctl enable --now serial-getty@ttyS1.service). The process on the line is agetty.
4. On AWS (Nitro-based instance types), serial console access is off by default: the account must enable it and IAM policies must grant it; it then works independently of the instance's network. Login needs an OS user with a password, since SSH keys do not apply there.
5. On Azure, Serial Console in the portal gives the equivalent access through the control plane; it requires boot diagnostics enabled and likewise a password-based local user.
6. For physical servers with a BMC, check SOL with ipmitool sol info, then connect with ipmitool -I lanplus -H <bmc-ip> -U <user> -E sol activate (-E reads the password from IPMI_PASSWORD, keeping it off the process list); sol activate requires the lanplus interface (IPMI v2.0). Leave with the escape sequence (default ~.); run sol deactivate if a previous session left SOL marked active.
7. Test every path once on a healthy system, including credentials and reachability of the BMC or cloud console independent of the host's own network.

Expected result

Boot-loader menu, kernel messages and a login prompt all appear on the serial line or provider console, so a boot hang or firewall lockout still leaves a usable console.

Limits and test basis

A serial console only shows text the kernel/userspace explicitly writes to that device; it does not substitute for journalctl after the fact if the console buffer scrolled past what a human could capture. BMC/SOL access is itself a credential to protect — treat the BMC network as sensitive as the host's own root access, since SOL, like a physical console, reaches the boot menu and rescue shells and is therefore root-equivalent in practice.