Recovery readiness before a hardening change: console access, a backup, a scheduled automatic rollback, and a change log

Goal

Reduce a network-, authentication- or firewall-affecting hardening change to something that self-reverts if it locks the operator out, instead of requiring physical or out-of-band access to fix.

Prerequisites

An out-of-band path to the host that does not depend on the change being made (serial/IPMI/hypervisor console, or a cloud provider's console access) confirmed reachable before starting; permission to schedule a one-shot job on the target OS.

Steps

1. Confirm out-of-band access works right now, before touching anything — not "it worked last month." If console access itself needs the change being tested (e.g. the change affects the jump host), stage the change from a session that does not depend on the thing being changed.
2. Back up exactly what is about to change: the specific config file, firewall rule set, or GPO export, with a timestamp in the filename, not just "recent backups exist somewhere."
3. Arm a scheduled, unconditional rollback before applying the risky change:
   • Linux, one-shot and transient (as root): systemd-run --unit=rollback-net --on-active=10min /usr/local/sbin/rollback-network.sh, using systemd's --on-active= timer to fire relative to now regardless of what happens to the current shell or SSH session; this creates rollback-net.timer and rollback-net.service.
   • Linux, classic queue: echo /usr/local/sbin/rollback-network.sh | at now + 10 minutes, using at, documented as a tool to "queue, examine, or delete jobs for later execution" — atq lists pending jobs, atrm cancels one once success is confirmed. The at package and a running atd are required and are missing on many minimal installs.
   • Windows: Register-ScheduledTask with a trigger built by New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(10), running a rollback script under a principal that does not depend on the change, e.g. New-ScheduledTaskPrincipal -UserId SYSTEM -RunLevel Highest.
4. Apply the hardening change.
5. Confirm success over the same channel that will be used going forward (not just the current session, which may be grandfathered in). If confirmed, cancel the scheduled rollback (atrm <job>, systemctl stop rollback-net.timer, or Unregister-ScheduledTask -TaskName <name> -Confirm:$false) and check that it is gone (atq, systemctl list-timers, Get-ScheduledTask).
6. Write the change, its timestamp, the backup location, and the rollback job ID to a change log before moving to the next task — this is what lets a later incident be traced to a specific, dated action.

Expected result

If the change breaks the access path being used, the scheduled job fires unattended and restores the prior configuration within the chosen window; if the change succeeds, the rollback is cancelled and never fires.

Limits and test basis

A scheduled rollback only helps if the rollback script itself is correct and was tested against the specific backup format taken in step 2 — test the rollback script in a non-production window at least once. Reboot behaviour differs: a transient systemd-run timer is lost at reboot, so a change that only takes effect after a restart is not covered by it; at jobs are kept in the spool directory and survive a reboot, so an overdue rollback can still run once atd starts again; a Windows scheduled task persists until it is unregistered. Check the atd/systemd/Task Scheduler state if a job seems not to have fired. None of these steps requires a reboot by itself.