Testing backups properly: the restore drill

Goal

Prove that the existing backups produce a working system, show how long that takes and how much data is lost in the process, before an outage forces the answer.

Prerequisites

A backup that is not stored on the same storage medium as the original; a fresh target environment (container, virtual machine, second instance) that cannot reach the production system; access to the keys for encrypted backups from a source other than the backup itself; a time window and a person who keeps a log.

Steps

1. Record the expectation: which point in time is to be restored, which databases, files and configurations are included, and how success will be recognized (row counts, checksums, a login, a sample of real records)?
2. Select the backup as you would in a real emergency: only via the documented storage location and the documented credentials, not via the machine that created the backup.
3. Check the integrity of the archive before restoring anything; restic provides the check command for this, which with --read-data also reads the data blocks instead of only the structure.
4. Restore into the empty environment: the PostgreSQL documentation states that non-text dumps are restored with pg_restore and that the owner roles must exist beforehand. Details like this are missing from runbooks until the drill uncovers them. Note every command and every deviation.
5. Measure the result against the expectation from step 1: tables and row counts, file checksums, start the application and run a real function (login, report, order in test mode).
6. Record the time from start until the application is usable and compare it with the agreed recovery target; likewise record the gap between the backup time and the last production state as the actual data loss window.
7. File the findings: date, backup version, duration, deviations, open items. Correct the runbook, schedule the next drill, delete the target environment.

Expected result

A dated proof that the backup is complete, readable and can be restored in a known time, or a list of concrete defects (missing key, forgotten configuration file, missing role, outdated script) to be fixed before a real emergency.

Limits and test basis

The drill checks the backup at the chosen point in time, not every future one; repeat it after changes to the schema, encryption or storage location. A small-scale restore says little about the duration with large data volumes. The commands follow the cited PostgreSQL and restic documentation; this article makes no statement on the right frequency of the drill, for which the wiki has an open question.