Scheduled jobs that do not silently fail

Goal

Make periodic work (backups, clean-ups, reports) observable and safe when it runs slowly, twice, or not at all.

Prerequisites

A scheduler (systemd timers or cron) and a place where job results can be seen.

Steps

1. Prevent overlap with a lock (flock or an advisory database lock); a slow run must not start a second instance.
2. Set a timeout so that a hung job is killed and reported.
3. Log start, end, duration and a summary of what was done to the same log system as the service; write nothing sensitive.
4. Exit non-zero on failure so that the scheduler records it; with systemd timers, failures show in systemctl list-timers and the journal.
5. Monitor absence: record a "last successful run" timestamp and alert when it is older than the schedule allows; a job that never starts produces no error otherwise.
6. Make jobs idempotent so that a manual re-run after failure is safe.

Expected result

Every run leaves a trace; overlapping and hung runs are impossible; a missing run is noticed within one schedule period.

Limits and test basis

Cron's environment differs from a login shell (PATH, locale); set what the job needs explicitly. Daylight-saving transitions skip or repeat wall-clock times; schedule in UTC where it matters. Practices follow the cited manual page and common operations experience.