Checking server time synchronisation: timedatectl, chronyc tracking and what to alert on

Goal

Know that every host's clock is disciplined against a trusted source and by how much it currently differs, and be alerted before the difference matters to applications.

Prerequisites

A time service on each host (systemd-timesyncd or chronyd are common on Linux), network access to NTP servers or an internal time source, and a monitoring system that can run a command or scrape a metric.

Steps

1. Check that a service is active and considers itself synchronised. timedatectl prints "System clock synchronized" and "NTP service" lines; its manual documents timesync-status for systemd-timesyncd details and show-timesync for machine-readable output.
2. On chrony hosts run chronyc tracking. The chrony documentation explains the fields: the reference ID and name of the server currently synchronised to, the stratum (hops from a computer with an attached reference clock), the current offset between chronyd's NTP clock and the system clock, last and RMS offsets, frequency, skew and leap status. It states that a reference ID of 7F7F0101 with no name or address means the computer is not synchronised to any external source and has the local mode operating.
3. Read the stratum. RFC 5905 assigns primary servers stratum one and each level below a number one greater, and states that accuracy degrades as the stratum number increases; a host far down the hierarchy, or one reporting stratum 16 (unsynchronized in RFC 5905's table), is not a usable clock.
4. Export two facts per host as metrics: synchronised (boolean) and current offset in seconds. Alert when synchronised stays false for several polling intervals, and when the absolute offset exceeds a bound chosen from what the applications tolerate (certificate validity checks, token lifetimes, lease expiry).
5. After each reboot and image rebuild, verify the service is enabled; a host that boots from a wrong hardware clock with no service running drifts unnoticed.
6. In containers the clock is the host's: check the host, and make sure images do not run a second, conflicting time service.
7. Occasionally compare hosts pairwise: two hosts each "synchronised" to different upstreams but far apart point to a bad upstream.

Expected result

A per-host view of synchronisation state and offset, an alert that fires when a host silently loses its time source, and no incident whose first clue is timestamps that do not line up.

Limits and test basis

Offsets reported by the daemon are estimates relative to its sources; a fleet synchronised to one wrong source shows zero offset everywhere. Virtual machines can jump after suspend or migration; chrony's makestep directive in chrony.conf governs whether a large offset is stepped rather than slewed. Commands and fields follow the cited manuals; no acceptable-offset value is asserted.