Bootstrapping NTP, DNS and DHCP after a full network outage: a dependency-ordered restart

Goal

Restart core infrastructure services on a small network in an order that lets each one start correctly, after a full outage where NTP, DNS and DHCP were all down together.

Prerequisites

Knowledge of which hosts run which service; console or out-of-band access to each host, since network-dependent remote access may not work yet; the hosts' own basic connectivity already restored (power, physical or virtual network links up); the NTP, DNS and DHCP servers themselves on static addresses, not on DHCP.

Steps

1. Bring up time first, on the host(s) that serve NTP. Certificate validation, Kerberos ticket lifetimes and log timestamp correlation across the rest of the recovery all depend on clocks being close to correct; a server whose clock drifted far during the outage should be allowed to step or slew before anything that checks certificate validity periods depends on it. A DNSSEC-validating resolver also fails with a badly wrong clock. The dependency is circular, though: an NTP server whose upstreams are hostnames (a pool name, NTS servers) cannot reach them until DNS works, so it serves only its hardware-clock time, or its local reference if configured. Give it at least one upstream by IP address, or accept that time is corrected again once DNS is up.
2. Bring up DNS next, on the host(s) that serve it. Many services locate each other, and validate TLS certificates, by name; starting DNS before DHCP means that even statically configured hosts — including the DHCP server itself, if its own startup resolves anything by name — can already resolve.
3. Bring up DHCP last of the three. Clients that lost their lease need DHCP to hand out addresses again; the DHCP server may benefit from correct time (lease timestamps, logs) and needs DNS only for dynamic DNS updates, if configured.
4. Recheck each service from a client before declaring the network recovered: a time check (does the client's clock converge toward the NTP server), a name lookup for an internal and an external name, and a DHCP lease renewal on a spare client.
5. Only after these three are confirmed, restart services that depend on all of them: authentication (Kerberos/LDAP, which need correct time and resolvable names), and application services that use short-lived certificates or service discovery by name.

Expected result

Each service should start without waiting on, or silently failing because of, one of the other two; clients should regain full connectivity in the same order the services came up, without a second round of restarts caused by a missed dependency.

Limits and test basis

This is a proposed general ordering, not a substitute for a site-specific runbook: a network where the DHCP server's own address is static and DNS is provided by the same host as NTP has fewer real dependencies to sequence. Virtualised or containerised deployments where all three run as application services behind their own load balancer may have additional dependencies — the hypervisor's or orchestrator's own control plane — that must come up before any of the three. No experiment or field measurement is claimed for this ordering; it is reasoned from the usual dependencies of these services on time and name resolution, and a given site may have fewer or different ones.