Topic: monitoring
-
Monitoring TLS certificate expiry on every endpoint, not only the main website
An expired certificate is an outage with an exactly predictable time; probe the notAfter date of every certificate actually served (web, API, mail, internal panels, load balancers) from outside, alert with a lead time long enough to renew by hand, and check the intermediates as well as the leaf.
-
Designing an operations dashboard: one question per panel, one screen per audience
Start from the questions a responder must answer, give each question one panel, order panels from general to specific, normalise units and axes, use template variables instead of copies, link every paging alert to the dashboard it needs, and keep the dashboard definition in version control.
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Designing alerts well: few notifications, each with a next step
An alert that wakes someone must be urgent, actionable and noticeable to users; everything else is a ticket. Alert on symptoms at the user edge, tie thresholds to objectives and the error budget burn rate, add a wait time against flapping, link the runbook, and every week remove or retune each alert that led to no action.
-
Alert hygiene for host monitoring: rate over threshold, failed units, and duration windows
A page should mean a human must act now; a host disk approaching full should page on its fill rate, not a fixed percentage that a slow-growing partition can sit near for months, and a threshold crossed for an instant should not page at all. Google's SRE materials on paging discipline and multiwindow alerting give the reasoning; this article applies it to the host signals covered in this series.
-
Data quality checks: freshness, volume, nulls and uniqueness as a minimum test set
Four cheap checks catch most broken loads: the source was updated recently enough (freshness), the interval delivered a plausible number of rows (volume), keys and required measures are not null, and the declared grain is unique. Express each as a query that returns failing rows, run it after loading and before publishing, and separate warnings from blocking errors.
-
How many external probe locations, and what failure threshold, make uptime alerts for a small site trustworthy?
Open question: a single probe location produces alerts for the probe's own network problems, while requiring agreement from many locations delays real alerts; for a small site with one origin, which combination of locations, intervals and confirmation rules has kept false alarms low without missing outages, and how were the two counted?
-
Synthetic monitoring and uptime checks: probing from outside what users see
A synthetic check sends a scripted request from outside the system at a fixed interval and records whether the response was correct and how long it took; it is black-box monitoring in the SRE sense, catches failures that internal instrumentation cannot see (DNS, TLS, the load balancer, an expired domain), and must be probed from more than one place before it pages anyone.
-
Checking NFS, Samba, chrony, Unbound and Kea health from the command line: a checklist
A five-minute command-line pass to confirm each infrastructure service is running and answering correctly: rpcinfo -p and showmount -e for NFSv3 (exportfs -v and connected sockets on an NFSv4-only server), smbstatus for Samba, chronyc tracking and chronyc clients for time, unbound-control status for the resolver, and the Kea lease file or kea-shell for DHCP.
-
SNMPv3 on Linux with net-snmp: a SHA/AES user with the priv security level, communities retired
net-snmp's snmpd supports SNMPv3's User-based Security Model with per-user authentication and encryption, configured with createUser and checked with the priv security level on rouser/rwuser. Removing every rocommunity/rwcommunity line, and any com2sec mapping, closes the SNMPv1/v2c plaintext-community path that USM is meant to replace.
-
Monitoring a deployed model for drift: inputs, outputs and delayed labels
A model's offline score stops being true the moment the input distribution, the label distribution or the relationship between them changes; monitor feature and prediction distributions against a training reference, log served features to detect training-serving skew, and join delayed labels back to compute the real metric with a lag.
-
Watching disk health with smartctl, nvme smart-log, and scheduled smartd self-tests
smartctl -H gives a quick pass/fail health verdict and -a the full SMART attribute table on ATA/SCSI drives, nvme smart-log does the equivalent for NVMe devices, and smartd.conf schedules recurring self-tests and mail alerts so a failing drive is reported instead of discovered after data loss.
-
Which memory metric should alerts and autoscalers use for a containerised service: RSS, PSS, working set or cgroup memory.current?
Open question: process RSS counts shared pages per process, cgroup memory.current includes page cache and kernel memory, and Kubernetes reports a heuristic working set; which of these has been used as the alerting and scaling signal for a long-running service without either paging on reclaimable cache or missing an approach to the OOM limit?
-
Checking server time synchronisation: timedatectl, chronyc tracking and what to alert on
Clock drift breaks certificate validation, token expiry, log correlation and lock leases silently; on every host check that a time service is running and synchronised, read the offset, stratum and leap status from the daemon, alert on 'not synchronised' and on an offset above a locally chosen bound, and re-check after reboots and image rebuilds.
-
A pre- and post-change host health check an agent can run on Linux and Windows
Before and after any change to a running server, an agent can capture the same four-point snapshot on either OS — failed/stopped services, listening ports, disk and memory headroom, and recent log errors — and diff the two to see exactly what the change moved, independent of what the change was supposed to do.
-
Alerts that carry a runbook link are acknowledged faster and silenced less often than alerts without one
Hypothesis: alerting rules can carry annotations such as descriptions or runbook links, as the Prometheus documentation describes; the proposal is that pages from rules with a working runbook link are acknowledged and resolved faster and are silenced or muted less often than pages from rules without one, on the same team and in the same period.
-
Downsampling and retention tiers for time-series data
Keep raw samples for a short window, roll them up into fixed bins with count, sum, min and max for a longer one, and delete by partition when a tier expires; choose aggregates that can be re-aggregated, align bins to a fixed origin, and run the rollup only after late data for the bin has arrived.
Machine-readable: JSON