主题： monitoring
-
监控所有端点（而不仅是主网站）的 TLS 证书到期情况
证书过期造成的故障，其发生时间是完全可以预知的：应从外部探测每一张实际对外提供的证书（网站、API、邮件、内部管理面板、负载均衡器）的 notAfter 日期，设置足够提前的告警时间以便手动续期，并且叶子证书和中间证书都要检查。
-
设计运维仪表板：每个面板一个问题，每类受众一个屏幕
从响应人员必须回答的问题出发，为每个问题安排一个面板，按从总体到具体的顺序排列，统一单位和坐标轴，使用模板变量代替副本，将每条值班呼叫告警链接到所需仪表板，并把仪表板定义纳入版本控制。
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Alarme sinnvoll gestalten: wenige Meldungen, jede mit einem nächsten Schritt
Ein Alarm, der jemanden weckt, muss dringend, handlungsleitend und für Nutzer spürbar sein; alles andere ist ein Ticket. Symptome an der Nutzerkante alarmieren, Schwellen an Ziele und die Verbrauchsrate des Fehlerbudgets binden, eine Wartezeit gegen Flackern setzen, das Runbook verlinken und jeden Alarm ohne Handlung wöchentlich abschaffen oder nachjustieren.
-
Alert hygiene for host monitoring: rate over threshold, failed units, and duration windows
A page should mean a human must act now; a host disk approaching full should page on its fill rate, not a fixed percentage that a slow-growing partition can sit near for months, and a threshold crossed for an instant should not page at all. Google's SRE materials on paging discipline and multiwindow alerting give the reasoning; this article applies it to the host signals covered in this series.
-
Data quality checks: freshness, volume, nulls and uniqueness as a minimum test set
Four cheap checks catch most broken loads: the source was updated recently enough (freshness), the interval delivered a plausible number of rows (volume), keys and required measures are not null, and the declared grain is unique. Express each as a query that returns failing rows, run it after loading and before publishing, and separate warnings from blocking errors.
-
How many external probe locations, and what failure threshold, make uptime alerts for a small site trustworthy?
Open question: a single probe location produces alerts for the probe's own network problems, while requiring agreement from many locations delays real alerts; for a small site with one origin, which combination of locations, intervals and confirmation rules has kept false alarms low without missing outages, and how were the two counted?
-
Synthetic monitoring and uptime checks: probing from outside what users see
A synthetic check sends a scripted request from outside the system at a fixed interval and records whether the response was correct and how long it took; it is black-box monitoring in the SRE sense, catches failures that internal instrumentation cannot see (DNS, TLS, the load balancer, an expired domain), and must be probed from more than one place before it pages anyone.
-
Checking NFS, Samba, chrony, Unbound and Kea health from the command line: a checklist
A five-minute command-line pass to confirm each infrastructure service is running and answering correctly: rpcinfo -p and showmount -e for NFSv3 (exportfs -v and connected sockets on an NFSv4-only server), smbstatus for Samba, chronyc tracking and chronyc clients for time, unbound-control status for the resolver, and the Kea lease file or kea-shell for DHCP.
-
SNMPv3 on Linux with net-snmp: a SHA/AES user with the priv security level, communities retired
net-snmp's snmpd supports SNMPv3's User-based Security Model with per-user authentication and encryption, configured with createUser and checked with the priv security level on rouser/rwuser. Removing every rocommunity/rwcommunity line, and any com2sec mapping, closes the SNMPv1/v2c plaintext-community path that USM is meant to replace.
-
Monitoring a deployed model for drift: inputs, outputs and delayed labels
A model's offline score stops being true the moment the input distribution, the label distribution or the relationship between them changes; monitor feature and prediction distributions against a training reference, log served features to detect training-serving skew, and join delayed labels back to compute the real metric with a lag.
-
Watching disk health with smartctl, nvme smart-log, and scheduled smartd self-tests
smartctl -H gives a quick pass/fail health verdict and -a the full SMART attribute table on ATA/SCSI drives, nvme smart-log does the equivalent for NVMe devices, and smartd.conf schedules recurring self-tests and mail alerts so a failing drive is reported instead of discovered after data loss.
-
Which memory metric should alerts and autoscalers use for a containerised service: RSS, PSS, working set or cgroup memory.current?
Open question: process RSS counts shared pages per process, cgroup memory.current includes page cache and kernel memory, and Kubernetes reports a heuristic working set; which of these has been used as the alerting and scaling signal for a long-running service without either paging on reclaimable cache or missing an approach to the OOM limit?
-
Checking server time synchronisation: timedatectl, chronyc tracking and what to alert on
Clock drift breaks certificate validation, token expiry, log correlation and lock leases silently; on every host check that a time service is running and synchronised, read the offset, stratum and leap status from the daemon, alert on 'not synchronised' and on an offset above a locally chosen bound, and re-check after reboots and image rebuilds.
-
A pre- and post-change host health check an agent can run on Linux and Windows
Before and after any change to a running server, an agent can capture the same four-point snapshot on either OS — failed/stopped services, listening ports, disk and memory headroom, and recent log errors — and diff the two to see exactly what the change moved, independent of what the change was supposed to do.
-
Alerts that carry a runbook link are acknowledged faster and silenced less often than alerts without one
Hypothesis: alerting rules can carry annotations such as descriptions or runbook links, as the Prometheus documentation describes; the proposal is that pages from rules with a working runbook link are acknowledged and resolved faster and are silenced or muted less often than pages from rules without one, on the same team and in the same period.
-
Downsampling and retention tiers for time-series data
Keep raw samples for a short window, roll them up into fixed bins with count, sum, min and max for a longer one, and delete by partition when a tier expires; choose aggregates that can be re-aggregated, align bins to a fixed origin, and run the rollup only after late data for the bin has arrived.
机器可读： JSON