Тема: monitoring
-
Мониторинг истечения TLS-сертификатов на всех узлах, а не только на основном сайте
Истёкший сертификат — это отказ с абсолютно предсказуемым моментом наступления; опрашивайте дату notAfter каждого реально обслуживаемого сертификата (веб, API, почта, внутренние панели, балансировщики нагрузки) снаружи, настраивайте оповещение с запасом времени, достаточным для ручного продления, и проверяйте не только листовой сертификат, но и промежуточные.
-
Designing an operations dashboard: one question per panel, one screen per audience
Start from the questions a responder must answer, give each question one panel, order panels from general to specific, normalise units and axes, use template variables instead of copies, link every paging alert to the dashboard it needs, and keep the dashboard definition in version control.
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Alarme sinnvoll gestalten: wenige Meldungen, jede mit einem nächsten Schritt
Ein Alarm, der jemanden weckt, muss dringend, handlungsleitend und für Nutzer spürbar sein; alles andere ist ein Ticket. Symptome an der Nutzerkante alarmieren, Schwellen an Ziele und die Verbrauchsrate des Fehlerbudgets binden, eine Wartezeit gegen Flackern setzen, das Runbook verlinken und jeden Alarm ohne Handlung wöchentlich abschaffen oder nachjustieren.
-
Data quality checks: freshness, volume, nulls and uniqueness as a minimum test set
Four cheap checks catch most broken loads: the source was updated recently enough (freshness), the interval delivered a plausible number of rows (volume), keys and required measures are not null, and the declared grain is unique. Express each as a query that returns failing rows, run it after loading and before publishing, and separate warnings from blocking errors.
-
How many external probe locations, and what failure threshold, make uptime alerts for a small site trustworthy?
Open question: a single probe location produces alerts for the probe's own network problems, while requiring agreement from many locations delays real alerts; for a small site with one origin, which combination of locations, intervals and confirmation rules has kept false alarms low without missing outages, and how were the two counted?
-
Synthetic monitoring and uptime checks: probing from outside what users see
A synthetic check sends a scripted request from outside the system at a fixed interval and records whether the response was correct and how long it took; it is black-box monitoring in the SRE sense, catches failures that internal instrumentation cannot see (DNS, TLS, the load balancer, an expired domain), and must be probed from more than one place before it pages anyone.
-
Monitoring a deployed model for drift: inputs, outputs and delayed labels
A model's offline score stops being true the moment the input distribution, the label distribution or the relationship between them changes; monitor feature and prediction distributions against a training reference, log served features to detect training-serving skew, and join delayed labels back to compute the real metric with a lag.
-
Which memory metric should alerts and autoscalers use for a containerised service: RSS, PSS, working set or cgroup memory.current?
Open question: process RSS counts shared pages per process, cgroup memory.current includes page cache and kernel memory, and Kubernetes reports a heuristic working set; which of these has been used as the alerting and scaling signal for a long-running service without either paging on reclaimable cache or missing an approach to the OOM limit?
-
Checking server time synchronisation: timedatectl, chronyc tracking and what to alert on
Clock drift breaks certificate validation, token expiry, log correlation and lock leases silently; on every host check that a time service is running and synchronised, read the offset, stratum and leap status from the daemon, alert on 'not synchronised' and on an offset above a locally chosen bound, and re-check after reboots and image rebuilds.
-
Alerts that carry a runbook link are acknowledged faster and silenced less often than alerts without one
Hypothesis: alerting rules can carry annotations such as descriptions or runbook links, as the Prometheus documentation describes; the proposal is that pages from rules with a working runbook link are acknowledged and resolved faster and are silenced or muted less often than pages from rules without one, on the same team and in the same period.
-
Downsampling and retention tiers for time-series data
Keep raw samples for a short window, roll them up into fixed bins with count, sum, min and max for a longer one, and delete by partition when a tier expires; choose aggregates that can be re-aggregated, align bins to a fixed origin, and run the rollup only after late data for the bin has arrived.
Машиночитаемо: JSON