주제: monitoring
-
메인 웹사이트뿐 아니라 모든 엔드포인트에서 TLS 인증서 만료 모니터링하기
만료된 인증서는 정확히 예측 가능한 시각에 발생하는 장애입니다. 실제로 서비스되는 모든 인증서(웹, API, 메일, 내부 관리 패널, 로드밸런서)의 notAfter 날짜를 외부에서 점검하고, 수동으로 갱신하기에 충분한 리드타임을 두고 경고를 울리며, 리프 인증서뿐 아니라 중간 인증서도 함께 확인해야 합니다.
-
Designing an operations dashboard: one question per panel, one screen per audience
Start from the questions a responder must answer, give each question one panel, order panels from general to specific, normalise units and axes, use template variables instead of copies, link every paging alert to the dashboard it needs, and keep the dashboard definition in version control.
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Alarme sinnvoll gestalten: wenige Meldungen, jede mit einem nächsten Schritt
Ein Alarm, der jemanden weckt, muss dringend, handlungsleitend und für Nutzer spürbar sein; alles andere ist ein Ticket. Symptome an der Nutzerkante alarmieren, Schwellen an Ziele und die Verbrauchsrate des Fehlerbudgets binden, eine Wartezeit gegen Flackern setzen, das Runbook verlinken und jeden Alarm ohne Handlung wöchentlich abschaffen oder nachjustieren.
-
Data quality checks: freshness, volume, nulls and uniqueness as a minimum test set
Four cheap checks catch most broken loads: the source was updated recently enough (freshness), the interval delivered a plausible number of rows (volume), keys and required measures are not null, and the declared grain is unique. Express each as a query that returns failing rows, run it after loading and before publishing, and separate warnings from blocking errors.
-
How many external probe locations, and what failure threshold, make uptime alerts for a small site trustworthy?
Open question: a single probe location produces alerts for the probe's own network problems, while requiring agreement from many locations delays real alerts; for a small site with one origin, which combination of locations, intervals and confirmation rules has kept false alarms low without missing outages, and how were the two counted?
-
Synthetic monitoring and uptime checks: probing from outside what users see
A synthetic check sends a scripted request from outside the system at a fixed interval and records whether the response was correct and how long it took; it is black-box monitoring in the SRE sense, catches failures that internal instrumentation cannot see (DNS, TLS, the load balancer, an expired domain), and must be probed from more than one place before it pages anyone.
-
Monitoring a deployed model for drift: inputs, outputs and delayed labels
A model's offline score stops being true the moment the input distribution, the label distribution or the relationship between them changes; monitor feature and prediction distributions against a training reference, log served features to detect training-serving skew, and join delayed labels back to compute the real metric with a lag.
-
Which memory metric should alerts and autoscalers use for a containerised service: RSS, PSS, working set or cgroup memory.current?
Open question: process RSS counts shared pages per process, cgroup memory.current includes page cache and kernel memory, and Kubernetes reports a heuristic working set; which of these has been used as the alerting and scaling signal for a long-running service without either paging on reclaimable cache or missing an approach to the OOM limit?
-
Checking server time synchronisation: timedatectl, chronyc tracking and what to alert on
Clock drift breaks certificate validation, token expiry, log correlation and lock leases silently; on every host check that a time service is running and synchronised, read the offset, stratum and leap status from the daemon, alert on 'not synchronised' and on an offset above a locally chosen bound, and re-check after reboots and image rebuilds.
-
Alerts that carry a runbook link are acknowledged faster and silenced less often than alerts without one
Hypothesis: alerting rules can carry annotations such as descriptions or runbook links, as the Prometheus documentation describes; the proposal is that pages from rules with a working runbook link are acknowledged and resolved faster and are silenced or muted less often than pages from rules without one, on the same team and in the same period.
-
Downsampling and retention tiers for time-series data
Keep raw samples for a short window, roll them up into fixed bins with count, sum, min and max for a longer one, and delete by partition when a tier expires; choose aggregates that can be re-aggregated, align bins to a fixed origin, and run the rollup only after late data for the bin has arrived.
기계 판독 가능: JSON