主题： observability
-
生产环境中的持续性能剖析：常开的采样剖析能回答什么问题
持续性能剖析（continuous profiling）会系统性地随时间采集 CPU 和内存剖析数据，并以带标签的序列形式存储，这样团队就能查询昨天整个集群中哪个函数消耗的 CPU 最多、或两个版本之间发生了什么变化之类的问题；采样式剖析器的开销足够低，可以一直保持开启，而 Go 的 /debug/pprof/ 之类的运行时端点或 eBPF 代理则负责提供这些剖析数据。
-
在低流量服务中，哪种链路采样策略能让罕见故障依然可见？
开放问题：现有的采样指南大多针对每秒成千上万条链路（trace）的服务而写，在那种规模下，1% 的采样比例仍具有代表性；但对于每秒只有几个请求的服务，头部采样（head sampling）、尾部采样（tail sampling）、按路由设置的采样率与数据保留策略要如何组合，才能以团队可以接受的成本，保住那条每周才出现一次的故障链路？
-
Alerts that page for symptoms, not causes
Alert on what users experience (error rate, latency, availability, freshness) with thresholds tied to objectives, route by urgency, and turn every noisy alert into either a fix or a deletion.
-
Which observability signals should a JVM or .NET service emit by default, and at what overhead?
Open question: both runtimes ship built-in telemetry (Flight Recorder and GC logging on the JVM; EventPipe counters and dotnet-trace on .NET) and both have OpenTelemetry auto-instrumentation, but there is little shared evidence on which of these should be always-on in production, what they cost, and which ones actually shortened incidents.
-
Log sampling for high-volume events: keep every error, sample the repetitive lines
Sampling drops a fraction of similar log events on purpose; the useful forms are one-in-N, burst-then-rate per period, per-level rules that leave warnings and errors untouched, and pipeline sampling keyed on a request ID so a whole request is kept or dropped together, with the applied rate written into the surviving events.
-
Verteiltes Tracing in Umrissen: Spans, Eltern-Kennungen und W3C Trace Context
Ein Trace ist ein Baum aus Spans, jeder mit Trace-Kennung, eigener Span-Kennung, Eltern-Span-Kennung, Zeitstempeln, Attributen und Status; der W3C-Header traceparent trägt Trace-Kennung, Eltern-Kennung und ein Sampled-Flag über Prozessgrenzen, und ein Dienst, der beide Header nur weiterreicht, hält Traces trotzdem zusammen.
-
Designing an operations dashboard: one question per panel, one screen per audience
Start from the questions a responder must answer, give each question one panel, order panels from general to specific, normalise units and axes, use template variables instead of copies, link every paging alert to the dashboard it needs, and keep the dashboard definition in version control.
-
Use correlation IDs across an agent task
Separate task identity, individual attempts and distributed trace context so retries can be followed without logging sensitive payloads.
-
Alarme sinnvoll gestalten: wenige Meldungen, jede mit einem nächsten Schritt
Ein Alarm, der jemanden weckt, muss dringend, handlungsleitend und für Nutzer spürbar sein; alles andere ist ein Ticket. Symptome an der Nutzerkante alarmieren, Schwellen an Ziele und die Verbrauchsrate des Fehlerbudgets binden, eine Wartezeit gegen Flackern setzen, das Runbook verlinken und jeden Alarm ohne Handlung wöchentlich abschaffen oder nachjustieren.
-
Replayable run logs for agents: recording every model and tool call
An agent run can only be debugged if every model request, response, tool call and tool result is recorded in order with identifiers and parameters; the OpenTelemetry GenAI semantic conventions name the fields, and a replayable log lets a failure be reproduced without paying for a new run.
-
Metric naming and label cardinality: units in the name, bounded values in the labels
Prometheus naming conventions put an application prefix, a base unit and a _total suffix for counters into the metric name and reserve labels for bounded dimensions; every distinct label combination is its own time series, so user IDs, raw URLs and error messages in labels multiply storage until the server degrades.
-
Correlation versus causation in incident and operations data
A correlation coefficient measures how two series move together; it says nothing about which one drives the other, whether a third factor such as traffic drives both, or whether the data were selected by the outcome. Plot first, condition on the obvious common causes, check timing, and confirm with an intervention such as a flag or canary before acting.
-
The RED method for request-driven services, with exemplars that link a slow bucket to a trace
Instrument every request-handling service identically with rate, errors and duration: one counter with route, method and status labels and one duration histogram; when tracing exists, attach exemplars (a trace ID with a recorded value) to the histogram so a slow bucket on the dashboard opens the trace that landed there.
-
Log rotation and retention limits
Logs must be bounded in size and age at every layer (application, container runtime, proxy, system journal), with retention chosen for debugging and legal needs rather than 'keep everything'.
-
Synthetic monitoring and uptime checks: probing from outside what users see
A synthetic check sends a scripted request from outside the system at a fixed interval and records whether the response was correct and how long it took; it is black-box monitoring in the SRE sense, catches failures that internal instrumentation cannot see (DNS, TLS, the load balancer, an expired domain), and must be probed from more than one place before it pages anyone.
-
Configuring the logging module once, at the entry point
Libraries obtain loggers and never configure handlers; the application configures handlers, levels and formats once at start-up, so that output is controlled in one place and duplicates are avoided.
-
Latency percentiles: why the average describes no real request
Latency distributions are skewed, so the mean sits between a fast majority and a slow tail and matches no actual request; p50, p99 and the maximum describe what users meet. Record histograms rather than pre-computed quantiles so percentiles can be aggregated across instances and recomputed for any window.
-
How should a dashboard show the uncertainty of a metric so that operators react to signal rather than noise?
Open question: dashboards draw a percentage from three requests with the same confidence as one from three million, and a p99 from a sparse histogram bucket as a precise line; which ways of showing sample counts, interval bands or estimation error have been shown to reduce false alarms and missed problems for on-call operators?
-
SLIs for queues and batch jobs: age of the oldest message, freshness, coverage and last success
Request-driven services measure availability and latency; queues and batch jobs need different indicators: how old the oldest unprocessed item is, what proportion of data is fresher than a threshold, what proportion of scheduled runs completed within their window, and when the job last succeeded. This methodology derives them from the pipeline SLIs in the SRE workbook.
-
Distributed tracing in outline: spans, parent IDs and W3C trace context propagation
A trace is a tree of spans, each with a trace ID, its own span ID, a parent span ID, timestamps, attributes and a status; the W3C traceparent header carries trace ID, parent ID and a sampled flag across process boundaries, and a service that only forwards both headers still keeps traces intact.
机器可读： JSON