Topic: observability
-
Continuous profiling in production: always-on sampling profiles and what they answer
Continuous profiling takes CPU and memory profiles systematically over time and stores them as labelled series, so a team can ask which function consumed the most CPU across the fleet yesterday or what changed between two versions; sampling profilers make it cheap enough to leave on, and runtime endpoints such as Go's /debug/pprof/ or eBPF agents supply the profiles.
-
Which trace sampling strategy keeps rare failures visible in a low-traffic service?
Open question: sampling guidance is written for services with thousands of traces per second, where one percent is still a representative sample; for a service with a few requests per second, which combination of head sampling, tail sampling, per-route rates and retention has kept the one failing trace a week available at a cost the team accepted?
-
Alerts that page for symptoms, not causes
Alert on what users experience (error rate, latency, availability, freshness) with thresholds tied to objectives, route by urgency, and turn every noisy alert into either a fix or a deletion.
-
Which observability signals should a JVM or .NET service emit by default, and at what overhead?
Open question: both runtimes ship built-in telemetry (Flight Recorder and GC logging on the JVM; EventPipe counters and dotnet-trace on .NET) and both have OpenTelemetry auto-instrumentation, but there is little shared evidence on which of these should be always-on in production, what they cost, and which ones actually shortened incidents.
-
Log sampling for high-volume events: keep every error, sample the repetitive lines
Sampling drops a fraction of similar log events on purpose; the useful forms are one-in-N, burst-then-rate per period, per-level rules that leave warnings and errors untouched, and pipeline sampling keyed on a request ID so a whole request is kept or dropped together, with the applied rate written into the surviving events.
-
Distributed tracing at a glance: spans, parent IDs, and W3C Trace Context
A trace is a tree of spans, each with a trace ID, its own span ID, a parent span ID, timestamps, attributes, and a status. The W3C traceparent header carries the trace ID, parent ID, and a sampled flag across process boundaries, and a service that merely forwards both headers still keeps traces intact.
-
journald forwarding and remote collection: ForwardToSyslog, systemd-journal-upload/-remote, and rate limits
journald can hand its entries to a local syslog daemon with ForwardToSyslog=, or ship them directly over HTTPS with systemd-journal-upload to a systemd-journal-remote listener that writes its own journal files. Both paths are subject to journald's own per-service rate limiting, which drops excess messages once a burst threshold is crossed and records only a count of what it dropped.
-
Designing an operations dashboard: one question per panel, one screen per audience
Start from the questions a responder must answer, give each question one panel, order panels from general to specific, normalise units and axes, use template variables instead of copies, link every paging alert to the dashboard it needs, and keep the dashboard definition in version control.
-
Windows Event Forwarding: a source-initiated subscription, wecutil, and the EventLog Readers group
In a source-initiated WEF subscription, each source computer pulls its configuration from a collector via Group Policy and pushes matching events to it; wecutil manages the subscription on the collector, and forwarding the Security log needs the NETWORK SERVICE account added to the EventLog Readers group on every source.
-
Use correlation IDs across an agent task
Separate task identity, individual attempts and distributed trace context so retries can be followed without logging sensitive payloads.
-
Designing alerts well: few notifications, each with a next step
An alert that wakes someone must be urgent, actionable and noticeable to users; everything else is a ticket. Alert on symptoms at the user edge, tie thresholds to objectives and the error budget burn rate, add a wait time against flapping, link the runbook, and every week remove or retune each alert that led to no action.
-
Alert hygiene for host monitoring: rate over threshold, failed units, and duration windows
A page should mean a human must act now; a host disk approaching full should page on its fill rate, not a fixed percentage that a slow-growing partition can sit near for months, and a threshold crossed for an instant should not page at all. Google's SRE materials on paging discipline and multiwindow alerting give the reasoning; this article applies it to the host signals covered in this series.
-
Replayable run logs for agents: recording every model and tool call
An agent run can only be debugged if every model request, response, tool call and tool result is recorded in order with identifiers and parameters; the OpenTelemetry GenAI semantic conventions name the fields, and a replayable log lets a failure be reproduced without paying for a new run.
-
Metric naming and label cardinality: units in the name, bounded values in the labels
Prometheus naming conventions put an application prefix, a base unit and a _total suffix for counters into the metric name and reserve labels for bounded dimensions; every distinct label combination is its own time series, so user IDs, raw URLs and error messages in labels multiply storage until the server degrades.
-
Forwarding syslog with rsyslog over TCP behind a disk-assisted queue
A drop-in file under /etc/rsyslog.d/ can forward every message to a central collector over TCP while a disk-assisted action queue buffers messages locally, so a collector outage or network blip does not drop logs. rsyslogd -N1 validates the syntax before the change is applied.
-
Correlation versus causation in incident and operations data
A correlation coefficient measures how two series move together; it says nothing about which one drives the other, whether a third factor such as traffic drives both, or whether the data were selected by the outcome. Plot first, condition on the obvious common causes, check timing, and confirm with an intervention such as a flag or canary before acting.
-
The RED method for request-driven services, with exemplars that link a slow bucket to a trace
Instrument every request-handling service identically with rate, errors and duration: one counter with route, method and status labels and one duration histogram; when tracing exists, attach exemplars (a trace ID with a recorded value) to the histogram so a slow bucket on the dashboard opens the trace that landed there.
-
Log rotation and retention limits
Logs must be bounded in size and age at every layer (application, container runtime, proxy, system journal), with retention chosen for debugging and legal needs rather than 'keep everything'.
-
Synthetic monitoring and uptime checks: probing from outside what users see
A synthetic check sends a scripted request from outside the system at a fixed interval and records whether the response was correct and how long it took; it is black-box monitoring in the SRE sense, catches failures that internal instrumentation cannot see (DNS, TLS, the load balancer, an expired domain), and must be probed from more than one place before it pages anyone.
-
Configuring the logging module once, at the entry point
Libraries obtain loggers and never configure handlers; the application configures handlers, levels and formats once at start-up, so that output is controlled in one place and duplicates are avoided.
Machine-readable: JSON