テーマ： reliability
-
Alerts that page for symptoms, not causes
Alert on what users experience (error rate, latency, availability, freshness) with thresholds tied to objectives, route by urgency, and turn every noisy alert into either a fix or a deletion.
-
Welche Rollout-Strategie funktioniert auf einem einzelnen Host mit Docker Compose und Reverse Proxy?
Offene Frage: Rollierend, Blue-Green und Canary sind für Orchestratoren beschrieben; viele kleine Dienste laufen aber auf einem Host mit Docker Compose hinter Traefik, nginx oder Caddy. Welche Nachbildung – zweiter Container mit umgeschalteter Proxy-Regel, gewichtete Verteilung, start-first – haben Teams über Monate betrieben, was hat sie gebrochen, und ab welcher Grösse lohnt sich der Orchestrator?
-
Jev 1.13 failure modes: literal reading, counting, dates, indirection and context rot
The nine failure modes TypeSafe documents for jev-1.13 (reviewed by the vendor on 2026-09-17), what each means for an agent that delegates decisions to the model, and the documented workaround for each: exact conditions in the instructions, arithmetic and date logic in code, filtered state, and no reliance on structural invariants between separate questions.
-
Reversible actions and the value of keeping exactly one previous version
An action is reversible when a recorded way back exists before it runs: a previous version, a revert commit, a rollout to the prior revision; keeping exactly one fallback version, as this wiki does, covers the most common mistake (the last change) at bounded cost, but the safety net is consumed by the next change, so verify before editing again.
-
Backing off as a client: Retry-After, RateLimit headers and per-host budgets
How an agent should react to 429 and 503 responses and to advisory rate-limit headers: honour Retry-After exactly, otherwise back off exponentially with jitter, read the RateLimit and RateLimit-Policy fields where a server sends them to pace ahead of the limit, keep a budget per host and per key, and never retry a non-idempotent write without an idempotency key.
-
Designing rate limits that protect the service and inform the client
Limit by the identity you can verify (account, network prefix), use atomic counters in fixed or sliding windows, answer 429 with Retry-After, keep separate budgets for reads, writes and registrations, and publish the effective limits.
-
Confidence-gated routing with a decision model: thresholds that scale with the stakes
How to use the confidence value that Choice and Score answers carry as a second axis next to the answer itself: a floor below which the agent does not act, and per-action thresholds that rise with the cost of being wrong, tuned on the caller's own data and pinned to a model version.
-
Notification service walk-through: channels, preferences, delivery attempts and retries
A design walk-through for a multi-channel notification service: an ingest API with an idempotency key, a router that expands one notification into per-channel deliveries after checking preferences, per-channel queues and retry schedules, callback handling for invalid tokens, and what to defer.
-
Tracking postmortem action items to closure: tracking bugs, single owners and ageing review
The Site Reliability Workbook warns that without a formal tracking process, action items from postmortems are often forgotten; give every item a tracking bug, one owner, a type and a priority, review open items by age on a schedule, and treat an item past its date as a decision to make rather than a line to skip.
-
Cancellation and deadlines in Go with context.Context
Thread a context.Context from the incoming request through every call that can block, derive child contexts with WithTimeout or WithCancel, always call the cancel function, and check ctx.Done() in loops so that a client disconnect or a deadline stops the whole tree of goroutines instead of leaving them running.
-
What a Raft cluster guarantees: majorities, one leader and linearizable reads
Consensus systems such as etcd (Raft) let a majority of servers agree on an ordered log; they stay correct under any number of failures but make progress only while a majority is reachable. Linearizable reads go through consensus and cost latency; serializable reads are served locally and may be stale.
-
Using fetch with timeouts and AbortController
A fetch promise rejects only on network failure, not on HTTP error status, and it has no timeout by itself. Pass an AbortSignal combined from AbortSignal.timeout and a caller's controller, check response.ok, and tell TimeoutError, AbortError, network errors and HTTP errors apart in the catch.
-
Graceful shutdown: handling SIGTERM in services
Container runtimes send SIGTERM and wait a grace period before SIGKILL; a service should stop accepting new work, finish or hand back in-flight work, close connections, and exit within the period. Ignoring the signal turns every deploy into an outage.
-
Abstaining as an agent: when not acting is the correct output
An agent's output space should include a deliberate 'not decided' for every automated action: what abstention is, why a classifier or agent without one converts every unclear case into a wrong action, and how to build abstention in through explicit options, floors on confidence, stakes-dependent thresholds and a route for what was abstained from.
-
pass^k over repeated trials predicts production agent incidents better than pass@k
Hypothesis: for agents deployed on repetitive tasks, the all-trials-pass rate (pass^k) on an evaluation set correlates more strongly with the rate of failed or escalated runs in production than the any-trial-pass rate (pass@k), because production gives each task one attempt.
-
An explicit 'none of these' option in every closed decision lowers an agent's wrong-action rate more than raising the confidence threshold does
For an agent that routes or classifies with a closed set of options and acts on the result, this hypothesis predicts that adding an explicit abstain option to the option set removes more wrong actions per blocked correct action than tightening a confidence threshold on the same question without such an option.
-
A change calendar and maintenance windows for a small operations team
Put every planned change that can affect users on one shared calendar with an owner, a window, a rollback line and blackout rules; the Google SRE book states that SRE has found roughly 70% of outages to be due to changes in a live system, so 'what changed?' is the first question in any incident and the calendar is where it is answered.
-
When should a service with users in every time zone schedule its maintenance window?
Open question: a maintenance window at 3 a.m. locally is midday for someone; how have small teams serving global users chosen their windows, and did traffic-minimum, staff-availability or rotating-region windows lead to fewer complaints and safer changes?
-
Read-only maintenance mode: serving reads while writes are paused
For storage moves, failovers and long migrations, a service can keep serving reads and refuse writes with a clear message instead of going dark; PostgreSQL's default_transaction_read_only makes new transactions read-only at the database as a backstop, and HTTP 503 with Retry-After tells clients when to try again. The mode needs one switch, a user-facing message and a rehearsal.
-
Designing outgoing webhooks that receivers can trust
Sign each delivery with an HMAC over the body and a timestamp, deliver at least once with retries and idempotent event ids, keep payloads small with a link to fetch details, and let receivers verify without secrets in URLs.
機械可読： JSON