テーマ： distributed-systems
-
Verteiltes Tracing in Umrissen: Spans, Eltern-Kennungen und W3C Trace Context
Ein Trace ist ein Baum aus Spans, jeder mit Trace-Kennung, eigener Span-Kennung, Eltern-Span-Kennung, Zeitstempeln, Attributen und Status; der W3C-Header traceparent trägt Trace-Kennung, Eltern-Kennung und ein Sampled-Flag über Prozessgrenzen, und ein Dienst, der beide Header nur weiterreicht, hält Traces trotzdem zusammen.
-
What a Raft cluster guarantees: majorities, one leader and linearizable reads
Consensus systems such as etcd (Raft) let a majority of servers agree on an ordered log; they stay correct under any number of failures but make progress only while a majority is reachable. Linearizable reads go through consensus and cost latency; serializable reads are served locally and may be stale.
-
Sagas: multi-step workflows across services with compensation instead of rollback
A saga is a sequence of local transactions in different services, each triggering the next; if a step fails, earlier steps are undone by compensating transactions the developer writes. Sagas restore consistency without distributed transactions but give up isolation, so intermediate states are visible and compensations must be designed, not assumed.
-
Tail latency amplification: when one request waits for the slowest of a hundred
A request that fans out to N backends is as slow as the slowest reply, so a 1-in-100 slow response per backend makes about 63% of hundred-way requests slow. The leaf's p99 becomes the root's median; the remedies are fewer leaves, hedged or tied requests after a percentile delay, deadlines with partial results, and removing the causes of leaf tails.
-
Publishing events reliably with a transactional outbox
Write the event into an outbox table in the same database transaction as the state change, then let a separate relay publish it to the broker. This removes the window in which state is saved but the event is lost (or the reverse), at the price of at-least-once delivery and a relay to operate.
-
Per-dependency bulkheads keep unrelated endpoints available when one dependency stalls
Hypothesis: when a service calls several dependencies from one shared pool of threads or concurrency slots, a dependency that becomes slow (not down) takes unrelated endpoints with it once the pool is exhausted; giving each dependency its own bounded pool keeps the other endpoints near their baseline latency and error rate during the same stall.
-
Read replicas and replication lag: what stale reads look like and how to bound them
A streaming replica applies the primary's log with some delay, so a read right after a write may not see the write. Route reads by how much staleness each caller tolerates, use synchronous replication modes only where their latency is acceptable, and understand that replaying the log can cancel long queries on the replica.
-
gRPC basics: protobuf contracts, streaming and where it fits
gRPC calls methods defined in a .proto file, with Protocol Buffers as interface definition and wire format; it offers unary and streaming calls, deadlines, metadata and a fixed status-code set over HTTP/2. It fits service-to-service and mobile backends; browsers need gRPC-Web and public integrators usually expect JSON over HTTP.
-
Consistent hashing: stable key placement when nodes come and go
Hashing a key modulo the node count remaps most keys whenever a node is added or removed. Consistent hashing places nodes and keys on a ring so that only the keys next to the changed node move; virtual nodes even out the load, and table-based variants such as Maglev trade some placement stability for faster lookup.
-
Distributed tracing in outline: spans, parent IDs and W3C trace context propagation
A trace is a tree of spans, each with a trace ID, its own span ID, a parent span ID, timestamps, attributes and a status; the W3C traceparent header carries trace ID, parent ID and a sampled flag across process boundaries, and a service that only forwards both headers still keeps traces intact.
-
CAP and PACELC as decision aids rather than slogans
CAP forbids only one corner of the design space: perfect availability and linearizable consistency while a network partition lasts. PACELC adds the everyday trade-off between latency and consistency when there is no partition. Used as questions rather than labels, both help decide, per operation, where a system may return stale data and where it must coordinate.
-
Backpressure and bounded queues: letting the slowest stage set the pace
An unbounded queue turns overload into memory exhaustion and unbounded latency. Backpressure means the consumer tells the producer how much it can take, from Reactive Streams demand to a Node.js write() returning false; a bounded queue plus a defined behaviour when it is full is the minimum every service stage needs.
-
Which overload signal should a small service shed load on: queue wait, in-flight count or CPU?
Open question: guidance lists CPU, latency, queue length and thread count as possible triggers for load shedding and calls the choice service-specific. For a service with a few instances and no global quota system, which signal, threshold and priority rule have teams actually kept in production, and how did they tune them?
-
Distributed locks and leader leases: expiry, fencing tokens and what a lock cannot promise
A distributed lock is a lease that expires; a process paused by garbage collection, CPU contention or a delayed network can keep acting after its lease has passed to someone else. Only a monotonically increasing fencing token checked by the protected resource makes such a lock safe for correctness, and a lock used merely to avoid duplicate work needs less.
-
Logical clocks: Lamport timestamps, vector clocks and hybrid clocks
Wall clocks on different machines disagree, so ordering events by timestamp loses updates. Lamport timestamps give an order consistent with causality, vector clocks additionally detect concurrent updates, and hybrid logical clocks keep a value close to wall time while preserving causal order.
-
Circuit breakers: failing fast when a dependency is down or slow
A circuit breaker counts failures and slow calls to a dependency; above a threshold it opens and rejects calls immediately, then lets a few trial calls through (half-open) before closing again. It protects the caller's threads and gives the dependency room to recover, but only with sensible windows, a fallback and a timeout underneath.
-
At-most-once, at-least-once and exactly-once delivery
Messaging systems deliver a message at most once (may lose), at least once (may duplicate) or effectively exactly once (deduplicated by the consumer); at-least-once plus idempotent consumers is the practical default, and 'exactly once' is a property of the whole pipeline, not of the broker.
-
Deletion pipelines across services, derived stores and backups
Model deletion of one person's data as a job with a state per store in the data map: fan out an event, require each owning service to report done with counts, handle versioned object storage and derived stores explicitly, bound how long backups keep the data or destroy per-subject keys, and keep a suppression list so restores can re-delete.
機械可読： JSON