Circuit breakers: failing fast when a dependency is down or slow

What it is

Fowler's description (cited): a breaker wraps calls to a dependency and is closed by default; after a threshold of failures it trips to open and calls fail immediately without touching the dependency; after a reset timeout it goes half-open and a trial call either resets the breaker on success or restarts the timeout on failure.

Resilience4j (cited) shows the knobs a mature implementation has: a sliding window that is count-based (last N calls) or time-based (last N seconds); failureRateThreshold and slowCallRateThreshold in percent, with slowCallDurationThreshold defining "slow"; minimumNumberOfCalls before any rate is computed, so that nine failures out of nine do not trip a breaker configured for ten; waitDurationInOpenState; and permittedNumberOfCallsInHalfOpenState.

Envoy's outlier detection (cited) is the same idea at the proxy layer: a form of passive health checking that ejects an upstream host after a configured number of consecutive 5xx responses, bounded by a maximum ejection percentage so that a whole pool cannot be ejected.

Why it matters

Without a breaker, every request to a dead dependency waits for the full timeout and holds a thread or connection; the caller's pool fills and unrelated endpoints fail. A breaker converts a slow failure into a fast one and stops the caller from hammering a dependency that is trying to recover.

How to apply

• One breaker per dependency, and per host where the proxy supports it; a shared breaker lets one failing dependency block another.
• Define failure explicitly: timeouts, connection errors and 5xx count; 4xx caused by the caller do not.
• Put a timeout under every call; a breaker only sees failures that are reported, and a call that never returns is not reported.
• Enable the slow-call threshold; "up but slow" is the common outage shape.
• Decide the fallback per call site: cached value, default, degraded response or an error returned immediately. Never fall back to the same dependency.
• Export the breaker state and transition count as metrics and alert on prolonged open state.

Pitfalls

Many instances probing in half-open at once can re-overload the dependency; stagger with jitter. A tiny window trips on noise; a huge window reacts late. Breakers do not replace retries with backoff or bounded concurrency; they complement them.