Per-dependency bulkheads keep unrelated endpoints available when one dependency stalls

Hypothesis

The bulkhead pattern (cited) isolates elements of an application into pools so that if one element fails the others continue to function; Resilience4j (cited) implements it as a semaphore limiting concurrent executions or as a fixed thread pool with a bounded queue. The hypothesis is that the pattern's benefit is largest for the slow-dependency case rather than the dead-dependency case: a dependency that answers just inside the client timeout holds a slot for the whole timeout, and in a shared pool this starves endpoints that never call it. With a bounded pool per dependency, calls to the slow dependency are rejected quickly once its pool is full, and the remaining endpoints see latency and error rates close to their baseline.

Prediction

In a service with endpoints /a (calls dependency A) and /b (calls dependency B) sharing one pool of size P, injecting a delay into A just below the client timeout will, under a constant open-loop request rate, raise the p99 latency and error rate of /b to the level of /a within a few timeout periods. With two bulkheads of size P/2, /b will stay within its baseline while /a shows fast rejections instead of timeouts. Total successful throughput for /a under normal conditions will be lower with the split pool, which is the cost of the isolation.

Proposed test

1. Build the two-endpoint service with a configurable shared pool or per-dependency semaphore bulkheads; keep timeouts identical in both configurations.
2. Put a fault-injection proxy in front of dependency A and add latency equal to about 90 percent of the client timeout.
3. Drive both endpoints at a fixed arrival rate (open model) for a period several times the timeout, once per configuration.
4. Record p50 and p99 latency, error type (timeout, rejection) and success rate per endpoint over time.
5. Repeat with A returning errors immediately instead of stalling, to test the claim that the difference between configurations is smaller in that case.

Status

No result claimed. The effect depends on the pool actually being the scarce resource; a fully asynchronous service may show the same starvation through connection limits or event-loop saturation instead, and the test should record which resource was exhausted.