Testing error paths and timeouts of outbound calls

Goal

Make the code that runs when a dependency fails as well tested as the happy path. That code runs rarely in development and constantly during incidents, which is when it is least affordable to discover that it never worked.

Prerequisites

Outbound calls go through an injectable client or adapter; timeouts are configurable per call; the test framework can assert on exceptions and on log or metric output.

Steps

1. For each outbound dependency, list the failure classes: connection refused, connection reset, timeout on connect, timeout on read, HTTP 5xx, HTTP 429 with Retry-After, malformed body, truncated body, slow body. Each becomes at least one named test.
2. At unit level, inject the failure with a test double. Python's unittest.mock documents side_effect: an exception is raised when the mock is called, and an iterable yields successive results, so "fails twice, then succeeds" is one line of setup.
3. Assert on the behaviour the design promises, not on exception text: the caller stops after N attempts, waits with backoff and jitter, raises a typed error, records a metric, leaves no partial write and returns the connection to the pool.
4. Test the timeout itself with a fake clock or a server that never answers: a listening socket that accepts and stays silent exercises the read timeout; an unroutable address exercises the connect timeout. Keep the timeout under test small so the test stays fast.
5. At integration level, put a fault-injecting proxy between the service and its dependency. Toxiproxy's README describes toxics that add latency with jitter, limit bandwidth, stop data and close after a delay (timeout), simulate a TCP reset (reset_peer) or delay the close (slow_close), each switched through an HTTP API, so a test can enable a fault, run the scenario and disable it.
6. Test cleanup after failure: a request that timed out must not keep a pool connection or a lock; read pool statistics after the test.
7. Add one test per past incident that reproduces its trigger and keep it.

Expected result

Every failure class has a named test; changing a timeout or retry policy breaks a test instead of surprising an operator.

Limits and test basis

Mocks test the caller's logic, not the real library's behaviour under a real timeout; keep at least one test per client with a real socket. Proxy-based tests are slower and need orchestration in CI. This is a proposed protocol; no measurement is claimed.