Backpressure and bounded queues: letting the slowest stage set the pace

What it is

Backpressure is flow control between stages of a pipeline: the consumer signals how much it can accept and the producer waits or slows down. Reactive Streams (cited) states its goal as governing the exchange of stream data across an asynchronous boundary so that the receiving side is not forced to buffer arbitrary amounts of data, which lets the queues between threads be bounded; in its interfaces the subscriber signals demand by requesting elements. In Node.js (cited), writable.write() returns false once the internal buffer reaches highWaterMark, and the 'drain' event says when writing may resume; stream.pipeline() wires this up between stages.

The Google SRE book (cited) describes the request-serving version: most thread-per-request servers keep a queue in front of a thread pool; if the queue is full the server rejects requests. Long queues raise latency and memory use, and for fairly steady traffic the book recommends small queue lengths relative to the thread pool so that the server rejects early when it cannot sustain the incoming rate.

Why it matters

Every unbounded buffer (an in-memory list of pending jobs, an unlimited channel, an HTTP server accepting without limit) hides overload until the process runs out of memory or its latency exceeds every client timeout, at which point clients retry and make it worse. Bounded queues make overload visible and early.

How to apply

• Bound every queue and choose one of three behaviours when full: block the producer (backpressure), reject the newest item (shed), or drop the oldest (only where fresh data supersedes old).
• Propagate the signal to the edge: a rejected request becomes an HTTP 503 or 429 with Retry-After, not a silent wait.
• In async code use bounded channels or semaphores around calls to slower dependencies; in stream code use the platform's pipeline helper rather than manual on('data') handlers.
• Size queues by acceptable wait: queue length divided by throughput is the added latency at saturation.
• Measure queue wait time (age of the oldest item) and the rejection count; both are better overload signals than CPU.

Pitfalls

Blocking a producer that holds a lock or a database connection can deadlock the system. Timeouts without rejection leave the queued work to be done after the client has left. A queue in front of a dependency that itself queues multiplies latency. Retries from upstream must be counted as load.