Load testing with open and closed workload models

What it is

A load test has to decide how new requests arrive. In a closed model a fixed number of virtual users each send a request, wait for the response and then send the next; the k6 documentation states that in the closed model a new iteration starts only when the previous one finishes, so the arrival rate is coupled to the response time. In an open model requests arrive at a configured rate whether or not earlier ones have completed, which is how anonymous traffic behaves: visitors do not wait for each other. The two models were contrasted in Schroeder, Wierman and Harchol-Balter's NSDI 2006 paper "Open Versus Closed: A Cautionary Tale", which analyses how system behaviour differs under each.

Why it matters

Under a closed model a slowing server throttles its own load generator: fewer requests per second arrive, and the slowest periods are sampled least. The k6 documentation names this coordinated omission; the wrk2 README explains that a generator that waits for each response coordinates with the server to avoid measuring during high-latency periods, and that wrk2 therefore measures latency from the moment a request should have been sent under the configured constant throughput. A closed test can report a healthy tail latency for a service that would collapse under the real arrival rate.

How to apply

• Derive the model from the question. "How does the service behave at N requests per second?" needs an open model (k6 constant-arrival-rate, wrk2's --rate option). "How do K workers with think time behave?" (batch clients, a fixed pool of API callers) is genuinely closed.
• In open-model tests, pre-allocate enough virtual users; if the tool cannot sustain the target rate, that is the finding, and the tool should say so.
• Report percentiles from full histograms, never averages, and state the arrival model, rate, ramp profile and duration next to every number.
• Ramp in steps and hold each step long enough for queues to settle before reading results.
• Run the generator on separate hardware and check it is not the bottleneck (CPU, ephemeral ports, file descriptors).

Pitfalls

Numbers from the two models are not comparable, so mixing them in one report misleads. Closed-model tools with zero think time produce back-to-back load no client generates. Open-model tests against an overloaded service grow unbounded queues until the client runs out of resources; that is the correct outcome, not a tool bug. Requests that always hit the same cached key make a service look faster than it is.