Speculative fan-out: asking a decision model every question in one request and deciding in code

What it is

A request to the TypeSafe API carries one state and a map of questions. The documentation states that every question is evaluated in parallel and in isolation against the same state, that adding questions barely changes response time, and that adding questions does not create context rot because no question sees another's answer. The fan-out pattern follows from this: instead of asking the category first and the severity in a second call once the category is known, ask both at once and read the severity only if the category turns out to be a bug report.

Why it matters

Sequential decision chains in an agent loop cost a round trip per hop, and each hop is a place where a partial failure leaves the workflow half-decided. One request with all questions has one latency, one failure mode and one log entry. The vendor's parallel-questions cookbook runs a thirteen-question regulatory briefing over the GDPR Wikipedia article and reports that batching every question into one call was 12.2x cheaper and 10.0x faster than separate calls with no change in the answers; the models page states that the state is ingested once per request and every question evaluated against it, which is where a cost difference between one request and many comes from, since pricing is per input token.

How to apply

• Collect every question a workflow may need for a given state, including speculative ones for branches that may not be taken, and send them together.
• Key the questions by what they decide and read them in code in the order the workflow needs; ignore answers whose precondition did not hold.
• Keep the state to what the questions need. Fan-out multiplies the value of a lean state, and the budget is 64k tokens for state plus all questions with 32k for state plus the longest question.
• Use the same state across the questions; if two decisions need different material, that is two requests, not one large state.
• Log the whole response, so that a later question about why a branch was taken can be answered from the recorded probabilities.

Pitfalls

Independence cuts both ways: a follow-up question cannot depend on an earlier answer inside the same request, so a genuinely conditional question (severity of a bug, given that it is a bug) is asked unconditionally and filtered afterwards. Answers to questions whose precondition is false are still real outputs with real probabilities; treat them as noise, not as evidence. The cookbook's factors are the vendor's own numbers on the vendor's example.