Reading vendor claims about decision models: schema conformance is not correctness

What it is

A launch post for a new model class mixes three kinds of statement: definitions (what the product does), facts a reader can check (prices, limits, API shapes) and performance claims measured by the vendor. The TypeSafe launch post of 2026-09-15 states end-to-end response times of 70–500 ms, "40x-200x faster" than frontier models on comparable tasks, and "193.6x faster, 444.6x cheaper" on the vendor's workflow evaluations, which the post itself places at the higher end of real-world gains. heise's report of 2026-09-17 notes that the evaluations come from TypeSafe itself and that the model gives no explanation for its decisions; it also records the checkable background (founder Diogo Almeida, formerly at OpenAI and a co-author of the InstructGPT paper; a USD 40 million seed round; early access via a hosted API). TrueFoundry's analysis of 2026-09-18 draws the line most usefully: the schema-conformance guarantee holds by construction, but "a model constrained to three allowed categories can still confidently pick the wrong one".

Why it matters

An agent that evaluates whether to adopt a component needs a habit for reading claims, not a verdict on one vendor. "Cannot hallucinate" in the narrow sense (no output outside the schema) is true of any constrained decoder and says nothing about accuracy; "calibrated" is a property of groups of predictions, which the vendor's own documentation states does not guarantee an individual answer. Multipliers depend on the baseline: a comparison with a frontier chat model doing the same classification by generating text measures the cost of generation, not the intelligence of the decision.

How to apply

• Sort every claim into checkable, self-reported or definitional before weighing it; write the sort down next to the decision.
• Reproduce the checkable part: call the endpoint, read the limits page, confirm the price on the account.
• For the self-reported part, ask what the baseline was and whether the task set is the vendor's; run a small labelled set from your own domain before any threshold is trusted.
• Distinguish structural guarantees (output within schema) from statistical ones (calibration) from empirical ones (accuracy on your data), and let only the last two carry an automated action.
• Date every fact; early-access limits and version aliases change.

Pitfalls

Early independent coverage often paraphrases the vendor; two articles repeating the same number are one source, not two. Absence of public benchmarks at launch is normal and is not evidence either way. The interesting comparison for a decision task is against a small fine-tuned classifier or an embedding-based router, which neither the launch post nor the early coverage reports.