Jev 1.13 failure modes: literal reading, counting, dates, indirection and context rot

What it is

TypeSafe publishes a page it calls "jaggedness" for jev-1.13 that lists known weaknesses with a workaround for each. In the vendor's own summary the model is good at common-sense judgment, quite literal, and weak at tasks that need numeric precision or several levels of indirection. The list: literal reading (scoping words, negations and implied conditions are taken at face value); math and numbers (it does not count reliably, and error grows with the size of what is counted; hex colours and low-level code representations do worse than names and high-level languages; score expectations must not be used to reconstruct exact magnitudes); date and time comparison (dates are read as text, so ordering, distance and window checks are unreliable); indirection (double negatives and multi-hop questions cost accuracy); large state full of irrelevant detail (accuracy falls as unrelated content grows); adversarial content (state is not treated as hostile by default, so injected instructions or text arguing for its own classification can move the answer); contradictory instructions and criteria; common-sense structural invariants that do not hold; and generation, which the model is not trained for.

Why it matters

An agent that delegates a judgment must know which judgments to keep. The invariants point is the least obvious: the page shows the same refund question asked as a Noul (0.22) and as a yes/no Choice (yes 0.01, no 0.99, confidence 0.97), and a question and its negation as two Nouls summing to 1.19. A threshold tuned on one question type does not carry over to another, and separate questions are not held to arithmetic identities.

How to apply

• Write the exact condition into the instructions and put boundary cases into the criteria; when a wrong answer makes you explain what you meant, that explanation is the missing half of the instruction.
• Count, compute and compare dates in code. For dates, extract the parts as Choices over enumerated values (twelve months, thirty-one days, a bounded year range, plus "not stated") and assemble in code.
• Filter the state first and send only the fields the question needs; the context limit is 64k tokens per request, but accuracy declines well before size becomes the problem.
• Ask each decision one way, and enforce identities (for example that mutually exclusive flags do not both hold) in code.
• Treat state as data. Test with inputs that argue for their own classification before exposing the decision to untrusted content.
• Use a generative model when text must be produced; use the decision model to pick among candidates that code or another model extracted.

Pitfalls

The page is versioned to jev-1.13 and dated; a later release may change the list, so re-read it before relying on a workaround being still necessary or still sufficient. "Extremely consistent" in the vendor's wording means similar outputs for semantically similar inputs, not that any particular invariant holds.