Where Jev falls short: decisions that need intermediate steps

The decision model Jev answers in a single pass and does not write out intermediate steps. A new preprint looks for the limit of this approach. Authors from Case Western Reserve University, Georgia Institute of Technology, Columbia University and Japan's NII LLMC found that the model reliably finds and selects a record, but makes errors on decisions that a language model only handles after writing out its procedure. S59 Version v1 on arXiv is dated 18 October 2026, and it is a preprint that has not completed peer review. S59 For a developer, this is a practical question: which decisions in an application can be left to Jev and which cannot.

What the authors studied

The authors define a decision model as one that returns the probabilities of the individual options after a single pass, without generating text. They tested jev-1.13.0 through the API, alongside 23 configurations of open Jev-style models. For comparison, they used the language model Qwen3.8-27B in thinking mode. S59

The first part is a synthetic set in which only one factor changes at a time: context length from 1,000 to 20,000 tokens and the number of options from 2 to 64. S59 The second part is based on real decision tasks from JevBench, PubMedQA, Banking77 and HelpSteer2. After exclusions, 738 items remained: 646 solvable in a single pass and 92 that require reasoning. S59

Finding a record, yes; assembling a state, no

When looking up and selecting a record, Jev answered correctly in all 105 tested combinations, even with 20,000 tokens and 64 options. S59

The problem appeared with state tracking. The model received a day-ordered event log in which the tracked user repeatedly joined and left projects. Entries from other users were mixed in with that user's entries. The task was to determine which projects the user belonged to at the end. S59 With one change, Jev was correct 100% of the time; with sixteen changes, 64% (context up to 8,000 tokens). S59 Of 145 wrong answers, 134 were an earlier state of the same user and only 3 confused users. S59 So the model found the entries but assembled the result from them incorrectly.

A better prompt did not help. Variants instructing the model to proceed step by step, describing the procedure or naming the error reached 54 to 64% at 16 changes, compared with 61% for the original wording. S59 Only changing the input helped. When it contained only the tracked user's entries, accuracy was 100%. When the other users' entries were replaced with unrelated text, it was 99%. S59 The problem was therefore similar records from other people, not the length of the text itself.

The model adopts a stale summary

The second weakness concerns verification. When Jev received a ready-made summary of the state in addition to the log and the summary was correct, it succeeded in all 30 combinations. With a stale summary, it chose the stale state in 61 to 100% of the items that offered it. S59 It accepted correct summaries in 91% of cases but rejected stale ones in only 62%. S59 According to the authors, none of the tested decision models verifies summaries reliably. S59

A language model with thinking handled both tasks. With context up to 8,000 tokens, Qwen3.8-27B solved 96% of the items with multiple changes and rejected 95% of stale summaries. The same model, read out in a single pass, reached 77% and 74%. S59

Real tasks showed a similar gap. Jev scored 97.5% on items solvable in a single pass and 73.9% on items that require reasoning, 94.6% overall. S59 On the hardest part of JevBench it reached 73.1%, against 99.2% for the thinking model. S59

Model confidence does not reveal this kind of error

The confidence value (confidence) helps detect errors the model makes on its own. The AUROC metric, which measures how well confidence separates correct answers from wrong ones, was 0.83 on state-tracking tasks and 0.94 on real tasks. For errors inherited from a stale summary it fell to 0.53 (logs up to 4,000 tokens), which is close to chance level. S59

In addition, Jev rarely used an explicit option to hand a case off. It chose it for 1.6% of real tasks and for 3 of its 40 errors. S59 Training open models to hand off helped only partly, and it did not catch errors from a stale input either. S59

What this means for a Slovak online shop

The authors recommend that decisions with intermediate steps, especially checking an input the model did not compute itself, be made by a model that writes out its reasoning. In their view, Jev belongs at the end of the chain: it selects quickly among options once the state has already been computed and checked by a model that reasons. S59

JEV.sk's own suggestion: an order history with several status changes resembles the logs from the experiment. The current order status should therefore be determined by code from the database, not by Jev from an event listing. If you send the model context, filter out entries from other orders and customers; in the preprint, exactly this kind of narrowing of the input raised accuracy to 100%. S59 Do not treat a summary from another system as verified just because Jev confirmed it. According to the preprint, a confidence threshold from the confidence and calibration guide will not catch this type of error. How the question types differ is explained on the Choice, Score and Noul page.

What the preprint does not measure

The authors studied isolated decisions, each posed and scored once. In agents, errors can accumulate across repeated calls, which the study did not measure. S59 The tested model was jev-1.13.0. Since 14 October 2026, jev-1.14.0 has also been available, but the jev-latest alias still points to jev-1.13.0. S01 Whether the newer version moves the limit is not examined in the preprint, and JEV.sk measured none of this.