One extra option shifts Jev's probabilities: what a new preprint found

A preprint posted on arXiv on 12 October 2026 looks at a property that is easy to overlook in decision models: whether the ratio of probabilities between two options stays the same when a third option is added to the question. The authors, from the company AltSlate Labs LLP, examined the model jev-1.13.0 and conclude that in their study it does not meet this condition. S54 Anyone who sets thresholds on confidence or compares numbers across different category lists should be aware of this. This is a preprint, and JEV.sk did not measure any of it.

What the authors measured and how

In decision theory this property is called independence of irrelevant alternatives (IIA). S54 The authors pre-registered five hypotheses and, before the first measurement call, froze the items, the decoy options and the analysis code. A pilot run on 200 items was not counted in the confirmatory tests. S54

They worked with the classification sets BANKING77 and CLINC150, 1,000 items each, and with 500 AG News items as a control. In total they sent 299,463 calls, including a check of whether the model changed during the run, and they report a total cost of USD 6.79. The model version was pinned on every call. S54 Each option set was run twice so that a real shift could be told apart from ordinary run-to-run variation. S54

The shift was tracked on so-called contested items, where both the correct answer and the strongest wrong answer received at least 0.05. There were 647 such items. S54 To this pair the authors then added one extra option: a topic from an entirely different domain, a random category, a category close to one of the pair, or a reworded version of the wrong option. S54

What the results showed

A single added option changed the probability ratio in all ten main combinations. The systematic shift reached 0.20 to 0.52 nats on the log-odds scale, while the variation between two identical runs was 0.15 to 0.17. S54 The ratio moved even when an option from another domain received at most 0.1% of the probability. S54

The direction of the change was not always unfavourable. A category close to the wrong answer moved the ratio by +0.099 nats in favour of the correct answer. The reworded wrong option received roughly a third of the share that it and the original wrong option held together. S54

The effect grew with the length of the list. The authors tried up to 255 options, and when they padded the list with unrelated categories to 255 entries, accuracy fell from 0.853 to 0.814. S54

Two findings are more reassuring. When they added 1 to 128 unrelated yes/no questions to a Choice question in the same call, the mean change in probability was below 0.0006 in every condition. S54 A separate yes/no question for each option, in turn, produced scores that did not depend on the other options, but accuracy when picking an answer was 0.087 lower than with Choice. S54 According to the authors, the number Jev returns for one option should therefore be read in the context of the whole list, not as a property of that option alone. S54

What this means for a Slovak online shop

A common deployment is sorting customer messages into categories such as complaint, shipping or invoice, with a confidence threshold above which the system acts on its own. The authors advise comparing probabilities only within one option set and re-tuning thresholds after every change to the list. S54 If a shop adds a new category, the preprint's results indicate that the ratio between the original categories may move as well, so thresholds need to be checked again on the shop's own data. S54

Further advice from the authors: drop near-duplicate options, do not pad the list with off-topic categories, use Choice to pick an answer and obtain comparable scores from separate yes/no questions. Independent questions, they say, can be sent in a single call. When the probabilities really matter, they recommend averaging repeated runs and both orderings of the options. S54 How Choice questions are written is explained in the guide Choice, Score and Noul, working with thresholds is covered in the guide confidence and calibration, and typical use in shops is described on the page Jev for e-commerce. An earlier preprint on option names also focused on how options are put together, but it examined other models.

Which version this applies to

The results apply to jev-1.13.0. On 14 October 2026 TypeSafe AI released Jev 1.14 with the identifier jev-1.14.0, available through the alias jev-preview, while the alias jev-latest still points to jev-1.13.0. S01 The authors themselves note that newer versions may behave differently. S54

Limits of the study

The authors tested one model version and one type of task: assignment to a single category, where the options had only a name and no longer description. S54 The materials were checked by a language model rather than a human, and the reworded options were also drafted by a language model. The shift analysis covers only the contested items, roughly a quarter of all items. They did not directly examine the mechanism by which the list affects the result; their explanation is inferred from the model's behaviour. S54 As this is a preprint, the text has not gone through peer review.