Shuffled options change Jev's answers more than accuracy reveals

If you reorder the options in a Choice question, Jev's average accuracy hardly changes. A new preprint shows, however, that the stable number hides a different problem: questions the model answered correctly in one order are often answered wrongly in another. S70 Anyone using Jev to sort messages, orders or products should therefore test their setup with more than a single accuracy figure.

What the authors studied

The preprint was written by Mingyang Song, Mao Zheng, Zhiyuan Ma and Xinpeng Zhou of the Foundation Model Department at Tencent. S70 The first version was submitted on 26 October 2026 and a second one followed on 27 October 2026. The figures in this article come from the second version. S71 The authors wanted to know whether a model that distributes probability directly across the supplied options depends less on the order in which it receives them. S70

They compared Jev 1.13 and Jev 1.14, used through their native decision interface, with six generative models, including Claude Sonnet 5.5, GPT-6.1-sol and Gemini 3.8 Flash, and with 12 open imitations of Jev. S70 The main collection had 1,700 questions: 1,200 from SuperGPQA and 500 from LEXam, where each question had 4, 8, 16 or 32 options. S70 Every question was given to the model eight times: in the original order, in reversed order and in six random orders. The wording of the question and the content of the options stayed the same. S70 For comparison, the authors also sent the same question eight times in unchanged order. S70

Accuracy holds, errors move around

According to the abstract, Jev's accuracy differed by less than two percentage points between orders. The set of questions it answered wrongly at least once, however, grew by 27.3 to 28.5%. S70 The errors therefore do not stay on the same questions; each order hits a slightly different set.

The comparison of repetition with reordering is the most telling. With repetition in the same order, Jev lost 8.8 to 9.2% of the questions it first answered correctly; with shuffled options it lost 27.0 to 27.8%. S70 The open imitations were almost unchanged under repetition and lost 0 to 1.3% of their successes, but under reordering they lost 23.2 to 64.2%. S70 Giving the same answer to the same request therefore says nothing yet about how a model copes with a different order.

In this study Jev came out neither clearly better nor clearly worse than the others. Of the questions it answered correctly, it kept the correct answer across all eight presentations for 72.2 to 73.0%. Four generative models kept more, 75.7 to 88.9%, and two kept less. S70 The authors therefore write that they found no general advantage for Jev. S70

Voting and agreement help little

A simple safeguard suggests itself: send the question in several orders and take the most frequent answer. For Jev, compared with the average accuracy of a single presentation, this added 0.96 points for version 1.13 and 1.01 points for version 1.14. For the generative models the gain ranged from 0.46 to 5.63 points. S70 Unanimity is not proof of correctness either. Jev picked the same option in all eight presentations for 57.7 to 59.1% of questions, and 62.1 to 63.3% of those answers were correct. That means 36.7 to 37.9% were wrong. S70

What this means for a Slovak online shop

The collection in the preprint consists of demanding expert test questions, not product categorisation or customer messages, so the numbers cannot be transferred directly. The preprint does show, however, that stability under repetition says nothing about stability under a different option order. S70

JEV.sk's own suggestion, which the preprint did not test: when you check Jev on your own data, for example when assigning products to categories, send the same examples in several option orders. Track which specific cases change their answer, not just the average accuracy. Repeated calls with the same input do not replace this check. In production, send the options in a fixed order so that results can be compared over time. Cases that flip when the order changes are good candidates for human review or a stricter threshold. Working with thresholds is described in the guide confidence and calibration, and question types on the page Choice, Score and Noul.

Every additional order is another request. TypeSafe charges 0.042 USD per million input tokens, and output tokens are free. S01 You can work out what sending a question several times would cost in the cost calculator.

Limitations of the preprint

This is a preprint. The results apply to a fixed set of tasks and eight presentations, and domain, language, difficulty and number of options varied together. S70 Option labels moved together with their content, so the study cannot separate the effect of position from the effect of the label. S70 How Jev is built and trained is not public, so the authors do not identify the cause of the differences. The models also differ in training and interface, so the comparison between them is not causal. S70

Versions 1.13 and 1.14 were tested, and the preprint names no clear winner between them. S70 The alias jev-latest currently points to jev-1.14.0, and the documentation notes that aliases can move to a new release. Anyone who needs stable answers should use a specific version ID. S01