When more than one answer is right: a preprint on Jev's probabilities

On 21 October 2026 a preprint appeared on arXiv about a situation that is more common in practice than it seems: a list of options in which more than one is correct. The author, Rushab Munot, listed as an independent researcher, compared jev-1.13.0 with two other decision models. According to his results, the probabilities from a Choice question can rank the correct options, but they do not reveal how correct each of them is. S60 This matters to anyone who picks several categories at once from a single Choice result, for example two departments for one message. It is a preprint that has not been peer reviewed, and JEV.sk did not measure any of it.

What the author studied

Choice returns a probability for every option, all values add up to 1, and the answer is the option with the highest number. S03 The preprint examines whether all correct options can be read from such a distribution when there are several, and whether a single threshold separates them from the incorrect ones. S60

The author called Jev through the TypeSafe API, and the models Liquid d1 and GPT-6 Luna Decisions through Vercel AI Gateway. For those two, the vendors do not publish version numbers, so the results describe what the gateway served in October 2026. S60 The data came from three English datasets. MetaTool, with 4,287 tool-selection queries, contained 497 queries with two correct tools. From HotpotQA the author used 7,405 lists where the answer needs two passages. The third was 1,000 cases of the European Court of Human Rights (ECtHR) with ten articles per list, 192 of them with two to four violated articles. S60

The second correct option often gets little

The main measurement covered lists with exactly two correct options. For Jev, the higher and the lower probability of the correct options differed on average by 74 percentage points on MetaTool, by 60 on HotpotQA and by 58 on ECtHR. S60 Even when both correct options ranked above all incorrect ones, one of them received less than its share under a uniform split (1/n) in 31 to 50% of such lists for Jev. Across all three models it was 28 to 50%. S60 The author notes that a value below 1/n is not an error in itself, because correct options need not be equally relevant. S60

A HotpotQA question that cannot be answered without two passages illustrates this. Jev assigned 0.93 to one of them and 0.07 to the other. S60

Another test sent five identical copies of one correct option. A sensible model would split the probability evenly among them. For Jev, the highest and the lowest copy differed on average by 17 to 26 points, for the other models by 37 to 79 points. Those two models also picked the first copy in 76 to 98% of answers. S60 On ECtHR, Jev picked the last copy in 56% of answers; the author describes the position effect as dependent on both the model and the data. S60

One threshold does not find the whole set

Anyone who wants to select all correct options from a list needs a threshold. The best threshold on list probabilities in the preprint fell between 0.02 and 0.08, well below the value of 0.25 in an example in the TypeSafe documentation. S60 In that example, the code sends a copy of the message to a second team when its probability exceeds 0.25. S03 Even at the best threshold, Jev matched the exact set of correct options no more often than in 0.57 of cases on MetaTool, 0.62 on HotpotQA and 0.52 on ECtHR. S60

Replacing the list with a separate yes/no question for each option did not help. Exact set accuracy dropped in 8 of 9 model and dataset combinations, for Jev to 0.31, 0.29 and 0.44. S60 The overall calibration error of these questions looked good at 3 to 9 points, but 46 to 78% of answers fell in the 0 to 0.1 bin, where at least 92% were incorrect options. Without that bin, the error rose to 8 to 18 points. S60

What this means for a Slovak online shop

A common case: a customer deals with both a return and an invoice in one message, or a product belongs to two categories. If the system adds the second category based on a fixed threshold on Choice probabilities, the correct second option can, according to the preprint, receive a very low number, 0.07 in the example above. S60 JEV.sk's suggestion: set the threshold for selecting several categories on your own labelled messages, including cases with more than one correct answer, and do not use the value from the documentation example as a starting point. How Choice questions are written is explained in the guide Choice, Score and Noul, working with thresholds in confidence and calibration, and message sorting itself in the article on intent routing.

The second trap is a list in which no option is correct. Without a "none of the above" option, the models showed mean confidence of 0.49 to 0.71 on HotpotQA and 0.74 to 0.87 on MetaTool and ECtHR. S60 For such situations, the TypeSafe documentation advises adding a separate Noul question to Choice, for example whether the message mentions any of your products at all, and reading the Choice answer only when the Noul says yes. S03 It is also worth noting that GPT-6 Luna Decisions refused to answer 11.7% of lists and questions, while neither Jev nor Liquid d1 refused any. S60 That the composition of the list matters even with a single correct answer was shown by an earlier preprint on the option list.

Model version and limits of the study

The results apply to jev-1.13.0. Since 14 October 2026, jev-1.14.0 has also been available via the jev-preview alias, while jev-latest still points to jev-1.13.0. S01 The author studied only three hosted models and three English datasets, in which the correct set is defined by each dataset's own labels. He tested one threshold per dataset and one yes/no question per option; per-example thresholds and fixed-count selection were not examined. Other tasks with several correct answers, such as medical diagnoses, were not included. S60 His conclusion is cautious: Choice probabilities are suitable for ranking, not as an estimate that a given option is correct. S60