Same case, different decision: a preprint on agreement between Choice and Noul

A preprint that appeared on arXiv on 28 October 2026 addresses a question that is easy to overlook with decision models: do we get the same decision when the same case is put to the model once as a Choice question and another time as a Noul question? The authors, from Zhejiang University and the company vivo, answer that this is not always so, and they treat disagreement between the answers as a useful signal of a wrong decision. S73 For a developer who builds on probability thresholds, this points to a simple safeguard. This is a preprint without peer review, and JEV.sk did not measure anything in it.

What the authors examined

They understand agreement across question types to mean that one and the same state should lead to the same decision regardless of which question type presents it to the model. S73 To measure this, they assembled a test suite called PrimaCons with four axes. S73

• Complement. On the HaluEval set, the model was asked whether an answer contains a hallucination, then the opposite, whether it is factually correct, and in addition it received a binary Choice question. S73
• Label hierarchy. It classified unsafe samples from the PKU-SafeRLHF set at the level of 2, 6 and 12 classes. S73
• Abstention. On AbstentionBench, it could pick the option that the question cannot be answered in a Choice question, or express unanswerability through a Noul question. S73
• Label descriptions. It assigned AG News messages in four categories via Choice with names, via Choice with descriptions and via a separate Noul question for each category. S73

Each axis had 6,000 items for calibration, and the test split also had 6,000 items, except for the hierarchy with 4,000. S73 The evaluated models are labelled JevK5-2B, JevK5-4B and JevK5-9B. The preprint does not mention TypeSafe AI or the identifiers jev-1.13.0 and jev-1.14.0 listed in the vendor documentation, so the text does not make clear whether this is a model that TypeSafe AI provides through its API. S73 S01

What the results showed

According to the authors, disagreements appeared on all four axes and at all three sizes. S73 On the complement axis, the average difference between the positively and the reversely phrased Noul question, after converting the reversed answer to 1 − p, was 0.1858 for the 2B model, 0.1360 for 4B and 0.0892 for 9B. S73 The larger model thus answered more consistently, but the difference did not disappear.

In the hierarchy, disagreement grew as the classification became finer. For the 4B model, the distance between distributions was 0.0870 with two classes, 0.5329 with six and 0.6323 with twelve. S73 On abstention, Choice and Noul differed in the final decision in 36.54% of cases for 2B, 26.69% for 4B and 17.18% for 9B. S73 On the fourth axis, Choice with descriptions differed little from Choice with names, by 0.0917 for 2B, but separate Noul questions for the individual categories gave a noticeably different distribution, with a distance of 0.4386 for 2B. S73

The most interesting point for practice is this: the authors used the difference between the positive and the reversed Noul question to rank cases by risk, and they report that it predicts wrong decisions with a ROC-AUC of 0.91. S73

The Cprc calibrator and its weak spots

The authors also propose a remedy. The Cprc calibrator is a small residual network attached to a frozen model that computes corrections to the logits. It is trained on the calibration data of three axes, and the fourth axis is held out for evaluation. S73 On the held-out descriptions axis, the accuracy of the 2B model rose from 84.90% to 87.42%, for 9B from 88.02% to 89.75%, and for 4B only from 84.08% to 84.12%. S73 The "+2.97%" figure in the abstract is a relative increase; in absolute terms, for 2B it amounts to 2.52 percentage points. According to the authors, latency increased by 0.75%. S73

The text also has shortcomings. The results from the first part come from the calibration data, some baseline values differ slightly between tables, and for the 2B model the gap on the abstention axis grew after calibration from 0.1884 to 0.2260. S73 The preprint has no separate section on limitations, and it evaluates a single model family in three sizes. S73

What this means for a Slovak online shop

The calibrator works directly with the model's logits. S73 According to the documentation, the answer to a Noul question is a single number, the probability of a yes answer, without a separate confidence value. S05 Anyone who calls Jev through the API can therefore take the disagreement check itself from the preprint rather than the calibrator. This is a suggestion by JEV.sk, not a recommendation of the authors: for a sensitive decision, for example whether a customer's message is a complaint or whether a chatbot's answer matches the product's specifications, ask the question twice with opposite meaning. When the results diverge clearly after conversion, send the case to a human.

The Noul documentation warns against negative phrasing, because it inverts the meaning and code that reads the value will then interpret it the wrong way round. It recommends that a high value should mean yes, and as a starting point it suggests thresholds of 0.7 and 0.3, with values in between going to a human. S05 Write both check questions as positive statements with opposite content, for example "The answer is factually correct." and "The answer contains a made-up detail.", not as a double negative. The difference between Choice and Noul questions is explained in the guide Choice, Score and Noul, and setting thresholds in the guide confidence and calibration.

The second question is not free. TypeSafe AI charges for input tokens, 0.042 USD per million, and output tokens are free of charge. S01 Each check question therefore adds more input tokens; the cost calculator shows what that costs at your volume. The results from the descriptions axis also suggest that thresholds set for Choice should not be carried over to separate Noul questions without verification. S73 How the set of options changes probabilities was also the focus of an earlier preprint.

Which version this concerns

According to the documentation, the jev-latest alias points to jev-1.14.0, available since 14 October 2026, and jev-1.13.0 also remains on offer. S01 Since the preprint names neither of these versions, its numbers cannot be directly mapped to the model the API returns today. Checking the agreement of answers on your own data is therefore more sensible than adopting specific values from the preprint.