Jev as a low-cost first judge: what a preprint on a cascade with GPT-6 showed

On 22 September 2026, a preprint by four authors appeared on arXiv that examines whether Jev can handle the role of a judge, that is, decide which of two AI answers is better or whether an answer is supported by facts. S34 It is a preprint without peer review, but with a described methodology and a dedicated chapter on limitations. For practitioners, it is interesting mainly because it does not test Jev as a replacement for a large model, but as a low-cost first filter that can tell when it is not sure.

What the authors measured and how

The authors tested Jev version 1.13.0 and collected the data on 21 September 2026. They compared it with sixteen judges based on generative models and reward models. The strongest of them was GPT-6 Astra in a low reasoning-effort setting. S34 They used four main datasets: 400 pairs from RewardBench, 350 from JudgeBench, 240 judgments from HaluEval on questions with supplied source text, and 150 items with existing labels from previous experiments. S34

Results in this test (Jev vs. GPT-6 Astra) S34:

• RewardBench: 92.2% vs. 93.5%,
• HaluEval: 87.5% vs. 86.7%,
• items from previous experiments: 94.0% vs. 96.7%,
• JudgeBench: 78.6% vs. 93.1%.

The big difference is in cost and speed. A thousand evaluations cost $0.044 with Jev and $12.182 with GPT-6 Astra, which corresponds to 0.36% of the fee of the model it was compared with. The median latency was 0.152 s versus 1.885 s. S34

Where Jev falls short

The wide gap on JudgeBench is no accident. The authors attribute it to tasks that require recalculating the steps of a solution, and to cases where a wrong answer is written at length and persuasively. S34 The RM-Bench set shows this directly: when both answers had the same style, Jev was right 84.0% of the time, but when the rejected answer was the more polished one, it dropped to 74.8%. S34

The second finding concerns order. When the authors swapped the order of the two answers, 3.25% of the decisions changed on RewardBench and 11.14% on JudgeBench. At the same time, Jev did not favor the first position; it chose it in 48.9% and 48.4% of cases, so this is not a simple position bias. S34

This fits the weaknesses that the vendor acknowledges and that we summarize in the guide on security and privacy: precise calculation and step-by-step verification belong to a different tool.

The cascade: accept when confident, pass on when unsure

The most important part of the study shows where Jev’s errors are concentrated. On items that Jev would accept at a confidence of at least 0.9, the two models barely differed (95.8% vs. 96.5%). On items it would pass on, GPT-6 Astra led by 15 points (67.9% vs. 82.6%). S34

The authors built a cascade on this. Jev assesses each pair in both orders and averages the probabilities. If the confidence is at least 0.9, the verdict stands; otherwise GPT-6 Astra decides. They chose the threshold in advance on 96 pilot pairs and fixed it before the test was expanded. S34 On 510 new pairs, the cascade accepted 53.7% of Jev’s verdicts, reached 92.5% versus 93.1% for GPT-6 Astra alone, and used 56.8% of its fee. S34

The vendor’s documentation describes a similar pattern. A decision is routed based on both the answer and the confidence; below a lower bound, the case goes to a human, and how strict the threshold is should depend on how serious the consequences of an error are. In a banking example, a confidence of 0.6 is enough for a low-risk action, while the system performs a risky action on its own only at a confidence above 0.85. S35 There is more on confidence in the guide on confidence and calibration.

What this means for online shops and developers

Anyone in Slovakia who uses a large language model to generate product descriptions or replies to customers often checks them with another expensive model. The study suggests a cheaper approach: leave the first check to Jev and call the more expensive model only for uncertain verdicts. However, the study’s numbers also point to limits. In the study, Jev could be fooled by a polished wrong answer and did worse on tasks that require recalculating the steps. S34 Checking calculations, prices or availability should therefore be left to code.

In conclusion, the authors recommend evaluating pairs in both orders, choosing the threshold on your own selection set and validating it on items that were not used to choose it. Before deploying on a new task, they say a small local validation is needed. S34 So the 0.9 threshold from this study is not a universal setting.

Limitations of the study

The authors list several limitations. They evaluated a single closed version of Jev and do not know whether the test sets appeared in its training data. The comparison is not balanced in terms of compute. They did not test specialized professional domains, and thresholds transfer unevenly between tasks. S34 The 183 disputed items were assessed blind by a single person who is also a co-author. S34 According to the authors, routing by confidence works where the first model is capable but uncertain. It fails where the model is wrong with high confidence. S34

We summarize other studies on Jev in our research overview.