Can Jev replace a language model as a judge? A preprint compares 14 models

Anyone who needs to decide automatically which of two language model answers is better usually hands that job to another generative model acting as a judge. A preprint by Shreyas Subramanian of Scale AI, published on 31 October 2026, examines whether decision models can do this work instead: models that return their result as probabilities in a single pass, without writing any text. S78 Jev 1.14 itself is among the 14 models studied, so the results are not limited to models built in its style. For anyone considering cheaper evaluation of AI outputs, it is a useful overview of where such a judge holds up and where it does not.

What was compared and how

This is a preprint, meaning it has not yet gone through peer review. Besides Jev, the study included, for example, pplx-decider 1.2 from Perplexity AI, the Clef model from Cloudflare and DReX-Max. Jev ran as the generally available release dated 20261014. S78 TypeSafe states that Jev 1.14 has been available since 14 October 2026. S01

On the other side were generative judges: Claude Haiku 5.5, GPT-5.6 without reasoning and with medium reasoning, and Claude Opus 5.5 with reasoning. They too were asked to return only a verdict and a probability, without a rationale. S78 The work rests on eleven task sets, among them JudgeBench, LLMBar, RM-Bench and RewardBench 2. Every model received answer pairs in both orders, to reveal whether it is swayed by which answer comes first. S78

Without a reference, on par with GPT-5.6 without reasoning

Five of the 14 decision models, Jev 1.14 among them, were statistically indistinguishable from GPT-5.6 without reasoning on five sets covering pairwise comparison and picking the best of four answers. S78 The decision models differed widely from one another, though. On JudgeBench, Choice accuracy ranged from 66.6% for Kev-4B to 84.6% for d1-ultra. S78

The largest gain came from a reference answer, prepared once per task by a generative model. With it, eleven models reached 90.3 to 93.4% on JudgeBench. The preprint assessed this as statistically indistinguishable from Opus 5.5 at 94.7% and from GPT-5.6 with reasoning at 93.1%. S78 On the PPE set, where the best of 16 answers is chosen, one GPT-5.6 reference plus 16 Jev calls came to 0.0051 USD per task. GPT-5.6 with reasoning cost 0.0827 USD and Opus 0.1308 USD. S78 The GPT-5.6 price is only an assumption in the paper, however, because the service tier used by the gateway was not known. S78

Without a reference, the decision models fell well behind GPT-5.6 with reasoning and Opus when finding the correct answer among several candidates. S78

Good calibration, weaker resistance to injected text

Jev 1.14 had a calibration error (ECE) of 0.019, while the generative judges ranged from 0.038 to 0.087. S78 In this test, its probabilities therefore matched more closely how often the model was right. What this means for setting thresholds is covered in the guide confidence and calibration.

The weak spot is text slipped into the answer being judged. When a note addressed to the judge was appended at its end, Jev 1.14 flipped 23.3% of otherwise correct Choice verdicts, compared with 0.4 to 2.3% for the generative judges. S78 When the question was posed as a checklist, this share fell to 4.9% for Jev. S78 Resistance varied widely among the decision models: 0.0 to 4.7% for the five most robust, 16.7 to 49.0% for the other nine. The preprint concludes that this is a property of the specific model rather than of the whole approach. S78 A similar risk is described in the article injected text can flip Jev's decisions.

On long inputs, the strongest decision models trailed GPT-5.6 by 10 to 12 points. For the answer-length bias on RM-Bench, Jev lost 12.9 points, similar to GPT-5.6 without reasoning at 11.3 points. S78

A cheap first stage and the limits of the study

The preprint also tried a cascade: Jev or pplx-decider decides and passes uncertain cases on to Opus. On LLMBar this reached 90.7% with 10% of cases escalated, while Opus alone scored 91.7%. S78 A similar approach with a different model is described in the article Jev as a low-cost first judge.

The paper itself lists several limitations. The results come from a single run at pilot scale, with 80 to 1,925 items per set. The injected texts were written by the author, not taken from an established test set. The checklists and references were produced by Haiku 5.5 and GPT-5.6, so these methods combine a decision model with a generative one. S78

What a Slovak developer can take from this

The following points are a JEV.sk suggestion derived from the preprint, not the result of a measurement. If an online shop lets a language model write product descriptions or replies to customers and wants to check them automatically, the preprint's results suggest Jev fits best where a model correct answer is available. Without one, it fell behind more expensive reasoning models at finding the correct answer. S78

Text that someone outside can influence, such as a supplier in a feed or a customer in a review, should be treated as untrusted. With Choice questions in the preprint, a single slipped-in note was enough for Jev to flip almost a quarter of correct verdicts; the checklist formulation reduced that share. S78 Uncertain cases are worth passing to a more expensive model or a human, as the cascade in the preprint did.

On cost, keep in mind that TypeSafe charges 0.042 USD per million input tokens for Jev 1.14 and output tokens are free. S01 The cost calculator gives an estimate for your own volume, and the guide Choice, Score and Noul explains the difference between the question types.