Jev against 19 language models: a preprint shows strengths and weak spots

A new preprint compared Jev with 19 general-purpose language models on 13 tests. Authors from Dalian Maritime University and Dalian University of Technology submitted it to arXiv on 8 October 2026. S46 It is interesting because it is not a vendor measurement: it shows the kinds of tasks where a decision model keeps pace with large models and where it falls behind. It is still a preprint, that is, a text whose peer review is not finished.

What the authors compared

According to the authors, Jev is a System One model: it picks one of the offered options and does not generate text. S46 They compared it with 19 language models split into six of the strongest, seven representative and six small ones. S46 There were 13 tests covering knowledge, reasoning and two multilingual sets. S46

The authors evaluated Jev and the strongest models themselves, each in a single run. Where they used published scores, they treat them only as a rough comparison, because those differ in prompts and settings. S46

Where Jev keeps pace

On MMLU-Redux Jev reached 95.0% against a median of 93.2% for the six strongest models, and on ARC-Challenge 97.6% against 95.5%. S46 In the multilingual tests it was below the median: MMLU-ProX 83.2% against 84.5% and MMMLU 85.0% against 88.6%. S46

From this the authors conclude that a specialised decision model can reach the level of general-purpose language models on decisions that depend mainly on knowledge. S46

Where it falls behind

The largest gap was on MathQA: Jev scored 75.0% and the median of the strongest models was 92.7%. The authors report that it ended below all 19 compared models. S46 It also trailed on MMLU-Pro, with 82.3% against a median of 88.1%. S46

The authors' conclusion is cautious: a decision model can handle many constrained decisions on its own, but tasks that need multi-step calculation still require a general-purpose language model. S46

What the preprint does not measure

The preprint does not evaluate prices, latency or calibration. S46 From its figures we therefore cannot estimate how much a decision would cost or how fast it would run. Jev does return a probability distribution over the options, but the authors did not evaluate it. In future work they want to study whether it can signal when a question should be passed to a model with reasoning. S46 The guide on confidence and calibration covers this topic.

In addition, the figures come from academic multiple-choice tests, not from real online shop data. JEV.sk did not measure any of this.

What this means for an online shop

The results are only a directional signal, not proof that Jev will cope with your operation. A suggestion from JEV.sk: questions that rely on the meaning of a text, such as sorting a customer message or a complaint, are related to the tasks where Jev held up in the preprint. Leave calculations of prices, VAT or deadlines to ordinary code, because the mathematical test MathQA turned out worst. S46

The practical rule is therefore to split the decision into two layers. Let code calculate and compare numbers and dates, let Jev judge the meaning of the message, and when confidence is low let a person or a stronger model take over the case. This split is JEV.sk's own suggestion; the preprint did not test it.

Check the model on your own messages before deployment. You can estimate costs in the cost calculator, and the question types Choice, Score and Noul are explained in the guide Choice, Score and Noul.