Building your own decision model instead of Jev: what a PayPal.AI preprint showed

A team at PayPal.AI has posted a preprint on arXiv about what an in-house decision model in the style of Jev offers compared with a ready-made service. The first part describes an internal judge; the second covers open models that the authors also compared with Jev 1.13.0. S68 For a team weighing self-hosting against calling an API, it is a rare text with concrete numbers on both cost and accuracy. It is a preprint, meaning work that has not gone through peer review.

Why the authors built their own judge

The authors needed to sort roughly 148,000 records from AI agents' coding sessions before they enter training data. Each record gets three questions with fixed answers. S68 Until then, a large language model acting as a judge did this work. By their estimate, one pass would cost on the order of 1,000 USD and take about 73 hours. S68 A check also showed that the stored labels matched a new joint verdict of two judges in only 26.9% of the cases where those two agreed. S68 And the data was not allowed to leave the organisation. S68

The solution was a frozen base model with 4 or 27 billion parameters, a small LoRA adapter and a selection head that does not write text but points directly at one of the options. S68 According to the authors, this judge processes 1,000 sessions in about three minutes on a single GPU. At a reference rate that comes to approximately 0.06 USD, roughly 1% of the cost of the original approach. S68 On a balanced set of 200 sessions, it agreed with the large model's labels in 83 to 85% of cases for the coding question, in 91 to 92.5% for the second question and in 68 to 69.5% when assigning one of eight domains. On the 144 sessions with no overlap with training data, results were 1.5 to 3.8 points lower. S68

Open models against Jev 1.13.0

The second part examines open models that read the answer directly from a language model's output layer at the position of the answer letter. Only LoRA adapters are trained. S68 The authors built on the public recipe and corpus of the Eikos project, with about 25,000 decisions, and tried models with 1, 2 and 35 billion parameters. S68 They evaluated on the public part of the JevBench benchmark, version 1.4.2.2, mainly on 111 hard items. S68

The largest model reached 79.3%, while Jev 1.13.0 scored 73.0%. The 6.3-point difference is not statistically significant, however; the unadjusted p-value is 0.12. S68 The 2-billion-parameter model reached 61.3% and the 1-billion model 54.1%. Generative language models scored 96.4% in the same table, 17.1 points more than the authors' best model. S68

On an H100 graphics card, answering a single question takes about 20.5 ms for the 2B model and about 83.2 ms for the 35B model. The 35B model needs approximately 80 GB of memory. S68 Only the 2B model is publicly available, on Hugging Face as Kailune-AI/pjev-2b; the authors did not release the 1B and 35B adapters. S68

What to take with caution

The authors list several limitations themselves. All JevBench numbers come from the public part. On the sealed part, they say, systems perform much worse, but they report no such result. S68 Each run used a single random seed, and the authors chose the comparison systems themselves. They therefore treat differences of up to roughly 10 points on the hard set as unresolved. S68

On 27 September 2026 the corpus author removed 1,994 rows derived from FinEntity, even though all models in the second part had already been trained on them. S68 The authors did not try a cheap classifier without a language model. Costs in the first part are GPU time converted at a reference rate, not an actual invoice. S68

What it means for a Slovak online shop

The preprint does not claim that an in-house model beats Jev. Rather, it shows the situation in which one is worth considering: a very large number of decisions, questions that do not change, and data that must stay inside. The authors had exactly this combination. S68 An online shop that, for example, sorts customer messages has to weigh that against the price of the service. TypeSafe charges 0.042 USD per million input tokens, and output tokens are free. S01 The cost calculator shows what that would mean at your volume. The 35B model with about 80 GB of memory S68 in turn means a powerful GPU of your own and the work of looking after it.

If data protection is the main reason, it helps to know that, according to TypeSafe, Jev is not trained on customers' requests or responses and is not fine-tuned on their data. S01 Whether that is enough for your obligations needs a separate assessment; the context is in the security and privacy guide.

Two more notes. The preprint compared Jev 1.13.0, but according to the documentation the jev-latest alias now points to jev-1.14.0. S01 The results may therefore not hold for the newer version. The preprint also does not state which language the JevBench items are in, and it says nothing about Slovak. S68 How to check behaviour in Slovak is described in testing in Slovak.

The most useful lesson may be the simplest. The authors first found that their expensive judge produced labels that mostly did not match a fresh check. S68 Before you change models, it is therefore worth checking on a small sample how accurate your current decisions are. That is a suggestion from JEV.sk, not a finding of the preprint.