Jev as a RAG answer checker: similar accuracy, more errors let through

A chatbot built on documentation or a product catalogue assembles its answer from retrieved passages (RAG). Even so, the answer can contain a claim that does not appear in those passages at all. A preprint on arXiv dated 30 September 2026 examined whether Jev can handle this kind of check instead of a large generative model. S55 Its value lies in not stopping at a single number: the overall accuracy of both systems came out similar, but each makes different mistakes.

Who compared what

The authors are Alden Do Rosario and Kirollos Atef of CustomGPT.ai. The first author founded and leads the company, and its product includes claim verification. The authors state that they are paying customers of both vendors and that neither vendor had any input into the design, data or results. S55 This is a preprint that has not gone through peer review. JEV.sk read the full text and did not test anything itself.

The comparison set version jev-1.13.0 against GPT-6 Astra, which worked as a binary classifier at low and high reasoning effort. The data consists of 495 claims from the development split of the LLM-AggreFact benchmark, 45 from each of 11 datasets. 297 claims are supported and 198 are not. Every claim went to both systems, so the comparison is paired. S55 Jev received the claim and the evidence as two items of its state (state) and chose between the answers SUPPORTED and NOT_SUPPORTED. S55

The headline metric is macro-averaged balanced accuracy. Jev scored 73.27, Astra 73.83 at low and 74.11 at high effort. Jev's difference from Astra is −0.56 points with a 95% interval from −5.6 to 4.5. The authors stress that they merely failed to detect a difference, which does not yet mean the systems are equivalent. S55

Where the systems differ

The systems disagreed on 15.2% of claims. S55 More important is the direction of their mistakes. Jev labelled an unsupported claim as supported in 23.23% of cases, Astra in 14.14%. The authors therefore describe the cheaper model as the more permissive guard. S55 For a chatbot this is exactly the unpleasant kind of error: an answer with an invented detail passes the check unnoticed.

The price gap is large. The authors calculated costs from price lists as of 20 September 2026; these are not actual invoices. Jev came to USD 0.04786 per 1,000 claims, Astra to USD 8.93 at low and USD 10.70 at high effort. S55 The study did not compare response speed. S55

The cascade saves money, but not thanks to confidence

The authors also tried a cascade: Jev decides first and passes the claims it is unsure about to Astra. They chose the threshold by cross-validation. The cascade reached 74.10 at USD 2.92 per 1,000 claims, passed on 29.9% of claims on average, and let through an unsupported claim in 19.22% of cases. S55 It is therefore roughly three times cheaper than Astra alone, yet it still lets more errors through.

The most instructive part is the control group with random selection. When the authors sent a randomly chosen 30.9% of claims to Astra, it cost USD 2.81 and scored 73.36. Selection by confidence at the same share cost USD 2.94 and reached 74.01. According to the authors, the saving comes from how many claims are passed on, not from the confidence signal, and at this sample size one cannot claim that confidence-based routing beats random routing. S55 The probability Jev returns does carry information, though: its AUROC against the correct labels is 0.868. S55

There is also a lesson in the threshold chosen on the same data it was then measured on. It yielded 75.82 at USD 1.62, but the authors consider this result too optimistic and replace it with the cross-validated one. S55

What this means for a Slovak online shop

If an online shop runs a chatbot that answers from product descriptions or terms and conditions, the study suggests two practical points. First, overall accuracy is not enough. You need to track how many unsupported answers the check lets through, because that is where Jev and Astra differed most in this test. S55 Second, the threshold for passing a case to a stronger model should be set on your own data and checked on data other than the data it was chosen from. How to work with confidence is explained in the guide confidence and calibration. A similar cascade for evaluation is also described in an earlier article on the GPT-6 preprint.

At a price of around five hundredths of a dollar per thousand claims, Jev can serve as a cheap first filter. But where an error that slips through means wrong information about a warranty or a price, the results show that Jev on its own was more permissive in this study. The authors themselves recommend checking the approach on your own task before generalising it. S55

Limitations of the study

The authors list the limitations openly. There is one benchmark, one pair of systems and one run per system. They used only the development split of the data, so it cannot be ruled out that the models saw it during training. The cascade costs are recalculations without network and orchestration overhead. The test harness was built with the help of AI assistants, and an appendix describes the corrections made during the work. S55

The results concern version jev-1.13.0. According to the documentation, it is still the stable version behind the alias jev-latest, while jev-1.14.0 has been available as a preview under the alias jev-preview since 14 October 2026. S01 The study does not say whether the behaviour changes in the newer version.