Jev as a citation checker for financial calculations: what a preprint showed

A language model can compute the correct number and still take it from the wrong row of a table. This is exactly the case examined by a preprint from six authors, published on arXiv on 6 October 2026. They ask whether Jev, acting as a low-cost checker, can judge that a GPT-4.1-mini calculation over a financial report relies on the correct figure, and what such a check adds to a plain comparison of numbers. S48 This is a preprint that has not completed peer review. The authors describe it as an exploratory study because they wrote the wording of the verifier question only after looking at the records under study. S48

How the experiment was set up

The basis is the FinQA dataset, with questions that require calculating with figures from financial statements. From its test split the authors selected 96 questions that can be solved with a single calculation operation. S48 GPT-4.1-mini produced two calculation traces for each question, 192 records in total. S48

Jev had two roles in the experiment. Through Choice it picked the answer from five numerical options: the correct value, one neighbouring value and three arithmetic distractors. Through Noul it answered yes or no to the question of whether the cited evidence supports the values used and their role in the financial statement. S48 The authors used the version typesafe/jev-1.13-20260917 through OpenRouter with TypeSafe as the provider. S48 The difference between Choice and Noul questions is explained in the guide Choice, Score and Noul.

Right number, unsupported citation

Of the 192 records, 170 matched the correct result. In a manual source review the authors marked 153 records as supported and 37 as unsupported. S48 A simple check that looks for the cited value literally in the text caught all 27 cases in which the result was right but the cited figure had the wrong role. At the same time, it rejected 33 records that were fine. S48

When comparing rules, the authors tracked how many records a rule lets through and how many of those have both a supported citation and a correct result. The literal check let through 122 records, of which 117 passed on both counts. A looser rule that compares values at the cited location, which the authors proposed only after the fact, let through 154 records with 141 successes. S48 Jev with a threshold of 0.8 let through 146 records with 142 successes and accepted one unsupported record. At a threshold of 0.5 it let through 159 records with 143 successes, but accepted six unsupported ones. S48

According to the authors, Jev's net gain over the numerical rule is a single record: two additional records on one question minus one valid record that Jev withheld on another. S48 Without that one question, Jev would have 140 successful records versus 141. S48

The same number in the wrong place

The second part of the experiment was a targeted stress test. The authors built 15 pairs in which the citation points to the same number but with a different financial role, for example to a value from a different year. S48 Of six clear wrong-role cases, Jev sent only two below the 0.8 threshold and accepted four. The opposite error also appeared: of six equivalent-fact cases, two fell below 0.8. An equivalent fact is, for example, the closing balance of one period and the opening balance of the next, which express the same amount. S48

The result also depended on how the table was presented to the model. When the authors added explicit column labels, Jev reacted more sensitively to period and status mismatches but supported some equivalent facts less. S48 In a follow-up check on 36 further questions, explicit columns removed both acceptances of unsupported citations, but the number of withheld valid records rose from five to six. S48

Cost and the study's limits

The calls are cheap. The authors report that the verifier extension, 196 calls, cost 0.01669 USD. S48 You can estimate costs for other volumes in the cost calculator.

The study has several limits, and the authors list them themselves. Only questions with a single calculation operation were selected, and Jev's Choice selection hit 95 of 96 correct answers, which left little room for improvement. S48 The original support labels were not assigned blind, the follow-up check was judged by a single person, and only one Jev version and one GPT-4.1-mini alias were tried. S48 For the actual release of an answer, the authors found no benefit on this set. S48 JEV.sk measured none of this.

What this means for a Slovak online shop

The authors recommend treating the check of the calculation and the check of the cited source as two separate tasks. A verifier should be evaluated from both sides: whether it rejects unsupported citations and whether it keeps valid equivalent facts. They advise caution with changes to the form of the input, such as adding column labels. S48

JEV.sk's own suggestion: if an online shop lets a language model fill in product parameters from supplier tables, such as weight or dimensions, the first check should be done by code. It verifies that the number exists in the source table at all. Noul can serve as a second filter for the question of whether it is the right column. Measure not only how many errors it catches but also how many correct records it holds back unnecessarily, and repeat the measurement after every change to the table format. How to choose a confidence threshold and when to hand a case to a human is covered in the guide confidence and calibration.