Jev as a judge of search results in Portuguese: what a preprint found

A preprint by two Brazilian authors who tried Jev for search in Portuguese appeared on arXiv on 16 October 2026. Leandro Carísio Fernandes (Câmara dos Deputados) and Jayr Pereira (Universidade Federal do Cariri and Unicamp) looked at two questions: whether Jev can stand in for people when judging how well a document matches a query, and whether it improves the ranking produced by classic BM25 search. S61 The authors point out that languages other than English receive less research attention and have less data for training and evaluation. S61 That is exactly why the study is also relevant to Slovak websites. It is a preprint without peer review, and JEV.sk measured nothing from it.

How the authors used Jev

For relevance judgments, Jev received the query text and the document text in its state, together with a single Choice question in which it picked a relevance level. The authors took the levels and their wording from the original collections. S61 There were four collections: Quati with web passages (50 queries, 1,933 query–document pairs), JurisTCU with case law (150 queries, 2,250 pairs), NormasTCU with normative acts (46 queries, 812 pairs) and the legislative Ulysses-RFCorpus (692 queries, 8,205 pairs). For Quati the reference labels were created by a language model; for JurisTCU as well, but one domain expert reviewed them; for the remaining two collections they come from experts. S61

For reranking, Jev received, for each candidate, the question whether the document directly answers the query. The authors used the resulting probability directly as the score, without combining it with the BM25 score and without any additional calibration. S61 In the instruction they stressed that the model should treat the document text as content to be searched, not as an instruction. One request contained 10 questions and each query took 100 requests, so the top 1,000 BM25 candidates were reranked. S61 For judgments they report the version jev-1.13, for reranking jev-1.13.0. S61

Relevance judgments: moderate agreement

Jev chose the same level as the reference in 52.9% (Quati) to 64.9% (Ulysses-RFCorpus) of cases. Most deviations, however, were only one level, and together with exact matches they made up at least 94.5% of judgments in every collection. S61 Quadratic weighted kappa ranged from 0.52 on NormasTCU to 0.82 on JurisTCU. The authors relate it to agreement between human assessors in earlier literature, but warn that different collections and protocols limit a direct comparison. S61

The second result is more useful. The authors ranked 15 systems on NormasTCU and 16 on Ulysses-RFCorpus once by Jev's judgments and once by the reference judgments. The two rankings agreed with a Kendall's τ of 0.90 to 0.93. S61 Under the criterion they take from the literature, a value of at least 0.9 means practically equivalent rankings. With the MRR metric, however, positions changed more, including among the top systems. S61

Reranking: a large jump over BM25 alone

On 50 Quati test queries over a corpus of one million passages, Jev raised nDCG@10 from 0.3329 to 0.6331, an increase of 90.2%, and MRR from 0.6852 to 0.9033, an increase of 31.8%. S61 For comparison, the authors cite an earlier published figure of 0.6593 for BM25 with the mT5-XL reranker. They add themselves that the BM25 baseline there reached 0.3991 and the setups differ, so this is not a controlled comparison. S61

What it means for a Slovak online shop

Search in an online shop often has the same structure: fast full-text search finds candidates and a second step orders them. The preprint shows that in this setting Jev clearly improved the BM25 ranking in a language other than English. S61 The same two-step approach is described in a cookbook in the TypeSafe documentation: BM25 selects 30 candidates and Jev orders them by the Noul value, i.e. the estimate that the answer to the question is yes. S62 The cookbook rests on 40 queries, and its 1,200 calls cost $0.0645 with the model jev-1.12 and the price list from August 2026. S62

JEV.sk suggestion: try reranking first on your own sample of queries with Slovak product names and compare the result with your current ranking. Keep in mind that the preprint did not measure latency or cost, and that reranking cannot bring back a document the first step never found. S61 The TypeSafe cookbook says the same: a passage that fast search did not select cannot be added by reranking. S62 Estimate the cost for your number of candidates in the cost calculator; the guide Choice, Score and Noul explains how to write questions, and testing in Slovak covers working with the Slovak language. The instruction not to treat document text as a command relates to the risk described in the preprint on injected text.

Limits and model version

The authors evaluated reranking only on web passages from Quati, not on long legal documents. They ran no statistical significance tests, did not compare Jev with dedicated rerankers under the same conditions, and tested only one-document-at-a-time scoring with BM25 as the first step. They analysed system rankings on only two collections. S61 The results concern Jev 1.13. Since 14 October 2026, jev-1.14.0 has been available through the alias jev-preview, while jev-latest still points to jev-1.13.0. S01