Jev and clinical trial criteria: admitting missing information made the difference

Authors from The Chinese University of Hong Kong and Capital Medical University posted a preprint on arXiv on 26 October 2026 that asks whether Jev can judge which conditions of a clinical trial a given patient meets. S72 It is a preprint, and the text does not state that it has gone through peer review. For developers it is interesting for two reasons. The authors used version jev-1.14.0, which the jev-latest alias points to today S01, and they tracked not only accuracy but also whether the model can admit that it lacks enough information to decide.

How the test was set up

For each patient–condition pair the model had to pick one class: the condition is met (met), not met (not_met), or information is missing (insufficient). One dataset added a class for conditions that do not apply to the patient. S72 The authors called Jev through a Choice question. The model returned the selected class, a probability distribution over the permitted classes and a confidence value derived from it, but it does not write a rationale. S72 The comparison model was the generative Qwen Flash (alias qwen3.8-flash), which had to answer in a fixed JSON schema and had thinking disabled in the main comparison. S72 How these question types differ is explained in the guide Choice, Score and Noul.

The data come from three sources. The first is a sample for the TrialGPT method, which pairs largely synthetic patient summaries with criteria from real trials. The other two are adapted questions over the MIMIC-III and MIMIC-IV clinical records. S72 Together they contain 3,135 conditions in 734 jobs, of which 2,048 conditions form the main test partition. S72 The authors excluded numerical questions: 468 in MIMIC-III and 700 in MIMIC-IV. S72 Each job ran in two modes: either one request per condition, or all conditions of the job in a single batch. S72

Results: a draw in some places, a clear lead in one

The main metric was macro-F1. In batch mode Jev scored 0.652 on TrialGPT against 0.660 for Qwen, and 0.807 against 0.814 on MIMIC-IV. On MIMIC-III the gap was large: 0.768 for Jev and 0.534 for Qwen. S72 The lead on MIMIC-III comes mainly from the missing-information class. Jev caught such conditions in 56.3% of cases, Qwen in 9.9%. S72

Jev changed little between the two modes, and for all three datasets the confidence interval of the difference included zero. Qwen gained more correct decisions in batch mode, but the authors report this only descriptively, without an interval of its own. S72

The bigger difference was in time. The median duration of one job in batch mode was 1.20 s for Jev against 4.55 s on TrialGPT, 1.18 s against 2.80 s on MIMIC-III and 1.40 s against 8.90 s on MIMIC-IV. S72 The authors also estimated an upper bound on cost per 1,000 conditions in batch mode: USD 0.0170 against USD 0.0645, USD 0.2045 against USD 0.4314, and USD 0.0216 against USD 0.0684. S72 When Qwen was allowed to think, the median job time grew by a factor of 2.33 to 4.30, yet the change in macro-F1 was not demonstrable: all intervals included zero. S72

Limitations the authors state themselves

The reference labels come from the source data and were not revised by clinicians, so agreement with them is not the same as clinical correctness. S72 Each mode ran only once, so the influence of the other questions in a batch cannot be separated from ordinary variation between repeated calls. Jev's architecture and training were not available to the authors, so they describe only its interface. S72 They phrase their conclusion cautiously: they see Jev as part of an initial assessment that flags possible information gaps, and its outputs must be checked against patient records before any decision on enrolling a patient. S72

What a Slovak online shop can take from this

A Slovak online shop does not deal with clinical trials, but the shape of the task is familiar: check whether a record satisfies a condition. That might mean whether a return claim contains the details needed to process it, or whether a product description from a feed states a required parameter. A suggestion from JEV.sk based on this preprint: include an explicit "information missing" class among the options of a Choice question. In this test, that class is where the models differed most. S72 Leave numerical conditions to code, much as the authors excluded numerical questions from the evaluation. S72

Treat the model's decision as a first sort, not a final verdict. How to set confidence thresholds for automatic processing and for escalation to a human is covered in the guide confidence and calibration. For costs, the cost calculator helps: TypeSafe charges USD 0.042 per million input tokens for jev-1.14.0, and output tokens are free. S01 The preprint did not examine Slovak texts, so check the model on your own data before deploying it. When moving to a new version, it is also worth going through the checklist in the article on the jev-latest alias.