Finding vulnerabilities in code: the Cyber-JEV preprint compares Jev with a custom model

Seven authors published the Cyber-JEV preprint on arXiv in early November 2026. S80 It looks at how to rank source-code functions by the risk that they contain a security flaw, so that reviewers can start with the most suspicious ones. Jev 1.13 serves as the reference point for the authors' own fine-tuned open model. S80 For developers, the work is useful mainly because it shows how Jev behaves on a task where the cases being searched for are very rare, and because it is a reminder that the default threshold may not make sense. This is a preprint that has not yet been peer reviewed; this article is based on its full text.

What the authors built

The basis is the open model Qwen3.5-4B, which the authors fine-tuned with LoRA and extended with an output for binary classification. S80 They saw the main problem in the data. Clean benchmark sets contain only a few vulnerable functions, while large patch archives contain only vulnerable ones. Once they are mixed, a model can learn to recognise where a function comes from rather than whether it is flawed. S80

The authors therefore built a small group from each fix commit: the vulnerable function, its patched version and up to two unchanged functions from the same file. This produced 2,829 groups with 10,099 functions. S80 Within each group, the model learns to rank the vulnerable version above the other members. According to the authors, this makes influences shared by the whole repository cancel each other out. S80

They back the need for this approach with a number. A model fine-tuned in the usual way on the reference data alone gave the patched code the higher score in 67.2% of vulnerable-versus-patch pairs. S80 This figure refers to their own baseline variant, not to Jev.

How Jev 1.13 performed

The main test set consists of 7,270 functions from 57 open-source projects. Of these, 322 are vulnerable, or 4.43%. S80 The authors called Jev 1.13 through the official API with the same prompt as the other models. The preprint, however, does not give the wording of the question or say whether Noul, Choice or Score was used. S80

When ranking functions, Jev reached an AUROC of 0.743 and the five-model Cyber-JEV ensemble 0.814. Average precision came out at 0.101 versus 0.197. S80 Under a strict cap of 1% false alarms, Jev caught 7 vulnerabilities and Cyber-JEV 33. S80

For practice, the most interesting figure concerns the threshold. At the default threshold of p > 0.5, Jev raised 4,031 alerts, which is more than half of all tested functions. S80 With the same number of alerts, Jev missed 39 vulnerabilities and Cyber-JEV 16. S80

The authors also examined pairs of a vulnerable and a patched function and checked whether the model scored the flawed one higher. Jev gave both versions the same probability in 18.8% of pairs, that is 51 of 271. Its result of 50.7% therefore includes half a point for every such tie. Without them, it won in only 41.3% of pairs, Cyber-JEV in 47.6%. S80 Both values are below one half, so telling a fix from a flaw was hard for both models in this test.

Where the limits are

The authors state that they did not deploy the system in CI, did not run it across consecutive releases and did not use it to find new vulnerabilities. The test pairs come from historical fixes, not from live pull requests. S80 A direct comparison with tools such as CodeQL or Semgrep on the same task is missing, and the authors make no claim about speed or accuracy relative to them. The preprint does not mention releasing the code or the model weights. S80

The version tested was Jev 1.13. TypeSafe's documentation now lists jev-1.14.0 as the newest version, available from 14 October 2026. S01 How the newer version would do on this task is not examined in the preprint.

What developers and online shops can take from this

Security review of code is not a typical task for an online shop, but the lesson of the preprint applies wherever the cases being searched for are rare. When they made up 4.43%, the 0.5 threshold flagged more than half of the functions. S80 The following recommendations are our own suggestion, not a result of the preprint. Set the threshold according to how many cases the team can actually review, and check it on your own data. The background is explained in the guide confidence and calibration.

If you use Jev's results for ranking, expect ties. When two cases get the same probability, the order between them has to be decided by another criterion in code, such as the size of the change or the age of the code.

The authors describe their model as a first stage that orders changed functions before deeper analysis. S80 Our article on the judge cascade describes the same role of a low-cost first filter. With long inputs such as source-code functions, cost also matters. Input tokens cost 0.042 USD per million and output is free. S01 The page pricing and limits makes it easier to estimate costs.