Jev as a guard in front of a language model: what a preprint found about thresholds

Two authors from the University of Southern California published a preprint on arXiv in early November 2026 on how Jev behaves as a safety guard in front of a generative language model. They were interested not only in classification accuracy but in the behaviour of the whole system: how many harmful requests get an answer and how many innocent questions the system refuses. S84 This is a preprint, and its results should be read together with the limitations the authors themselves state. The topic concerns anyone who wants to put a low-cost filter in front of a chatbot or another generative model.

How the experiment was set up

The protected model was Qwen/Qwen2.5-1.5B, with answers of at most 120 tokens. The set had 970 prompts: 520 harmful ones from AdvBench, 200 harmful ones from XSTest-unsafe and 250 innocent ones from XSTest-safe. S84

The authors compared three setups:

• the model alone, with no filter,
• a single input gate, where Jev returns the probability that a prompt is harmful and the system blocks the prompt from a value of 0.50,
• a tiered gate, which blocks from 0.90 and lets a prompt through without review below 0.30. In the band between, the model answers and Jev assesses the answer itself, blocking it from 0.50. S84

The preprint names the alias jev-latest but does not state the concrete version or the date of the experiments. S84 Today this alias points to jev-1.14.0, available since 14 October 2026. S01 The text therefore does not show which version ran during the experiments.

What came out at the original settings

The model alone answered 527 of 720 harmful prompts (73.2%) and refused 3 of 250 innocent ones (1.2%). A single input gate cut answers to harmful prompts to 17 of 720 (2.4%), but it refused 27 of 250 innocent ones (10.8%). The tiered gate let through 40 of 720 (5.6%) and refused 10 of 250 (4.0%). S84

What the word answer means matters here. The authors count every output that was not a refusal, and they found refusals with a simple search for 14 text patterns. This is therefore not verified harmfulness. In a manual check of a random 40 such answers from the model alone, they found 34 harmful and 6 safe. S84

Jev's score on its own separated the groups clearly: the mean was 0.98 for AdvBench, 0.87 for XSTest-unsafe and 0.14 for XSTest-safe. Sensitivity to the threshold differed between the sets, however, and AdvBench reacted less than XSTest-unsafe. S84

Output review added little

In the tiered gate, 79 of 970 prompts (8.1%) went through output review. The review blocked 13 answers, and on manual assessment 7 of them were harmful and 6 safe. The authors also reconstructed from recorded results what would have happened without it: there would have been 47 of 720 answers to harmful prompts (6.5%) instead of 40 (5.6%), but only 4 refusals of innocent prompts instead of 10. S84 At the original settings, then, the tiered scheme did not outperform the simple gate.

Tuning the three thresholds jointly raised the macro-F1 of the tiered gate from 0.9351 to 0.9512 and lowered the number of answers to harmful prompts to 27 of 720. The authors tuned the thresholds on the same data, however, and describe the result as exploratory. In a check on held-out data with a 60/40 split, both schemes gave 2.8% answers to harmful prompts and 14.0% refused innocent prompts, and the selected tiered scheme did not use output review at all. The authors add that one split does not establish that the schemes are equivalent. S84 In their conclusion they write that careful threshold selection may be as important as the complexity of the guard as a whole. S84

What a Slovak online shop can take from this

For a chatbot operator, the most instructive part is the other side of the results. The input gate with a threshold of 0.50 refused more than a tenth of the innocent prompts in this set. S84 It is therefore not enough to copy a threshold from the paper. Slovak is not mentioned in the preprint, and the authors do not state the language of the prompts. S84 The numbers are therefore no basis for estimating behaviour in Slovak.

The authors did not measure latency or cost, and they state explicitly that review frequency does not stand in for them. S84 For a cost estimate, the TypeSafe price list says that jev-1.14.0 costs 0.042 USD per million input tokens and that output tokens are free. S01 Output review also sends the text of the model's answer to Jev, S84 so the costs of both calls should be counted in the cost calculator.

A suggestion from JEV.sk, not a conclusion of the preprint: prepare a sample of real innocent requests from your own operation, measure the harmful cases let through and the innocent cases refused separately on that sample, and select the threshold on a different part of the data from the one you evaluate it on. The page Confidence and calibration describes working with probabilities in more detail. Resistance to targeted manipulation is a separate topic, covered in the article on injected text.

Limitations stated by the authors

The experiment uses one target model and one generation setup. The automatic refusal detector was wrong at least once, and the check on held-out data rests on one split with a small set of innocent prompts. According to the authors, further research needs more models and benchmarks, assessment of actual harmfulness, and measurement of latency and cost. S84 The authors describe the code, anonymised per-prompt results and analysis scripts as publicly available. S84