Jev for moderating hateful posts: what a new preprint found

Three authors from the University of Cyprus posted a preprint on arXiv on 2 October 2026 that examines whether decision models can recognise hateful posts without anyone fine-tuning them for the task. Among the six configurations studied is Jev in version jev-1.13.0. S42 It is a preprint without peer review, but with a detailed description of its methodology. The topic concerns anyone who publishes reviews, product questions or comments on their website: the content has to be checked, and sending every post to a large language model is expensive.

How the authors proceeded

They called Jev through the hosted API and ran the other five decision models (Laya, Decider, two SemIf variants on Qwen3.5 2B and 4B, and Bespoke-Nimble-9B) locally. For comparison they used two commercial models, GPT-5.6 Luna and GPT-6 Sol, as well as the specialised moderation model CoPE, the BART model without task-specific training, and two classic classifiers trained on data (TF-IDF with logistic regression, and DistilBERT). S42

They tested on four English datasets: Dynamic with 4,120 texts, MHS with 3,203, HateXplain with 1,924 and HateCheck with 3,728 texts. HateCheck consists of deliberately constructed test cases. S42 For the statistics, the authors resampled the data 10,000 times and adjusted the results with a Bonferroni correction for the number of comparisons. S42

How Jev performed

The main metric is macro-F1 on a scale from 0 to 1 for the simple question of whether a text is hateful. Results in this test (Jev, GPT-6 Sol, GPT-5.6 Luna) S42:

• Dynamic: 0.826, 0.897 and 0.858,
• MHS: 0.691, 0.682 and 0.691,
• HateXplain: 0.738, 0.776 and 0.677,
• HateCheck: 0.960, 0.976 and 0.974.

The commercial models had the best result on three of the four datasets, but they were statistically significantly ahead of all decision models only on the Dynamic dataset. On MHS, the locally run Decider model beat them with 0.801. S42 So no model won everywhere.

Cost and latency

The difference lies mainly in cost. On the HateCheck dataset, a thousand texts cost 0.033 USD with Jev, 0.120 USD with GPT-5.6 Luna and 1.177 USD with GPT-6 Sol. Median latency was 336 ms for Jev, 844 ms for Luna and 1,426 ms for Sol; the locally run Laya needed 48 ms. S42 The authors sum this up by saying that Jev trailed GPT-6 Sol by 1.6 macro-F1 points at a cost lower by approximately 97%. S42

According to its documentation, TypeSafe AI charges 0.042 USD per million input tokens and does not charge for output tokens. S01 The pricing and limits guide shows an estimate for your own volume of posts.

Definitions and extra questions did not reliably improve results

The authors also tried two adjustments that one would intuitively expect to help. The first was to add the dataset's definition of hate speech to the question. The definition changed 2.8% to 28% of predictions, but significant improvements and deteriorations occurred equally often, in eight of 28 comparisons each. S42

The second adjustment was decomposition into several sub-questions, for example whom the text targets and what kind of conduct it involves. The result improved significantly in only 20% of comparisons and got worse in approximately 37%. For Jev, decomposition also raised costs by 64% on HateCheck and by 139% on Dynamic. S42 The authors see the benefit of decomposition elsewhere: for a disputed post, it records which criterion was decisive, which makes human review easier. S42 The sub-questions themselves varied in difficulty. Jev identified the target group on HateCheck with a score of 0.989, the direction of the statement with 0.682, and the type of conduct on Dynamic with only 0.538. S42 How Choice and Score questions are composed is explained in the Choice, Score and Noul guide.

What this means for a Slovak online shop

All four datasets are in English, and the authors state explicitly that no conclusions about other languages can be drawn from them. S42 The vendor also states that Jev's primary language is English and that other languages have been tested but are not optimised to the same degree. S01 The figures from the study therefore cannot be transferred to Slovak reviews or comments. The testing in Slovak guide describes how to check the model's behaviour in Slovak.

The authors' recommendation is measured: in their view, a good decision model without fine-tuning is a reasonable starting point, provided the operator validates it on a sample labelled according to their own moderation rules. S42 They also point to the cost of both kinds of error. A strict filter can hide legitimate posts, including replies that push back against hate, while a lenient filter leaves the affected groups unprotected. S42

A JEV.sk suggestion for practice: take a few hundred real reviews and comments from your website, label them by hand according to your own rules, and only then compare Jev with the model you use today. Leave disputed cases to a human. How this relates to the model's confidence is explained in the confidence and calibration guide; this study does not evaluate confidence thresholds in its main text. Further use cases are in the Jev for online shops guide.

Limitations of the study

Besides the language restriction, the authors note that HateCheck is artificially constructed, so it is not certain how its results apply to real posts. They had to substitute the definition for HateXplain, the decomposition rules reflect their own interpretation, and nobody checked the answers to the sub-questions by hand. They also do not know whether the test data made it into the models' training. S42 We summarise further studies on Jev in the research overview.