Does Jev share human biases? A preprint finds sensitivity to anchors and order

A preprint posted on arXiv on 14 October 2026 asks whether the decision model Jev repeats errors known from human judgment: anchoring on an unrelated number, the effect of wording, or regard for costs already spent. The authors Ilaria Imperatrice, Taisia Kameneva, Adam Eric Greenberg and Alessandro Romano examined version jev-1.13.0 in 41,500 responses. S57 The result looks reassuring at first glance and less so on closer inspection, and it matters to anyone who builds automation on confidence thresholds. This is a preprint that describes itself as a working draft, and JEV.sk measured none of it. S57

How the study was run

The authors collected data from 30 September to 5 October 2026 through the TypeSafe API. They split the responses into seven task groups: cognitive reflection, anchoring, conjunction, framing, sunk cost, zero risk and social bias in pricing. They used a Score question for anchoring and pricing and Choice for the other tasks. S57 Most experiments set a classic task from the psychology literature next to an edited version of it. Choice tasks were also asked with the order of options reversed. S57 The results are compared with earlier experiments on people, not with other models. S57

Where Jev held up

On nine cognitive reflection tasks, which lure people towards a quick but wrong answer, Jev answered correctly in all 500 responses for every item. Mean confidence was above 85% for every item. S57 In framing, the selected option barely moved: the share of the certain choice fell from 100% to 97%, while in the original experiment on people it fell from 72% to 22%. S57 In sunk cost, it rejected further investment in all 2,000 responses. S57

Where the cracks appeared

The selected answer is only part of the result. In the original sunk-cost version, the prior investment raised the probability of investing further from 1.00% to 7.12%, or to 5.92%, depending on the order of options. In the original version, framing shifted the probability of the risky option by 34.31 points on average. S57

For anchoring, Jev had to propose a sentence for shoplifting on a scale of 3 to 9 months, and the task included the sum of two thrown dice. Although the dice have nothing to do with the case, the mean derived sentence rose from 6.338 months at a sum of 2 to 6.811 months at a sum of 12. S57 The effect was smaller than in people: with anchors of 3 and 9 the difference was 0.302 months, compared with 2.53 months in the study on people. S57

The conjunction task produced the starkest result. Jev solved the original, well-known task in all 500 responses, but did not solve the edited version even once, with mean confidence of 94.28% and 95.42% for the two option orders. S57 In an extended check across 30 scenarios and 15,000 responses, it chose the wrong answer in 53.96% of cases. The authors conclude that results on well-known tests may overstate reliability. S57

Option order also mattered in the zero-risk task. In the edited version, the share choosing the option with the larger harm reduction fell from 99.6% to 65.6% simply because the order of options changed. S57

Prices by customer name

The largest part of the study, 25,000 responses, concerned prices. Jev had to recommend an opening offer for buying a car from ten options between 28,750 and 30,895 USD. The authors rotated 125 buyer names from six groups by race and gender, once with the name only and once with race and gender stated explicitly. S57 Group means ranged from 29,529.28 to 29,539.90 USD with the name only and from 29,540.71 to 29,560.75 USD with explicit details. The only difference whose confidence interval excluded zero was between Black and White customers with explicit details: 12.30 USD on an option range of 2,145 USD. Some differences ran in the opposite direction to classic experiments on people. S57 The authors derived the price as a probability-weighted average of the individual options, not as an amount the model returned directly. S57

What this means for a Slovak online shop

In their discussion, the authors give the example of a customer-support system that handles a request automatically when the probability of the selected option reaches at least 80% and otherwise passes it on. They point out that a drop from 83% to 77% changes what the application does, even though the selected answer stays the same. S57 If an online shop sorts messages or return claims with such a threshold, these results suggest that an unrelated detail in state or a different order of categories can decide whether a case is handled by automation or by a person. The authors therefore recommend testing sensitivity to irrelevant context and to answer order. S57

JEV.sk suggestion: before going live, send the same question with both option orders and once with an unrelated detail added, and compare how confidence moves. Put only the data that belongs to the decision into the state. Working with thresholds is covered in the guide confidence and calibration, and writing questions in the guide Choice, Score and Noul. The effect of the option set on probabilities was also described in an earlier preprint on an added option.

Limits and version

Most experiments have only one original and one edited task, so the repeated calls describe behaviour for these specific texts, not for the whole family of similar tasks. Derived values such as the sentence or the price depend on a fixed response scale, and the authors note that, according to TypeSafe documentation, Score outputs have limited numerical calibration. The original and edited versions were collected in separate runs, so the authors cannot rule out a change in the state of the service between them. S57 The results apply to jev-1.13.0. TypeSafe has since made Jev 1.14 (jev-1.14.0) available from 14 October 2026 through the alias jev-preview, while jev-latest still points to jev-1.13.0. S01 The preprint has not been peer reviewed.