When the data changes: a preprint on when a decision model should not answer

A decision model is usually tried on a data sample and then deployed. Over time, however, the data changes: new kinds of requests appear, the ratio between categories moves, or the list of options is edited. The SHIFT-JEV preprint, posted on 25 October 2026, examines what happens in such a situation to models that spread probability across a fixed set of options, and proposes when a model should rather not answer. S77 Anyone who uses Jev to sort messages or orders gets a useful reminder that a successful test before launch is not enough for long-term operation.

Who tested what

The authors are Muhammad Atif Saeed, Azka Atiq and Akhtar Jamil from the National University of Computer & Emerging Sciences in Islamabad, Adarsh Kumar from the University of Arkansas at Little Rock, Uzair Iqbal from the University of Malaya and Roland Aydin from Germany's DFKI. It is a preprint, meaning the work has not yet gone through peer review. S77

An important limitation: the results in the preprint come from the open models Qwen3-0.6B, Qwen3-4B-Instruct-2507 and Phi-4-mini-instruct. The authors gave them an interface that imitates Jev, so the model receives numbered options and picks one of them. By their own account, the JevBench suite was not available to them. S77 The work therefore describes Jev-style models, and its figures cannot be transferred directly to Jev itself.

The authors distinguished four kinds of shift. In a domain shift, tasks came from other areas, for example bank customer questions from the Banking77 set, financial news from Twitter, or MMLU. In a share shift, the AG News set had strongly imbalanced categories, at ratios of 10 and 100. New categories were represented by the CLINC150 set with a "none of the above" option. The last kind changed the option list itself: its order and wording, adding a similar-sounding wrong option, or leaving out the correct answer entirely. S77

When it is better not to answer

The main part of the work deals with when the model should abstain. It answers only when the estimated risk of error is low enough; otherwise it passes the case on. The target was that the error rate among answered inputs should not exceed 10%. S77

A learned uncertainty score with a certified threshold did best: it exceeded the target in 3 to 5 of 27 environment-and-run combinations. A threshold based only on the top probability exceeded it in 5 to 15 cases, and conformal prediction sets in 9 to 14. S77 On the smaller Qwen3-0.6B model, the top probability with the same certification had an average realised error rate of 16.0%, while the learned score had 6.4%. S77

The fewest violations, 3 and 2 of 27, came from the strictest rule, which applies the stricter of two thresholds to all inputs. The authors, however, present it as a finding reached only after the fact that still needs to be confirmed. S77

The case where the correct answer was missing from the list, so every option was wrong, is also instructive. The larger model then abstained in 14 to 72% of cases across individual runs, and the authors acknowledge that a dedicated "none of the above" mechanism may be needed. S77

A simple test reveals changes in category shares

For monitoring the data stream, the authors compared several detectors. A shift in category shares that detectors based on model confidence or on vector representations of the text missed was caught by a χ² test over the histogram of predicted categories in 95 to 99% of cases at full shift strength. S77 The test also raises false alarms, however: on data without a shift they reached 6% and 10%, more than the target of 5%. S77 The authors nevertheless recommend that monitoring of changes in category shares run by default. S77

Fine-tuning and its limits

The third part proposes ongoing fine-tuning of the model to new conditions. A safeguard accepts a change only if it shows a statistically supported improvement on held-out examples. On Qwen3-0.6B this approach reached an accuracy of 70.5% and outperformed the compared methods in every run. On Qwen3-4B, by contrast, it fell behind by 0.4 to 1.4 points. S77 Fine-tuning with anchoring to the original model and temperature scaling reduced the smaller model's average calibration error under shift from 0.289 to 0.087. S77 This part mainly concerns those who have access to model weights, as the authors did with open models.

The authors themselves list several limitations. They created the shifts in category shares and in the option list artificially, ran each model in three runs, and tested only models of up to 4 billion parameters from two families. The code is to be released only with the final version of the paper. S77

What an online shop can take from this

The following points are a JEV.sk proposal derived from the preprint, not the result of a measurement on Jev. If an online shop uses Jev to sort customer messages by intent, as described in the message routing pattern, it is worth storing the predicted categories and comparing their shares between periods. If, for example, delivery questions increase before Christmas, this is exactly the kind of share shift that confidence-based detectors missed in the preprint. S77 Since the test also produces false alarms, an alert should lead to a review by a person, not to an automatic change of settings.

The second lesson concerns thresholds. For Choice and Score questions, Jev returns a confidence value. The TypeSafe documentation advises splitting it into three bands: act automatically at high confidence, proceed carefully at medium confidence, and hand the case to a human or another procedure at low confidence. Thresholds should scale with risk, start conservative and be checked on your own data. S06 The preprint adds a warning: a threshold certified only on the original data did not hold up in most runs at the stricter target of 5%. S77 You can read more about thresholds in the guide confidence and calibration.

The third proposal: if the option list may not contain the correct answer, add an "other" type of option to the Choice question. Without it, the model has to pick one of the offered options even when all of them are wrong. S77 How Choice, Score and Noul questions differ is explained in the guide Choice, Score and Noul.