Option labels can override definitions: what a new preprint shows

A preprint dated 1 October 2026 examines what happens when a typed decision model receives options that have a short label and a longer description of the rule. The authors, from the University of Southern California, claim that most of the models they tested follow the option labels rather than the definitions the developer wrote. S45 For anyone assembling the list of options for a Choice question, this is a practical warning. It is, however, a preprint about open-weight models; the authors did not measure Jev itself. S45

What the authors examined

A typed decision model returns a probability for each option supplied by the caller. Every option has a short label and a written definition containing the rule. S45 The authors asked which of the two influences the result more, and they named the tendency to follow the label option-label bias. S45

They tested four open-weight models: three from the laya family and the model von. They used eleven classification tasks and their own synthetic suite called PolicyBench, in which the rule appears only in the definitions. As a control they also added the language models Qwen2.5, Llama and Mistral. S45

What the numbers show

For the laya-td model, deleting all definitions barely changed accuracy: 0.8559 compared with 0.8487 with definitions. The definitions alone, without meaningful labels, gave 0.7971. S45 When a label and its definition contradicted each other, laya-td accuracy fell to 0.1357. S45

On the control version PolicyBench-XF, renaming the options to neutral A and B raised laya-td accuracy by 0.1511 (interval from 0.1377 to 0.1646). The authors call this their main quantity. For the language model Qwen2.5-7B the difference was +0.1203. S45

The model von behaved the other way round. It had an accuracy of 0.8511 under every labelling and fell to 0.5102 when the definitions were removed, so it drew only on the definitions. When the authors inserted a label that contradicted the definition, its accuracy on the classification tasks dropped from 0.8511 to 0.2281. S45

The mechanism is interesting. When the authors removed the label from the input of the laya models, the difference disappeared entirely. In their view the cause is therefore not the constrained decision head of the model but the way the options are rendered into the input. S45

Confidence can mislead

The preprint also touches on the confidence value. For laya-td, the area under the ROC curve (AUROC) of the confidence score was 0.8123 with aligned labels and 0.2433 with misleading ones. A value below 0.50 means that confidence points in the opposite direction when the model errs. S45 The authors therefore warn that thresholding confidence alone is not enough; more about it is in the guide on confidence and calibration. S45

What the authors recommend

The first step is a two-call test: ask the same question twice with different option labels and unchanged definitions. If the probability distributions differ, the labels are influencing the decision. S45

If the rule lives in the definitions, the authors recommend arbitrary labels such as A and B, provided the task allows it. When the labels must carry meaning, they suggest a correction using an empty state and a comparison of two renderings. S45 A bare instruction to ignore the labels helped only a little: an improvement of 0.0138, which the authors say does not replace arbitrary labels. S45

Impact for a Slovak online shop

A typical case is sorting customer messages into categories such as complaint, return or a question about shipping. If the rule for when a message belongs under complaints sits in the definition of the option, and the label sounds similar to the content of the message, the label may, according to the preprint, decide instead of the rule. This holds for the models the authors tested. S45

The authors did not measure Jev, because it is a hosted commercial model, and their conclusions about the mechanism apply to open implementations. S45 Whether this also appears in Jev therefore cannot be known from the source. JEV.sk measured none of this. The two-call test is nevertheless a cheap procedure that a developer can run on their own data; how options are written is explained in the guide on Choice, Score and Noul, and typical deployments in online shops are described on the page Jev for online shops.

Limits of the study

PolicyBench is a synthetic suite built from templates, so it lacks the naturalness of real data. The definitions are the authors' own wording, and a different wording would shift the absolute numbers. The laya-ml model was close to chance level, so the authors draw no conclusions from it. They measured neither latency nor cost. S45 Because this is a preprint, the text has not gone through peer review.