Confidence is not a guarantee of truth

What the model returns and what the application concludes from it

For Choice and Score, confidence is calculated from the probability distribution. It expresses how concentrated that distribution is, not an independent verification of the answer. Thresholds should be adapted to the specific task and to the consequences of an error. S06

TypeSafe describes its RLCD training as an effort to obtain well-calibrated decisions. Calibration is assessed across a group of predictions; it is no guarantee that a particular high-probability answer will be correct. S09

How we propose to validate

First, prepare manually evaluated examples. Set aside separate data for designing the questions, setting the thresholds and the final test. With a small sample, do not use percentages with decimal places as evidence of high certainty.

Track the accuracy of automatically accepted cases and their share. A model that hands almost everything over to a human can have high accuracy on its small automated portion but poor practical value. Conversely, a large share of automation can hide too many costly errors.

The simulator on JEV.sk

The simulator on JEV.sk explains this trade-off using hand-made values. It switches between an unambiguous inquiry, an unclear inquiry and a deliberately confident mistake. No number in the simulation is a measured Jev probability, and the simulation does not validate the model.

An automatic decision should have limits set in advance. For money transfers, data deletion and permission changes, a threshold alone is not enough. We recommend independent rules and approval.