How to test Jev in Slovak

What we know so far

The vendor names English as the main training language and notes that other languages do not achieve the same results. S01 Accuracy in Slovak cannot be inferred from an English banking test. The closest public data point from our language region is a Czech test of internal linking between articles on vibecoding.cz; however, it concerns a different language and a single task. S27

The test set we propose

As a start, we recommend 300 to 500 manually checked examples, not as universal proof of quality. Cover common inquiries, abbreviated messages, text without diacritics, typos, negations and incomplete information. We evaluate rare high-risk categories separately.

Original customer messages require an appropriate legal basis and data minimization. Synthetic examples can be prepared for a first technical test; however, they must be labeled as such and must not replace a real, representative sample when deciding on production use.

Separate data splits

Use one part to formulate the criteria, a second for the thresholds, and leave the third untouched until the end. Record the versions of the rules and models. If you find an error in the test set, do not reuse it as a “new independent test” without disclosing this.

Comparing variants

We propose a Slovak task definition, English instructions over Slovak content and, if needed, a translation-based workflow. When translating, account for the cost, the time and possible new errors. All variants should receive the same cases.

The results should show accuracy by category, the number of critical errors, the share of automation and the cost per correct case. When Jev does not know, a correct handover to a human can count as a success, rather than a forced category.