Composite scoring with Jev: several criteria, weights in code, and what the sum hides

The TypeSafe documentation describes a pattern called composite scoring. A complex judgment is split into separate dimensions, Jev assesses each one with its own Score question, and only your code calculates the overall ranking, using weights you choose. S76 For an online shop that compares items on several criteria at once, this is a useful scheme: the model assesses the meaning of the text, and readable code decides what matters more. A weighted sum has a weak spot, however, which the pattern page does not mention.

The documentation example: screening resumes

The example evaluates job applicants' resumes. A single request contains the resume text and four Score questions: depth of Python knowledge, team leadership, system design, and the applicant's generalist breadth. Jev evaluates them in parallel and returns four scores. S76 The code divides each of them by four so that it falls between 0 and 1, and adds up the results with weights. S76

The weights change with the role. For a senior developer without a management role (senior IC in the example), Python and system design get 40% each, team leadership and generalist breadth 10% each. For an engineering manager, team leadership gets 40%, generalist breadth 25%, system design 20%, and Python 15%. S76 In both cases the weights add up to 100%.

The documentation sees the advantage in the weights sitting directly in the code, so it is clear how each result came about. When the ranking of the top applicants does not match expectations, you adjust the weights and the individual scores stay intact. S76 The page does not say explicitly that changing the weights requires no new model call; that follows from the weights being applied only in code. S76

How the score is calculated

A Score question contains an ordered list of level descriptions from lowest to highest. There must be at least two levels, the API accepts at most ten, and they are numbered by position starting from zero. S04 The score value in the response is not a single selected level but the mean of the level numbers weighted by their probabilities. The response also returns the probabilities of the individual levels (probabilities), their descriptions (legend), and a confidence value from 0 to 1. S04

The Score page describes composite scoring in three steps: divide each score by the top level number, that is the number of levels minus one, multiply it by a weight, and add up the results. In its ticket triage example, normalized scores of 0.62, 0.64, and 1.0 with weights of 0.6, 0.3, and 0.1 give 0.664. S04 According to the vendor, extra questions in a request barely lengthen the response time and cost only a few extra tokens; the page gives no measured figure for this. S04

Normalization matters most when dimensions have scales of different length. A score of 2 is the top of the scale with three levels and its middle with five. Without dividing, a dimension with a longer scale would quietly get more weight than you intended to give it. This is our own calculation, not a vendor figure.

What a weighted sum hides

The Score documentation points out that the same score can arise from different probability distributions. A value of 1.0 can mean that all probability is on level 1, but also that half falls on level 0 and half on level 2. It therefore advises reading probabilities and confidence together with the score. S04 Even high confidence does not mean the answer is correct; it only describes the model's answer itself. S04 The composite scoring pattern page does not mention confidence at all and treats each dimension as a single number. S76

In a composite score this problem gets worse. From a weighted sum of four means you can no longer tell whether the model judged a dimension clearly or wavered between the extreme levels. Our suggestion: store the confidence of each question alongside the overall score. An item where a heavily weighted dimension has low confidence should not be ranked automatically; flag it for review instead. According to the documentation, low confidence often means that the levels overlap for the given input, the question measures more than one thing, or the state lacks detail. S04 How to work with confidence is described in the guide confidence and calibration.

How to write levels for each dimension

The documentation recommends that each Score question measure only one dimension and that multi-part judgments be split into separate questions. S04 That is exactly the principle of composite scoring. Levels should describe concrete situations, not degrees such as "somewhat" or "very". S04 The model receives only the level descriptions and judges each one against the state on its own, without knowing its number or the neighbouring levels. S04 Numbers instead of descriptions work poorly: in an example from the documentation, levels "0", "1", and "2" gave a score of 0.55 at confidence 0.33, while descriptive levels for the same report gave 0.0 at confidence 1.0. S04 The basic question types are explained in the guide Choice, Score and Noul.

What this means for a Slovak online shop

The pattern fits wherever an online shop ranks items on several dimensions at once. Our example: for product descriptions from supplier feeds, separate Score questions can assess the completeness of technical data, the clarity of the text, and how well the description matches its category. The code builds from them a ranking of which descriptions to rewrite first, and the editorial team can change the weights without touching the questions. The documentation does not describe this use; it is our suggestion built on its scheme. S76

All questions go in a single request with shared text. S76 TypeSafe charges input tokens at 0.042 USD per million for both jev-1.14.0 and jev-1.13.0, and output is free. S01 You can estimate your own volume with the cost calculator; how to group several questions into one request is covered in our article on the fan-out pattern.

The documentation says nothing about Slovak, and the vendor itself advises checking levels against your own data. S04 For Slovak texts, set the level descriptions and weights only after a trial on a sample; the procedure is described in the guide testing in Slovak.