Calibration
Choosing a confidence threshold
· 5 min read
Jev returns a probability with every answer, and the loudest criticism of the launch is that nothing public backs the calibration claim. No reliability curve, no expected calibration error, no ablation separating the training method from the architecture. The criticism is fair. It also does not stop you shipping, as long as you stop treating the number as a promise and start treating it as a threshold you own.
What a calibrated probability is supposed to mean
A calibrated model that says 0.70 should be right about 70 percent of the time across every case where it says 0.70. That is a different property from accuracy, and it is the property that lets software make a decision about the decision: act, abstain, or escalate.
Ordinary language models are famously bad at this. Ask one how sure it is and you get a number shaped like confidence with no relationship to being correct. The pitch for System One models is that the probability is load bearing. The gap is that the pitch has not been demonstrated in public, and the vendor has said plainly that being wrong with high confidence remains possible.
So measure it on your own data
The useful response is not to wait for someone else's reliability curve. It is to build yours, on your own traffic, for your own question. Calibration is not a property of a model in the abstract; it is a property of a model on a distribution, and your distribution is not Amazon ESCI.
The method is unglamorous. Log the answer and the probability for every call. Label a few hundred of them by hand. Bucket by confidence and compare each bucket to the accuracy you actually observed.
confidence n accuracy 0.50-0.60 412 0.58 0.60-0.70 690 0.71 0.70-0.80 1103 0.79 0.80-0.90 2204 0.88 0.90-1.00 5817 0.96
That table is the shape you are looking for, not real numbers from our traffic. If your buckets track the diagonal, the probability is telling you something and you can pick a threshold from it. If they do not, you have learned something more valuable than a threshold: on your question, the number is decoration, and you should be gating on something else.
What a threshold is actually protecting
Ours sits on one question. The assistant classifies whether a visitor message is a product search at all, and a search only proceeds when SEARCH both wins the vote and clears 0.75.
a = answers['intent'] allowed = a['choice'] == 'SEARCH' and a['probabilities']['SEARCH'] >= .75
Note what this is not. It is not a quality filter on the results. It is a gate on whether to act, and the asymmetry behind it is specific: running a confident wrong search inside somebody else's storefront is worse than asking the shopper to rephrase. A bad search shows a stranger the wrong products under a merchant's brand. A clarifying question costs a turn.
Pick the number from that asymmetry, not from a default. If abstaining is cheap and being wrong is expensive, gate high. If abstaining is expensive because every abstention is a lost sale, gate low and put the safety somewhere else. A threshold is a business decision wearing a decimal point.
Only gate the questions that deserve it
We do not threshold every answer. Most of the typed questions in a turn take the winning choice directly, because the cost of getting them slightly wrong is a slightly worse result rather than a wrong action. Whether a message continues the previous search or starts a new one, whether the shopper wants alternatives, which currency a budget is in: these degrade gracefully.
Gating everything would be worse than gating nothing. Every gate needs a branch for what happens when it fails, and a pile of abstention paths nobody has thought through is how an assistant becomes unable to answer anything. One gate on the question that decides whether to act at all carries most of the value.
Give the model somewhere to put uncertainty
A threshold catches low confidence. It does not help when the model is confidently answering a question that has no good answer, which happens whenever the options you supplied do not contain the right one.
So supply it. Our ranking candidates include an explicit NONE meaning no candidate satisfies the current constraints, and the intent question has an UNCLEAR option next to SEARCH, SUPPORT and OTHER. A model forced to choose among wrong options will choose one, and it will look confident doing it. Abstention has to be expressible inside the answer, not only outside it.
The honest summary
There is no published evidence that Jev's probabilities are well calibrated. There is also no published evidence that they are not, and the mechanism is at least the right shape for it. The way through is not to believe or disbelieve the claim. It is to stop needing it: log the probabilities, label a sample, check the buckets, and set a threshold you can defend from your own numbers.
Navlu is a conversational product discovery assistant for ecommerce stores. One Jev call per turn answers eight to sixteen typed questions about the conversation; a single confidence gate decides whether a search runs at all.