Venn-Abers calibration

Venn-Abers predictors turn any real-valued score for a yes/no question — a raw model probability, a logit, a judge rating — into a calibrated probability interval $[p_0, p_1]$. They come from the same exchangeability theory as conformal prediction (Vovk, Gammerman & Shafer) and, like it, need no assumption about how good the underlying score is.

CLI uses Venn-Abers for Belief, for the per-option venn_abers intervals on every Set answer, and inside Judge and Route cascades to decide when a cheaper stage is not sure enough.

The IVAP construction

Take $n$ calibration pairs $(s_i, y_i)$: a score $s_i$ (higher means “more likely 1”) and an observed label $y_i \in \{0, 1\}$. For a test score $s$:

1. For each postulated label $\ell \in \{0, 1\}$, add the point $(s, \ell)$ to the calibration set.
2. Fit an isotonic (non-decreasing) least-squares regression of label on score to those $n+1$ points, using pool-adjacent-violators.
3. Let $p_\ell$ be the fitted value at $s$.

The Inductive Venn-Abers Predictor (IVAP) outputs $[p_0, p_1]$, with $p_0 \le p_1$ by construction. This is exactly what cli_sdk.stats.venn_abers.ivap.calibrate_and_predict computes: one isotonic fit per postulated label per test point.

What is guaranteed

Let $p_Y$ be the endpoint computed by postulating the true label $Y$ (so $p_Y = p_0$ when $Y = 0$ and $p_Y = p_1$ when $Y = 1$). If calibration and test data are exchangeable, then

$\mathbb{E}\big[\,Y \mid p_Y\,\big] = p_Y,$

that is, $p_Y$ is perfectly calibrated (Vovk & Petej, 2014). You never know which endpoint is $p_Y$ for a given item, which is why the output is an interval: the calibrated probability is one of its two endpoints.

Interval width

$w = p_1 - p_0$ measures how much a single extra observation at $s$ could move the isotonic fit. It is small where many calibration scores sit near $s$ and the label rate there is stable, and large where calibration data is sparse or the relationship is noisy. The width is an ambiguity signal in its own right, available before any threshold is chosen (BeliefAnswer.interval_width, venn_abers.interval_width).

import numpy as np
from cli_sdk.stats.venn_abers import interval_width, ivap, merge_to_probability
 
rng = np.random.default_rng(3)
cal_scores = rng.uniform(size=2000)                               # any score
cal_labels = (rng.uniform(size=2000) < cal_scores).astype(int)    # observed 0/1 labels
test_scores = np.array([0.10, 0.50, 0.90])
 
p0, p1 = ivap.calibrate_and_predict(cal_scores, cal_labels, test_scores)
# p0 = [0.098 0.497 0.935], p1 = [0.113 0.502 0.946]
 
q0, q1 = ivap.calibrate_and_predict(cal_scores[:50], cal_labels[:50], test_scores)
# with only 50 examples: q0 = [0. 0.2 0.875], q1 = [0.167 0.706 1.]
 
print(interval_width(p0, p1), interval_width(q0, q1))
print(merge_to_probability(p0, p1))

Merging to one number

When downstream code needs a single probability, merge the interval with

$p = \frac{p_1}{1 - p_0 + p_1},$

the rule that minimizes worst-case log loss over the two endpoints (Vovk & Petej). For $[0.90, 0.96]$ this gives $0.96 / 1.06 \approx 0.906$. The probability field on a Belief answer is this merged value; the offline equivalent is venn_abers.merge_to_probability(p0, p1). Prefer the interval itself whenever the width matters to the decision.

The straddle rule

To act on a threshold $t$ (merge two records when the match probability is at least $t$, say), use the whole interval:

Because $p_Y$ is always one of the two endpoints, the first two rows never act on an item whose calibrated probability could lie on the other side of $t$, whichever label turns out to be true. The third row is exactly the case where the calibration data cannot tell you which side you are on. BeliefAnswer.straddles(t) returns True in that case.

The straddle rule is a decision rule on calibrated probabilities, not a bound on the error rate of the decisions it takes. If you need a stated bound on how often acted-on decisions are wrong, use a Gate, or audit the rule on fresh labelled data. The entity matching guide walks through both.

Cross Venn-Abers (CVAP)

A single IVAP uses one calibration split. The Cross Venn-Abers Predictor fits an IVAP on each of $K$ leave-one-fold-out subsets and combines the $K$ intervals, using the calibration data more fully at $K$ times the cost:

from cli_sdk.stats.venn_abers import cvap
 
p0, p1 = cvap.calibrate_and_predict(cal_scores, cal_labels, test_scores, n_folds=5, seed=0)

The combination rule for $K$ intervals is not uniquely prescribed in the literature. cvap averages each endpoint in log-odds space. Treat the result as an aggregate signal in the same spirit as a single IVAP, not as a strictly stronger guarantee.

Where it appears in the API

A Belief answer’s guarantee card has type: "calibration" with method: "IVAP", the profile name, and calibration_n, and no alpha. CVAP has no finite-sample validity guarantee, so a CVAP answer is labelled heuristic with method: "CVAP". See Guarantee reference.