The three engines

CLI has three statistics engines. All three live in one network-free package, cli_sdk.stats, which is also what the hosted service runs; every hosted guarantee card reports the engine version in guarantee.stats_version.

Conformal prediction

Split conformal sets

Given any nonconformity score $s(x, y)$ — higher means “less plausible” — and $n$ labelled calibration pairs $(x_i, y_i)$ drawn exchangeably with production data, split conformal prediction computes

$\hat q = \text{the } \lceil (n+1)(1-\alpha) \rceil\text{-th smallest of } s(x_1,y_1), \dots, s(x_n,y_n)$

and returns $C(x) = \{y : s(x,y) \le \hat q\}$. This set satisfies

$1-\alpha \;\le\; P\big(Y_{n+1} \in C(X_{n+1})\big) \;\le\; 1-\alpha+\frac{1}{n+1}$

for any score and any underlying model, in finite samples, under exchangeability alone (the upper bound needs continuous scores without ties). The probability is over the calibration draw and the test point together. If $\lceil (n+1)(1-\alpha) \rceil > n$, then $\hat q = +\infty$ and every option is included; this is why a profile needs $n \ge \lceil (1-\alpha)/\alpha \rceil$ examples.

Risk control

A gate accepts an input when its confidence score clears a threshold $\lambda$. Write $L_i(\lambda)$ for the loss on calibration example $i$ at threshold $\lambda$ (for a gate: 1 if the example would be accepted and is wrong, 0 otherwise), with $L_i \le B$ and $L_i$ non-increasing in $\lambda$, and $\hat R_n(\lambda) = \frac{1}{n}\sum_i L_i(\lambda)$.

Conformal risk control (CRC) picks

$\hat\lambda = \inf\Big\{\lambda : \tfrac{n}{n+1}\hat R_n(\lambda) + \tfrac{B}{n+1} \le \alpha\Big\}$

and guarantees $\mathbb{E}\big[L_{n+1}(\hat\lambda)\big] \le \alpha$ — a bound on the expected loss, over the calibration draw and the test point. Powers Gate(guarantee="risk").

Risk-controlling prediction sets (RCPS) instead test, for each candidate $\lambda$ from most to least conservative, the null $H_0(\lambda) : R(\lambda) > \alpha$ with a Hoeffding-Bentkus p-value, and stop at the first $\lambda$ that cannot be rejected at level $\delta$ (fixed-sequence testing). The result satisfies

$P_{\text{calibration}}\big(R(\hat\lambda) \le \alpha\big) \ge 1 - \delta,$

a high-probability bound: with probability at least $1-\delta$ over the draw of the calibration set, the deployed threshold’s true risk is at most $\alpha$. Powers Gate(guarantee="risk_high_probability") and the agreement guarantee of Judge.

Learn-then-Test (LTT) generalizes RCPS to several jointly tuned knobs (for example a sample count and a quality threshold) and to non-monotone losses, by testing every configuration and controlling the family-wise error rate (fixed-sequence or Bonferroni).

Group-conditional (Mondrian) calibration

Fitting one quantile per group $g$ (account tier, language, product line) gives $P(Y \in C(X) \mid G = g) \ge 1-\alpha$ for every group with enough calibration data. Groups below $\lceil (1-\alpha)/\alpha \rceil$ examples fall back to the marginal threshold and are reported as underpowered (profile.groups[g].status == "underpowered"), so a marginal 90% cannot hide a much lower rate on a group you care about.

Selection with FDR control

When a gate auto-approves many decisions, the relevant quantity is the false discovery rate: the expected fraction of approved decisions that are wrong. Conformal selection computes a conformal p-value for each candidate from the calibration set and applies Benjamini-Hochberg, giving $\mathrm{FDR} \le q$. When the decisions may be dependent (related inputs, the same underlying model), CLI uses e-BH on e-values instead, which keeps $\mathrm{FDR} \le q$ under arbitrary dependence; see E-values: e-BH. Powers Gate(guarantee="fdr").

Venn-Abers calibration

An Inductive Venn-Abers Predictor (IVAP) fits isotonic regression twice — once postulating that the test label is 0, once that it is 1 — and returns the two fitted values $[p_0, p_1]$. The endpoint selected by the true label is perfectly calibrated on average over exchangeable data, whatever the quality of the underlying score; a poor score only widens the interval. Full construction, the merge rule $p = p_1/(1 - p_0 + p_1)$, and the straddle decision rule are on Venn-Abers calibration.

E-values

An e-value is a nonnegative statistic $E$ with $\mathbb{E}[E] \le 1$ under a null hypothesis. E-values support safe optional stopping (monitor continuously, stop whenever you like), e-BH FDR control under arbitrary dependence, and post-hoc choice of the significance level. CLI’s drift monitors are betting martingales, and its label-efficient calibration uses prediction-powered inference. Full construction, Ville’s inequality, the e-BH rule, the p-to-e calibrator, and the PPI estimator are on E-values.

How they compose

A single CLI deployment typically uses all three together: Venn-Abers turns a raw score into a calibrated probability interval; the conformal engine turns scores into coverage-, risk-, or FDR-bounded decisions; the e-value engine watches the deployed decision stream for drift and lets the calibration set grow from a small human sample plus a large judge-labelled pool.