E-values

An e-value for a null hypothesis $H_0$ is a nonnegative statistic $E$ with

$\mathbb{E}_{H_0}[E] \le 1.$

Large values are evidence against $H_0$: by Markov’s inequality, $P_{H_0}(E \ge 1/\delta) \le \delta$. E-values multiply across independent pieces of evidence, stay valid under optional stopping when built as e-processes, allow the significance level to be chosen after the fact, and give false-discovery-rate control under arbitrary dependence. CLI uses them for drift monitors, batch FDR control, and label-efficient calibration. All of it is in cli_sdk.stats.evalues.

Betting martingales

A drift monitor watches a stream of 0/1 outcomes $X_1, X_2, \dots$: a miss (the returned set did not contain the label) for a coverage monitor, or a loss for a risk monitor. The null hypothesis is that the rate has not risen above its calibrated value $p_0$:

$H_0 : \mathbb{E}[X_t \mid X_1, \dots, X_{t-1}] \le p_0 \quad \text{for every } t.$

For a betting fraction $\lambda \in [0, 1/p_0]$, define

$M_t(\lambda) = \prod_{i=1}^{t} \big(1 + \lambda (X_i - p_0)\big).$

Each factor is nonnegative (because $X_i \ge 0$ and $\lambda p_0 \le 1$) and has conditional expectation $1 + \lambda(\mathbb{E}[X_t \mid \text{past}] - p_0) \le 1$ under $H_0$, so $M_t(\lambda)$ is a nonnegative supermartingale starting at 1. Read it as the wealth of a gambler who bets a fraction of their wealth, each round, that the next outcome will be a miss: if misses really occur at rate at most $p_0$, the bet cannot be expected to make money.

CLI does not require you to guess the size of the drift. It averages five bets, $\lambda_k = c_k / p_0$ with $c_k \in \{0.10, 0.25, 0.45, 0.65, 0.85\}$:

$M_t = \frac{1}{5}\sum_{k=1}^{5} M_t(\lambda_k).$

An average of nonnegative supermartingales is again one, so $M_t$ keeps the same guarantee. This is anytime.BettingMartingale.

Ville’s inequality

For any nonnegative supermartingale $M_t$ with $M_0 = 1$,

$P_{H_0}\big(\exists\, t \ge 1 : M_t \ge 1/\delta\big) \le \delta.$

The bound is over the whole stream: it holds however long the monitor runs and however often you look at it. A monitor alarms the first time $M_t \ge 1/\delta$, where $\delta$ is the false_alarm_rate (so the threshold is 20 at the default 0.05). The probability that it ever alarms while $H_0$ holds is at most $\delta$.

from cli_sdk.stats.evalues.anytime import BettingMartingale
 
m = BettingMartingale(null_rate=0.10, false_alarm_rate=0.05)
for miss in [0, 0, 1, 0, 1, 1, 0, 1]:
    m.update(miss)
print(m.n, round(m.e_value, 2), m.triggered)   # 8 27.85 True

A classical test controls its error only at one pre-committed sample size. Re-running it after every new outcome and alarming the first time it rejects inflates the false-alarm rate far above its nominal level. In a simulation with 400 in-spec streams of 3,000 outcomes each (miss rate exactly 0.10), a one-sided z-test at level 0.05, re-run after every outcome from the 30th on, alarmed on 43% of the streams. LocalMonitor at false_alarm_rate=0.05 alarmed on 4.25%.

Coverage and risk monitors

CoverageMonitor.update and RiskMonitor.update return an alert dict whenever the e-value is at or above the threshold (and None otherwise); act on the first one. LocalMonitor wraps them and returns a typed Alert only the first time.

Detection delay

The monitor tests “the rate has been at most $p_0$ since the monitor started”. It is not a change-point detector with a bounded delay: during a long in-spec period the e-value shrinks toward zero and has to be rebuilt after a change. In simulation (coverage target 0.90, miss rate doubling from 0.10 to 0.20), a freshly started monitor fired after a median of 59 labelled outcomes; one that had first watched 1,000 in-spec outcomes fired a median of about 740 outcomes after the change. The further the pre-change rate sits inside the null, the faster the e-value shrinks and the longer the delay: with a 0.88 target and 90% coverage before the change, the median delay was 1,750 outcomes. Start a new monitor with each new profile version, and see the silent model update guide for a worked example and for windowed monitoring.

Post-hoc significance levels

For any e-value $E$ and any level $\alpha > 0$,

$\frac{\mathbf{1}\{E \ge 1/\alpha\}}{\alpha} \le E,$

so for a level $\tilde\alpha$ chosen after looking at the data, $\mathbb{E}\big[\mathbf{1}\{E \ge 1/\tilde\alpha\}/\tilde\alpha\big] \le 1$. The ratio of the chance of a false rejection to the level you reported stays bounded in expectation, which fixed-level p-value procedures cannot offer. In practice: report $1/E$ as the smallest level at which the evidence rejects. An Alert carries its e_value for this purpose.

e-BH: FDR control under arbitrary dependence

Given e-values $e_1, \dots, e_m$ for $m$ hypotheses and a target false-discovery rate $q$, sort them in decreasing order $e_{(1)} \ge \dots \ge e_{(m)}$ and let

$k^* = \max\Big\{k : e_{(k)} \ge \frac{m}{q\,k}\Big\}$

(zero if no $k$ qualifies). Rejecting the $k^*$ hypotheses with the largest e-values controls $\mathrm{FDR} \le q\, m_0 / m \le q$, where $m_0$ is the number of true nulls, whatever the dependence between the e-values (Wang & Ramdas, 2022). Benjamini-Hochberg on p-values needs independence or positive dependence for the same statement. CLI uses e-BH for Gate batches in "fdr" mode when the decisions may be correlated.

import numpy as np
from cli_sdk.stats.evalues import ebh
 
e_values = np.array([45.0, 0.3, 120.0, 2.0, 18.0, 0.9, 60.0, 1.1, 0.2, 35.0])
print(ebh.select(e_values, q=0.10))   # [0 2 6 9]

Here $m/q = 100$: the sorted e-values 120, 60, 45, 35 clear $100/1, 100/2, 100/3, 100/4$, and the fifth (18) misses $100/5 = 20$.

From p-values to e-values

A valid p-value $P$ (super-uniform under $H_0$) becomes a valid e-value through a calibrator:

$f_\kappa(p) = \kappa\, p^{\kappa - 1}, \qquad \kappa \in (0, 1).$

Since $\int_0^1 \kappa\, p^{\kappa-1}\, dp = 1$ and $f_\kappa$ is decreasing, $\mathbb{E}[f_\kappa(P)] \le 1$. The naive $1/p$ is not a valid e-value (its expectation under a uniform p-value is infinite). With the default $\kappa = 1/2$, $f(p) = 1/(2\sqrt{p})$:

from cli_sdk.stats.evalues import ebh
 
ebh.calibrator_from_p_value(0.01)     # 5.0
ebh.calibrator_from_p_value(0.0001)   # 50.0

Prediction-powered inference

Label-efficient calibration estimates a rate (coverage, error rate, judge-human agreement) from a small human-labelled sample plus a large pool labelled by a judge model. Let $Y_i$ be human labels and $f_i$ the judge’s labels on the same $n$ items, and $\tilde f_j$ the judge’s labels on $N$ further items. The prediction-powered estimate of $\theta = \mathbb{E}[Y]$ is

$\hat\theta_\lambda = \lambda \cdot \frac{1}{N}\sum_{j=1}^{N} \tilde f_j \;+\; \frac{1}{n}\sum_{i=1}^{n} \big(Y_i - \lambda f_i\big).$

The second term is a rectifier: it measures the judge’s bias on the items humans labelled and subtracts it. For a fixed $\lambda$ the estimate is unbiased whenever the human-labelled items are a uniformly random subset of the same population as the pool, whatever the judge’s quality. Its standard error is

$\widehat{\mathrm{SE}} = \sqrt{\frac{\lambda^2\, \widehat{\mathrm{Var}}(\tilde f)}{N} + \frac{\widehat{\mathrm{Var}}(Y - \lambda f)}{n}},$

and the reported $1-\alpha$ interval is $\hat\theta_\lambda \pm z_{1-\alpha/2}\, \widehat{\mathrm{SE}}$. With tune_lambda=True (PPI++), CLI sets $\hat\lambda = \mathrm{clip}\big(\widehat{\mathrm{Cov}}(f, Y) / \widehat{\mathrm{Var}}(f),\ 0,\ 1\big)$, which is never worse than the human-only estimate ($\lambda = 0$) up to estimation error; tune_lambda=False uses classical PPI ($\lambda = 1$).

from cli_sdk.stats.evalues import ppi
 
result = ppi.estimate_mean(
    labelled_predictions=judge_on_human_sample,   # (n,) judge labels on the human-labelled items
    labelled_labels=human_labels,                 # (n,) human labels on the same items
    unlabelled_predictions=judge_on_pool,         # (N,) judge labels on the pool
    alpha=0.05,
)
print(result.estimate, result.ci_lower, result.ci_upper, result.lam, result.standard_error)

See the label-efficient calibration guide for an end-to-end run.