Offline statistics package

cli_sdk.stats is the network-free statistics engine: the same conformal, Venn-Abers, and e-value implementations the hosted service runs, as plain NumPy functions with no side effects. It never imports the HTTP client and never opens a connection, so it runs on a laptop, in a notebook, or inside an air-gapped network, on precomputed score arrays from any model — LLM or not.

from cli_sdk.stats.conformal import aps, cqr, crc, lac, ltt, mondrian, raps, rcps
from cli_sdk.stats.venn_abers import cvap, ivap, interval_width, merge_to_probability
from cli_sdk.stats.evalues import CoverageMonitor, RiskMonitor, anytime, ebh, ppi

The hosted service reports its engine version in every guarantee card as guarantee.stats_version. Pin the same cci-sdk version offline to reproduce a hosted result exactly. The math behind each module is on The three engines, Venn-Abers, and E-values.

Conformal prediction

Classification sets take cal_probs of shape (n, k) (probabilities over k classes), cal_labels of shape (n,) (true class indices in [0, k)), and return, for each test row, an array of class indices in the set.

import numpy as np
from cli_sdk.stats.conformal import aps, lac, raps
 
q_hat = aps.calibrate(cal_probs, cal_labels, alpha=0.10)
sets = aps.predict(test_probs, q_hat)            # list of index arrays, one per row
coverage = np.mean([y in s for y, s in zip(test_labels, sets)])

Shared helpers, importable from cli_sdk.stats.conformal:

Notes:

• aps randomizes calibration scores by default and returns deterministic sets at prediction time. With these defaults coverage is valid (at least $1-\alpha$) but conservative; pass randomize=True to predict as well for coverage close to exactly $1-\alpha$, at the cost of randomized sets.
• cqr wraps any lower/upper quantile predictions; the interval is (test_lower - q_hat, test_upper + q_hat).
• crc and rcps take one confidence score per calibration example (accept when the score is at least the threshold) and the 0/1 loss that example would incur if accepted. crc bounds expected risk; rcps bounds risk with probability $1-\delta$ over the calibration draw.
• ltt.calibrate_grid calls risk_fn(config) -> (risk_hat, n) for each candidate configuration. "conservative_first" runs fixed-sequence testing (configurations ordered from most to least conservative); "bonferroni" tests each at delta / len(configurations).
• mondrian.calibrate takes the nonconformity score of the true label for each calibration example (for LAC, 1 - cal_probs[i, cal_labels[i]]) and a group label per example. The result has group_thresholds, group_sizes, fallback_threshold, underpowered_groups, and threshold_for(group), which returns the marginal threshold for underpowered or unseen groups.

import numpy as np
from cli_sdk.stats.conformal import lac, mondrian
 
cal_scores = 1.0 - cal_probs[np.arange(len(cal_labels)), cal_labels]
cal = mondrian.calibrate(cal_scores, cal_groups, alpha=0.10)
print(cal.underpowered_groups)
 
for group in np.unique(test_groups):
    rows = test_groups == group
    group_sets = lac.predict(test_probs[rows], cal.threshold_for(str(group)))

Venn-Abers

from cli_sdk.stats.venn_abers import cvap, interval_width, ivap, merge_to_probability
 
p0, p1 = ivap.calibrate_and_predict(cal_scores, cal_labels, test_scores)
p = merge_to_probability(p0, p1)          # p1 / (1 - p0 + p1)
width = interval_width(p0, p1)            # p1 - p0

cal_scores is any real-valued score, higher meaning “more likely label 1”; cal_labels are binary. For a multiclass classifier, a common use is calibrating “the top-1 prediction is correct”: score each example by its top probability and label it 1 when the top class is right. See the offline calibration guide.

E-values

from cli_sdk.stats.evalues import CoverageMonitor, ebh
 
selected = ebh.select(e_values, q=0.10)   # indices selected at FDR level q
 
monitor = CoverageMonitor(target=0.88, false_alarm_rate=0.05)
for label, prediction_set in production_stream:
    alert = monitor.update(covered=label in prediction_set)
    if alert:
        page_oncall(alert)
        break

The monitor classes keep returning an alert dict on every update while the e-value stays above the threshold; act on the first one. cli_sdk.LocalMonitor wraps them, returns a typed Alert, and fires only once.

Network-free helpers outside stats

These live elsewhere in cli_sdk but make no network calls either: