Offline calibration

Problem

A data science team has a proprietary, already-trained classifier (not an LLM) that routes documents to one of five queues. They suspect its softmax outputs are overconfident, they know it is weaker on traffic from one region, and no data may leave their network — not even to a CLI-hosted service.

Primitives

No hosted primitive: everything runs in the network-free engine cli_sdk.stats, plus the local audit and monitor helpers.

conformal.aps for prediction sets with a coverage guarantee.
conformal.lac and conformal.mondrian for one threshold per region.
venn_abers.ivap for calibrated probability intervals on “the top prediction is correct”.
calibration.audit_coverage and LocalMonitor for audits and drift alarms.

Code

The script below runs as-is: load_scores generates a stand-in for the classifier’s outputs. Replace it with np.load of your own arrays.

import json
 
import numpy as np
 
from cli_sdk import LocalMonitor
from cli_sdk.calibration import audit_coverage
from cli_sdk.stats.conformal import (
    aps,
    conformal_quantile,
    coverage_confidence_interval,
    lac,
    minimum_calibration_size,
    mondrian,
)
from cli_sdk.stats.venn_abers import ivap
 
 
def load_scores(seed: int = 42):
    """Stand-in for your classifier's outputs. Replace with np.load(...) of:
    probs (n, k) softmax outputs, labels (n,) true class indices, groups (n,) segment labels."""
    rng = np.random.default_rng(seed)
    n, k = 6000, 5
    groups = rng.choice(["amer", "emea", "apac"], size=n, p=[0.5, 0.4, 0.1])
    labels = rng.integers(0, k, size=n)
    logits = rng.normal(size=(n, k))
    logits[np.arange(n), labels] += np.where(groups == "apac", 1.0, 3.0)   # weaker on apac
    probs = np.exp(2.0 * logits) / np.exp(2.0 * logits).sum(axis=1, keepdims=True)  # overconfident
    return probs, labels, groups
 
 
probs, labels, groups = load_scores()
rng = np.random.default_rng(0)
order = rng.permutation(len(labels))
cal, test = order[:3000], order[3000:]          # calibrate on one random half, evaluate on the other
alpha = 0.10
 
# 1. Is there enough calibration data?
print("minimum n:", minimum_calibration_size(alpha), "| calibration n:", len(cal))
print("realized-coverage interval:", np.round(coverage_confidence_interval(len(cal), alpha), 3))
 
# 2. APS prediction sets.
q_hat = aps.calibrate(probs[cal], labels[cal], alpha=alpha, seed=0)
sets = aps.predict(probs[test], q_hat)
covered = np.array([y in s for y, s in zip(labels[test], sets)])
print("APS coverage:", round(covered.mean(), 3), "| mean set size:", round(np.mean([len(s) for s in sets]), 2))
 
# 3. One marginal threshold versus one threshold per region (Mondrian), with LAC scores.
cal_scores = 1.0 - probs[cal, labels[cal]]
marginal = conformal_quantile(cal_scores, alpha)
by_region = mondrian.calibrate(cal_scores, groups[cal], alpha=alpha)
print("underpowered regions:", sorted(by_region.underpowered_groups))
for region in ["amer", "emea", "apac"]:
    rows = test[groups[test] == region]
    one = np.mean([y in s for y, s in zip(labels[rows], lac.predict(probs[rows], marginal))])
    per = np.mean([y in s for y, s in zip(labels[rows], lac.predict(probs[rows], by_region.threshold_for(region)))])
    print(f"  {region}: n_cal={by_region.group_sizes[region]} marginal={one:.3f} per-region={per:.3f}")
 
# 4. Venn-Abers intervals for "the top-1 prediction is correct".
cal_top = probs[cal].max(axis=1)
cal_correct = (probs[cal].argmax(axis=1) == labels[cal]).astype(int)
test_top = probs[test[:4]].max(axis=1)
p0, p1 = ivap.calibrate_and_predict(cal_top, cal_correct, test_top)
for raw, lo, hi, region in zip(test_top, p0, p1, groups[test[:4]]):
    print(f"  {region}: raw top-1 {raw:.2f} -> calibrated [{lo:.2f}, {hi:.2f}]")
 
# 5. Audit on held-out labelled data, as a CI job would.
audit = audit_coverage(covered.tolist(), target=1 - alpha)
print("audit:", audit.result, round(audit.realized_coverage, 3), (round(audit.ci_lower, 3), round(audit.ci_upper, 3)))
 
# 6. Version the result yourself: this JSON is your offline calibration profile.
with open("routing-classifier-v7.json", "w", encoding="utf-8") as handle:
    json.dump({"method": "APS", "alpha": alpha, "q_hat": q_hat, "n": int(len(cal)),
               "lac_group_thresholds": by_region.group_thresholds}, handle, indent=2)
 
# 7. Watch production coverage with the local e-process monitor.
monitor = LocalMonitor(type="coverage", target=0.88, false_alarm_rate=0.05, profile="routing-classifier-v7")
alarm = next((t for t, ok in enumerate(covered, start=1) if monitor.update(bool(ok))), None)
print("drift alarm at outcome:", alarm)

Output:

minimum n: 9 | calibration n: 3000
realized-coverage interval: [0.891 0.909]
APS coverage: 0.978 | mean set size: 1.47
underpowered regions: []
  amer: n_cal=1531 marginal=0.960 per-region=0.912
  emea: n_cal=1176 marginal=0.954 per-region=0.896
  apac: n_cal=293 marginal=0.510 per-region=0.906
  amer: raw top-1 0.94 -> calibrated [0.95, 0.97]
  emea: raw top-1 0.56 -> calibrated [0.53, 0.55]
  emea: raw top-1 0.99 -> calibrated [0.99, 0.99]
  emea: raw top-1 0.94 -> calibrated [0.95, 0.97]
audit: pass 0.978 (0.972, 0.983)
drift alarm at outcome: None

Walkthrough

Sizing. 3,000 calibration examples is far above the minimum of 9 at $\alpha = 0.10$ . The realized-coverage interval $[0.891, 0.909]$ says how far this particular calibration’s coverage can plausibly sit from 90%.

APS sets. The sets contain the true queue 97.8% of the time on held-out data, with 1.47 queues on average — valid (at least 90%) but conservative, because aps.predict is deterministic by default while calibration scores are randomized. aps.predict(..., randomize=True, seed=...) brings coverage close to exactly 90% (0.908 on this data, 1.26 queues on average), at the cost of sets that can differ between two calls on the same input.

One threshold hides a region. With a single marginal LAC threshold, apac traffic — where the classifier is weaker and still overconfident — is covered only 51% of the time, while the other regions are covered 95% or more; the average still looks fine. One threshold per region (mondrian.calibrate) brings every region to about 90%. With 293 calibration examples, apac is well above its minimum; a region below it would be listed in underpowered_groups and fall back to the marginal threshold.

Calibrated probabilities. IVAP maps the raw top-1 probability to an interval for “the top-1 prediction is correct”, learned from the calibration set. Narrow intervals here reflect 3,000 calibration points.

Audit and monitor. audit_coverage gives the Clopper-Pearson interval $[0.972, 0.983]$ for realized coverage and passes because it is not below the target. The same check runs from a shell with cli calibration audit-local (Command-line tool). LocalMonitor replays the held-out outcomes and, as expected on exchangeable data, never alarms.

Calibrate on the exact artifact you deploy. Re-exporting the model, changing its quantization, or changing preprocessing shifts the scores, and the thresholds in your JSON file stop meaning what they meant. Retrain or re-export, then recalibrate, then re-audit.

What the guarantee means

Offline there is no guarantee card; the JSON file you version is the profile. What it backs:

APS sets, $\alpha = 0.10$ . For documents exchangeable with the 3,000 calibration documents, sets built with q_hat contain the true queue at least 90% of the time, averaged over documents — not for any single document.
Per-region LAC thresholds. The same statement within each region, for every region at or above its minimum size.
IVAP intervals. The endpoint selected by the true outcome is calibrated on average over exchangeable documents (Venn-Abers).
The monitor. If coverage stays at or above 88%, the chance it ever alarms is at most 5%, however long it runs.

Cost routing Entity matching with Belief