Exchangeability

Every guarantee CLI produces rests on one statistical assumption: calibration data and production data are exchangeable — their joint distribution is unchanged by permuting which points are “calibration” and which are “test.” This is weaker than assuming the data is i.i.d., and it is the only assumption CLI’s engines make.

What exchangeability does not require

• It does not require the underlying model to be accurate.
• It does not require the underlying model’s probabilities to be well calibrated.
• It does not require you to know anything about how the model was trained.

A worse or less calibrated model degrades the usefulness of a CLI guarantee — sets get bigger, intervals get wider, more decisions escalate — but it never invalidates the guarantee itself. This is why CLI can offer the same guarantee shape on Anthropic Claude, which exposes no token-level signal at all, and on a self-hosted checkpoint with full access to logits and hidden states: the guarantee’s validity comes from the calibration procedure, not from the model.

What breaks exchangeability

In practice, exchangeability fails when the pipeline that produced your calibration data differs from the pipeline serving production traffic. Concretely:

• Model version drift. A vendor updates the model behind an alias, or a self-hosted deployment is upgraded to a new engine version.
• Prompt or template changes. Editing a system prompt or few-shot examples changes the scoring function even if the model is unchanged.
• Decoding parameter changes. Temperature, sampling count, or reasoning effort are part of the score, not incidental settings.
• Quantization or hardware changes. A self-hosted model’s numerical precision, batch size, or GPU type can shift token probabilities enough to move a calibrated threshold.
• Domain shift. Calibrating on one customer segment or language and serving another.
• Context-induced shift in agentic pipelines. Calibrating a model alone and then deploying it inside a multi-agent context — with other agents’ outputs or tool results in the prompt — changes the scoring mechanism even when the question distribution is unchanged.

How CLI protects against this

• Backend fingerprints. Every CalibrationProfile records the exact model identifier, prompt-template hash, and decoding parameters it was calibrated against (profile.backend_fingerprint). With strict_fingerprint=True (the default when you create a profile), a request whose live fingerprint does not match is refused rather than served against a stale calibration. See Backends: pinning and fingerprinting.
• Coverage audits. Profiles are re-checked against fresh labelled examples, and the realized-coverage confidence interval is exposed, not just a point estimate. See Calibration profiles: auditing.
• Anytime-valid drift monitors. Built on e-processes, so continuous monitoring of production traffic never inflates the false-alarm rate, no matter how often or for how long you check. See Drift monitoring and E-values.
• Fail-closed behavior. A primitive that cannot back its guarantee returns a heuristic-labelled answer rather than a silently invalid one, and the SDK can turn that into an exception. See Errors and retries.

Minimum calibration size

For a target miscoverage $\alpha$, a non-trivial (non-universal) prediction set requires at least

$n \ge \left\lceil \frac{1-\alpha}{\alpha} \right\rceil$

labelled, exchangeable calibration examples — 9 at $\alpha = 0.1$, 19 at $\alpha = 0.05$, 99 at $\alpha = 0.01$. Below that, the conformal quantile $\lceil (n+1)(1-\alpha) \rceil$ exceeds $n$ and the only valid set is the set of every option. CLI enforces this floor per profile (profile.minimum_n, profile.can_serve_guarantees), and per group when a profile uses group-conditional (Mondrian) calibration.

In practice, realized coverage given a fixed calibration set is itself a random variable: it follows a $\mathrm{Beta}(n+1-\ell,\ \ell)$ distribution with $\ell = \lfloor (n+1)\alpha \rfloor$. CLI recommends roughly 1,000 labelled examples for a stable $\alpha = 0.10$ guarantee, and exposes the resulting coverage interval on every profile (profile.realized_coverage_ci; see Calibration profiles). The offline engine computes the same interval with cli_sdk.stats.conformal.coverage_confidence_interval(n, alpha).