Guarantee reference

Every CLI answer includes a guarantee object, the guarantee card. This page is the canonical reference for what each field means and what it does — and does not — promise. In the SDK the card is answer.guarantee, a frozen Guarantee dataclass.

Schema

{
  "type": "coverage | risk | risk_high_probability | fdr | anytime | cost_budget | heuristic",
  "method": "LAC | APS | RAPS | CQR | ordinal-aps | CRC | RCPS | LTT | IVAP | CVAP | conformal-selection | e-BH | e-process | conformal-factuality | trust-or-escalate | calibrated-cascade",
  "alpha": 0.10,
  "delta": 0.05,
  "target": 0.05,
  "target_cents": 0.4,
  "realized_upper_bound": 0.047,
  "statement": "human agreement >= 0.90 on non-escalated verdicts",
  "calibration_profile": "support-routing-v3",
  "calibration_n": 1204,
  "coverage_ci": [0.886, 0.914],
  "last_audited": "2026-09-18T00:00:00Z",
  "stats_version": "0.1.0"
}

Only the fields that apply to a given guarantee type are present.

Two helpers read the card: guarantee.is_heuristic (also on every answer as answer.is_heuristic) and guarantee.describe(), which returns a one-line plain-language statement:

card = result.answers["department"].guarantee
print(card.describe())
# Contains the correct answer at least 90% of the time on profile 'support-routing-v3' (n=1204).

Guarantee types, precisely

coverage

“The returned set or interval contains the correct answer at least $1-\alpha$ of the time, averaged over exchangeable data.” Produced by Set and Interval. The upper bound is $1-\alpha+\frac{1}{n+1}$ when scores have no ties.

calibration

Produced by Belief with method IVAP, and no alpha. It states Venn-Abers validity: of the pair $[p_0, p_1]$, the probability computed under the true label is calibrated on data exchangeable with the profile. It is not a coverage statement and not a $1-\alpha$ interval for one item’s “true probability” (see Venn-Abers calibration). IVAP has this finite-sample property. CVAP averages several IVAP folds, which is usually sharper in practice but has no finite-sample validity guarantee, so CVAP answers are labelled heuristic with method CVAP.

risk

“The expected value of a stated loss is at most target.” Produced by Gate in "risk" mode (0/1 loss on auto-approved decisions), by Claim (loss 1 when any retained claim is false), and by Route in "accuracy" mode (loss 1 when the served answer is wrong). This is a statement about an expectation over the calibration draw and the test point, not a high-probability bound.

risk_high_probability

“With probability at least $1-\delta$ over the draw of the calibration set, the risk of the deployed threshold is at most target.” Strictly stronger than risk, and the form most audit and compliance processes want. Produced by Gate in "risk_high_probability" mode and by Judge.

fdr

“Among the decisions the gate auto-approves, the fraction that are wrong is at most target.” A statement about the population of approved decisions, not about any one of them. Produced by Gate in "fdr" mode, in two forms:

• One decision per request (method LTT, with delta): with confidence $1-\delta$ over the calibration draw, the error rate among approved decisions is at most target (a selective Learn-then-Test bound). describe() prints the “With … confidence” form.
• A batch decided together (method conformal selection with Benjamini-Hochberg, or e-BH, no delta): the expected fraction of approved items that are wrong is at most target.

anytime

“This monitor’s false-alarm rate is controlled at $\delta$ no matter when, or how often, it is checked.” A property of drift monitors, never of a per-call query; a monitor’s Alert carries its e_value and false_alarm_rate.

cost_budget

”$P(\text{cost per request} \le \text{target\_cents}) \ge 1-\alpha$.” Produced by Route. Up to an $\alpha$ fraction of requests can exceed the budget.

heuristic

Reserved for any value CLI returns that does not carry one of the above guarantees — for example, an answer served against a profile with too few calibration examples. A heuristic answer is always labelled as such. The SDK also treats as heuristic any answer whose card is missing or has a type it does not recognize, so nothing is ever mistaken for a proven statement. See Errors and retries for how to refuse heuristic answers outright.

Reading calibration_n and coverage_ci

calibration_n is the number of labelled examples backing the current threshold. coverage_ci is the interval for realized coverage implied by that sample size (see Calibration profiles: sizing). Read it as “how far the actual coverage of this particular calibration could plausibly sit from the target, given how much data it was calibrated on”, not as an estimate that the guarantee itself might be wrong: the guarantee is the average over calibration draws, and coverage_ci describes how realized coverage varies from one calibration draw to another at this $n$.