Calibration profiles

A CalibrationProfile is a versioned, auditable resource — the labelled data, the backend fingerprint, the method, and the target $\alpha$ that back every guarantee CLI issues. Every primitive names one with calibration_profile=..., and every guarantee card names it back.

All profile operations live on client.calibration_profiles (the same methods, awaited, on AsyncCLIClient).

Creating a profile

from cli_sdk import CLIClient, OpenAIBackend
 
with CLIClient() as client:
    profile = client.calibration_profiles.create(
        name="support-routing-v3",
        backend=OpenAIBackend(model="gpt-4.1-2025-04-14"),
        method="APS",
        alpha=0.10,
        prompt_template_hash="sha256:9f2c...",   # optional, recorded in the fingerprint
    )
    print(profile.status)    # "collecting"

Argument	Default	Meaning
`name`	required	Profile name, referenced by queries.
`backend`	required	The backend the profile is calibrated against (object or dict).
`method`	`"APS"`	One of `LAC`, `APS`, `RAPS`, `CQR`, `ordinal-aps`, `CRC`, `RCPS`, `LTT`, `IVAP`, `CVAP`, `conformal-factuality`, `trust-or-escalate`, `calibrated-cascade`.
`alpha`	`0.10`	Target level, in $(0, 1)$ .
`group_by`	`None`	Context field for group-conditional (Mondrian) calibration.
`prompt_template_hash`	`None`	Hash of the prompt template, part of the backend fingerprint.
`strict_fingerprint`	`True`	Refuse requests whose live backend fingerprint does not match.

An unknown method or an alpha outside $(0, 1)$ raises ValueError before any request is sent.

Adding examples

from cli_sdk import CalibrationExample
 
profile = client.calibration_profiles.add_examples(
    "support-routing-v3",
    examples=[
        CalibrationExample(context={"ticket": "I was charged twice."}, label="billing"),
        {"context": {"ticket": "The export button does nothing."}, "label": "technical"},
    ],
)
print(profile.n)

Examples can be CalibrationExample objects or plain dicts with context and label keys (Route profiles use tier_outputs alongside context; see Route).

`CalibrationExample` field	Default	Meaning
`context`	required	The same shape you pass to `evaluate`.
`label`	required	The correct answer.
`source`	`"human"`	`"human"` or `"judge"`: who produced the label. Anything else raises `ValueError`.
`group`	`None`	Mondrian group, when the `group_by` field is not in `context`.
`metadata`	`{}`	Free-form metadata stored with the example.

Calibration examples must be exchangeable with the traffic the guarantee will cover: a random sample of real production inputs, labelled, not a curated set of hard or easy cases. See Exchangeability.

Reading a profile

profile = client.calibration_profiles.get("support-routing-v3")
print(profile.n)                     # 1204
print(profile.minimum_n)             # 9
print(profile.recommended_n)         # 1000
print(profile.realized_coverage_ci)  # (0.886, 0.914)
print(profile.can_serve_guarantees)  # True
print(profile.last_audit)            # {"date": "2026-09-18", "sample": 300, "result": "pass"}
print(profile.backend_fingerprint)   # {"model": "gpt-4.1-2025-04-14", "prompt_template_hash": "sha256:..."}
 
for p in client.calibration_profiles.list():
    print(p.name, p.version, p.status)

Attribute	Meaning
`name`, `version`, `method`, `alpha`	Identity and configuration.
`n`	Labelled examples currently backing the threshold.
`minimum_n`	Hard floor $\lceil (1-\alpha)/\alpha \rceil$ .
`recommended_n`	Size for a stable guarantee (see Sizing).
`realized_coverage_ci`	Interval for realized coverage given this calibration set.
`can_serve_guarantees`	`n >= minimum_n`.
`last_audit`	The most recent audit record, if any.
`backend_fingerprint`	What the profile was calibrated against.
`group_by`, `groups`	Mondrian field, and per-group `GroupStatus(n, status)`.
`status`	`"collecting"`, `"serving"`, or `"stale"`.

Sizing

$\alpha$	Hard minimum $\lceil (1-\alpha)/\alpha \rceil$	Recommended for a stable guarantee
0.20	4	about 300
0.10	9	about 1,000
0.05	19	about 1,500
0.01	99	about 2,500

A profile below its hard minimum cannot serve coverage, risk, or fdr guarantees at all — queries against it return a heuristic-labelled answer instead (see Errors and retries). Below the recommended size, CLI still serves the guarantee but reports a wider coverage_ci, so callers see the real uncertainty in the calibration itself rather than a false sense of precision.

The same numbers are available offline:

from cli_sdk.calibration import recommended_size
from cli_sdk.stats.conformal import coverage_confidence_interval, minimum_calibration_size
 
minimum_calibration_size(0.10)                 # 9
recommended_size(0.10)                         # 1000
coverage_confidence_interval(1204, 0.10)       # 90% interval for realized coverage

Realized coverage given a fixed calibration set of size $n$ follows a $\mathrm{Beta}(n+1-\ell,\ \ell)$ distribution with $\ell = \lfloor (n+1)\alpha \rfloor$ , centred just above $1-\alpha$ , and its spread shrinks roughly as $1/\sqrt{n}$ . The guarantee is the average over calibration draws; the interval tells you how far your particular draw can sit from it.

Group-conditional (Mondrian) profiles

client.calibration_profiles.create(
    name="support-routing-by-tier",
    backend=OpenAIBackend(model="gpt-4.1-2025-04-14"),
    method="APS",
    alpha=0.10,
    group_by="account_tier",
)
 
profile = client.calibration_profiles.get("support-routing-by-tier")
for tier, status in profile.groups.items():
    print(tier, status.n, status.status)   # e.g. "enterprise 212 calibrated", "startup 6 underpowered"

Each distinct value of account_tier seen in the calibration examples gets its own quantile, and its own minimum- $n$ enforcement. Groups below the minimum fall back to the marginal (ungrouped) threshold and are reported with status == "underpowered": their answers carry the marginal guarantee, not a per-group one.

Label-efficient calibration

Human labelling is almost always the bottleneck. A profile can combine a large judge-labelled pool with a small, random human-labelled sample through prediction-powered inference (PPI). The combined estimate is unbiased whatever the judge’s quality: a poor judge widens its interval, it does not bias it. The interval is a large-sample (central limit theorem) interval, not a finite-sample one. The finite-sample conformal guarantees on the profile’s answers rest on the human-labelled examples; the judge-labelled pool narrows estimates and monitoring, it does not replace them.

job = client.calibration_profiles.label_with_judge(
    "support-routing-v3",
    judge=OpenAIBackend(model="gpt-4.1-2025-04-14"),
    unlabelled_examples=unlabelled,       # contexts, or dicts with a "context" key
    human_labelled_sample_size=300,
)
print(job)   # the human-labelling task list and a judge-quality diagnostic

label_with_judge returns the job as a dict. human_labelled_sample_size must be at least 2. Label the items it selects for humans and upload them with add_examples(..., CalibrationExample(..., source="human")).

You can run the same estimate locally before paying for labels:

from cli_sdk.calibration import estimate_rate_with_judge, judge_quality
 
quality = judge_quality(human_labels, judge_labels_on_same_items)
print(quality.agreement_with_humans, quality.correlation, quality.effective_label_multiplier)
 
estimate = estimate_rate_with_judge(
    human_labels,
    judge_labels_on_same_items,
    judge_labels_on_pool,
    alpha=0.05,
)
print(estimate.estimate, estimate.ci_lower, estimate.ci_upper)

A judge that is no more accurate than the model being evaluated cannot reduce the required number of human labels by more than about half — a mathematical limit of debiasing, not a CLI restriction. The judge-quality diagnostic shows whether you are near that limit. The label-efficient calibration guide and E-values: prediction-powered inference explain the estimator.

Auditing

audit = client.calibration_profiles.audit(
    "support-routing-v3",
    fresh_examples=recent_labelled_sample,
)
print(audit.result)             # "pass" or "fail"
print(audit.passed)             # True
print(audit.realized_coverage)  # e.g. 0.903
print(audit.ci_lower, audit.ci_upper, audit.sample, audit.target)

An audit fails only when the whole confidence interval for realized coverage sits below the target, that is, when the fresh sample gives real evidence of under-coverage rather than landing a little under by chance. The offline equivalents are cli_sdk.calibration.audit_coverage(covered, target, confidence=0.95) and, for 0/1 losses, audit_risk(losses, target), which reports realized_coverage = 1 - risk so both results have the same shape.

Run audits on a schedule, or wire them into CI with the command-line tool so a deploy is blocked when an audit fails:

cli calibration audit support-routing-v3 --examples fresh.jsonl --fail-below 0.88

Backends Drift monitoring