Local mode

LocalCLIClient runs CLI’s calibration and evaluation inside your own process, against a model you bring. It takes the same queries as the hosted CLIClient, returns the same typed answers with the same guarantee cards, and fails closed the same way. Nothing is sent anywhere except to the model provider you configure, and every calibration profile is a plain JSON file you can commit, review and ship with your service.

Local mode is what every agent framework example uses, and it is the quickest way to try CLI: a model API key, or a self-hosted model on vLLM or SGLang, is all it needs.

from cli_sdk.local import LocalCLIClient    # also importable as: from cli_sdk import LocalCLIClient

Every code block on this page is part of one script that was run against MockEvidenceBackend (the SDK’s keyless, deterministic stand-in model) on synthetic data; the output shown is what it printed. Replace the mock with any evidence backend and nothing else changes. The script starts with a refund policy, the context builder, a stand-in scoring function, and synthetic labelled history:

import json
import random
 
POLICY = "Refunds up to 100 USD are within policy. Larger refunds need a receipt on file."
 
 
def refund_context(args):
    return {"policy": POLICY, "amount_usd": float(args["amount_usd"]),
            "receipt_on_file": bool(args["receipt_on_file"])}
 
 
def demo_scorer(context, instructions, options):
    if context["amount_usd"] <= 100:
        p = 0.95
    else:
        p = 0.70 if context["receipt_on_file"] else 0.10
    return {"true": p, "false": 1.0 - p}
 
 
def synthetic_history(n, seed):
    rng = random.Random(seed)
    rows = []
    for _ in range(n):
        args = {"amount_usd": rng.choice([15, 40, 80, 250, 600, 1400]),
                "receipt_on_file": rng.random() < 0.5}
        correct = args["amount_usd"] <= 100 or args["receipt_on_file"]
        if rng.random() < 0.04:
            correct = not correct
        rows.append({"context": refund_context(args), "label": correct})
    return rows
 
 
labelled = synthetic_history(300, seed=7)
fresh = synthetic_history(200, seed=8)

When to use it

Use local mode when	Use the hosted `CLIClient` when
You want to try CLI with nothing but a model key	Several services or teams share one set of calibration profiles
The model runs inside your network and nothing may leave it	You want managed drift monitors with alerting, including fingerprint monitors
Profiles should live in version control next to the code	You want label-efficient calibration jobs (`label_with_judge`)
Tests and CI must run offline and deterministically	You need evidence above L1: prompt scoring (L2), exact label-token probabilities (L3) or hidden-state probes (L4); see the access ladder
	You need `Interval(method="CQR")` or `Route(guarantee="cost_budget")`, which local mode does not implement

The agent integrations accept either client: ToolGuard(client, rules) only needs an object with evaluate(context, queries), so moving a guard from local to hosted is a one-line change.

Create a client

from cli_sdk import Gate
from cli_sdk.evidence import MockEvidenceBackend
from cli_sdk.local import LocalCLIClient
 
client = LocalCLIClient(
    MockEvidenceBackend(scorer=demo_scorer),   # the evidence model: any cli_sdk.evidence backend
    store=".cli_profiles",                     # one JSON file per calibration profile
    sample_count=20,                           # samples per score for L0 (sampling-only) backends
    strict_guarantees=False,                   # True raises InsufficientCalibrationError instead of heuristic answers
    max_workers=4,                             # concurrent model calls while calibrating
)

Parameter	Default	Meaning
`backend`	`None`	The default evidence backend: any `CustomBackend`, usually one from `cli_sdk.evidence`. Every method also takes `backend=` per call.
`backends`	`None`	Named backends for `Judge` and `Route` cascades, for example `{"small": ..., "large": ...}`. With exactly one entry and no `backend`, that entry becomes the default.
`store`	`".cli_profiles"`	A directory (relative to the working directory) or a `LocalProfileStore`. Holds one `<profile name>.json` per profile.
`sample_count`	`20`	Samples per score when the backend is at access level L0. At L0 it is part of the profile’s fingerprint. Must be at least 1.
`strict_guarantees`	`False`	Raise `InsufficientCalibrationError` instead of returning a `heuristic` answer.
`max_workers`	`4`	Concurrent model calls while calibrating.

Calibrate

refund_gate = Gate(instructions="Is approving this refund correct under the policy?",
                   calibration_profile="refunds-demo-v1", guarantee="risk", target=0.05)
profile = client.calibrate(refund_gate, labelled,
                           progress=lambda done, total: done == total and print(f"scored {done}/{total}"))
print(profile.name, profile.method, profile.n, profile.version, profile.minimum_n, profile.recommended_n,
      profile.status)

scored 300/300
refunds-demo-v1 CRC 300 1 19 1500 serving

calibrate(query, examples, *, backend=None, replace=True, progress=None) scores every labelled example once with the evidence model and stores the scores and labels as the query’s profile, under the name in query.calibration_profile. It returns a CalibrationProfile.

replace=True (the default) starts a new version of the profile: the version number increments and the old records are discarded.
add_examples(query, examples, *, backend=None, progress=None) appends to the existing profile instead (it is calibrate(..., replace=False)). It raises ConfigurationError if the stored profile was built with a different query or backend, because the old and new scores would then come from different scoring functions.
progress(done, total) is called after each scored example.

Examples are {"context": ..., "label": ...} dicts or CalibrationExample objects. An optional group sets the Mondrian group when the query’s group_by field is not in the context. The label depends on the query type:

Query	Label
`Set`	The correct option key
`Interval`	The correct level
`Belief`	`True` if the statement is true
`Gate`	`True` if approving would be correct (the loss is approving a wrong one)
`Claim`	A list of `{"text": str, "supported": bool}`, one per claim in the answer
`Judge`	`"response_a"` or `"response_b"`: the human preference
`Route`	The correct option key of `Route.task`

Scoring cost is one model request per example (per claim for Claim, per stage or tier for Judge and Route) at L1, and sample_count sampled replies per example at L0. Calibration is the expensive step: do it once, commit the profile file, and reuse it until something in the fingerprint changes.

The guarantee holds only for contexts exchangeable with the calibration contexts, scored by the same function. Build the calibration contexts with the same code that builds contexts at run time (in the agent integrations, the GuardRule.context builder), and draw the examples at random from real traffic, not from a curated set of easy or hard cases. See Exchangeability.

Change a level without recalibrating

The profile stores evidence, not a threshold. Thresholds are recomputed from the stored scores each time a query is evaluated, so changing alpha, target, delta, method or a Gate’s guarantee mode needs no new model calls:

stricter = Gate(instructions="Is approving this refund correct under the policy?",
                calibration_profile="refunds-demo-v1", guarantee="risk_high_probability",
                target=0.03, delta=0.10)
answer = client.evaluate(refund_context({"amount_usd": 600, "receipt_on_file": True}),
                         {"refund": stricter}).answers["refund"]
print(answer.decision, answer.raw["threshold"])
print(answer.guarantee.describe())

escalate 0.95
With 90% confidence, the rate of decisions that are auto-approved and wrong is at most 0.03 (RCPS, Hoeffding-Bentkus, n=300).

Changing the instructions, options or levels is different: those define the scoring function, so the profile becomes stale (see below).

Evaluate

result = client.evaluate(refund_context({"amount_usd": 600, "receipt_on_file": True}), {"refund": refund_gate})
answer = result.answers["refund"]
print(answer.decision, answer.raw["confidence"], answer.raw["threshold"])
print(answer.guarantee.describe())
print(result.usage.backend_calls, result.backend, result.warnings)

auto_approve 0.7 0.7
Expected rate of decisions that are auto-approved and wrong is at most 0.05 (conformal risk control, n=300).
1 {'provider': 'local', 'model': 'mock-model', 'access_level': 'L1'} []

evaluate(context, queries, *, backend=None, response_model=None) answers every query in queries (a dict of query id to query) about one context, and returns an EvaluateResponse with the same typed answers the hosted client returns:

result.answers[qid]: a SetAnswer, GateAnswer, BeliefAnswer, IntervalAnswer, ClaimAnswer, JudgeAnswer or RouteAnswer, each with its guarantee card in .guarantee.
result.warnings: why an answer is heuristic, a Mondrian group that fell back to the marginal threshold, or a Gate for which no threshold meets the target.
result.heuristic_answers: the ids of answers without a formal guarantee.
result.usage.backend_calls: model requests made for this call.
result.backend: {"provider": "local", "model": ..., "access_level": ...}.
result.request_id: local- followed by 12 hex characters.

A local GateAnswer is always auto_approve or escalate; local mode never returns abstain. Its raw["confidence"] is the evidence model’s score and raw["threshold"] the calibrated threshold. When no threshold meets the target on the stored data, threshold is None, every request escalates, and result.warnings says so. From 20 calibration examples on, Gate answers also carry a Venn-Abers interval in raw["venn_abers"].

Decide a batch with FDR control

When decisions arrive as a batch (a queue of alerts, a list of candidate matches), gate_batch(contexts, query, *, backend=None) decides all of them together with conformal selection and Benjamini-Hochberg, and returns one GateAnswer per context:

queue = [refund_context(row["context"]) for row in fresh[:8]]
for item, decision in zip(queue, client.gate_batch(queue, refund_gate)):
    print(item["amount_usd"], item["receipt_on_file"], decision.decision, round(decision.raw["p_value"], 4))
print(decision.guarantee.describe())

40.0 True auto_approve 0.0133
40.0 False auto_approve 0.0133
40.0 False auto_approve 0.0133
250.0 False escalate 0.2525
250.0 True auto_approve 0.0365
250.0 True auto_approve 0.0365
15.0 False auto_approve 0.0133
250.0 False escalate 0.2525
Across this batch of 8, the expected fraction of approved items that are wrong is at most 0.05 (conformal selection with Benjamini-Hochberg, n=300).

Each item gets a conformal p-value against the calibration examples that were labelled wrong, and BH at level query.target selects the items to approve. The statement is about the batch: in expectation, at most target of the approved items in it are wrong. gate_batch reads the query’s target and ignores its guarantee mode. It needs a usable profile with at least one example labelled wrong; otherwise every item escalates with a heuristic card.

Check a profile before you rely on it

reworded = Gate(instructions="Should this refund be approved?",
                calibration_profile="refunds-demo-v1", guarantee="risk", target=0.05)
print(client.calibration_status(reworded))
result = client.evaluate(refund_context({"amount_usd": 40, "receipt_on_file": False}), {"refund": reworded})
answer = result.answers["refund"]
print(answer.decision, answer.is_heuristic, result.heuristic_answers)
print(answer.guarantee.statement)

(False, "profile 'refunds-demo-v1' was calibrated with different instructions or options; recalibrate after changing the prompt")
escalate True ['refund']
No guarantee: profile 'refunds-demo-v1' was calibrated with different instructions or options; recalibrate after changing the prompt

calibration_status(query, *, backend=None) returns (True, "") when the query’s profile exists, calibrates a query of the same type, and matches both the query’s scoring fields and the backend’s fingerprint. Otherwise it returns (False, reason). It does not check the sample size, because that depends on alpha or target and is checked per answer. The examples’ _shared/calibration.py uses it to calibrate only when needed.

get_profile(name) returns the CalibrationProfile for a stored profile (and raises ConfigurationError if there is none); list_profiles() returns all of them.

`CalibrationProfile` attribute	Meaning in local mode
`name`, `version`, `method`	Identity. `method` is the procedure answers use, for example `CRC`, `RCPS` or `LTT-selective` for a Gate.
`alpha`	The query’s `alpha`, or its `target` for a Gate.
`n`	Stored labelled examples.
`minimum_n`	The smallest `n` for which this query can return a guaranteed answer (see sizes).
`recommended_n`	The size recommended for a stable guarantee at this level.
`realized_coverage_ci`	The spread of realized coverage implied by `n` (sizing).
`backend_fingerprint`	What the profile was scored with.
`group_by`, `groups`	The Mondrian field, and `GroupStatus(n, status)` per group: `calibrated` or `underpowered`.
`status`, `can_serve_guarantees`	`"serving"` when `n >= minimum_n`, otherwise `"collecting"`.

Methods and guarantees

Each primitive runs one method. The statement on the guarantee card is generated from the templates below, filled in with the query’s levels and the profile size. Read them literally: each bounds a rate over data exchangeable with the calibration set, not the outcome of one decision.

Query	Method (`guarantee.method`)	Statement on the card
`Set`	`APS` by default; `LAC` or `RAPS` with `method=`; per-group (Mondrian) thresholds with `group_by=`	`describe()`: “Contains the correct answer at least 90% of the time on profile ’…’ (n=…).”
`Belief`	`IVAP` (inductive Venn-Abers)	`Venn-Abers pair [{p0}, {p1}]: the probability computed under the true label is calibrated on data exchangeable with the {n} calibration examples; a wide pair means the calibration data cannot pin the probability down.`
`Interval`	`ordinal-aps`: contiguous intervals over ordered levels	`describe()`: “Contains the correct answer at least 90% of the time on profile ’…’ (n=…).”
`Gate`, `guarantee="risk"`	`CRC` (conformal risk control)	`Expected rate of decisions that are auto-approved and wrong is at most {target} (conformal risk control, n={n}).`
`Gate`, `guarantee="risk_high_probability"`	`RCPS` (Hoeffding-Bentkus)	`With {1-delta} confidence, the rate of decisions that are auto-approved and wrong is at most {target} (RCPS, Hoeffding-Bentkus, n={n}).`
`Gate`, `guarantee="fdr"`	`LTT-selective` (Learn-then-Test, fixed-sequence)	`With {1-delta} confidence, at most {target} of auto-approved decisions are wrong (Learn-then-Test, fixed-sequence, n={n}).`
`gate_batch(...)`	`conformal-selection-BH`	`Across this batch of {m}, the expected fraction of approved items that are wrong is at most {target} (conformal selection with Benjamini-Hochberg, n={n}).`
`Claim`	`conformal-factuality`	`With probability at least {1-alpha}, every retained claim is supported (conformal factuality, n={n}).`
`Judge`	`trust-or-escalate` cascade	`With {1-delta} confidence, verdicts that are not escalated agree with human labels at least {1-alpha} of the time (n={n}).`
`Route`	`conformal-cascade` (needs `task=Set(...)` and `guarantee="accuracy"`)	`The served answer is wrong at most {alpha} of the time: each of the {k} tiers answers only when its {1-alpha/k} conformal set is a single option (n={n}).`

Two Gate distinctions matter in practice:

Per request or per approval. risk and risk_high_probability bound the rate of decisions that are auto-approved and wrong, counted over all requests. fdr bounds the error rate among auto-approved decisions. A gate that approves 40% of requests at a per-request risk of 0.05 can have up to 0.05 / 0.4 = 12.5% of its approvals wrong.
Expectation or confidence. risk bounds an expectation over the calibration draw. risk_high_probability and fdr hold for the threshold you actually deployed, except with probability delta over the calibration draw, and need more data.

When alpha is omitted it defaults to 0.10; a Gate in fdr mode without delta uses 0.10; Judge always uses delta = 0.10.

Minimum and recommended sizes

Below its minimum, a query cannot return a guaranteed answer and fails closed. The minimum is a floor, not a target: at the minimum the answer is valid but usually uninformative (a set of every option, a gate that escalates everything), and the realized coverage of one particular calibration draw varies widely.

Query	Minimum `n`	At common levels
`Set`, `Interval`, `Claim`	$\lceil (1-\alpha)/\alpha \rceil$	9 at $\alpha = 0.10$ , 19 at $\alpha = 0.05$
`Belief`	20	20
`Gate`, `risk`	$\lceil 1/\text{target} - 1 \rceil$	19 at target 0.05
`Gate`, `risk_high_probability`	$\lceil \ln(1/\delta)/\text{target} \rceil$	47 at target 0.05, $\delta = 0.10$ ; 300 at target 0.01, $\delta = 0.05$
`Gate`, `fdr`	$\lceil \ln(5/\delta)/\text{target} \rceil$	40 at target 0.10, $\delta = 0.10$
`Judge` ( $s$ model stages)	$\lceil \ln(5s/0.1)/\alpha \rceil$	40 at $\alpha = 0.10$ with one stage
`Route` ( $k$ tiers)	the `Set` minimum at $\alpha/k$	19 at $\alpha = 0.10$ with two tiers
`gate_batch`	at least one example labelled wrong

The recommended size for a stable guarantee is about 300 examples at $\alpha = 0.20$ , 1,000 at 0.10, 1,500 at 0.05 and 2,500 at 0.01 (profile.recommended_n; see Calibration profiles: sizing). The bundled agent examples calibrate on 250 to 400 synthetic examples so they run quickly; their guarantees are valid, but a production profile should be calibrated on at least the recommended number of labelled examples from your own traffic.

Fail-closed behavior

When a profile is missing, too small for the requested level, built for a different query type, or stale (a different prompt or a different backend fingerprint), every answer is labelled heuristic and takes its safe form:

Query	Safe form of a heuristic answer
`Set`	Every option
`Gate`	`escalate`
`Belief`	Venn-Abers interval `[0, 1]`
`Interval`	Every level
`Claim`	Nothing retained
`Judge`, `Route`	Sent to the human queue

The card’s statement gives the reason (No guarantee: ...), the same reason is added to result.warnings, and answer.is_heuristic is True. The tool guard never allows a call on a heuristic answer.

With strict_guarantees=True, the same situations raise instead:

strict = LocalCLIClient(MockEvidenceBackend(scorer=demo_scorer), store=".cli_profiles", strict_guarantees=True)
try:
    strict.evaluate(refund_context({"amount_usd": 40, "receipt_on_file": False}), {"refund": reworded})
except Exception as exc:
    print(type(exc).__name__)

InsufficientCalibrationError

Profile files

A profile is one JSON file, <store>/<profile name>.json, holding everything its guarantee depends on. The file written by the calibration above, with its 300 records shortened to two:

{
  "format_version": 1,
  "name": "refunds-demo-v1",
  "version": 1,
  "query_type": "gate",
  "method": "CRC",
  "group_by": null,
  "created_at": "2026-09-24T13:00:16+00:00",
  "updated_at": "2026-09-24T13:00:16+00:00",
  "query": {
    "type": "gate",
    "instructions": "Is approving this refund correct under the policy?",
    "calibration_profile": "refunds-demo-v1",
    "guarantee": "risk",
    "target": 0.05
  },
  "query_fingerprint": "9f0af94f7a457278",
  "backend_fingerprint": {
    "provider": "mock",
    "model": "mock-model",
    "access_level": "L1",
    "temperature": 1.0,
    "prompt_version": "61fa8201d8ea",
    "seed": 0,
    "sharpness": 2.5,
    "noise": 0.8,
    "keywords": {},
    "custom_scorer": true
  },
  "n": 300,
  "records": [
    {"evidence": {"score": 0.95}, "label": true, "source": "human"},
    {"evidence": {"score": 0.95}, "label": true, "source": "human"}
  ]
}

query is the query as calibrated, and query_fingerprint a hash of its scoring fields.
backend_fingerprint records the evidence model and its settings.
records holds, per example, the evidence (a score, or per-option probabilities), the label, the label source and, for Mondrian profiles, the group. It does not hold the calibration contexts, with two exceptions: Claim records keep each claim’s text, and Mondrian records keep the group value.

Treat the file as the audit record of the guarantee. Commit it next to the code that uses it, review changes to it in pull requests like code (a new version, a changed fingerprint, a different n), and ship it with the service. Profile names may use letters, digits, ., _ and - (up to 128 characters, starting with a letter or digit). Files are written atomically, so a crash never leaves a half-written profile. LocalProfileStore(root) (load, save, exists, delete, names) gives direct access to the directory.

Fingerprints and what invalidates a profile

A conformal guarantee holds for the scoring function the calibration set was scored with. Local mode stores two fingerprints of that function and refuses to serve a guarantee when either no longer matches.

The query fingerprint is a hash of the query fields that define the question: type, instructions, options, levels, criteria, support_source and task. Levels and modes (alpha, target, delta, method, guarantee) are not part of it, which is why they can change without recalibrating.

The backend fingerprint is whatever the evidence backend reports from fingerprint():

Backend	Fingerprinted
Every `cli_sdk.evidence` backend	`provider`, `model`, `access_level`, `temperature`, and `prompt_version` (a hash of the SDK’s scoring prompt)
OpenAI, Azure OpenAI, Gemini, vLLM, SGLang, OpenAI-compatible	plus `base_url`, `top_logprobs`, `reasoning_effort` and `extra_body` (which carries `chat_template_kwargs`, for example `enable_thinking`); Azure adds `azure_endpoint` and `api_version`
`AnthropicEvidenceBackend`	plus `effort`, the sampling scheme and whether thinking is disabled
`LangChainEvidenceBackend`	plus the chat model’s class and sampling settings (`temperature`, `top_p`, `top_k`, `reasoning_effort`, base URL)
`MockEvidenceBackend`	plus `seed`, `sharpness`, `noise`, `keywords` and whether a custom scorer is set
Your own `CustomBackend`	`provider: "custom"`, its class, `name` and `access_level`, unless you implement `fingerprint()` (example)
Any backend at access level L0	plus the client’s `sample_count`

So a profile goes stale, and its answers fail closed, when you change the prompt or options, switch model or provider, point at a different server, change the sampling temperature, the reasoning effort or thinking setting, upgrade to an SDK whose scoring prompt changed, or change sample_count for a sampling-only model.

A fingerprint only sees the query and the backend. It does not see the context, so it cannot detect a change to your context builder or to policy text you carry inside the context: recalibrate whenever you change either. Nor can it detect a provider changing the weights behind an unchanged model name, a self-hosted server restarted with different weights under the same served name, a change inside a mock scorer function, or a shift in the traffic itself. Pin dated model snapshots, set Azure deployments to NoAutoUpgrade, version your served model names, audit on fresh labelled data, and run a drift monitor.

Audit and monitor

audit(query, examples, *, backend=None, confidence=0.95) re-checks the guarantee on fresh labelled examples that were not used to calibrate, and returns an AuditResult:

audit = client.audit(refund_gate, fresh)
print(audit.result, audit.sample, round(audit.realized_coverage, 3), round(audit.ci_lower, 3),
      round(audit.ci_upper, 3), audit.target)

pass 200 0.96 0.923 0.983 0.95

An audit fails only when the whole Clopper-Pearson interval shows real evidence of a violation, not when a finite sample lands a little on the wrong side of the target by chance. For a Gate the result is reported in coverage form: realized_coverage is 1 minus the realized risk (here 4% of the 200 fresh requests were auto-approved and wrong) and target is 1 minus the Gate’s target. In fdr mode the realized risk is the error rate among the approved examples, and the audit raises ValueError if none was approved. Audits are supported for Set, Interval, Gate, Claim, Judge and Route; a Belief audit raises ConfigurationError, because Venn-Abers calibration is not a coverage or risk statement. An audit on a stale or undersized profile raises InsufficientCalibrationError.

monitor(query, false_alarm_rate=0.05) returns a LocalMonitor, an anytime-valid e-process you feed with production outcomes as labels arrive:

monitor = client.monitor(refund_gate, false_alarm_rate=0.05)
alert = None
for row in fresh:
    decision = client.evaluate(row["context"], {"refund": refund_gate}).answers["refund"]
    alert = monitor.update(decision.approved and not row["label"])  # approved and wrong
    if alert:
        break
print("alert:", alert)

alert: None

Query	Monitor	Feed `update()` with
`Gate`, `risk` or `risk_high_probability`	risk, at `target`	`True` when a decision was auto-approved and wrong, for every decision
`Gate`, `fdr`	risk, at `target`	only auto-approved decisions: `True` when the approval was wrong
`Set`, `Interval`, `Claim`	coverage, at `1 - alpha`	`True` when the set or interval contained the label, or every retained claim was supported

update returns an Alert the first time the evidence crosses 1 / false_alarm_rate, and None otherwise. The chance that the monitor ever alarms while the guarantee holds is at most false_alarm_rate, no matter how often you check it. client.monitor tests the nominal level exactly; to follow the advice on Drift monitoring and test at the lower end of the profile’s realized_coverage_ci, construct LocalMonitor(type="coverage", target=..., false_alarm_rate=...) directly. When a monitor alarms, stop auto-allowing (send every guarded action to review), recalibrate on recent labelled data, and start a new monitor for the new profile version.

Moving to the hosted service

The queries, answers and guard rules do not change. Create the profile in the service, add the same labelled examples, and give the guard a CLIClient instead:

from cli_sdk import CLIClient
 
hosted = CLIClient()                     # CLI_API_KEY from the environment
guard = ToolGuard(hosted, rules, backend=evidence_backend)

With a CustomBackend (every cli_sdk.evidence backend is one), the hosted client computes evidence in your process and sends the evidence, with the context and the query definition, to the service; see Backends: bring your own model and Calibration profiles.

Overview Bring your own model