High-stakes approvals

Problem

A fintech assistant handles account requests. Read-only balance checks can be completed liberally without confirmation; wire transfers cannot. The team wants a different auto-approval bar for each, and a compliance reviewer wants to see exactly what each bar guarantees.

A spreadsheet of hand-picked thresholds (0.6 for balance checks, 0.9 for transfers) answers neither question: nobody can say what error rate 0.9 corresponds to, or how confident anyone is in it.

Primitives

Set to classify the request’s intent.
Gate with guarantee="risk", target=0.15 for low-stakes actions: a bound on the expected rate of wrong auto-approvals.
Gate with guarantee="risk_high_probability", target=0.01, delta=0.05 for transfers: the deployed threshold’s rate of wrong auto-approvals is at most 1%, with 95% confidence over the calibration draw.

Code

Create one profile per decision

from cli_sdk import AnthropicBackend, CalibrationExample, CLIClient
 
BACKEND = AnthropicBackend(model="claude-sonnet-5", sample_count=20)
 
with CLIClient() as client:
    profiles = client.calibration_profiles
    profiles.create(name="assistant-intents-v2", backend=BACKEND, method="APS", alpha=0.05)
    profiles.create(name="low-stakes-actions", backend=BACKEND, method="CRC", alpha=0.15)
    profiles.create(name="high-stakes-transfers", backend=BACKEND, method="RCPS", alpha=0.01)
 
    # Gate examples: the request as it reached the gate, and whether
    # auto-approving it would have been correct.
    profiles.add_examples(
        "high-stakes-transfers",
        examples=[
            CalibrationExample(
                context={"request": row["request"], "account_id": row["account_id"]},
                label=row["approval_was_correct"],
            )
            for row in reviewed_transfers          # about 2,400 reviewed transfer requests
        ],
    )
    print(profiles.get("high-stakes-transfers").n)   # 2400

Classify, then gate

from cli_sdk import CLIClient, Gate, Set
 
INTENTS = {
    "check_balance": "Read an account balance or recent transactions",
    "approve_transfer": "Move money out of the account",
    "other": "Anything else",
}
 
GATES = {
    "check_balance": Gate(
        instructions="Can this read-only request be completed without confirmation?",
        calibration_profile="low-stakes-actions",
        guarantee="risk",
        target=0.15,
    ),
    "approve_transfer": Gate(
        instructions="Should this transfer be auto-approved?",
        calibration_profile="high-stakes-transfers",
        guarantee="risk_high_probability",
        target=0.01,
        delta=0.05,
    ),
}
 
def handle(client: CLIClient, request: str, account_id: str) -> str:
    context = {"request": request, "account_id": account_id}
 
    intent = client.evaluate(
        context=context,
        backend=BACKEND,
        queries={"intent": Set(instructions="What is the user asking for?",
                               options=INTENTS,
                               calibration_profile="assistant-intents-v2")},
    ).answers["intent"]
 
    if intent.is_heuristic or not intent.is_singleton or intent.top not in GATES:
        return "human_review"
 
    gate = client.evaluate(
        context=context,
        backend=BACKEND,
        queries={"gate": GATES[intent.top]},
    ).answers["gate"]
 
    if gate.is_heuristic:
        return "human_review"
    if gate.approved:
        return f"execute:{intent.top}"
    if gate.decision == "abstain":
        return "ask_user_to_confirm"
    return "human_review"
 
with CLIClient() as client:
    print(handle(client, "What's my checking balance?", "acct_1042"))           # execute:check_balance
    print(handle(client, "Wire 4,800 EUR to the new supplier.", "acct_1042"))  # human_review

Walkthrough

The intent step is a Set: a singleton set is required before any gate runs, so an ambiguous request (“move my balance to savings and tell me what’s left”) goes to a human instead of being forced into one intent.

Each intent then has its own gate and its own profile, with its own labelled examples, method, and target. The two-step call means only one gate is scored per request, which matters on an L0 backend like Claude where every query costs sample_count backend calls.

The transfer gate is the expensive one to calibrate. A high-probability bound at target=0.01, delta=0.05 cannot certify any threshold with fewer than about 400 calibration examples, even if none of them is an error; in practice, 2,400 reviewed transfers leave room for the few errors a real model makes. Until the profile has enough data, its answers are heuristic and handle sends every transfer to a human.

Calibrate each gate on requests that reached it through the same intent step — requests the Set classified as a singleton transfer, not a hand-picked set of transfers. The gate’s guarantee holds for the population its calibration examples were drawn from; if the intent step changes, recalibrate the gates too.

What the guarantee card means

A compliance reviewer reading the transfer gate’s card sees:

{
  "type": "risk_high_probability",
  "target": 0.01,
  "delta": 0.05,
  "method": "RCPS",
  "calibration_profile": "high-stakes-transfers",
  "calibration_n": 2400,
  "last_audited": "2026-09-18T00:00:00Z"
}

What is bounded. The per-request rate at which a transfer that reaches the gate is auto-approved and should not have been. describe() renders it as “With 95% confidence, risk at most 0.01 on profile ‘high-stakes-transfers’ (n=2400).”
What the 95% is. The probability, over the draw of the 2,400 calibration transfers, that the procedure picked a threshold whose true risk exceeds 1%. It is not a per-transaction confidence.
Per request, not per approval. If the gate approves 30% of transfers, a 1% per-request risk allows up to about 3.3% of approved transfers to be wrong. If policy is written per approved transfer, use guarantee="fdr" instead.
The low-stakes card (type: "risk", target: 0.15, method CRC) bounds only the expected risk, averaged over calibration draws — a weaker statement, appropriate for an action that is cheap to undo.

RAG claim filtering LLM-judge cascade