Risk-sensitive domains

This page is a playbook for putting a calibrated guard in front of an agent that acts in healthcare, finance and lending, insurance, or financial-crime compliance: which primitive and guarantee fit which kind of decision, what each guarantee says in plain words, what gets escalated, and the governance a deployment needs around it. The eight examples are the worked cases.

Choosing a primitive and guarantee

Three questions settle most choices:

1. Is the policy written per request or per action taken? The Gate’s risk modes count wrong automatic approvals over all requests; fdr counts them among the automatic approvals. A gate that auto-approves 40% of requests at a per-request risk of 0.05 can have up to 0.05 / 0.4 = 12.5% of its approvals wrong. If the rule is “at most 10% of the holds we place automatically may be unwarranted”, that is fdr.
2. Must the statement hold for the deployed threshold? risk bounds an expectation over calibration draws. risk_high_probability and fdr hold for the threshold you actually deployed, except with probability delta, which is the form audit and model-risk reviews usually ask for. They need more data: 47 examples at the minimum for target 0.05 and delta 0.10, against 19 for risk (sizes).
3. What should happen when the model is unsure? Set, Interval and Belief answers have a width. The guard lets a call run only when the answer is narrow enough to decide it (one allowed label, an interval inside the allowed levels, a probability interval entirely above the bar) and sends everything else to a person.

A useful property of Set and Interval rules: the guard allows a call only when the whole set or interval supports it, so an automatic action is wrong only when the set or interval missed the correct answer. The coverage guarantee therefore also bounds the rate of requests that are acted on automatically and wrongly by alpha, counted over all requests, like a Gate’s risk mode. For a Set rule this needs match_argument whenever the tool argument carries the label: without it, a confident set for one allowed label would let a call with a different label run.

Gate, Belief, Set and Interval guard tool calls. Claim checks generated text, so it runs in its own step (a graph node in the discharge-summary example), not as a GuardRule.

Healthcare

Four examples: portal triage, discharge summaries, trial pre-screening and pharmacovigilance. In each, the model reads a record and proposes an action that reaches a patient, a study or a regulator, and a clinician, coordinator or safety physician decides everything the guard escalates.

Patient-portal triage

langchain/healthcare_patient_triage.py (LangChain). The agent reads a portal message and proposes send_triage_advice(message_id, level) with one of self_care, routine_appointment, urgent_care or emergency.

• Guarantee. Set(method="APS", alpha=0.05): the set contains the level the written protocol assigns for at least 95% of messages like the calibration set.
• Rule. GuardRule(allow_labels=["self_care", "routine_appointment"], match_argument="level"): a reply is sent automatically only when the set is exactly the level the agent proposed, and that level is one of the two low-acuity ones. So at most 5% of messages like the calibration set get an automatic reply with the wrong level, counted over all messages.
• Escalated to a nurse. Every set with two or more levels, every confident set for a different level than the agent proposed, and every urgent_care or emergency proposal. From the mock run:

    review request for M-2004:
      action:    send_triage_advice({"level": "self_care", "message_id": "M-2004"})
      why:       the calibrated prediction set is {routine_appointment}, but the agent proposed 'self_care'.
      guarantee: Contains the correct answer at least 95% of the time on profile 'portal-triage-v1' (n=320).

• Emergency pathway. A review queue must never delay an emergency. A production portal should show fixed emergency instructions to the patient immediately, whatever the agent or the guard does, and route the message to a nurse at the same time.

Discharge summaries

langgraph/clinical_summary_graph.py (LangGraph). A model drafts a patient-friendly discharge summary from the chart; a verify_claims node splits it into atomic claims and keeps only those the chart supports.

• Guarantee. Claim(alpha=0.10): “With probability at least 90%, every retained claim is supported”. In at most about 10% of drafts like the calibration set does an unsupported claim survive the filter.
• Escalated to a clinician. Any draft with a dropped claim, or with no calibrated guarantee available, pauses the graph for a clinician, who approves publishing the verified claims only, or rejects. The portal only ever receives retained claims. In the mock run, a draft with an invented dose increase and a wrong appointment date loses exactly those two claims before the clinician sees it.
• What it does not cover. The guarantee is about what is kept, not about what is missing. A filtered summary can omit something the patient needs, which is one reason every draft with a dropped claim goes to a clinician, and why portal text still falls under your clinical governance.

Clinical-trial pre-screening

google_adk/clinical_trial_screening_agent.py (Google ADK). The agent reads a pre-screening record and proposes mark_eligible(patient_id, trial_id) for a fictional protocol.

• Guarantee. Belief returns a Venn-Abers interval $[p_0, p_1]$ for “meets every inclusion criterion and no exclusion criterion”, calibrated against eligibility confirmed at the screening visit.
• Rule. allow_above=0.85, block_below=0.15: mark automatically when the whole interval is at or above 0.85; do not mark automatically when it is entirely below 0.15; otherwise a research coordinator decides. A blocked mark is not a finding of ineligibility: the record stays in the coordinator’s standard queue, and the study team confirms eligibility at the visit in every case.
• What it does not say. A Belief is a calibrated probability, not an error-rate bound: it does not cap the rate of wrong automatic marks. It acts alone only where the calibration data are unambiguous. Where a stated bound on wrong automatic decisions is required, use a Gate.

Pharmacovigilance

agent_framework/pharmacovigilance_agent.py (Microsoft Agent Framework). The agent reads an adverse-event case for a fictitious product and proposes submit_expedited_report(case_id, seriousness_grade).

• Guarantee. Interval over five ordered grades, ordinal APS, alpha=0.10: the contiguous interval contains the grade an assessor records for at least 90% of cases like the calibration set.
• Rule. Submit automatically when the whole interval lies in the grades that qualify for expedited reporting under the synthetic guide (grade_3 to grade_5); block when it lies entirely in grade_1 and grade_2; otherwise a safety physician decides. Both automatic outcomes are wrong only when the interval missed, so at most 10% of cases like the calibration set are handled automatically in the wrong direction.
• Choosing alpha by direction. Over-reporting is the conservative direction; a missed expedited report is not. The example’s alpha=0.10 is a demonstration setting: a real deployment would choose a much smaller alpha for the blocking direction, or route blocked cases to routine physician review as well.
• Scope. The guard decides whether the report is expedited. It does not check the grade written into the report, so the calibrated interval should travel with the case to the physician’s routine review.

Finance and lending

Refunds

langchain/finance_refund_agent.py (LangChain). A refunds assistant looks up the order (unguarded) and proposes issue_refund(order_id, amount, reason).

• Guarantee. Gate(guarantee="risk", target=0.05), conformal risk control: “Expected rate of decisions that are auto-approved and wrong is at most 0.05”, over all refund requests like the calibration set.
• Why CRC. Refunds are high-volume and, within limits, reversible, and an expectation bound is cheap to calibrate. For a payment that cannot be clawed back, use RCPS as in the insurance example.
• Escalated. Every refund below the calibrated threshold pauses for a refunds specialist, who approves, edits (for example, removes a non-refundable shipping fee) or rejects. With --no-human-review the same calls are refused with an error ToolMessage and queued, for batch jobs.
• Context integrity. The guard scores the order record from the order system and the written policy, never the agent’s summary of them, so a model that misreads the record cannot talk the guard into an approval.

Credit risk tiers

agent_framework/credit_underwriting_agent.py (Microsoft Agent Framework). The agent reads a consumer-loan application file and proposes assign_risk_tier(application_id, tier), from A (lowest risk) to E.

• Guarantee. Set(method="APS", alpha=0.10, group_by="channel"): Mondrian calibration, one threshold per application channel. For each channel with enough calibration data, the set contains the tier the policy assigns for at least 90% of that channel’s applications.
• Rule. allow_labels=["A", "B", "C"], match_argument="tier": a tier is recorded automatically only when the set is exactly the proposed tier and it is A, B or C. Every D or E proposal, and every wider set, goes to an underwriter through the framework’s native tool approval; approvals are bound to the exact call, so an altered approval never runs.
• Group-conditional calibration. A channel with fewer examples than the minimum falls back to the marginal threshold, with a warning; its answers then carry the marginal guarantee, not a per-channel one. Each decision record names its group and the group’s calibration size. In the example the broker channel has 40 files, so its realized coverage is visibly less certain than the others.
• Fair lending. The features exclude protected characteristics and obvious proxies. Per-channel coverage is not a fair-lending analysis: review the policy, and the guard’s escalation rates by segment, with your fair-lending and model-risk teams.

Insurance

Claim payments

langgraph/insurance_claims_graph.py (LangGraph). A claims agent reads the claim file and proposes approve_claim_payment(claim_id, amount); add_cli_guard puts a guard node between the agent and its tools.

• Guarantee. Gate(guarantee="risk_high_probability", target=0.05, delta=0.10), RCPS: “With 90% confidence, the rate of decisions that are auto-approved and wrong is at most 0.05”. The 90% is over the draw of the calibration set; it is not a per-payment confidence.
• Why RCPS. A payment is money out the door, and a claims or audit review will ask about the threshold that is deployed, not about an average over hypothetical calibration sets.
• Escalated to an adjuster. Every payment below the threshold pauses the graph. In the mock run the adjuster corrects a jewelry payment to the sub-limit and rejects a payment with fraud indicators and a loss nine days after inception, which the guideline refers to special investigations. A reviewer’s edit is final: the edited payment is not re-scored.
• Audit trail. The checkpoint keeps, per turn, the guard’s decision, its evidence and guarantee, and the adjuster’s outcome and note, in state["cli_guard"].

Financial-crime compliance

AML account holds

google_adk/aml_account_hold_agent.py (Google ADK). An investigator agent reads an alert and proposes place_account_hold(account_id, alert_id, reason).

• Guarantee per decision. Gate(guarantee="fdr", target=0.10, delta=0.10), Learn-then-Test: “With 90% confidence, at most 0.1 of auto-approved decisions are wrong”. At most 10% of the holds placed automatically on alerts like the calibration set are unwarranted.
• Guarantee per batch. With --batch, a nightly queue is decided at once with gate_batch: “Across this batch of 40, the expected fraction of approved items that are wrong is at most 0.1”. In the mock run, 16 of 40 alerts were held automatically and 24 went to analysts; the synthetic labels show 1 of the 16 automatic holds was unwarranted. The bound is on the expected fraction, so one batch can land above or below it.
• Why FDR. An unwarranted hold freezes an innocent customer’s money, and the policy question is per action taken: of the holds we place without an analyst, how many are wrong?
• What it does not cover. The guard sees only the holds the agent proposes. An alert the agent closes without proposing a hold never reaches it, so measure the agent’s missed-hold rate separately. Account actions and regulatory filings remain the responsibility of qualified compliance staff.

Governance

A calibrated guard is one control in a governed process, not a replacement for it.

• Keep a qualified human in the loop. Everything the guard escalates goes to a person with the authority to decide it, who sees the proposed action, the reason and the guarantee statement. Record their decision and note: the examples do so in the framework’s own state.
• Keep the audit trail. The calibration profile JSON is the record of what the guarantee rests on (the question, the evidence model, every score and label): commit it and review changes. Each decision is recorded in guard.log and in the framework’s state: cli_guard in the LangChain and LangGraph checkpoints, session.state["cli_guard_audit"] in Google ADK, and approval tickets in session.state["cli_guard"] in Microsoft Agent Framework.
• Audit on held-out data. Before go-live and on a schedule, run client.audit(query, fresh_examples) on labelled cases the profile has not seen. An audit fails only on real evidence of a violation.
• Monitor for drift. Feed labelled production outcomes to client.monitor(query) (an anytime-valid e-process, safe to check continuously). When it alarms, stop auto-allowing, send every guarded action to review, and recalibrate on recent data.
• Calibrate per group where the guarantee must hold per group. Use group_by on a Set for per-site, per-channel or per-language coverage. Each group needs its own minimum number of examples; smaller groups fall back to the marginal guarantee, with a warning.
• Size the calibration set. The minimum makes a guarantee possible; the recommended size (about 1,000 labelled cases at alpha=0.10, 1,500 at 0.05) makes it stable. The bundled sets (240 to 400) are demo-sized.
• Label by policy. Labels should come from qualified reviewers applying a written policy, sampled at random from real traffic. Noisy labels do not invalidate the procedure, but the guarantee is then about agreement with those labels.
• Protect exchangeability. The guarantee holds for traffic like the calibration set, scored by the same function. It breaks when the query’s instructions or options change, the evidence model or its settings change, the policy text or the context builder changes, the agent model changes which calls reach the guard, or the traffic shifts (a new product, channel, population, season or fraud pattern). Profile fingerprints detect the first two and fail closed. They do not see the context: in the examples the policy text travels inside the context, so recalibrate whenever you edit the policy or the context builder. Audits and monitors are how you catch the rest.

What a guarantee does not say

• It does not say any single decision is correct. “At most 5% of approvals like these are wrong” says nothing about which 5%.
• It does not hold for data unlike the calibration set, and it is not conditional on a case’s features, except per declared group with Mondrian calibration.
• “With 90% confidence” is over the draw of the calibration set, not a per-decision confidence.
• A risk bound counts wrong automatic approvals over all requests, not among approvals; only fdr bounds the error rate among automatic actions.
• It covers only the guarded action. It says nothing about what the agent did not propose, the tools it calls without a rule, or, for Claim, what a filtered text leaves out.
• A Belief interval is a calibrated probability, not an error-rate bound.
• A heuristic answer carries no guarantee at all, which is why the guard never lets one run a tool.
• The examples’ mock evidence models are stand-ins. Their scores, and the thresholds calibrated on them, mean nothing outside the demonstration.