Agent frameworks

An agent’s tool call is a decision: a refund is issued, an account is frozen, a patient receives advice, a claim is paid. CLI puts a calibrated check in front of the tools that carry that kind of risk. Each proposed call is allowed, sent to a human, or blocked by a rule whose error rate has a stated, finite-sample guarantee on data like your calibration set, and anything the calibration cannot vouch for goes to a person.

The integrations work with LangChain, LangGraph, Google ADK and Microsoft Agent Framework, with closed models (OpenAI, Azure OpenAI, Anthropic Claude, Gemini) and open weights (Gemma on vLLM or SGLang), fully in-process with local mode or against the hosted service.

Why agents need calibrated guardrails

• A tool call is not a chat message. Once the tool runs, money has moved or a message has been sent. The question is not whether the model sounds sure, but how often actions like this one are wrong when they run without review.
• Model confidence is not a guarantee. Verbalized confidence (“I am 95% sure”) and raw token probabilities are often miscalibrated, and they shift when the model version, the prompt or the traffic changes. Nothing about them bounds an error rate.
• Hand-picked thresholds have unknown error rates. “Auto-approve above 0.9” is a number nobody can translate into “at most X% of auto-approved refunds are wrong”, and nobody can tell when it stops holding.
• CLI calibrates the decision itself. You label a few hundred to a few thousand real cases. CLI turns the evidence model’s scores on them into a decision rule with a guarantee, for example “the expected rate of decisions that are auto-approved and wrong is at most 0.05”, attaches that statement to every decision, and fails closed: when calibration is missing, too small, or was built for a different model or prompt, the guard never lets the call run on its own.

How it works

Guard rules

from cli_sdk.integrations import GuardRule, ToolGuard
 
guard = ToolGuard(client, [GuardRule(tool="issue_refund", query=refund_gate, context=refund_context)])
decision = guard.check("issue_refund", {"order_id": "A-1002", "amount_usd": 600.0, "receipt_on_file": False})

ToolGuard(client, rules, *, backend=None, on_error="escalate", cache_size=1024) takes a LocalCLIClient or a hosted CLIClient, one rule per guarded tool, an optional backend passed through to client.evaluate, and what to do when evaluation fails ("escalate", or "raise"). check(tool, arguments, state=None) returns a GuardDecision; acheck(...) is the same without blocking the event loop.

How a decision is made

The mapping from answer to action, exactly as cli_sdk.integrations implements it:

Fail-closed rules

• A heuristic answer (no profile, too few examples for the requested level, a different prompt or a different evidence model) escalates, or blocks with on_heuristic="block". It never allows.
• An exception while building the context or evaluating escalates, or raises with on_error="raise". It never allows. Failures are not cached, so the next attempt evaluates again.
• A tool without a rule is allowed: guard the actions that carry risk.
• When one model turn proposes several calls, the LangGraph adapter applies the most severe decision to the whole turn (block over escalate over allow).

Caching and the audit trail

Frameworks re-run nodes and predicates when a paused run resumes. The guard caches each decision per (tool, arguments, context), up to cache_size entries, so a resumed run gets the decision the reviewer saw without a second model call; the LangChain adapter also stores decisions in the checkpointed agent state, so a resume on another worker honours exactly what the reviewer saw. Build a new ToolGuard after recalibrating, because cached decisions (including heuristic ones) are kept for the guard’s lifetime. Every decision is appended to guard.log, and each adapter records it where the framework keeps state: the agent state’s cli_guard key in LangChain and LangGraph (checkpointed), session.state["cli_guard_audit"] in Google ADK, and approval tickets in session.state["cli_guard"] in Microsoft Agent Framework.

Framework integrations

Install the SDK with the extra for your framework (Python 3.10 or later for the frameworks), plus the provider package for your agent model:

pip install "cci-sdk[langchain,openai]" langchain-openai     # LangChain
pip install "cci-sdk[langgraph,openai]" langchain-openai     # LangGraph
pip install "cci-sdk[google-adk,openai]" "google-adk[extensions]"   # Google ADK (LiteLLM for non-Gemini models)
pip install "cci-sdk[agent-framework,openai]" agent-framework-openai  # Microsoft Agent Framework

The openai extra installs the OpenAI SDK used by the OpenAI, Azure, Gemini, vLLM and SGLang evidence backends; use anthropic for Claude evidence, or all for every extra.

What you get

• A guarantee card on every decision. Each allow, escalation and block carries the statement, method, profile and calibration size behind it, in the reviewer’s view and in the audit record.
• Calibrated human-review routing. People see only the calls the calibration cannot settle, and the escalation rate is itself a measurable property of the profile rather than a side effect of a guessed threshold.
• Any model. Access level L0 (sampling) covers models without log-probabilities, such as Claude and Gemini; L1 (first-token log-probabilities) covers OpenAI non-reasoning models and open weights on vLLM or SGLang. The agent model and the evidence model can differ. The hosted service adds L2 to L4 evidence. See Bring your own model.
• Fingerprinted calibration profiles. Each profile records the question and the exact evidence model and settings it was calibrated with; a change makes it stale and every guarded call escalates until you recalibrate. Local profiles are JSON files you commit and review.
• Audits and drift monitors. client.audit(query, fresh_examples) re-checks a guarantee on held-out labels, and client.monitor(query) is an anytime-valid e-process you can check continuously.
• Local or hosted. Local mode runs in your process with nothing but a model; the hosted CLIClient adds shared profiles, managed monitors and label-efficient calibration. The guard code is the same.
• Native human-in-the-loop. Escalations use each framework’s own interrupt, confirmation or approval mechanism, so they work with its checkpointers, sessions and UIs.

A minimal guarded agent

A LangChain agent with one guarded tool. It runs offline as written, in about a second: a scripted fake chat model proposes the tool calls, and MockEvidenceBackend with a small scoring function stands in for the evidence model. The calibration data is synthetic.

import random
 
from langchain.agents import create_agent
from langchain_core.language_models.fake_chat_models import FakeMessagesListChatModel
from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.tools import tool
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command
 
from cli_sdk import Gate
from cli_sdk.evidence import MockEvidenceBackend
from cli_sdk.integrations import GuardRule, ToolGuard
from cli_sdk.integrations.langchain import cli_middleware, pending_reviews
from cli_sdk.local import LocalCLIClient
 
POLICY = "Refunds up to 100 USD are within policy. Larger refunds need a receipt on file."
 
 
def refund_context(args):
    """Builds the guard's context. Calibration examples use the same function."""
    return {"policy": POLICY, "amount_usd": float(args["amount_usd"]),
            "receipt_on_file": bool(args["receipt_on_file"])}
 
 
def demo_scorer(context, instructions, options):
    """Keyless stand-in for an evidence model: reads the request, never the label."""
    if context["amount_usd"] <= 100:
        p = 0.95
    else:
        p = 0.70 if context["receipt_on_file"] else 0.10
    return {"true": p, "false": 1.0 - p}
 
 
def synthetic_history(n=300, seed=7):
    """Synthetic reviewed refunds: label is True when approving was correct."""
    rng = random.Random(seed)
    rows = []
    for _ in range(n):
        args = {"amount_usd": rng.choice([15, 40, 80, 250, 600, 1400]),
                "receipt_on_file": rng.random() < 0.5}
        correct = args["amount_usd"] <= 100 or args["receipt_on_file"]
        if rng.random() < 0.04:  # reviewers disagree on a few cases
            correct = not correct
        rows.append({"context": refund_context(args), "label": correct})
    return rows
 
 
refund_gate = Gate(instructions="Is approving this refund correct under the policy?",
                   calibration_profile="refunds-demo-v1", guarantee="risk", target=0.05)
 
client = LocalCLIClient(MockEvidenceBackend(scorer=demo_scorer), store=".cli_profiles")
ready, reason = client.calibration_status(refund_gate)
if not ready:  # calibrate once; the profile is cached as .cli_profiles/refunds-demo-v1.json
    client.calibrate(refund_gate, synthetic_history())
guard = ToolGuard(client, [GuardRule(tool="issue_refund", query=refund_gate, context=refund_context)])
 
executed = []
 
 
@tool
def issue_refund(order_id: str, amount_usd: float, receipt_on_file: bool) -> str:
    """Refund an order to the original payment method."""
    executed.append(order_id)
    return f"Refunded {amount_usd:.2f} USD on {order_id}."
 
 
class ScriptedModel(FakeMessagesListChatModel):
    """Keyless stand-in for the agent model: proposes one tool call, then answers."""
 
    def bind_tools(self, tools, **kwargs):
        return self
 
 
def run(order_id, amount_usd, receipt_on_file, reviewer_decision):
    call = {"name": "issue_refund", "id": "call_1", "type": "tool_call",
            "args": {"order_id": order_id, "amount_usd": amount_usd, "receipt_on_file": receipt_on_file}}
    model = ScriptedModel(responses=[AIMessage("", tool_calls=[call]), AIMessage("Done.")])
    agent = create_agent(model, tools=[issue_refund], middleware=cli_middleware(guard),
                         checkpointer=InMemorySaver())
    config = {"configurable": {"thread_id": order_id}}
    result = agent.invoke({"messages": [HumanMessage(f"Refund order {order_id}")]}, config)
    for review in pending_reviews(result):
        print(review["description"])
        result = agent.invoke(Command(resume={"decisions": [reviewer_decision]}), config)
    tool_result = [m for m in result["messages"] if m.type == "tool"][-1]
    print(f"{order_id}: tool executed: {order_id in executed}; tool result: {tool_result.content}\n")
 
 
run("A-1001", 40.0, False, None)
run("A-1002", 600.0, False, {"type": "reject", "message": "No receipt on file."})

Output:

A-1001: tool executed: True; tool result: Refunded 40.00 USD on A-1001.

Calibrated check: escalate
Why: below the calibrated auto-approval threshold (confidence 0.100, calibrated threshold 0.700).
Guarantee: Expected rate of decisions that are auto-approved and wrong is at most 0.05 (conformal risk control, n=300).
Tool: issue_refund
Arguments: {"amount_usd": 600.0, "order_id": "A-1002", "receipt_on_file": false}
A-1002: tool executed: False; tool result: User rejected the tool call for `issue_refund` with reason: No receipt on file.

The 40 USD refund scores 0.95, above the calibrated threshold of 0.70, and runs. The 600 USD refund without a receipt scores 0.10: the agent run pauses on a LangChain interrupt, the reviewer sees the calibrated reason and the guarantee, rejects, and the tool never runs. The threshold 0.70 was not chosen by hand: it is the least conservative threshold that satisfies the conformal risk control bound on these 300 examples, which is what makes the expected rate of auto-approved-and-wrong decisions at most 0.05 on requests like them.

To use real models, replace the evidence backend and the chat model; the guard, the rule and the tool stay the same (pip install "cci-sdk[langchain,openai]" langchain-openai):

import os
 
from langchain_openai import ChatOpenAI
 
from cli_sdk.evidence import OpenAIEvidenceBackend
 
evidence = OpenAIEvidenceBackend("gpt-4.1-mini-2025-04-14",
                                 api_key=os.environ.get("OPENAI_API_KEY", "YOUR_OPENAI_API_KEY"))
client = LocalCLIClient(evidence, store=".cli_profiles")
guard = ToolGuard(client, [GuardRule(tool="issue_refund", query=refund_gate, context=refund_context)])
model = ChatOpenAI(model="gpt-4.1-mini-2025-04-14",
                   api_key=os.environ.get("OPENAI_API_KEY", "YOUR_OPENAI_API_KEY"), temperature=0,
                   use_responses_api=False)
agent = create_agent(model, tools=[issue_refund], middleware=cli_middleware(guard), checkpointer=InMemorySaver())

The profile calibrated with the mock does not match the OpenAI backend’s fingerprint, so calibration_status reports it as stale and the first run recalibrates: 300 requests to the evidence model, once. Calibrate on your own labelled cases, not on the synthetic history above.

Examples

Eight complete examples in examples/agents, two per framework, each in a risk-sensitive domain with a synthetic dataset and a written policy. Each runs offline in seconds by default, and with any provider by flag. See Risk-sensitive domains for what each one guards and why it uses the primitive it does.

From the cci-sdk repository:

python examples/agents/langchain/finance_refund_agent.py                    # mock: offline, keyless (default)
python examples/agents/langchain/finance_refund_agent.py --provider openai  # or azure, anthropic, gemini, vllm, sglang
python examples/agents/langchain/finance_refund_agent.py --provider anthropic --evidence-provider vllm
python examples/agents/langchain/finance_refund_agent.py --interactive      # you are the reviewer
python examples/agents/langchain/finance_refund_agent.py --recalibrate --store /tmp/cli_profiles

Every example takes --provider, --evidence-provider, --store, --recalibrate and --interactive; setup, per-provider commands and costs are in the examples README.

Next

• Local mode: LocalCLIClient in depth, profile files, fingerprints and fail-closed behavior.
• Bring your own model: agent and evidence models, access levels, and every provider including Gemma on vLLM and SGLang.
• Risk-sensitive domains: which primitive fits which decision, and the governance around it.
• Framework guides: LangChain, LangGraph, Google ADK, Microsoft Agent Framework.