Bring your own model

A guarded agent involves two models, and you choose both. This page covers what each one does, which access level each provider reaches, how to construct an evidence backend for OpenAI, Azure OpenAI, Anthropic Claude, Gemini and open-weight Gemma on vLLM or SGLang, and how to plug in anything else. Every snippet here was run with placeholder keys (constructing a backend makes no network call), and all keys come from environment variables.

Two models, two jobs

They can be the same model or different ones. Common reasons to split them:

• Claude as the agent, a log-probability model as the evidence model. Claude exposes no log-probabilities, so as an evidence model every score costs sample_count requests. Scoring with Gemma on vLLM, or with an OpenAI non-reasoning model, costs one request per score.
• A large agent model, a small evidence model. The evidence model answers one closed question per guarded call; a 12B open-weight model is usually enough, and calibration makes whatever it is honest.
• Data residency. The evidence model sees the guard’s context. Running it on your own vLLM or SGLang server keeps that context in your network, whatever the agent model is.

In the examples, --provider picks the agent model and --evidence-provider the evidence model (default: the same provider):

python examples/agents/langchain/finance_refund_agent.py --provider anthropic --evidence-provider vllm

Access levels in local mode

A higher access level never changes what a guarantee means, only how sharp the answers are and what they cost (see the access ladder). At L0 a score can take at most sample_count + 1 distinct values per option, so thresholds are coarser and more decisions escalate than with the same model at L1. A CustomBackend that declares L2 to L4 is scored through score_options in local mode; the evidence those levels add (prompt scoring, exact label-token probabilities, hidden states) is used by the hosted service.

OpenAI

import os
 
from cli_sdk.evidence import OpenAIEvidenceBackend
 
evidence = OpenAIEvidenceBackend(
    "gpt-4.1-mini-2025-04-14",          # a dated snapshot, so the scoring function cannot change underneath you
    api_key=os.environ.get("OPENAI_API_KEY", "YOUR_OPENAI_API_KEY"),
)
 
# A reasoning model that accepts reasoning_effort="none" also returns log-probabilities (L1).
evidence = OpenAIEvidenceBackend("gpt-6-sol", api_key=os.environ.get("OPENAI_API_KEY", "YOUR_OPENAI_API_KEY"),
                                 reasoning_effort="none")
 
# Any other reasoning configuration returns no log-probabilities: sample instead (L0).
evidence = OpenAIEvidenceBackend("gpt-6-astra", api_key=os.environ.get("OPENAI_API_KEY", "YOUR_OPENAI_API_KEY"),
                                 use_logprobs=False, reasoning_effort="low", max_tokens=4096)

• L1 needs a non-reasoning configuration: the gpt-4.1 family, or a model that accepts reasoning_effort="none" (for example gpt-6-sol, gpt-6-luna, and gpt-5.1 and later). Other reasoning configurations (gpt-6-astra at any effort, gpt-5, the o-series) return no log-probabilities; construct the backend with use_logprobs=False. If you leave L1 on with such a model, scoring fails with an EvidenceError that says so, and the guard escalates.
• Reasoning tokens count against the completion limit. When sampling a reasoning model, raise max_tokens (256 by default) so the answer is not cut off; a truncated reply counts as an invalid sample.
• Pin a dated snapshot such as gpt-4.1-mini-2025-04-14. An alias can move to new weights without changing its name, which the fingerprint cannot detect.
• The backend sends max_completion_tokens by default. Other constructor options: base_url, top_logprobs (at most 20), supports_n (set it to False for a model that rejects n greater than 1), seed, extra_body, temperature (1.0; used for sampling), timeout, and client for a preconfigured openai.OpenAI instance.

Azure OpenAI

import os
 
from cli_sdk.evidence import AzureOpenAIEvidenceBackend, OpenAIEvidenceBackend
 
evidence = AzureOpenAIEvidenceBackend(
    os.environ.get("AZURE_OPENAI_DEPLOYMENT", "YOUR_DEPLOYMENT_NAME"),     # the deployment name, not the model family
    azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT", "https://YOUR-RESOURCE.openai.azure.com"),
    api_key=os.environ.get("AZURE_OPENAI_API_KEY", "YOUR_AZURE_OPENAI_API_KEY"),
    api_version=os.environ.get("AZURE_OPENAI_API_VERSION", "2024-10-21"),
)
 
# The Azure OpenAI v1 API: the same OpenAI backend with a base_url and no api_version.
evidence = OpenAIEvidenceBackend(
    os.environ.get("AZURE_OPENAI_DEPLOYMENT", "YOUR_DEPLOYMENT_NAME"),
    base_url="https://YOUR-RESOURCE.openai.azure.com/openai/v1/",
    api_key=os.environ.get("AZURE_OPENAI_API_KEY", "YOUR_AZURE_OPENAI_API_KEY"),
)

• The first argument is your deployment name. The same access-level rules as OpenAI apply to the model behind it.
• Set the deployment’s version-upgrade policy to NoAutoUpgrade. Otherwise Azure can move the deployment to a new model version under the same name, and the profile would describe a model that is no longer serving.
• The fingerprint records azure_endpoint and api_version. For Microsoft Entra ID instead of a key, pass your own client=openai.AzureOpenAI(azure_endpoint=..., azure_ad_token_provider=..., api_version=...).

Anthropic Claude

import os
 
from cli_sdk.evidence import AnthropicEvidenceBackend
from cli_sdk.local import LocalCLIClient
 
evidence = AnthropicEvidenceBackend(
    "claude-sonnet-5",
    api_key=os.environ.get("ANTHROPIC_API_KEY", "YOUR_ANTHROPIC_API_KEY"),
)
client = LocalCLIClient(evidence, sample_count=8)   # 8 requests per score instead of the default 20

Claude is L0 only. The Messages API returns no token log-probabilities and has no n parameter, so every score is built from sample_count separate requests (run concurrently, up to max_concurrency=8). Each request uses structured output: a JSON schema whose answer is an enum of the option keys, so every reply is a valid option. The Anthropic Python SDK 1.x has no temperature parameter and current Claude models sample at their default, which is what gives the samples their diversity; the backend never sets one.

• effort (optional) is passed through and fingerprinted: hold it fixed between calibration and serving.
• thinking="auto" (the default) disables extended thinking for evidence requests where the model allows it, and otherwise raises the token limit to leave room for it. A reply that stops on max_tokens or is refused counts as an invalid sample, never as a guess. Whether thinking was disabled is part of the fingerprint.

What calibration costs at L0. Every example costs sample_count requests, and so does every guarded call at run time:

To reduce it:

1. Score with an L1 model and keep Claude as the agent (--provider anthropic --evidence-provider vllm or openai): one request per score.
2. Lower sample_count. Scores get coarser (at most sample_count + 1 values), so more calls escalate, but the guarantee stays valid. sample_count is fingerprinted at L0: use the same value to calibrate and to serve.
3. Calibrate once. Profiles are cached in the store directory and reused until the prompt or the backend changes; commit them.
4. Guard only the actions that carry risk. Tools without a rule are never scored.

Google Gemini

import os
 
from cli_sdk.evidence import gemini_backend
 
evidence = gemini_backend("gemini-2.5-flash", api_key=os.environ.get("GOOGLE_API_KEY", "YOUR_GOOGLE_API_KEY"))

gemini_backend calls Gemini through its OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/) and samples (L0), with the same costs as Claude. It needs the OpenAI SDK (cci-sdk[openai]), not google-genai. Samples are requested with n in one request; if an endpoint rejects n greater than 1, pass supports_n=False to send one request per sample. Use stable model codes, never -latest aliases.

Gemma on vLLM

vllm serve google/gemma-4-12B-it --max-logprobs 20 --generation-config vllm

vllm serve google/gemma-4-12B-it --max-logprobs 20 --generation-config vllm \
  --enable-auto-tool-choice --tool-call-parser gemma4

from cli_sdk.evidence import vllm_backend
 
evidence = vllm_backend("google/gemma-4-12B-it", base_url="http://localhost:8000/v1")

Gemma 4 instruction models (google/gemma-4-E2B-it, -E4B-it, -12B-it, -26B-A4B-it, -31B-it) are Apache-2.0 and ungated. Gemma 3 repositories are gated, and Gemma 3’s chat template has no system role and requires strictly alternating turns, which breaks agent frameworks that insert system messages mid-conversation; prefer Gemma 4 for agents.

Gemma on SGLang

python -m sglang.launch_server --model-path google/gemma-4-12B-it --port 30000
# with tool calling, when it also serves the agent model:
python -m sglang.launch_server --model-path google/gemma-4-12B-it --port 30000 --tool-call-parser gemma4

from cli_sdk.evidence import sglang_backend
 
evidence = sglang_backend("google/gemma-4-12B-it", base_url="http://localhost:30000/v1")

• Temperature-scaled log-probabilities. At temperature above 0, SGLang returns log-probabilities computed after dividing the logits by the temperature, unless the server runs with SGLANG_RETURN_ORIGINAL_LOGPROB=1. The backend scores at temperature 0, where SGLang returns the unscaled values, so L1 evidence is unaffected. Keep the server’s setting fixed between calibration and serving.
• Seeds are honoured only with --enable-deterministic-inference (FlashInfer, FA3 or Triton attention backends, for example --attention-backend fa3).
• SGLang listens on 127.0.0.1:30000 by default; sglang serve --model-path ... is an equivalent launcher. disable_thinking and api_key behave as for vLLM.

Anything else

Other OpenAI-compatible servers

import os
 
from cli_sdk.evidence import OpenAICompatibleEvidenceBackend
 
evidence = OpenAICompatibleEvidenceBackend(
    "my-served-model",
    base_url="https://llm-gateway.internal.example/v1",
    api_key=os.environ.get("GATEWAY_API_KEY", "YOUR_GATEWAY_API_KEY"),
    use_logprobs=False,          # True only if the server returns top_logprobs
)

Use it for any server that speaks the OpenAI Chat Completions API. Keep use_logprobs=True only if the server returns top_logprobs for the first token; otherwise it samples. It sends max_tokens by default (token_limit_param="max_completion_tokens" for servers that want the newer name).

The agent’s own LangChain chat model

LangChainEvidenceBackend(chat_model, use_logprobs=False) scores with any LangChain chat model. It is L0 unless the model returns log-probabilities: only ChatOpenAI and AzureChatOpenAI on the Chat Completions API (use_responses_api=False) do, including ChatOpenAI(base_url=...) pointed at vLLM or SGLang. The wrapped model samples at its own settings, and those settings are fingerprinted: pass an instance at temperature=1.0 for sampled evidence, not the agent’s temperature=0 instance, which would return the same reply every time.

Your own CustomBackend

Subclass CustomBackend, declare the access level, and implement score_options (L1) or sample (L0). Implement fingerprint() too, so a new model version makes old profiles stale:

import random
 
from cli_sdk import CustomBackend, LocalCLIClient, Set
 
 
class InHouseTriageModel(CustomBackend):
    """Wraps a model you already run. Local mode calls score_options at L1 and above."""
 
    access_level = "L1"
    name = "in-house-triage"
 
    def __init__(self, model, version):
        super().__init__()
        self.model = model
        self.version = version
 
    def score_options(self, context, instructions, options):
        return self.model(context, list(options))          # one probability per option key
 
    def fingerprint(self):
        # Everything that defines the scoring function; a change makes stored profiles stale.
        return {"provider": "in-house", "model": self.name, "version": self.version,
                "access_level": self.access_level}
 
 
def stand_in_model(context, options):
    """Stands in for your model in this snippet: leans towards the ticket's hint, with noise."""
    rng = random.Random(context["id"])
    weights = {o: rng.random() + (2.0 if o == context["hint"] else 0.0) for o in options}
    total = sum(weights.values())
    return {o: w / total for o, w in weights.items()}
 
 
query = Set(instructions="How urgent is this ticket?", options={"urgent": None, "routine": None},
            calibration_profile="in-house-triage-v1", alpha=0.10)
rng = random.Random(0)
examples = []
for i in range(200):  # synthetic labelled history; use your own in practice
    label = rng.choice(["urgent", "routine"])
    hint = label if rng.random() < 0.85 else rng.choice(["urgent", "routine"])
    examples.append({"context": {"id": i, "hint": hint}, "label": label})
store = ".cli_profiles"

client = LocalCLIClient(InHouseTriageModel(stand_in_model, version="2026-09-01"), store=store)
client.calibrate(query, examples)
answer = client.evaluate({"id": 999, "hint": "urgent"}, {"q": query}).answers["q"]
print(answer.set, answer.guarantee.describe())
upgraded = LocalCLIClient(InHouseTriageModel(stand_in_model, version="2026-10-01"), store=store)
print(upgraded.calibration_status(query))

['routine', 'urgent'] Contains the correct answer at least 90% of the time on profile 'triage-inhouse-v1' (n=200).
(False, "profile 'triage-inhouse-v1' was calibrated with a different backend or settings (in-house:in-house-triage at L1); recalibrate for this backend")

In local mode, score_options(context, instructions, options) receives the query’s options (for Belief and Gate, {"true": ..., "false": ...}; for Interval, the levels) and returns a score per option key; scores are clipped at 0 and renormalized. At L0, local mode calls sample(context, prompt, n) with a prompt that lists the options, and parses each reply to an option key with a conservative rule; replies it cannot parse count as invalid. Without fingerprint(), the profile only records the class, name and access_level. See Backends: bring your own model for the same class used with the hosted service.

Agent models per framework

The examples build the agent model from the same provider settings. The classes they use:

Framework specifics, including install lines, are on the LangChain, LangGraph, Google ADK and Microsoft Agent Framework pages.

Environment variables

The examples read every key, model and endpoint from the environment, with placeholder defaults. A real provider stops with a clear message, before any network call, while its key is still a placeholder; --provider mock needs none of them. examples/agents/.env.example lists them all:

The examples do not read a .env file themselves. Export the variables in your shell, or load a copy of the file into it:

cp examples/agents/.env.example examples/agents/.env    # then replace the placeholders
set -a; . examples/agents/.env; set +a