Backends

CLI is model-agnostic. Every backend adapter reports the access level it can reach; the calibration engine is identical regardless of which one you use.

A backend can be passed as a typed object, as a plain dict with a "provider" key, or as a CustomBackend that runs your own model. Pass it to client.evaluate(backend=...), to the client constructor (CLIClient(backend=...)) as a default, to calibration_profiles.create(backend=...), or inside a Judge or Route cascade stage.

from cli_sdk import OpenAIBackend
 
backend = OpenAIBackend(model="gpt-4.1-2025-04-14")
# equivalent dict: {"provider": "openai", "model": "gpt-4.1-2025-04-14"}

Fields every hosted backend has

These backends are configuration only: the CLI service calls the provider on your behalf with the stored credential. Constructing one without model, or with an unknown access_hint, raises ConfigurationError.

Closed APIs

OpenAI

from cli_sdk import OpenAIBackend
 
OpenAIBackend(model="gpt-4.1-2025-04-14")
OpenAIBackend(model="gpt-6-astra", reasoning_effort="medium")

Reaches L1 (top-20 generated-token log-probabilities per position) only on non-reasoning configurations: gpt-4.1 and models run with reasoning_effort="none". Any other reasoning effort, and gpt-6-astra at any effort, reaches L0, and CLI falls back to sampling-based scoring.

Azure OpenAI

from cli_sdk import AzureOpenAIBackend
 
AzureOpenAIBackend(model="gpt-4.1", deployment="support-gpt41", api_version="2025-04-01-preview")

Same access levels as OpenAI. Set the deployment’s version-upgrade policy to NoAutoUpgrade for any deployment backing a calibration profile. CLI records the deployment’s reported model version in the profile’s backend fingerprint, since Azure deployment versions can differ from the underlying model’s own version string.

Anthropic Claude

from cli_sdk import AnthropicBackend
 
AnthropicBackend(model="claude-sonnet-5", sample_count=20)

Reaches L0 only: the Messages API returns no token probabilities and has no n parameter, so CLI builds every score from repeated sampling (Set, Belief) or resampled claim support (Claim). Current Claude models reject non-default sampling temperature, so CLI never sets it.

Google Gemini

from cli_sdk import GeminiBackend
 
GeminiBackend(model="gemini-3.8-flash", thinking_level="low")

Covers the Gemini API and Vertex AI. Log-probabilities are deprecated for Gemini 3.x, so treat 3.x models as L0; Gemini 2.5-generation models can reach L1, subject to your project’s access. Use stable model codes, never -latest aliases.

AWS Bedrock

from cli_sdk import BedrockBackend
 
BedrockBackend(model="anthropic.claude-sonnet-5", region="us-east-1")
BedrockBackend(model="arn:aws:bedrock:us-east-1:123456789012:imported-model/abc123",
               custom_model_import=True)

Natively hosted models reach L0. Models you import through Bedrock Custom Model Import (your own open weights) can reach L2, including scoring of text you supply.

OpenRouter

from cli_sdk import OpenRouterBackend
 
OpenRouterBackend(model="openai/gpt-4.1", upstream="openai", pin_upstream=True)

The access level depends on the upstream provider. Keep pin_upstream on: without it, a silent fallback to another upstream would change the scoring model between calibration and serving and break exchangeability.

Self-hosted engines

The CLI service calls these servers directly, so base_url must be reachable from the service (from inside your VPC when you self-host). Both classes raise ConfigurationError without base_url.

vLLM

from cli_sdk import VLLMBackend
 
VLLMBackend(
    model="meta-llama/Llama-3.3-70B-Instruct",
    base_url="http://vllm.internal:8000",
    engine_version="0.11.0",
    quantization="fp8",
)

Reaches up to L4: logprob_token_ids for exact label-token probabilities, prompt_logprobs for scoring text you supply, and hidden-state extraction for probe-based scores. Run the server with VLLM_BATCH_INVARIANT=1 for bitwise-reproducible calibration runs.

SGLang

from cli_sdk import SGLangBackend
 
SGLangBackend(
    model="meta-llama/Llama-3.3-70B-Instruct",
    base_url="http://sglang.internal:30000",
    deterministic=True,
)

Reaches up to L4: token_ids_logprob for exact label probabilities, logprob_start_len for cheap candidate scoring against a shared prefix, and return_hidden_states for probe-based scores. SGLang returns temperature-scaled log-probabilities by default and ignores per-request seeds unless deterministic inference is enabled.

Bring your own model

A CustomBackend runs your model inside your process. You implement the evidence methods your access level supports; the SDK calls them before sending the request and sends only the resulting evidence (probabilities, samples, feature vectors) with backend provider "client_evidence". Your model, its weights, and its API credentials never leave your infrastructure.

from cli_sdk import CLIClient, CustomBackend, Set
 
class MyEngine(CustomBackend):
    access_level = "L1"
    name = "in-house-classifier-v7"
 
    def score_options(self, context, instructions, options):
        # a probability for each option key (need not sum to 1)
        return my_model.option_probabilities(context, instructions, list(options))
 
    def sample(self, context, instructions, n):
        # n independently sampled answers
        return [my_model.generate(context, instructions) for _ in range(n)]
 
client = CLIClient(backend=MyEngine(), sample_count=20)

What each query type asks your methods for:

sample receives n = sample_count from the client (default 20). Route queries are skipped: they declare their own cascade. Methods you do not implement raise NotImplementedError, so a query that needs one fails with a clear error instead of silently downgrading its guarantee. The constructor accepts base_url=None and arbitrary keyword options (stored on self.base_url and self.options) for your own use, and validates access_level ("L0" to "L4"), raising ConfigurationError otherwise.

Pinning and fingerprinting

Every CalibrationProfile records a backend fingerprint: provider, model identifier, prompt-template hash, and for self-hosted engines the engine version, quantization, and log-probability mode (profile.backend_fingerprint). A request whose live fingerprint does not match its profile’s stored fingerprint is flagged, and with strict_fingerprint=True (the default in calibration_profiles.create) it is refused rather than served against a stale calibration. A fingerprint drift monitor alerts on the change.

Listing configured backends

with CLIClient() as client:
    for backend in client.list_backends():
        print(backend)       # provider, model, and detected access level