Provider connections

The hosted service calls model providers with your credentials. A provider connection is one stored credential (or, for a self-hosted engine, one server URL) that belongs to your organization. There is no shared provider key: a request that needs a model and finds no connection fails with 422 connection_required.

Connections are managed under Dashboard > Provider connections. Owners and admins can add, change, test and delete them; every member can see them and use them through API keys.

Add a connection

Connections to an endpoint you control, vllm, sglang, openai-compatible and azure-openai, must pass Test before a request can use them; until then they fail with 422 connection_unverified. Other connections can be used untested, and a wrong key surfaces on the first request as 422 provider_auth_failed. Run Test again after you change a self-hosted server, since its capabilities are recorded when you test.

Defaults and naming a connection from code

The first connection you add for a provider type becomes its default; you can make another one the default at any time. A backend without connection uses the default for its provider. To pick one explicitly, set connection on any typed backend, or "connection" in a backend dict:

from cli_sdk import AnthropicBackend, CLIClient, OpenAIBackend
 
with CLIClient() as client:
    client.list_backends()      # providers, their connections, and the access levels they reach
 
eval_backend = OpenAIBackend(model="gpt-4.1-2025-04-14", connection="openai-eval")
judge = AnthropicBackend(model="claude-sonnet-5-5", connection="claude-main", sample_count=8)
dict_backend = {"provider": "openai", "model": "gpt-4.1-2025-04-14", "connection": "openai-eval"}

A name that does not exist fails with 422 connection_not_found. The connection is resolved on every request, so rotating a key in the dashboard takes effect immediately. The connection is not part of a profile’s backend fingerprint, but its endpoint is: swapping one OpenAI key for another keeps a profile serving, while moving to a different Azure resource or self-hosted server is a backend change (see Calibration profiles).

How secrets are stored

• Write-only. A secret is never returned by any API, the dashboard included. The dashboard shows a hint: the last four characters of a key, or a service account’s client_email. Changing a secret replaces it.
• Encrypted at rest with AES-256-GCM. The encryption key is held only by the API service (never by the dashboard), and each ciphertext is bound to its organization, provider and connection name, so it cannot be decrypted under another row.
• Never echoed. Provider error bodies are never passed back to the caller; the service returns a classified error code instead, and logs the detail against the request id.
• No ambient credentials. The service never falls back to credentials from its environment: Vertex AI tokens are minted from your stored service account, Bedrock uses only your stored keys.
• Guarded egress. Endpoints you supply (Azure, vLLM, SGLang, OpenAI-compatible) must use https and resolve only to public addresses; the service connects to the address it checked and does not follow redirects. Private or plain-HTTP endpoints work only in a self-hosted deployment that allows them.

Every change to a connection (created, updated, tested, made default, deleted) is recorded in the audit log.

OpenAI

from cli_sdk import OpenAIBackend
 
OpenAIBackend(model="gpt-4.1-2025-04-14")                          # L1
OpenAIBackend(model="gpt-6-luna", reasoning_effort="none")         # L1
OpenAIBackend(model="gpt-6-luna", reasoning_effort="medium")       # L0: sampling

L1 is reached by gpt-4.1 (and gpt-4o, gpt-4-turbo, gpt-3.5) and by GPT-5.x and GPT-6 Sol and Luna at reasoning_effort="none". When you leave reasoning_effort unset on those models, the service sends "none" so the query reaches L1. gpt-6-astra and gpt-6.1-sol reach L0 at every effort. At L0 the service samples, and sends neither temperature nor logprobs to a reasoning configuration.

Azure OpenAI

The connection holds the endpoint and key; the backend names the model and, optionally, the deployment.

from cli_sdk import AzureOpenAIBackend
 
AzureOpenAIBackend(model="gpt-4.1", deployment="gpt-4.1")                       # v1 API, L1
AzureOpenAIBackend(model="gpt-4.1", deployment="support-prod", access_hint="logprobs")
AzureOpenAIBackend(model="gpt-4.1", deployment="support-prod", api_version="2025-04-01-preview")

The endpoint must be https://<resource>.openai.azure.com, .cognitiveservices.azure.com or .services.ai.azure.com. The deployment is the backend’s deployment, else the connection’s default deployment, else model. The access level follows model, the model the deployment serves (set it to the model family, e.g. gpt-4.1), whatever the deployment is called. The fingerprint records the endpoint, the API version and model; set the deployment’s upgrade policy to NoAutoUpgrade.

Anthropic

from cli_sdk import AnthropicBackend
 
AnthropicBackend(model="claude-sonnet-5-5", sample_count=8)
AnthropicBackend(model="claude-haiku-4-5", effort="low")    # effort is not sent to Haiku models

Claude is L0: every score is built from sample_count samples. The service sends each model the thinking settings it accepts (Claude Sonnet 5.5 rejects disabled thinking and runs with thinking between tools). Profiles calibrated on claude-sonnet-5 keep their original fingerprint and keep serving.

Gemini API and Vertex AI

A gemini connection has two modes.

from cli_sdk import GeminiBackend
 
GeminiBackend(model="gemini-3.8-flash", thinking_level="low")
vertex = {"provider": "vertex", "model": "gemini-3.8-flash", "connection": "vertex-eu", "sample_count": 8}

In Vertex mode the service mints short-lived OAuth tokens from the stored service account (the JSON file you download from Google Cloud; its token_uri must be https://oauth2.googleapis.com/token). A model name without a publisher gets google/ in front. provider: "vertex" against a connection that is not in Vertex mode fails with 422 connection_mode_mismatch. Both modes are L0; the location is part of the fingerprint.

AWS Bedrock

from cli_sdk import BedrockBackend
 
BedrockBackend(model="us.amazon.nova-pro-v1:0", region="us-east-1")              # Converse API, L0
BedrockBackend(model="arn:aws:bedrock:us-east-1:123456789012:imported-model/abc123",
               custom_model_import=True)                                            # L1
BedrockBackend(model="arn:aws:bedrock:us-east-1:123456789012:imported-model/abc123",
               custom_model_import=True, access_hint="prompt-scoring")              # L2

The connection authenticates with access keys (plus an optional session token) or a Bedrock API key; the region comes from the backend, else the connection. Natively hosted models go through the Converse API at L0. Models you import with Custom Model Import (custom_model_import=True, with the imported model’s ARN as model) reach L1 from first-token log-probabilities, and L2 prompt scoring with access_hint="prompt-scoring". Run Test with the ARN as the model: the probe records whether the imported model returns log-probabilities and prompt log-probabilities.

OpenRouter

from cli_sdk import OpenRouterBackend
 
OpenRouterBackend(model="openai/gpt-4.1")                        # pinned to the "openai" upstream
OpenRouterBackend(model="meta-llama/llama-3.3-70b-instruct", upstream="together", quantization="fp8")

With pin_upstream=True (the default) the service routes only to one upstream, with fallbacks off and required parameters on. The upstream is upstream, or the vendor prefix of the model slug when upstream is unset; a slug with no prefix and no upstream fails with 422. The level is L1 only for an OpenAI model that reaches L1 on its own (for example openai/gpt-4.1) served by a pinned openai, together or fireworks upstream; every other OpenRouter configuration is L0. Keep pinning on: a silent fallback to another host would change the scoring model between calibration and serving.

vLLM

The connection holds the server’s base URL (and an API key if the server has one). The SDK’s VLLMBackend requires base_url; it must have the same origin as the connection, otherwise the request fails with 422 egress_denied.

from cli_sdk import VLLMBackend
 
VLLMBackend(model="google/gemma-4-12B-it", base_url="https://gemma.example.com/v1",
            engine_version="0.11.0", quantization="fp8")                          # L1
VLLMBackend(model="google/gemma-4-12B-it", base_url="https://gemma.example.com/v1",
            access_hint="exact")                                                   # L3

L2 and L3 need the server’s tokenizer endpoint (/tokenize), which Test looks for. engine_version, quantization and logprobs_mode (raw_logprobs or processed_logprobs) are recorded in the fingerprint. The service disables the chat template’s thinking mode for scoring.

SGLang

from cli_sdk import SGLangBackend
 
SGLangBackend(model="google/gemma-4-12B-it", base_url="https://gemma-sglang.example.com/v1",
              deterministic=True, access_hint="hidden-state")                      # L4

SGLang reaches L1 to L3 the same way as vLLM, and L4 when the server was started with --enable-return-hidden-states (Test detects it). At L4 the service trains a probe on half of the calibration rows and conformalizes its scores on the other half, so the split-conformal guarantee still holds; L4 serves Set, Belief, Gate and Interval queries.

OpenAI-compatible servers and presets

gateway = {"provider": "openai-compatible", "model": "llama-3.3-70b", "use_logprobs": True}
mistral = {"provider": "mistral", "model": "mistral-medium-2604", "sample_count": 8}
together = {"provider": "together", "model": "meta-llama/Llama-3.3-70B-Instruct-Turbo"}

openai-compatible covers any server speaking the Chat Completions API (gateways, Ollama, LM Studio) at its connection’s base URL: L0 by default, L1 with "use_logprobs": true when the server returns top_logprobs. The presets have fixed endpoints and need only an API key:

There are no typed SDK classes for these providers; pass a dict.

Errors

Inside a cascade or a judge, the field path names the stage, so you can tell which connection failed. In a self-hosted deployment without the dashboard, connections are managed over the service’s internal API.