The access ladder

Different backends expose different signal about their own outputs. CLI calls this the access ladder, and every backend adapter reports which level it can reach so the calibration engine can pick the strongest valid nonconformity score automatically.

Level	What the backend returns	Score CLI can build	Example backends
L0	Sampled text only	Sample-frequency and semantic-cluster nonconformity	Anthropic Claude; any reasoning-effort OpenAI/Azure config; Gemini 3.x
L1	Generated-token log-probabilities (top-k)	LAC / APS set scores directly from option probabilities	OpenAI/Azure non-reasoning configs; Gemini 2.5
L2	Log-probabilities of text you supply	Candidate-likelihood and claim-support scores	AWS Bedrock Custom Model Import, self-hosted vLLM/SGLang; Together and Fireworks via `CustomBackend`
L3	Exact probabilities for chosen label tokens, full-vocabulary logits	Exact APS/RAPS, token-level scores	Self-hosted vLLM, SGLang; TensorRT-LLM via `CustomBackend`
L4	Hidden states	Trained-probe nonconformity scores	Self-hosted vLLM (pooling / hidden-state extraction), SGLang; Hugging Face `transformers` via `CustomBackend`

A higher access level never changes what a guarantee means — it only changes how tight the resulting set or interval can be, and how cheaply it can be produced. A Set primitive on Claude (L0) and the same Set primitive on a self-hosted vLLM deployment (L3) both carry a coverage guarantee of exactly the same shape; the vLLM version typically returns smaller sets at lower cost per call.

How CLI picks a score

The backend adapter reports the highest access level it can reach for the current request (a reasoning-effort OpenAI config, for instance, reports L0 even though the same model reports L1 with reasoning effort set to none).
The requested primitive names the guarantee it needs (coverage, risk, FDR, anytime).
CLI selects the cheapest nonconformity score at or above the minimum access level that guarantee requires, and falls back to sampling-based scores when a higher level is unavailable.
The chosen method is recorded in the answer’s guarantee.method field — nothing about score selection is hidden from the caller.

Setting the access hint explicitly

Two knobs pin the score family instead of letting CLI choose:

access_hint on any hosted backend (OpenAIBackend, VLLMBackend, and the others), which applies to every query sent to that backend.
backend_access_hint on a single Set query.

from cli_sdk import OpenAIBackend, Set
 
backend = OpenAIBackend(model="gpt-4.1-2025-04-14", access_hint="logprobs")
 
department = Set(
    instructions="Which team should handle this ticket?",
    options={
        "billing": "Payments, invoicing, refunds",
        "technical": "Bugs, outages, integrations",
    },
    calibration_profile="support-routing-v3",
    backend_access_hint="logprobs",  # force L1 scoring; error if unavailable
)

Hint	Level it pins
`"auto"`	CLI picks (the default for `access_hint`; `backend_access_hint` left unset means the same)
`"sampling"`	L0
`"logprobs"`	L1
`"prompt-scoring"`	L2
`"exact"`	L3
`"hidden-state"`	L4

Leave hints on "auto" unless you have a specific reason to pin the score family, for example to guarantee that a calibration profile is never silently served with a different score type after a model configuration change. A pinned hint the backend cannot satisfy fails with a BackendError rather than downgrading the guarantee.

For a CustomBackend, you declare the level yourself with the access_level class attribute ("L0" to "L4").

Exchangeability The three engines