Testing without keys

Everything CLI does can be exercised without a model provider key. Pick the recipe by what you are testing:

You are testing	Recipe	Needs
Your code’s handling of answers, `424`s, retries and errors	`httpx.MockTransport`	`pip install cci-sdk`
Calibration and guarantees end to end, offline	`LocalCLIClient` with `MockEvidenceBackend`	`pip install cci-sdk`
The hosted request path with your own model	`CustomBackend` client evidence	A CLI API key, no provider key
The full service, including faults	The mock provider on a local service	The service image or source

Unit tests with httpx.MockTransport

CLIClient(http_client=...) takes any httpx.Client, so a MockTransport stands in for the API. The SDK parses the canned responses exactly as it parses real ones.

import json
 
import httpx
import pytest
from cli_sdk import CLIClient, InsufficientCalibrationError, RetryConfig, Set
 
DEPARTMENT = Set(
    instructions="Which team should handle this ticket?",
    options={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    calibration_profile="support-routing-v3",
    alpha=0.10,
)
BACKEND = {"provider": "openai", "model": "gpt-4.1-2025-04-14"}
 
 
def answer(set_, guarantee):
    return {"answers": {"department": {"type": "set", "set": set_, "probabilities": {}, "guarantee": guarantee}},
            "backend": {**BACKEND, "access_level": "L1"}, "usage": {"backend_calls": 1, "backend_tokens": 296},
            "warnings": [], "request_id": "req_0123456789abcdef"}
 
 
COVERAGE = {"type": "coverage", "method": "APS", "alpha": 0.1, "calibration_profile": "support-routing-v3",
            "calibration_n": 1204}
HEURISTIC = {"type": "heuristic", "calibration_profile": "support-routing-v3", "calibration_n": 0,
             "statement": "No guarantee: profile 'support-routing-v3' is being calibrated"}
 
 
def client_for(handler, **kwargs):
    return CLIClient(api_key="cci_live_test", http_client=httpx.Client(transport=httpx.MockTransport(handler)),
                     **kwargs)
 
 
def test_singleton_is_routed():
    def handler(request):
        body = json.loads(request.content)
        assert body["queries"]["department"]["calibration_profile"] == "support-routing-v3"
        return httpx.Response(200, json=answer(["billing"], COVERAGE),
                              headers={"x-request-id": "req_0123456789abcdef"})
 
    result = client_for(handler).evaluate(context={"ticket": "I was charged twice."}, backend=BACKEND,
                                          queries={"department": DEPARTMENT})
    assert result.answers["department"].top == "billing"
    assert result.request_id == "req_0123456789abcdef"
 
 
def test_heuristic_answers_are_returned_or_refused():
    def handler(request):
        return httpx.Response(424, json=answer(["billing", "technical"], HEURISTIC))
 
    result = client_for(handler).evaluate(context={"ticket": "?"}, backend=BACKEND, queries={"department": DEPARTMENT})
    assert result.heuristic_answers == ["department"]
    with pytest.raises(InsufficientCalibrationError):
        client_for(handler, strict_guarantees=True).evaluate(
            context={"ticket": "?"}, backend=BACKEND, queries={"department": DEPARTMENT})
 
 
def test_a_retried_call_reuses_its_idempotency_key():
    seen = []
 
    def handler(request):
        seen.append(request.headers["Idempotency-Key"])
        if len(seen) == 1:
            return httpx.Response(502, json={"message": "provider failed", "code": "provider_unavailable"})
        return httpx.Response(200, json=answer(["billing"], COVERAGE))
 
    client = client_for(handler, retry=RetryConfig(base_delay=0.0, max_delay=0.0))
    client.evaluate(context={"ticket": "I was charged twice."}, backend=BACKEND, queries={"department": DEPARTMENT})
    assert len(seen) == 2 and seen[0] == seen[1]

Shape canned responses from API reference and Errors and retries, so your tests cover the codes your code branches on.

Offline with LocalCLIClient

LocalCLIClient runs the guarantee engine in your process, and MockEvidenceBackend is a deterministic, keyless evidence model: it scores options by word overlap between the context and each option (plus any keywords you give it), with seeded noise so calibration behaves as it would on a real, imperfect model. Calibration, evaluation, audits and the e-process all run with no network.

import random
 
from cli_sdk import Set
from cli_sdk.evidence import MockEvidenceBackend
from cli_sdk.local import LocalCLIClient
 
DEPARTMENT = Set(
    instructions="Which team should handle this ticket?",
    options={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    calibration_profile="support-routing-test",
    alpha=0.10,
)
PHRASES = {"billing": ["I was charged twice", "refund my last invoice", "my payment failed"],
           "technical": ["the app crashes on launch", "the integration is down", "export shows an error"]}
 
 
def tickets(n, seed):
    rng = random.Random(seed)
    rows = []
    for _ in range(n):
        label = rng.choice(list(PHRASES))
        rows.append({"context": {"ticket": rng.choice(PHRASES[label])}, "label": label})
    return rows
 
 
def test_guarantee_holds_offline(tmp_path):
    client = LocalCLIClient(MockEvidenceBackend(seed=0), store=str(tmp_path))
    profile = client.calibrate(DEPARTMENT, tickets(300, seed=1))
    assert profile.can_serve_guarantees
 
    answer = client.evaluate({"ticket": "please refund my invoice"}, {"department": DEPARTMENT}).answers["department"]
    assert not answer.is_heuristic
 
    audit = client.audit(DEPARTMENT, tickets(200, seed=2))
    assert audit.passed

MockEvidenceBackend(scorer=...) takes your own scoring function instead, for tests that need a particular score distribution. The local mode page covers profiles, fingerprints and audits in depth.

Your own model with client evidence

With a CustomBackend, the SDK computes evidence in your process and the hosted service only runs the statistics: it never calls a model, so no provider connection is needed. That makes it the way to test the hosted path end to end with a small stand-in model.

from cli_sdk import CLIClient, CustomBackend, Set
 
DEPARTMENT = Set(
    instructions="Which team should handle this ticket?",
    options={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    calibration_profile="triage-keyword-v1",
    alpha=0.10,
)
WORDS = {"billing": ("charged", "refund", "invoice", "payment"), "technical": ("crash", "error", "down", "bug")}
 
 
class KeywordModel(CustomBackend):
    access_level = "L1"
    name = "keyword-triage-v1"
 
    def score_options(self, context, instructions, options):
        text = context["ticket"].lower()
        hits = {key: 1.0 + sum(word in text for word in WORDS[key]) for key in options}
        total = sum(hits.values())
        return {key: value / total for key, value in hits.items()}
 
 
model = KeywordModel()
 
with CLIClient(backend=model) as client:
    client.calibration_profiles.create("triage-keyword-v1", backend=model, method="APS", alpha=0.10)
    client.calibration_profiles.add_examples("triage-keyword-v1", examples=[
        {"context": row["context"], "label": row["label"],
         "evidence": model.collect_evidence(DEPARTMENT.to_payload(), row["context"], sample_count=client.sample_count)}
        for row in labelled_rows
    ])
    result = client.evaluate(context={"ticket": "I was charged twice"}, queries={"department": DEPARTMENT})
    print(result.answers["department"].set, result.answers["department"].guarantee.describe())

Every human-labelled example of a client-evidence profile must carry the evidence your backend computes for that context, exactly as evaluate sends it; the calibration page explains why. Client-evidence calibration needs no model calls, so it always runs inline.

The mock provider on a local service

A local copy of the service can enable a keyless mock provider: a deterministic keyword scorer that behaves like a real provider in every other respect (it is metered, traced, and can fail on demand). It is refused in production, so it never exists on cci.gitdate.ink; start the service with CCI_ENV=development:

docker run --rm -p 127.0.0.1:8000:8000 \
  -e CCI_ENV=development -e CCI_AUTH_MODE=none -e CCI_ALLOW_ANONYMOUS=1 \
  -e CCI_ENABLE_MOCK_PROVIDER=1 \
  cci-stats-api:latest

or, from the service’s source directory, as its README describes:

CCI_ENV=development CCI_AUTH_MODE=none CCI_BIND_HOST=127.0.0.1 CCI_ENABLE_MOCK_PROVIDER=1 \
CCI_SECRETS_KEY=$(.venv/bin/python -m app.admin secrets generate) \
DATABASE_PATH=/tmp/cci/cci.db CCI_STATS_DB_PATH=/tmp/cci/cci-stats.db \
CCI_OBSERVABILITY_DB_PATH=/tmp/cci/cci-observability.db CCI_PROFILES_DIR=/tmp/cci/profiles \
.venv/bin/uvicorn --factory app.main:create_app --port 8000

Then use the SDK as you would against the hosted service:

from cli_sdk import CLIClient, Set
 
MOCK = {"provider": "mock", "model": "mock-v1",
        "keywords": {"billing": ["charged", "refund", "invoice"], "technical": ["crash", "error", "down"]}}
DEPARTMENT = Set(
    instructions="Which team should handle this ticket?",
    options={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    calibration_profile="support-routing-mock",
    alpha=0.10,
)
 
client = CLIClient(api_key="local", base_url="http://127.0.0.1:8000/v1", backend=MOCK)
client.calibration_profiles.create("support-routing-mock", backend=MOCK, method="APS", alpha=0.10)
client.calibration_profiles.add_examples("support-routing-mock", examples=labelled_rows)   # up to 200 for inline
result = client.evaluate(context={"ticket": "I was charged twice"}, queries={"department": DEPARTMENT})

The mock backend accepts keywords (option key to extra words), access_level (L1, the default, or L0), sample_count, seed, sharpness, noise, usage (fixed token counts to report) and fault. A fault is not part of the backend’s fingerprint, so you can add one to a calibrated mock backend and the profile keeps serving:

flaky = {**MOCK, "fault": {"mode": "fail_first", "times": 1}}
result = client.evaluate(context={"ticket": "I was charged twice"}, backend=flaky,
                         queries={"department": DEPARTMENT})
# the first attempt fails with 502; the SDK retries with the same Idempotency-Key and succeeds

`fault.mode`	Each model call	What the SDK does
`error`	Fails: `502 provider_unavailable`	Retries, then raises `BackendError`
`rate_limit`	Fails: `429 provider_rate_limited` with `Retry-After` (`retry_after`, default 2 s)	Retries after the wait, then raises `RateLimitError`
`overloaded`	Fails: `529 provider_overloaded` (`retry_after`, default 3 s)	Retries, then raises `BackendError`
`quota`	Fails: `402 provider_quota_exhausted`	Raises `CLIError` at once
`auth`	Fails: `422 provider_auth_failed`, field `backend.connection`	Raises `ValidationError` at once
`slow`	Waits `latency_ms` (default 1,500) first	Exercises your timeouts
`fail_first`	Fails the first `times` calls (default 1) of each `Idempotency-Key` with `status` (`502` by default, or `429`), then succeeds	Exercises retries: the retry carries the same key and succeeds

Requests on a local service are traced like hosted ones, with is_mock: true; GET /v1/requests?q=mock:true lists them.

The service’s own test suites run the same way, with no provider credentials: the stats-api unit and integration tests (pytest in the service directory), a full-stack run of the dashboard, proxy, service and SDK against the mock provider, and a wire-compatibility suite that replays every SDK call from published cci-sdk wheels.

Organizations and SSO Exchangeability