Command-line tool

pip install cci-sdk installs a cli command for inspecting and auditing calibration profiles from a terminal or a CI job. The same entry point is available as python -m cli_sdk.cli_tool.

cli calibration show support-routing-v3
cli calibration audit support-routing-v3 --examples fresh.jsonl --fail-below 0.88
cli calibration audit-local --covered outcomes.txt --target 0.90

show and audit call the API with the same configuration as CLIClient(): CLI_API_KEY, and CLI_BASE_URL for a self-hosted deployment. audit-local makes no network calls.

cli calibration show

cli calibration show NAME

Prints a profile’s size, coverage interval, and audit status:

profile            support-routing-v3 (v3)
method / alpha     APS / 0.1
n                  1204 (minimum 9, recommended 1000)
coverage CI        [0.886, 0.914]
serving guarantees yes
last audit         {'date': '2026-09-18', 'sample': 300, 'result': 'pass'}

serving guarantees is no (below minimum n) when the profile is below $\lceil (1-\alpha)/\alpha \rceil$ examples and its answers are heuristic.

cli calibration audit

cli calibration audit NAME --examples FILE [--fail-below X]

Audits a hosted profile against fresh labelled examples, through client.calibration_profiles.audit. FILE is JSONL: one example per line, with at least context and label, and optionally source, group, and metadata.

{"context": {"ticket": "I was charged twice."}, "label": "billing"}
{"context": {"ticket": "The export button does nothing."}, "label": "technical"}

Output:

result             pass
realized coverage  0.9033 on 300 examples
confidence interval [0.8642, 0.9343]

cli calibration audit-local

cli calibration audit-local --covered FILE --target T [--confidence C] [--fail-below X]

Runs the same audit offline on a file of coverage outcomes, one per line. 1, true or yes counts as covered and 0, false or no as not covered (case-insensitive); blank lines are skipped. Any other token is an input error: the command prints the file and line and exits 2. --target, --confidence and --fail-below must be strictly between 0 and 1; --confidence defaults to 0.95.

To audit a Gate or other risk profile, write one line per decision with 1 when the decision incurred no loss, and pass --target as one minus the risk target (0.99 for a 1% risk target).

How an audit decides

The audit computes the realized coverage on the file and its two-sided Clopper-Pearson interval at --confidence. It fails when the upper end of that interval is below --target, that is, when the sample gives real evidence of under-coverage. A small sample that lands a little under the target passes.

--fail-below X adds a stricter, point-estimate check: the command also fails when the realized coverage itself is below X, and prints below --fail-below X.

Exit codes

Status 1 and 2 are deliberately different, so a CI job can tell “the guarantee does not hold” apart from “the check could not run”. Treat both as “do not ship” in a deploy gate.

Blocking a deploy in CI

# .github/workflows/calibration-audit.yml
name: calibration-audit
on: [pull_request]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install cci-sdk
      - run: cli calibration audit support-routing-v3 --examples eval/fresh.jsonl --fail-below 0.88
        env:
          CLI_API_KEY: ${{ secrets.CLI_API_KEY }}

For an air-gapped pipeline, compute coverage outcomes on the build machine and run cli calibration audit-local instead; it needs no key and no network.