Observability

The hosted service records every request made with one of your API keys as a trace: what was asked, which backend and connection served it, how the calibration profile was resolved, every model call with its tokens and latency, the statistic that produced each answer, and the guarantee card that came back. You read it in the dashboard under Observability, through the events read API, or in your own tracing backend through OTLP export.

What is recorded

Every API-key event becomes a request record, including the ones that fail: validation errors, 401 responses to a revoked key, rate-limited requests and fail-closed 424 answers. A background job (a calibration, an append, a large audit or a label-with-judge pool) gets its own trace, linked to the request that queued it.

Alongside the record: the span tree, one answer record per query (type, profile and version, guarantee type, the binding decision, and why an answer was heuristic), the request and response payloads when capture allows it, and any guard events your agent sent for the same request or thread.

Reading a trace

A trace is a tree of spans under one root span named after the method and route. The spans you will see:

Model-call spans follow the OpenTelemetry generative-AI conventions: gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.response.id, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, gen_ai.request.choice.count, plus cci.purpose (answer, calibration, audit, judge or outcome), cci.query.id, cci.connection.id, the provider’s own request id, the estimated cost, and on failure error.type and http.response.status_code. When payload capture allows it, the first 4,000 characters of the prompt and the reply are kept as gen_ai.input.preview and gen_ai.output.preview.

Fan-outs are compacted: a calibration or audit over thousands of examples keeps the first 25 item spans and every failed one, and folds the rest into one (compacted) span per parent with counts and the slowest time. Retries that share an Idempotency-Key stay separate requests, and each request lists its sibling attempts, so a 502, 502, 200 sequence reads as one logical call.

Correlation headers

Send these headers to group requests into sessions and threads, and to filter by them later. The SDK sends whatever you pass in default_headers on every request:

from cli_sdk import CLIClient
 
client = CLIClient(default_headers={
    "X-CCI-Session-Id": "triage-run-42",
    "X-CCI-Environment": "production",
    "X-CCI-Agent-Name": "claims-triage",
    "X-CCI-Tags": "pilot,eu",
    "X-CCI-Metadata": '{"team": "ops", "release": "2026.10.1"}',
})

default_headers are fixed per client. To tag each call with the current thread or tool call, pass your own httpx.Client with a request hook:

import contextvars
 
import httpx
from cli_sdk import CLIClient
 
current_thread = contextvars.ContextVar("current_thread", default=None)
 
 
def tag_thread(request: httpx.Request) -> None:
    thread = current_thread.get()
    if thread:
        request.headers["X-CCI-Thread-Id"] = thread
 
 
http = httpx.Client(timeout=60.0, event_hooks={"request": [tag_thread]})
client = CLIClient(http_client=http, default_headers={"X-CCI-Environment": "production"})
 
current_thread.set("th_77")
# client.evaluate(...) now carries X-CCI-Thread-Id: th_77

The SDK does not close an http_client you pass in. With AsyncCLIClient, use an httpx.AsyncClient and an async def hook.

Payload capture and PII masking

Observability > Data settings controls what is kept of request and response bodies. Owners and admins can change it; every change is in the audit log.

Redaction runs before anything is written:

• Values under key names that look like credentials (authorization, api_key, password, secret, token, cookie, private_key, access_key, session_id, credential, and your own denylist) become [REDACTED], whatever the masking setting.
• Values that look like secrets (provider API keys, cci_live_ keys, AWS access key ids, Google API keys, Slack tokens) become [SECRET], whatever the masking setting.
• With PII masking on: email addresses become [EMAIL], phone numbers [PHONE], US social security numbers [SSN], and card numbers that pass the Luhn check [CARD].
• Matches of the organization’s custom patterns (up to 50 regular expressions, stored with the data settings) become [REDACTED].
• Each body is capped at 64 KB; a larger one is stored as a truncated summary and marked truncated.

The same rules apply to model-call previews and guard-event content. X-CCI-Omit-Payload: 1 skips capture for one request, and a request that touches a zero-retention profile is never captured.

Retention

Requests, spans, answers, payloads and guard events are deleted after the organization’s retention period (30 days by default, 1 to 90). Profile events (calibrations, versions, stale markers, audits, monitor alarms) are kept about a year longer, and hourly usage rollups for 400 days, so the Usage page and profile history outlive the detailed traces. An owner can delete all telemetry, or only captured content, from Data settings.

Zero retention

A zero-retention profile keeps no calibration contexts once they are scored. Set it per profile, or make it the organization default in Data settings:

import os
import uuid
 
import httpx
 
API = os.environ.get("CLI_BASE_URL", "https://cci.gitdate.ink/api/v1").rstrip("/")
HEADERS = {"Authorization": f"Bearer {os.environ['CLI_API_KEY']}", "Idempotency-Key": str(uuid.uuid4())}
 
# On an existing profile: only "retention" can be changed ("standard", "zero", or null to inherit).
httpx.patch(f"{API}/calibration-profiles/patient-triage-v1", json={"retention": "zero"},
            headers=HEADERS, timeout=30).raise_for_status()

A new profile can also be created with "retention": "zero" in the POST /v1/calibration-profiles body (the SDK’s create does not send this field). What changes:

• After an example is scored, its raw context is discarded. The stored row keeps a salted hash of the context, the label, the source, the group and the evidence. Contexts of examples that are not scored yet stay only until their scoring runs.
• Payload capture is off for every request that touches the profile, and an organization-wide zero-retention default turns capture off for every request of the organization.
• Audits, outcomes and monitors keep working, because callers send the contexts with each call and nothing is stored. Audits run synchronously only, and so does label-with-judge (a pool large enough to need a job fails with 422 contexts_required); its human-labelling tasks are returned once and stored without their contexts.
• A backend change cannot be re-scored without contexts: the profile becomes stale with reason contexts_required, and POST .../calibrate with a new backend fails with 422 contexts_required. Upload the examples again (or use a new profile) to calibrate the new backend.
• Only owners and admins can switch a zero-retention profile back to standard retention.

Dashboard pages

Provider connections and API keys sit next to Observability in the sidebar; the API keys page shows each key’s last 24 hours of use.

Saved views and columns

On Requests, Columns chooses which fields the table shows, and Save view stores the current search, filters, columns and sort under a name. A view is personal (visible to you) or shared with the organization. Five presets are always available:

Alerts and signed webhooks

An alert rule watches one metric over a trailing window and opens an incident when the comparison holds and the window has at least the minimum number of requests. Rules are evaluated every minute. An incident notifies the rule’s channels once when it opens and again when it resolves; the cooldown keeps a flapping rule from reopening too often.

A rule has a name, a metric, a window (60 seconds to 7 days; 15 minutes by default), a comparator (>, >=, <, <=), a threshold, a minimum request count, a cooldown, an optional scope (one API key, one profile or one operation) and up to 10 channels. Channels are a webhook (JSON over HTTPS) or a Slack incoming webhook. There are no email alerts.

A webhook receives:

{
  "event": "incident.opened",
  "incident_id": "inc_01k...",
  "rule_id": "rule_...",
  "rule_name": "Fail-closed spike",
  "metric": "fail_closed_rate",
  "value": 0.083,
  "threshold": 0.05,
  "comparator": ">",
  "window_s": 900,
  "org_id": "org_...",
  "opened_at": "2026-10-01T09:41:00.000Z",
  "resolved_at": null,
  "dashboard_url": "https://cci.gitdate.ink/dashboard/observability/alerts"
}

event is incident.opened or incident.resolved. When the channel has a signing secret (16 to 256 characters, set when you create the channel), each request carries

x-cci-signature: t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>." followed by the raw body>

keyed with the secret. Verify it against the raw request body, before parsing, and reject stale timestamps:

import hashlib
import hmac
import time
 
 
def verify_cci_signature(secret: str, header: str, body: bytes, tolerance_s: int = 300) -> bool:
    """True when `header` (the x-cci-signature value) signs `body` with `secret`."""
    try:
        fields = dict(part.split("=", 1) for part in header.split(","))
        timestamp = int(fields["t"])
        received = fields["v1"]
    except (KeyError, ValueError):
        return False
    if abs(time.time() - timestamp) > tolerance_s:
        return False
    expected = hmac.new(secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, received)

For example, in a FastAPI receiver:

import json
import os
 
from fastapi import FastAPI, HTTPException, Request
 
app = FastAPI()
SECRET = os.environ["CCI_ALERT_SECRET"]
 
 
@app.post("/hooks/cci")
async def cci_alert(request: Request) -> dict:
    body = await request.body()
    if not verify_cci_signature(SECRET, request.headers.get("x-cci-signature", ""), body):
        raise HTTPException(status_code=401, detail="bad signature")
    incident = json.loads(body)
    if incident["event"] == "incident.opened":
        page_oncall(incident["rule_name"], incident["value"], incident["dashboard_url"])
    return {"ok": True}

Slack channels take no secret; they post one line naming the rule, the value, the threshold and the window, with a link to the dashboard.