Drift monitoring

A calibration profile is only as valid as its exchangeability assumption. CLI’s drift monitors are anytime-valid e-processes: sequential tests that stay honest under continuous checking, so you can look at a monitor every hour for a year without inflating its false-alarm rate above the number you configured. The math is on E-values.

Creating a monitor

from cli_sdk import CLIClient
 
with CLIClient() as client:
    monitor = client.calibration_profiles.monitors.create(
        "support-routing-v3",
        type="coverage",
        target=0.88,
        false_alarm_rate=0.05,
        labelled_sample_rate=0.02,   # fraction of production traffic to label
    )
    print(monitor.id, monitor.status)
 
    fingerprint = client.calibration_profiles.monitors.create(
        "support-routing-v3",
        type="fingerprint",
    )
 
    for m in client.calibration_profiles.monitors.list("support-routing-v3"):
        print(m.id, m.type, m.target, m.false_alarm_rate)

Argument	Default	Meaning
`profile`	required	The profile to watch.
`type`	`"coverage"`	`"coverage"`, `"risk"`, or `"fingerprint"`.
`target`	required except for `fingerprint`	Coverage floor or risk ceiling the monitor tests.
`false_alarm_rate`	`0.05`	$\delta$ in $(0, 1)$ : the chance of ever alarming while the guarantee holds.
`labelled_sample_rate`	`None`	Fraction of production traffic to label for the monitor.
`use_judge_pseudo_labels`	`False`	Supplement true labels with judge pseudo-labels (see below).

An invalid type, a missing target, or a false_alarm_rate outside $(0, 1)$ raises ValueError before any request is sent.

What it watches

Monitor type	What it detects
`coverage`	Realized coverage of a `Set` or `Interval` profile drifting below `target`
`risk`	Realized risk of a `Gate` profile drifting above `target`
`fingerprint`	Backend model version, prompt-template hash, or engine configuration changing without a matching recalibration

Set a coverage monitor’s target at or below the lower end of the profile’s realized_coverage_ci (0.88 for a profile whose interval is $[0.886, 0.914]$ ). The monitor tests the coverage of the calibration you actually deployed, which scatters around the nominal $1-\alpha$ ; a target set exactly at $1-\alpha$ can eventually, and correctly, flag a profile that landed slightly under it.

Why anytime-valid matters here

A classical significance test controls its false-alarm rate only at a single, pre-committed sample size. Checking it repeatedly as data arrives — the natural way to watch a production system — inflates the true false-alarm rate well above the stated number (to 43% in the simulation on E-values). E-process monitors avoid this by construction: the probability that the monitor ever alarms while the guarantee holds is at most false_alarm_rate, for any stopping rule, including “check continuously and alert the instant the evidence crosses the threshold”.

Responding to an alert

for alert in client.calibration_profiles.monitors.poll("support-routing-v3"):
    if alert.type == "coverage" and alert.severity == "critical":
        pause_auto_actioning(profile=alert.profile)
        notify_oncall(alert.message, e_value=alert.e_value, n=alert.n)

poll(profile, monitor_id=None) yields the alerts raised since the last poll, across every monitor on the profile (or one monitor when monitor_id is given). The async client returns a list.

`Alert` field	Meaning
`type`	`"coverage"`, `"risk"`, or `"fingerprint"`
`severity`	Default `"critical"`
`profile`	The watched profile
`n`	Outcomes observed when the alert fired
`e_value`	The evidence; at or above `1 / false_alarm_rate` when it fired
`target`, `false_alarm_rate`	The monitor’s configuration
`detected_at`, `message`	When, and a human-readable summary
`details`	Any other fields the service sent

A fingerprint alert typically fires before a coverage alert does — it catches a model swap or configuration change as soon as it shows up in the backend’s response metadata, rather than waiting for enough mislabelled traffic to accumulate statistical evidence. The coverage monitor is what catches a change that does not show up in metadata; see the silent model update guide.

Running a monitor locally

LocalMonitor runs the same e-process in your own process, with no network access, for air-gapped deployments or for replaying logs:

from cli_sdk import LocalMonitor
 
monitor = LocalMonitor(type="coverage", target=0.88, false_alarm_rate=0.05,
                       profile="support-routing-v3")
for label, prediction_set in labelled_stream:
    alert = monitor.update(label in prediction_set)
    if alert:
        page_oncall(alert)
        break

type is "coverage" (feed covered booleans) or "risk" (feed is_loss booleans). update returns an Alert the first time the evidence crosses 1 / false_alarm_rate, and None otherwise. Start a new monitor for each new profile version: a monitor that has watched a long in-spec period needs longer to fire after a change (see E-values: detection delay).

Label-free monitoring

When production labels are expensive, pass use_judge_pseudo_labels=True so the monitor combines judge pseudo-labels with a small true-labelled trickle, using the same prediction-powered machinery as label-efficient calibration. A monitor can then run continuously on a few true labels per hour rather than a full human-labelling pipeline.

The anytime-valid false-alarm guarantee of a monitor rests on its true labels. Judge pseudo-labels make the monitor react sooner, through a prediction-powered correction that is large-sample rather than finite-sample, so treat a judge-assisted alarm as strong evidence to check, and confirm it on true labels before acting on it:

client.calibration_profiles.monitors.create(
    "support-routing-v3",
    type="coverage",
    target=0.88,
    false_alarm_rate=0.05,
    labelled_sample_rate=0.005,
    use_judge_pseudo_labels=True,
)

Calibration profiles Guarantee reference