Detecting a silent model update

Problem

A support-routing profile was calibrated against a vendor model. Weeks later the vendor changes what serves that model — a new snapshot behind an alias, a serving-stack change, a different quantization — and coverage quietly drops from 90% to 80%. Nothing errors. The first sign is a quarterly review.

Checking a dashboard of “coverage over the last N tickets” every day does not fix this. Each look is a new chance for noise to cross the line, so a daily check either alarms constantly or is tuned so loosely it misses real changes.

Primitives

• A fingerprint monitor: alerts as soon as the backend’s reported model version, prompt-template hash, or engine configuration stops matching the profile.
• A coverage monitor: an anytime-valid e-process on a small labelled sample of traffic, for changes that do not show up in any metadata.
• LocalMonitor, the same e-process in your own process.

Code

from cli_sdk import CLIClient
 
PROFILE = "support-routing-v3"
 
with CLIClient() as client:
    profile = client.calibration_profiles.get(PROFILE)
    floor = round(profile.realized_coverage_ci[0], 2) if profile.realized_coverage_ci else 0.88
 
    monitors = client.calibration_profiles.monitors
    monitors.create(PROFILE, type="fingerprint")
    monitors.create(PROFILE, type="coverage", target=floor, false_alarm_rate=0.05,
                    labelled_sample_rate=0.02)

import numpy as np
from cli_sdk import LocalMonitor
from cli_sdk.stats.evalues.anytime import normal_ppf
 
 
def first_alarm(covered, target=0.88, false_alarm_rate=0.05):
    """The outcome index at which a fresh coverage monitor first alarms, or None."""
    monitor = LocalMonitor(type="coverage", target=target, false_alarm_rate=false_alarm_rate)
    for t, ok in enumerate(covered, start=1):
        if monitor.update(bool(ok)):
            return t
    return None
 
 
def naive_first_alarm(covered, target=0.88, level=0.05, start=30):
    """A one-sided z-test on the running miss rate, re-run after every outcome."""
    misses = 1.0 - np.asarray(covered, dtype=float)
    t = np.arange(1, len(misses) + 1)
    null = 1.0 - target
    z = (np.cumsum(misses) / t - null) / np.sqrt(null * (1 - null) / t)
    hits = np.flatnonzero(z[start - 1:] > normal_ppf(1 - level))
    return int(hits[0]) + start if hits.size else None
 
 
rng = np.random.default_rng(7)
 
# 1. No change, coverage sitting exactly at the monitor's target: how often does anything alarm?
streams = [rng.uniform(size=3000) > 0.12 for _ in range(400)]
print("e-process false alarms: ", np.mean([first_alarm(s) is not None for s in streams]))
print("naive test false alarms:", np.mean([naive_first_alarm(s) is not None for s in streams]))
 
# 2. A silent update after 1,000 outcomes drops coverage from 90% to 80%.
delays = []
for _ in range(200):
    stream = np.concatenate([rng.uniform(size=1000) > 0.10, rng.uniform(size=5000) > 0.20])
    alarm = first_alarm(stream)
    if alarm is not None and alarm > 1000:
        delays.append(alarm - 1000)
print("alarms after the update:", len(delays), "of 200; median delay:", int(np.median(delays)),
      "outcomes; 90th percentile:", int(np.percentile(delays, 90)))
 
# 3. The same drop seen by a monitor started fresh at the update.
fresh = [first_alarm(rng.uniform(size=5000) > 0.20) for _ in range(200)]
print("fresh monitor median delay:", int(np.median([d for d in fresh if d is not None])), "outcomes")

e-process false alarms:  0.0225
naive test false alarms: 0.385
alarms after the update: 196 of 200; median delay: 1750 outcomes; 90th percentile: 2450
fresh monitor median delay: 143 outcomes

from cli_sdk import CalibrationExample, CLIClient
 
with CLIClient() as client:
    for alert in client.calibration_profiles.monitors.poll(PROFILE):
        pause_auto_routing(PROFILE)                     # every ticket goes to humans for now
        notify_oncall(alert.type, alert.message, e_value=alert.e_value, n=alert.n)
 
        audit = client.calibration_profiles.audit(PROFILE, fresh_examples=fresh_labelled_tickets)
        if not audit.passed:
            client.calibration_profiles.create(
                name="support-routing-v4",
                backend=current_backend,                # the model that is actually serving now
                method="APS",
                alpha=0.10,
            )
            client.calibration_profiles.add_examples(
                "support-routing-v4",
                examples=[CalibrationExample(context=c, label=y) for c, y in relabelled_sample],
            )
            # then switch queries to support-routing-v4 and create fresh monitors on it

Walkthrough

The fingerprint monitor is the first line. If the vendor’s response metadata changes — a new model version string, a different deployment version on Azure — the fingerprint monitor alerts on the first request, before a single mislabelled ticket accumulates. Pin dated snapshots and upgrade policies (see Backends) so that fewer changes are silent in the first place.

The coverage monitor catches what metadata does not. It labels 2% of traffic and bets, outcome by outcome, against “coverage is still at least the floor”. Its target is the lower end of the profile’s realized_coverage_ci, so it tests for drift rather than for the ordinary scatter of one calibration draw.

Why continuous checking is safe. In the simulation, with coverage sitting exactly at the monitor’s target and no change at all, the e-process alarmed on 2.25% of 400 streams — under its 5% budget — even though it was checked after every one of 3,000 outcomes. Re-running a fixed-sample z-test after every outcome alarmed on 38.5% of the same kind of streams. Ville’s inequality is what makes the difference: the 5% holds over the whole stream, not at one pre-chosen sample size.

Detection is not instant, and history matters. After a drop from 90% to 80%, a monitor started fresh at the update needed a median of 143 labelled outcomes. One that had already watched 1,000 in-spec outcomes needed a median of 1,750: while coverage is healthy, its bets lose and its e-value shrinks, and it has to win that back after the change. At a 2% labelling rate, 1,750 labelled outcomes is about 87,500 tickets. Two practices keep the delay down without spending the false-alarm budget:

• Start a new monitor at every profile version and after every known change (a deploy, a vendor notice). The start time must be chosen without looking at the monitored outcomes.
• For long-lived profiles, run consecutive monitors over fixed windows (for example, a new monitor every 2,000 labelled outcomes). Each window spends its own false_alarm_rate, so over $K$ windows the chance of any false alarm is at most $K$ times the per-window rate; set the per-window rate from the budget you want for the year.

Responding. Pausing auto-routing is always safe: escalated tickets carry no guarantee claim at all. The audit on fresh labels confirms whether coverage really dropped; if so, the fix is a new profile version calibrated against the model that is actually serving, with new monitors.

What the guarantee means

A monitor’s guarantee is of type anytime:

• If coverage stays at or above the monitor’s target, the probability that the monitor ever alarms is at most false_alarm_rate (5%), however long it runs and however often it is polled.
• When it does alarm, the Alert carries e_value (at least 20 at a 5% rate) and n, the number of outcomes seen. $1/e$ is the smallest false-alarm rate at which this evidence would still have alarmed, and can be reported after the fact (post-hoc levels).
• The guarantee is about the monitor’s false alarms. It does not promise any particular detection delay; the simulation above shows what to expect.