Support routing with an FDR gate

Problem

A support team wants to auto-route tickets to billing, technical, or sales without a human in the loop, but only if the fraction of auto-routed tickets that land in the wrong queue stays under 5% — a number the ops lead has to defend in a quarterly review.

A hand-picked “route automatically above 0.6 confidence” rule makes no statement about that fraction. The threshold was chosen by eyeballing a few examples, not calibrated against a target error rate, and nobody knows what happens to it after the model changes.

Primitives

Set with a coverage guarantee: the plausible teams for a ticket, shown to a human when the ticket is escalated.
Gate with guarantee="fdr", target=0.05, delta=0.05: the auto-route-or-escalate decision on the queue the Set proposes, calibrated on the Set’s own proposals so that, with 95% confidence, at most 5% of auto-routed tickets go to the wrong team.
Drift monitors on the profile, so a change in traffic or model is caught between reviews.

Code

Create and fill the calibration profile

Calibrate on a random sample of historical tickets with the team a human finally assigned, not a curated set of easy or hard cases.

import json
 
from cli_sdk import AzureOpenAIBackend, CalibrationExample, CLIClient, Gate, Set
 
PROFILE = "support-routing-v3"          # the Set: which team
GATE_PROFILE = "support-auto-route-v3"  # the Gate: may this proposed team be used without review?
BACKEND = AzureOpenAIBackend(model="gpt-4.1", deployment="support-gpt41")
 
DEPARTMENTS = {
    "billing": "Payments, invoicing, refunds",
    "technical": "Bugs, outages, integrations",
    "sales": "Pricing, upgrades, new accounts",
}
DEPARTMENT = Set(
    instructions="Which team should handle this ticket?",
    options=DEPARTMENTS,
    calibration_profile=PROFILE,
    alpha=0.10,
)
AUTO_ROUTE = Gate(
    instructions="Is proposed_queue the correct team for this ticket?",
    calibration_profile=GATE_PROFILE,
    guarantee="fdr",
    target=0.05,
    delta=0.05,
)
 
with CLIClient() as client:
    client.calibration_profiles.create(
        name=PROFILE,
        backend=BACKEND,
        method="APS",
        alpha=0.10,
    )
 
    with open("labelled_tickets.jsonl", encoding="utf-8") as handle:
        rows = [json.loads(line) for line in handle if line.strip()]
 
    profile = client.calibration_profiles.add_examples(
        PROFILE,
        examples=[
            CalibrationExample(context={"ticket": row["ticket"]}, label=row["team"])
            for row in rows
        ],
    )
 
    print(profile.n)                     # 1204
    print(profile.realized_coverage_ci)  # (0.886, 0.914)
    print(profile.can_serve_guarantees)  # True

Calibrate the gate on the Set’s own proposals

The gate judges one exact action: this ticket, sent to the team the Set proposes. Its calibration examples are built the same way, so its guarantee covers the queue the code actually routes to.

with CLIClient() as client:
    client.calibration_profiles.create(name=GATE_PROFILE, backend=BACKEND, method="LTT", alpha=0.05)
    gate_examples = []
    for row in rows:
        department = client.evaluate(context={"ticket": row["ticket"]}, backend=BACKEND,
                                     queries={"department": DEPARTMENT}).answers["department"]
        gate_examples.append(CalibrationExample(
            context={"ticket": row["ticket"], "proposed_queue": department.top},
            label=department.top == row["team"],   # True: auto-routing this proposal is right
        ))
    client.calibration_profiles.add_examples(GATE_PROFILE, examples=gate_examples)

Route each ticket

def triage(client: CLIClient, ticket: str) -> dict:
    department = client.evaluate(context={"ticket": ticket}, backend=BACKEND,
                                 queries={"department": DEPARTMENT}).answers["department"]
    if department.top is not None:
        route = client.evaluate(context={"ticket": ticket, "proposed_queue": department.top},
                                backend=BACKEND, queries={"route": AUTO_ROUTE}).answers["route"]
        if route.approved and not route.is_heuristic:
            return {"action": "auto", "queue": department.top}
    return {"action": "human", "candidates": department.set}
 
with CLIClient() as client:
    print(triage(client, "I was charged twice for order A-104."))
    # {'action': 'auto', 'queue': 'billing'}
    print(triage(client, "Can I get a discount if I add 40 seats and fix the SSO bug?"))
    # {'action': 'human', 'candidates': ['sales', 'technical']}

Watch the profile

with CLIClient() as client:
    monitors = client.calibration_profiles.monitors
    monitors.create(PROFILE, type="fingerprint")
    monitors.create(GATE_PROFILE, type="fingerprint")
    monitors.create(PROFILE, type="coverage", target=0.88, false_alarm_rate=0.05,
                    labelled_sample_rate=0.02)
 
    for alert in monitors.poll(PROFILE):
        pause_auto_routing()
        page_ops_lead(alert.type, alert.message, alert.e_value)

The wrong-route rate among auto-routed tickets can also be watched in your own process, from the ticketing system’s outcomes:

from cli_sdk import LocalMonitor
 
wrong_routes = LocalMonitor(type="risk", target=0.05, false_alarm_rate=0.05, profile=GATE_PROFILE)
 
def on_ticket_closed(auto_routed: bool, reassigned_by_agent: bool) -> None:
    if not auto_routed:
        return
    alert = wrong_routes.update(reassigned_by_agent)   # is_loss: the auto-route was wrong
    if alert:
        pause_auto_routing()
        page_ops_lead(alert.type, f"wrong-route rate above 5% (n={alert.n})", alert.e_value)

Walkthrough

The ops lead creates support-routing-v3 from 1,204 randomly sampled, human-labelled tickets — above the roughly 1,000 examples recommended for a stable $\alpha = 0.10$ guarantee — and the profile reports a realized-coverage interval of $[0.886, 0.914]$ .

Every ticket takes two evaluate calls. The Set proposes a queue (department.top) and supplies the shortlist a human sees on escalation; the Gate then judges that exact proposal. Because the gate was calibrated on the Set’s own proposals, its guarantee covers the queue the code routes to. A gate asked a generic “auto-route this ticket?” question would bound something else, and nothing would tie its answer to department.top. A human choosing between two teams is faster than one reading a ticket cold.

The routing code reads route.is_heuristic before acting: if the profile ever drops below its minimum size (for example after a version bump), the gate answer is labelled heuristic and every ticket goes to a human rather than being auto-routed without a guarantee. For a hard failure instead, construct the client with strict_guarantees=True (Errors and retries).

The monitors replace the quarterly surprise. The fingerprint monitor fires the moment the deployment’s reported model version changes. The coverage monitor labels 2% of traffic and alarms the first time there is strong evidence that coverage has fallen below 88%. The local wrong-route monitor alarms when the error rate among auto-routed tickets has risen above 5%. The two e-process monitors keep their false-alarm rate at 5% however often they are checked. The coverage monitor’s target sits below the nominal 90% (0.88 against a 0.886 lower bound), because the Set’s coverage is an average over calibration draws and this particular draw can sit a little under it. The gate’s bound is different: it already holds for this calibration with 95% confidence, so its monitor runs at the 5% target itself. Both then report drift, not the ordinary scatter of one calibration draw.

“Reassigned by an agent” is only a valid loss signal if agents reliably reassign every misrouted ticket. If they do not, feed the monitor from a small random sample of auto-routed tickets that a human re-checks.

What the guarantee card means

Inside triage, the gate’s card reads:

card = route.guarantee
card.type                  # "fdr"
card.method                # "LTT"
card.target                # 0.05
card.delta                 # 0.05
card.realized_upper_bound  # 0.047
card.describe()
# With 95% confidence, the wrong fraction among approved decisions is at most 0.05 on profile 'support-auto-route-v3' (n=1204).

Gate, fdr, target 0.05, delta 0.05. One ticket arrives per request, so the threshold was fixed at calibration by Learn-then-Test. With probability at least 95% over the calibration draw, at most 5% of the tickets the gate auto-routes are sent to the wrong team, over tickets exchangeable with the 1,204 calibration tickets. realized_upper_bound is the bound the calibration achieved at the selected threshold. The card does not say that any one auto-routed ticket is 95% likely to be right, and it says nothing about escalated tickets.
Set, coverage, alpha 0.10. Sets built this way contain the right team at least 90% of the time over exchangeable tickets. coverage_ci $[0.886, 0.914]$ is how far this particular calibration’s realized coverage can plausibly sit from that.
Both hold only while production tickets remain exchangeable with the calibration tickets — the assumption the monitors are there to check.

Overview RAG claim filtering