Cost routing

Problem

A team serves most requests from a self-hosted open-weight model and wants to pay for a frontier API call only when the cheap model is genuinely unsure. Finance wants a guarantee on the resulting cost per request, not “usually cheap”.

A static rule (“escalate below 70% confidence”) has no cost story: nobody can say how often it will escalate next month, or what that does to the bill.

Primitives

• Route with guarantee="cost_budget": a calibrated cascade with $P(\text{cost per request} \le \text{target\_cents}) \ge 1-\alpha$.
• LocalMonitor on the over-budget rate, to catch drift in the traffic mix.

Code

from cli_sdk import CLIClient, OpenAIBackend, VLLMBackend
 
PROFILE = "cost-routing-v1"
CHEAP = VLLMBackend(
    model="meta-llama/Llama-3.3-70B-Instruct",
    base_url="http://internal-vllm:8000",
    engine_version="0.11.0",
    quantization="fp8",
)
STRONG = OpenAIBackend(model="gpt-4.1-2025-04-14")
 
with CLIClient() as client:
    client.calibration_profiles.create(
        name=PROFILE,
        backend=CHEAP,
        method="calibrated-cascade",
        alpha=0.10,
    )
    client.calibration_profiles.add_examples(
        PROFILE,
        examples=[
            {
                "context": {"query": q},
                "tier_outputs": {"cheap": cheap_out, "strong": strong_out},
                "label": gold_answer,
            }
            for q, cheap_out, strong_out, gold_answer in historical_data   # about 900 queries
        ],
    )

from cli_sdk import CLIClient, EvaluateResponse, LocalMonitor, Route, RouteAnswer
 
TARGET_CENTS = 0.4
 
class Routed(EvaluateResponse):
    answer: RouteAnswer
 
ROUTE = Route(
    cascade=[{"backend": CHEAP}, {"backend": STRONG}],
    calibration_profile=PROFILE,
    guarantee="cost_budget",
    target_cents=TARGET_CENTS,
    alpha=0.10,
)
 
over_budget = LocalMonitor(type="risk", target=0.12, false_alarm_rate=0.05, profile=PROFILE)
 
def serve(client: CLIClient, user_query: str):
    result = client.evaluate(context={"query": user_query}, queries={"answer": ROUTE},
                             response_model=Routed)
    answer = result.answer
    record_cost(answer.served_by, answer.cost_cents, escalated=answer.escalated)
 
    alert = over_budget.update(answer.cost_cents is not None and answer.cost_cents > TARGET_CENTS)
    if alert:
        notify_finance(f"over-budget rate above 12% after {alert.n} requests")
    return answer.output
 
with CLIClient() as client:
    print(serve(client, "How do I export invoices?"))

Walkthrough

Route needs no backend argument: the cascade names its own backends, cheapest first. For each request, the self-hosted model answers first. Because vLLM reaches access levels up to L4, CLI can score the cheap stage’s answer from exact token probabilities, which is inexpensive. When that score clears the calibrated escalation threshold, the request is served by vLLM (escalated=False, cost_cents around 0.03). Otherwise it is escalated to the OpenAI stage.

The escalation thresholds are learned from the calibration set so that at most an $\alpha = 10\%$ fraction of requests is expected to exceed 0.4 cents. Nothing caps the cost of one particular request; the bound is on how often requests exceed the target.

The local monitor turns that bound into an alarm. Each request is a 0/1 outcome, “over budget”, and the monitor tests whether their rate has risen above 12% (a margin above the 10% the profile allows, so it reports drift rather than the scatter of one calibration). If the traffic mix shifts toward questions the cheap model cannot handle, escalations rise, the rate climbs, and the monitor fires with its false-alarm rate still at 5%.

What the guarantee card means

card = result.answer.guarantee
card.type            # "cost_budget"
card.target_cents    # 0.4
card.alpha           # 0.1
card.describe()
# Cost per request at most 0.4 cents with 90% probability on profile 'cost-routing-v1' (n=900).

• Over requests exchangeable with the 900 calibration queries, at least 90% cost at most 0.4 cents. Up to 10% can cost more, including the full price of the strong stage.
• The card says nothing about answer quality. To bound how often the served answer is wrong instead, use guarantee="accuracy", whose card is a risk card with target equal to alpha.
• Provider price changes change the cost of each stage, and so the meaning of the calibrated thresholds. Recalibrate when prices change.