Route

A calibrated cascade across model backends — a cheap or self-hosted model first, escalating to a stronger or more expensive one only when needed — with either a cost guarantee or an accuracy guarantee, instead of a static “always call the expensive model” or “always call the cheap one” rule.

Request

from cli_sdk import CLIClient, OpenAIBackend, Route, VLLMBackend
 
with CLIClient() as client:
    result = client.evaluate(
        context={"query": user_query},
        queries={
            "answer": Route(
                cascade=[
                    {"backend": VLLMBackend(model="meta-llama/Llama-3.3-70B-Instruct",
                                            base_url="http://internal-vllm:8000")},
                    {"backend": OpenAIBackend(model="gpt-4.1-2025-04-14")},
                ],
                calibration_profile="cost-routing-v1",
                guarantee="cost_budget",
                target_cents=0.4,
                alpha=0.10,
            ),
        },
    )
 
answer = result.answers["answer"]
print(answer.output)
print(answer.served_by, answer.escalated, answer.cost_cents)

Route does not need a backend argument: its backends come from cascade. The same holds for a Judge that declares a cascade. A request whose queries all declare their own cascade can omit backend; any other query in the same request still needs one.

Response

{
  "answer": {
    "type": "route",
    "output": "You can export invoices from Settings, then Billing.",
    "served_by": { "provider": "vllm", "model": "meta-llama/Llama-3.3-70B-Instruct" },
    "escalated": false,
    "cost_cents": 0.03,
    "guarantee": {
      "type": "cost_budget",
      "target_cents": 0.4,
      "alpha": 0.10,
      "method": "calibrated-cascade",
      "calibration_profile": "cost-routing-v1",
      "calibration_n": 900
    }
  }
}

Guarantee modes

`guarantee`	Statement	Card
`"cost_budget"`	$P(\text{cost per request} \le \text{target\_cents}) \ge 1-\alpha$	`type: "cost_budget"` with `target_cents` and `alpha`
`"accuracy"`	$P(\text{served answer is wrong}) \le \alpha$ , escalating whenever the cheap stage’s Venn-Abers interval is too wide to support that bound	`type: "risk"` with `target` equal to $\alpha$

Both are marginal over requests exchangeable with the calibration set. A cost budget does not cap the cost of any single request: up to an $\alpha$ fraction of requests can exceed target_cents.

Calibrating a cascade

A Route profile is built from historical queries with every stage’s output and the correct answer recorded, so CLI can learn a calibrated per-query escalation rule rather than a single global confidence threshold. Add examples the same way as any other profile:

client.calibration_profiles.add_examples(
    "cost-routing-v1",
    examples=[
        {
            "context": {"query": q},
            "tier_outputs": {"cheap": cheap_out, "strong": strong_out},
            "label": gold_answer,
        }
        for q, cheap_out, strong_out, gold_answer in historical_data
    ],
)

See the cost routing guide for an end-to-end setup, including a spend monitor.

Answer attributes

Attribute	Type	Meaning
`output`	`Any`	The answer from the stage that served the request
`served_by`	`dict`	Provider and model of that stage
`escalated`	`bool`	Whether the request moved past the first stage
`cost_cents`	`float \| None`	Cost of this request
`guarantee`, `is_heuristic`		The guarantee card, and whether it is heuristic

Parameters

Parameter	Type	Description
`cascade`	`list[dict]`	Required, at least one stage. Each stage is `{"backend": ...}`, cheapest first.
`calibration_profile`	`str`	Required.
`guarantee`	`"cost_budget" \| "accuracy"`	Default `"cost_budget"`.
`target_cents`	`float`	Required for `"cost_budget"`. Per-request cost target.
`alpha`	`float`, optional	Allowed violation rate.

Judge Overview