LLM-judge cascade

Problem

An eval team compares two model versions on thousands of prompts a day with an LLM judge. They need a number for “how often does the judge agree with our human raters” that survives an audit, and they do not want to pay for the strongest judge on every comparison.

An uncalibrated judge gives neither: its agreement rate is measured once, on a convenience sample, and there is no rule for when it should defer.

Primitives

• Judge with a three-stage cascade: a small judge, a large judge, then a human queue. Each stage’s verdict is accepted only when its Venn-Abers interval clears that stage’s calibrated threshold.
• The guarantee: with probability at least $1-\delta$ over the calibration draw, human agreement on non-escalated verdicts is at least $1-\alpha$.

Code

from cli_sdk import CalibrationExample, CLIClient, OpenAIBackend
 
PROFILE = "pairwise-judge-v2"
SMALL = OpenAIBackend(model="gpt-4.1-mini")
LARGE = OpenAIBackend(model="gpt-4.1")
 
with CLIClient() as client:
    client.calibration_profiles.create(
        name=PROFILE,
        backend=SMALL,
        method="trust-or-escalate",
        alpha=0.10,
    )
    client.calibration_profiles.add_examples(
        PROFILE,
        examples=[
            CalibrationExample(
                context={"prompt": row["prompt"], "response_a": row["a"], "response_b": row["b"]},
                label=row["human_choice"],        # "response_a" or "response_b"
                source="human",
            )
            for row in human_preferences          # about 1,800 randomly sampled comparisons
        ],
    )

from collections import Counter
 
from cli_sdk import CLIClient, EvaluateResponse, Judge, JudgeAnswer
 
class Verdict(EvaluateResponse):
    verdict: JudgeAnswer
 
JUDGE = Judge(
    instructions="Which response better follows the prompt? Answer response_a or response_b.",
    calibration_profile=PROFILE,
    alpha=0.10,
    cascade=[
        {"backend": SMALL},
        {"backend": LARGE},
        {"backend": "human_queue"},
    ],
)
 
def run_eval(client: CLIClient, comparisons: list[dict]) -> dict:
    wins, resolved_by, pending = Counter(), Counter(), []
    for item in comparisons:
        result = client.evaluate(
            context={"prompt": item["prompt"], "response_a": item["a"], "response_b": item["b"]},
            backend=SMALL,
            queries={"verdict": JUDGE},
            response_model=Verdict,
        )
        verdict = result.verdict
        if verdict.needs_human:
            pending.append(item["id"])
            continue
        wins[verdict.winner] += 1
        resolved_by[verdict.escalated_to or "first stage"] += 1
    return {"wins": dict(wins), "resolved_by": dict(resolved_by), "pending_human": pending}
 
with CLIClient() as client:
    summary = run_eval(client, todays_comparisons)
    print(summary["resolved_by"])   # e.g. {'first stage': 8412, 'gpt-4.1': 1106}
    print(len(summary["pending_human"]))

Walkthrough

pairwise-judge-v2 is calibrated on about 1,800 comparisons with human preference labels, drawn at random from the same prompt distribution the eval runs on. The labels must be human: the guarantee is about agreement with humans, so judge labels cannot stand in for them here (see the label-efficient calibration guide for how judge labels can still reduce the number of human labels when the target is a rate rather than a per-verdict guarantee).

For each comparison, the small judge answers first. If its Venn-Abers interval clears the calibrated threshold, its verdict is returned with escalated_to=None. If the interval straddles the threshold, the large judge runs; if that one is also unsure, the comparison lands in the human queue (needs_human). The expensive stages run only on the comparisons the cheap stage could not resolve, which is where the cost saving comes from; resolved_by shows the split.

What the guarantee card means

{
  "type": "risk_high_probability",
  "statement": "human agreement >= 0.90 on non-escalated verdicts",
  "target": 0.10,
  "delta": 0.05,
  "method": "trust-or-escalate",
  "calibration_profile": "pairwise-judge-v2",
  "calibration_n": 1800
}

• With probability at least 95% over the draw of the 1,800 calibration comparisons, the rate at which non-escalated verdicts disagree with human raters is at most 10%, over comparisons exchangeable with the calibration set.
• It is not a statement that any single verdict is 90% likely to match a human, and it says nothing about comparisons that were escalated.
• “Human agreement” means agreement with the raters who produced the calibration labels, following their rubric. A new rubric or rater pool is a new profile.
• A new candidate model under evaluation can shift the comparison distribution. Audit the profile on a fresh human-labelled sample when the models being compared change substantially.