Judge

An LLM-as-judge verdict with a calibrated guarantee on agreement with human raters, and an optional escalation cascade so an expensive judge only runs on genuinely ambiguous comparisons.

Request

from cli_sdk import CLIClient, Judge, OpenAIBackend
 
with CLIClient() as client:
    result = client.evaluate(
        context={"prompt": prompt, "response_a": a, "response_b": b},
        backend=OpenAIBackend(model="gpt-4.1-mini"),
        queries={
            "verdict": Judge(
                instructions="Which response better follows the prompt?",
                calibration_profile="pairwise-judge-v2",
                alpha=0.10,
                cascade=[
                    {"backend": OpenAIBackend(model="gpt-4.1-mini")},
                    {"backend": OpenAIBackend(model="gpt-4.1")},
                    {"backend": "human_queue"},
                ],
            ),
        },
    )
 
verdict = result.answers["verdict"]
if verdict.needs_human:
    enqueue_for_review(prompt, a, b)
else:
    record(verdict.winner)       # "response_a" or "response_b"

A cascade stage’s backend can be a backend object (as above), a plain dict such as {"provider": "openai", "model": "gpt-4.1"}, or the string "human_queue".

Response

{
  "verdict": {
    "type": "judge",
    "winner": "response_a",
    "escalated_to": null,
    "guarantee": {
      "type": "risk_high_probability",
      "statement": "human agreement >= 0.90 on non-escalated verdicts",
      "target": 0.10,
      "delta": 0.05,
      "method": "trust-or-escalate",
      "calibration_profile": "pairwise-judge-v2",
      "calibration_n": 1800
    }
  }
}

How the cascade works

Each stage in cascade computes a Venn-Abers interval for its verdict. If the interval clears the calibrated threshold for that stage, the verdict is returned with escalated_to: null when the first stage resolved it, or naming the stage that did. If the interval straddles the threshold, the next stage runs — up to and including a human review queue, in which case escalated_to is "human_queue" and needs_human is True. This spends the expensive stage only on the fraction of comparisons the cheap stage could not resolve.

The guarantee is about non-escalated verdicts: with probability at least $1-\delta$ over the calibration draw, the rate at which those verdicts disagree with human raters is at most target, over comparisons exchangeable with the calibration set. It says nothing about any one comparison, and nothing about comparisons sent to the human queue.

Building the calibration profile

A Judge profile is calibrated on comparisons with human preference labels — not judge labels, since the guarantee is specifically about agreement with humans:

from cli_sdk import CalibrationExample, CLIClient, OpenAIBackend
 
with CLIClient() as client:
    client.calibration_profiles.create(
        name="pairwise-judge-v2",
        backend=OpenAIBackend(model="gpt-4.1-mini"),
        method="trust-or-escalate",
        alpha=0.10,
    )
    client.calibration_profiles.add_examples(
        "pairwise-judge-v2",
        examples=[
            CalibrationExample(
                context={"prompt": p, "response_a": ra, "response_b": rb},
                label=human_choice,          # "response_a" or "response_b"
                source="human",
            )
            for p, ra, rb, human_choice in human_preferences
        ],
    )

See label-efficient calibration for combining a modest human-labelled sample with a larger pool that a judge labels first, and the judge cascade guide for an eval pipeline built on this.

Answer attributes

Attribute	Type	Meaning
`winner`	`str \| None`	The verdict, for example `"response_a"`; `None` while pending human review
`escalated_to`	`str \| None`	`None` if the first stage resolved it; otherwise the stage that did, or `"human_queue"`
`needs_human`	`bool`	`escalated_to == "human_queue"`
`guarantee`, `is_heuristic`		The guarantee card, and whether it is heuristic

Parameters

Parameter	Type	Description
`instructions`	`str`	Required. The comparison or verdict to produce.
`calibration_profile`	`str`	Required; calibrated against human labels.
`alpha`	`float`, optional	Target: human agreement $\ge 1-\alpha$ on non-escalated verdicts.
`cascade`	`list[dict]`, optional	Ordered stages, each `{"backend": ...}`, cheapest first; end with `"human_queue"` to guarantee a resolution.

Claim Route