Label-efficient calibration

Problem

A team needs to know how often its support assistant’s answers are correct, with an interval they can defend, and needs a calibration profile for it. Expert review costs a few dollars per item and they have 20,000 unreviewed conversations. A strong LLM judge can grade all 20,000 cheaply, but its grades are biased in ways nobody has measured.

Averaging the judge’s grades gives a precise-looking number with an unknown bias. Using only human labels is unbiased but needs thousands of them for a tight interval.

Primitives

• calibration.judge_quality and calibration.estimate_rate_with_judge: local, network-free prediction-powered inference, to decide how many human labels are worth buying.
• calibration_profiles.label_with_judge: the hosted job that labels a pool with a judge, selects a random subset for humans, and combines them validly.
• Monitors with use_judge_pseudo_labels=True, to keep watching with a small trickle of human labels.

Code

import numpy as np
from cli_sdk.calibration import estimate_rate_with_judge, judge_quality
from cli_sdk.stats.evalues.anytime import normal_ppf
 
 
def load_items(seed: int = 0):
    """Stand-in data. Replace with your own: for every item, the judge's 0/1 verdict
    ("the assistant's answer was correct"), plus the human verdict on a random subset."""
    rng = np.random.default_rng(seed)
    n_items = 20_300
    truth = rng.uniform(size=n_items) < 0.83                         # unknown in real life
    judge = np.where(rng.uniform(size=n_items) < 0.90, truth, ~truth)  # judge agrees 90% of the time
    return truth.astype(float), judge.astype(float)
 
 
truth, judge = load_items()
rng = np.random.default_rng(1)
human_idx = rng.choice(len(judge), size=300, replace=False)          # uniformly random subset
pool_idx = np.setdiff1d(np.arange(len(judge)), human_idx)
 
human = truth[human_idx]                  # the 300 labels you pay for
judge_on_human = judge[human_idx]         # the judge on the same 300 items
judge_on_pool = judge[pool_idx]           # the judge on the other 20,000
 
quality = judge_quality(human, judge_on_human)
print(f"agreement {quality.agreement_with_humans:.3f}  correlation {quality.correlation:.3f}  "
      f"label multiplier {quality.effective_label_multiplier:.2f}")
 
ppi = estimate_rate_with_judge(human, judge_on_human, judge_on_pool, alpha=0.05)
print(f"prediction-powered: {ppi.estimate:.3f}  95% CI [{ppi.ci_lower:.3f}, {ppi.ci_upper:.3f}]  lambda {ppi.lam:.2f}")
 
se = human.std(ddof=1) / np.sqrt(len(human))
z = normal_ppf(0.975)
print(f"humans only:        {human.mean():.3f}  95% CI [{human.mean() - z * se:.3f}, {human.mean() + z * se:.3f}]")
print(f"judge only (biased): {judge_on_pool.mean():.3f}")

agreement 0.910  correlation 0.723  label multiplier 2.10
prediction-powered: 0.828  95% CI [0.799, 0.858]  lambda 0.65
humans only:        0.837  95% CI [0.795, 0.879]
judge only (biased): 0.767

from cli_sdk import CalibrationExample, CLIClient, OpenAIBackend
 
PROFILE = "assistant-correctness-v1"
 
with CLIClient() as client:
    client.calibration_profiles.create(
        name=PROFILE,
        backend=OpenAIBackend(model="gpt-4.1-mini"),
        method="IVAP",
        alpha=0.10,
    )
    job = client.calibration_profiles.label_with_judge(
        PROFILE,
        judge=OpenAIBackend(model="gpt-4.1-2025-04-14"),
        unlabelled_examples=[{"context": conversation} for conversation in conversations],
        human_labelled_sample_size=300,
    )
    print(job)    # the human-labelling task list and a judge-quality diagnostic
 
    # After reviewers finish the selected items:
    client.calibration_profiles.add_examples(
        PROFILE,
        examples=[
            CalibrationExample(context=item_context, label=reviewer_verdict, source="human")
            for item_context, reviewer_verdict in completed_reviews
        ],
    )

with CLIClient() as client:
    client.calibration_profiles.monitors.create(
        PROFILE,
        type="risk",
        target=0.20,
        false_alarm_rate=0.05,
        labelled_sample_rate=0.005,
        use_judge_pseudo_labels=True,
    )

Walkthrough

Diagnose before you pay. On the 300 items both humans and the judge labelled, the judge agrees with humans 91% of the time, with correlation 0.72. effective_label_multiplier is $1/(1-\rho^2) \approx 2.1$: with a large judge-labelled pool, each human label is worth about two human-only labels. A judge with correlation 0.3 would be worth about 1.1, and buying more human labels would be the better use of money.

The judge alone is wrong in a way you cannot see. The judge’s own average over the pool is 0.767, well below the true rate of about 0.83 in this stand-in data; nothing in the judge’s grades reveals that. The prediction-powered estimate uses the 300 human labels to measure and remove the judge’s bias (the rectifier term), then uses the 20,000 judge grades to shrink the variance. Its interval, $[0.799, 0.858]$, is about 30% narrower than the human-only interval $[0.795, 0.879]$ from the same 300 labels: roughly what twice as many human labels alone would give.

The hosted job does the same at profile scale. label_with_judge labels every example with the judge, selects a random subset for human review, and combines the two with the same prediction-powered estimator: unbiased whatever the judge’s quality, with a large-sample interval. Upload the human verdicts with source="human" so the profile records who produced each label; the finite-sample conformal guarantees on the profile’s answers rest on those human-labelled examples.

Monitoring stays cheap. A risk monitor with judge pseudo-labels needs only 0.5% of traffic reviewed by humans, instead of a full labelling pipeline.

What the guarantee means

• The estimate. $[0.799, 0.858]$ is a large-sample 95% confidence interval for the correct-answer rate over the population the 20,300 items were drawn from. It is valid whatever the judge’s quality — a worse judge gives a wider interval, not an invalid one — but it relies on a normal approximation, so it is not a finite-sample or anytime-valid interval.
• lam (0.65 here) is the weight the estimator put on the judge. It is chosen from the data to minimize variance; 0 would ignore the judge entirely.
• The profile’s cards carry the usual finite-sample guarantee for the profile’s method, computed on the human-labelled examples, with the same exchangeability assumption: the calibration pool, the human subset, and production traffic must all come from the same distribution. Estimates that use the judge pool (PPI) are large-sample, not finite-sample.