Interval

Rate the context along an ordinal or continuous rubric, and get back a conformalized interval with a coverage guarantee, alongside a point estimate — instead of a single probability-weighted score with no stated error bound.

Request

from cli_sdk import AzureOpenAIBackend, CLIClient, Interval
 
with CLIClient() as client:
    result = client.evaluate(
        context={"ticket": "This is the third time this has happened. Fix it."},
        backend=AzureOpenAIBackend(model="gpt-4.1", deployment="support-gpt41"),
        queries={
            "frustration": Interval(
                instructions="How frustrated is the customer?",
                levels=["Calm", "Frustrated", "Very angry"],
                calibration_profile="frustration-rubric-v1",
                alpha=0.10,
            ),
        },
    )
 
frustration = result.answers["frustration"]
print(frustration.interval)                 # (1.0, 2.0)
print(frustration.point_estimate)           # 1.6
print(frustration.contains(2))              # True
print(frustration.legend[str(int(frustration.interval[1]))])   # 'Very angry'

Response

{
  "frustration": {
    "type": "interval",
    "point_estimate": 1.6,
    "interval": [1.0, 2.0],
    "legend": { "0": "Calm", "1": "Frustrated", "2": "Very angry" },
    "guarantee": {
      "type": "coverage",
      "alpha": 0.10,
      "method": "CQR",
      "calibration_profile": "frustration-rubric-v1",
      "calibration_n": 640
    }
  }
}

Levels are indexed from 0 in the order you list them; legend maps each index back to its name.

How it differs from a plain probability-weighted score

A probability-weighted average across rubric levels can land between levels with no stated error bound on where the true rating falls. Interval instead conformalizes a regression or ordinal score (CQR, conformalized quantile regression, or an ordinal-APS variant for discrete rubrics), so intervals produced this way contain the true rating at least $1-\alpha$ of the time over exchangeable inputs, calibrated on your own labelled examples. The offline equivalent is cli_sdk.stats.conformal.cqr.

Answer attributes

Attribute	Type	Meaning
`interval`	`tuple[float, float]`	The conformalized interval, in level indices
`point_estimate`	`float`	The point estimate
`legend`	`dict[str, str]`	Level index (as a string) to level name
`width`	`float`	`interval[1] - interval[0]`
`contains(value)`	`bool`	Whether `value` lies in the closed interval
`guarantee`, `is_heuristic`		The guarantee card, and whether it is heuristic

Parameters

Parameter	Type	Description
`instructions`	`str \| dict \| list`	Required. What to rate.
`levels`	`list[str]`	Required. Ordered rubric levels, 2 to 10 entries.
`calibration_profile`	`str`	Required.
`alpha`	`float`, optional	Target miscoverage rate.
`method`	`"CQR" \| "ordinal-aps"`, optional	Default chosen from the profile’s data type.

Set Gate