Set

Choose among a fixed list of options, and get back a set of options that contains the correct one at least $1-\alpha$ of the time over exchangeable traffic — not a single best guess with a heuristic confidence score.

When to use it

Any classification decision: routing, intent detection, categorization, triage. Reach for Set wherever you would previously have used a top-1 prediction plus a confidence threshold.

Request

from cli_sdk import CLIClient, OpenAIBackend, Set
 
with CLIClient() as client:
    result = client.evaluate(
        context={
            "ticket": "I upgraded to the annual plan but was billed monthly.",
            "account_tier": "enterprise",
        },
        backend=OpenAIBackend(model="gpt-4.1-2025-04-14"),
        queries={
            "department": Set(
                instructions="Which team should handle this ticket?",
                options={
                    "billing": "Payments, invoicing, refunds",
                    "technical": "Bugs, outages, integrations",
                    "sales": "Pricing, upgrades, new accounts",
                },
                calibration_profile="support-routing-v3",
                alpha=0.10,
                method="APS",             # or "LAC", "RAPS"
                group_by="account_tier",  # optional Mondrian conditioning field
            ),
        },
    )
 
department = result.answers["department"]
print(department.set)            # ['billing', 'sales']
print(department.top)            # 'billing'
print(department.is_singleton)   # False

Response

{
  "department": {
    "type": "set",
    "set": ["billing", "sales"],
    "probabilities": { "billing": 0.61, "sales": 0.33, "technical": 0.06 },
    "venn_abers": {
      "billing": [0.57, 0.66],
      "sales": [0.29, 0.38],
      "technical": [0.02, 0.09]
    },
    "guarantee": {
      "type": "coverage",
      "alpha": 0.10,
      "method": "APS",
      "calibration_profile": "support-routing-v3",
      "calibration_n": 1204,
      "coverage_ci": [0.886, 0.914],
      "last_audited": "2026-09-18T00:00:00Z"
    }
  }
}

set is the conformal set. probabilities are the per-option estimates the set was built from, and venn_abers gives a calibrated interval for each option being correct. The guarantee is about set: over tickets exchangeable with the 1,204 calibration tickets, sets built this way contain the correct team at least 90% of the time.

Answer attributes

Attribute	Type	Meaning
`set`	`list[str]`	The options in the conformal set
`probabilities`	`dict[str, float]`	Per-option probability estimates
`venn_abers`	`dict[str, tuple[float, float]]`	Per-option Venn-Abers intervals
`is_singleton`	`bool`	Exactly one option. Coverage is not conditional on set size, so singletons alone carry no error bound; gate them with `Gate(guarantee="fdr")`
`is_empty`	`bool`	No option cleared the threshold: abstain
`top`	`str \| None`	The highest-probability option inside the set
`guarantee`, `is_heuristic`		The guarantee card, and whether it is heuristic

Choosing a score method

Method	Behavior
`LAC`	Smallest average set size; conditional coverage can be uneven across easy and hard inputs. Sets can be empty.
`APS`	Adaptive: larger sets on ambiguous inputs, smaller on clear ones. The server default.
`RAPS`	APS with a rank penalty; smaller sets than APS at a small conditional-coverage cost, useful with long option lists.

The math for each score is on The three engines.

Group-conditional (Mondrian) calibration

Set group_by to a field present in context to calibrate a separate threshold per group, so a marginal 90% guarantee cannot hide a much lower rate on a subgroup you care about. Each group needs its own $n \ge \lceil (1-\alpha)/\alpha \rceil$ examples; CLI enforces this per group, and groups below it fall back to the marginal threshold (reported in profile.groups). When calibration examples do not carry the field in their context, set it explicitly with CalibrationExample(group=...).

Option limits

Conformal set methods handle option lists of any size the underlying backend can score. When scoring comes from generated-token log-probabilities (access level L1), the practical limit is the backend’s top-k log-probability cap — see The access ladder. Larger option lists fall back to sampling-based scoring automatically, or you can force sampling with backend_access_hint="sampling".

Give every option a short description. A None description is sent as null and works, but a few words usually help the backend score the options.

Parameters

Parameter	Type	Description
`instructions`	`str \| dict \| list`	Required. The decision to make.
`options`	`dict[str, str \| None]`	Required, at least 2. Option key to a short description.
`calibration_profile`	`str`	Required.
`alpha`	`float`, optional	Target miscoverage rate; defaults to the profile’s.
`method`	`"LAC" \| "APS" \| "RAPS"`, optional	Server default `"APS"`.
`group_by`	`str`, optional	Context field to condition calibration on (Mondrian).
`backend_access_hint`	`str`, optional	Pin the score family; see access hints.

Belief Interval