Claim

Decompose a long-form answer into atomic claims and filter to the subset that is jointly supported, with a guarantee on the retained set — rather than trusting an entire generated answer because it “sounds right.”

Request

from cli_sdk import AnthropicBackend, CLIClient, Claim
 
with CLIClient() as client:
    result = client.evaluate(
        context={
            "question": "What's your refund policy for annual plans?",
            "retrieved_docs": retrieved_docs,
            "draft_answer": draft_answer,
        },
        backend=AnthropicBackend(model="claude-sonnet-5"),
        queries={
            "filtered_answer": Claim(
                instructions="Filter the draft answer to only fully-supported claims.",
                calibration_profile="rag-factuality-v1",
                alpha=0.05,
                support_source="retrieved_docs",
            ),
        },
    )
 
answer = result.answers["filtered_answer"]
print(answer.as_text())
print(f"kept {answer.retention_rate:.0%} of claims")
for dropped in answer.dropped_claims:
    print(dropped.text, dropped.reason, dropped.score)

Response

{
  "filtered_answer": {
    "type": "claim",
    "retained_claims": [
      "Annual plans can be refunded within 30 days of purchase.",
      "Refunds are issued to the original payment method."
    ],
    "dropped_claims": [
      { "text": "Refunds typically take 2 business days.", "reason": "unsupported", "score": 0.31 }
    ],
    "guarantee": {
      "type": "risk",
      "target": 0.05,
      "method": "conformal-factuality",
      "calibration_profile": "rag-factuality-v1",
      "calibration_n": 140
    }
  }
}

The card is a risk card with a 0/1 loss: 1 when any retained claim is false. Expected loss at most 0.05 is the same statement as $P(\text{all retained claims true}) \ge 0.95$, over question/answer pairs exchangeable with the calibration set.

How claim support is scored

CLI selects the strongest available scoring method for the connected backend:

• L2 and above (self-hosted vLLM and SGLang, Bedrock Custom Model Import): claim log-likelihood from prompt scoring, plus resampled support frequency.
• L1 (OpenAI and Azure non-reasoning configurations): generated-token log-probabilities on the support question, plus resampled support frequency.
• L0 backends (Anthropic Claude, reasoning-effort configurations) — resampled support frequency and semantic-cluster agreement only. This costs a handful of extra generations per claim, reported in result.usage.backend_calls, but produces the same guarantee shape.
• Retrieval-augmented contexts — claim-to-source support computed from the support_source field of context, when the backend and context support it.

Calibration data for Claim

Claim-level calibration needs every claim in each calibration example labelled true or false against a reference. This is labour-intensive per example, so typical profiles are smaller than classification profiles: 50 to 200 fully annotated question/answer pairs is a reasonable starting point. See label-efficient calibration for combining a small human-labelled set with judge-labelled examples, and the RAG claim filtering guide for a full walkthrough.

Answer attributes

Parameters