Errors and retries

Every exception the SDK raises for an API, connection or configuration problem subclasses CLIError, importable from cli_sdk. Invalid query or resource arguments raise ConfigurationError at construction time, before anything reaches the network; ConfigurationError is also a ValueError, so existing except ValueError code keeps working.

Exception map

Not wrapped in CLIError:

from cli_sdk import (
    AuthenticationError, BackendError, CLIClient, CLIError, ConfigurationError,
    InsufficientCalibrationError, RateLimitError, ValidationError,
)
 
try:
    result = client.evaluate(context=ticket, backend=backend, queries=queries)
except ValidationError as err:
    log.error("bad request field=%s: %s", err.field, err)
except RateLimitError as err:
    schedule_retry(after=err.retry_after or 30)
except BackendError:
    fall_back_to_human_queue()
except InsufficientCalibrationError:
    fall_back_to_human_queue()        # strict mode: no heuristic answers
except CLIError as err:
    log.exception("evaluate failed: %s", err)
    fall_back_to_human_queue()

Heuristic answers and 424

When a query names a profile that does not yet have enough labelled examples for its guarantee (below $\lceil (1-\alpha)/\alpha \rceil$, or underpowered for the requested bound), the service answers with HTTP 424 Failed Dependency and a body that still contains an answer for every query, with the affected answers’ cards labelled heuristic.

By default, evaluate returns that response instead of raising: the answers are usable, and they say plainly that no formal guarantee backs them.

result = client.evaluate(context=ticket, backend=backend, queries=queries)
 
if result.heuristic_answers:                       # e.g. ["route"]
    for qid in result.heuristic_answers:
        print(qid, result.answers[qid].guarantee.describe())
        # Heuristic value: no formal statistical guarantee.
 
gate = result.answers["route"]
if gate.approved and not gate.is_heuristic:
    auto_route()
else:
    send_to_human()

The SDK applies the same rule to every response, whatever its status: an answer whose card is missing, or whose type is not one of the eight known guarantee types, is treated as heuristic. A Gate decision other than auto_approve, escalate, or abstain is read as escalate. The SDK never turns an unrecognized answer into an approval.

strict_guarantees

Pass strict_guarantees=True to refuse heuristic answers outright:

client = CLIClient(strict_guarantees=True)

evaluate then raises InsufficientCalibrationError whenever any answer in the response is heuristic — on a 424, and also on a 200 response that contains a heuristic answer. The message lists the query ids. Use strict mode in paths where acting on an unguaranteed answer is never acceptable, and handle the exception by falling back to human review.

Retries

429, 502 and 529, connection errors and timeouts are retried automatically; everything else (authentication, validation, insufficient calibration) is returned immediately, because retrying cannot change the outcome. Every POST carries an Idempotency-Key header, one value per logical call reused across its retries, so a retried write (for example add_examples after a 502 that the server had already committed) is applied once.

from cli_sdk import CLIClient, RetryConfig
from cli_sdk.client import NO_RETRY
 
client = CLIClient(retry=RetryConfig(max_attempts=5, base_delay=0.5, max_delay=8.0))
no_retries = CLIClient(retry=NO_RETRY)          # RetryConfig(max_attempts=1)

Before retry $k$ (the wait after failed attempt $k$), the client sleeps:

• the server’s retry-after value in full, if it is numeric (and at most max_retry_after); otherwise
• a uniform random time in $\big[0,\ \min(\text{max\_delay},\ \text{base\_delay} \cdot 2^{k-1})\big]$ (full-jitter exponential backoff).

When the last attempt still fails, the matching exception is raised (RateLimitError for 429, BackendError for 502 and 529, APIConnectionError for a connection error or timeout).

A few consequences worth knowing:

• Size timeout for L0 backends, which make one model call per sample.
• With a CustomBackend, evidence is computed once, before the first attempt; retries resend the same evidence and do not call your model again.
• RetryConfig is a frozen dataclass. Share one instance across clients.

Request ids

Every response carries a request id (result.request_id, from the x-request-id header). Log it next to any decision you act on: together with the guarantee card’s calibration_profile and last_audited, it lets you trace a decision back to the exact calibration that backed it.