Entity matching with Belief

Problem

A CRM has duplicate company records (“Acme Corp.”, “ACME Corporation”, “Acme Corp - Berlin office”). A blocking step already proposes candidate pairs. Merging two different companies is expensive and hard to undo; leaving a duplicate is cheap. Today, pairs are merged when a string similarity score exceeds a threshold someone picked.

A raw similarity score, or a model’s stated confidence, cannot tell you how often a “0.9” is actually a match, or when the evidence is simply too thin to decide.

Primitives

Belief: one calibrated Venn-Abers interval $[p_0, p_1]$ per candidate pair, for “these records are the same company”.
The straddle rule: merge only when the whole interval is at or above the threshold, keep separate when it is at or below, and send pairs whose interval straddles the threshold to a human.

Code

Calibrate on pairs from your own blocking step

Label a random sample of the candidate pairs the blocking step actually produces, not a hand-built set of obvious matches and obvious non-matches.

from cli_sdk import CalibrationExample, CLIClient, OpenAIBackend
 
PROFILE = "entity-match-v2"
BACKEND = OpenAIBackend(model="gpt-4.1-2025-04-14")
 
with CLIClient() as client:
    client.calibration_profiles.create(name=PROFILE, backend=BACKEND, method="IVAP", alpha=0.10)
    client.calibration_profiles.add_examples(
        PROFILE,
        examples=[
            CalibrationExample(context={"left": left, "right": right}, label=same_company)
            for left, right, same_company in labelled_pairs      # about 2,000 reviewed pairs
        ],
    )

Score candidate pairs concurrently

import asyncio
 
from cli_sdk import AsyncCLIClient, Belief, BeliefAnswer
 
MATCH = Belief(
    instructions="Do these two records refer to the same company?",
    calibration_profile=PROFILE,
    criteria={
        "true": "The same legal entity, even with different name formatting or address details.",
        "false": "Different companies, including a parent and its subsidiary.",
    },
)
 
async def score_pairs(pairs: list[tuple[dict, dict]], concurrency: int = 16) -> list[BeliefAnswer]:
    limit = asyncio.Semaphore(concurrency)
    async with AsyncCLIClient(backend=BACKEND) as client:
        async def one(left: dict, right: dict) -> BeliefAnswer:
            async with limit:
                result = await client.evaluate(context={"left": left, "right": right},
                                               queries={"match": MATCH})
                return result.answers["match"]
        return await asyncio.gather(*(one(left, right) for left, right in pairs))

Apply the straddle rule

MERGE_AT = 0.9
 
def decide(match: BeliefAnswer) -> str:
    if match.is_heuristic or match.venn_abers is None:
        return "review"
    p0, p1 = match.venn_abers
    if p0 >= MERGE_AT:
        return "merge"
    if p1 <= MERGE_AT:
        return "keep_separate"
    return "review"                      # match.straddles(MERGE_AT) is True
 
pairs = [
    ({"name": "Acme Corp.", "domain": "acme.com", "city": "Berlin"},
     {"name": "ACME Corporation", "domain": "acme.com", "city": "Berlin"}),
    ({"name": "Acme Corp.", "domain": "acme.com", "city": "Berlin"},
     {"name": "Acme Logistics GmbH", "domain": "acme-logistics.de", "city": "Hamburg"}),
]
for (left, right), match in zip(pairs, asyncio.run(score_pairs(pairs))):
    print(left["name"], "|", right["name"], match.venn_abers, decide(match))
# Acme Corp. | ACME Corporation (0.95, 0.97) merge
# Acme Corp. | Acme Logistics GmbH (0.58, 0.93) review

Check the rule before you trust it

The straddle rule decides on calibrated probabilities; it does not by itself bound how often a merge is wrong. Measure that on held-out labelled pairs, entirely offline:

import numpy as np
from cli_sdk.calibration import clopper_pearson
from cli_sdk.stats.venn_abers import ivap
 
# Stand-in for labelled candidate pairs: a similarity score per pair and
# whether a human confirmed it as the same company.
rng = np.random.default_rng(11)
is_match = rng.uniform(size=4000) < 0.3
similarity = np.clip(np.where(is_match, rng.normal(0.8, 0.12, 4000), rng.normal(0.45, 0.18, 4000)), 0, 1)
 
cal, held_out = np.arange(2000), np.arange(2000, 4000)
p0, p1 = ivap.calibrate_and_predict(similarity[cal], is_match[cal].astype(int), similarity[held_out])
 
MERGE_AT = 0.9
merge = p0 >= MERGE_AT
keep_separate = p1 <= MERGE_AT
review = ~merge & ~keep_separate               # the interval straddles MERGE_AT
 
truth = is_match[held_out]
false_merges = int(np.sum(merge & ~truth))
lo, hi = clopper_pearson(false_merges, int(merge.sum()))
print(f"merge {merge.mean():.1%}  keep separate {keep_separate.mean():.1%}  review {review.mean():.1%}")
print(f"false merges: {false_merges} of {merge.sum()} (95% CI {lo:.3f} to {hi:.3f})")
print(f"matches left unmerged: {np.sum(truth & ~merge)} of {truth.sum()}")
# merge 9.8%  keep separate 86.7%  review 3.5%
# false merges: 15 of 197 (95% CI 0.043 to 0.122)
# matches left unmerged: 413 of 595

Walkthrough

Each candidate pair is one evaluate call with both records in context, the same shape as the calibration examples, so production pairs are exchangeable with calibration pairs as long as the blocking step and record sources do not change. AsyncCLIClient with a semaphore scores many pairs concurrently without overrunning rate limits (429 responses are retried automatically; see Errors and retries).

The first example pair shares a name, domain, and city: the interval $[0.95, 0.97]$ sits entirely above 0.9, so the pair is merged. The second shares a brand but not a domain or city: $[0.58, 0.93]$ straddles 0.9. The calibration data cannot say which side of the threshold this pair’s calibrated probability lies on, so a human decides. A wide interval like this is common where calibration pairs are sparse, for example unusual naming patterns.

The offline check puts numbers on the rule. With MERGE_AT = 0.9, 15 of 197 merges on held-out pairs were wrong: 7.6%, with a 95% interval of 4.3% to 12.2%. That is in line with merging only when the calibrated match probability is at least 0.9, and it is not zero. Raise MERGE_AT for fewer false merges and more reviews, and re-run the check.

If policy requires a stated bound — “fewer than 2% of automatic merges are wrong” — express it directly as a Gate with guarantee="fdr", target=0.02, calibrated on the same labelled pairs, and keep Belief for the reviewer’s view of how ambiguous a pair is.

What the guarantee card means

{
  "type": "coverage",
  "method": "IVAP",
  "calibration_profile": "entity-match-v2",
  "calibration_n": 2000
}

The card states Venn-Abers validity: for pairs exchangeable with the 2,000 calibration pairs, the endpoint of $[p_0, p_1]$ selected by the true answer is calibrated on average. It has no alpha, because a Venn-Abers interval is not a $1-\alpha$ confidence interval.
It does not say that a merged pair is a match with probability at least 0.9. It says that for every merged pair, whichever endpoint is the calibrated one is at least 0.9. Measure the realized false-merge rate on held-out pairs, as above, rather than reading it off the card.
It holds for pairs from the same blocking step and record sources. A new data source, or a change to the blocking rules, is a new population: recalibrate.

Offline calibration Detecting a silent model update