thisness 0.1.0__tar.gz → 0.2.0__tar.gz

thisness-0.2.0/CHANGELOG.md

@@ -0,0 +1,22 @@
+# Changelog
+
+## 0.2.0 (2026-10-15)
+
+Supplier lists (design 08).
+
+- **The mapping export.** `engine.mapping(entity_type, as_of_seq=None)` and `thisness export mapping` give every record, its entity, the entity's name and a status (`resolved`, `in_review`, `provisional`, `unresolved`), as CSV or Parquet, as of any point in the ledger; the Workbench's entity lists download it.
+- **Supplier-name cleaning.** The organization name normalizer (version 3) removes payment-processor prefixes, store and location suffixes, reference numbers and ERP notes before comparing names; a note becomes the record's status. Numbers inside a name ("Intercede 2296 Ltd", "Omega Holdings II") are kept. A trade name ("dba", "t/a") is an alias that proposes candidates and is a relationship signal, not a match. A card descriptor cut at its length limit is never taken for a whole name. On messy names the name-only plan's recall rises from .865 to .964 (synthetic) and .803 to .93 (UK).
+- **Readable names.** Organization labels prefer a mixed-case spelling. A model can suggest display names for entities written in capitals (`thisness labels suggest`); a suggestion must be a spelling of the entity's own names, and accepting it is a reversible commit (`ENTITY_LABELED`), kept through later merges and splits.
+- **Benchmarks.** `thisness eval --names-only` takes the UK benchmark; `--messy` adds what real extracts carry; `--plan rewrite` runs a rewrite-and-hash baseline of the kind supplier-consolidation pipelines use (design 05 §11.5).
+- **Changed.** Name comparison version 5 (levels `trade_name` and `truncated_prefix`); the organization starter model is rebuilt. The name-only plan collapses on the cleaned name and no longer drops every number of three or more digits, so numbered companies reach the model apart. A store built with 0.1.0 keeps working: opening it recomputes its organization features with the new normalizer, and a model fitted for the old comparisons is replaced on the next resolve, by refitting or by the starter.
+
+## 0.1.0 (2026-10-10)
+
+The first release.
+
+- **Core** (design 01 to 03). Observations, stable entities, and a ledger of reversible merges and splits in DuckDB. Candidate generation by blocking keys, typed comparisons, Fellegi-Sunter weights fitted by EM with term-frequency adjustments, and validated-components consensus that commits only what the evidence supports and opens a review for the rest. Every decision is stored with its evidence and can be explained.
+- **Review and the Workbench** (design 04). Pair constraints, a review queue ranked by impact, dossiers, relationship candidates (a shared brand, a household), and a local web Workbench to browse entities, answer reviews by dragging records, relate or merge entities, and page through possible relationships.
+- **Three domains.** Organizations, people, and taxa (scientific plant names, with World Flora Online and the Catalogue of Life as authorities; design 06). Each ships a starter model for stores too small to fit their own.
+- **The language-model layer** (design 05). Anthropic and OpenAI-compatible providers, budgets, retries on transient errors, record and replay. An adjudicator for contested pairs (shadow or decide; a model's "same" yields to its own "different" and to cannot-link constraints), a review assistant that suggests answers, name-only resolution, and a per-profile `model_role`.
+- **Benchmarks.** Synthetic organizations and people, WFO plant names, and real UK companies from GLEIF and Companies House with company numbers as truth (design 07). `thisness eval` reports precision, recall, false-merge clusters, and a model's accuracy against the truth.
+- **Scale.** Tested to 46,565 real records: bounded boundary reviews and relationship neighborhoods, a paginated relationships page, penalty-only common-value adjustments for shared addresses.

{thisness-0.1.0 → thisness-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: thisness
-Version: 0.1.0
+Version: 0.2.0
 Summary: Identity resolution for imperfect observations: evidence, stable identities, reversible history.
 Project-URL: Homepage, https://thisness.dev
 Project-URL: Repository, https://github.com/mfbaig35r/thisness
@@ -95,8 +95,11 @@ thisness ui                              # the Workbench: entities, reviews, his
 thisness --store demo.duckdb demo && thisness ui demo.duckdb    # try it on synthetic data
 thisness --store people.duckdb demo --type person                # synthetic people
 thisness --store taxa.duckdb demo --type taxon --reference       # real plant names, labelled by WFO
+thisness export mapping > mapping.csv   # every record, its entity and the entity's name
 ```
 
+Supplier lists and spend extracts work as they are: payment-processor prefixes ("SQ *", "PAYPAL *"), store and reference numbers and ERP notes ("DO NOT USE", "ZZ ") are cleaned before names are compared, and a note becomes the record's status in the mapping. A model can suggest readable names for entities whose records are all in capitals (`thisness labels suggest`), which you accept or clear (design 08).
+
 The Workbench is a local web app over the same API: browse entities with the evidence for every value, answer the review queue by dragging records between entities, relate or merge entities that share a brand, page through possible relationships, and see exactly what a language model was given and what it answered.
 
 ## With a language model
@@ -164,6 +167,7 @@ The UK benchmark's hard cases are real: 62% of live UK companies share a registe
   - [05 the language-model layer](docs/design/05-m3-llm-layer.md)
   - [06 taxonomy, the third domain](docs/design/06-m4-taxonomy.md)
   - [07 real organizations](docs/design/07-m5-real-organizations.md)
+  - [08 supplier lists](docs/design/08-supplier-lists.md)
 - [Changelog](CHANGELOG.md)
 
 ## Development

{thisness-0.1.0 → thisness-0.2.0}/README.md

@@ -63,8 +63,11 @@ thisness ui                              # the Workbench: entities, reviews, his
 thisness --store demo.duckdb demo && thisness ui demo.duckdb    # try it on synthetic data
 thisness --store people.duckdb demo --type person                # synthetic people
 thisness --store taxa.duckdb demo --type taxon --reference       # real plant names, labelled by WFO
+thisness export mapping > mapping.csv   # every record, its entity and the entity's name
 ```
 
+Supplier lists and spend extracts work as they are: payment-processor prefixes ("SQ *", "PAYPAL *"), store and reference numbers and ERP notes ("DO NOT USE", "ZZ ") are cleaned before names are compared, and a note becomes the record's status in the mapping. A model can suggest readable names for entities whose records are all in capitals (`thisness labels suggest`), which you accept or clear (design 08).
+
 The Workbench is a local web app over the same API: browse entities with the evidence for every value, answer the review queue by dragging records between entities, relate or merge entities that share a brand, page through possible relationships, and see exactly what a language model was given and what it answered.
 
 ## With a language model
@@ -132,6 +135,7 @@ The UK benchmark's hard cases are real: 62% of live UK companies share a registe
   - [05 the language-model layer](docs/design/05-m3-llm-layer.md)
   - [06 taxonomy, the third domain](docs/design/06-m4-taxonomy.md)
   - [07 real organizations](docs/design/07-m5-real-organizations.md)
+  - [08 supplier lists](docs/design/08-supplier-lists.md)
 - [Changelog](CHANGELOG.md)
 
 ## Development

{thisness-0.1.0 → thisness-0.2.0}/docs/design/05-m3-llm-layer.md

@@ -224,3 +224,28 @@ Benchmarks: the contested pairs of the Organization and Person benchmarks (adjud
 
 - **What limits precision.** With a right model, every remaining false merge is distinct companies with identical names after collapse ("Premier Services AG" twice, "Premier Services" thirteen times): the benchmark's 25 "Premier Services" companies. From names alone they cannot be told apart; Fellegi-Sunter declines them through its common-name adjustment, which is why its precision is 1.0 and its recall half. The plan has no such guard: a key shared by distinct entities collapses. Systematic folding of related companies adds false merges as it did for the adjudicator. Both are measurements a real model run should repeat; the oracle groups common names by truth, which a real model told to treat common names with suspicion may not.
 - **Tests:** `test_name_only.py` (windows; the collapse key; one vote per stage, echoes and derived votes; recall far above Fellegi-Sunter on names with the oracle; later records joining existing entities; discarded answers and an exhausted budget leaving names provisional; folding related companies).
+
+### 11.5 A rewrite-and-hash baseline, measured
+
+Supplier-consolidation pipelines often resolve names another way: a model rewrites every name in overlapping chunks to a cleaned, consolidated form, in one to three passes, and records whose final rewrites are equal (after keeping letters and digits) are one organization; a last call standardizes each group's name, and groups whose standardized names match merge. Identity is string equality of model output: no pairwise evidence, no votes, no reviews. The name-only plan was measured against it, with a real model, before deciding what to take from it.
+
+- **The baseline** (`benchmark/rewrite.py`, `thisness eval --plan rewrite`): distinct names keyed by their letters and digits; passes over chunks of 50, 75 and 100 overlapping by 10, the first in input order and later ones in key-hash order as such pipelines build it (`--rewrite-order as_built`) or by name (`sorted`); the last answered rewrite per name; one standardization call per group. Its prompts are written for it. Records are shuffled first, since the generators emit them organization by organization and an AP extract is in no such order.
+- **Results** (gpt-6-luna for every model call, names-only benchmarks):
+
+  | Benchmark | Method | Precision | Recall | False-merge clusters | Reviews | Cost |
+  |---|---|---|---|---|---|---|
+  | Synthetic, 805 organizations, 1,776 records | Fellegi-Sunter on names | 1.000 | .528 | 0 | 219 | |
+  | | name-only plan | .984 | .964 | 7 | 12 | $0.14 |
+  | | rewrite, as built | .983 | .762 | 10 | | $0.10 |
+  | | rewrite, passes by name | .983 | .976 | 8 | | $0.10 |
+  | UK, 400 sampled, 3,507 records (design 07) | Fellegi-Sunter on names | .9996 | .344 | 1 | 455 | |
+  | | name-only plan | .9966 | .944 | 6 | 32 | $0.16 |
+  | | rewrite, as built | .9977 | .825 | 5 | | $0.17 |
+  | | rewrite, passes by name | .9961 | .949 | 6 | | $0.16 |
+
+- **String equality does not over-merge.** Precision is the same for every method with a model; the false merges they share are the ones names cannot settle (the synthetic "Premier Services" companies, real UK namesakes), most fixed before any model sees them by the key or the collapse.
+- **Order decides the baseline's recall.** A model consolidates the variants a chunk shows together. Ordered by key hash, later passes scatter one organization's variants across chunks, and 12 to 21 points of recall are lost; ordered by name, the baseline matches the name-only plan. The standardization call merges almost nothing (1,094 groups before it, 1,092 after), so it does not repair the scatter.
+- **What the name-only plan adds is not a better score.** It holds contested clusters for review (12 and 32 here) where the baseline commits everything, and every link it makes cites the votes and the exact batches behind it. It was also faster in these runs (154 s against 184 and 406 s on the synthetic benchmark, 435 s against 1,150 and 1,620 s on the UK one), since the baseline's passes run one after another.
+- **What to take from the baseline** is not its identity rule but what it does around it: cleaning names of payment prefixes, store and account numbers and notes, and a standardized display name per entity. Both are design 08's subject.
+- **Reproducing.** The model runs are recorded; with the recording, `thisness eval --names-only --plan rewrite --rewrite-order sorted --size 805 --seeds 20261007 --adjudicator openai --model gpt-6-luna --replay CASSETTE` (and `--source uk --size 400 --seeds 20261009`) reproduces the table's rows exactly, offline. Without a model, `--plan rewrite` uses an oracle that consolidates only what one chunk shows (and standardizes without merging): on 805 organizations it gives recall .61 as built and .996 by name, the same effect.
+- **Tests:** `test_rewrite.py` (the key; chunks overlapping and covering every name; hash order losing recall to name order at equal precision; identical names of different companies merging; unanswered chunks keeping their names; a spent budget stopping calls without failing the run; names-only UK benchmarks; `eval --plan rewrite`). `names_only()` now takes the UK benchmark as well as the synthetic one.

thisness-0.2.0/docs/design/08-supplier-lists.md

@@ -0,0 +1,187 @@
+# Design 08: supplier lists
+
+Status: accepted 2026-10-14, with the recommendations in section 10. Complete 2026-10-15 at step 4; step 5 (model cleaning) not built (§13.5).
+
+Builds on: design 05 (the name-only plan, §6; the rewrite-and-hash baseline, §11.5), design 07 (real organizations) and the organization profile as of commit fbe761c.
+
+## 1. Why
+
+A vendor master, an accounts-payable extract or a spend cube is a list of supplier names and little else. Thisness resolves such lists with the name-only plan, which on the names-only benchmarks matches the best rewrite-and-hash pipeline on precision and recall and adds what that pipeline lacks: a review queue for what it cannot settle and the votes behind every link (design 05 §11.5). Identity is not what is missing.
+
+What is missing is everything around it that a supplier list needs:
+
+- **Clean names.** Real extracts carry payment-processor prefixes ("PAYPAL *ACME", "SQ *JOES CAFE"), store and account numbers ("ACME CORP #4411", "Acme - 812773"), and notes ("DO NOT USE", "DUPLICATE OF 10233"). The organization normalizer strips a legal form only from the end of a name, so "Harjas Foods PLC - 812773" keeps both the number and the "PLC" in its key.
+- **A readable name per entity.** An entity's label is its most frequent spelling, so a company most often written in a registry's capitals is labelled "HALEON UK FINANCE LIMITED".
+- **The mapping.** What a spend analyst takes away is a table: every source record, the entity it belongs to, and that entity's name. Thisness has the data and no export.
+
+## 2. What was measured first
+
+- **Name-only resolution works without cleaning, but rules cost little.** With account numbers on a quarter of the names, Fellegi-Sunter on names loses 2 to 4 points of recall (synthetic: .571 to .528; UK: .367 to .344); the name-only plan already strips account numbers in its collapse key. The benchmarks carry no payment prefixes or notes, so their cost is unmeasured.
+- **Blocking is not the bottleneck.** With names only, the candidate generators propose 96.8% of true pairs on the synthetic benchmark and 96.4% on the UK one (98.0% and 96.5% with full records). Names-only recall is low (.53 and .34 for Fellegi-Sunter) because a name is weak evidence, not because pairs go unproposed. An embedding candidate generator could raise the ceiling by at most 3.5 points here.
+- **A model's string equality does not over-merge** (design 05 §11.5), so a model asked to clean names does not threaten precision by itself, provided it only cleans and the identity decision stays with the plan.
+
+## 3. Name cleaning by rules
+
+A new version of the organization name normalizer (`organization_name` v3) cleans before it tokenizes. Each rule is a dictionary or pattern in the normalizer's configuration, so changing one is a new component version and a feature rebuild, as now. The raw name is never changed; cleaning only shapes the features.
+
+| Rule | Examples | Removed |
+|---|---|---|
+| Payment-processor prefixes | "PAYPAL *ACME", "SQ *JOES CAFE", "TST* BLUE DOOR", "SP * SHOPNAME" | the prefix through its asterisk |
+| Trailing reference numbers | "Acme - 812773", "ACME CORP #4411", "Acme Corp 812773", "Acme (Vendor 10233)" | a number at the end after a separator ("-", "#", ":") or a reference word, a bare trailing number of five or more digits, or a trailing parenthesis with a reference word |
+| Store and location suffixes | "WALGREENS STORE 1022", "SHELL OIL 57442 BLDG 3" | "store", "unit", "bldg", "loc" and a number |
+| Notes | "DO NOT USE", "INACTIVE", "DUPLICATE OF 10233", "(OLD)", "USE VENDOR 4411" | the note |
+
+What the rules must not remove: numbers that are part of a name ("3M", "7-Eleven", "Studio 54"), numbered vehicles ("Omega Holdings II", "Landmark Mortgage Securities No. 3"), which the `number_differs` level relies on (design 07 §13.2), and years ("Mayfield 1966 Investments"). Trailing numbers are stripped only after a separator or a reference word, or when they have five or more digits; Roman numerals and "No. n" are never stripped.
+
+Two things the rules find are kept rather than dropped:
+
+- **Notes are observation metadata.** "DO NOT USE" and "INACTIVE" say something about the vendor record, not the vendor. They become a `record_status` feature, shown on the record in the Workbench and in the export, never compared.
+- **Trade names are a second name, but weak.** "Acme Holdings LLC dba Acme Tools" and "T/A" forms yield two names for one record. The record's name stays as written; the trade name becomes an alias feature that candidate generation reads, and the name comparison scores a match on it at a capped level (0 bits) and treats as a relationship signal: a trade name is often a group's brand, shared by sister companies (§13.2, decision 3).
+
+## 4. Name cleaning by a model, only if the rules leave enough behind
+
+What rules miss (mangled processor descriptors, free-text notes, names run together with addresses) a model can clean. It is built only if the measurement in §8 shows a residue worth the cost and the machinery:
+
+- One call per batch of distinct raw names not already clean by the rules, cached by the raw string, so a name is cleaned once per store and the answer is replayed thereafter.
+- The answer is stored as an **annotation**: a value derived from an observation by a recorded component or model invocation, never an edit to the observation. Normalizers read a name's annotation when one exists. Annotations are new (decision 4): they need a table, a ledger-free rebuild rule (dropping an annotation component rebuilds features without it) and a place in the dossier.
+- The prompt only cleans; it is never asked to make two names equal. Consolidation stays with the name-only plan, where it is voted and reviewable.
+
+## 5. Display names
+
+Two steps, the first free:
+
+1. **Prefer a readable spelling.** The organization canonicalizer picks, among the spellings of the entity's most supported name key, the one in mixed case when one exists, else the most frequent. "HALEON UK FINANCE LIMITED" (3 records) and "Haleon Uk Finance Limited" (1) share a key; the label becomes the second. Support counts are unchanged: the dossier still shows how many records carry each value.
+2. **A model's suggestion, recorded and reviewable.** For an entity whose best spelling is all capitals, abbreviated or truncated, the review assistant can suggest a display name from its variants ("Haleon UK Finance Limited"). An accepted suggestion is a ledger event (`ENTITY_LABELED`) citing the suggestion, so it is reversible and the snapshot shows it; the canonicalizer never overrides an accepted label. Suggestions are asked for in bulk, like review recommendations, and cost one short call per entity.
+
+## 6. The mapping export
+
+```bash
+thisness export mapping --type organization > mapping.csv      # or --format parquet
+```
+
+`engine.mapping(entity_type, as_of_seq=None)` returns the same table:
+
+| Column | |
+|---|---|
+| `source_type`, `source_id`, `locator` | the record as it was ingested |
+| `name` | the record's name as written |
+| `observation_id` | |
+| `entity_id`, `entity_name` | its entity, and the entity's display name |
+| `status` | `resolved`, `in_review` (a member of an open review), `provisional` |
+| `record_status` | the note the cleaning rules found, if any (§3) |
+| `since_seq`, `decision_id` | when the membership began, and the decision behind it |
+
+`as_of_seq` exports the mapping as it stood at a point in the ledger, so a spend report can be reproduced after later merges. The Workbench gets a download link on each entity list.
+
+## 7. Embedding candidates, deferred
+
+§2 shows the generators already propose nearly every true pair on these benchmarks, so an embedding generator is not part of this milestone. It stays the answer to a measured gap: if real supplier lists show pairs that share no blocking key (a renamed company with its old and new names in one list is not one; nothing in two unrelated names says they are the same), an `EmbeddingNeighbors` generator would embed cleaned names through the provider layer (OpenAI-compatible `/embeddings`, so a local server works too), cache vectors by text, and propose nearest neighbours above a similarity, to be scored like any other candidate.
+
+## 8. Benchmark
+
+The names-only benchmarks gain a **messy** variant (`--names-only --messy`): on a share of records, a payment prefix, a store or location suffix, a reference number in one of the forms of §3, a note, or a trade name. The same generator settings apply to the synthetic and UK benchmarks. Measured before and after each step:
+
+1. Fellegi-Sunter on names and the name-only plan, on clean and messy names, before any rule (the cost of mess).
+2. The same after the rules (§3).
+3. If the residue warrants it, model cleaning (§4).
+4. Display names: the share of entities whose label is all capitals, before and after §5.1.
+
+A regression test fixes the rules' behaviour on a list of names that must and must not change, including every example in §3.
+
+## 9. Order
+
+1. **The mapping export** (§6). Small, useful on its own, and the table every later step is checked against.
+2. **The messy benchmark and the first measurement** (§8.1).
+3. **Cleaning rules and the readable-spelling canonicalizer** (§3, §5.1), measured (§8.2, §8.4).
+4. **Display-name suggestions** (§5.2).
+5. **Model cleaning**, only if §8.2 leaves a residue worth it (§4).
+
+## 10. Decisions
+
+| # | Question | Recommendation |
+|---|---|---|
+| 1 | What to take from rewrite-and-hash pipelines | Cleaning and display names; not their identity rule (design 05 §11.5) |
+| 2 | Where cleaning lives | In the organization name normalizer, versioned; the raw name never changes |
+| 3 | Trade names ("dba", "t/a") | An alias feature read by candidates; matched only at a capped level (0 bits) and as a relationship signal, since a trade name is often a group's brand. The record's name stays as written (revised 2026-10-14, §13.2) |
+| 4 | Model cleaning | Deferred until the rules are measured; if built, as annotations (a new record type), never edits to observations |
+| 5 | Display names | Readable spelling by rule now; model suggestions accepted as a reversible ledger event |
+| 6 | Export | `thisness export mapping` and `engine.mapping()`, with `as_of_seq` |
+| 7 | Embedding candidates | Deferred: blocking proposes 96 to 98% of true pairs on names alone |
+| 8 | Prompts | Written for Thisness; no prompt or dictionary is copied from other pipelines |
+
+## 11. Risks
+
+- **Rules that strip part of a name.** A number, a word like "Store" or a note-like phrase can be the name itself ("Store 54 Records", "The Inactive Club"). The must-not-change list in §8 is the guard, and every rule is a dictionary entry that can be removed in a new version.
+- **Locale.** Payment descriptors and note conventions differ by country and by ERP. The first dictionaries are US and UK forms; others are additions, not code.
+- **Annotations, if built, are a new kind of state.** They must rebuild features deterministically from recorded answers and must not let a model's cleaning become a merge by the back door. The prompt cleans one name at a time, and §2's finding about string equality is checked again on the messy benchmark before they ship.
+
+## 12. Not in scope
+
+Supplier classification (industry, commodity), web enrichment, and parent-subsidiary hierarchies: relationships already carry "subsidiary of", and Level 2 data is design 07's.
+
+## 13. Implementation notes
+
+### 13.1 Step 1: the mapping export
+
+- **As built.** `engine.mapping(entity_type, as_of_seq=None)` returns a pyarrow table with the columns of §6, from one query over the ledger: records ingested by the point, memberships valid at it, each entity's latest snapshot by then, entities created provisional and not yet promoted, and reviews open at it. `thisness export mapping [--type] [--format csv|parquet] [--output] [--as-of SEQ]` writes it (CSV to standard output without `--output`), and the Workbench's entity lists have a Mapping download. `record_status` is empty until the cleaning rules (step 3) fill it.
+- **Status.** `in_review` when the record or its entity is a member of an open review; then `unresolved` (no entity), `provisional`, `resolved`. On the 46,565 UK records of design 07 §13.4, 69% of records are `in_review`: 2,629 boundary reviews are open there, and a boundary review holds every record of the entities it asks about. That is what the status says (those records' entity may still change), but an analyst may read it as doubt about the records themselves; a `review_id` column, so a report can group by the open question, is a small addition if it proves needed.
+- **Cost.** 40 ms for the 46,565 records, 70 ms as of an earlier point.
+- **Tests** (`test_mapping.py`): one row per record with its current entity and that entity's name; status agreeing with open reviews and entity states; a point before resolution (every record unresolved) and before ingest (no rows); a merge changing the mapping from its seq on and not before; the command in CSV and Parquet; the Workbench download, its 404 and its link.
+
+### 13.2 Step 2: the messy benchmark, and what mess costs
+
+- **The variant** (`messy()`, `thisness eval --names-only --messy`): on 35% of names-only records, one of a payment-processor descriptor ("SQ *", "TST* ", "PAYPAL *" and the name in capitals, half of them cut at 22 characters as card statements cut them), an ERP note ("ZZ ", "XX-", "*INACTIVE* ", "DO NOT USE", "(OLD)", "DUPLICATE OF n", "USE VENDOR n"), a store or location suffix, a reference number ("(VENDOR n)", "REF n", "A/C n"), or a trade name ("dba", "T/A" and the name's first one or two words). It applies to the synthetic and UK benchmarks alike, on top of the account numbers names-only already appends.
+- **Results** (gpt-6-luna; seeds 20261007 and 20261009; every model run recorded under `~/thisness-demo/messy`):
+
+  | Benchmark | Method | Clean P | Clean R | Messy P | Messy R | False merges, clean to messy | Reviews, clean to messy |
+  |---|---|---|---|---|---|---|---|
+  | Synthetic, 1,776 records | Fellegi-Sunter on names | 1.000 | .528 | 1.000 | .507 | 0 to 0 | 219 to 137 |
+  | | name-only plan | .985 | .968 | .987 | .865 | 6 to 5 | 20 to 21 |
+  | | rewrite, passes by name | .983 | .976 | .982 | .967 | 8 to 8 | |
+  | UK, 3,507 records | Fellegi-Sunter on names | .9996 | .344 | 1.000 | .335 | 1 to 0 | 466 to 340 |
+  | | name-only plan | .995 | .944 | .996 | .803 | 7 to 7 | 21 to 47 |
+  | | rewrite, passes by name | .996 | .949 | .986 | .923 | 6 to 28 | |
+
+- **Mess costs the name-only plan recall, not precision.** 10 points on the synthetic benchmark and 14 on the UK one, at the same precision. The collapse key no longer folds a name's variants together once one of them carries a descriptor or a note (UK nodes 1,574 to 2,524), so more of them reach the model as separate items, and some are never grouped. The UK run also made more than twice the calls ($0.15 to $0.32). This is what the rules of §3 are for: a cleaned key collapses them again before any model sees them.
+- **A model that cleans while it consolidates keeps recall and loses precision.** The rewrite baseline's false merges on the UK names went from 6 to 28. Read one by one, they are the cases §3 and §11 anticipated: a truncated descriptor ("TST* BRIDGEHAVEN EUROP") taken for a prefix of another company's name; numbered companies cleaned into one ("Intercede 2296 Ltd", "Intercede 2302 Ltd", "Intercede 2343 Limited"); a qualifier dropped ("BRIP 6 (Poole BC) LLP" and "BRIP 6 (Poole) LLP"); and trade names that are a group's brand ("BRIP 6 (POOLE) LLP dba BRIP 6"). It is the evidence for decision 4's rule that a model only cleans, and for the must-not-change list of §8.
+- **Decision 3 revised** (accepted 2026-10-14). A trade name as an alias the name comparison reads as a second name would merge sister companies that trade under one brand, as the last case shows. Proposed: the trade name is an alias feature that candidate generation reads, and the name comparison scores at a capped level (0 bits, like `token_subset`) and treats as a relationship signal, so two records sharing only a trade name are related until other evidence says they are the same.
+- **Tests** (`test_messy.py`): determinism and unchanged truth; the share of changed names and every kind of mess, truncated descriptors included; the UK variant; `--messy` requiring `--names-only`.
+
+### 13.3 Step 3: cleaning rules and the readable label
+
+- **As built.** `clean_name()` in the organization normalizer (version 3) removes, before tokenizing: payment-processor prefixes; ERP notes before or after the name, which become the record's status (`do_not_use`, `inactive`, `blocked`, `old`, `duplicate`, `use_other`; a feature, shown in the mapping export, never compared); store and location suffixes; reference numbers after a separator or a reference word, bare trailing numbers of five or more digits, and any number after a legal form ("Amazon, Inc. 4411"); and trade names after "dba", "t/a", "trading as", which become an alias key. Suffixes stack and are removed until none is left. A name that is only a note keeps itself. Every pattern is in the normalizer's configuration. `test_cleaning.py` holds the lists of names that must change and must not ("3M", "7-Eleven", "Studio 54", "Omega Holdings II", "No. 3", "Mayfield 1966 Investments", "Intercede 2296 Ltd", "Old Mutual", "The Inactive Club", "Unit 4 Ltd", "Building 3").
+- **Trade names** (decision 3 as revised): candidate generators pair an alias with the name it equals and records sharing an alias; the name comparison (version 5) has a `trade_name` level, neutral, capped at 0 bits, a relationship signal. It and `truncated_prefix` are placed before `else` so the term-frequency levels keep their indices.
+- **A card descriptor cut short is not a name.** The first version dropped the possibly cut last word of a descriptor 22 characters long. Measured, that made two companies' descriptors identical ("SP * ASSOCIATED BRITIS", "TST* ASSOCIATED BRITIS" both "Associated") and one company's descriptor another's name ("SQ *LOW PROFILE FASHIO" read "Low Profile", a different company's): six false-merge clusters for Fellegi-Sunter and twenty for the name-only plan on messy UK names, most collapsed before any model saw them. Now the cut word is kept and the name marked (`name.truncated`); the name comparison gives a cut name that could begin the other only `truncated_prefix` (neutral, capped at 0), even when two cut names read alike; and the name-only plan never collapses one, so each reaches the model on its own.
+- **The name-only plan** collapses on the cleaned key, without its former rule of dropping every number of three or more digits, so numbered companies ("Intercede 2296 Ltd", "Intercede 2302 Ltd") reach the model apart; and its windows follow the cleaned key, so "SQ *HOWDEN JOINERY" sits beside "Howden Joinery".
+- **Results** (gpt-6-luna; recordings under `~/thisness-demo/cleaned2`; before = step 2, §13.2):
+
+  | Benchmark | Method | Before P | Before R | After P | After R | False merges, before to after |
+  |---|---|---|---|---|---|---|
+  | Synthetic, clean names | Fellegi-Sunter | 1.000 | .528 | 1.000 | .571 | 0 to 0 |
+  | | name-only plan | .985 | .968 | .990 | .956 | 6 to 5 |
+  | Synthetic, messy names | Fellegi-Sunter | 1.000 | .507 | 1.000 | .548 | 0 to 0 |
+  | | name-only plan | .987 | .865 | .986 | .964 | 5 to 6 |
+  | UK, clean names | Fellegi-Sunter | .9996 | .344 | .9995 | .367 | 1 to 1 |
+  | | name-only plan | .995 | .944 | .995 | .948 | 7 to 6 |
+  | UK, messy names | Fellegi-Sunter | 1.000 | .335 | .9995 | .364 | 0 to 1 |
+  | | name-only plan | .996 | .803 | .993 | .930 | 7 to 13 |
+  | Synthetic, full records | Fellegi-Sunter | .9994 | .923 | .9994 | .923 | 1 to 1 |
+  | UK, full records | Fellegi-Sunter | .9992 | .9119 | .9992 | .9116 | 1 to 1 |
+
+  On messy names the name-only plan recovers most of what mess cost (synthetic to the clean level; UK 12.7 of 14 points), at about half the UK cost ($0.32 to $0.23). The UK's 13 false-merge clusters, read one by one, are mostly the model joining a cut descriptor to a whole name it fits ("PAYPAL *ZODIAC MARITIM" and Zodiac Maritime Limited; "TST* ROYAL FREE HOSPIT" and Royal Free Hospital) where the benchmark gave the descriptor to a namesake: nothing in the record can tell them apart. Before the rules those descriptors sorted among the S's and P's, out of their names' windows, so they were neither merged wrongly nor found. Guidance telling the model that a cut descriptor may name another company is a possible next step. The synthetic clean run's recall fell a point (.968 to .956) with one fewer false merge, from the new window order.
+- **The readable label.** The organization canonicalizer (`ReadableSpelling`) labels an entity with a spelling of its most supported cleaned key that needed no cleaning, then one in mixed case (most of its words with a lower-case letter, so "AXA XL UNDERWRITING AGENCIES Ltd." does not count), then the most frequent. On 548 UK entities, labels in capitals fell from 77% to 56%; 41% have no spelling but capitals, which is what step 4's suggestions are for. Case it cannot know stays wrong ("Axa Xl"), for the same reason.
+- **The shipped organization starter** was rebuilt for name comparison version 5.
+- **Tests** (`test_cleaning.py`): every name that must change, and every one that must not; statuses; trade names and descriptors; features from the cleaned name; the trade-name level, its cap and relationship signal, and the term-frequency levels' places; a cut descriptor never exact or collapsed; numbered companies apart in the collapse key; the readable label; the record status in the mapping.
+
+### 13.4 Step 4: display names
+
+- **As built.** `engine.suggest_labels(entity_type, assistant=LabelAssistant(client, model), limit)` asks a model for a display name for live entities whose label is written all in capitals or all in lower case and that have no name set or suggested, twenty entities to a call, each with its eight most frequent spellings (`display_names.py`, prompt `suggest_labels` v1). Each answer is a `LabelSuggestion` decision citing its model call. It changes nothing. `engine.accept_label_suggestion(id)` labels the entity (its survivor, if it has been merged since) by a commit: an operator decision (`LABEL`) and an `ENTITY_LABELED` event carrying the name and the suggestion; `engine.label(entity, name, reason=...)` sets any name a person chooses; `engine.clear_label` undoes either (`ENTITY_LABEL_CLEARED`). Both events are identity events, so the entity's version and snapshot advance, the invariant suite counts them, and the snapshot written by any later commit takes the latest accepted name over the canonicalizer's. A merge keeps the survivor's name; an entity split off starts without one. `thisness labels suggest | list | accept [--all | IDS] | clear IDS`; in the Workbench, an entity with a suggestion shows it with "Use this name", one with a name set shows "Clear", and its history shows "Named ...".
+- **A suggestion must be a spelling of the entity's own names.** Its key, by the profile's value keys, must equal one of the shown spellings' keys, or it is not recorded: casing, spacing and the legal form's spelling may change, words may not. On the trial below the one rejection was the model correcting the typo in "perion limiqted", the entity's only spelling, to "Perion Limited": a better name, but a different key, and the rule is that a model never changes a name, only how it is written.
+- **Trial** (gpt-6-luna, the 548 UK entities of §13.3's label measurement; recording `~/thisness-demo/labels/suggest-gpt-6-luna.jsonl`): 300 entities asked in 15 calls for $0.0055; 299 suggestions, 1 rejected. Acronyms and initialisms stay in capitals ("DAE Aviation Holdings UK Limited", "GS Capital Partners UK LLP", "B2C2 OTC Ltd", "Haleon UK Finance (USD) Limited", "M.U. Enterprises Limited"); the rest reads in title case. Accepting every suggestion takes the entities whose label needs a name from 58% to 3% (the rest were beyond the trial's limit of 300, or the rejection).
+- **A correction to step 3.** Its rule marked a descriptor as cut when it was 22 characters or longer; a descriptor longer than the limit was plainly not cut ("PAYPAL *HALEON UK FINANCE LIMITED", 33 characters, was kept apart from its company). A descriptor is now cut only when it is exactly 22 or 25 characters long, trailing spaces counted. Re-measured on the messy names (recordings under `~/thisness-demo/cleaned3`): synthetic Fellegi-Sunter R .548 to .558, name-only plan unchanged at .964 (P .985, 6 false-merge clusters); UK Fellegi-Sunter unchanged at .364, name-only plan R .930 to .928 with 15 false-merge clusters against 13, within a model run's variation.
+- **Naming.** The new module is `display_names.py`: `labels.py` already holds review answers as training labels (design 04 §2.7), and `engine.labels()` returns those; accepted names are `engine.display_labels()`.
+- **Tests** (`test_display_names.py`): which labels need a name; a name as a commit, its version, history and clearing; a name kept through a merge and a split, not inherited by the entity split off, refused on a retired entity; suggestions recorded with their model call and changing nothing until accepted, and not asked twice; a suggestion that is not a spelling of the entity's names rejected; accepting after a merge names the survivor; the mapping export showing the name; the `labels` command; the Workbench's accept and clear.
+
+### 13.5 Closing: step 5 not built
+
+Step 5 was to be built only if the rules left a residue worth a model's cleaning (§4). They did not: on messy names the name-only plan reaches .964 recall on the synthetic benchmark, its clean level, and .93 on the UK one, and what remains is mostly card descriptors cut short, which no cleaning can complete, and the model's own judgments on them. Annotations are therefore not built. Two follow-ups remain open: guidance telling the model that a cut descriptor may name another company (§13.3), and an embedding candidate generator if a real supplier list shows pairs that share no blocking key (§7).

{thisness-0.1.0 → thisness-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "thisness"
-version = "0.1.0"
+version = "0.2.0"
 description = "Identity resolution for imperfect observations: evidence, stable identities, reversible history."
 readme = "README.md"
 license = "Apache-2.0"

{thisness-0.1.0 → thisness-0.2.0}/src/thisness/benchmark/organizations.py

@@ -17,7 +17,7 @@ Proportions follow the design 03 dataset; ``n_orgs`` scales the whole set.
 from __future__ import annotations
 
 import random
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, replace
 from typing import ClassVar
 
 SYLLABLES = [
@@ -323,10 +323,11 @@ def generate(n_orgs: int = 805, seed: int = 20261007) -> Benchmark:
     return Benchmark(seed, orgs, records)
 
 
-def names_only(benchmark: Benchmark, *, account_numbers: float = 0.25, seed: int = 0) -> Benchmark:
+def names_only(benchmark, *, account_numbers: float = 0.25, seed: int = 0):
     """The supplier-names benchmark (design 05 §7): the same organizations and records with
     every attribute but the name removed, and a vendor or account number appended to a
-    quarter of the names ("Harjas Foods PLC - 812773"), as in AP and ERP extracts."""
+    quarter of the names ("Harjas Foods PLC - 812773"), as in AP and ERP extracts. Works
+    on the synthetic benchmark and on the UK one (design 07)."""
     rng = random.Random(f"{seed}|names_only")
     records = []
     for r in benchmark.records:
@@ -334,5 +335,61 @@ def names_only(benchmark: Benchmark, *, account_numbers: float = 0.25, seed: int
         if rng.random() < account_numbers:
             number = rng.randrange(100_000, 999_999)
             name = rng.choice([f"{name} - {number}", f"{name} {number}", f"{name} #{number}"])
-        records.append(Record(r.record_id, r.org_id, {"name": name}))
+        records.append(replace(r, attributes={"name": name}))
+    if not isinstance(benchmark, Benchmark):
+        return replace(benchmark, records=records, variant="names_only")
     return Benchmark(benchmark.seed, benchmark.organizations, records, variant="names_only")
+
+
+PROCESSORS = ("PAYPAL *", "SQ *", "TST* ", "SP * ", "PP*")
+NOTES_BEFORE = ("ZZ ", "*INACTIVE* ", "XX-")
+NOTES_AFTER = (" - DO NOT USE", " (OLD)", " - DUPLICATE OF {n}", " INACTIVE", " - USE VENDOR {n}")
+STORES = (" STORE {n}", " #{n}", " LOC {n}", " UNIT {n}", " - BLDG {n}")
+REFERENCES = (" (VENDOR {n})", " REF {n}", " A/C {n}")
+MESS = (
+    ("processor", 0.25),
+    ("note", 0.25),
+    ("store", 0.2),
+    ("reference", 0.15),
+    ("trade_name", 0.15),
+)
+
+
+def _mess(name: str, kind: str, rng: random.Random) -> str:
+    n = rng.randrange(10, 99_999)
+    if kind == "processor":
+        # A card descriptor: a processor prefix, capitals, often cut at the descriptor's
+        # length, so no rule can restore the rest of the name.
+        descriptor = rng.choice(PROCESSORS) + name.upper()
+        return descriptor[:22] if rng.random() < 0.5 else descriptor
+    if kind == "note":
+        if rng.random() < 0.4:
+            return rng.choice(NOTES_BEFORE) + name
+        return name + rng.choice(NOTES_AFTER).format(n=n)
+    if kind == "store":
+        return name + rng.choice(STORES).format(n=rng.randrange(2, 9_999))
+    if kind == "reference":
+        return name + rng.choice(REFERENCES).format(n=rng.randrange(1_000, 999_999))
+    words = name.split()
+    trade = " ".join(words[: max(1, min(2, len(words) - 1))])
+    return f"{name} {rng.choice(('DBA', 'dba', 'T/A'))} {trade}"
+
+
+def messy(benchmark, *, share: float = 0.35, seed: int = 0):
+    """Names-only records as real extracts write them (design 08 §8): on ``share`` of the
+    records, one of a payment-processor descriptor (half of them truncated), an ERP
+    note ("ZZ ", "DO NOT USE", "DUPLICATE OF 10233"), a store or location suffix, a
+    reference number, or a trade name ("Acme Holdings LLC dba Acme"). Apply it to a
+    names-only benchmark."""
+    rng = random.Random(f"{seed}|messy")
+    kinds, weights = zip(*MESS, strict=True)
+    records = []
+    for r in benchmark.records:
+        name = r.attributes["name"]
+        if rng.random() < share:
+            name = _mess(name, rng.choices(kinds, weights)[0], rng)
+        records.append(replace(r, attributes={"name": name}))
+    variant = f"{benchmark.variant}+messy" if benchmark.variant else "messy"
+    if not isinstance(benchmark, Benchmark):
+        return replace(benchmark, records=records, variant=variant)
+    return Benchmark(benchmark.seed, benchmark.organizations, records, variant=variant)

thisness-0.2.0/src/thisness/benchmark/rewrite.py

@@ -0,0 +1,221 @@
+"""A rewrite-and-hash baseline for name-only resolution (design 08, measured first).
+
+The method common in spend-analysis pipelines, reproduced to be measured against the
+name-only plan (design 05 §6):
+
+1. Each distinct name is keyed by its letters and digits, upper-cased.
+2. One to three passes over overlapping chunks: a model rewrites every name in a chunk
+   to a cleaned, consolidated name, giving names it judges to be one organization the
+   same text. Pass 1 sees the names in input order; later passes see the previous pass's
+   names, ordered by key hash (as built) or by name.
+3. Each name's last answered rewrite (else the original), keyed the same way, is its
+   intermediate group: two records are one organization when their rewrites are equal.
+4. For each group, a model picks one standardized name from its variations; groups
+   whose standardized names key alike merge.
+
+Identity is string equality of model output: there are no pairwise comparisons, votes
+or reviews. The prompts here are written for this baseline.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import random
+import re
+from collections import Counter, defaultdict
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass, field
+from typing import Any
+
+from pydantic import BaseModel
+
+from thisness.benchmark.evaluate import ClusterMetrics, cluster_metrics
+from thisness.llm import BudgetExhausted
+from thisness.llm.prompts import Prompt
+from thisness.llm.providers.function import FunctionProvider
+
+_NOT_ALNUM = re.compile(r"[^A-Z0-9]")
+
+
+def key(name: str) -> str:
+    return _NOT_ALNUM.sub("", name.upper())
+
+
+class Rewrite(BaseModel):
+    ref: str
+    name: str
+
+
+class RewriteAnswer(BaseModel):
+    items: list[Rewrite]
+
+
+class Standardized(BaseModel):
+    name: str
+
+
+REWRITE = Prompt(
+    "baseline_rewrite_names",
+    "1",
+    system="""You clean and consolidate supplier names taken from purchasing and accounts-payable data.
+
+For every item, write a cleaned name:
+- Remove what is not part of the name: account, vendor, store or reference numbers; payment-processor prefixes; internal notes.
+- Fix spacing and casing, and expand obvious abbreviations (Mfg to Manufacturing) when that does not change the name.
+- Keep legal forms (Ltd, Inc, GmbH, PLC).
+
+Then consolidate: when items in the list very likely name the same organization, give them exactly the same cleaned name. Differences of punctuation, casing, spacing, abbreviation or small typos usually mean the same organization; a different word usually means a different one.
+
+Return every item once, by its ref, with its cleaned name.""",
+    user="{payload}",
+    response=RewriteAnswer,
+)
+
+STANDARDIZE = Prompt(
+    "baseline_standardize_name",
+    "1",
+    system="""You pick one standardized name for a supplier from the ways different records wrote it.
+Return the clearest full form of the name, with its legal form, in normal casing. Use only the variations given.""",
+    user="{payload}",
+    response=Standardized,
+)
+
+SIZES = (50, 75, 100)  # chunk size per pass
+OVERLAP = 10
+
+
+@dataclass
+class RewriteResult:
+    assignment: dict[str, str]  # record id -> group
+    metrics: ClusterMetrics
+    stats: dict[str, Any] = field(default_factory=dict)
+
+
+def _chunks(items: list[Any], size: int, overlap: int) -> list[list[Any]]:
+    step = max(1, size - overlap)
+    out, start = [], 0
+    while start < len(items):
+        out.append(items[start : start + size])
+        if start + size >= len(items):
+            break
+        start += step
+    return out
+
+
+def run(benchmark, client, *, model: str, passes: int = 3, order: str = "as_built",
+        concurrency: int = 16, shuffle_seed: int | None = 13) -> RewriteResult:  # fmt: skip
+    """Resolve a names-only benchmark by rewriting and hashing; score it like any run.
+    Records are shuffled first (``shuffle_seed``): the generators emit them organization
+    by organization, which would hand pass 1 every variant in one chunk; an AP extract
+    is in no such order."""
+    records = [(r.record_id, r.attributes["name"]) for r in benchmark.records]
+    if shuffle_seed is not None:
+        random.Random(shuffle_seed).shuffle(records)
+    display: dict[str, str] = {}
+    for _, name in records:
+        display.setdefault(key(name), name)
+    keys = list(display)  # input order: the first record carrying each key
+    calls = Counter()
+
+    def ask(prompt: Prompt, payload: dict) -> Any:
+        try:
+            answer = client.call(prompt, model=model, payload=payload)
+        except BudgetExhausted:
+            calls["budget"] += 1
+            return None
+        calls["calls"] += 1
+        if not answer.ok:
+            calls["failed"] += 1
+            return None
+        return answer.parsed
+
+    passes_out: list[dict[str, str]] = []
+    current = {k: display[k] for k in keys}
+    for p in range(passes):
+        if p == 0:
+            sequence = keys
+        elif order == "as_built":  # the next pass's input, sorted by its key's hash
+            sequence = sorted(current, key=lambda k: hashlib.sha256(k.encode()).hexdigest())
+        else:
+            sequence = sorted(current, key=lambda k: current[k].upper())
+        batches = _chunks(sequence, SIZES[min(p, len(SIZES) - 1)], OVERLAP)
+
+        def one(batch: list[str], current: dict[str, str] = current) -> list[tuple[str, str]]:
+            refs = {f"N{i + 1}": k for i, k in enumerate(batch)}
+            parsed = ask(
+                REWRITE, {"items": [{"ref": r, "name": current[k]} for r, k in refs.items()]}
+            )
+            if parsed is None:
+                return []
+            out = []
+            for item in parsed.items:
+                if item.ref in refs and item.name.strip():
+                    out.append((refs[item.ref], item.name.strip()))
+            return out
+
+        with ThreadPoolExecutor(max_workers=concurrency) as pool:
+            answers = list(pool.map(one, batches))
+        this_pass: dict[str, str] = {}
+        for answer in answers:  # first answer per key, in chunk order
+            for k, name in answer:
+                this_pass.setdefault(k, name)
+        passes_out.append(this_pass)
+        current = {k: this_pass[k] for k in current if k in this_pass}
+
+    # Intermediate groups: the last answered rewrite, else the original name.
+    group_of: dict[str, str] = {}
+    variations: dict[str, set[str]] = defaultdict(set)
+    for k in keys:
+        final = next((out[k] for out in reversed(passes_out) if k in out), display[k])
+        g = key(final) or k
+        group_of[k] = g
+        variations[g] |= {display[k], *(out[k] for out in passes_out if k in out)}
+
+    def standardize(g: str) -> tuple[str, str]:
+        names = sorted(variations[g])
+        parsed = ask(STANDARDIZE, {"variations": names})
+        return g, (key(parsed.name) or g) if parsed is not None else g
+
+    with ThreadPoolExecutor(max_workers=concurrency) as pool:
+        final_of = dict(pool.map(standardize, sorted(variations)))
+
+    assignment = {rid: final_of[group_of[key(name)]] for rid, name in records}
+    metrics = cluster_metrics(assignment, benchmark.truth, benchmark.kind)
+    stats = {
+        "names": len(records), "keys": len(keys),
+        "intermediate_groups": len(variations), "final_groups": len(set(final_of.values())),
+        "answered_per_pass": [len(o) for o in passes_out],
+        "calls": calls["calls"], "failed": calls["failed"], "budget_stops": calls["budget"],
+        "spent_usd": round(client.spent_usd, 6), "order": order, "passes": passes, "model": model,
+    }  # fmt: skip
+    return RewriteResult(assignment, metrics, stats)
+
+
+def oracle(benchmark) -> FunctionProvider:
+    """A model that consolidates only what it sees together, from the truth (design 05
+    §11.5). In a rewrite chunk, items of one organization get the name of the first of
+    them; an item alone with its organization keeps its name. Standardization returns a
+    group's own first variation, so it merges groups only when those coincide, as the
+    real runs found (1,094 groups before it, 1,092 after). A name is read as the organization most of its
+    records belong to, as a model cannot tell two companies of one name apart."""
+    owners: dict[str, Counter] = defaultdict(Counter)
+    for r in benchmark.records:
+        owners[key(r.attributes["name"])][benchmark.truth[r.record_id]] += 1
+
+    def org_of(name: str) -> str | None:
+        found = owners.get(key(name))
+        return found.most_common(1)[0][0] if found else None
+
+    def answer(request) -> dict:
+        payload = json.loads(request.messages[0].content)
+        if "variations" in payload:
+            return {"name": payload["variations"][0]}
+        first: dict[str | None, str] = {}
+        items = []
+        for item in payload["items"]:
+            org = org_of(item["name"]) or item["ref"]
+            items.append({"ref": item["ref"], "name": first.setdefault(org, item["name"])})
+        return {"items": items}
+
+    return FunctionProvider(answer, name="oracle")