trajectories 0.0.0 → 0.1.2

package/docs/research/ledger/docs/research/2026-09-16-c3-echo-filter.md

@@ -0,0 +1,305 @@
+# C3: the self-referential echo filter, 2026-09-16
+
+## Outcome
+
+An agent's own lookups of Ledger no longer come back as search hits. The
+transcripts of `ledger search` / `status` / `sync` / `scan` / `build` / `watch`
+and of the Ledger MCP tools are now **canonical but unsearchable**: `records`
+keeps every row exactly as the adapter wrote it, `records_fts` never receives
+them, and the store reports how many it keeps out (`ledger status`:
+`Echo filtered: 188 records (agent lookups of this store; never searchable)`;
+robot: `echo_filtered: 188`). The exclusion is one pure rule over three canonical
+fields — `record_type`, `tool_name`, `tool_arguments_json` — so ingest, the
+index re-derivation, the parity gate, and the status verb classify a record
+identically, and the rule change rides the C2 derive counter as `FTS_FORMAT`
+`"1"` → `"2"`: every store written before the filter re-derives itself once, on
+its next writer pass.
+
+On a byte-consistent copy of the production store (1,121,468,416 bytes; 227,474
+records; 2,604 sessions) the whole self-echo footprint is **188 records** — 94
+`assistant-tool-call`s and their 94 paired `tool` results, across 14 sessions
+(this morning's scout measured 40; the store grew fourfold while the cycles
+ran). The re-derivation costs **10.0–10.4 s** (whole-table re-projection
+5.76 s · rule pass 0.37 s · derived-set insert 1 ms · index subtraction 0.16 s ·
+parity + FTS5 integrity gates 2.82 s), parity is exact
+(`227,474 − 188 = 227,286`), and a query whose rank-1 hit was the agent's own
+previous identical search now returns only genuine records — with the genuine
+hits intact. This repository's own 21 development sessions (9,176 records)
+survive intact: every kept record is in the index (10/10 sampled by exact
+`record_id`), every one of their 146 lookups is not (0/10 sampled), and
+per-session `indexed = records − excluded` holds for all 21.
+
+Measurements are timestamped by fixture: 40 at the morning scout, 188 on this
+cycle's copy, 192 on the independent verifier's later copy — the store grows
+while the cycles run; the rule's zero-false-positive property, not the count,
+is the invariant.
+
+The tree's own state proved the upgrade path once for real before this audit: the
+production store already carried `fts_format = 2` and a `records_echo` of 188,
+and re-running the rule against those records reproduced that set **exactly**
+(188 stored, 188 recomputed, 0 on either side of the difference) — the shipped
+rule is what derived the real store.
+
+## The rule
+
+A record is a **lookup** (and so excluded from `records_fts`) when it is an
+`assistant-tool-call` that either
+
+- **names a Ledger MCP tool**: the name *ends* with a Ledger tool name at a word
+  boundary (`ledger_search`, `ledger_recent`, …; client prefixes vary:
+  `mcp__ledger__ledger_search`, `mcp_ledger_ledger_recent`, bare
+  `ledger_search`; `awake_search` and `xledger_search` do not match), or carries
+  the Claude Code server segment `mcp__ledger__` — the server's own namespace,
+  whatever the tool is called; or
+- **runs a Ledger binary in command position** in a shell tool's arguments: the
+  basename `ledger`, `ledger-core`, or `traj-bridge` (extension `.ts`/`.js`/
+  `.exe`, any path form) as the first word of a command, or the word after a
+  separator (`&&`, `||`, `;`, `|`, `&`, newline), a command substitution
+  (`$(`, backtick), a prefix word (`sudo`, `env`, `time`, `nohup`, `nice`,
+  `exec`, `command`, `then`, `do`, …), or a runner (`bun run ./bin/ledger.ts`).
+
+A `tool` record is excluded only through its **call**: `tool_call_id` paired to
+an excluded call in the same session — never a guess from the result text.
+
+Everything else is kept, deliberately: user and assistant dialogue (however much
+it talks about Ledger), `Read`/`Edit`/`Grep`/`Write` calls, a `git commit -m
+"…ledger…"`, a `cd` into this checkout, `bun test`, `cat ledger.ts`, `grep
+ledger`, a quoted alternation (`rg -n "ard|ledger|review"`), a heredoc body that
+begins `ledger search …` (a document, not a run), and any command whose
+`ledger` is an argument. Reads and writes are treated alike (`status`, `sync`,
+`scan`, `build`, `watch`, `--help`, `--version` all count): an agent dumping the
+store's own counters and cursors into the corpus pollutes it exactly as a search
+does. Quoting is lexical — a separator inside quotes is text — and heredoc
+bodies are removed before segmentation.
+
+Two shapes are knowingly not seen, and both err toward indexing: a Ledger
+invocation from *inside* a script (a `subprocess.run(['ledger', …])` in a python
+heredoc — measured: **3** of 4,728 declined candidates), and a runner this rule
+does not model (`node`, `npx`). They cost a noisy hit, never a hidden record.
+Wake makes the same trade with a preview fallback (`command_names_binary`) this
+port declines: when a shell tool carries no structured arguments there is
+nothing here to read, so the call stays indexed — measurably free, since **0**
+shell-tool calls in the store carry null arguments.
+
+## Files
+
+- `packages/core/src/filter.ts` (new) — the rule, and the three forms the store
+  uses it in: `isEchoCall` (pure), `echoExcludedIds` (one streaming pass over
+  `records`, used by the re-derivation, the parity gate, and the status count),
+  `echoExcludedBatchIds` (an ingest batch, resolving a result whose call sits in
+  an earlier batch through the same rule, and returning the call ids the writer
+  needs to repair a result committed before its call).
+- `packages/core/src/schema.ts` — `FTS_FORMAT = "2"`; the `records_echo` DDL
+  (derived set, both its columns cascading on delete); `DERIVE_COUNTERS` grows an
+  `outputs` list so a counter that *matches* but whose output table is missing
+  still re-derives.
+- `packages/core/src/db.ts` — open-time counter detection against the tables the
+  store already held (before the DDL creates the ones this build needs);
+  `insertRecordsInternal` writes canonical rows first, then consults
+  `echoExcludedBatchIds`, skipping the FTS row and writing the id into
+  `records_echo` — and taking back any already-committed result that pairs with
+  a call the batch just classified (index row deleted, id marked), so arrival
+  order cannot decide what a pair derives to; `rebuildRecordsFts` re-projects the whole index, recomputes the
+  rule set, rewrites `records_echo`, subtracts exactly those rows, and runs the
+  gates; `verifyRecordsFtsRebuild` checks `records − excluded == indexed` plus
+  “no excluded id without its record”.
+- `packages/core/src/cli.ts` — `build` takes the pending re-derivation before its
+  import loop (and reports the wall time); `status` publishes `echo_filtered`
+  in both modes.
+- `packages/core/src/watch.ts` — a drain takes the pending re-derivation under
+  the lease it already holds, before its batch, reporting the wall on stderr.
+- `packages/core/src/store-status.ts` — the status field, published only when
+  the derivation under this build's rules has completed (see the gap below).
+- `packages/core/tests/echo-filter.test.ts` (new) — the rule's positive and
+  negative table (15 lookups / 17 not-lookups), ingest
+  (canonical completeness + index exclusion + status count), the C2 bump
+  (flag → one-shot → parity → exactly once, including the status count's `null`
+  while the derivation is pending), and the counter-without-output state.
+- Re-pinned to the new contract: `cli-canonical.test.ts` (`echo_filtered`),
+  `derive-counters.test.ts` (the refusal message now formats `FTS_FORMAT`),
+  `integration-migrations.test.ts` (the parity message names the excluded term).
+
+## Ported mechanics (Wake MIT, `iAmCorey/Wake@d9f702b6` tag v0.6.5 — mechanics with attribution, no code copied)
+
+Pin: `d9f702b6df0e85` (`Release v0.6.5`), `crates/wake-core/src/{adapters/mod.rs,db.rs}`,
+MIT License © 2026 Corey Chiu.
+
+| Ledger | Wake pin | What was kept |
+| --- | --- | --- |
+| The lookup is dropped from the derived unit; dialogue is not | `adapters/mod.rs#L425-L447` (`units_from_messages`) | a tool call that *is* a lookup contributes nothing to the unit; the message text around it stays, and a message with nothing left is dropped whole — Ledger's analogue is record-level: the call and its paired result leave the index, every other record stays |
+| MCP tools by word-boundary suffix, not substring | `#L469-L535` (`is_wake_lookup`, `ends_with_word`) | client prefixes vary, so the name must *end* with the tool name at a word boundary (`awake_search` is not `wake_search`); the server segment (`mcp__ledger__`) is kept as its own stronger signal |
+| Shell binaries named in command position of the structured `command` field | `#L469-L535` (`WAKE_BINARIES`, `command_names_binary`, `names_binary`) | the structured field is read for *any* tool that carries one (string or argv array), never a substring of the preview; the command is segmented so a binary in argument position (`grep ledger`) is not a run, and a `wake-cli-old`-style near-miss is not a name |
+| The narrowness is a test, not a hope | `#L599-L653` (`wake_lookup_detection_is_narrow`): `cargo test` not a lookup; `man wake-cli-old` not, even with its description mentioning the binary; `Grep`/`Read` not | `echo-filter.test.ts` ports the shape: our own `bun test`, a commit message, a checkout, a quoted alternation, a heredoc body, a `Read` of `ledger.md`, an `Edit` of `ledger.ts` — all kept |
+| A derivation rule change bumps an index-format counter; an old store flags and re-derives once | `db.rs#L134-L196` (`FTS_FORMAT`, whose doc says “'2' = 过滤自指回声”; `open_conn` stamps and flags when `stored != FTS_FORMAT` and sessions existed) | exactly the C2 machinery, consumed: the value is the rule's name, the flag is transient and durable across a crash, a reader serves the stale index while a writer re-derives. C3 is the first bump (below) |
+
+## Deviations from the scout design (each with its reason)
+
+- **The filter is a derived set (`records_echo`), not a SQL predicate inside
+  `FTS_REPOPULATE_SQL`.** The scout allowed “pure SQL or a derived marker”; this
+  is the marker, because the rule lives in TypeScript and a SQL restatement of it
+  would be a second implementation that can drift — the one thing a
+  self-referential filter must not do. The set is also what makes ingest and the
+  re-derivation *one decision*: the ingest path writes the id where it skips the
+  FTS row, the rebuild recomputes it from the same rule and subtracts it, the
+  parity gate and the status count read the same table back. The projection stays
+  whole-table on purpose — the rebuild is then a *repair*: re-project every
+  record, drop exactly what the rule names, and an index that drifted in either
+  direction comes back equal to `records` minus the filter.
+- **The parity gate gained a second term, and an orphan check.** `records −
+  excluded == indexed` (scout) plus “every id in `records_echo` still backs a
+  record”. The foreign key already forbids an orphan, so the check exists to
+  catch the one thing the key cannot: a writer running with enforcement off.
+- **A matched counter with a missing output re-derives.** `DERIVE_COUNTERS`
+  records the tables each rule derives, and a store whose counter names the
+  current rule but which cannot produce the rule's table is treated as an
+  unfinished derivation, not a derived store. This is beyond the scout sketch
+  but falls out of the second output: without it, a store that recorded `"2"`
+  and lost `records_echo` would report `0` filtered forever. The matching value
+  is left in place — it is not the counter that was wrong, the derivation is
+  incomplete — and the flag is raised for the next pass.
+- **The rule is broader in shell shapes than the sketch.** Beyond first token /
+  separator / prefix word (`ledger/ledger-core/traj-bridge`, `bun run …`), the
+  port reads quoted strings correctly (`rg -n "a|ledger|b"` is one command
+  running `rg`), strips heredoc bodies, segments on newlines, `$(`, and
+  backticks, joins `bash -lc` argv arrays, and reads the Codex `cmd:"…"` strings
+  inside an `exec` input. Each extension is in the conservative direction —
+  structural command position, never a substring — and is pinned in the test
+  table.
+- **`build` and `watch` take the pending re-derivation too.** C2's boundary note
+  left exactly this to C3 (`sync` was the only pass): a daemon-only machine
+  would otherwise keep serving an index derived under the old rules until some
+  sync ran. The watcher reports the wall time on stderr; `build` reports it in
+  its summary.
+- **No per-query filtered count.** Per the contract and Wake: the envelope is
+  unchanged; the count lives where it is a property of the *store*, not of a
+  query.
+
+## Audit against the scout design
+
+| Contract item | State |
+| --- | --- |
+| (a) MCP tool naming, suffix + server namespace | as designed |
+| (b) shell binary in command position (`&& ; \|\| \| sudo time env`, `bun` runner) | as designed, extended (quotes, heredocs, substitutions, wrappers, argv, embedded `cmd:`) |
+| (c) result paired by `tool_call_id`, same session | as designed; batch form resolves a call from an earlier batch through the store, store-wide form buffers results so source order cannot matter |
+| never filter dialogue / read / edit / grep; reads *and* writes are lookups | as designed; 4,728 LIKE-superset candidates declined, spot-audited (checkout, commit message, `bun test`, tool reads/writes, quoted alternation, heredoc bodies) |
+| record-level granularity; `records` canonical; `sessionRecords`/export untouched | as designed; fixture proves all 7 rows stay; 10/10 sampled dev records still indexed |
+| derivation-time placement: ingest skip + re-derivation exclusion | as designed, via the marker table (deviation above) |
+| `FTS_FORMAT` `1` → `2`, one re-derivation per store | as designed |
+| parity `records − excluded == indexed` | as designed, plus orphan check |
+| `echo_filtered` in `status` robot output | **gap → fixed**: the count was published from the derived set alone, so a store whose derivation was *pending* (the critical-pressure skip `sync.ts` documents, or a crash between open and the pass) reported `0` while the served index still held the lookups. `echo_filtered` is now published only when `fts_format` equals this build's **and** no `fts_reindex` is pending; otherwise `null` (“unknown … the next writer pass derives it”) |
+| daemon paths wired (`build`, watch drain) | as designed (C2's boundary #1 taken) |
+| envelope: no per-query filtered count | as designed (`reader.ts` untouched) |
+
+## Review round 2 (adversarial findings → fixes)
+
+An adversarial review of this tree found four issues; all four are fixed here,
+with regression tests for the first three and the API-surface document updated
+for the fourth. None of them changes the design; three are ordering/coherence
+holes in the rule's edges, one is documentation drift.
+
+| # | Finding | Fix | Test |
+| --- | --- | --- | --- |
+| 1 (P2) | The batch form never repaired an **indexed result whose call arrives later**: a result committed in an earlier batch stayed in `records_fts` forever (the marker set cannot see it, and the parity gate trusts the set, so `records − excluded == indexed` still held). | On a batch that classifies a call, the same transaction now takes back the committed `tool` rows pairing those `tool_call_id`s — delete from `records_fts`, `INSERT OR REPLACE` into `records_echo` — so arrival order cannot decide what a pair derives to. Rows of the batch itself are touched idempotently (the delete matches nothing yet; the marker insert precedes the loop's own). | `a result committed before its call is repaired when the call arrives in a later batch` (both orders) |
+| 2 (P2) | `bash -lc "ledger search x"` was a **false negative**: the wrapper's quoted script reaches `segmentWords` as one word, and the quote-stripping `basename` then rejected it for containing spaces — while the argv form and a script with an internal separator were detected. Incoherent for the most common wrapper shape. | The extracted script is quote-stripped and judged by `commandSegments` (the same segmentation as every other command string), which also makes quoting *inside* the script meaningful — a quoted alternation (`bash -lc "rg -n 'ard|ledger|review' docs"`) is no longer split raw and mis-read as a run. | `a shell wrapper's quoted -c script is one more command string` |
+| 3 (P3) | `echoCallOf` hardcoded `'assistant-tool-call'`, so a `tool` row carrying a Ledger tool name was excluded at ingest and then **flipped back into the index** on the next re-derivation — the batch and store-wide forms disagreeing about one row. | The record's own `record_type` decides (`echoCallOf` is now the single place the rule meets the row, used by both forms), and the store lookup selects `record_type` it had previously only typed. A tool row is never a call: an orphan result with a Ledger tool name stays indexed, errs toward indexing, and is stable across a re-derivation. | `a tool result that carries a Ledger tool name is not a call, at ingest or after a re-derivation` |
+| 4 (P3) | `docs/api-surface.md` was stale where robots read the contract: the `status` envelope lacked `echo_filtered` (and its `null` semantics), §3.1 still called `fts_format` `"1"`, and the re-derivation was described as `sync`'s alone. | Updated, plus the C3 surface the same file now has to describe: `build`'s summary `deriveMs` and its `Re-derived` prose line, the `records_echo` table (new §3.2 entry), the parity count's excluded term, and both counter sections naming `sync`/`build`/`watch` as the executors. | — (documentation; covered by the two code tests above) |
+
+Gates after the round: **`bun test` 568 pass / 0 fail across 39 files**,
+`bun run typecheck` clean, `bun run build` clean.
+
+## Measurements
+
+Host: Apple M1, macOS 25.6, APFS. Fixture: a byte-consistent copy of the
+production store (`sqlite3 …?mode=ro ".backup /tmp/ledger-c3/ledger.sqlite"`;
+1,121,468,416 bytes, 227,474 records, 2,604 sessions, 0 embeddings) — the
+production store is never opened for writing, and the measurement harnesses
+(throwaway scripts under `/tmp/ledger-c3`) were deleted afterwards.
+
+The store was found **already derived** (`fts_format = 2`, `records_echo` = 188,
+`records_fts` = 227,286) — the C3 build had run against it during development.
+The “before” state below is therefore reconstructed from the store's own
+projection: drop `records_echo` and `records_fts`, recreate the v11 DDL,
+re-project every record whole (`5,783 ms`), set `fts_format = '1'` — a
+bit-consistent stand-in for what a pre-C3 build leaves.
+
+**OBSERVED**
+
+| measurement | value |
+| --- | --- |
+| echo set, production store | **188 records** = 94 calls + 94 results, in **14 sessions** (omp 146, codex 40, claude-code 2); the calls are `bash` 73, `exec` 20, `Bash` 1 |
+| scout's count this morning vs now | 40 → **188** (this repo's own sessions supplied the growth) |
+| rule pass over the copy (`echoExcludedIds`) | **2,979 ms** first (cold) read, **372–500 ms** warm |
+| stored set vs fresh rule computation | 188 vs 188, **0 differences** either way (the real store's derivation is exactly the shipped rule) |
+| false positives | **0** across all 188: every call runs a Ledger binary in command position or names an MCP Ledger tool; every result pairs to a call in its own session. An independent separator-anchored pattern matched **92/94**; the two exceptions are path-prefixed runners (`/Users/mac/.bun/bin/bun …/ledger-core.ts`), which the rule normalizes by basename and the check's pattern did not |
+| declined candidates (LIKE superset the rule keeps) | **4,728**; among them 3 script-embedded invocations (documented blind spot) and 7 direct `sqlite3` reads of the store file (kept by design, not lookups) |
+| dev sessions of this checkout (`cwd` match) | **21 sessions, 9,176 records, 146 excluded**; per-session `indexed = records − excluded` for **all 21**, 0 mismatches; exact-membership sample: **10/10 kept records in `records_fts`, 0/10 excluded** |
+| writer open of the pre-C3 copy | **3 ms**; `needsFtsReindex()` true (stored `"1"` ≠ `"2"`) |
+| `reindexRecordsFts()`, four runs | **10,049 / 10,051 / 10,162 / 10,374 ms** |
+| phase breakdown (same statements, timed individually) | repopulate (drop + DDL + `INSERT … SELECT`) **5,759 ms** · rule pass **372 ms** · derived-set insert **1 ms** · index subtraction (188 deletes) **159 ms** · parity + FTS5 `integrity-check` **2,818 ms** |
+| state after | `records` 227,474 · `records_fts` 227,286 · `records_echo` 188 · parity true · `verifyRecordsFtsRebuild` passed · `schema_meta`: `fts_format=2`, `embeddings_format=1`, flag cleared |
+| searches, before → after (totals bounded at 1000; marker = the rule's set) | `search_meta` **95 → 91** (4 echo results in the top 200 → 0) · `coverage` 1000 → 1000 (**16 → 0**) · `canonical trajectory` **30 → 28** (2 echo calls in the top 5 → 0; the previous identical search was rank 1) · `writer lease` **230 → 229** · `record_candidate` **45 → 43** (echo at rank 4 → gone) · `rebase token passthrough` **3 → 2** (echo at rank 1 → gone) · `ledger` 1000 → 1000 (**2 → 0**). Every non-echo hit stayed; the top-5 list of `canonical trajectory` went from `[ECHO, real, real, ECHO, real]` to five genuine tool records |
+| CLI, copy | `status --robot` → `"echo_filtered":188`; human → `Echo filtered: 188 records (agent lookups of this store; never searchable)`; `search "canonical trajectory" --limit 3` → 3 genuine hits across 2 sessions, no echo |
+| gap fix, verified live in the reconstructed pending state | raw set 0, served index still returning the rank-1 echo record, `status.echoFiltered` → **null** (was `0`); after the one-shot → **188** and parity |
+| existing suite | **565 pass / 0 fail** across 39 files (`bun test`), `bun run typecheck` clean, `bun run build` clean — the round-1 tree; after the review round's three regression tests the same gates are **568 pass / 0 fail**, clean (see “Review round 2”) |
+
+**INFERRED** (not directly measured, listed so reviewers can attack them)
+
+- The 10.0–10.4 s band is **warm** and idle-host: the copy had been fully read
+  and rewritten several times before the timed runs. C2 measured the same
+  re-projection path at 19.4–33.3 s on the older store; the C3 addition is only
+  the rule pass + subtraction (~0.5 s of the 10 s), so the difference between the
+  two bands is page cache and host load, not a different mechanism. The first
+  cold post-merge pass on a user's machine should be expected in the C2 band.
+- The status gate's *motivation* is derived from code, not from a failing
+  scenario observed in production: the pending state is reachable by design
+  (`sync` skips the one-shot under critical pressure, and the flag is durable)
+  and by crash between open and pass; only the reconstructed window was
+  measured. Its other reachable case — an older build's writer pass writing the
+  index back whole, leaving `fts_format` different from this build's — is the
+  version-crossing cost C2 already documents as “one more re-derivation”; the
+  gate now reports `null` instead of a stale count while it lasts.
+- Any *further* change to the rule with `FTS_FORMAT` left at `"2"` would not
+  re-derive anything: the set is derived state keyed by the counter, not hashed
+  from the rule. The bump is the contract (as in Wake); the follow-up below
+  records it.
+- The vectors lane is unaffected and cannot currently surface echo records: the
+  store holds 0 embeddings, so semantic/hybrid search has nothing to rank. If
+  the lane ships, a lookup's embedding would need the same exclusion — the set
+  is the natural input, and no decision is needed until then.
+
+## Position in the program (C3 row, `docs/research/2026-09-16-wake-port-program.md`)
+
+- Failure it fixes: *“We index the transcripts of the agents that call `ledger
+  search` — your own queries come back as hits. Unnamed bug, observed daily.”*
+- Open-source case: *“Self-hosting agents are our primary users; a store that
+  echoes itself reads as broken. Wake ships the same filter (`wake_*` tools,
+  `FTS_FORMAT` counter).”*
+- Ordering: C3 follows C2 because it changes index shape and needs the one-shot;
+  it bumps `FTS_FORMAT` and is the first real consumer of the counter, plus the
+  first cycle to take C2's own boundary — the daemon passes now re-derive too.
+  C4 and C5 will bump the same counter and can rely on the marker-table pattern
+  introduced here (a derived set beside the index, maintained by the same
+  decision that writes the index).
+
+## Boundaries and follow-ups (not in this cycle)
+
+1. **In-script invocations stay unfiltered** (3 measured: python heredocs
+   running `subprocess.run(['ledger','sync',…])`), as do unmodelled runners
+   (`node`, `npx`) and — unlike Wake — an unstructured shell tool whose only
+   text is a preview. All err toward indexing; the test table pins the current
+   shape, so widening it is a deliberate change plus a bump.
+2. **Rule changes need the bump.** The rule's identity is `FTS_FORMAT`, not a
+   hash of `filter.ts`. A future edit to the rule without a bump leaves every
+   store's set as-is; the cycle that edits it must bump.
+3. **The vectors lane, if it ships, inherits the question**: a lookup's
+   embedding would be rankable while its text is not. The natural input is
+   `records_echo` (delete the embeddings with the record, or exclude at query
+   time); not decided here, because the store holds no vectors today.
+4. **`status` pays one extra presence check + one `schema_meta` row read** on a
+   store that has the table (both trivial, both read-only); the read path
+   (`search`, `recent`, …) is untouched, and the envelope carries no filtered
+   count by design.
+5. **The echo set is one row per excluded record.** At 188 rows for 227k records
+   it is nothing; a store where lookups dominate would grow it linearly with
+   their count (one row per pair, two small text columns), and the rebuild's
+   subtraction step is bounded by the set size, not the store.