trajectories 0.0.0 → 0.1.2

package/docs/research/ledger/docs/research/2026-09-16-c4-title-index.md

@@ -0,0 +1,349 @@
+# C4: the title index and title-first ranking, 2026-09-16
+
+## Outcome
+
+A session can be found by its name. `titles_fts` is a second FTS5 index beside
+`records_fts` — one row per session whose title the store holds — and a lexical
+query now runs two lanes: the title lane first, then the record lane, with both
+sharing `--limit`. A title hit is *session-level* and says so: `match_type:
+"title"`, `record_id: null` (no record is fabricated for it, Wake's rule),
+`record_type: "title"`, the whole title as the snippet with the matched phrase in
+`«»`, and the session's `started_at` as its timestamp. The lane rides every
+surface — human output (`Role:  title`), `--json`, robot v1 and v2 — because
+`--output-version` selects formatting, never lanes.
+
+The query the lane exists for, on a byte-consistent copy of the production store:
+`"Hermes Feishu Thread Reply Behavior"` (a session's own name, from a harness
+that titles its sessions) went from **zero genuine matches** — the zero-hit
+widening path substituted five `implicit_wildcard` records, none of them that
+session — to **rank 1**, one `title` hit naming it. A common word (`Feishu`) now
+ranks the sessions that carry it in their *name* above the records that merely
+mention it, and a body-only query (`wal_autocheckpoint`) returns the same top
+hits it did before: the record lane is untouched.
+
+The index is derived state on the C2 counter, so the change ships as
+`FTS_FORMAT` `"2"` → `"3"`: every store written before it re-derives itself once
+on its next writer pass, in the *same* transaction that re-projects
+`records_fts`. On the production copy (1.125 GB, 229,103 records, 2,609
+sessions) that one-shot cost **15.8 s** — the `records_fts` re-projection C2/C3
+already measured — of which **9.4 ms** was the title rebuild and **7.3 ms** the
+title gate. The store's own numbers make the case concrete: **2,609 of 2,609
+sessions carry a title** (277,385 characters, 859 distinct titles) and **2,348 of
+them (90%) are a prefix of one of their own user records** — the title *is* the
+first instruction, which the record index could only ever find as text inside
+that record, never as the session's name.
+
+## The mechanism (five pieces)
+
+**1. A standalone index, not a column on `records_fts`.**
+`titles_fts(session_id UNINDEXED, title, tokenize="porter unicode61")` — one row
+per session with a non-empty title; `NULL` and `''` are the same non-title.
+A title belongs to the session, has no record id to point at, and is derived from
+harness metadata or the first user message, so folding it into the record index
+would mean either a fabricated row (inventing a record no adapter wrote) or the
+same title indexed once per record of that session. Wake splits the same way and
+for the same reason (`db.rs#L126-L130`: `titles_fts(key UNINDEXED, title)`, the
+key carried only to find and delete the row — "a title is not a transcription
+message and has no `seq` to pair with"). Our tokenizer is the record index's, not
+Wake's trigram: this is the trigram-free first pass, and the tokenizer decision
+is C5's (docs/research/2026-09-16-wake-port-program.md).
+
+**2. Maintenance at every write point, as triggers on `sessions`.**
+The index is a projection of one canonical column, so the rule that keeps it
+equal to that column is a predicate rather than a decision — the case where a
+trigger is right and a call site is not. Three triggers
+(`trg_sessions_titles_insert` / `_update` / `_delete`, in `CREATE_TABLES_SQL`, so
+every store gets them on its next writer open) cover `saveSession`'s three paths,
+`appendSessionRecords`, the staged promotion, and any raw connection that
+bypasses the helpers — including one running with foreign keys off. The update
+trigger fires only when the title actually changed (`IS NOT`, so an absent title
+is compared too) and re-indexes that session's one row by delete-then-insert,
+which is Wake's shape at its single write point (`db.rs#L1968-L1976`: `DELETE
+FROM titles_fts WHERE key = ?` then an insert when the title is not empty). A
+deleted session's title row goes with it: no foreign key can enforce that for a
+virtual table, which is the same reason `sessions` already carries
+`trg_sessions_diagnostics_delete`.
+
+**3. Re-derivation in the counter's one transaction, with its own gate.**
+`reindexFts()` (renamed from `reindexRecordsFts`: the counter names the *rules*,
+and one rule version is one derivation) now rebuilds both indexes — drop,
+recreate, re-project `records_fts` from `records`, recompute and subtract the
+echo set, repopulate `titles_fts` from `sessions` — and gates both before it
+clears the flag. The title gate is exact parity
+(`COUNT(titles_fts) == COUNT(sessions WHERE title != '')`, against the
+repopulation's own predicate, kept as one exported constant), an orphan check
+(every indexed row still names a live session — the one thing a virtual table
+cannot enforce for itself), and FTS5's `integrity-check`. Because the triggers
+live on `sessions`, dropping and recreating the index leaves them in place. A
+store whose counter matches but whose `titles_fts` is missing was not derived by
+these rules, so `DERIVE_COUNTERS[0].outputs` names the new table and the flag is
+raised exactly as it is for a moved counter.
+
+**4. Two lanes, one limit.**
+`queryLexical` probes for the index (a store whose derivation is pending has
+none), runs the title lane first, and emits its hits ahead of the record lane's.
+The title lane is bounded by the caller's limit and the record lane gets what is
+left, so `totalHits` stays what `meta.count_precision` describes — a title hit
+displaces a record hit at the boundary rather than extending the result past the
+limit. Ordering is `bm25(titles_fts) ASC, s.started_at DESC`: the same shape the
+record lane uses. Wake multiplies this lane's bm25 by a time decay
+(`db.rs#L1676-L1700`, `RECENCY_BOOST` at `#L1552`); the decay is C6's cycle, so
+this lane keeps the ordering every other query in the store has. Widening stays a
+*whole-query* zero-hit recovery: it runs only when both lanes found nothing, and
+it widens the record lane (a title is matched by its words or not at all).
+
+**5. The lane never invents a record, and the echo filter has no opinion about
+titles.**
+`record_id: null`, `record_type: "title"`, `content: null` — Wake's `seq: 0` /
+`role: "title"` rule (`db.rs#L1749-L1757`: the title hit jumps to the session's
+start because there is no transcription row to jump to, and no `seq` is
+faked). In the type system this is a union, not a nullable field
+(`RecordSearchHit | TitleSearchHit`), so a consumer that needs a record is forced
+to narrow rather than silently reading a null.
+
+Titles stay searchable for **every** session, including sessions whose records
+the self-referential echo filter keeps out of `records_fts` (C3): a title is
+provenance metadata — who asked what — not echo content, and the filter is a
+record-level rule. An agent's own lookup sessions remain findable by name while
+the lookups themselves stay unindexed; `tests/title-index.test.ts` pins exactly
+that pair.
+
+## Files
+
+- `packages/core/src/schema.ts` — `FTS_FORMAT = "3"`; `TITLES_FTS_DDL`,
+  `TITLES_FTS_REPOPULATE_SQL`, `TITLES_FTS_REBUILD_SQL`,
+  `TITLES_FTS_TRIGGERS_SQL`, `INDEXABLE_TITLE_SQL` (the one indexability
+  predicate: repopulation, triggers, and gate all apply `title != ''`);
+  `DERIVE_COUNTERS[0].outputs` grows `titles_fts`; `CREATE_TABLES_SQL` creates the
+  table and its three triggers.
+- `packages/core/src/db.ts` — `rebuildTitlesFts()` (rebuild + gate) beside
+  `rebuildRecordsFts()`; `reindexFts()` runs both then clears the flag;
+  `configureFtsAutomerge` pins the same automerge policy for both indexes;
+  `verifyTitlesFtsRebuild` is the exported gate (parity, orphan, integrity),
+  module scope so tests can drive it against a doctored store.
+- `packages/core/src/types.ts` — `SearchHit` becomes
+  `RecordSearchHit | TitleSearchHit`; `RecordMatchType` and `SearchMatchType`
+  (`… | "title"`); `TitleSearchHit` documents the no-fabrication rule.
+- `packages/core/src/reader.ts` — the title lane (`titlePrepass`, presence probe,
+  `highlight(titles_fts, 1, …)`), the two-lane budget in `queryLexical`, the
+  title pre-pass ahead of RRF in `queryHybrid`; `escapedTermExpression` is shared
+  by both lanes so a fallback expression cannot drift between them;
+  `sessionsCount` is now computed once from the returned hits instead of being
+  threaded through each engine.
+- `packages/core/src/sync.ts`, `cli.ts`, `watch.ts` — the renamed one-shot call
+  sites, and the re-derivation prose now names both indexes.
+- `packages/core/tests/title-index.test.ts` (new) — the index's parity and repair
+  (including the two gate refusals), title-first emission with the session-level
+  hit's fields, the shared limit, the source filter and the recency tiebreak, the
+  echo interplay, the rename/append/delete and staged-promotion maintenance, the
+  format bump (pending state tolerated → one-shot → parity → exactly once), the
+  hybrid pre-pass with a real vector leg, and the robot/human surfaces through
+  the CLI.
+- Re-pinned to the new contract (each because the assertion's subject is the
+  *record* lane, which now has a second lane beside it): `derive-counters`,
+  `ingest-transaction`, `ingest-staging`, `sync`, `watch`, `e2e` (record-lane
+  counts), `cli-skill-wrappers` (the stream now carries the title hit),
+  `api.test.ts` (the robot hit shape holds for either lane), `search-quality`
+  (the fusion's provenance filter narrows on `record_id`).
+- `docs/api-surface.md` — §1.4.4's lane contract (fields, budget, v1/v2, the
+  widening interaction), §3.1's `titles_fts` table, the `"3"` counter text, the
+  `REQUIRED_READ_SCHEMA` note, the renamed method, and a C4 ADR.
+
+## Ported mechanics (Wake MIT, `iAmCorey/Wake@d9f702b6` tag v0.6.5 — mechanics with attribution, no code copied)
+
+Pin: `d9f702b6df0e85` (`Release v0.6.5`), `crates/wake-core/src/db.rs`, MIT
+License © 2026 Corey Chiu.
+
+| Ledger | Wake pin | What was kept |
+| --- | --- | --- |
+| A standalone title index keyed by session, the key UNINDEXED | `db.rs#L126-L130`, and the comment above it (`#L124-L125`) | a title is not a record and has no `seq` to pair with, so it gets its own table rather than a fake row; the key is carried only to find and delete the row |
+| Title hits emitted ahead of body hits, at session level | `db.rs#L1676-L1700` (the title query, `ORDER BY bm25(titles_fts) * RECENCY_BOOST LIMIT ?`) | the lane runs first and its hits precede the record lane's; the query is session-level and joins `sessions` for the metadata the hit reports |
+| `seq: 0`, `role: "title"`, never a fabricated `seq` | `db.rs#L1749-L1757` | our analogue is `record_id: null` + `record_type: "title"`: the hit jumps to the session's start because there is no record to jump to, and none is invented |
+| One write point keeps the index equal to the title column | `db.rs#L1968-L1976` (`upsert_session`: delete by key, insert when non-empty) | we make the write point total instead of single by expressing it as triggers on `sessions`, so every path — and every raw connection — maintains the pair |
+| A rule change bumps the index-format counter and the old store flags itself | `db.rs#L133-L136` (`FTS_FORMAT`, doc: "改了派生规则就换个值"), `#L176-L196` (compare at open, flag when sessions existed) | the C2 machinery, consumed a second time: `FTS_FORMAT` `"2"` → `"3"`, exactly one re-derivation per store |
+| The derivation is the *index's* whole re-projection | `db.rs#L212-L223` (an empty index beside non-empty sessions is repopulated in one `INSERT … SELECT`), `tests/db_roundtrip.rs#L1015-L1038` (`title_index_is_backfilled_for_older_databases` simulates the table just created and asserts the titles are searchable again) | the same `INSERT … SELECT` from the sessions table is what our one-shot runs — but as the counter's derivation rather than an open-time backfill (see deviations) |
+
+## Deviations from the scout design (each with its reason)
+
+- **Triggers instead of maintenance calls in the upsert paths.** The scout
+  sketched "`saveSession`/append paths/`commitStagedSession` maintain `titles_fts`
+  rows in the same transaction (C3's `records_echo` shows the pattern)". The tree
+  has both patterns — `records_echo` is written by the ingest decision that skips
+  the FTS row, and `trg_sessions_diagnostics_delete` exists because a session's
+  rows must not outlive it for *any* writer. Titles have no decision to make
+  (the rule is a column predicate) and three write paths plus one delete path to
+  cover, so the total form is the trigger set: it leaves `saveSession`,
+  `appendSessionRecords`, and `promoteStagedIngest` untouched, cannot be missed by
+  a future path, and makes an orphaned title row impossible rather than merely
+  detected. The gate still checks for orphans, because a store can be written
+  with the triggers dropped.
+- **The lane is shared with the record lane's limit.** Wake emits title hits up to
+  `limit` *and* body hits up to `limit` (two bounded lists, up to 2× the caller's
+  limit). Our envelope's documented invariant is that `totalHits` is bounded by
+  `--limit`, so the title hits come out of that budget and the record lane fills
+  what is left. The cost is real and deliberate: at the boundary a title hit
+  displaces a record hit. The alternative — up to 2× the limit — would break
+  `count_precision`'s single meaning for every consumer.
+- **No session-level dedup, in either direction.** A session matched by title
+  *and* body appears once per lane (Wake's shape), and its record hits are not
+  suppressed by its title hit. Suppressing them would hide the record evidence
+  behind a 120-character name — the title is a prefix of the first message for
+  90% of the store's sessions, so the suppressed row is usually the text the user
+  was looking for — and it would make the record lane's answer depend on whether a
+  title happened to match. `search-quality.test.ts` pins the fusion's provenance;
+  `title-index.test.ts` pins that both rows survive.
+- **No open-time backfill of an empty index (Wake's `#L212-L223`).** Wake tests
+  "the table is empty and sessions exist" at every open because an empty table is
+  its only signal that the feature is new. Our counter already carries that signal
+  and owns the derivation: `"2"` ≠ `"3"` raises `fts_reindex`, and the pass that
+  holds the lease re-derives in the transaction that clears the flag. An
+  open-time backfill would be a second derivation point outside the pass's
+  accounting (it would not appear in `derive_ms`), which is exactly the
+  first-class switch C2 shipped. The visible difference is one pass: between the
+  writer open and the pass, reads serve body hits and no title hits — the same
+  window C3 documented for `records_echo`, and the same state `status.echo_filtered
+  → null` covers ("unknown, not zero").
+- **The title gate's parity is exact rather than subtracted.** `records_fts`
+  gates on `records − excluded == indexed`; the title index has no excluded set,
+  so it gates on `COUNT(titles_fts) == COUNT(sessions WHERE title != '')` plus the
+  orphan check. Both terms use the same exported predicate the repopulation and
+  the triggers apply.
+- **The title lane is skipped, not refused, when the index is missing.** The read
+  contract deliberately requires no index *presence* (§3.1 `REQUIRED_READ_SCHEMA`),
+  so the lane probes `sqlite_master` once per query and answers "no title hits"
+  for a store whose derivation is pending, instead of refusing a store the record
+  lane can serve.
+- **`reindexRecordsFts` → `reindexFts`.** The one-shot derives both indexes now;
+  keeping the old name would have made the method's name a lie at the exact place
+  a reader looks for the contract. The flag and the counter keep their names
+  (`fts_reindex`, `fts_format`) because they were never records-specific.
+
+## Audit against the scout design
+
+| Contract item | State |
+| --- | --- |
+| Standalone `titles_fts(session_id UNINDEXED, title, porter unicode61)` | as designed (trigram-free first pass; C5 owns the tokenizer) |
+| `FTS_FORMAT` → `"3"`; `DERIVE_COUNTERS` outputs grow to `[records_fts, records_echo, titles_fts]` | as designed; a matched counter with the table missing re-derives |
+| Rebuild populates from the sessions' title source, one-shot | as designed, from `sessions.title` — the column the normalizer writes and the reader projects, so there is exactly one title implementation in the tree (`normalizer.ts` derives it; nothing recomputes it at query time) |
+| Parity gate: `COUNT(titles_fts)` == sessions with a non-empty title + orphan check, beside the existing gates | as designed, plus FTS5 `integrity-check` |
+| Lexical: titles first (bm25 ASC, recency tiebreak), ahead of body hits, within the limit | as designed |
+| Body hits exclude sessions already hit by title? | decided: **no** — Wake emits both, and record-level hits were never session-deduped (see deviations) |
+| Hybrid: title pre-pass as session-level hits preceding RRF fusion | as designed (title hits do not enter the fusion: they carry no record id to key it) |
+| Semantic lane untouched | as designed — except its lexical *fallback*, which is the lexical engine and therefore carries the lane |
+| `SearchMatchType += "title"`; `record_id: null`; `record_type: "title"`; snippet = the matched title; `started_at` = session start | as designed; the type is a discriminated union so a consumer must narrow |
+| Robot v2 + human CLI both surface it | as designed; v1 too, because `--output-version` is documented as formatting-only |
+| Echo: all session titles stay searchable | as designed and documented as a decision: a title is provenance metadata, not echo content |
+| Maintenance in the same transaction | as designed, via triggers on `sessions` (see deviations) |
+| Reader/status tolerate a pending derivation | as designed (probe per query; `title-index.test.ts` pins the pending read, the empty index, and `echo_filtered: null`) |
+
+## Measurements
+
+Host: Apple M1, macOS 25.6, APFS. Fixture: a byte-consistent copy of the
+production store (`sqlite3 "file:…?mode=ro" ".backup /tmp/ledger-c4/ledger.sqlite"`;
+1,125,158,912 bytes, 229,103 records, 2,609 sessions, 0 embeddings). The
+production store is never opened for writing — the read-only numbers below come
+from `?mode=ro` queries and the trajectory readers used for the search
+observations; the measurement harness (`/tmp/ledger-c4`) was deleted afterwards.
+
+The tree's own state proved the upgrade path once for real before this audit: the
+production store already carried `fts_format = 3`, a `titles_fts` of **2,609**
+rows, and the three maintenance triggers (a scheduled pass had run against this
+working tree), and its parity holds — 2,609 indexed titles for 2,609 titled
+sessions. The "before" state below was therefore reconstructed on the copy, not
+observed on the live store.
+
+**OBSERVED**
+
+| measurement | value |
+| --- | --- |
+| production store, read-only | 2,609 sessions, **2,609 with a non-empty title** (277,385 characters, 859 distinct titles); `fts_format = 3`; `titles_fts` = 2,609 rows; `trg_sessions_titles_{insert,update,delete}` present |
+| titles that are a prefix of one of their own user records | **2,348 of 2,609 (90.0%)** — the title *is* the first instruction for the common case; 261 titles come from elsewhere (harness metadata, or a first message that is not a `user` record) |
+| copy, canonical side | 229,103 records, 228,903 indexed = records − 200 echo exclusions |
+| reconstructed pre-C4 state on the copy | `fts_format = '2'`, `DROP TABLE titles_fts`, flag cleared: the C3 shape |
+| search before — the session's own name, `"Hermes Feishu Thread Reply Behavior"` | **0 genuine matches**; `totalHits: 5`, every hit `implicit_wildcard` (the zero-hit widening path substituted prefix matches from other sessions), `widened.applied` set |
+| search before — `Feishu` (a word in that title) | 5 hits, all `lexical` record hits; the session that is *named* Feishu is not the answer |
+| search before — `wal_autocheckpoint` (body-only) | 5 hits across 2 sessions |
+| writer open of the pre-C4 copy | **3 ms**; stamped `fts_format = 3`, raised `fts_reindex = 1`, recreated `titles_fts` empty (0 rows); reads still served 5 hits with no title lane (pending is not a refusal) |
+| one-shot `reindexFts()` on the copy | **15,835 ms**; `titles_fts` 2,609 = titled sessions 2,609; `records_fts` 228,903 = records − excluded; flag cleared; `FTS_FORMAT` stamped; `records_fts` content unchanged (228,903 before and after) |
+| title phase, timed separately (warm) | rebuild (drop + create + `INSERT … SELECT` 2,609 rows) **9.4 ms**; gate (parity + orphan + `integrity-check`) **7.3 ms** — 0.1% of the one-shot, whose cost is the `records_fts` re-projection |
+| search after — the session's own name | **`totalHits: 1`, `sessions_count: 1`, rank 1**: one `match_type: "title"` hit naming `hermes:20260622_161708_b6f70dc8`, snippet `«Hermes Feishu Thread Reply Behavior»`; `widened: null` (the lane's hit is a hit, so the widening path is not entered) |
+| search after — `Feishu` | 5 hits; the first three are `title` hits (the sessions named after Feishu, best first), the record hits follow |
+| search after — `wal_autocheckpoint` (body-only) | **the same 5 hits across the same 2 sessions, in the same order** as before the derivation |
+| robot v2, copy | `{"type":"search_meta", …, "total_hits":1, "sessions_count":1, "widened":null}` then `{"session_id":"hermes:20260622_161708_b6f70dc8","source":"hermes","title":"Hermes Feishu Thread Reply Behavior","timestamp":"2026-06-22T08:17:06.724Z","score":-10.541302855850118,"snippet":"«Hermes Feishu Thread Reply Behavior»","match_type":"title", …}` — no `record_id` field (the v2 stream has never carried one, so nothing is fabricated), `timestamp` is the session's start |
+| human output, copy | `[1] [hermes] Session: hermes:20260622_161708_b6f70dc8 (2026-06-22T08:17:06.724Z)` / `Title: …` / `Role:  title` / `Match: «Hermes Feishu Thread Reply Behavior»` |
+| `status --robot`, copy | `echo_filtered: 200` (the derivation under this build's rules completed); in the *pending* window the same field is `null`, pinned by `title-index.test.ts` rather than measured here |
+| gate on the derived copy | passed (parity 2,609 == 2,609, no orphan, `integrity-check` clean) |
+| second `reindexFts()` | `false` — exactly once per store |
+| existing suite | **579 pass / 0 fail across 40 files** (568 at HEAD; `tests/title-index.test.ts` adds 11), `bun run typecheck` clean, `bun run build` clean |
+
+**INFERRED** (not directly measured, listed so reviewers can attack them)
+
+- The 15.8 s one-shot is the *warm* `records_fts` re-projection band C2/C3
+  measured (10.0–33.3 s across runs and hosts), plus ~17 ms of title work. Nothing
+  in this cycle's measurement separates the title phase from the record phase's
+  page-cache state, so a cold first pass on a user's machine should be expected in
+  the C2 band, title work included.
+- The 90% title/record overlap is measured on *this* store, whose titles derive
+  from first user messages for `pi`, `codex`, and `cursor` sessions. A store of
+  renamed or harness-titled sessions (claude-code summaries, Wake-style browser
+  titles) would show a different ratio, and the lane's value grows as that ratio
+  falls.
+- The budget interaction is measured only at the boundary the tests construct
+  (three titled sessions against a limit of two): the note's claim that a title
+  hit displaces a record hit at the limit is arithmetic from the implementation
+  (`limit − titles.length`), not a production observation.
+- The orphan branch of the gate is unreachable through a store that was merely
+  written (the triggers maintain the pair, and the rebuild drops the table whole),
+  so it is exercised only by the test that doctors the index directly.
+- Wake's trigram tokenizer would change this lane's recall (mid-token substrings,
+  CJK) and its index size; C5 owns that measurement and will re-derive under a
+  fourth counter bump. Nothing here depends on the choice beyond the DDL string.
+- A raw connection that drops `titles_fts` by hand while the triggers exist makes
+  session writes fail until the next writer open recreates the table
+  (`CREATE_TABLES_SQL` is idempotent and runs on every writer open). This is the
+  same class of coupling `trg_sessions_diagnostics_delete` already has with
+  `ingest_diagnostics`, and it is not measured in production because nothing does
+  it.
+
+## Position in the program (C4 row, `docs/research/2026-09-16-wake-port-program.md`)
+
+- Failure it fixes: *"Titles render in results but are unsearchable
+  (`records_fts` has no title column)."*
+- Open-source case: *"Session-find-by-name is the human's first query; every
+  session browser ships it."*
+- Ordering: C4 follows C3 on the same counter (the second bump, the second
+  consumer of the marker/one-shot pattern) and precedes C5, which will change the
+  tokenizer behind the same DDL family. C6's recency prior will land on top of
+  this lane's `ORDER BY` (Wake already multiplies this exact query by its time
+  decay, `db.rs#L1685`), and C7's tombstone work inherits the open question this
+  cycle's maintenance answers for titles: a delete path that must reach a derived
+  index the store's foreign keys cannot.
+
+## Boundaries and follow-ups (not in this cycle)
+
+1. **Rule changes need the bump.** The indexability predicate (`title != ''`) and
+   the lane's shape are the rules `FTS_FORMAT` `"3"` names; editing either without
+   a bump leaves every store's title index as-is. The maintenance triggers are
+   DDL, so a change to them rides `CREATE_TABLES_SQL` (idempotent, every writer
+   open), not the counter.
+2. **A store that records `"3"` with a hand-emptied `titles_fts`** is not detected:
+   the counter names the rules, not the rows, and the output table's *presence* is
+   all the open-time check reads — the same stance C3 documented for
+   `records_echo`. Raising `fts_reindex` by hand (or the next bump) repairs it;
+   parity is what would notice.
+3. **Titles are indexed as text, not as names.** The lane matches the same
+   tokenized words the record lane does, so a title word the query misspells, or
+   a mid-token substring, still misses — C5's tokenizer decision is where that
+   changes for both indexes at once.
+4. **No title-lane tuning knobs.** No `--title-limit`, no weight: the lane's cost
+   is one extra bounded index query per lexical search (the whole rank-1 title
+   query above — probe, title pass, record pass, projection — reported
+   `elapsed_ms: 1.471` on the production copy), and its share of the limit is the
+   documented rule. A weight or a separate budget is C6's territory (relevance),
+   not a hidden default.
+5. **The v1 robot stream carries title hits.** By the documented contract
+   `--output-version` selects formatting, never lanes, so v1 consumers that
+   assumed "every hit is a record" meet a row whose snippet is the session's name
+   (`tests/cli-skill-wrappers.test.ts` pins it). The alternative — filtering the
+   lane for v1 — would have made the version select the search algorithm, which
+   that contract forbids.

package/docs/research/ledger/docs/research/2026-09-16-c5-trigram.md

@@ -0,0 +1,268 @@
+# C5: the trigram swap, measured and refused, 2026-09-16
+
+## Outcome
+
+The decision round approved a trigram swap **with binding conditions**, and the
+conditions were measured first, on `/tmp` copies of the production store. They
+fail: nothing in this repo changes, `FTS_FORMAT` stays `"3"`, and both indexes
+stay `tokenize="porter unicode61"`. The program's C5 row is closed as
+*measured, refused* rather than refused on judgement — the capability case is
+real, and it is the cost that refuses it.
+
+Every binding condition of the decision round fails on at least one
+measured case, on the store as it stands today (1,134,215,168 bytes, 230,220
+records, 2,612 titled sessions, 198 MB of indexed text):
+
+| bar (from the decision round) | threshold | observed | verdict |
+| --- | --- | --- | --- |
+| total store file size after the one-shot re-derivation | ≤ +35% (≤ 1,531 MB) | **1,729,425,408 B = +52.5%** | **FAIL** |
+| FTS shadow index growth, the doc's own basis (`_data` + `_idx`) | ≤ 5.5× | **6.93×** (100,798,464 → 698,441,728 B) | **FAIL** |
+| one-shot re-derivation wall, idle (both indexes, gates included) | ≤ 45 s | **54.3 s / 54.6 s** (two runs) | **FAIL** |
+| identifier-substring query (class 1) | ≤ 15 ms median | 3.2–15.7 ms for rare identifiers, **190–200 ms for a common one** | **FAIL** (class) |
+| degraded short-segment `LIKE` (class 3) | ≤ 50 ms median / ≤ 150 ms p95 | 0.15–1.1 ms when the term matches early, **248–289 ms when the scan is total** | **FAIL** (class) |
+| CJK, 3+ codepoints (class 2) | ≤ 20 ms median | 0.39 ms | pass |
+| natural-language AND (class 4) | ≤ 20 ms median / ≤ 50 ms p95 | 6.8 / 39.3 / 18.5 ms | fail on one of three |
+| #34 common-prefix floor (class 5) | ≤ 25 ms median | **283–456 ms** | **FAIL** |
+
+Two findings deserve naming before the tables:
+
+- **Trigram does not fix the #34 floor.** The hypothesis was that `"the"`
+  becomes a single cheap trigram. It does become a single trigram, and that
+  trigram's posting list is 79,122 of 230,220 records — the cost is match-set
+  and bm25 bound, exactly the class #34's closing comment named, and the measured
+  numbers (390–456 ms) sit at parity with or slightly above the widened porter
+  path they were meant to retire (214–418 ms). Substring *recall* is bought; the
+  floor stands.
+- **The decision round's own bar table was internally unsatisfiable.** Its
+  baseline (811 MB store, ~150 MB shadow, +35% total) allows a shadow of at most
+  ~434 MB — a 2.9× growth — while the same table allows 5.5× (~825 MB), which is
+  a +83% store. Any re-decision should derive both bars from one budget.
+
+## The design that was going to ship (kept for a future cycle)
+
+The full contract, for the record and for whoever reopens this:
+
+- **Both indexes**: `tokenize="trigram case_sensitive 0"` — `records_fts` and
+  `titles_fts` together (Wake's line: `db.rs#L63-L67` and `#L126-L130` both use
+  it), because the two lanes are handed the same compiled expression and a
+  porter title lane beside a trigram record lane would answer differently for
+  the same identifier.
+- **`FTS_FORMAT = "4"`** — C2's counter does the migration: the next writer pass
+  re-derives both indexes in one transaction, gates unchanged (they are
+  count-based), and stamps the new format. Rollback is the same machinery
+  downwards: a `"3"` build stamps `"4"` → `"3"` and re-derives back to porter.
+- **Query compilation**: whitespace segments, each quoted as a phrase and
+  `AND`-joined (`db.rs#L1587-L1591`); C4's title/record lane split and limit
+  budget unchanged.
+- **Whole-query `LIKE` degradation**: any segment shorter than 3 codepoints
+  (`Array.from(seg).length < 3`, Wake's `chars().count() < 3` at `db.rs#L1544`)
+  sends the whole query for that lane to `LIKE '%…%' ESCAPE '\'` — the record
+  lane over `content`/`tool_arguments_json`/`tool_result_json`, the title lane
+  over `sessions.title` (Wake degrades titles too, `db.rs#L1706`), with the
+  echo filter as an anti-join against `records_echo` because canonical `records`
+  *does* hold echo rows (74 of the store's 216 echo records match `'%db%'` — the
+  leak would be real).
+- **Disclosed, never swallowed**: the envelope gains `degraded: boolean` +
+  `degraded_reason: "short_segment_under_3_chars" | null`, hits carry
+  `match_type: "like_degraded"`, and the CLI prints the caveat on stderr for v1
+  robot consumers that carry no meta.
+- **Prefix-OR widening retired** for the trigram lanes (substring subsumes
+  prefix): `LEDGER_WIDEN_MAX_RECORDS` and the widening pass go away, the
+  envelope's `widened` field stays for compatibility and reports
+  `{applied: false, skip_reason: "trigram_native"}` on a zero-hit lexical query,
+  and `match_type: "implicit_wildcard"` stops being produced.
+- **CJK semantics**: ≥3-codepoint substrings match natively at any position
+  (measured: `搜索测试` → 3 hits, 0.39 ms); a 2-codepoint CJK query has no
+  trigram to build and degrades to `LIKE` — that is the honest behaviour, and
+  it measures 189–289 ms.
+
+Ordering and snippet generation for the degraded path, per Wake
+(`db.rs#L1632-L1650`, `#L2016-L2043`): segments escape `\`, `%`, `_`; the plan
+must early-exit (ordering by `rowid DESC` with the `LIMIT` measured 0.2 ms for a
+common term where ordering by session time measured 310 ms); snippets are
+windowed in JS around the first matching segment (`make_like_snippet`: 40 chars
+before, match in `«»`, 80 after).
+
+## Measurements (OBSERVED)
+
+Host: Apple M1, macOS 25.6, APFS; Bun 1.3.14 (SQLite **3.51.0**, trigram and
+`case_sensitive 0` verified compiled in — the decision round verified the same
+SQLite under Bun 1.4.2). Fixture: a byte-consistent copy of the production store
+taken with `sqlite3 ".backup"` (the live store is only ever read). The
+one-shot was run exactly as `reindexFts()` runs it — drop, recreate, repopulate,
+echo-subtract, automerge, both gates, one transaction — twice, on two copies.
+
+**Size.** `dbstat` per shadow table, after checkpoint; the file is the page
+count × 4096, no significant freelist left (241 pages).
+
+| shadow table | porter (format "3") | trigram | ratio |
+| --- | --- | --- | --- |
+| `records_fts_data` | 100,507,648 | 697,552,896 | 6.94× |
+| `records_fts_idx` | 290,816 | 888,832 | 3.06× |
+| `records_fts_content` | 280,817,664 | 280,829,952 | 1.00× |
+| `records_fts_docsize` | 3,985,408 | 4,091,904 | 1.03× |
+| `records_fts`, all | 385,605,632 | 983,367,680 | 2.55× |
+| `titles_fts`, all | 700,416 | 1,806,336 | 2.58× |
+| all FTS shadow | 386,306,048 | 985,174,016 | 2.55× |
+| **`_data` + `_idx` (the bar's basis)** | **100,798,464** | **698,441,728** | **6.93×** |
+| **store file** | **1,134,215,168** | **1,729,425,408** | **+52.5%** |
+
+The indexed text is 197,656,948 bytes (59.1 MB `content`, 93.1 MB
+`tool_result`, the rest tool names and arguments): trigram costs 3.5 bytes of
+index per byte of text where porter cost 0.51. The `_content` shadow is a
+1.42× duplicate of that text and is the only large removable term (below).
+
+**One-shot re-derivation** (two copies, identical statement set):
+
+| phase | run 1 | run 2 |
+| --- | --- | --- |
+| drop + DDL `records_fts` | 3.84 s | — |
+| repopulate from `records` | 40.05 s | — |
+| echo subtraction | 0.58 s | — |
+| automerge (both) | 0.01 s | — |
+| `titles_fts` rebuild | 0.05 s | — |
+| FTS5 `integrity-check` (records / titles) | 9.80 s / 0.02 s | 8.81 s / 0.02 s |
+| **total, flag clear included** | **54.34 s** | **54.56 s** |
+
+For scale, the same one-shot under format "3" measured **15.8 s** on this store
+(C4's note): the trigram re-derivation is 3.4× the porter one, and the porter
+gate alone was 5.0 s where the trigram gate is 9.8 s.
+
+**Query classes.** The reader's own SQL shape (join `sessions`, `bm25`, three
+`snippet()` columns, `ORDER BY rank, started_at DESC LIMIT 50`), 3 warmups + 50
+iterations; hit counts are `COUNT(*) … MATCH`.
+
+| query | trigram hits | median | p95 | porter baseline (same query) |
+| --- | --- | --- | --- | --- |
+| `Session` | 36,242 | 190.4 ms (re-run 200.6) | 202 ms | 163.9 ms (34,391 hits) |
+| `getSessionStore` | 19 | 16.4 ms (re-run 15.7) | 17.7 ms | — |
+| `useEffect(` | 39 | 6.0 ms | 6.1 ms | — |
+| `content_snippet` | 14 | 4.5 ms | 4.8 ms | 1.2 ms (16 hits) |
+| `readWidenMaxRecords` | 9 | 3.5 ms | 3.7 ms | — |
+| `搜索测试` (3+ CJK) | 3 | 0.39 ms | 0.40 ms | 1 hit (token-granular) |
+| `中文搜索` (3+ CJK) | 2 | 0.21 ms | 0.22 ms | 0 hits (token-granular) |
+| `搜索` (2 CJK → LIKE) | 0 native | 189 ms | 202 ms | — |
+| `db` / `fs` / `ui` / `io` → LIKE | early exit | 0.15–1.10 ms | 1.18 ms | — |
+| absent term → LIKE (`%zqxq%`) | 0 | 248 ms | — | — |
+| `fair AND use AND throttle` | 8 | 6.8 ms | 7.1 ms | 0.88 ms (5 hits, phrase) |
+| `worktree AND premortem AND fix` | 703 | 39.3 ms | 39.7 ms | 1.58 ms (5 hits, phrase) |
+| `rebase AND token AND passthrough` | 25 | 18.5 ms | 18.7 ms | 2.22 ms (21 hits, phrase) |
+| `the` | 79,122 | 456 ms (re-run 391) | 484 ms | 418 ms (`the*` widened) |
+| `run` | 70,707 | 394 ms | 454 ms | 277 ms (`run*`) |
+| `test` | 47,735 | 283 ms | 309 ms | 214 ms (`test*`) |
+
+Supporting observations, same fixture:
+
+- `snippet()`/`highlight()` work under trigram (a `getSessionStore` hit came
+  back as `… («getSessionStore» → s…`), bm25 produced sane ranks.
+- The `LIKE` plan's two cost regimes: `rowid DESC` + `LIMIT` early-exits on
+  common terms (0.2 ms for `db`), and scans everything when few or no rows
+  match (248–289 ms for 0–72 matching rows). `NOT IN (SELECT … records_echo)`
+  and `NOT EXISTS` measured identically (0.20 ms / 13.5 ms).
+- Ordering the degraded `LIKE` by session time instead of `rowid` removes the
+  early exit: 310 ms for `db`, 267 ms even for a term with no hits.
+- The title lane is trivial either way (2,612 rows): trigram `trigram` 0.05 ms,
+  `LIKE '%tr%'` 0.07 ms.
+- A FTS5 `integrity-check` passes on a trigram table (it is the 9.8 s line
+  above), so the C2/C3/C4 gates stay valid under the swap.
+
+## Why the costs are where they are (INFERRED, marked as such)
+
+- The 190–200 ms for `Session` and the ~400 ms for `the` are match-set bound:
+  bm25 must score every matching row and `ORDER BY rank` cannot stop at the
+  `LIMIT`. Porter pays the same on the same query (163.9 ms / 418 ms), so
+  trigram is at parity, not a regression — but it is not the ≤15 ms / ≤25 ms the
+  decision round projected, and no tokenizer choice alone changes that.
+- The re-derivation cost is dominated by tokenization (40 s) plus the
+  `integrity-check` (9.8 s), both O(index size). It scales with the corpus and
+  the store keeps growing (156,801 records at the decision round, 230,220
+  measured here), so a re-decision should re-measure, not rescale.
+- The full-scan `LIKE` band is O(rows × indexed columns) with no index to help
+  it; 248–289 ms on 198 MB of text is the floor for a total scan, and a store
+  with larger tool results would be worse.
+- Class 4 is not just slower, it is looser: the compiled expression changes the
+  current phrase semantics to AND-joined segments, and the hit counts move with
+  it (`worktree premortem fix`: 5 phrase hits → 703 AND hits; `fair use
+  throttle`: 5 → 8). That is Wake's shape and would be a deliberate
+  recall-for-precision trade for lexical search — worth deciding explicitly if
+  the cycle reopens, but it is not what the class-4 bar was measuring.
+- The measured runs were warm (page cache) and the host was not idle (load
+  average ~2.5 with the agent stack live). That cuts both ways: a cold or busier
+  machine is slower, and "idle" in the bar's wording was not available; the
+  porter baseline measured on the same host (15.8 s) is the honest comparison,
+  and the 3.4× multiplier is tokenizer work, not host noise.
+
+## The one measured size lever (a variant, not the decided design)
+
+`records_fts` stores its indexed columns in `records_fts_content`, a 281 MB
+copy. A contentless variant (`content='', contentless_delete=1`, same trigram
+tokenizer) rebuilds to **702,992,384 B** of shadow (data 697,335,808 + docsize
+4,775,936 + idx 880,640) and a **logical** store of 1,448,067,072 B (+27.7%) —
+under the +35% bar. It is not free: the file only shrinks with a `VACUUM`
+(the variant left 281 MB of freelist; the decided DDL leaves none), `snippet()`
+and every projected column come back `NULL` from a contentless table (measured:
+`record_id`/`content`/`snippet` null, `MATCH` and `bm25` still correct), so the
+record lane would have to join `records` and window snippets in JS — the
+mechanism the degraded `LIKE` path needs anyway. That is a reader redesign, a
+different cycle, and it does not move the 6.93× shadow bar or the 54 s
+one-shot: the index itself and its build are the cost.
+
+## What would reopen C5
+
+1. **A re-derived bar table** (one budget, both metrics) and an owner decision to
+   accept either +52.5% / 1.73 GB or the contentless redesign.
+2. **Candidate capping** for the common-substring classes (`Session`, `the`):
+   #34's closing comment already names candidate-capping as the fix shape for
+   the match-set-bound floor; it is orthogonal to the tokenizer and would make
+   the recall half of this design affordable.
+3. **A corpus whose shape differs** (fewer, smaller tool results; a
+   shadow-share far below this store's 34%) — the arithmetic, not the
+   tokenizer, is what fails here.
+
+## Roadmap context and the open-source case (program doc, C5 row)
+
+> **C5** | Trigram tokenizer decision + swap (with whole-query `LIKE`
+> degradation, surfaced in the envelope) | #34 common-prefix floor: mid-token
+> substrings and CJK are unmatchable under `porter unicode61` | Code-identifier
+> search is the agent's query shape (`useEffect(`); Wake proves it works in
+> SQLite FTS5 alone
+
+Both halves of that rationale were tested. The failure half is confirmed:
+mid-token substrings and CJK are unmatchable today, and the swap fixes recall
+(`Session` inside `getSessionStore`, `useEffect(`, 3-codepoint CJK). The cost
+half is what the program doc's C5 row did not price: the swap costs +52.5% of
+the store, a 54 s one-shot, and it leaves the common-prefix floor standing at
+~0.4 s. Its dependency note — "C6 after C5 (ranking experiments on the final
+tokenizer)" — resolves to *no tokenizer change*: C6 can proceed on the porter
+index, and any ranking work should be evaluated on this store's measured
+match-set classes rather than on a hoped-for cheaper tokenizer.
+
+## Method (for reproducibility)
+
+Copy the live store with SQLite's own backup API (`.backup /tmp/…` — a read
+transaction against the original; the store was never opened for writing), then
+on the copy: drop and recreate both FTS tables with the candidate DDL,
+repopulate from `records` / `sessions.title`, subtract `records_echo`, reset
+automerge (`INSERT INTO records_fts(records_fts, rank) VALUES('automerge', 4)`),
+run both gates, and time the whole transaction; sizes are `dbstat` sums plus
+`page_count` × `page_size` after `wal_checkpoint(TRUNCATE)`; query timings use
+the reader's own SQL shape with warmups. The harness lived in `/tmp` and was
+deleted; the numbers above are the runs described here, and the tree is
+untouched by this cycle.
+
+## Wake pins (MIT, `iAmCorey/Wake@d9f702b6` tag v0.6.5)
+
+| mechanism | pin |
+| --- | --- |
+| trigram tokenizer on the message index | `crates/wake-core/src/db.rs#L63-L67` (`tokenize="trigram case_sensitive 0"`) |
+| trigram tokenizer on the title index | `db.rs#L126-L130` |
+| whole-query degradation predicate | `db.rs#L1540` (segments), `#L1544` (`chars().count() < 3`) |
+| AND-joined quoted segments | `db.rs#L1587-L1591` |
+| degraded body `LIKE … ESCAPE '\'`, `ORDER BY ts DESC LIMIT ?` | `db.rs#L1632-L1650` |
+| degraded title `LIKE … ESCAPE '\'` | `db.rs#L1706` |
+| `escape_like` / `make_like_snippet` windowing | `db.rs#L2016-L2043` |
+
+## Files
+
+None. This cycle changed no source, no DDL, no counter, no test, and no
+`api-surface.md` statement; this note is its whole artifact.