trajectories 0.0.0 → 0.1.1

package/docs/research/ledger/docs/research/2026-09-16-c10-kimi.md

@@ -0,0 +1,414 @@
+# C10: Kimi as a first-class source, 2026-09-16
+
+## Outcome
+
+Kimi CLI is now a source this store discovers, reads, normalizes, and covers like
+any other: `packages/core/src/native/kimi.ts` (discovery of
+`~/.kimi/sessions/<workspace-md5>/<session-uuid>/wire.jsonl`, the replay of its
+append-only event protocol into canonical records, the `state.json` sidecar, the
+shared workspace registry, and the `subagents/<agent-id>/` registry), wired
+through `scanner.ts`, `native.ts`, `normalizer.ts`, `parentage.ts`, and
+`watch.ts`. A top-level session is keyed `kimi:<session-uuid>`; a child is keyed
+`kimi:<parent-uuid>:<agent-id>`, so an agent id is never assumed globally unique.
+The source gets its own `source_coverage` row, and
+`packages/core/tests/native-kimi.test.ts` (15 cases) covers the fixture plan.
+
+The sentences this cycle exists to make impossible:
+
+1. **"A `ToolCall` row carries the arguments."** It usually carries a *prefix* of
+   them: on this host 220 of 309 real calls open a JSON object they never close,
+   and the remainder streams into `ToolCallPart` rows that follow, interleaved
+   with other calls' results. The call's arguments are the *shortest prefix of
+   its own argument bytes that parses as an object* — a rule that resolves 309/309
+   real calls (89 already complete, 220 completed from their fragments).
+2. **"`ToolCallPart` rows are noise."** They are the call's own argument bytes;
+   they are counted as lifecycle only because the call row — not they — carries
+   the record's identity and timestamp.
+3. **"A subagent's parent is written on the child."** The path proves the parent
+   (`<parent-session>/subagents/<agent>/`), and the *parent's* wire binds the
+   spawning call (`SubagentEvent.parent_tool_call_id` → `agent_id`). Nothing on
+   the child names either: 5/5 real `meta.json` files have no parent or tool
+   field. The scan reads the binding once per parent, while it is already walking
+   the store.
+4. **"A `SubagentEvent` is a second copy of the conversation."** It is: the
+   parent's mirror carries exactly the child's own rows (per-type counts equal for
+   5/5 children), so the mirror is state — counted, never projected. Projecting it
+   would store the child's transcript twice under two ids.
+5. **"`context.jsonl` is the transcript."** It is compacted working memory,
+   rewritten when the context fills; `wire.jsonl` is append-only and is what this
+   reader reads. None of the three source files is ever written.
+6. **"An interrupted session has no metadata, so it is not a session."**
+   `state.json` is optional by construction (an aborted session has none:
+   `e5bb8737` on this host), so the title falls back to the first user turn and
+   the session's span falls back to its own stream.
+7. **"What the adapter drops, nobody knows."** Every drop — lifecycle rows, the
+   subagent mirrors, repeated result rows, unrecognized kinds, non-text parts,
+   invalid or torn lines — becomes a typed, counted diagnostic in
+   `ingest_diagnostics`, with the kind names in the bounded sample.
+
+### Evidence, and what kind it is
+
+**OBSERVED** (this tree, this cycle):
+
+- `bun test` — 654 pass / 0 fail, 4,962 assertions, 45 files (the new file
+  contributes 15 cases / 111 assertions; `cli-canonical` and `watch-targets` grew
+  their source lists); `bun run typecheck` and `bun run build` clean.
+- The real-corpus pass over the read-only `~/.kimi` store into a scratch store
+  under `/tmp` (below): 8 sessions (3 top-level + 5 children), 806 records, 5
+  parent edges all tool-call-bound, 1 real failed tool result preserved, FTS
+  present with the reassembled arguments searchable, second pass fully deduped.
+- Argument reassembly measured over every real call: 309/309 resolved to a JSON
+  object, none left as a fragment, and the resolved calls carry the keys their
+  tool name implies (`ReadFile`→`path`, `Shell`→`command`, `Grep`→`pattern`,
+  `Agent`→`description`/`prompt`/`subagent_type`).
+- The `TurnBegin` shape split: 29 part-array rows and 5 whole-string rows — the
+  whole-string rows are exactly the 5 children.
+- Every other shape fact in the tables below, counted from the 8 real wires
+  (2,013 event rows, 0 torn lines) with content never printed. The test suite's
+  fixtures are synthetic models of those shapes; no real transcript text is
+  copied into this repo.
+- `@letta-ai/trajectory` 0.3.0 was inspected in `packages/core/node_modules`:
+  its adapter registry (`atif`, `claude-code`, `codex`, `copilot-cli`, `cursor`,
+  `droid`, `gemini-cli`, `hermes`, `letta-code`, `openclaw`, `opencode`, `pi`,
+  `deepagents`) has no Kimi entry and is not an extension point.
+
+**INFERRED** (not run here):
+
+- A workspace whose path is not in `~/.kimi/kimi.json` (a relocated directory, a
+  registry edited by hand) leaves that session's cwd unknown and changes nothing
+  else; the registry is a shared index, so a torn one costs cwd only and is not
+  diagnosed per session. The fallback the scout floated — parsing the
+  `<git-context>` block out of a prompt — was not implemented: the registry maps
+  every workspace on this host (3/3), and a second, weaker cwd mechanism is not
+  worth its ambiguity.
+- `ToolCallPart` attribution is *positional*, not a documented rule. A fragment
+  is appended to the call whose window it lands in — after that call's row, before
+  the next call's row — and the first prefix that parses wins. Under the protocol
+  observed here a call's fragments always precede the next announcement, so the
+  rule is exact (309/309 real calls resolved, 0 starved, 0 leftover). A future
+  protocol that emitted an *earlier* call's completing fragment after a later
+  call's row would give that fragment to the later call: with
+  `A:{"pattern":"`, then `B:{"path":"`, then a fragment `hi"}`, B's window
+  consumes it and B stores `{"path":"hi"}` — a wrong-but-valid object — while A
+  keeps its announced prefix verbatim. Nothing in the wire distinguishes the two
+  (fragments carry no call id), so the position *is* the contract; the sequence is
+  pinned in the suite so a future change to the rule is visible. The risk is
+  future-protocol only.
+- Compaction never truncates the wire: `CompactionBegin`/`CompactionEnd` are
+  lifecycle rows beside an append-only stream, so replay preserves history. No
+  compaction row exists on this host to observe directly.
+- A resumed or retried call could re-emit a `ToolResult`; no such row exists on
+  this host (309 results / 309 distinct call ids), so the one-result-per-call rule
+  is defensive and proven only synthetically.
+
+**Historical reference, deliberately not reused:** Wake's
+`crates/wake-core/src/adapters/kimi.rs` targets a layout and protocol that no
+longer exist — root `~/.kimi-code`, a `session_index.jsonl`, transcripts at
+`session_<uuid>/agents/main/wire.jsonl`, protocol `"1.3"` with `turn.prompt` /
+`context.append_message` rows. Shipping Kimi CLI (protocol `"1.10"` on this host)
+writes `~/.kimi/sessions/<md5>/<uuid>/wire.jsonl` with `TurnBegin` / `ContentPart`
+/ `ToolCall` / `ToolResult`. No line parser, path rule, or type name survived the
+port; Wake is cited here as the reason to distrust a stale adapter, not as a
+source of code.
+
+**Not done at all:** no upstream `@letta-ai/trajectory` adapter was written or
+proposed; `~/.kimi` was never modified (read-only, mtimes untouched); the
+production Ledger store was never opened by this cycle; no embedding or semantic
+measurement was taken; `StatusUpdate` token accounting (`token_usage`,
+`context_usage`, `context_tokens`) is not stored, because the store has no field
+for it — it is a counted lifecycle row instead; no cwd flattening was implemented
+(each session keeps the cwd it ran in).
+
+**Production first ingest is still blocked, by a known wall.** A scoped
+`ledger sync --sources kimi` pass discovers and enqueues this source but its queue
+drain is not source-scoped (issue #69: `--sources` scopes discovery, not the claim
+order), so a new bank's rows sit behind an existing backlog. That is the failure
+C9 hit in production and recorded; it is unchanged by this cycle, and it is why
+every measurement here was taken on scratch stores under `/tmp` (the C9 pattern),
+never on the production store. Kimi will ingest in production when #69 (or the
+#49 D1/D4 work) threads the source filter into the queue claim.
+
+**A second gate is the operator, not the code.** The installer defaults pin the
+source list a deployed service scrapes, and they have never been extended for the
+new banks: `ops/macos/install-launchd.sh:9` and
+`ops/linux/install-systemd-user.sh:48` both still say the same fourteen sources
+(claude-code, codex, cursor, pi, omp, hermes, openclaw, letta-code, opencode,
+droid, deepagents, openhands, gemini-cli, copilot-cli) — no `grok`, no `kimi` —
+while `watch`'s `DEFAULT_SOURCES` and the scan's default set now carry both. A
+host installed through those scripts therefore never sees either bank, even after
+#69 lands, and a `watch` service beside a scheduled-`sync` service diverges in
+what it covers. Whether to extend the installer defaults (and which sources a
+default deployment should read at all) is the owner's call; this cycle did not
+change service behavior, and the omission is recorded here so it is a decision
+rather than an accident.
+
+## The format, as observed
+
+Layout (`~/.kimi/sessions`): `<workspace-md5>/<session-uuid>/` holds `wire.jsonl`
+(the transcript), `state.json` (metadata sidecar: `custom_title` and
+`title_generated`, approval settings, todos, archive flags), `context.jsonl`
+(compacted working memory — not a source), and, when the session spawned
+children, `subagents/<agent-id>/` with `wire.jsonl`, `meta.json`, `prompt.txt`,
+and `output`. `~/.kimi/kimi.json` sits beside the sessions root and registers the
+workspace paths whose MD5 names the directories above it; `~/.kimi/device_id`
+is installation identity and Kimi's own index lives nowhere else — there is no
+SQLite in the store at all. No `subagents/` directory exists without a parent.
+
+`wire.jsonl` line 1 is the file's protocol stamp (`{"type":"metadata",
+"protocol_version":"1.10"}`, `"1.1"` in children); every later row is
+`{"timestamp": <unix seconds>, "message":{"type": <event>, "payload": <object>}}`,
+appended in place (`"a"` mode), one event per line.
+
+Aggregate counts over the 8 wires / 2,013 event rows (content never printed; only
+types, shapes, and counts):
+
+| fact | observed |
+| --- | --- |
+| event types | `ToolCall` 309, `ToolResult` 309, `SubagentEvent` 726, `ToolCallPart` 227, `ContentPart` 146 (113 `think`, 33 `text`), `StepBegin` 114, `StatusUpdate` 113, `TurnBegin` 34, `TurnEnd` 34, `StepInterrupted` 1 |
+| torn/invalid lines | 0 |
+| `TurnBegin.user_input` | 29 part arrays (`[{type:"text", text}]`), 5 whole strings — the strings are exactly the 5 children |
+| `ToolCall` arguments | 89 complete JSON, 220 prefixes completed by following fragments, 0 unresolvable; keys match the tool name in 309/309 |
+| `ToolCall.function` | `{name, arguments: string}`, `type: "function"`, `extras: null` |
+| `ToolResult.return_value` | `{is_error, output, message, display, extras}`; 308 `is_error: false` / 1 `true`; 4 rows with an empty `output` and a non-empty `message`; 2 rows with `display` renderings (a diff, an exit-code brief) that are not the model-visible text |
+| `StepBegin.payload` | `{n}` — a step counter, not a boundary of speech |
+| `StatusUpdate.payload` | `context_usage`, `context_tokens`, `max_context_tokens`, `token_usage{input_other, output, input_cache_read, input_cache_creation}`, `message_id`, `plan_mode`, `mcp_status` |
+| `SubagentEvent.payload` | `parent_tool_call_id`, `agent_id`, `subagent_type`, `event:{type, payload}`; the inner stream's per-type counts equal the child's own wire for 5/5 children |
+| child `meta.json` keys | `agent_id`, `subagent_type`, `status`, `description`, `created_at`, `updated_at`, `last_task_id`, `launch_spec{agent_id, subagent_type, model_override, effective_model, created_at}` — **no** parent session or tool field |
+| child extras | `prompt.txt` and `output` in 5/5 children; `output` is a stage/tool log, not a transcript |
+| `state.json` keys | `version`, `approval`, `additional_dirs`, `custom_title`, `title_generated`, `title_generate_attempts`, `plan_mode`, `plan_session_id`, `plan_slug`, `wire_mtime`, `archived`, `archived_at`, `auto_archive_exempt`, `todos`; absent for the interrupted session |
+| `kimi.json` | `work_dirs: [{path, kaos, last_session_id}]`; each path's MD5 equals its sessions subdirectory name (3/3); one registered workspace (flow-v2) has a hash directory with no sessions |
+| sessions | 3 top-level, 5 children; per session: 1144/139/4 rows (top-level) and 111/141/144/156/174 rows (children) |
+
+Row grammar → canonical record:
+
+| wire event | record |
+| --- | --- |
+| `TurnBegin` (parts or a whole string) | `user` (projected as the lane's user-prompt event) |
+| `ContentPart` (`type: "think"`) | `reasoning` |
+| `ContentPart` (`type: "text"`) | `assistant` |
+| `ToolCall` (`id`, `function.{name, arguments}`) | `assistant-tool-call` (`tool_call_id` = `id`, `tool_arguments_json` = the resolved arguments) |
+| `ToolResult` (`tool_call_id`, `return_value`) | `tool` (`tool_result_ok` = `!is_error`, `tool_result_json` = the result text) |
+| `SubagentEvent` | dropped, counted — the child's own wire holds that transcript |
+| `StepBegin`, `StepInterrupted`, `ToolCallPart`, `StatusUpdate`, `TurnEnd`, `ApprovalRequest`, `ApprovalResponse`, `CompactionBegin`, `CompactionEnd` | dropped, counted (lifecycle) |
+| anything else | dropped, counted |
+| the line-1 `metadata` stamp | neither projected nor counted: it is the file's header, not an event |
+
+## Assembly: the rules, and why each one
+
+**Parts, not fragments.** Unlike Grok's token chunks, a `ContentPart` is one whole
+part: 79 adjacent `think` rows on this host each end in a finished sentence and
+begin a new one (`"...structure as mentioned in AGENTS.md."` followed by
+`"The user wants me to use a multi-agent s..."`), and no two `text` rows are ever
+adjacent. Concatenating them the way Grok's run rule does would merge two
+separate reasoning blocks into one record; each part therefore becomes its own
+record, and only the wire's own row boundaries decide where a message ends.
+
+**Tool arguments are a stream that usually outruns its row.** A call is announced
+with the arguments known at that moment. `{"path":` followed by
+`ToolCallPart{"arguments_part":" \"air-cloud/webhook.go\"}"}` is one call's
+arguments split across rows, and the fragments are *not* adjacent to the row that
+announced them — other calls' `ToolResult` rows and `StatusUpdate` rows land in
+between. The reader walks a call's own window (from its row to the next call row),
+appends fragments, and takes the **first prefix that parses as a JSON object**:
+the row's own text if it is already complete, otherwise the shortest completion
+its fragments provide. Fragments carry no call id, so that window *is* the
+attribution rule — its one failure mode, and why it is safe against the protocol
+observed here, is recorded below. Measured over 309 real calls: 89 complete at
+announce time, 220 completed from fragments, 0 left unfinished, and the resolved
+arguments are semantically right for every tool (`ReadFile` 200, `Shell` 65,
+`Grep` 36, `Agent` 5, `Glob` 2, `WriteFile` 1). Storing only the row's prefix was
+rejected: it would truncate 220 of 309 real calls' arguments, which are the
+most-queried part of a coding transcript. Guessing past what parses was rejected
+too: a call whose stream never completes keeps the prefix — the library then
+stores it under its own `{"_raw": …}` convention, which is lossless and marks
+itself as unshaped.
+
+**Tool results.** `ok = !is_error` (309 real results: 308/1), keyed by
+`tool_call_id`. The result's text is `return_value.output` when non-empty, else
+`return_value.message` — 4 real rows have an empty output with the answer in the
+message (`"No matches found."`), and the `display` renderings (a diff, an
+exit-code brief) are presentation, not text. The first result for a call id is
+that call's outcome and a repeated result row is counted, so one call can never
+carry two results — the same discipline C9 landed for Grok's repeated terminals,
+here proven only synthetically because no real wire repeats a result.
+
+**Identity.** Rows have no ids of their own, so each projected row is identified
+by its **line number** (`kimi-line-<n>`): the file is append-only, so a line never
+moves, whereas a byte offset would shift when an earlier run grows. Sequence and
+timestamps come from the row's position and its `timestamp` (unix seconds →
+ISO-8601 milliseconds). The listing's own id is stamped on a `session.start`
+envelope row emitted for **every** session, so a session that projects no
+conversation rows at all still carries its own group.
+
+**cwd is a hash lookup, not a guess.** The session directory is the MD5 of the
+workspace path, so the workspace is recovered by hashing the paths in
+`kimi.json`; a workspace that is not registered leaves cwd unknown. This is the
+only mechanism used: the prompt-borne `<git-context>` block (visible in the
+children's whole-string turns) is *not* parsed for it.
+
+**Sidecar.** `state.json` supplies the title (`custom_title`, which the CLI writes
+as the display title — truncated with an ellipsis, and not a copy of the first
+prompt). A missing sidecar is not an error; a malformed one is a diagnostic and
+costs the title only. The wire's own bounds are the session's span when the
+sidecar holds no timestamps (it holds none at all — `wire_mtime` is `null`).
+
+## The projection lane, and the decision behind it
+
+The pinned normalizer (`@letta-ai/trajectory` 0.3.0) has **no** Kimi adapter, and
+its registry is a frozen object, not an extension point. Ledger's established
+answer is the one it already uses for Cursor, OpenCode, and Grok: project the
+native store into a contract the library *documents* and let the library do
+identity, hashing, bounds, and validation. C9's lane comparison is unchanged and
+applies verbatim — `atif` loses the terminal outcome, `letta-code` cannot separate
+a later result from its call, `cursor` has no timestamps — so Kimi joins **Grok's
+lane**:
+
+The chosen contract is the same shape as ACP and Copilot CLI: one event per line,
+an id per event, calls and results as separate events, a boolean outcome on the
+result. The projection is a rename:
+
+| Kimi wire | copilot-cli event |
+| --- | --- |
+| `session.start` envelope (listing id, first timestamp, cwd) | `session.start` (`sessionId`, `startTime`, `context.cwd`) |
+| `TurnBegin` | `hook.start` (`hookType: "userPromptSubmitted"`, `input.prompt`) |
+| `ContentPart` `think` / `text` | `assistant.message` (`reasoningText` / `content`) |
+| `ToolCall` | `assistant.message` (`toolRequests[]` = `{toolCallId, name, arguments}`) |
+| `ToolResult` | `tool.execution_complete` (`toolCallId`, `success`, `result.content` / `error.message`) |
+
+`KIMI_PROJECTION_SOURCE` in `native/kimi.ts` names the lane; `normalizer.ts` maps
+each unreadable source to its lane in one `PROJECTION_LANES` table and restores
+the records' provenance (`record.source_type = "kimi"`) right after
+normalization. Recorded consequences, so nobody has to rediscover them:
+
+- `records.source_type` is `kimi` and `sessions.source` is `kimi`; the store's
+  provenance axis is the source, never the lane.
+- The single `meta` record's `record_json` names the *projection lane*
+  (`"source":"copilot-cli"`), and both `record_hash` and the meta `content_hash`
+  are the library's hashes over it. Ledger does not rewrite library-owned hashes;
+  the note is the disclosure.
+- **A projected source's group is authoritative**, and kimi is in that set (the
+  C9 P1 fix). The reader's own envelope always agrees with it, so
+  `source_group_conflict` cannot fire; a reader change that stopped stamping the
+  envelope would still land on `kimi:<uuid>` rather than the library's `default`.
+- Tool results and arguments are subject to the library's default bounds: the
+  real-corpus pass stored 184 `tool_result_truncated` occurrences (2,500
+  characters, head-tail) over 7 sessions and exactly one
+  `tool_arguments_truncated` (the one `WriteFile` call whose reassembled arguments
+  passed 20,000 characters). Both are durable diagnostics, not silent losses.
+- An argument string the library cannot parse is stored as `{"_raw": …}` — the
+  unfinished-prefix case above, which is a *fidelity* outcome, not an error.
+- The session-level `model` stays null: Kimi's wire carries token accounting but
+  no model id, and `meta.json`'s `effective_model` is `null` on this host.
+
+## Parentage: the path proves the parent, the parent's wire proves the call
+
+Two pieces of evidence, both read from outside the child:
+
+1. **The path.** A child transcript lives at
+   `<parent-session>/subagents/<agent-id>/wire.jsonl`, so the enclosing session
+   directory is the parent — the same `parentIdFromSubagentsPath` shape Claude
+   Code and Cursor use. A session is never its own parent. The child's listing id
+   is `<parent-uuid>:<agent-id>`: agent ids are 9 characters (`"a"` + 8 hex) and
+   unique only within a parent, so a bare agent id would collide across sessions.
+2. **The parent's wire.** The parent records `SubagentEvent` rows binding
+   `parent_tool_call_id` → `agent_id`; the first binding for a registered child
+   wins, and only children that exist as directories are bound. The scan reads
+   this once per parent that has a `subagents/` registry — not once per child, and
+   it stops as soon as every registered child is bound — one bounded pass over
+   that session's own wire (≤ 64 MiB; the largest real parent wire here is
+   2.6 MB). The binding is attached to the child's listing as
+   `native.parentToolUseId`. A missing or unreadable parent wire costs the
+   binding, never the edge or the child.
+
+`agent_nickname` comes from the child's own `meta.json` (`subagent_type`, `explore`
+in 5/5 real children) and `thread_role` is `subagent`. The binding participates in
+the child's native fingerprint — `nativeSourceFingerprint` hashes `item.native` —
+so learning (or losing) a binding re-ingests exactly that child. The store's
+lineage backfill has no scan context and re-derives the path-provable edge only;
+`updateSessionParentage` merges, so a backfill never erases a stored tool id.
+
+**One child, one id.** A child is keyed by the parent its *path* names, on every
+route that can list it: the parent's registry, or the store walk reaching
+`subagents/<agent>/` because the parent's wire is missing (a partial copy, a
+deleted transcript). A bare agent id is never used — it is unique only within one
+parent, and it would be a second ledger id (and a second full ingest) for one
+transcript the moment the parent's wire came back. Losing the parent's wire costs
+the spawn binding, not the child's identity: the regression case *keeps one
+identity for a child while the parent's wire comes and goes* ingests it before,
+during, and after the loss and asserts one session, stable record ids, and a
+`deduped` verdict each time.
+
+## Verification
+
+**Suite.** `packages/core/tests/native-kimi.test.ts` — 15 cases:
+
+| case | what it pins |
+| --- | --- |
+| discovery | the documented root, newest-first order, state title, `native.kind`, the `listable` outcome and the `missing_root` outcome for an absent root |
+| dedup/symlink/page | a symlinked tree is not listed, an id is listed once, `--limit` bounds the page |
+| single turn | the exact projected transcript, the cwd resolved through the workspace registry, the protocol stamp ignored, the lifecycle diagnostic with its kinds |
+| child turn shape | a whole-string `TurnBegin` becomes the user turn; adjacent `think` parts stay separate records; a missing `state.json` is not a diagnostic |
+| tools | call/result pairing by id, `tool_result_ok` 1 and 0, compact `tool_arguments_json`, the empty-output → message fallback, and the repeated-result count |
+| argument reassembly | fragments across interleaved rows complete a call's arguments; a fragment is attributed by wire position (including the wrong-but-valid completion the contract allows); an unfinished prefix is never invented |
+| resilience | lifecycle and unknown kinds counted, a torn line counted, the transcript still complete |
+| interrupted session | no `state.json` → first-user title and stream-bounded span; a malformed sidecar → diagnostic, title from the user turn |
+| lifecycle-only | a session with only lifecycle rows projects its envelope and no conversation |
+| parentage | child listed under its parent-scoped id with the spawning tool call, title from `meta.json`, edges landing on both sessions' rows; a child with no wire is not a session; an unbound child keeps its path-provable parent |
+| child identity | a child keeps its parent-scoped id, session, and record ids while the parent's wire disappears and returns; a bare agent id never becomes a second session for one transcript |
+| identity/stability | unchanged re-read is `unchanged`; an appended turn adds records without moving or rewriting stored ones; a `state.json`-only change is `deduped` with a stable record set; the reader's diagnostics are stored in `ingest_diagnostics`; state-less sessions keep distinct `kimi:<uuid>` ids and no `kimi:default` exists |
+
+The default-scan tests in `cli-canonical.test.ts` were extended with `kimi`
+(source list, seeded root, and the "no root at all" set), and `watch-targets`
+now covers `kimi` (and the `grok` root C9 missed) in its adapter-roots case.
+
+**Real corpus.** Read-only `~/.kimi`, scratch store `/tmp/c10-kimi-scratch`
+(never the production store; `~/.kimi` untouched):
+
+| measurement | value |
+| --- | --- |
+| discovery | 8 sessions from `~/.kimi/sessions` (3 workspaces; one hash directory holds no sessions) — 3 top-level + 5 children |
+| first `build --sources kimi --embedding none` | 8 scanned / 8 imported / 0 failed / 0 tombstoned, 806 records, wall ≈ 0.36 s |
+| second pass (same store) | 8 scanned / 0 imported / 8 deduped — idempotent |
+| records | `assistant-tool-call` 309, `tool` 309, `reasoning` 113, `user` 34, `assistant` 33, `meta` 8 |
+| `source_type` | 806/806 `kimi` |
+| failed tools preserved | 1 (`tool_result_ok = 0`) — the real `Shell` failure with its stderr output |
+| parent edges | 5, all tool-call-bound (`parent_tool_use_id` non-null), all `agent_nickname = explore`, all `thread_role = subagent`, each binding matching a call row the parent stored; 0 edges for the 3 top-level sessions |
+| cwd | resolved for 8/8 sessions (3/3 workspaces) |
+| diagnostics | `noise_record_dropped` 1,215 occurrences over 8 sessions (489 lifecycle rows + 726 subagent mirrors), plus the library's `tool_result_truncated` 184 and `tool_arguments_truncated` 1 |
+| coverage | `kimi` / `/Users/mac/.kimi/sessions` / `listable` / discovered 8 |
+| search | 806 FTS rows; `consolidator` 20 hits across 4 sessions with the parent edge rendered, and `subagent_type` 11 hits inside a *reassembled* `Agent` call's arguments — the arguments the row alone did not carry are searchable |
+
+## Decisions
+
+1. **A borrowed lane, named in one table.** Kimi's records go through the same
+   identity, hashing, bounds, and validation path as every other source; the
+   divergence (which lane) is one exported constant plus one row in
+   `PROJECTION_LANES`. Building canonical records in Ledger remains rejected as a
+   second convention that would drift from the library's identity rules.
+2. **The shortest parseable prefix wins.** Tool arguments are reconstructed from
+   the source's own bytes, never extended past what parses and never invented when
+   the stream stops. Both alternatives were rejected: storing the announced prefix
+   (truncated arguments for 220/309 real calls) and treating fragments as noise.
+3. **A mirror is not a transcript.** The parent's `SubagentEvent` rows equal the
+   child's own wire row-for-row, so they are counted state. The parent still shows
+   the call that spawned the child, and the child is its own searchable session
+   under a parent-scoped id.
+4. **The path is the primary parent evidence.** The tool-call binding is a bonus
+   that makes the child attachable to the parent's call; it is read at scan time
+   and rides in `native_json`, so ingest pays no extra I/O and a binding learned
+   later re-ingests only the child.
+5. **Metadata degrades, transcripts do not.** `state.json`, `meta.json`, and
+   `kimi.json` each cost at most a title, a nickname, or a cwd. A torn line, an
+   unknown event, or an oversized sidecar never costs the session.
+6. **The skip discipline is C9's, unchanged.** Every dropped row is typed and
+   counted into `ingest_diagnostics`; the bounded sample names the kinds. A new
+   Kimi event becomes a visible count, not a silent hole.
+7. **This source is a bank for the open-source pitch and the Trajectory-format
+   gap.** Kimi is the second harness Ledger reads natively that the pinned
+   library cannot, and the second whose parentage comes from a registry rather
+   than a field on the child — the model generalizes, and the roadmap row is now
+   closed with it (see [the program](2026-09-16-wake-port-program.md), C10).
+8. **Production ingestion is gated on #69, and the cycle says so.** Everything
+   measured here is on scratch stores; the first production ingest will wait on
+   the queue-claim fix, exactly as C9's did. Recording the wall in the note is
+   cheaper than rediscovering it from an empty `records` table.

package/docs/research/ledger/docs/research/2026-09-16-c2-derive-counter.md

@@ -0,0 +1,216 @@
+# C2: the derive-rule counter, 2026-09-16
+
+## Outcome
+
+Index-shape changes no longer ride schema migrations. A store now records, in
+`schema_meta`, the **rules** each derived index was produced under
+(`fts_format`, `embeddings_format`), stamped by every writer open; a counter that
+contradicts the build's raises a transient re-derivation flag, and the next
+`sync` pass executes it **in store** — drop `records_fts`, recreate it,
+re-project it from canonical `records` in one transaction that also clears the
+flag. On the production store (225,959 records, 1.12 GB) that one-shot costs
+**19.4–33.3 s, 381.8 MB of WAL, and zero store growth**, with the store's
+cursors, queue, canonical records and vectors byte-identical, **no source file
+re-read**, and the re-derived index answering **10/10 probe queries exactly like
+the production index** (same totals, same ordered hits).
+
+The lifecycle the cycle adds, every stage pinned by a case in
+`packages/core/tests/derive-counters.test.ts`:
+
+| | open (writer) | pass (`sync`, lease held) | read (`TrajectoryReader`, `status`) |
+| --- | --- | --- | --- |
+| counter equals this build's | nothing | nothing | serves |
+| counter **older**, or **absent** | stamp this build's; raise `fts_reindex` when absent (see below) | re-derives once, clears the flag | serves the stale index as written |
+| counter **newer** | stamp this build's; raise the flag | re-derives to this build's rules | refuses `FORMAT_MISMATCH` |
+| flag pending, work interrupted | leaves it set | retries; the previous index survives | serves; never clears anything |
+
+Files:
+
+- `packages/core/src/schema.ts` — `FTS_FORMAT`, `EMBEDDINGS_FORMAT`,
+  `DERIVE_COUNTERS`, the `schema_meta` DDL (fresh store **and** every migration,
+  because `migrate()` runs the same `CREATE_TABLES_SQL`).
+- `packages/core/src/db.ts` — `detectDeriveCounters` (open-time),
+  `needsFtsReindex`, `reindexRecordsFts` (one transaction: rebuild, gates,
+  automerge, flag clear).
+- `packages/core/src/sync.ts` — the pass executes a pending re-derivation before
+  admission, under the writer lease it already holds, skipped under critical
+  pressure like the maintenance child; `timing.derive_ms` accounts for it.
+- `packages/core/src/schema-gate.ts`, `reader.ts`, `store-status.ts` — the
+  `format_mismatch` state and `FORMAT_MISMATCH` (`ReaderError` /
+  `StoreStatusError`), one shared implementation and one message for both gates.
+- `packages/core/src/cli.ts` — one prose line on a pass that re-derived
+  (`Index re-derivation: <ms> (records_fts rebuilt from this store's records; no
+  source was re-read).`), so a 20–30 s upgrade is not an unexplained stall.
+- Gates: `bun test` **561 pass / 0 fail** (552 tests in the tree at HEAD; the 9
+  cases in `tests/derive-counters.test.ts` are this cycle's), `bun run typecheck`,
+  `bun run build`.
+
+## Ported mechanics (Wake MIT, `iAmCorey/Wake@d9f702b6` tag v0.6.5 — mechanics with attribution, no code copied)
+
+Pin: `d9f702b6df0e85`, `crates/wake-core/src/{db,scanner}.rs`, MIT License © 2026
+Corey Chiu.
+
+| Ledger | Wake pin | What was kept |
+| --- | --- | --- |
+| `schema_meta` table beside `SCHEMA_VERSION` for derived-state versions | `db.rs#L25` (`schema_meta`), `#L128-L136` (`FTS_FORMAT` + its doc: bump the value when the derivation rules change) | the counter is an index-format version *distinct* from the schema version, with the value as the rules' name |
+| counter comparison, stamp, and flag at **open** | `db.rs#L176-L196` | compare the stored value, flag a mismatch **only when the store already had sessions**, stamp the current value in the same transaction |
+| flag as a transient key, not a column | `db.rs#L411-L427` (`has_meta_flag`/`clear_meta_flag`), `#L439-L447` (`needs_fts_reindex`/`finish_fts_reindex`) | a flag the pass reads at start and the work clears; crashes leave it set |
+| the scan reads the flag once, forces the work, and clears it after | `scanner.rs#L179` (read at scan start), `#L281` (forced work), `#L439-L441` (clear after the scan) | executed by a **scan-path pass** under its lease, not by an open; a pass that fails leaves the flag for the next one |
+| flag reprocesses rows the incremental path would skip, exactly once; a fresh store is not flagged | `tests/scanner_finale.rs#L309-L356` | the cycle's test skeleton: fresh store seeds silently, drift re-derives once, the second pass has nothing to do |
+| old consumers refuse a store they do not understand, up front | `db.rs#L229-L233` (`NEWEST_COLUMN`/`NEWEST_TABLE`), `#L320-L334` (read-only open bails by name) | our read gate refuses a counter newer than this build derives, naming the counter and both values, instead of dying deep in a query |
+
+Deliberate deviations, each with its reason:
+
+- **The re-derivation is in-store SQL, not a re-parse.** Wake's flag forces
+  every source file to be re-parsed (`scanner.rs#L179` → `#L281`), because a
+  file is the only place its units exist. Ledger keeps canonical records in the
+  store, so the same increment is `DROP`+`CREATE`+`INSERT INTO records_fts
+  SELECT … FROM records` through the projection ingest already writes
+  (`FTS_REPOPULATE_SQL`, shared with the v11 migration so the two cannot drift).
+  Measured: zero source bytes read where the sources behind the same records
+  total 1,555,222,435 bytes (1,479,531,453 ingested) across 2,664 cursors.
+- **The flag clear is inside the rebuild transaction.** Wake clears after the
+  scan (`scanner.rs#L439-L441`), which leaves a window where the flag is clear
+  and the index is not the one it names. Here the drop, the re-projection, the
+  post-copy gates, and the clear commit or roll back together — so an
+  interrupted pass leaves a consistent store by construction, and an unreachable
+  `finish` step cannot exist. (`finish_fts_reindex` has no Ledger counterpart.)
+- **A pre-counter store is flagged for FTS but not for vectors.** An unrecorded
+  counter is not a *moved* one, but Wake's rule (stored ≠ current → flag)
+  applies to an absent value too, and for FTS that is the right direction: the
+  re-derivation is executable in this build, cheap relative to a re-parse, and
+  it repairs drift it cannot otherwise detect. For vectors it is not: the lane
+  that would consume `embeddings_rederive` is deferred, so raising the flag on
+  absence would leave every upgraded store carrying a promise no pass can keep.
+  `DERIVE_COUNTERS[].flagsWhenUnrecorded` records the difference, per counter.
+- **A newer counter is stamped down by a writer, refused only by a reader.**
+  `records` is canonical, so a writer can restore the index to its own rules
+  losslessly in either direction, and an older `sync` keeps working across a
+  version crossing instead of hard-failing; a reader cannot re-derive anything,
+  so it refuses by name. The cost of crossing in either direction is one more
+  re-derivation, never data loss.
+- **Counters are judged numerically, and an unparsable value is refused.** `"10"`
+  is newer than `"2"`, which a string comparison would invert — worth pinning
+  now, because C3/C4/C5 will reach two digits. A value this build cannot compare
+  is a value it cannot vouch for, so the read gate treats it as newer (the next
+  writer open stamps a comparable one).
+- **No operator surface (yet).** The counters and flags are store state, and the
+  pass reports `derive_ms`; a `status` field or robot envelope for the counters
+  was deliberately left out of the cycle.
+
+## Measurements
+
+Host: Apple M1, macOS 25.6, APFS. Fixture: a **byte-consistent copy** of the
+production store (`~/.local/share/ledger/ledger.sqlite`: 1,121,468,416 bytes,
+225,959 records, 2,600 sessions, 2,664 ingest cursors, 4,487 queued items, 0
+embeddings, WAL 0 B) — the user's store is never opened for writing, and the
+copy was deleted afterwards. Harnesses were throwaway; the properties they
+measured are pinned by `tests/derive-counters.test.ts`.
+
+Reproduction: copy the store byte-for-byte (`db` + `-wal` + `-shm`) to a scratch
+path — the production store is never opened for writing — then (1) `new
+TrajectoryDb(copy)` and read `schema_meta` back (counter detection), (2) time
+`reindexRecordsFts()`, (3) for the breakdown, re-raise the flag by hand and run
+the same statements from `schema.ts` (`RECORDS_FTS_DDL`, `FTS_REPOPULATE_SQL`,
+`verifyRecordsFtsRebuild`) individually inside one transaction, (4) compare
+`TrajectoryReader.queryLexical(q, {limit: 50})` on the copy and on the
+production store (totals plus ordered `record_id`s), and (5) diff the counts and
+an `ingest_cursors` digest before/after, then `PRAGMA wal_checkpoint(TRUNCATE)`
+and compare the database file size.
+
+**OBSERVED**
+
+| measurement | value |
+| --- | --- |
+| writer open of the 1.12 GB pre-counter store (WAL pragma, DDL, trigger repair, counter detection, stamp + flag) | **8.3 ms**, `schema_meta` after it: `fts_format=1`, `embeddings_format=1`, `fts_reindex=1` (no `embeddings_rederive`: unrecorded, not moved) |
+| `reindexRecordsFts()` on the copy, first (cold) pass | **33,253 ms** |
+| the same statements in one transaction, warm (two runs) | **19,443.8 / 19,521.1 ms** |
+| phase breakdown (warm, same statements, timed individually) | drop+DDL **105.8 ms** · project (`INSERT INTO records_fts SELECT …`) **11,072.8 ms** · gates (`records`↔`records_fts` parity + FTS5 `integrity-check`) **5,036.8 ms** · automerge reset **14.2 ms** · flag clear **0.3 ms** |
+| index parity after the rebuild | `records` 225,959 = `records_fts` 225,959 |
+| store size | **1,121,468,416 bytes before and after** (identical: the rebuild reuses the freed pages); WAL peak **381.8–382.0 MB**; `wal_checkpoint(TRUNCATE)` afterwards 0.8 s |
+| untouched durable state | records 225,959→225,959 · sessions 2,600→2,600 · embeddings 0→0 · cursors 2,664→2,664 with a **byte-identical digest** (mtime, size, `bytes_ingested`, `record_count`, `status`, `updated_at`) · queue 4,487→4,487 |
+| query equality, re-derived copy vs production index | 10/10 probes identical in **total hits and ordered hit lists** (`zirconium` 0, `porter unicode61` 49, `index-shape` 17, `derive counter` 4, `ledger sync pending` 1, `records sqlite` 22, `canonical trajectory` 23, `trigram`/`wal_autocheckpoint`/`ERROR ENOENT` at the limit 50) — 0 mismatches |
+| second pass, nothing pending | `timing.derive_ms` **0**, `records_fts` content byte-identical (test-pinned) |
+| CLI, real store (`ledger sync` prose + `--robot`, `/tmp` fixture) | upgrade pass: `Index re-derivation: 1ms (records_fts rebuilt from this store's records; no source was re-read).`, `0` workers, `0` source bytes read, `0` admissions; the pass after it prints no line and reports `derive_ms: 0` |
+| CLI, refusal surface | `search --robot` and `status --robot` on a store whose `fts_format` is `2`: `{"error":{"code":"FORMAT_MISMATCH","message":"Database at … was derived under newer rules (fts_format: 2; this build derives 1). Upgrade Ledger to read this store."}}`, exit 1 |
+| existing suite | 561 pass / 0 fail (552 tests in the tree at HEAD) |
+
+**INFERRED** (not directly measured, listed so reviewers can attack them)
+
+- The cold/warm spread (33.3 s vs 19.4–19.5 s) is page cache and host load, not
+  a different code path: the cold run was the copy's first touch, the warm runs
+  followed it in separate processes. The projected cost on the production store
+  is the warm band; a first post-merge `ledger sync` on a cold page cache should
+  be expected at the high end.
+- The gates are 26% of the warm cost (5.0 s of 19.5 s) because FTS5's
+  `integrity-check` walks the whole index. It is kept: the same gate is what
+  makes a drifted index a *repair* rather than a silent inheritance, and the
+  parity count alone cannot see a table that answers `MATCH` while its internals
+  disagree with its rows.
+- The unrecorded-counter one-shot on a real upgrade is, in the typical case, a
+  repair rather than a fix: the v11 rebuild and ingest write the same projection,
+  so a pre-counter store's index is very likely already conforming. It is
+  charged anyway because the store cannot name its rules — and it is not wasted
+  work: the test plants a missing row *and* an orphan row, and re-projection
+  removes both.
+- Detection cost is one indexed `schema_meta` lookup per counter per open
+  (primary key), measured only as part of the 8.3 ms open on a 1.12 GB store.
+- Version skew costs one rebuild per crossing (older writer stamps its own value
+  and re-derives; the newer build re-derives again on its next pass). Not
+  measured — it needs two builds — but both directions are the same code path
+  the tests pin for a *newer* counter, and neither can lose canonical records.
+- The WAL peak (382 MB) scales with the indexed text (~222 MB across the
+  counter's columns), not with the store size (1.12 GB), because the rebuild
+  rewrites only FTS pages; a store with much larger `record_json` would show the
+  same WAL with a bigger database file.
+- On this host the deferred vectors flag cannot fire: the production store holds
+  0 embeddings. The lane's absence is therefore not observable in production
+  today, only in tests.
+
+## Position in the program (C2 row, `docs/research/2026-09-16-wake-port-program.md`)
+
+- Failure it fixes: *"Index-shape changes currently must ride schema migrations;
+  re-derivation has no first-class switch."*
+- Open-source case: *"Lets contributors evolve the index without migration
+  ceremonies — the difference between 'rebuild your DB' and 'it re-derives once
+  on next scan'."*
+- Ordering: C2 precedes C3/C4/C5 because they change index shape and need the
+  one-shot. Each of them now bumps one string (`FTS_FORMAT`) and gets exactly
+  one re-derivation per store on the next pass; C3's echo filter additionally
+  has to accept that a daemon-only machine re-derives when its next `sync` runs
+  (see below).
+
+## Boundaries and follow-ups (not in this cycle)
+
+1. **`sync` is the only pass that executes the flag.** `ledger build` and a
+   `ledger watch` drain are writer passes that could call the same one-liner
+   (`reindexRecordsFts`), and a machine that runs the watcher plus the scheduled
+   sync gets the re-derivation through the scheduled pass anyway. The program's
+   C2 row names the scan pass and this cycle keeps its blast radius to `sync`;
+   wiring the other two passes is a one-line change each, and the cycle that
+   first *needs* it on a daemon-only machine (C3) should take it.
+2. **The embeddings lane's execution is deferred, by the design's allowance.**
+   The counter, the flag, the stamp, and the refusal exist and are tested; no
+   pass clears `embeddings_rederive`, and none is raised by an unrecorded
+   counter, so no live store carries it today. Making it real needs a decision
+   the lane cannot avoid: vectors carry provider and dim per row but **no
+   per-row derivation-rule marker**, so "which rows are stale" is unanswerable
+   from the store alone. The two shapes are (a) a rule-generation stamp in
+   `schema_meta` plus rows written since it (a resumable, bounded re-embed), or
+   (b) whole-store re-embedding in one call, cleared only on completion
+   (Wake's scanner shape, idempotent but unbatched). Either way the query path's
+   `ensureQueryEmbeddings` gap-fill must not become an unbounded re-embed.
+3. **No operator surface for the counters.** `derive_ms`, the prose line, and
+   the store's own `schema_meta` are the visibility this cycle ships; a
+   `status`/robot field for the counters and pending flags is a candidate when a
+   bump actually ships (C3+).
+4. **Wall budget.** The one-shot is charged to the pass's wall (`elapsed_ms`
+   includes it) and is not interruptible mid-transaction; a dedicated budget
+   knob was not added. A pass killed during it rolls back and leaves the flag.
+5. **`parentage.test.ts` store reset.** The v3-migration case deleted the store
+   file but left its `-wal`/`-shm`, so the fixture written at the same path was
+   read through a WAL describing a different database (`SQLITE_IOERR_SHORT_READ`
+   once the counter's own writes changed the WAL's shape). The reset now removes
+   all three files, mirroring the file's own `beforeEach` — the latent hazard was
+   in the test, not in the migration path it checks (that path is covered by
+   `integration-migrations.test.ts` across every historical shape).