trajectories 0.0.0 → 0.1.1

package/docs/research/ledger/docs/research/2026-09-16-c9-grok.md

@@ -0,0 +1,380 @@
+# C9: Grok as a first-class source, 2026-09-16
+
+## Outcome
+
+Grok Build is now a source this store discovers, reads, normalizes, and covers
+like any other: `packages/core/src/native/grok.ts` (discovery of
+`~/.grok/sessions/<url-encoded-cwd>/<session-id>/updates.jsonl`, the replay
+that turns its append-only ACP update stream into canonical records, the
+`summary.json` sidecar, and the parent-side subagent registry), wired through
+`scanner.ts`, `native.ts`, `normalizer.ts`, `parentage.ts`, and `watch.ts`.
+Sessions are keyed `grok:<uuidv7>`, the source gets its own `source_coverage`
+row, and `packages/core/tests/native-grok.test.ts` (13 cases) covers the fixture plan.
+
+The sentences this cycle exists to make impossible:
+
+1. **"Grok's streamed chunks are turns."** `user_message_chunk`,
+   `agent_thought_chunk`, and `agent_message_chunk` arrive as token fragments,
+   not messages: a record is a *run* of same-role fragments, concatenated with
+   no separator, flushed on a role change or when tool activity interrupts it.
+2. **"A tool result is whatever the last `tool_call_update` said."** Only a
+   terminal update (`completed` / `failed`) is a result, at most one per call
+   id: `in_progress` and status-less progress rows are superseded, and a
+   terminal row for a call that already has a result is counted rather than
+   projected — so a retry or a re-emitted update cannot put two results behind
+   one call.
+3. **"the exit is the evidence."** `ok = status !== "failed"` survives into
+   `records.tool_result_ok` (0/1), so a failed tool is a stored fact.
+4. **"What the adapter drops, nobody knows."** Every drop — lifecycle updates,
+   superseded progress, repeated terminal rows, unrecognized kinds, non-text
+   content, invalid or torn lines — becomes a typed, counted diagnostic in
+   `ingest_diagnostics`, with the kind names in the bounded sample.
+5. **"A subagent's parent is written on the child."** Grok records the edge
+   *under the parent* (`<parent>/subagents/<child>/meta.json`); the scan reads it
+   while it is already walking the store, and the child carries it into ingest.
+
+### Evidence, and what kind it is
+
+**OBSERVED** (this tree, this cycle):
+
+- `bun test` — 639 pass / 0 fail, 4,845 assertions, 44 files (the new file
+  contributes 13 cases / 97 assertions); `bun run typecheck` and
+  `bun run build` clean.
+- The real-corpus pass over the read-only `~/.grok` store into a scratch store
+  under `/tmp` (below): 5 sessions, 103 records, 0 parent edges, 2 failed tool
+  results preserved, FTS hits present, second pass fully deduped.
+- The primary fixture shapes, read (never written) from `~/.grok/sessions`:
+  layout, row grammar, `_meta` keys, tool statuses, and `summary.json` keys.
+
+**INFERRED** (not run here):
+
+- Compaction never truncates the transcript: `auto_compact_*` /
+  `compaction_checkpoint` rows are lifecycle, and the stream is append-only, so
+  replaying it preserves history (the scout's risk list).
+- Grok's GC can prune ephemeral worktree directories while session directories
+  survive; the reader never resolves a cwd on disk, so a pruned worktree cannot
+  break a read.
+- The registry edge is proven for a child whose session directory lives in a
+  *different* cwd folder than its parent: exercised synthetically (no real
+  subagent exists on this host), following Wake's contract.
+
+**Not done at all:** no upstream `@letta-ai/trajectory` adapter was written or
+proposed; `~/.grok` was never modified; the production Ledger store was never
+opened by this cycle; no embedding or semantic measurement was taken; no cwd
+flattening was implemented (each session keeps the cwd it ran in).
+
+## The format, as observed
+
+Layout (`~/.grok/sessions`): `<url-encoded-cwd>/<session-uuidv7>/` holds
+`updates.jsonl` (the transcript), `summary.json` (metadata sidecar),
+`events.jsonl`, `chat_history.jsonl` (a compacted snapshot — not a source),
+`signals.json`, `prompt_context.json`, and `*.lock` files;
+`session_search.sqlite` sits at the sessions root (Grok's own index, not a
+source). `subagents/<child>/meta.json` appears under a *parent's* session
+directory when a subagent was spawned.
+
+`updates.jsonl` rows are ACP notifications:
+`{"timestamp": <unix seconds>, "method": "session/update" | "_x.ai/session/update",
+"params": {"sessionId": "<uuid>", "update": {"sessionUpdate": "<kind>", …},
+"_meta": {"eventId": "<uuid>-<n>", "agentTimestampMs": <ms>, …}}}`.
+
+Aggregate counts over the 5 sessions / 226 rows on this host (content never
+printed; only kinds, statuses, key names, and counts):
+
+| fact | observed |
+| --- | --- |
+| kinds | `tool_call_update` 119, `tool_call` 27, `hook_execution` 22, `agent_thought_chunk` 19, `agent_message_chunk` 14, `user_message_chunk` 11, `turn_completed` 11, `retry_state` 2, `session_recap` 1 |
+| tool statuses | `completed` 25, `in_progress` 66, absent (progress) 26, `failed` 2 |
+| `params._meta` keys | `eventId`, `agentTimestampMs` on all 226 rows; `totalTokens`, `promptId`, `streamStartMs`, `turnStartMs`, `updateType` on 113; `chunkId` 33, `updateParams` 80, `cancelTrigger` 1 |
+| ids | `eventId` unique on 226/226 rows, prefixed by the session id; every row's `params.sessionId` and every directory name agree (5/5 sessions) |
+| text | every chunk's `content` is `{type: "text", text}`; tool results are `content: [{type: "content", content: {type: "text", text}}]` |
+| `summary.json` keys | `info{id,cwd}`, `session_summary`, `generated_title`, `created_at`, `updated_at`, `last_active_at`, `current_model_id`, `num_messages`, `next_trace_turn`, `chat_format_version`, `request_id`, `grok_home`, `agent_name`, `reasoning_effort`, `head_branch` (2/5 sessions), `git_root_dir`/`head_commit`/`git_remotes` (3/5) |
+| registry | 0 `subagents/` directories across all 5 sessions (no real parent edge exists on this host) |
+
+Row grammar → canonical record:
+
+| `sessionUpdate` | record |
+| --- | --- |
+| `user_message_chunk` run | `user` |
+| `agent_thought_chunk` run | `reasoning` |
+| `agent_message_chunk` run | `assistant` |
+| `tool_call` (`toolCallId`, `title`, `rawInput`) | `assistant-tool-call` (`tool_name` = `title`, falling back to `_meta["x.ai/tool"].name`; `tool_arguments_json` = `rawInput`) |
+| `tool_call_update` terminal | `tool` (`tool_call_id` = the call, `tool_result_ok` = `status !== "failed"`, `tool_result_json` = the result text) |
+| `hook_execution`, `turn_completed`, `session_recap`, `retry_state`, `task_*`, `auto_compact_*`, `compaction_checkpoint`, `plan`, `current_mode_update` | dropped, counted |
+| anything else | dropped, counted |
+
+## Assembly: the rules, and why each one
+
+**Runs.** Fragments are concatenated with no separator and no trimming: Grok
+splits tokens *including the spaces between them* across rows, so
+`"Read the"` + `" config"` is one record `"Read the config"`. Wake's replay
+states the same rule and the same hazard — its comment on `append_message_chunk`
+says the incremental fragments must not go through the generic content join or
+"the space the model wrote at a token boundary gets eaten"
+(`crates/wake-core/src/adapters/grok.rs#L198-L218`, `#L159-L196`). A run flushes
+on a role change or when a tool call, a terminal tool result, or the end of the
+stream interrupts it, which is Wake's `ensure`/`flush` pair (`#L159-L196`)
+narrowed to *role* boundaries: Ledger keeps a tool call as its own record
+(`assistant-tool-call`) and its result as a `tool` record, so the run cannot
+straddle them. Lifecycle rows deliberately do *not* split a run, and that is
+enough for the observed stream: every one of the three conversational local
+sessions opens with a user chunk, and its user-chunk runs are exactly its turns
+(1, 1, and 9), so a real turn boundary is a role change.
+
+**Tool results.** `tool_call` projects the call (id, name, arguments) and
+terminal `tool_call_update` projects the result, keyed by `toolCallId`;
+`completed` → `ok: true`, `failed` → `ok: false`, and no other status produces a
+record — an unknown status is counted instead of being guessed at. The
+`in_progress` rows (66 of the 119 updates here) and the 26 status-less progress
+rows carry partial or empty content and are superseded by the terminal row that
+follows (Wake back-fills the call's output the same way, overwriting with the
+final state: `#L226-L250`). The replay also remembers which call ids have
+produced a result, because nothing proves ACP never re-emits a terminal row (a
+retry after a failure is the obvious way it could): the *first* terminal row is
+the call's result and every later one is counted, so one call never carries two
+`tool` records and a later row cannot rewrite an outcome already stored. Ada's
+round-1 finding — the replay originally emitted every terminal row — was fixed
+here, with the regression below confirmed failing before the dedupe went in
+(4 projected results for two calls instead of 2). A result's text is the joined
+`content` blocks; when a row has no content block, its structured `rawOutput` is
+serialized rather than lost.
+
+**Identity.** Each projected event carries its update's own `eventId` as the
+row id, so the canonical record's identity is `native`
+(`stable_source_record_id` = the id), which is stable across appends and
+re-reads. A row without an id falls back to a synthetic
+`grok-line-<n>` — a line number in an append-only file never moves, whereas a
+byte offset would shift when an earlier run grows. Sequence (`source_order_id`)
+and per-record timestamps come from the row's position and
+`_meta.agentTimestampMs`.
+
+**Sidecar.** `summary.json` supplies the title (`generated_title`, then
+`session_summary` — Wake's precedence, `grok.rs#L73-L110`), `cwd`
+(`info.cwd`), model (`current_model_id`), branch (`head_branch`, absent in 3/5
+local summaries), and the session's span (`created_at` … `updated_at`, which
+contains the update stream's own bounds in all 5 local sessions: e.g.
+10:35:20.032Z…10:36:02.858Z around a stream of 10:35:31Z…10:36:02Z). The
+sidecar is metadata only: a missing, oversized, or malformed one costs title,
+cwd, model, and the envelope — never the transcript.
+
+## The projection lane, and the decision behind it
+
+The pinned normalizer (`@letta-ai/trajectory` 0.3.0) has **no** Grok adapter:
+`normalizeToCanonical({source: "grok", …})` fails with
+`NormalizationError(unknown_source)`, and the adapter registry is a frozen
+object, not an extension point. Ledger's established answer is the one it
+already uses twice — project the native store into a contract the library
+*documents*, and let the library do identity, hashing, bounds, and validation
+(Cursor IDE stores → capture JSONL; OpenCode SQLite → export JSON).
+
+| candidate lane | what it keeps | what it loses |
+| --- | --- | --- |
+| `atif` (the library's generic interchange format) | steps, prose, reasoning, tool calls, per-step timestamps, session + trajectory ids | **the terminal outcome**: ATIF observations carry `source_call_id` and content only, so `ok` is always `null` and a failed tool is indistinguishable from a successful one |
+| `letta-code` (kind-tagged client rows) | per-row ids, `captured_at`, `resultOk` | calls and results share one row: a result that arrives in a later update than its call cannot be projected without duplicating the call id |
+| `cursor` (role/message content blocks) | blocks, `is_error` → `ok`, native row ids | **timestamps**: the adapter reads none of them, so every record's time would be synthesized |
+| **`copilot-cli` (event-per-line JSONL)** — taken | per-event `id` and `timestamp`, `assistant.message` with `reasoningText`/`content`/`toolRequests[]`, `tool.execution_complete` with `toolCallId` + `success`, `session.start` context (`sessionId`, `cwd`, `branch`), and unknown event types dropped with diagnostics | nothing this source carries |
+
+The chosen contract is not a coincidence of taste: an ACP update stream and
+Copilot CLI's event stream are the same shape — one notification per line, an
+id per event, calls and results as separate events, and a boolean outcome on the
+result — which is why the projection is nearly a rename rather than a rewrite.
+`GROK_PROJECTION_SOURCE` in `native/grok.ts` is the single place the lane is
+named, and `normalizer.ts` restores the records' provenance
+(`record.source_type = "grok"`) right after normalization.
+
+Recorded consequences, so nobody has to rediscover them:
+
+- `records.source_type` is `grok` and `sessions.source` is `grok`; both match
+  the store's provenance axis. `ledgerId` already prefixes record ids with the
+  ledger source, so identity was never the lane's.
+- The single `meta` record's `record_json` names the *projection lane*
+  (`"source":"copilot-cli"`), because that field is the library's record of the
+  transcript it was handed, and both `record_hash` (`sha256(record_json)`) and
+  the meta `content_hash` are computed over it. Ledger does not rewrite
+  library-owned hashes; the note is the disclosure.
+- The session-level `model` comes from the sidecar (the `meta` record carries
+  cwd/branch only, since the projection's `session.start` has no model field).
+- Tool results are subject to the library's default bounds (2,500 characters,
+  head-tail); the real-corpus pass recorded `tool_result_truncated` on 15
+  records, the same policy every other source gets.
+- **A projected source's group is authoritative.** The reader stamps the
+  listing's own id on every group carrier the projection can emit
+  (`session.start`, `hook.start`), but an agent-only session with a missing or
+  torn sidecar emits neither, and the library would then fall back to its
+  `default` sentinel: every such session would take the ledger id
+  `grok:default`, and the next one would replace the previous one's records
+  through the store's same-id path — permanently, because that session's cursor
+  still matches its unchanged fingerprint. `normalizeDiscoveredSession` therefore
+  passes the group as authoritative for projected sources
+  (`options.groupId !== undefined || projected`), which is where the
+  projected-source set lives. The reader's own rows always agree with it, so
+  `source_group_conflict` cannot fire; non-projected sources are byte-identical
+  to before. This was found by the cycle's adversarial review, which reproduced
+  the loss with two sidecar-less agent-only sessions, and the regression below
+  was confirmed failing with the flag removed (`grok:default` present) before
+  being kept.
+
+## Parentage: the edge lives under the parent
+
+`<parent-session-dir>/subagents/<child-id>/meta.json`:
+`{"parent_session_id", "child_session_id", "subagent_id", "created_at"}`.
+Wake's loader is the pinned contract (`grok_group.rs#L179-L242`): the parent is
+`parent_session_id` when non-empty and the enclosing session directory otherwise
+(`#L218-L222`), *both* `child_session_id` and `subagent_id` name the child
+(`#L227-L237`), and the first registration wins (`#L235`). Ledger adds a
+self-edge drop (a session is never its own parent) and keeps Wake's chain
+walking out of scope on purpose: the edge recorded is the **direct** edge, not
+the root parent — flattening is a reporting decision, and flattening the *cwd*
+(Wake's `canonical_project`) is not implemented at all: each session keeps the
+cwd it ran in.
+
+The lookup is a *reverse* one — the child's own directory says nothing about its
+parent — so someone has to walk the store. Ledger reads the registries **in the
+scanner**, which is already walking `<root>/<cwd>/<session>/`:
+`discoverGrokSessions` collects `child → parent` while it lists, and attaches the
+edge to the child's listing as `native.parentSessionId`. That buys two things a
+per-session read cannot: the edge is fresh on every pass (a registry written
+between two syncs is visible to the next scan), and ingest pays no extra I/O —
+no per-session walk, no process-lifetime cache that could go stale. The
+extractor (`parentage.ts`, `case "grok"`) consumes the edge, and keeps the
+`…/subagents/<child>/…` path shape as its second piece of evidence for a
+listing that arrives without one. The edge participates in the session's native
+fingerprint, so learning (or losing) a parent re-ingests exactly that session.
+
+## Verification
+
+**Suite.** `packages/core/tests/native-grok.test.ts` — 13 cases:
+
+| case | what it pins |
+| --- | --- |
+| discovery | the documented root, newest-first order, sidecar title/`updated_at`, `native.kind`, the `listable` outcome and the `missing_root` outcome for an absent root |
+| dedup/symlink/page | a symlinked tree is not listed, an id is listed once, `--limit` bounds the page |
+| single turn | the exact projected transcript, the run concatenation (no separators), reasoning vs assistant records, `source_type = grok`, the sidecar envelope, and the lifecycle diagnostic with its count |
+| tools | call/result pairing by id, `tool_result_ok` 1 and 0, `tool_arguments_json`, and the superseded-progress count |
+| repeated terminals | `[completed, completed]` projects one result and counts the repeat; `[failed, completed]` keeps the first terminal (`success = false`) even when the later row contradicts it |
+| resilience | `retry_state`/`task_backgrounded` counted, an unknown kind counted, a torn line counted, the transcript still complete |
+| sidecar | no sidecar → stream-bounded span + first-user title; malformed sidecar → diagnostic, session still readable |
+| lifecycle-only | a session whose updates are all lifecycle rows projects no conversation rows |
+| parentage | explicit parent, `subagent_id`-only fallback to the enclosing directory, self-edge dropped, edges landing on both sessions' rows |
+| registry-rooted listing | a child laid out under its parent's registry is not listed by the whole-store walk, and is listed — with the parent taken from the path shape — when the scan is rooted at the registry |
+| identity/stability | unchanged re-read is `unchanged`; an appended turn adds records without moving or rewriting stored ones; sidecar-only change is `deduped` with stable record ids |
+| group identity | two agent-only sidecar-less sessions (plus one with a torn sidecar) keep distinct `grok:<uuid>` ids with both record sets intact, and no `grok:default` session exists |
+| durable skips | the reader's diagnostics are stored in `ingest_diagnostics` |
+
+The default-scan tests in `cli-canonical.test.ts` were extended with `grok`
+(source list, seeded root, and the "no root at all" set): the source is part of
+the bare `scan`/`sync` contract, so its absence is reported where the other
+fourteen are.
+
+**Real corpus.** Read-only `~/.grok`, scratch store `/tmp/c9-grok-scratch`
+(never the production store; `~/.grok` untouched):
+
+| measurement | value |
+| --- | --- |
+| discovery | 5 sessions from `~/.grok/sessions` (3 cwd folders) |
+| first `build --sources grok --embedding none` | 5 scanned / 5 imported / 0 failed / 0 tombstoned, 103 records, wall ≈ 0.12 s |
+| second pass (same store) | 5 scanned / 0 imported / 5 deduped / 0 new records — idempotent |
+| records | `assistant-tool-call` 27, `tool` 27, `reasoning` 19, `assistant` 14, `user` 11, `meta` 5 |
+| `source_type` | 103/103 `grok` |
+| failed tools preserved | 2 (`tool_result_ok = 0`) in the largest session |
+| parent edges | 0 (no `subagents/` registry exists on this host — the path is covered synthetically) |
+| diagnostics | `noise_record_dropped` 128 occurrences over 5 sessions (lifecycle + superseded progress + unknown kinds); `tool_result_truncated` 15 (library default bounds) |
+| coverage | `grok` / `/Users/mac/.grok/sessions` / `listable` / discovered 5 |
+| search | 103 FTS rows; `read_file` 3 hits, `cache` 4 hits |
+| sessions with no conversation | 2 sessions hold a single `hook_execution` row (378 bytes) and normalize to a metadata-only record set, with the session's own span from the sidecar |
+
+## Decisions
+
+1. **A borrowed lane, named in one place.** No wrapped or forked copy of the
+   library, no private canonical-record builder: Grok's records go through the
+   same identity, hashing, bounds, and validation path as every other source,
+   and the divergence (which lane) is one exported constant plus one restore
+   line. Building canonical records in Ledger was rejected as a second
+   convention: it would duplicate `buildCanonicalRecords`' identity rules and
+   drift from them silently.
+2. **`copilot-cli` rather than `atif`.** The library's generic interchange
+   format would have been the tidy choice, and it cannot express a failed tool
+   result. A stored `tool_result_ok = 0` is worth more than a tidy lane name,
+   and the lane is documented rather than implied.
+3. **Terminal-only results.** Projecting an `in_progress` row would invent a
+   result the run had not produced, and would leave a stale record behind when
+   the terminal row replaced it (`ok` would flip in place), so progress rows are
+   counted, not projected.
+4. **Native identity from `eventId`.** The update id is the only per-record
+   native identity Grok writes; using it makes appends idempotent without a
+   content digest, and the line-number fallback is append-stable by
+   construction.
+5. **Registry reads at scan time.** One walk, already paid for, beats a
+   per-session walk (`O(sessions²)` on a large store) or a process-lifetime
+   cache (a registry written mid-process would be invisible until restart).
+   The edge rides in `native_json`, so `ingest_queue`'s existing "native
+   metadata changed → reset retry state" rule re-ingests just the child.
+6. **Sidecar as envelope, stream as records.** `started_at`/`ended_at` come from
+   the sidecar when it exists: it is Grok's own record of the session's span, it
+   is never narrower than the stream, and it is the only span a hook-only
+   session has. Per-record times stay the stream's.
+7. **Skips are diagnostics, not silence.** The reader's counts land in the same
+   `ingest_diagnostics` rows as the library's diagnostics, with the dropped kind
+   names in the bounded sample, so "what did we not read?" is answerable from
+   the store.
+8. **The scan probes `~/.grok/sessions` as a directory root**, like the other
+   native sources: an absent root is `missing_root` (counted, not listed), an
+   unreadable one is `unreadable_root`, and a healthy one makes the coverage row
+   real.
+
+## Ported mechanics (Wake MIT, `iAmCorey/Wake@d9f702b6df0e85763f8ab597c09bf4ca6a168db4` v0.6.5 — mechanics with attribution, no code copied)
+
+| Ledger | Wake pin | What was kept |
+| --- | --- | --- |
+| chunk runs concatenated with no separator, flushed on role change / tool activity | `crates/wake-core/src/adapters/grok.rs#L159-L196`, `#L198-L218` | the rule and its reason (token fragments split at spaces), narrowed to role boundaries because Ledger keeps calls and results as records |
+| tool updates back-fill the call, terminal state wins, `status: "failed"` is the error | `grok.rs#L226-L250` | "the terminal update supersedes the partial ones"; Ledger stores the outcome as `tool_result_ok` rather than folding the output into the call |
+| known lifecycle kinds are named, everything else is counted, not guessed | `grok.rs#L323-L335` | the closed kind list and the counter; Ledger reports the count and the names as a diagnostic |
+| summary sidecar: `generated_title` then `session_summary`, `info.cwd`, `current_model_id`, `head_branch` | `grok.rs#L73-L110` | the precedence and the field choices |
+| parent from `parent_session_id`, else the enclosing session directory; both `child_session_id` and `subagent_id` name the child; first registration wins | `grok_group.rs#L179-L242` (`#L218-L222`, `#L227-L237`) | the whole fallback rule, plus a self-edge drop |
+| the child's registry entry is read from the parent's side, once, for the whole store | `grok_group.rs#L179-L186` | the direction of the read and its one-walk cost; Ledger moves the walk into the scan it already runs |
+
+**Adapted:** Wake's transcript message keeps thinking and tool calls *inside*
+one assistant message; Ledger's canonical contract has separate `reasoning`,
+`assistant-tool-call`, and `tool` record types, so the replay is the same state
+machine with different grains. Wake's group context also resolves a canonical
+*project* (percent-decoding the cwd folder, `grok_group.rs#L151-L178`,
+`#L265-L283`) and flattens a nested subagent to its root parent
+(`#L244-L263`); Ledger records the direct edge and never rewrites the session's
+cwd.
+
+**Not ported:** worktree registry/`worktrees.db` (`grok_worktree_slug`,
+`is_ephemeral_worktree`) and `relocations` handling, image attachment decoding
+(`apply_tool_update`'s `decode_images` path — Ledger's projection carries text
+only), and `chat_history.jsonl` reading (a compacted snapshot; `updates.jsonl`
+is the source of truth).
+
+## Boundaries
+
+- **The upstream adapter is still missing.** `@letta-ai/trajectory` 0.3.0 lists
+  fourteen sources and not `grok`; a future adapter would make this projection
+  unnecessary, and its records would not be hash-compatible with these. Cycle
+  9's "first-class" is Ledger-side: discovery, assembly, coverage, parentage,
+  identity.
+- **No cwd flattening** — a session's `cwd` is the directory it ran in, and a
+  worktree session keeps its worktree path.
+- **Discovery does not descend into a session directory.** Its other
+  subdirectories hold a session's own artifacts (`terminal/*.log`, `recap_requests/`),
+  and the child of a subagent is expected in its own cwd folder, which is the
+  layout the format spec and Wake's loader assume.
+- **Two local sessions are metadata-only** (a single `hook_execution` row each,
+  378 bytes): they are listed, with a sidecar-derived span, and the library's
+  partial-transcript path admits their meta record. Not special-cased.
+- **Unknown kinds are counted, not mapped.** A future Grok kind is a diagnostic
+  row until someone maps it here.
+- **A result whose call never appeared** is dropped by the library itself
+  (`orphan_tool_result`), and a call whose updates never reached a terminal
+  state stays a call with no result: Ledger adds no second mechanism for either.
+- **`auto_compact_*`, `retry_state`, and `turn_completed` are dropped**
+  including their token/usage payloads: usage accounting is a different feature
+  with its own consumer, not a transcript record.
+- **Embeddings were not exercised for this source** (`--embedding none` in the
+  measured pass); records are ingest-complete without them.
+- **The append path is not used.** Grok's `updates.jsonl` is incrementally
+  appendable, but the run accumulator needs the whole stream (a run can span any
+  chunk boundary), so a changed session is re-read whole under its snapshot
+  fingerprint — the same gradient OpenCode and Cursor native sources use.

package/docs/research/ledger/docs/research/2026-09-16-wake-port-program.md

@@ -0,0 +1,38 @@
+# Wake port program (2026-09-16)
+
+Ten bounded cycles porting patterns from `iAmCorey/Wake@d9f702b6` (v0.6.5,
+MIT; evidence in [confer/2026-09-14-wake/wake.md](../../confer/2026-09-14-wake/wake.md))
+plus two new harness banks. Each cycle carries: the observed Ledger failure it
+fixes (or the measured trade it buys), the pinned upstream mechanism, and the
+open-source positioning case. Order encodes dependencies: the derive-rule
+counter (C2) precedes the index-shape changes that need a one-shot re-derivation
+(C3, C4, C5).
+
+Protocol per cycle (owner-approved 2026-09-16, rewound from the 2026-09-12
+loop command via our own ledger): subagents handle explore → implement →
+adversarial review → improve → independent verify; the parent owns architecture,
+design, final review and git. Each increment opens a PR immediately, monitors
+cloud review within 10–15 min, and triages **two rounds** — fix the valid,
+explicitly defer the rest. Merge history-preserving at LGTM or only-lows,
+fast-forward main (that IS the CLI rebuild), file the cycle's follow-up issue
+for remaining nits, then continue. One consolidated follow-up issue closes the
+program if nits remain at the end. `codegraph` before grep; Wake's MIT source
+at `/tmp/ledger-confer/wake` is the authority for ported mechanisms; Gemini
+embeddings (`--embedding gemini`) only where a cycle genuinely needs vectors.
+
+| # | Cycle | Fixes / buys (concrete case) | Open-source case |
+| --- | --- | --- | --- |
+| C1 | Copy-fallback read-only SQLite (probe → copy `db`+`-wal`+`-shm` → read copy; never `immutable=1`) + `db_cache_stamp = max(mtime(db), mtime(-wal))` invalidation | A store held by its own CLI costs a coverage row today (`read_error`, `native/common.ts`). Wake's gradient converts locked stores to working reads | "Works on stores while the agent is using them" — first-run experience on any live machine; the #1 operational complaint class for tools like ours |
+| C2 | Derive-rule counter (index-format version in `schema_meta`, distinct from `SCHEMA_VERSION`; mismatch → one full re-derivation) | Index-shape changes currently must ride schema migrations; re-derivation has no first-class switch | Lets contributors evolve the index without migration ceremonies — the difference between "rebuild your DB" and "it re-derives once on next scan" |
+| C3 | Self-referential echo filter (exclude Ledger/MCP-lookup transcripts from the index; C2 bumps force re-derivation) | We index the transcripts of the agents that call `ledger search` — your own queries come back as hits. Unnamed bug, observed daily | Self-hosting agents are our primary users; a store that echoes itself reads as broken. Wake ships the same filter (`wake_*` tools, `FTS_FORMAT` counter) |
+| C4 | Title index (`titles_fts` analog, trigram-free first pass), title hits rank above body, `seq=0`, no fabricated seq | Titles render in results but are unsearchable (`records_fts` has no title column) | Session-find-by-name is the human's first query; every session browser ships it |
+| C5 | Trigram tokenizer decision + swap (with whole-query `LIKE` degradation, surfaced in the envelope) | #34 common-prefix floor: mid-token substrings and CJK are unmatchable under `porter unicode61` | Code-identifier search is the agent's query shape (`useEffect(`); Wake proves it works in SQLite FTS5 alone — **CLOSED 2026-09-16: measured, refused** (all binding bars failed — +52.5% store, 54 s re-derivation, 190–200 ms common substrings; and trigram does not fix #34, whose floor is match-set-bound: see the [C5 note](2026-09-16-c5-trigram.md) for measurements, the kept design contract, and reopen conditions) |
+| C6 | Recency prior before RRF (bm25 × time-decay, arithmetic-only) | All lanes order by relevance with `started_at DESC` tiebreak only; "the session from today" loses to an old strong match | Cheap relevance win every user feels; no extension dependency, single ORDER BY term |
+| C7 | Tombstone double-track (path + logical key, same transaction as removal) | Deletion today is undefined; source truncation is not authorization to erase, and user deletion must survive re-scans | Privacy posture: explicit delete semantics before anyone trusts us with their history; Wake's tombstones + system Trash is the reference UX |
+| C8 | Remote truthfulness primitives (unreachable ≠ empty, partial ≠ complete, per-path probe, exit-24 handling, disposable cache lifecycle) | #14/ADR-0005 blocked on the decision, but the failure-semantics vocabulary has no owner; port it as tested primitives without shipping a transport — **SHIPPED 2026-09-16: `packages/core/src/sync-truth.ts` + 25 test cases, no transport, no schema version spent** (see the [C8 note](2026-09-16-c8-remote-truth.md)) | Every multi-machine user hits this; honest sync failure is a moat vs "it silently showed nothing" — Wake's `remote.rs` is the pinned authority |
+| C9 | Grok first-class (discovery, assembly, canonical records via the Trajectory shape, registry-backed parentage, coverage row) | #51: owner-approved first-class bank; `~/.grok` carries real data on this host; its subagent registry is source-provable parentage (rare) — **SHIPPED 2026-09-16: `packages/core/src/native/grok.ts` + scan-time registry edges + 13 test cases; 5 real sessions / 103 records ingested read-only, 0 real edges, 2 failed results preserved (see the [C9 note](2026-09-16-c9-grok.md))** | Breadth is the obvious open-source pitch; parentage from a second registry source proves the model generalizes |
+| C10 | Kimi (#51; Claude-shaped transcripts, `parent_tool_use_id`) | Owner-approved bank; needs fixture sourcing before implementation — **SHIPPED 2026-09-16: `packages/core/src/native/kimi.ts` + streamed-argument assembly + path/tool-call-bound child edges + 15 test cases; 8 real sessions (3 top-level + 5 subagents) / 806 records ingested read-only into a scratch store, 5 real parent edges all tool-call-bound, 1 failed result preserved (see the [C10 note](2026-09-16-c10-kimi.md))** | Same; plus Trajectory-format conformance for another harness — production first ingest still gated on #69 |
+
+Out of scope per owner: Qoder, Antigravity (encrypted — structurally impossible), WorkBuddy; DSH deferred behind #49. Never port: whole-session replace without retained generations, stats-only dedup, in-process scan lock, dropped diagnostics counters.
+
+Order note: C1 is independent and mechanical — first. C2 must precede C3/C4/C5 (they change index shape). C5 is the largest and touches the same DDL as C4 — sequence C4 then C5, with C5's decision round explicit (index size vs substring coverage, measured). C6 after C5 (ranking experiments on the final tokenizer). C7/C8 are boundary work, independent. C9/C10 last so new banks land on the finished index.

package/docs/research/ledger/docs/research/2026-09-17-cursor-blobstore.md

@@ -0,0 +1,157 @@
+# Cursor Blob-Store Chats
+
+Audited **2026-09-17** against a local corpus of **589 `store.db` files, 492 MB**
+under `~/.cursor/chats/`, written by Cursor between 2026-05-25 and 2026-09-17.
+Every claim below was measured across all 589 files, not sampled. Nothing in the
+corpus was modified; the reader opens these databases read-only.
+
+## Why this was unsupported
+
+`discoverCursorSessions` previously walked to `store.db`, emitted
+`incomplete_evidence` with "Cursor store.db schema is unverified and unsupported",
+and moved on. Cursor's older `state.vscdb` composer store was supported, so the
+ledger held 609 Cursor sessions while 589 newer agent chats were skipped
+entirely — roughly a quarter of the local Cursor corpus.
+
+## Layout
+
+```
+~/.cursor/chats/<workspaceHash>/<agentId>/store.db
+  meta (key TEXT PRIMARY KEY, value TEXT)   -- one row, key '0'
+  blobs (id TEXT PRIMARY KEY, data BLOB)    -- id is the content's SHA-256, hex
+```
+
+`meta.value` is **hex-encoded ASCII JSON**, not raw bytes — the only field in the
+format carrying that extra layer. Decoded:
+
+```json
+{ "agentId": "<uuid>", "latestRootBlobId": "<64 hex>", "name": "New Agent",
+  "mode": "default", "isRunEverything": true, "createdAt": 1779717061154 }
+```
+
+`blobs.data` is raw: message blobs are UTF-8 JSON, the root blob is binary.
+
+## Root blob: protobuf
+
+`latestRootBlobId` names a protobuf-encoded blob. Observed fields:
+
+| field | wire | meaning |
+| --- | --- | --- |
+| 1 | 2 (32 bytes) | repeated, **ordered** message blob IDs |
+| 5 | 2 | nested record containing `system_prompt` |
+| 8 | 2 (32 bytes) | a further blob reference |
+| 9 | 2 | workspace URI (`file:///…`) — the session's `cwd` |
+| 18 | 2 | repeated filesystem paths (open/context files) |
+| 10, 21, 22, 26, 27 | 0/2 | not interpreted |
+
+The reader takes fields 1 and 9 and skips everything else **by wire type**, so a
+field Cursor adds later cannot break the walk.
+
+A naive scan for the `0A 20` byte pair instead of real varint parsing looks
+plausible and is wrong: it matched inside other fields, inflating 15,860 real
+message IDs to 16,202 and inventing 16 "missing" blobs and 176 "non-JSON" ones.
+Parsed correctly the corpus is clean — **15,860 messages, zero missing, zero
+non-JSON, zero missing roles** — and no root repeats an ID, so the content hash
+is a stable per-record identity.
+
+## Messages
+
+Each message blob is `{ "role", "content", … }` where `content` is a string or a
+block list. Roles across the corpus: `tool` 9,388 · `assistant` 4,690 ·
+`user` 1,193 · `system` 589 (one per session).
+
+Block names differ from the capture format `@letta-ai/trajectory`'s cursor
+adapter decodes, which is the substance of the translation:
+
+| Cursor block | canonical block |
+| --- | --- |
+| `text {text}` | `text {text}` (unchanged) |
+| `reasoning {text, signature}` | `thinking {thinking}` |
+| `tool-call {toolCallId, toolName, args}` | `tool_use {id, name, input}` |
+| `tool-result {toolCallId, result, experimental_content}` | `tool_result {tool_use_id, content}` |
+| `redacted-reasoning {data}` | none — see below |
+
+`providerOptions.cursor.modelName` carries the model (`composer-2.5-fast`,
+`cursor-grok-4.5-high-fast`), which the adapter reads from `message.model`.
+
+A tool-call ID is **not** a path and is not validated with `safeNativeId`: 292
+IDs in the corpus contain a separator that check rejects.
+
+## Deliberate losses
+
+Two kinds of record cannot be carried, and both are handed to the adapter
+unchanged so its own `noise_record_dropped` diagnostic records the drop rather
+than this reader hiding it:
+
+- **`system` rows** (589, one per session). The upstream cursor adapter accepts
+  only `user` and `assistant` rows. Carrying the system prompt would require a
+  change in `@letta-ai/trajectory`, not here.
+- **`redacted-reasoning` blocks.** The payload is provider-encrypted; there is no
+  plaintext to hand on, and inventing one would fabricate trajectory content.
+
+`tool` rows *are* carried, remapped to the `user` role, because the canonical
+shape keeps a tool result on the turn that carries it. Without that remap the
+adapter would drop 9,388 records — 59% of the corpus.
+
+## Measured result
+
+Against the full local corpus, after the change:
+
+- 589/589 discovered, 589/589 read
+- **22,274 canonical records** from 588 sessions
+- `cwd` on 589, title on 589, model on 577
+- one session yields no records: it has no assistant turn, an empty draft
+- normalization diagnostics: `noise_record_dropped` 5,225 (the deliberate losses
+  above), `tool_result_truncated` 3,629, `timestamps_synthesized` 588
+
+Timestamps are synthesized for every session because message blobs carry no
+per-message time; only `meta.createdAt` exists, which the reader maps to
+`startedAt`.
+
+## ACP sessions (issue #70)
+
+Cursor's ACP layout is a different path with the same file name:
+`~/.cursor/acp-sessions/<id>/store.db`, beside a `meta.json`. Measured
+**2026-09-17 on host `charles`** (Cursor CLI ACP, Linux), read-only over SSH:
+**171 session directories, 30 with a `store.db`, 1.4 GB total**. The same host
+also holds 80 `chats/` stores.
+
+The format is the same as `chats/` in every respect that matters:
+
+- schema is `blobs(id TEXT PRIMARY KEY, data BLOB)` + `meta(key TEXT PRIMARY KEY, value TEXT)`
+- `meta.value` is hex-encoded JSON in 28 of 28 readable rows
+- the root blob is the same protobuf: field 1 the ordered message IDs, field 9 the
+  workspace URI, present on all 30
+- the same block vocabulary: `tool-call`, `tool-result`, `reasoning`, `text`,
+  `redacted-reasoning`
+
+Across all 30 stores — 2,451 messages — there are **no missing blobs, no non-JSON
+message blobs, no roots with trailing bytes, no field-1 value that is not 32
+bytes, and no tool call or result missing its ID or name.** None of the conditions
+that make the reader throw occur, so the `chats/` implementation reads this layout
+unchanged, with no ACP-specific code.
+
+Two differences worth recording:
+
+- **ACP meta carries `blobEncryptionKey`**, which `chats/` meta does not. This is
+  the credential #70 warns about. The reader's allowlist — `agentId`,
+  `latestRootBlobId`, `name`, `createdAt` — already excludes it, and a fixture
+  with a planted key confirms nothing from meta reaches the transcript or title.
+- **ACP also keeps `cwd` and `title` in the sibling `meta.json`** (141 dirs have
+  `{cwd, schemaVersion}`, the 30 with a store add `title`). The reader does not
+  need it: `cwd` comes from root field 9 and the title from `meta.name`, both
+  present on all 30.
+
+### A size worry that does not hold
+
+#70 reports a live session at ~619 MB and infers a reader needs to cope with that
+volume; it has since grown to 705 MB. Measured, that database contains **185
+messages totalling 1.0 MB**. The remaining ~704 MB is blobs the current root does
+not reference — superseded roots and out-of-line content the store keeps.
+
+Because the reader walks the root's message list rather than the file, the
+budget that matters is the referenced bytes, and **no session of the 30 comes near
+`MAX_NATIVE_BYTES`**: the largest referenced total is 5.0 MB, against a 64 MB
+limit. An earlier draft of this note claimed the 705 MB session would be refused
+on the byte limit. That was wrong, and it was wrong because it reasoned from file
+size instead of measuring the referenced set.