trajectories 0.0.0 → 0.1.1

package/docs/research/ledger/docs/research/2026-09-09-trajectory-and-letta-code.md

@@ -0,0 +1,128 @@
+# Trajectory and Letta Code: integration evidence
+
+Source refresh: 2026-09-09. Retain published `@letta-ai/trajectory` for canonical
+normalization; adapt selected Letta Code lifecycle contracts independently. Letta
+Code is not an adopted dependency or a replacement for Ledger's durable store.
+This addendum supplements the [prior comparison](2026-09-07-confer-inspirations.md)
+and [service review](2026-09-07-letta-continuous-service.md), without repeating them.
+
+## Pins and scope
+
+| Inspected source | Exact commit | Version / boundary |
+| --- | --- | --- |
+| letta-ai/trajectory | `21ae92d10579499b183d93929c85da0b91233167` | GitHub HEAD and npm latest were still `0.3.0`, unchanged from the prior research; installed package agreed. `NORMALIZER_VERSION=0.3.0`, `CANONICAL_SCHEMA_VERSION=3`. [Constants](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/src/version.ts#L1-L16) |
+| letta-ai/letta-code | `170a6d190eb70b65ebf0d4e1683f971e18633997` | Package `0.31.14`; pins Trajectory **`0.2.0`**, not `0.3.0`. [Package](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/package.json#L1-L3), [dependency](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/package.json#L123-L135) |
+| Ledger research baseline | `571c14498acad05c89d2fefc14b8ee3ae775fd11` | Original `main` inspected during the source refresh. |
+| Ledger PR2 research head | `94f70c5fa46fcc8f5ccbdf477d67c0cd2c4a32cc` | Historical comparison point, not the final merged PR2 head. |
+| Ledger PR6 research head | `1efa46d9ac7164f2c32b2a47957b6ea732322f8d` | Historical comparison point, before current integration repairs. |
+
+At preservation time, this worktree starts at PR2's merge commit
+`7fe199ce9d62e4a9e966aff18b7c33eda917ac40`. The coordinator reports PR6 integration
+underway with Ledger DB schema 6, atomic cursor commits, and bounded chunks. Those
+changes were not evaluated by this source pass: the old PR findings below must
+not be presented as current-main failures or as verified repairs.
+
+Evidence came from fresh primary GitHub source/API reads, with both upstream HEADs
+rechecked during the pass. Confer's direct shell was unavailable; registrar and
+doctor checks found a usable queue, but job
+`499e3ea4-cc67-4619-b200-c895c4231a14` failed because Letta Code was absent from its
+catalog. That job supplies no findings. Upstream test source was inspected, not
+run. No daemon/runtime verification is claimed here; main owns that observation.
+No upstream clones, native transcript inspection, or secret values are included.
+
+## Canonical contract already available in 0.3.0
+
+- **Identity and versions are separate.** `record_id` hashes native group,
+  source-record identity, and semantic component key, without source type.
+  Native/location identities support conflict detection; content fallback only
+  supports exact dedup. `content_hash` covers normalized semantic content;
+  `record_hash` covers the full normalized record, including timestamps.
+  Ledger supplies source scope, content versions, and durable ingestion order.
+  [Implementation](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/src/canonical.ts#L43-L153)
+- **Partial is not arbitrary streaming.** Explicit `partial:true` permits an
+  initial single-role fragment; a positive `baseByteOffset` implies partial mode.
+  Canonical meta appears only at offset zero, and cross-chunk tool results retain
+  their call IDs. Byte offsets affect byte anchors, not ordinals: id-less Letta
+  Code rows are ordinal-anchored. Preserve native group context and test each
+  adapter's chunk equivalence rather than generalizing from JSONL syntax.
+  [API](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/src/index.ts#L94-L172),
+  [tests](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/test/canonical.test.ts#L468-L584),
+  [Letta ordinal fallback](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/src/adapters/letta-code/index.ts#L60-L69)
+- **Normalization is neither raw retention nor an ingestion engine.** The API
+  accepts a complete string; JSONL parsing builds arrays and skips malformed rows
+  with diagnostics. Tool argument/result bounds cap fields, not peak memory.
+  Canonical JSON preserves emitted normalized records, not original source bytes.
+  Persist diagnostics and the returned normalizer/schema/effective configuration.
+  [Parser](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/src/adapters/shared.ts#L10-L45),
+  [bounds](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/src/bounds.ts#L17-L24),
+  [result envelope](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/src/canonical.ts#L338-L349)
+
+## Letta Code's actual use, not inferred infrastructure
+
+- **Historical export:** discovery feeds whole-input `normalizeTranscript` calls,
+  JSON files, and a manifest, defaulting to `/tmp/letta-trajectories`. Re-export
+  removes previous export contents. Search scans user/assistant prose for a
+  case-insensitive substring. This path is a replaceable export, not canonical
+  persistence, incremental sync, or a continuously maintained search index.
+  [CLI](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/subcommands/trajectories.ts#L23-L62),
+  [export implementation](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/subcommands/trajectories/export.ts#L150-L334),
+  [query implementation](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/subcommands/trajectories/review.ts#L167-L200)
+- **Own-turn capture and correction to the prior review:** native rows append to
+  `$LETTA_TRANSCRIPT_ROOT` or `~/.letta/transcripts/<agent>/<conversation>/transcript.jsonl`,
+  with a separate `state.json`. Reflection later calls `normalizeTranscript` with
+  `partial:true`, not `normalizeToCanonical`. The listener attempts capture before
+  reflection, but **catches append failure and still evaluates reflection**.
+  Therefore the prior capture-before-trigger observation is ordering, not a
+  capture-success gate. Ledger must define its own committed-evidence prerequisite.
+  [Root](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/utils/transcript-paths.ts#L4-L21),
+  [capture](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/helpers/reflection-transcript.ts#L914-L952),
+  [normalization](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/helpers/reflection-transcript.ts#L630-L641),
+  [decisive caller](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/websocket/listener/turn-completion.ts#L67-L105)
+- **Sync, watch, and query boundaries:** completion sync sends Cloud a
+  `reflected_through_message_id`; the inspected filesystem watcher watches new
+  worktrees, not harness history. Separate local-backend search treats vector and
+  hybrid requests as FTS-lite without a vector index. None proves fleet transcript
+  synchronization or continuous cross-harness indexing.
+  [Completion sync](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/helpers/reflection-completion.ts#L40-L130),
+  [watcher](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/websocket/listener/worktree-watcher.ts#L120-L158),
+  [local query](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/backend/local/transcript-search.ts#L409-L460)
+- **Useful later Dream contracts:** reflection completion is gated on memory
+  integration, not merely subagent success. Cron has persisted PID/token ownership
+  and run outcomes, but per-minute dedup remains in memory. Adapt explicit
+  ownership and completion semantics; do not inherit an exactly-once claim.
+  [Integration gate](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/helpers/reflection-launcher.ts#L628-L634),
+  [completion caller](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cli/helpers/reflection-launcher.ts#L868-L906),
+  [lease](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cron/cron-file.ts#L646-L682),
+  [dispatch](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/src/cron/scheduler.ts#L498-L594)
+
+## Historical PR findings versus remaining architecture
+
+| Pinned observation before integration | Disposition / minimal Ledger-owned contract |
+| --- | --- |
+| PR2 scopes Ledger IDs; PR6 passes stored session IDs as native `groupId`. The branches also assigned incompatible meanings to DB migration v4. [PR2 identity](https://github.com/archeism/ledger/blob/94f70c5fa46fcc8f5ccbdf477d67c0cd2c4a32cc/packages/core/src/normalizer.ts#L102-L106), [PR6 handoff](https://github.com/archeism/ledger/blob/1efa46d9ac7164f2c32b2a47957b6ea732322f8d/packages/core/src/ingest.ts#L196-L217), [PR2 migration](https://github.com/archeism/ledger/blob/94f70c5fa46fcc8f5ccbdf477d67c0cd2c4a32cc/packages/core/src/schema.ts#L76-L107), [PR6 migration](https://github.com/archeism/ledger/blob/1efa46d9ac7164f2c32b2a47957b6ea732322f8d/packages/core/src/schema.ts#L117-L145) | Integration concern, not a newly asserted main defect. Schema-6 work is underway. Keep Ledger-scoped keys distinct from native group identity, including append paths and relationships; cover both historical DB shapes. |
+| PR6 commits records, optionally awaits embeddings, then writes its cursor separately. [Sequencing](https://github.com/archeism/ledger/blob/1efa46d9ac7164f2c32b2a47957b6ea732322f8d/packages/core/src/ingest.ts#L132-L144) | Atomic-cursor integration is underway. Commit canonical records and normalization progress together; keep embeddings retryable enrichment outside that boundary. |
+| PR6 assumes growth means append continuity, initializes progress from full file size, and allocates the entire suffix. A skipped incomplete initial row can be lost if its later suffix contains other valid rows. [Initial cursor/growth](https://github.com/archeism/ledger/blob/1efa46d9ac7164f2c32b2a47957b6ea732322f8d/packages/core/src/ingest.ts#L119-L168), [suffix read](https://github.com/archeism/ledger/blob/1efa46d9ac7164f2c32b2a47957b6ea732322f8d/packages/core/src/ingest.ts#L225-L241) | Bounded-chunk integration is underway. Complete-byte cursors, UTF-8/partial-line handling, and adapter-specific replay equivalence are the contracts to validate, not claims established here. |
+| PR6 skips existing record IDs without comparing semantic hashes; changed full sessions still replace prior records. [Append dedup](https://github.com/archeism/ledger/blob/1efa46d9ac7164f2c32b2a47957b6ea732322f8d/packages/core/src/db.ts#L385-L418), [replacement](https://github.com/archeism/ledger/blob/1efa46d9ac7164f2c32b2a47957b6ea732322f8d/packages/core/src/db.ts#L306-L349) | Separate archival/versioning followup. Atomic cursors and bounded chunks alone do not establish retained source generations, content-conflict handling, or rebuildable evidence. |
+
+Remaining architecture: retain raw source generations/range digests before
+advancing capture progress; bind normalization to normalizer/schema/config
+versions and diagnostics; version or quarantine same-identity semantic conflicts.
+Keep source timestamps descriptive, not ingestion cursors. Dream needs durable
+jobs, evidence-bound publication/invalidation, and separate promotion authority;
+fleet/live capture needs explicit identity and privacy contracts. These are
+separately scoped Ledger-owned followups, not supplied upstream features. This
+source note does not certify implementation status or merge readiness.
+
+## Adoption and licensing
+
+Adopt the published Trajectory API; independently implement Ledger's storage and
+progress contracts. Learn from Letta Code's explicit ownership and integration
+completion gate, not its replaceable export directory, in-memory cron dedup, or
+best-effort capture ordering. No upstream implementation was copied or vendored.
+
+Both inspected upstream repositories declare Apache-2.0; applicable licenses and
+notices still apply. [Trajectory license](https://github.com/letta-ai/trajectory/blob/21ae92d10579499b183d93929c85da0b91233167/LICENSE),
+[Letta Code license](https://github.com/letta-ai/letta-code/blob/170a6d190eb70b65ebf0d4e1683f971e18633997/LICENSE).
+Ledger's [repository guidance](https://github.com/archeism/ledger/blob/43081aef54c23b4bd017b6c0e11fc42a9a1cb381/AGENTS.md#licensing) still records no selected
+project license and prohibits copying/vendoring/deriving Obelisk code without an
+explicit licensing decision. No new dependency or licensing decision is made here.

package/docs/research/ledger/docs/research/2026-09-13-diagnostics-coverage.md

@@ -0,0 +1,116 @@
+# Durable normalization diagnostics as coverage records, 2026-09-13
+
+## Outcome
+
+Ledger now persists the normalization diagnostics of `@letta-ai/trajectory`
+as durable, attributable coverage records (schema v9 `ingest_diagnostics`)
+and surfaces them through `ledger status`. Before this change the diagnostics
+were collected by the normalizer and dropped by `ingest.ts`; the only honest
+signal of lossiness (for example `tool_result_truncated` under the default
+2,500-code-point tool-result bound) was invisible in the store, the CLI, and
+the robot contract.
+
+The increment is stacked on PR16's `codex/ledger-cli-evaluation` head because
+the status surface and `store-status.ts` live there. PR8-owned behavior is
+untouched; the one PR8-file edit (`normalizer.ts`) is an additive export.
+
+## Evidence chain
+
+- Round-1 Confer research pinned the upstream contract:
+  diagnostic codes are additive and never repurposed, and diagnostic text
+  never contains raw transcript content
+  (`letta-ai/trajectory@21ae92d10579499b183d93929c85da0b91233167:CANONICAL.md#L204-L209`,
+  `src/types.ts#L96-L118`), which makes the messages safe to persist verbatim.
+- The dropped-plumbing defect was verified locally at base `988f52e`:
+  `packages/core/src/normalizer.ts#L98` returns
+  `NormalizedSessionResult.diagnostics`; `ingest.ts` never read the field.
+- The installed `Diagnostic` shape (`dist/types.d.ts#L67-L73`) is
+  `{ code, message, inputLine?, recordIndex?, count? }` — there is no severity
+  field anywhere in the package, so the design's severity column was dropped
+  rather than stored permanently NULL. `count` is pre-aggregated by upstream
+  for `timestamps_interpolated`/`timestamps_synthesized` and summed honestly.
+- Gap alignment: `docs/inspirations.md` names "expose unsupported/failed/
+  pending sources" and evidence-bearing completeness reporting as remaining
+  work; AgentsView's doctor is the per-root ok/missing/error pattern and Cass
+  carries a `_meta.search_completeness` field for the same reason. This is
+  Ledger's first durable piece of that surface.
+
+## Design
+
+- `ingest_diagnostics` is keyed `(session_id, code)`; `source` is stored for
+  filtering. Rows carry `occurrence_count`, a bounded code-point-safe
+  `sample` (last message, ≤200 code points), and `first_seen_at` /
+  `last_seen_at`. Sessions are globally unique by id, so replacement and
+  deletion key on `session_id` alone.
+- Replacement semantics, not accumulation: each successful whole-file
+  normalization replaces the session's rows inside the same transaction as
+  records and cursor. Identical re-imports are idempotent; counts never
+  double; a session that normalizes cleanly loses its old rows.
+- Chunked honesty: a pass publishes live coverage only when it normalized the
+  whole file itself (`!append && complete`). Multi-chunk rebuilds merge each
+  chunk's aggregate into the stage (`ingest_staged_sessions.diagnostics_json`,
+  durable across restarts) and publish the union at promotion. A promotion
+  with no aggregate clears rows instead of keeping the predecessor's —
+  unknown beats wrong. Live append continuations keep the last whole-file
+  coverage and are documented as not re-counting appended content.
+- `ledger build` (the `saveSession` path) publishes the same coverage so a
+  build-populated store cannot silently report zero findings.
+- Status surface: robot/JSON gains an additive `diagnostics` object
+  (`{ totalSessions, totalOccurrences, byCode[] }`), `null` on stores that
+  predate the table (read-only `sqlite_master` probe — `status` still never
+  migrates); human output distinguishes `unknown (store predates diagnostics
+  schema)`, `none`, and the findings block (top codes by occurrences).
+- Invariant hardening: an `AFTER DELETE ON sessions` trigger removes a
+  session's diagnostics rows even when a raw connection has
+  `foreign_keys=OFF`; promotion also deletes explicitly.
+- `commitIngest`/`commitStagedIngest` take an options object for the new
+  payload (`{ diagnostics? }`) so later extensions (PR8 adds
+  `allowPathReplacement` positionally) cannot hit parameter-order traps.
+
+## Verification
+
+Independent verifier round (scripts under `/tmp/dx-verify`, base worktree at
+`988f52e`, changed tree uncommitted; every number below is from that round):
+
+- Stability: full suite 289 pass / 0 fail / 1846 assertions across 26 files,
+  three consecutive runs (25.9s, 25.5s, 25.3s). Base tree: 269 / 0 / 1670
+  across 25 files, three runs. Typecheck and build clean.
+- Transient-failure chase: one new e2e test initially re-downloaded a 22 MB
+  ONNX model per run under an isolated HOME (35.5 s wall, bun 5 s timeout,
+  5/5 individual timeouts); root-caused and fixed to inherit HOME like the
+  sibling subprocess tests (0.49 s with the shared model cache).
+- Capability: 16/16 assertions on a fresh mixed corpus via the canonical CLI
+  (`build` + `status --robot`): 3 truncation occurrences attributed to exactly
+  the loud session; clean corpus reports `none`; rewriting the source clean
+  makes findings disappear (no accumulation). Negative control on the base
+  tree scores 2/16 — the base payload has no diagnostics key at all.
+- Atomicity probe: a poisoned aggregate (null code) throws inside
+  `commitIngest`; sessions, records, cursor, and diagnostics all roll back
+  together (7/7). Staged-union probe: a rebuild consumed across five chunk
+  passes published exactly the whole-file ground truth at promotion (5/5).
+- Overhead (N=30 passes, 12-session corpus, embeddings disabled, same script
+  bytes on both trees): ingest median 44.1 ms base vs 44.4 ms changed
+  (+0.8%, ranges overlap); records/pass and sessions/pass identical
+  (236/12). In-process `readStoreStatus` gains ~50 µs; CLI `status --robot`
+  wall time is indistinguishable (bun startup dominates).
+- Migration: 19/19 assertions. A v8 store stays v8 under read-only status
+  (`diagnostics: null`, human `unknown`), upgrades on the next writer open
+  with zero fabricated rows, and a raw `DELETE FROM sessions` under
+  `foreign_keys=OFF` still removes the session's diagnostics rows (trigger).
+
+## Boundaries
+
+- Occurrence units are per-code heterogeneous (records-affected for
+  `timestamps_*`, diagnostic counts elsewhere) — documented on the type, not
+  normalized away.
+- Ledger's own `warning()` strings (paths, budgets) remain transient ingest
+  results by scope decision; persisting them is follow-up work with a
+  privacy decision attached.
+- Brand-new multi-chunk listings publish no coverage until a whole-file
+  normalization happens (documented residual; closing it requires staging
+  cold partial passes, which the existing ingest-staging contract pins
+  against). Zero rows therefore render as `none`, indistinguishable from an
+  observed-clean pass for such sessions.
+- No raw source retention: this records the normalizer's observations, not
+  source bytes. The archival boundary remains open with its own policy.
+- No upstream code copied; Trajectory remains the only adopted dependency.

package/docs/research/ledger/docs/research/2026-09-13-fts-rebuild.md

@@ -0,0 +1,83 @@
+# FTS rebuild migration (schema v11) and the measured rejection of the prefix index, 2026-09-13
+
+## Outcome
+
+Cycle 2's query widening is inert on the production store: 156,801
+records always exceeded `LEDGER_WIDEN_MAX_RECORDS` (10,000), and widened
+queries measured 0.14–1.5 s there. This cycle set out to add an FTS5
+`prefix=` index and **measurement rejected it**: on a byte-consistent
+copy of the live store, prefix options ('2 3', '2 3 4') added
++106–151 MB (+13–19% store) and ~+35% ingest write cost with **no
+measurable latency benefit** — the residual cost for common short
+prefixes is match-set/bm25 evaluation, not the index seek. The shipped
+increment is what the measurement actually validated: a one-time
+**rebuild of `records_fts`** (schema v11), which made widened queries
+5–10× faster and exact-term queries 8–80× faster at near-zero size cost,
+plus a measured-basis raise of the widening ceiling to 250,000 records.
+Closes #21.
+
+## Design
+
+- FTS5 virtual tables cannot be `ALTER`ed; the migration (inside the
+  existing crash-atomic `migrate()` transaction, under the writer lease)
+  drops and recreates `records_fts` with the **unchanged** DDL and
+  repopulates from `records` via a projection mirroring
+  `insertRecordsInternal` value-for-value (single DDL constant shared by
+  fresh stores and the rebuild so they cannot drift). Post-copy gates
+  inside the transaction: row parity (`records` vs `records_fts` counts)
+  and the FTS5 `integrity-check` command — a failure aborts and leaves
+  v10 authoritative. `user_version = 11` is the marker. The gates also
+  heal drift (lost/orphan FTS rows).
+- Widening ceiling `DEFAULT_WIDEN_MAX_RECORDS` 10,000 → 250,000, on the
+  measured basis below; `LEDGER_WIDEN_MAX_RECORDS` semantics unchanged.
+
+## Verification (two independent measurement rounds + adversarial audit)
+
+All numbers from byte-consistent copies of the live store
+(156,801 records / 811 MB):
+
+- **Rebuild cost**: wall 11.5 s idle / 35.6–38.1 s under host load; WAL
+  peak 274 MiB (~2.8% of volume headroom), truncated after commit; net
+  store +1.8 MB; parity and integrity-check pass inside the migration.
+  Concurrent writers refuse fast via the cycle-3 lease (250 ms → typed
+  `WRITER_BUSY`) instead of blocking behind the ~27 s write-lock hold;
+  readers unaffected (WAL).
+- **Latency (engine ms, old → rebuilt)**: widened zero-hit probes
+  'records sqlite' 890 → 85/60; 'daemon watchdog' 178 → 19/37;
+  'canonical trajectory' 219 → 106/130 (idle round) and 840 → 474 first /
+  42 → 40 warm, 3456 → 1731 widened (loaded round — magnitudes are
+  load-dependent; direction unanimous). Exact-term probes 88 → 5.6,
+  82 → 10.8, 55 → 0.8, 34 → 1.0, 78 → 9.9 ms. Common short prefixes
+  ('the*', 'run*') remain ~0.46–0.69 s in every shape — match-set-bound,
+  documented as the residual cost class.
+- **Result equality**: 10/10 and 8/8 probe queries (widened and exact)
+  byte-identical hits, order, ranks, snippets across the rebuild.
+  FTS rowids are remapped; no consumer depends on them (verified).
+- **Prefix option (rejected)**: rebuild-without-prefix vs
+  '2 3 4' on identical content — widened queries within noise;
+  +151 MB size, 10,772 → 32,129 idx rows; 5,000-record ingest
+  1,919 → 2,595 ms (+35%), WAL 65.5 → 87.9 MB.
+- **Rollback atomicity**: killed mid-migration → `user_version` stays
+  10, index DDL and rows byte-identical, next open migrates cleanly.
+- **Audit fixes before shipping**: both post-copy gates now pinned by
+  real-mismatch tests (mutations that neutralize either gate fail the
+  suite); the fresh-segments rationale was itself falsified by
+  measurement (the rebuilt index has *more* segments on the maintained
+  store — the win is empirical, mechanism-neutral); binary-skew and
+  marker-only-revert documented; tool-result fallback pinned by column
+  value.
+- **Suite**: 522+ pass / 0 fail across 35 files (post-fix count per the
+  final round); typecheck/build clean.
+
+## Boundaries
+
+- The daemon's first post-merge writer open performs the one-time
+  migration on the live store (~30–40 s on this volume, under the
+  lease); pre-change binaries refuse a v11 store with the standard
+  too-new error — rollback is the `user_version` marker alone (the DDL
+  is unchanged).
+- Common-prefix widened queries remain ~0.5 s at this corpus size — a
+  ranking/candidate-cap question, not an index question; recorded.
+- Rebuild cadence: the v11 rebuild is one-time; periodic re-defragment
+  (Cass-style reconciler) is not included — `automerge=4` plus the
+  existing maintenance merge remain the steady-state discipline.

package/docs/research/ledger/docs/research/2026-09-13-read-path-split.md

@@ -0,0 +1,74 @@
+# Read-path connection split, 2026-09-13
+
+## Outcome
+
+Read verbs (`search`, `recent`, `emit-pi`) opened the write-bearing
+`TrajectoryDb`, whose constructor always writes (WAL pragma,
+`CREATE_TABLES_SQL`, trigger repair; migrations on old schemas) — so a
+search could create or upgrade a store outside the writer lease, and a
+`--semantic` query backfilled embeddings mid-read. This cycle introduces
+`TrajectoryReader`: a strictly read-only connection (`readonly: true` open
+flag — engine-enforced, irreversible; never `query_only`) with typed
+refusals (`STORE_NOT_INITIALIZED`, `SCHEMA_MISMATCH`, `STORE_UNREADABLE`),
+no DDL/migration/repair, and no embedding writes. Semantic/hybrid without
+usable embeddings answers lexically and says so via the cycle-2 envelope
+trio (`search_mode: "lexical"`, `requested_search_mode` as requested,
+`fallback_reason: "embeddings_unavailable"`) — the designed first use of
+that field pair. Closes issue #23's read-side; the lease boundary from
+cycle 3 is now complete.
+
+Query internals were extracted and shared between the writer and reader
+handles, so widening, envelope, provenance, and fusion cannot drift; the
+cycle-2 suite passes unchanged against the reader path. The writer handle
+keeps its pre-query backfill (`ensureQueryEmbeddings`) and its documented
+provider-failure asymmetry.
+
+## Evidence chain
+
+[Confer round 5](../../confer/2026-09-13-round-5/README.md) pinned the
+read-connection discipline from all six inspirations: Obelisk's
+`readOnly:true` recipe and typed schema refusals (AGPL — patterns only),
+AgentsView's reader-pool/DSN discipline and the stranded-reader-on-swap
+hazard avoided by per-invocation readers, Cass's "a strict read never
+repairs" rule and byte-invariance test (rider — patterns only),
+Trajectory's `openWithWalFallback` writable-retry identified as the
+anti-pattern behind the round-4 stale-`-shm` failure, letta-code's
+manifest refusal with remedy in the message.
+
+## Verification (combined adversarial + independent rounds)
+
+- Write-freedom confirmed: db and `-wal` byte-identical after every read
+  verb across WAL/non-WAL shapes, refusal paths, and a live writer with an
+  open transaction; no sidecar or lock file ever created; `user_version`
+  never bumped. (`-shm` is shared wal-index bookkeeping SQLite may refresh
+  once on first open of a dirty WAL — documented, excluded from the
+  data-hash contract.) Engine-level test proves the handle rejects writes
+  (SQLITE_READONLY) — mutation-verified (removing the readonly flag fails
+  two tests).
+- Behavioral drift: none — 14 lexical query shapes plus the full envelope
+  identical to baseline `a19916b`; semantic/hybrid with embeddings
+  byte-identical between handles. Intended changes only: read path never
+  backfills; provider failure with vectors present now answers lexically
+  (documented) instead of throwing on the writer handle too.
+- Concurrency: reads serve the last committed snapshot during a held
+  `BEGIN IMMEDIATE`; 120 concurrent read verbs against a write burst with
+  periodic checkpoints all succeeded; the live 820 MB store read correctly
+  while a real `ledger sync` held the lease.
+- Suite: 454 pass / 1 fail of 455 across 35 files — the failure is the
+  pre-existing native-cursor classification (#18, reproduced identically
+  on baseline). Typecheck and build clean. Reader suite: 22 tests / 181
+  assertions; cycle-2 and cycle-4 suites pass unchanged.
+- Human-mode note branches on reason (FTS wording pinned; other reasons
+  print `answered lexically (<reason>)`); empty stores report plain
+  zero-hit results; dimension-mismatched provider vectors fall back rather
+  than rank zeros.
+
+## Boundaries
+
+- The "initialized store" gate recognizes a `sessions` table (consistent
+  with `store-status`); a foreign file with only that table fails per-verb
+  with a raw SQLite error — a shared weakness with status, recorded as
+  follow-up (AgentsView's required-schema diff is the known shape).
+- `-shm` refresh on first open of a dirty WAL is SQLite's own wal-index
+  bookkeeping, not a Ledger write.
+- Writers unchanged: `TrajectoryDb` + lease + backfill-first semantics.

package/docs/research/ledger/docs/research/2026-09-13-runtime-rollout.md

@@ -0,0 +1,102 @@
+# Verified Mac CLI and service rollout
+
+The Mac service now uses `main` at
+`51a071ee5af3b722955260edbc669998eb025d6e` and the existing XDG database.
+[PR31](https://github.com/archeism/ledger/pull/31) preserves robot output
+compatibility and fixes Cursor schema classification, closing issue 18.
+Its merge tree exactly matched the tested `3a610b5` tree.
+The service checkout subsequently received documentation-only commit `0680cb5`.
+Upstream PR30 landed during report publication and was preserved in the report's
+merge, but was not deployed; the test and rollout claims here apply to the pinned
+source revision above, not that newer scan-outcomes change.
+
+## Changes and checks
+
+- Default robot search remains legacy hits-only JSONL or a JSON array. New agents
+  explicitly select `--output-version 2` for metadata and match provenance.
+- Cursor checks required SQLite columns before projection. Invalid schema errors
+  no longer depend on runtime-specific error metadata; transient errors retain
+  their classification. The original test passed on this Mac, but four
+  metadata-stripped regressions failed before the fix and passed afterward.
+- Full core suite: **494 passed, zero failed**, 3,393 assertions, 53.03 seconds.
+  Build, package typecheck, helper typecheck, and diff checks passed. The separate
+  observer suite passed 12 tests.
+- Independent review found embedded personal fixtures and omitted cursor/staging
+  comparison fields in the first helper. Both were corrected before publication;
+  the supplemental comparison covered the omitted fields without repeating the
+  expensive full archive scan.
+
+## Preservation and retrieval
+
+An online SQLite backup was rehearsed separately from the live store. The v6-to-v10
+migration took 23.4 seconds on that copy. Full row digests matched for 1,333,616
+records, 12,880 sessions, 14,874 embeddings, and 12,992 cursors. Old cursor identity
+and prefix hashes and staged-session metadata also matched. Before/after SQLite
+quick checks passed, with no foreign-key violations.
+
+Eleven frozen checks passed both before and after migration: original ACTI eval
+and Cursor Git-notes retrieval, the broad evaluation-repository keyword, one
+original session from each of seven populated harnesses, and a negative control.
+All positive target sessions ranked first within source-filtered top-50 results.
+Database, WAL, and writer-lock hashes were unchanged by the query batches.
+
+After stopping the old watcher, a fresh quiescent backup held 1,334,038 records
+and 12,884 sessions. The live migration preserved those counts exactly. All 11
+live checks then passed with unchanged database/WAL/lock hashes while the service
+was idle. The live negative was generated privately before querying: an earlier
+visible control literal could itself enter the archive through this agent's
+transcript. Positive cases and limits were unchanged, and both fixtures remain
+available in the private evidence bundle.
+
+These checks establish preservation and sampled retrieval, not full historical
+coverage or general ranking quality. Widening retains its 10,000-record ceiling;
+this archive is larger, so it does not automatically receive that fallback.
+
+## Installed operation
+
+`~/.local/bin/ledger` now invokes the stable workspace checkout. The launchd job
+`com.archeism.ledger` retains the existing database, all 14 configured sources,
+and discovery limit of 1,000,000 per source. It runs bounded `sync` every 900
+seconds, with one worker, 25 admitted sessions, 64 MiB source budget, 120-second
+pass budget, and 30-second worker deadlines; the old always-on watcher is stopped.
+
+Two real passes exited zero and their process trees terminated:
+
+| Pass | Duration | Published | Staged Records After | Worker Timeouts |
+| --- | --- | --- | --- | --- |
+| Initial scheduled pass | 115.1 s | No completed sessions | 1,593 | 3 |
+| Explicit resumability check | 111.6 s | 1 session / 31 records | 2,868 | 3 |
+
+Final observed totals were 1,334,069 records and 12,885 sessions. Backfill is
+**not complete**: 14,785 queue entries remained, including six timed-out items
+scheduled for retry. Existing failed source artifacts were not silently cleared.
+The service is expected to be idle between passes, with last exit code zero.
+
+## Resource and Evidence Limits
+
+The first observer had five database-aggregate timeouts on the large store;
+its incomplete result is retained. A second process-only observation reused
+the repository's PID/start-time tracking, sampled every second without database
+aggregation, and had no collection errors. It attached after startup: its
+330,727,424-byte sampled peak RSS and 29.67 observed CPU-seconds are lower bounds,
+not whole-pass maxima or a capacity guarantee. No descendants remained at exit.
+
+No model was called by the measured operations. Early Bun `resourceUsage()`
+objects serialized as `{}`; they are unavailable measurements, not zeros.
+A replay explicitly materialized their non-enumerable getters and CPU bigint
+counters. Physical process I/O bytes remain unavailable on this Mac. The evidence
+validator reports **valid, incomplete**, with only those two I/O dimensions
+unavailable; no percentile or throughput claim follows from these probes.
+
+Private artifacts are under the rollout worktree's
+`.local/runtime-upgrade-20260913/`: frozen fixtures, before/after digests,
+supplemental preservation checks, query attempts, full test log, both observations,
+service summaries, and checksummed `run.json`. They are excluded from Git.
+See [the replay protocol](https://github.com/archeism/ledger/blob/43081aef54c23b4bd017b6c0e11fc42a9a1cb381/lab/runtime-rollout/README.md).
+
+Paired rollback files are under
+`~/.local/share/ledger/backups/20260913-runtime-upgrade/`, including
+`quiescent-pre-upgrade.sqlite`, `pre-upgrade.plist`, and `rollout-state.json`.
+Stop the new job first, retain the latest database and sidecars, then restore the
+old backup with code `6a33ca80e921040981114bf3a94e9995566a0f15` and its plist.
+Never run an old writer on the migrated database or discard post-upgrade data.

package/docs/research/ledger/docs/research/2026-09-13-scan-outcomes.md

@@ -0,0 +1,80 @@
+# Per-source scan outcomes, 2026-09-13
+
+## Outcome
+
+`scanSessions` collapsed every listing failure into one `read_error`
+diagnostic delivered to an `onDiagnostic` callback that no caller passed —
+sources failed silently and reported 0 sessions; the extended directory
+branch swallowed filesystem errors with no diagnostic at all; and the
+installed Trajectory 0.3.0 cannot fix this from below: a verified probe
+of the package under Bun shows 8 of 10 file listers return silent
+`items=0` for missing, empty, and unreadable roots alike, and the SQLite
+listers' most common real failures escape as raw `SQLiteError` with no
+code. This cycle computes scan outcomes scanner-side and reports one row
+per source.
+
+- `scanSessions` returns `{sessions, outcomes}` (clean cutover; every
+  caller migrated). Each `ScanOutcome` row carries `source`, `root`,
+  a closed status (`listable`, `missing_root`, `root_not_directory`,
+  `unreadable_root`, `listing_failed`), `discovered`, and a capped
+  content-free `detail` (error class + code only; the only path in a row
+  is the root itself). Rows are sorted by source; a zero row is a signal,
+  not absence.
+- Ledger probes each resolved root itself (stat: ENOENT vs non-directory
+  vs access error) before listing, mirroring exactly what each lister
+  reads — verified against the installed adapters for all 10 library
+  sources, both native roots, and the extended locations.
+- Native sources funnel their existing diagnostics into the rows:
+  a diagnostic alongside discovered sessions keeps `listable` with the
+  message in `detail`; `listing_failed` fires only when the scan threw or
+  discovered nothing. Extended sources replaced silent skips with rows.
+- `scan --robot` JSONL emits a leading `{"type":"scan_outcomes",...}`
+  line (the `search_meta` precedent); `--json` gains `outcomes`; human
+  output is byte-identical to baseline when every source is listable and
+  otherwise prints one stderr note per anomalous source — `missing_root`
+  rows stay silent (absent defaults are normal; robots get every row).
+
+## Evidence chain
+
+[Confer round 7](../../confer/2026-09-13-round-7/README.md): the installed-
+package behavior matrix (trajectory), Obelisk's errno split and
+provenance analysis (AGPL — patterns only), AgentsView's typed
+per-source status shape over doctor's three-way text ladder, Cass's
+all-count-keys and sorted/deduped diagnostics discipline with a curated
+typed channel only (rider — patterns only), and letta-code recorded as
+the deliberate counterexample (listing hides unusable sources).
+
+## Verification (combined adversarial + independent rounds)
+
+- Root resolution: 14/14 default roots mirror the actual listers
+  (compared against installed adapter code); a fake HOME with real-shaped
+  stores for all fourteen sources yields 14/14 `listable` with discovered
+  counts equal to the library's; the real HOME yields 8 `listable`
+  (claude-code 113, codex 833, cursor 640, hermes 124, omp 77, opencode
+  932, pi 873, droid 68) and 6 `missing_root`, each verified genuinely
+  absent — no false verdicts.
+- Invariance: human stdout byte-identical to baseline `cb673ea` across
+  three HOME shapes (all-listable, partial, real); robot JSONL delta is
+  exactly the leading outcomes line; `--json` strictly additive.
+- Mutations: removing the probe or the sort each fails multiple tests;
+  the native partial-evidence edge (diagnostic + sessions → `listable`
+  with detail) is pinned.
+- Suite: 486 pass / 1 fail of 487 — the pre-existing native-cursor #18.
+  Typecheck/build clean. The audit's findings were fixed before shipping:
+  native partial-evidence no longer mislabels working Cursor installs
+  (640 sessions reported with a diagnostic), lab controller probes
+  destructure the new shape (their deeper fixture rejection is
+  pre-existing and out of scope), the root/status doc widened, default-
+  scan note noise suppressed, sort made code-unit stable.
+
+## Boundaries
+
+- Outcomes are recomputed per scan and never persisted; durable per-source
+  coverage state (a natural fit for `ingest_queue.last_outcome`) is a
+  later cycle.
+- `status` describes the source's scan; `root` names the primary
+  location — a multi-location source can be `listable` while `detail`
+  records a failure elsewhere (documented).
+- Two historical lab controllers were migrated to the new shape but
+  remain non-runnable on their original fixtures for a pre-existing
+  OpenCode validator reason (identical before this change; recorded).