mlx-bun 0.2.0 → 0.3.0

package/docs/design/dreaming-nightly-pipeline.md

@@ -1,291 +1,684 @@
-# The Dreaming — nightly memory pipeline (CANONICAL design)
-
-Status: **canonical**, 2026-07-01. This document states the system design as
-settled with Josh and **supersedes any conflicting statement** in
-[the-dreaming-master-plan.md](./the-dreaming-master-plan.md) (task breakdown still
-useful; several decisions overridden below),
-[memory-synthesis.md](./memory-synthesis.md) (superseded), and
-[the-dreaming-handoff.md](./the-dreaming-handoff.md) (session log, mostly aligned).
-
-The system is the second iteration of Lucien's memory pipeline, rebuilt to run
-**entirely on local models** inside mlx-bun (`src/memory/`). Same methodology and
-pipeline at the core; the local constraint drives two design consequences:
-
-1. **Small tasks.** Every AI step is decomposed to the smallest single-purpose
-   unit a small local model can do well — one chunk, one bucket, one section,
-   ideally a binary judgment. Never a whole-article blob in or out.
-   **And every AI step is pure text-in/text-out — no tool calls, no agent
-   loop.** The model never edits a file, never searches, never decides what to
-   do next; the pipeline hands it exactly the context it needs and applies its
-   output deterministically. This lowers the capability bar dramatically: a
-   small model that could never drive an agentic edit session can absolutely
-   rewrite one section given the section and the new information — and pure
-   completion tasks are exactly what per-stage LoRA SFT trains well.
-2. **Deterministic gates between AI steps.** AI does judgment; deterministic
-   code enforces invariants (citations, footnotes, wikilinks, infobox shape,
-   word floors). A weak AI output is gated to NO-OP, never allowed to corrupt
-   the vault.
-
-## Fixed principles (do not re-derive, do not contradict)
+---
+status: active
+axis: USING
+canonical-for: memory-synthesis (the Dreaming)
+plan-anchor: none
+last-verified: 2026-08-23
+---
+
+# The Dreaming — memory write path + synthesis (canonical design)
+
+This is THE design doc for mlx-bun's memory write path: the pipeline that
+turns conversations into a personal wiki (`src/memory/`, `scripts/memory/`),
+the local-model inference seam it runs on, the read path it feeds, and the
+evaluation methodology that sits beside it. It consolidates, in supersession
+order, `docs/design/dreaming-nightly-pipeline.md` → `docs/design/dreaming-nightly-pipeline.md` →
+`docs/design/dreaming-nightly-pipeline.md` → this doc, plus the architecture half of
+`docs/design/dreaming-nightly-pipeline.md`, `docs/design/dreaming-nightly-pipeline.md`, and the still-open task list
+from `docs/design/dreaming-nightly-pipeline.md`. Where those docs disagree, this doc wins.
+
+- User-facing behaviour (CLI, tools, vault layout as the user sees it) lives
+  in `docs/reference/memory.md`. Status and changelog prose belong in
+  `PLAN.md`; this doc carries design plus a short dated History.
+- Every code claim below was re-read against `src/memory/` on 2026-08-23.
+  Where the code and the settled design differ, both are stated
+  ("as built" vs "target") — the code is the truth about what runs; the
+  design is the truth about where it goes.
+
+The system is the second iteration of Lucien's memory pipeline
+(`~/Code/lucien`, cloud models), rebuilt to run entirely on local models. Same
+methodology at the core — do whatever Wikipedia does — with two consequences
+of the local constraint:
+
+1. **Small tasks.** Every AI step is the smallest single-purpose unit a small
+   local model does well — one chunk, one section, ideally a binary judgment.
+   Never a whole-article blob in or out. And every AI step is pure
+   text-in/text-out: no tool calls, no agent loop. The pipeline hands the
+   model exactly the context it needs and applies the output
+   deterministically.
+2. **Deterministic gates between AI steps.** AI does judgment; code enforces
+   invariants (citations, footnotes, wikilinks, infobox shape, word floors).
+   A weak output is gated to NO-OP, never allowed to corrupt the vault.
+   "LoRAs propose, gates dispose."
+
+## 1. Contract and fixed principles
+
+**The memory contract (ported from Lucien):** chat-time agents READ memory;
+the synthesis pipeline WRITES memory. Articles are durable priors,
+conversations are evidence, synthesis is the deliberate, auditable process
+that turns evidence into updated articles. Normal chat never mutates an
+article; writes happen only through `mlx-bun memory synthesize` (or its
+per-stage subcommands) and the user's own edits in the vault.
+
+Fixed principles (do not re-derive, do not contradict):
 
 - **Conversations are the base.** The pipeline's only input is conversation
-  transcripts (pi sessions daily; historical imports are a one-time onboarding
-  path through the same code).
-- **No notability filter.** *"If I talk about it, it's notable."* Everything
-  surfaces; nothing is dropped or parked as trivia. Any prior "owned/decided/≥3
-  chunks else PARK" gate is dead — the catch-all bucket exists only so that
-  low-volume topics remain searchable until they warrant their own article.
-- **No embeddings, anywhere in the project.** Not for routing, not for
-  clustering, not as "offline instruments." Grouping and routing are done by
-  deterministic name/alias/category lookup plus small local-model judgments.
-- **Articles are real single-topic wiki articles with sections** — one subject
-  per article, a lead, an infobox, `##` sections, See also, References. All
-  subject mentions are `[[wikilinks]]`.
-- **Vault = truth, DB = derived cache.** Markdown in the vault is canonical and
-  git-tracked; the SQLite store is pipeline state + rebuildable index
+  transcripts — pi sessions in steady state, a one-time historical import
+  through the same code for onboarding.
+- **Articles are THINGS (subjects), with sections.** One subject per
+  article: a lead, an infobox, `##` sections, See also, References. Subject
+  mentions are `[[wikilinks]]`. Every article is "the thing AND how the user
+  relates to it"; general world-facts link out.
+- **No notability filter.** "If I talk about it, it's notable." Nothing is
+  dropped as trivia. A thin subject is CAPTURED (kept searchable in
+  `_captured`) until it recurs; stubs are a feature.
+- **Capacity by decomposition.** A small model's effective capacity scales
+  inversely with task size; you don't make the model smarter, you make each
+  task narrower. Decompose to binary where possible and run M×N — affordable
+  because local inference is nearly free. Coarse gates prune fine ones
+  (chunk → article → section).
+- **No embeddings in the pipeline.** Routing, grouping, and reading are
+  deterministic name/alias/category lookup plus small-model judgments. The
+  hot path asserts the `src/embed.ts` tripwire (`getEmbedCounter()`) stays
+  at 0. Embeddings survive only as offline eval instruments (§8) — see the
+  contradiction note there.
+- **Vault = truth, DB = derived cache.** Markdown in the vault is canonical
+  and git-tracked; SQLite is pipeline ledger + rebuildable index
   (`reindex`).
-- **Self-healing through ordinary use.** Corrections arrive as ordinary chunks;
-  chronological processing (`updated_at ASC`) resolves claims toward the user's
-  latest position.
-- **Prompts and Meta pages are SCHEMATIC** (no real names/specs/dates) — the
-  small model parrots concrete example values into unrelated articles.
-- **Runs nightly**, locally, unattended (launchd, `src/memory/schedule.ts`).
-- **Model evals are not part of this design.** The eval harnesses (chunk-eval,
-  judges, F1/Jaccard gates, cloud-judge sampling) are tooling for *choosing and
-  training the models used by* the pipeline. They live beside the system, never
-  inside it, and belong in their own docs.
-
-## The pipeline
+- **Chronology is load-bearing.** Every stage walks
+  `conversations.updated_at ASC, chunks.start ASC`. This is what makes
+  self-healing correct: the user's latest statement dominates.
+- **Self-healing through ordinary use.** Corrections arrive as ordinary
+  chunks; a later-dated contradiction resolves the article toward the
+  user's latest position. The user's evolving positions keep a `## History`
+  trace; an AI factual error is silently overwritten.
+- **Prompts and Meta pages are SCHEMATIC** — no real names, specs, or dates.
+  The small model parrots concrete example values into unrelated articles
+  (observed five times during bring-up). Every prompt carries a no-copy
+  guard; leaky drafts are rejected (`isLeakyDraft`).
+- **Runs nightly, locally, unattended** (launchd, `src/memory/schedule.ts`).
+- **Lucien is a read-only oracle.** `~/Code/lucien/.lucien/lucien.db` is
+  opened read-only; `~/Dreaming` is never written. Never suggest modifying
+  Lucien's code, DB, or vault.
+- **Model evals are beside the system, not inside it.** Judges, gold sets,
+  and signal scripts choose and train the models the pipeline uses; they are
+  never a pipeline stage.
+
+## 2. Data model
+
+### 2.1 Vault (truth)
+
+- Root `~/.mlx-bun/wiki/` (override `MLX_BUN_WIKI`, honoured by `vaultRoot()`
+  — every script and test redirects the whole pipeline by setting it).
+- `articles/*.md` user articles · `Meta/*.md` editable policy pages ·
+  `Talk/` restructure suggestions · `Reference/*.md` read-only symlinks to
+  mlx-bun's own docs (chat tools may read them; synthesis must never write
+  through them).
+- Plain Markdown, git-tracked, usable as an Obsidian vault. Every pipeline
+  write ends in `commitVault`.
+- Meta pages seeded by `setupVault` (`src/memory/vault.ts` `META_PAGES`):
+  Editorial_Guidelines, Bucketing, Article_Conventions, Infobox_Schemas,
+  Category_Definitions, Entities, Summary_Style, Buckets, Chunking,
+  Topics_to_Ignore. Policy lives in the vault, not in code:
+  `loadMetaPolicy(names)` (`src/memory/prompts.ts`) inlines pages into a
+  stage prompt (single-turn, bounded prefill — never "go read the page")
+  and throws if a requested page is missing.
+
+### 2.2 Article grammar (`src/memory/article.ts`)
+
+Fixed skeleton, top to bottom: `# H1` (== filename stem with `_`→space) →
+optional series banner `*Part of a series on [[X]].*` → optional infobox
+(fenced ```` ```info ```` block of `key: value` lines) → lead (present-tense
+relationship abstract) → `##` sections → `## See also` → optional `## Notes`
+→ `## References` last. Templates the parser and NORMALIZE recognise:
+`{{stub}}`, `*Main article: [[Child]]*`, `*Redirects to [[X]]*`.
+
+- **Infobox** — `snake_case` keys; entity-valued fields are `[[wikilinks]]`
+  (they feed the link graph); `aliases:` is a comma list feeding the entity
+  index; `kind:` is the closed set `ENTITY_KINDS = thing | person | domain |
+  project | standard`; `type:` and every other field label are EMERGENT
+  (governed by `Infobox_Schemas.md`, never a code enum). Fields are either
+  world-facts (citable specs) or relationship-facts (owned / used_for /
+  opinion / status … — the personal layer). The infobox is CONTENT the model
+  reads, never a query/filter engine; there is no field query anywhere.
+- **Categories** — emergent and explicit: an article declares
+  `[[Category:Name]]`; the deriver accumulates names into `categories`. A
+  fresh vault has zero.
+- **Citations** — footnote defs in the Lucien standard
+  `` [^N]: `conv:HASH` (YYYY-MM-DD, source) — desc ``, HASH = first 8 hex of
+  the conversation UUID. A hash identifies the CONVERSATION only; chunk →
+  section provenance is recovered from the ledger, never by parsing
+  footnotes. The model only PLACES markers; code writes every definition
+  line (`buildFootnoteMap`, `footnoteDefLine` in `synthesize.ts`).
+- One shared fence scanner (`isFenceLine`/`fenceLabel`) so `article.ts`,
+  `vault.ts`, `normalize.ts`, `crosslink.ts` agree on "inside a code block";
+  `repairFences` closes a dangling fence deterministically.
+
+### 2.3 Store (`src/memory/db.ts`, `~/.cache/mlx-bun/memory.sqlite`)
+
+bun:sqlite, WAL, `PRAGMA foreign_keys = ON`, `ON DELETE CASCADE`. A DB
+carrying the pre-rewrite schema (`chunks.text` column or a
+`synthesized_bucket_chunks` table) is dropped and recreated on open.
+
+- **Derived cache** (rebuilt from markdown by `reindex`, never the reverse):
+  `entities` · `entity_aliases` · `categories` · `article_categories` ·
+  `infobox_facts` · `links` (`via ∈ prose|infobox|seealso|series|main`).
+- **Synthesis ledger** (authoritative pipeline progress, not derivable from
+  the vault): `conversations` · `messages` · `chunks` · `chunk_entities` ·
+  `chunk_sections` · `synthesized_chunk_sections` · `reconciled_articles` ·
+  `watermarks` · `buckets` / `chunk_buckets` · `subject_redirects`.
+- **Pointers, not copies.** `messages` holds text once, keyed
+  `(conv, position)`. A chunk is a `[start,end]` range plus a label, id
+  `${conv}:${start}-${end}` (deterministic, idempotent re-chunking);
+  `chunkText(id)` reassembles on demand. Assignments are join tables
+  (many-to-many at every layer). The article markdown is the only
+  materialised synthesis; its `conv:` citations point back at sources.
+- `reindex` (`src/memory/reindex.ts`) drops and rebuilds exactly the six
+  derived tables from the vault with foreign keys off (so pipeline
+  `chunk_entities` rows survive), in sorted-stem order, deterministically.
+  It also PROJECTS `subject_redirects` into `entity_aliases` so a subject
+  that folded into or was captured near another article still resolves.
+
+Turso/sqlite-vec were considered as a vector-capable engine and are moot
+under the no-embeddings rule; the driver stays bun:sqlite.
+
+## 3. The pipeline as built (verified 2026-08-23)
 
 ```
-conversations                                    (ingest, watermark/cursor-gated)
-     │
-     ▼  SEGMENT ──────────────── AI: find the topics in each conversation
-  chunks: single-topic segments with labels (pointer ranges, no text copies)
-     │
-     ▼  BUCKET ───────────────── AI + deterministic: group similar topics
-  buckets: one per STORY/TOPIC; everything discussed about that topic since the
-  last run accumulates here (a bucket usually has NO article of its own)
-     │
-     ▼  RESOLVE ──────────────── which articles does each story affect?
-  bucket delta → the subject articles it touches (update · create · sections)
-     │
-     ▼  SYNTHESIZE ───────────── AI: per-section create/patch, bounded in and out
-  create: outline → per-section draft · update: patch one section, rest byte-identical
-     │
-     ▼  ── deterministic gates ── citation survival · footnote bijection ·
-     │     word floor · wikilink resolution · weak output → NO-OP
-     │
-     ▼  RECONCILE ────────────── infobox = ground truth; lead/body/infobox agree
-     ▼  LINK ─────────────────── cross-link: mentions → [[links]], co-occurrence → See also
-     ▼  NORMALIZE ────────────── deterministic: footnote renumber, wikilink canonicalize,
-     │                           structure guards (References last, stub marking)
-     ▼  EDITORIAL (wikify) ───── AI: remove redundancy, organize, make it read well —
-     │                           per-section improve + structural pass over the TOC
-     ▼
-  commit to vault (git) + run changelog
+pi sessions / Lucien import   ingest.ts  (watermark per source)          [NOT wired into the nightly run — §9]
+        │
+        ▼  SEGMENT ─────────── AI (memory-chunk adapter): one conversation → single-topic chunks + labels
+        ▼  ENTITY-EXTRACT ──── AI (base+Entities.md): one chunk → the THINGS it is about (name list)
+        ▼  ROUTE ────────────── deterministic resolve → binary disambiguate → create | routed | capture
+        ▼  SYNTHESIZE
+        │    CREATE ─────────── AI: sub-cluster → OUTLINE → per-section DRAFT → INFOBOX → assemble
+        │    SECTION-ROUTE ──── AI: M×N binary "does this chunk update THIS section?" (+ name a new one)
+        │    PATCH ──────────── AI: one (chunk, section) → replacement section, rest byte-identical
+        │    RECONCILE ──────── AI+code: infobox = ground truth; lead/body/infobox agree; History
+        │    ── every write: NORMALIZE (code) → conservative gate (code) → weak ⇒ NO-OP ──
+        ▼  LINK ─────────────── code: first-mention [[links]] + co-occurrence See also (idempotent)
+        ▼  commit (git)
+        ▼  WIKIFY sweep ─────── AI (editor): per-section tighten + infobox refresh, every article
 ```
 
-Every stage is an independent, resumable, DB-state-keyed worker
-(`src/memory/stages.ts`), runnable separately or as the full DAG
-(`mlx-bun memory synthesize`).
-
-### SEGMENT — conversations → topics
-
-The e4b chunk adapter (`e4b-chunk-300`, symlinked `memory-chunk`) segments each
-conversation into single-topic chunks with short labels. One chunk = one
-coherent topic; a multi-stage process is ONE topic; long single-topic
-conversations stay whole. Chunks are pointer ranges over `messages` — text is
-materialized exactly once. Deterministic anchor repair validates every boundary.
-
-### BUCKET — group similar topics (first-class stage)
-
-**The Wikipedia-editor analogy (this is the model — Josh's framing).** A news
-story gets reported seven times in a day — Fox says one thing, MSNBC another,
-CBS a third, each report carrying a bit of different information. A Wikipedia
-editor who heard ALL of the reports incorporates everything they learned into
-every article the story affects. In our system: the news shows are
-**conversations**, the stories are **topics (chunks)**, and the **bucket is the
-editor's brain** — the place where everything reported TODAY about THAT TOPIC
-accumulates before any article is touched.
-
-**A bucket is keyed by the STORY/TOPIC, not by a subject.** The worked example:
-the topic is *"2026 Lincoln Memorial Reflecting Pool controversy"* (the algae
-issue in the pool that Trump renovated). Every report about it that day goes in
-that one bucket. There is **no article for that topic** — and there may never
-be — but the story affects THREE articles: *Lincoln Memorial Reflecting Pool*,
-*Second presidency of Donald Trump*, and *John J. Cafaro*. Each article
-receives the facet of the delta relevant to it (renovation/algae facts vs.
-abuse-of-power framing vs. the contract angle). This is why bucket ≠ article is
-**structural**: articles are THINGS (subjects); buckets are STORIES (what got
-discussed); the story→things relationship is inherently many-to-many.
-
-Assignment mechanics: group chunks by topic — normalize/compare chunk labels
-deterministically first, with a small-model binary judgment ("are these two
-chunks about the same story? yes/no") for near-misses, and a new-bucket
-proposal for genuinely new topics. The taxonomy is emergent: buckets come into
-existence because the user talked about something, never from a predefined
-list. Buckets are naturally short-lived relative to articles — a story
-accumulates while it's live and goes quiet when the user stops discussing it;
-the articles it fed persist.
-
-`_captured` is the reserved catch-all bucket: chunks whose subject doesn't yet
-justify assignment stay there, searchable, and are revisited on later runs —
-**nothing is dropped**.
-
-### RESOLVE — bucket deltas → articles/sections
-
-The unit of work from here on is the **bucket delta**: all of a bucket's new
-(un-synthesized) chunks since the last run, taken TOGETHER. For each non-empty
-delta, RESOLVE answers the editor's question: **"which articles does this story
-affect?"** (the Reflecting Pool story → the pool article, the presidency
-article, the contractor article). For each affected subject:
-
-- **The article exists** → UPDATE. Section-route the delta's *facet for that
-  article* via binary per-section judgments over the TOC ("does this new
-  information update THIS section? yes/no"); all-no → propose a new section.
-- **The article doesn't exist yet** → CREATE the subject article (a THING —
-  never an article named after the story itself). Thin is fine — 2–3 chunks
-  make a legitimate stub (stubs are a feature, not a failure).
-
-The per-section ledger (`synthesized_chunk_sections`) records which chunks each
-fold incorporated, making every fold idempotent and every claim traceable to
-its source chunks.
-
-### SYNTHESIZE — one edit per (bucket, article), never chunk-at-a-time
-
-**The cardinal rule:** synthesis is the editor sitting down ONCE per article
-with everything the bucket learned. Never patch a section from one chunk and
-then re-patch it from the next chunk five minutes later — sequential
-single-chunk edits overwrite each other, churn citations, and produce
-patchwork prose. The whole point of the bucket layer is that the delta arrives
-at the article **already aggregated**.
-
-- **CREATE**: group the bucket's chunks into proto-sections (by label/subject
-  affinity — small-model judgment, no embeddings) → outline (title, lead,
-  section names) → draft each section from only its chunks. The model never
-  holds a whole article in its head.
-- **UPDATE (patch)**: hand the model one section + the delta's facet relevant
-  to that article and section (all of it, with per-conversation `[^N]` citation
-  material); it rewrites that one section once, weaving everything in. Facet
-  extraction is implicit: the same story becomes algae/renovation facts in the
-  Reflecting Pool article and framing in the presidency article, because the
-  model writes what's relevant to the section it's editing.
-  Contradiction = correction: resolve toward the latest statement, including
-  contradictions *within* the delta (latest chunk wins — the editor heard all
-  seven reports and writes the reconciled version, not seven amendments).
-  User's evolving positions keep a `## History` trace; AI factual errors are
-  silently overwritten.
-- **Edit mechanics (SETTLED 2026-07-01, Josh-confirmed — and how Lucien always
-  worked): the model never edits the article; it is handed the current section
-  + the new information and outputs a complete REPLACEMENT SECTION.** Not an
-  in-place file edit (the
-  model would have to hold the whole article — the exact blob failure this
-  design avoids) and not a diff (small local models produce brittle patch
-  formats, and a diff is hard to gate). A full replacement section is bounded
-  in and out, trivially appliable (`extractSection` swap — the rest of the file
-  byte-identical by construction), and directly gateable: the gate compares old
-  section vs. new section for word floor and citation survival. Deterministic
-  code performs the actual write. CREATE works the same way — the model emits
-  one section at a time; code assembles the article.
-- **Multiple buckets converging on one article** (the algae-controversy bucket
-  AND the olympian-arrest bucket both affect *Reflecting Pool*): each bucket's
-  facet folds as its own section-level edit within the run — different stories
-  usually land in different sections. If two buckets touch the SAME section,
-  fold chronologically (latest wins, per self-healing) and let the nightly
-  EDITORIAL pass smooth the seam; the editorial pass runs after all folds, so
-  the article the user reads next morning is one coherent account.
-- **Conservative write gate** on every output: ≥70% word floor on the touched
-  section, every pre-existing `conv:` citation survives, H1/References survive,
-  footnote bijection. Weak output → NO-OP; the delta retries next run.
-
-### RECONCILE · LINK · NORMALIZE — nightly deterministic/structural stages
-
-These run **every night**, not periodically:
-
-- **RECONCILE** (`reconcile.ts`): the infobox is the article's ground truth;
-  lead, body sections, and infobox must agree; date-aware (latest wins).
-  Infobox extraction/refresh is part of this — every article carries a current,
-  parseable infobox whose entity-valued fields are `[[wikilinks]]`.
-- **LINK** (`crosslink.ts`): cross-linking is a dedicated stage, NOT
-  synthesis's job (synthesis sees one article and cannot know the others).
-  Mention-based prose links + co-occurrence See-also. Idempotent. Ensures **all
-  subjects are wikilinked** across the vault.
-- **NORMALIZE** (`normalize.ts`): pure code — footnote renumber/merge, wikilink
-  canonicalization, orphan repair, structure guards (`## References` last,
-  `{{stub}}`/`{{Main}}` recognition, title == H1 == stem).
-
-### EDITORIAL (wikify) — make the article good
-
-The final AI pass, and the reason articles stay readable instead of accreting
-patches: per-section "tighten this — remove redundancy, keep every citation and
-every specific detail" plus a structural pass over the TOC (reorder sections,
-refresh the lead, split a section that outgrew the article into its own article
-with a `{{Main}}` summary, merge duplicate stubs). Behind the same conservative
-gate; a weak edit is rejected and the original kept.
-
-## Nightly run contract
-
-- Trigger: launchd (`com.mlx-bun.memory`), logs under `~/.mlx-bun/logs/`.
-- Scope: only conversations past the per-source watermark — a normal night is
-  minutes of work, not hours.
-- Inference: one resident base model (Gemma-4-e4b), per-stage adapter mounted
-  iff `~/.cache/mlx-bun/adapters/memory-<stage>` exists, else base+policy.
-  Route generation through the persistent server (load-once) when available;
-  no-server fallback works but pays model load per invocation.
-- Every article change is a git commit; the run ends with a changelog summary.
-- Failure isolation: stages are resumable; a failed bucket/article never blocks
-  the rest of the run.
-
-## Decisions overridden from the master plan (explicit, so no agent re-derives them)
-
-| Master-plan statement | Superseded by |
+Orchestration: `runSynthesis` (`src/memory/pipeline.ts`) drives
+`runPipeline` = `runSegmentStage → runExtractStage → runRouteStage →
+runSynthesizeStage → runLinkStage`, then `wikifyVault` over every article.
+Each stage worker (`src/memory/stages.ts`) pulls its eligible work from the
+DB by state, processes a bounded batch, persists, and is individually
+runnable: `mlx-bun memory segment | extract | route | synthesize-stage |
+link`; `mlx-bun memory synthesize` is the full DAG; `--dry-run` plans without
+model calls. A failed article never blocks the rest of a run.
+
+### 3.1 INGEST (`ingest.ts`)
+
+Two sources, watermark-gated per `source`: pi sessions via
+`SessionManager.listAll()` (`ingestSessions`) and the one-time Lucien
+bootstrap (`ingestLucien`, read-only; maps Lucien's UUID-anchored chunks to
+positions). Idempotent — a re-touched conversation is rewritten in place.
+**As built, neither entry point is called by `runSynthesis` or any
+`scripts/memory/` driver** (only `tests/memory-ingest.test.ts` exercises
+`ingestLucien`); `runPipeline` emits "ingest: corpus already in the store".
+The nightly fold-in of new pi sessions is therefore not yet end-to-end — see
+§9.
+
+### 3.2 SEGMENT (`chunk.ts`, `chunk-validation.ts`)
+
+The `memory-chunk` adapter (`e4b-chunk-300`, symlinked to
+`~/.cache/mlx-bun/adapters/memory-chunk`) segments each conversation whose
+`chunked_at` is NULL or older than `updated_at`. Prompt = `CHUNK_SYSTEM`
+(the exact trained system turn, byte-for-byte) + `CHUNK_PROMPT` with
+`Chunking.md` + `Topics_to_Ignore.md` inlined. Output is JSON of message-UUID
+anchors + a 4–10-word label; anchors are validated and repaired
+(`validateChunks`) before pointer rows are written; a conversation whose
+output fails validation is left unmarked and retried next run. Re-run is a
+no-op by watermark. Chunk purity is the foundation of everything downstream:
+a chunk that fuses two topics pollutes every article it routes to.
+
+### 3.3 ENTITY-EXTRACT (`entity.ts`, `resolve.ts`)
+
+One chunk per call, base model + `Entities.md`: "what things is this chunk
+about? canonical wiki-title names, one per line, specific things AND the
+broad domain, NONE if nothing nameable." Plain newline list parsed by
+`parseLines` — no multi-item JSON, no batch indices (the batch-index drift
+that sank Lucien's batched assign). Persists `chunk_entities(chunk_id,
+entity_name, surface_form)` and a candidate `entities` row (article_stem
+NULL). Canonicalisation is deterministic and conservative
+(`EntityResolver`): `canonicalize` (casefold, whitespace, leading
+articles/possessives) → `squeeze` (strip all non-alphanumerics) → alias index
+(store + `goldens/dreaming-entities-gold.json` seeds) → token-subset fuzzy
+match requiring a distinctive token (the over-merge guard). When in doubt it
+mints a new canonical rather than fusing two things.
+
+### 3.4 ROUTE (`route.ts`, `redirect.ts`)
+
+Per surface: (1) RESOLVE against the known-entity index — a hit folds into
+that article with no model call; (2) DISAMBIGUATE a miss with a trigram
+shortlist of existing entities and one bounded binary per candidate ("is X
+the same thing as Y? yes/no", `parseBinary`, existing wins); (3) CREATE /
+CAPTURE for a surface matching nothing: it earns its own article when it
+RECURS (`RECURRENCE_THRESHOLD = 2` routed chunks) or, with the optional
+`useSubjectGate`, when a chunk engages it as a genuine subject; otherwise it
+is CAPTURED — its chunks go to the reserved `_captured` bucket, `notable = 0`,
+still searchable, never dropped, promoted later if it recurs. There is no
+ownership/usefulness test. `runRouteStage` also registers
+`subject_redirects` so every discussed subject resolves to a home (a folded
+subject → its article; a captured subject → the article it most co-occurs
+with). A chunk fans out to every article it names — this is where facet
+extraction falls out: the same chunk lands on several articles, each keeping
+only what is relevant to it at patch time.
+
+### 3.5 SYNTHESIZE (`stages.ts` `runSynthesizeStage`, `synthesize.ts`, `cluster.ts`)
+
+Chronological throughout; commits once at the end.
+
+- **CREATE** (`synthesizeCreate`) for each `notable = 1` entity with no
+  article file, oldest entity first, capped by `DEFAULT_ARTICLE_CAP = 20`
+  per pass: `subClusterChunks` groups the entity's chunks into proto-sections
+  by label/text token overlap (deterministic, no embeddings) → OUTLINE (2–6
+  headings) → per-section DRAFT from only that section's chunks → LEAD →
+  seed INFOBOX (`buildInfoboxFields`, kind + declared aliases) → assemble →
+  NORMALIZE → gate → write + `synthesized_chunk_sections` ledger. The model
+  never holds a whole article. Leaky drafts retry once, then NO-OP.
+- **SECTION-ROUTE** (`routeSections`) for each chunk routed to an existing
+  article and not yet in the ledger: one binary per routable section
+  (structural tails References/See also/Notes excluded; the lead is a
+  first-class target, `LEAD_ANCHOR`), prompt = title + heading + section's
+  first ~2 sentences + chunk label + gist. All-no + substantive → "name a new
+  section (2–5 words) or NONE"; a named section is honoured
+  (`synthesizeNewSection`), never a silent drop. Hierarchical pruning is
+  structural: only articles ROUTE matched are ever gridded.
+- **PATCH** (`synthesizePatch`, `patchSection`) — the model is handed the
+  current section + the chunk (with its pre-assigned `[^N]`) and outputs a
+  complete REPLACEMENT SECTION; code swaps exactly that section
+  (`replaceSection`), appends the definition line, and leaves the rest of
+  the file byte-identical. Contradiction = correction: rewrite toward the
+  latest statement, classified by `classifyContradiction` (`history.ts`) as
+  EVOLUTION (keep a `## History` entry citing both sources; default when
+  ambiguous) or ERROR (silent overwrite, the wrong value's citation may be
+  pruned via `allowDroppedHashes`). Idempotent via the ledger PK.
+- **RECONCILE** (`reconcileArticle`) runs after the patch loop over every
+  article touched this run: the infobox is ground truth; relationship
+  fields (favourite/opinion/status/owned … — never `PHYSICAL_SPEC_KEYS`) and
+  sibling body sections still asserting a superseded value are refreshed
+  toward the latest-dated position, provenance-preserving (every `[^k]`
+  survives, phrased as change over time). Consistent article ⇒ NO-OP;
+  unresolved present-tense assertions are logged, not forced.
+
+### 3.6 The conservative write gate (`gate.ts`) and NORMALIZE (`normalize.ts`)
+
+`gateEdit(before, after)` is a faithful port of Lucien's editorial gate and
+guards every write (create, patch, new section, reconcile, link, wikify):
+every pre-existing `conv:` citation survives (unless deliberately pruned),
+footnote bijection, edited word count ≥ `floor` (default 0.7) of the
+original, H1 and `## References` survive, at most one `## See also`. It can
+only reject, never rewrite; it catches citation/structure/bulk-prose loss,
+not nuance loss (that is the judge's job, §8).
+
+`normalizeArticle` runs inside every write before the gate, pure and
+idempotent: footnote renumber/merge (`normalizeFootnotes`), wikilink
+canonicalisation to on-disk stems (`normalizeWikilinks`), infobox key sort,
+section order (`## See also` before `## References` last), `{{stub}}` guard
+for thin articles (`articleThinness`), `title = H1 = stem`, fence repair.
+Non-fatal per step. `normalizeVault` exists for a whole-vault pass but is not
+a nightly stage; normalisation is constitutive of each write instead.
+
+### 3.7 LINK (`crosslink.ts`)
+
+Cross-linking is its own deterministic stage because synthesis is bounded to
+one article and cannot know the others. (1) MENTION edges: wrap the first
+whole-word mention of another article's title/alias as
+`[[Stem|surface]]` — never inside an existing link, code span, marker,
+infobox, heading, or References. (2) CO-OCCURRENCE edges from shared folded
+chunks (`synthesized_chunk_sections` ⋈ `chunk_entities`). (3) Rebuild
+`## See also` idempotently (mentions first, then top co-occurring articles
+that exist). Every edit passes `normalizeWikilinks` + `gateEdit`; a fully
+linked article re-runs to a byte-identical no-op. Measured on a 587-article
+snapshot of the import (2026-06-28, M1 Max 32 GB): See-also coverage 18% →
+96%, resolved edges 309 → 7026.
+
+### 3.8 WIKIFY — the editorial pass (`wikify.ts`)
+
+The periodic sweep, distinct from NORMALIZE: the model supplies judgment, the
+gate disposes of weak judgment. As built, `runSynthesis` runs `wikifyVault`
+over EVERY article every night after the write branch; each article commits
+its own change or NO-OPs, and a wikify defect cannot lose the committed
+create/patch work.
+
+- Per-section IMPROVE (`improveSections`, `editor` stage): "tighten this
+  section — remove redundancy, keep every citation and every specific
+  detail." Behind the gate.
+- Infobox EXTRACT/REFRESH (`refreshInfobox`): creates a missing infobox,
+  refreshes present ones without losing a grounded fact, entity-valued
+  fields as wikilinks, declared `aliases:` feed the entity index on reindex.
+- Summary-style SPLIT (`detectSplitCandidate`, `applySplit`): when one body
+  section dominates the article it becomes a child article with the full
+  cited prose; the parent keeps a summary + `*Main article: [[Child]]*`
+  (a `via=main` edge). Skips the word floor (a sanctioned bulk move); gated
+  structurally instead. Series banner maintenance (`setSeriesBanner`).
+- MERGE signals (`detectMergeCandidates`) and structure suggestions are
+  written to `Talk/<stem>.md` (`writeTalkPage`) — the pass never
+  restructures across articles on its own. `sectionCohesion` (an injected
+  `EmbedFn` silhouette) exists here as an offline triage instrument only;
+  `wikify.ts` never imports `src/embed.ts`.
+
+### 3.9 Nightly run contract
+
+- Trigger: launchd `com.mlx-bun.memory` (`schedule.ts`; `mlx-bun memory
+  schedule --at HH:MM`, default 03:00), running `mlx-bun memory synthesize`;
+  logs `~/.mlx-bun/logs/memory-synthesis.{out,err}.log`.
+- Scope: only work past each stage's state (watermark / missing downstream
+  row / ledger). A normal night is minutes, not hours; throughput only
+  matters for the one-time import.
+- Inference: one resident Gemma-4-e4b (§4); a stage's adapter is mounted iff
+  `~/.cache/mlx-bun/adapters/memory-<stage>` exists, else base + policy.
+- Every article change is a git commit; the run ends with a summary line.
+- Never started by an agent: the full-corpus import and any GPU training are
+  user actions.
+
+## 4. The inference seam (`src/memory/model.ts`)
+
+One shared runtime: one `RuntimeModel` (Gemma-4-e4b OptiQ-4bit, snapshot
+resolved by globbing for `config.json`), the `memory-chunk` adapter mounted
+once and ACTIVATED per call (`loraState.active`), one in-process
+`BatchScheduler`. Two seams: `callLocal(stage, {system?, user})` (bit-exact
+greedy via the eval runner's raw-forward decode) and `callLocalBatch(stage,
+inputs[])` (order-preserving; batched only when `MLX_BUN_MEMORY_BATCH > 1`
+and N > 1, else the same serial loop). Never auto-downloads; throws with the
+`hf download` hint until the snapshot lands.
+
+**Templating.** Each stage renders a `[system?, user]` message array through
+the model's chat template exactly as the server and the trainer's
+prompt-region render do (render with generation prompt → encode → dedupe a
+double BOS; `memoryPromptIds`). This fixed the bring-up defect where the
+chunk adapter — SFT'd with a system turn that Gemma-4 renders as a distinct
+`<|turn>system…<turn|>` block — was decoded with the system segment dropped.
+`tests/memory-templating.test.ts` asserts `memoryPromptIds("chunk", …)` is
+id-identical to a real training row's prompt region. Base stages get concise
+default systems (`DEFAULT_STAGE_SYSTEM`); only `chunk` must match training
+byte-for-byte.
+
+**Token budget.** `MAX_OUTPUT_TOKENS = 64_000` is the single backstop; stages
+pass it rather than per-call caps, because a finished answer stops at EOS and
+a cap can only truncate an unfinished one. (The per-stage 4/16/128/2048 caps
+in the original inference-path design are superseded; `extractEntityNamesRaw`
+still defaults to 128 when called directly.)
+
+**Batching — implemented, off by default (decision: Josh, 2026-07-01).**
+Measured 2026-06-28 on Gemma-4-e4b OptiQ-4bit, held-out
+`chunk-data-le4000/valid.jsonl` + the live 2096-conversation store, M-series
+GPU: 8 extract-style prompts (~400-token prefills) batched 9345 ms vs serial
+5001 ms (≈1.9× slower, 8/8 exact); 8 short replies batched 2085 ms vs 1067 ms
+(≈2× slower, 7/8 match — one documented near-tie divergence); only 16 short
+uniform one-liners won (≈1.2× faster). The warm uniform chunk workload on the
+M1 Max 32 GB measured 0.607 vs 0.357 s/chunk (~1.7× slower). Root cause: the
+scheduler pads heterogeneous prefills to the batch max, losing to one
+efficient single-sequence prefill per row. The real efficiency lever for
+daily use is load-once (the shared runtime above; a persistent-server route
+remains open, §9), not batching.
+
+**Adapter constraint.** `loraState.active` is a single global field read per
+forward, so a batch is necessarily single-adapter; `callLocalBatch` is
+per-stage and stage→adapter is fixed. The seam must never be shared with a
+live server process.
+
+## 5. Target design: the BUCKET layer (settled with Josh 2026-07-01, not built)
+
+**The Wikipedia-editor analogy.** A news story is reported seven times in a
+day, each report carrying a little different information; an editor who
+heard ALL of them incorporates everything into every article the story
+affects. Conversations are the news shows, topics (chunks) are the stories,
+and the **bucket is the editor's brain** — where everything reported TODAY
+about THAT STORY accumulates before any article is touched.
+
+**A bucket is keyed by the STORY, not by a subject.** Worked example: the
+story "2026 Lincoln Memorial Reflecting Pool controversy" has no article of
+its own and may never get one, but it affects three: *Lincoln Memorial
+Reflecting Pool*, *Second presidency of Donald Trump*, *John J. Cafaro*. Each
+receives the facet relevant to it. Bucket ≠ article is structural: articles
+are THINGS, buckets are STORIES, story→things is many-to-many. Buckets are
+emergent (never a predefined list) and short-lived relative to articles.
+
+Target stage order:
+
+```
+SEGMENT → BUCKET → RESOLVE → SYNTHESIZE (delta-at-once) → RECONCILE → LINK → NORMALIZE → EDITORIAL
+```
+
+- **BUCKET** — group chunks by story: normalise/compare labels
+  deterministically first, a binary "same story? yes/no" for near-misses, a
+  new-bucket proposal for genuinely new stories. `_captured` remains the
+  catch-all so nothing is dropped.
+- **RESOLVE** — per non-empty bucket delta (all new chunks since the last
+  run, TOGETHER): "which articles does this story affect?" Existing article →
+  UPDATE (section-route the delta's facet for that article); missing →
+  CREATE the subject article (a THING, never an article named after the
+  story). The as-built entity-keyed ROUTE becomes RESOLVE.
+- **SYNTHESIZE, one edit per (bucket, article)** — the cardinal rule. Never
+  patch a section from one chunk and re-patch it from the next five minutes
+  later: sequential single-chunk edits overwrite each other, churn
+  citations, and produce patchwork prose. The delta arrives aggregated; the
+  model rewrites one section once, weaving everything in, resolving
+  contradictions within the delta toward the latest chunk. Edit mechanics
+  stay as built (replacement section, code performs the write, same gate).
+  Two buckets touching the SAME section of one article fold
+  chronologically; the editorial pass smooths the seam.
+
+As-built vs target, so nobody re-derives it:
+
+| Target (2026-07-01) | As built (2026-08-23) |
+| --- | --- |
+| Story-keyed BUCKET stage between SEGMENT and article fan-out | No bucket stage; `buckets`/`chunk_buckets` hold only `_captured` |
+| RESOLVE: bucket delta → affected articles | ROUTE: chunk-entity surfaces → articles (`route.ts`) |
+| One edit per (bucket, article) per run | One PATCH per (chunk, section), chronological |
+| RECONCILE every night over every article | RECONCILE over articles touched by this run's patches |
+| EDITORIAL over touched articles (cadence open) | `wikifyVault` over every article every run |
+
+## 6. The read path (design; user-facing details in `docs/reference/memory.md`)
+
+The read path is deliberately trivial — all the acceptance pressure is on
+the write side keeping each article a good, current account. FIND → READ →
+ANSWER in small reads, never a context dump: find the one article (name /
+alias / category / a `[[link]]` hop), open its TOC, read the ONE relevant
+section, answer from the user's recorded position there. Silent-colleague
+contract (ported from Lucien): look up before answering; each article is the
+consolidated position, don't reconstruct from raw history; speak as a
+continuation, never "per the wiki…".
+
+- Tools (`src/memory/tools.ts`, registered by the shared pi surface for
+  terminal and web, read-only, auto-allowed): `memory_resolve`,
+  `memory_category`, `memory_read` (TOC + lead by default; full text only on
+  request), `memory_section` (the default read), `memory_links`,
+  `memory_infobox` (readable content, not a query), `memory_list`,
+  `memory_status`, `memory_search` (last-resort substring fallback). There is
+  deliberately no infobox-field query tool. mlx-bun's own docs are a
+  separate `reference_*` surface, never mixed into personal memory.
+- Navigation index (`src/memory/query.ts`): mtime-incremental in-memory maps
+  (alias → stem, category/series/kind/type → stems, infobox, lead, link
+  graph) mirroring the derived tables; zero embedding calls by contract.
+- Exposure: tools and the bundled memory skill appear only when the vault
+  exists; `memoryIndexHint()` adds a one-line presence hint, never a title
+  dump. Memory is for personal continuity, not weather, public facts, or
+  ordinary coding tasks.
+- Acceptance budget (the bar Lucien's traces failed at 33–150 KB per query,
+  one 671-line blob read up to 42× in a query): mean context < 5 KB (8 KB
+  hard ceiling), no single read > 2 KB, `getEmbedCounter() == 0`, local
+  only, silent. Harness: `scripts/memory/eval-northstar.ts` over the frozen
+  `scripts/memory/northstar-queries.json` (Q1–Q6; Q6 is the negative
+  control), graded by `scripts/memory/judge-answer.ts`; traversal proof
+  `scripts/memory/north-star-traversal.ts`.
+
+## 7. Adapters and training
+
+One base model for every stage. Per-stage adapters are product assets (the
+creator trains them, publishes to Hugging Face, `memory init` may offer to
+download); they never get write authority — they produce candidates, the
+gates decide.
+
+| stage string | task | adapter |
+| --- | --- | --- |
+| `chunk` | one conversation → single-topic chunks + labels | **`memory-chunk` = `e4b-chunk-300` (exists, the only trained adapter)** |
+| `entity` | one chunk → the things it is about | base + `Entities.md` (`memory-entity` only on a measured recall miss) |
+| `route` | binary "same thing?" / "engages as subject?" | base |
+| `section` | binary "does this chunk update THIS section?" / name a new one | base (`memory-section` only if the binary misfires) |
+| `synthesis` | draft one section / integrate one chunk into one section / reconcile | base (`memory-synthesis` is the highest-value adapter to train) |
+| `editor` | tighten one section / refresh an infobox / summarise a split | base (`memory-editor` if base misfires) |
+
+Staged LoRA decision, per stage: measure the base + inlined policy on the
+stage's own judge first; train only on a measured miss; pick checkpoints by
+the task judge, never validation loss. Recipe: `scripts/examples/chunk-finetune.ts`
+(sft, rank 16, lr 1e-5, seq 8192, `ops.sdpa`; e4b needs the segmented-backward
+path — `SEG=2` as `e4b-chunk-300` was trained; see
+`docs/design/orpo-training.md`). Training data uses the
+trainer's `messages` row shape with loss masked to the assistant turn.
+Section-granular synthesis gold can be mined from Lucien's structured tables
+plus `~/Dreaming` git history (chunk identity from `chunks`/`chunk_buckets`/
+`synthesized_bucket_chunks`, section before/after from integration commits) —
+never from footnote parsing. All training is a user action.
+
+Recorded chunk-adapter numbers (`reports/orpo-chunk-paper/research-journal.md`,
+via `scripts/examples/chunk-eval.ts`): `e4b-chunk-300` purity 92.2 /
+cohesion 87.8 vs base e4b 99.5 / 58.8; the e4b metric is purity × cohesion.
+The chunk eval score is well-formedness, not accuracy — see the memory note
+"chunk-eval 95.10 = format, not accuracy".
+
+## 8. Evaluation methodology (beside the system)
+
+- **Lucien bucket-gold is NOT the write-path oracle.** Its 379-bucket /
+  385-article topic-bin taxonomy (245 `X_and_Y` bins, 0 infoboxes) is
+  exactly what this pipeline replaces; bucket-F1 / Jaccard against it are
+  routing-instrument numbers, never a pipeline-health gate.
+- **Cloud LLM-judge on real output + hand-curated gold.** Quality is judged
+  by a strong cloud model reading END articles (the article the user reads
+  next morning, after every stage has run — not intermediate drafts), against
+  `goldens/dreaming-entities-gold.json` (hand-curated variant groups and
+  notable entities, grounded by reading `~/Dreaming` and live chunks; NOT
+  derived from bucket-gold). The judge is never in the hot path.
+- **Metrics that mean something:** topic aggregation (many conversations →
+  one article), coverage (0 engaged-with things lost even as article count
+  drops), structural invariants (title is a thing, infobox present and
+  resolvable, footnote bijection, every claim cited), gate pass-rate, and
+  the read-path byte/embedding budgets. Do not measure articles per
+  conversation — there is no conv→article link.
+- **Embeddings as instruments only.** They measure cohesion/dispersion well
+  (silhouette ρ≈0.68 vs the LLM judges, per-conversation) and purity poorly
+  (ρ≈0.32) — so purity rides on the chunker, never on cosine geometry. The
+  P4 label-proxy routing measured a pessimistic ~39% recall@12 on chunk
+  labels, which is why ENTITY-EXTRACT reads full chunk text.
+
+Instruments, all under `scripts/memory/` (moved 2026-08-23; Lucien read-only
+throughout):
+
+| script | what it measures |
 | --- | --- |
-| Personal-notability checklist; CREATE gate "≥3 chunks OR owned/decided, else PARK as trivia"; ≤40-article first-run cap (P1-T5, P5-T4, R5, P9-T3) | **No notability filter.** Everything surfaces; `_captured` keeps the tail searchable; thin buckets → stubs. |
-| Buckets demoted to "optional staging, may ship empty" (L4) | **Buckets are first-class**: the per-subject accumulation layer between chunks and articles; synthesis consumes buckets. |
-| Embeddings allowed as offline instruments (P4 recall tripwire experiments, P8-T4 silhouette triage) | **No embeddings anywhere in the project.** The tripwire (`getEmbedCounter()==0`) remains as a guard, permanently. |
-| Routing/section-route acceptance gated on Lucien bucket-F1 / Jaccard (P4-T2, P7-T1) | Model-selection evals live **outside** the system design; pipeline health is judged on real output quality, never Lucien's bin taxonomy. |
-| Cross-linking + infobox work assigned to periodic WIKIFY (P8) | RECONCILE + LINK are **nightly stages** in the DAG; editorial wikify is the readability pass. |
-
-## Work plan (docs-only reconciliation done; implementation next)
-
-Ordered; none of it touches the running full-corpus import
-(`~/.mlx-bun/wiki-full`, cursor file `~/.mlx-bun/full-run-cursor.txt`) or the
-uncommitted `src/model/universal` / trainer work.
-
-1. **Bucket layer made explicit** — a topic-keyed BUCKET stage between SEGMENT
-   and the article fan-out (`buckets` + `chunk_buckets` tables already exist):
-   group chunks by story (label normalization + binary "same story?"
-   judgments), `_captured` catch-all, and **delta-at-once synthesis** — the run
-   collects each bucket's new chunks, then makes ONE edit per (bucket, article).
-   The as-built entity-keyed ROUTE becomes RESOLVE (bucket → affected subject
-   articles); the per-chunk patch loop is replaced, not wrapped. (Settled
-   2026-07-01 — the Wikipedia-editor / Reflecting-Pool framing above; buckets
-   are STORIES, articles are THINGS, many-to-many by nature.)
-2. **Nightly DAG order** — ensure `mlx-bun memory synthesize` runs
-   segment → bucket → resolve → synthesize → reconcile → link → normalize →
-   editorial every run (reconcile/link/infobox currently periodic/partial).
-3. **Editorial stage** — wire the per-section improve + structural pass as the
-   closing stage behind the conservative gate.
-4. **Persistent-server inference routing** — load-once for all memory
-   generation, no-server fallback (the in-progress inference-path rework;
-   finish, don't fork).
-5. **Promote + schedule** — after the import is judged good: point
-   `~/.mlx-bun/wiki` at the bootstrapped content, enable the nightly launchd
-   fold-in from pi sessions.
-6. **Known quality fixes** — schematic-prompt leaks (<1% of articles), run-on
+| `bucket-cohesion-signal.ts` | embedding shortlist recall ceiling chunk→bucket, per bucket-size bin (the bucketing-stage foundation check) |
+| `chunk-embedding-signal.ts` · `chunk-judge-correlate.ts` | boundary-local embedding features vs the two LLM chunk judges |
+| `eval-name-recall.ts` · `dreaming-lex.ts` | lexical (trigram + token-overlap) shortlist recall over names, no model |
+| `eval-route.ts` | no-vector routing proof; `--phase embed` and `--phase route` run as separate processes so the embedder and e4b are never co-resident |
+| `eval-entity-extract.ts` | precision/recall/F1 + canonicalisation rate for ENTITY-EXTRACT |
+| `eval-section-route.ts` | per-pair accuracy + section-set Jaccard on frozen proxy gold; new-section-not-drop |
+| `eval-patch.ts` | only-that-section-changed + self-healing scenarios, judged on output |
+| `eval-wikify-split.ts` · `p8-judge-wikify.ts` · `wikify-smoke.ts` | split/series/merge/Talk mechanics; cloud-judge fixture for improve-not-degrade |
+| `audit-bootstrap.ts` · `eval-coverage.ts` | structural audit of a bootstrap; migration parity against the notable-entity gold |
+| `north-star-traversal.ts` · `eval-northstar.ts` · `judge-answer.ts` · `northstar-queries.json` | read-path proof, acceptance harness, grader |
+| `segment-smoke.ts` · `dreaming-create-smoke.ts` · `dreaming-coldstart-smoke.ts` · `dreaming-stages-independent.ts` | wiring smokes (real adapter; one entity; whole DAG; stage independence, model-free) |
+| `dreaming-selfheal-demo.ts` | the timestamp-ordered self-healing demonstration |
+| `dreaming-full-run.ts` · `dreaming-slice-select.ts` · `dreaming-slice-run.ts` · `dreaming-slice-inventory.ts` · `dreaming-chunk-diag.ts` | the resumable full-corpus import driver (user-launched, writes `~/.mlx-bun/wiki-full`, cursor `~/.mlx-bun/full-run-cursor.txt`) and its slice tools |
+| `mine-gold-entities.ts` | the frozen entity vocabulary (`goldens/entities.json`) from `~/Dreaming` titles/lead bolds/bucket names |
+| `eval-pi-memory-fixation.ts` | the pi surface exposes memory only when enabled and names the negative cases |
+
+**Contradiction with the no-embeddings principle, recorded rather than
+hidden:** `wikify.ts sectionCohesion` (injected `EmbedFn`), the
+`bucket-cohesion-signal` / `chunk-embedding-signal` / `chunk-judge-correlate`
+scripts and `eval-route.ts --phase embed` still use an embedder. None of them
+is on the pipeline or read path (`src/memory/` never imports `src/embed.ts`;
+the tripwire asserts 0). The design rule as settled 2026-07-01 says "no
+embeddings anywhere in the project"; the code keeps them as offline
+instruments. Deleting them is an open decision (§9), not a silent rewrite.
+
+## 9. Open work (verified against `src/memory/` 2026-08-23)
+
+Ordered. None of it touches the user's real vault (`~/.mlx-bun/wiki`) without
+the promotion step.
+
+1. **Wire ingest into the nightly run.** `ingestSessions` has no caller
+   outside tests; `runSynthesis` assumes the corpus is already in the store.
+   Without this the launchd job cannot fold in new pi sessions. Sequence:
+   ingest → the existing DAG; watermark per source already exists.
+2. **Bucket layer** (§5): story-keyed BUCKET stage, RESOLVE, delta-at-once
+   synthesis replacing the per-chunk patch loop (not wrapping it). The
+   `buckets`/`chunk_buckets` tables already exist.
+3. **Nightly DAG order**: RECONCILE over every article (today: touched
+   stems only) and a single NORMALIZE/EDITORIAL closing pass in the settled
+   order.
+4. **Persistent-server inference routing**: route memory generation through
+   the resident server when one is running (load-once for daily use), with
+   the in-process runtime as the no-server fallback. Batching is not the
+   lever (§4).
+5. **Chunk user-turn drift**: `CHUNK_PROMPT` still differs from the trained
+   user turn (an added no-copy line; 2 inlined Meta pages vs 8 in training;
+   abstracted label examples; the training vault-path line). Either
+   reconcile the prompt to training or retrain `e4b-chunk-300` on the live
+   prompt — pick one, don't stay half-aligned. Re-segmenting conversations
+   chunked under the old user-only template is principled but low-urgency
+   (old output was valid in-range).
+6. **Promote + schedule**: after the import (`~/.mlx-bun/wiki-full`) is
+   judged good on a broad, diverse cloud-judged sample — `reindex`,
+   `mlx-bun memory link`, browse in Obsidian — point `~/.mlx-bun/wiki` at
+   it and enable the nightly job. The `Reference/` symlinks into the repo
+   are non-portable; drop them for the real personal vault.
+7. **Known quality fixes**: schematic-prompt leaks (<1% of articles), run-on
    leads, `owned:` framing on abstract concepts (reframe the lead spec per
-   subject kind).
-
-## Open questions (for discussion with Josh)
-
-- ~~Bucket granularity vs article granularity~~ — **SETTLED 2026-07-01**:
-  buckets are a separate layer from articles (many-to-many both ways); the
-  bucket is the editor's brain that aggregates the day's information about a
-  subject, and RESOLVE fans the delta out to every affected article. Bucket is
-  never a 1:1 alias for an article.
-- **When does a `_captured` chunk graduate?** Re-scan `_captured` every night
-  for subjects that have since accumulated enough related chunks, or only when
-  a new bucket's name matches?
-- **Editorial cadence.** Every night over every touched article, or touched
-  articles nightly + a slower full-vault sweep?
+   subject kind), and stale in-code comments that still describe the dead
+   "≥3 chunks or owned/decided" CREATE rule (`synthesize.ts` header;
+   `prompts.ts` M1 stub constants) — the live rule is
+   `RECURRENCE_THRESHOLD = 2` in `route.ts`.
+8. **Per-stage adapters** (§7), each a user-launched training run gated on a
+   measured base miss: `memory-synthesis` first (RECONCILE logs
+   "UNRESOLVED … a memory-synthesis LoRA would close it" when the base model
+   leaves a stale present-tense assertion), then section, entity, editor.
+   Prerequisite: give `chunk-finetune.ts` a `TASK` selector defaulting
+   DATA/ADAPTER per stage and an e4b (not MiniCPM5) default model.
+9. **Embedding residue** (§8): decide whether `sectionCohesion` and the
+   embedding signal scripts stay as instruments or go.
+10. **Jobs-runner integration**: move `memory synthesize` onto the jobs
+    runner with SSE progress once manual synthesis is boring.
+
+## 10. Decisions overridden (explicit, so no agent re-derives them)
+
+| Earlier statement | Superseded by |
+| --- | --- |
+| Personal-notability checklist; CREATE gate "≥3 chunks OR owned/decided, else PARK as trivia"; ≤40-article first-run cap (master plan P1-T5, P5-T4, R5, P9-T3) | **No notability filter.** Everything surfaces; `_captured` keeps the tail searchable; recurrence (2 chunks) or subject engagement earns an article; thin subjects → stubs. |
+| Buckets demoted to "optional staging, may ship empty" (master plan L4, write-pipeline-entity) | **Buckets are first-class** story-keyed accumulators between chunks and articles (§5). |
+| Embeddings as routing narrower / offline instruments (bucketing-stage, memory-synthesis section-routing, master plan P4/P8-T4) | **No embeddings in the pipeline**; the tripwire is permanent. Residual instruments are catalogued in §8 pending deletion. |
+| Routing / section-route acceptance gated on Lucien bucket-F1 / Jaccard (P4-T2, P7-T1) | Model-selection evals live outside the design; pipeline health is judged on real END output by a cloud judge, never Lucien's bin taxonomy. |
+| Cross-linking + infobox work assigned to periodic WIKIFY (P8) | LINK and RECONCILE are nightly stages; wikify is the readability pass. |
+| Whole-article synthesis (Lucien `synthesize-update`) and per-chunk sequential patches (as built) | One replacement section per (bucket, article) per run, code performs the write (§5). |
+| Per-stage `maxTokens` caps 4/16/128/2048 (inference-path design) | One `MAX_OUTPUT_TOKENS` backstop; never cap below it. |
+| Batching as the import throughput lever | Measured slower for the real workload; default serial; load-once is the lever. |
+| Turso / vector-capable store evaluation | Moot without embeddings; bun:sqlite stays. |
+
+## 11. Open questions
+
+- **When does a `_captured` chunk graduate?** Re-scan `_captured` nightly for
+  subjects that have since accumulated related chunks, or only when a new
+  bucket's name matches? (Today: promotion happens when ROUTE recomputes and
+  the subject recurs.)
+- **Editorial cadence.** Every night over every article (as built), or
+  touched articles nightly plus a slower full-vault sweep?
+- **Provisional / temporal facts.** A convention (e.g. "as of YYYY-MM" tied
+  to the citation date) so "work set to begin July" updates cleanly to
+  "began / completed" rather than contradicting.
+- **Domain vs specific granularity.** ENTITY-EXTRACT emits both; the
+  summary-style split decides what becomes its own page. Is that enough?
+- **Bootstrap on a stronger model?** Is first creation good enough locally,
+  or is the one-time import the one place a stronger model earns its cost?
+
+## History
+
+- 2026-06-19 — M0 landed: vault, read tools, CLI, launchd scheduling
+  (`docs/design/dreaming-nightly-pipeline.md`); synthesis a safe no-op stub.
+- 2026-06-26 — M1 design (`docs/design/dreaming-nightly-pipeline.md`): pointer DB, create/update
+  split, wikify as its own node; bucketing decomposition
+  (`docs/design/dreaming-nightly-pipeline.md`); entity-keyed routing (`docs/design/dreaming-nightly-pipeline.md`).
+- 2026-06-28 — Pipeline built end-to-end on real data; stage workers,
+  chronological order, self-healing, LINK stage, templating fix; batching
+  measured and shelved (`docs/design/dreaming-nightly-pipeline.md`); full import resumed
+  from 720/2096 into `~/.mlx-bun/wiki-full`.
+- 2026-07-01 — Settled with Josh: no notability filter, no embeddings,
+  buckets are stories, one edit per (bucket, article), replacement-section
+  edit mechanics; `MLX_BUN_MEMORY_BATCH` default 1. This doc became
+  canonical.
+- 2026-08-23 — Consolidated the source docs into this one; scripts moved to
+  `scripts/memory/`; every code claim re-verified; ingest-not-wired and
+  embedding-residue contradictions recorded.