mlx-bun 0.0.4 → 0.0.6

package/docs/design/memory-system.md

@@ -1,339 +1,237 @@
-# Memory system — design sketch
-
-Status: **design** (nothing built yet; capture prerequisite already landed)
-Owner: assistant layer (pi-web / pi-terminal / web-tools / jobs)
-Default: **off** until the user opts in (vault + schedule are explicit, consented actions)
-
-Ported from **lucien** (`~/Code/lucien`, vault at `~/Dreaming`) — a separate,
-already-working system at real scale (thousands of conversations, hundreds of
-articles). We are **not importing lucien**; we are reimplementing its ideas as a
-first-class, reusable mlx-bun module that runs on the **local model** and wires
-into mlx-bun's own substrate (sessions, jobs, web-tools, skills, CLI).
-
-## Goal
-
-Give the local assistant durable, user-specific memory so it can **resume
-instead of restart**. The product thesis is "local to you" — the moat is
-private accumulated context, not the weights ([[local-to-you-thesis]]). Memory
-is how that context accumulates.
-
-Three steps, the **flywheel**:
-
-1. **Capture** — conversations during sessions are saved to disk.
-2. **Synthesize** — nightly, a scheduled pass turns those conversations into
-   *articles* (a small personal wiki).
-3. **Consult** — the assistant (web + CLI) has read tools + a skill that make it
-   read the relevant articles when answering.
-
-Talk → assistant reads the wiki → you add/correct context → nightly synthesis
-folds it into articles → richer next conversation. Articles are *priors*,
-conversations are *evidence*, synthesis updates the priors over time; chat-time
-access is read-only.
-
-## Naming: "memory", not "the Dreaming"
-
-Recommendation: user-facing name is **memory** (`mlx-bun memory …`), the store
-is `~/.mlx-bun/memory/`. We keep lucien's *good descriptive concepts* (articles,
-wikilinks, Meta governance pages, Talk pages, conservative integrate-don't-
-overwrite synthesis) but drop the mythology ("the Dreaming", "librarian",
-"dreaming"). Reasons:
-
-- The Dreaming mythology is Josh's personal branding for a *different* project.
-  mlx-bun has its own identity (MLX-only, bun/JS, Apple Silicon, "local to
-  you" — [[mlx-bun-scope-and-framing]]). Importing the cosmology muddies the
-  product story.
-- "Memory" is the term users already hold (ChatGPT/Claude memory). It's
-  self-explanatory in `--help` and in the assistant's "what can you do?" list —
-  no glossary required.
-- This is a port/reimplementation, not lucien-the-dependency. A plain name
-  avoids implying we ship or require the external lucien MCP.
-
-## Port, not import: what carries over and what changes
-
-| Aspect | lucien | mlx-bun memory |
-| --- | --- | --- |
-| Read interface | MCP server (stdio) | native web-tools + a bundled skill |
-| Synthesis engine | spawns `claude -p --model opus` (cloud) | **the local model** (in-process / local server) |
-| Conversation sources | claude-code JSONL + claude.ai (Playwright) + codex | **mlx-bun's own sessions** only (web + pi terminal) |
-| Pipeline orchestration | `nightly.sh` + ~36 standalone scripts | a **job runner** on `src/jobs/*` (progress → web UI via SSE) |
-| Scheduling | user-added crontab line | `mlx-bun memory schedule` installs a **launchd** agent |
-| Setup | `lucien_setup` MCP tool | `mlx-bun memory setup` **or** the assistant via a skill |
-| Vault | `~/Dreaming` markdown + git | `~/.mlx-bun/memory/` markdown + git (same format) |
-| State DB | `.lucien/lucien.db` (sqlite) | `memory.sqlite` (sqlite) — see storage decision below |
-
-The hard, proven parts we keep wholesale: the **article format** (lead +
-sections + `## References` with `conv:` footnotes + `[[wikilinks]]`), the
-**Meta governance pages** (editorial conventions live *in* the vault, read by
-the synthesis prompts), **idempotency** (watermarks + per-chunk synthesized
-markers so re-runs are safe), and the **conservative synthesis gate** (preserve
-prose, ≥70% word-count floor, every original citation survives).
-
-The big departure is **synthesis runs locally**. lucien outsources the hard NLP
-to Opus; we do it on e4b/12B. That fits the privacy thesis (nothing leaves the
-machine) and lines up with the planned **memory LoRAs** — each pipeline stage
-(chunk / classify / synthesize / wikify) is a future specialized adapter routed
-via `adapterScoped` ([[lora-routing-architecture]]). Synthesis quality on a
-local model is the main open *quality* risk; it's a nightly batch job where
-latency doesn't matter, so we gate output the way lucien does. Graceful
-degradation applies
-([[three-tier-parity-and-degradation]]): a weak synthesis pass should no-op, not
-corrupt the vault.
-
-## Architecture
-
-### Storage layout
-
-mlx-bun already splits durable user state from rebuildable cache:
-
-- `~/.mlx-bun/` — **durable, user-owned**: `sessions/` (web chat JSONL),
-  `pi/` (terminal sessions), `pi-sessions/` (web agentDir), `skills/`,
-  `hf.json`.
-- `~/.cache/mlx-bun/` — **rebuildable / derived**: `registry.sqlite`,
-  `evals.sqlite`, `jobs.sqlite`, `jobs/`, `datasets/`, `quants/`, `adapters/`.
-
-Proposed:
+# Memory system
 
-```
-~/.mlx-bun/memory/              # the vault — precious, user-owned, git-tracked
-  articles/                     #   Topic_Name.md (underscored stems)
-  Meta/                         #   Editorial_Guidelines.md, Article_Conventions.md,
-                                #   Buckets.md, Topics_to_Ignore.md, Synthesis_Pipeline.md
-  Talk/                         #   per-article discussion / conflict notes
-  README.md
-  .git/                         #   every synthesis run commits here
-  .gitignore                    #   ignores memory.sqlite* (don't commit a binary)
-  memory.sqlite                 #   synthesis bookkeeping (cursors, chunks, buckets)
-```
+Status: **M0/M0.5 landed (2026-06-19)**. The vault, read tools, CLI, Obsidian open flow, and launchd scheduling are implemented. The actual synthesis stage bodies are still **M1 TODO**; `mlx-bun memory synthesize` is currently a safe no-op stub. The planned M1/M2 synthesis engine uses the same local models the user chats with, plus stage-specific LoRA adapters trained by Josh from the existing Lucien-scale memory corpus and distributed from Hugging Face.
 
-**Everything lives under `~/.mlx-bun/memory/`** — the vault *and* the bookkeeping
-DB (decided 2026-06-14). The win is that "your memory" is one self-contained,
-portable folder: copy it and your memory moves to another machine, intact. It
-also keeps the privacy story literal — one place to point at and one place to
-delete. The DB is derived state (it can be rebuilt from the transcripts +
-articles), so the seeded `.gitignore` excludes `memory.sqlite*` — git tracks the
-articles (the source of truth, with history), not the binary index. This is a
-deliberate, documented exception to the codebase's cache-vs-durable split
-(`evals`/`jobs`/`registry` dbs live in `~/.cache/mlx-bun/`), justified by
-treating memory as a single portable unit.
+## Contract
 
-### The vault
+The memory contract is ported from `../lucien`:
 
-Identical in spirit to lucien's. `mlx-bun memory setup` seeds it idempotently:
-create dirs, write Meta pages (only if missing — never clobber user edits),
-write README, `git init` + initial commit. Articles are plain markdown:
+> **Chat-time agents read memory. The synthesis pipeline writes memory.**
 
-```markdown
-# Archie Project
+Memory articles are durable priors. Conversations are evidence. Synthesis is the deliberate, auditable process that turns evidence into updated articles. Normal chat must not mutate articles.
 
-The **Archie Project** is an internal AI agent platform Josh built at
-[[AlphaPoint]] beginning spring 2026.[^1][^2]
+This gives both terminal and web the full pi agent experience without turning memory into an unsafe scratchpad:
 
-## Origin
-...[^1]
+- Terminal and web expose the same read-only memory tools through one shared pi surface factory.
+- Memory tools/skill are exposed only when the user has enabled the vault.
+- When enabled, memory is scoped to user-specific continuity: explicit recall, named personal projects/people/preferences/decisions/history, or missing personal context.
+- Memory is **not** the default first step for weather, current public facts, generic web research, or ordinary coding/file tasks where the current files are sufficient.
+- Mutating/coding tools remain full-power and approval-gated by pi/web as usual.
+- Memory article writes happen only through `mlx-bun memory synthesize` once M1 lands.
+- Users can always browse/edit their own Markdown vault directly, especially via Obsidian.
 
-## References
-[^1]: `conv:a1b2c3d4` (2026-05-10, mlx-bun) — Archie design overview
-```
+## User-facing model
 
-Citations point at `conv:<hash>` — a stable id derived from the source session,
-so an article is traceable back to the conversation that produced it. The vault
-is a valid Obsidian vault for free (browse with Obsidian, edit by hand; the next
-synthesis run respects your edits).
+- User-facing capability: **memory**.
+- Store on disk: `~/.mlx-bun/wiki/` (override: `MLX_BUN_WIKI`).
+- User articles: `~/.mlx-bun/wiki/articles/*.md`.
+- Read-only reference residents: `~/.mlx-bun/wiki/Reference/*.md` symlinked to mlx-bun's own docs.
+- Operational pages: `~/.mlx-bun/wiki/Meta/*.md`.
+- Talk pages: `~/.mlx-bun/wiki/Talk/`.
+- Synthesis DB: `~/.cache/mlx-bun/memory.sqlite`.
 
-### Ingestion — mlx-bun sessions only
+The vault is plain Markdown and git-tracked. It is intentionally usable as an Obsidian vault. `Reference/` is a separate read-only namespace: chat tools may search/read it, but synthesis must skip it and never write through those symlinks. The SQLite DB is derived state and lives in the rebuildable cache, not in the user-owned Markdown vault.
 
-No Playwright, no web scraping. The capture prerequisite **already exists**:
-web chat persists each session to `~/.mlx-bun/sessions/*.jsonl` via
-`SessionManager.create(...)` ([pi-web.ts:398](../../src/pi-web.ts)), and the
-comment there already names this as "the durable transcript store the nightly
-memory pipeline reads." Terminal pi persists to `~/.mlx-bun/pi`. Both are pi's
-own JSONL format, enumerable via `SessionManager.list/listAll`.
+Stage-specific memory LoRAs are product assets, not user-authored vault content. The creator trains them from the existing memory-pipeline data, publishes them to Hugging Face, and `memory init` can offer to download/cache them for local synthesis.
 
-So ingestion = read pi sessions → normalize to `{conv uuid, messages[]}` →
-upsert into `memory.sqlite`, tracking a per-source watermark so each nightly run
-only processes what's new. pi's session format becomes the ingestion contract
-for free.
+## What is implemented now
 
-### The nightly pipeline (on `src/jobs/*`)
+### Vault and article helpers
 
-Reuse the existing background-job substrate (`src/jobs/{db,runner,sse}.ts`):
-register a `memory-synthesis` runner and `submitSubprocess(...)` it so model
-state is GPU-isolated. Stages mirror lucien, each emitting `stage`/`log` events
-that stream to the web UI through the existing SSE endpoint — synthesis shows up
-in the same jobs panel as quantize/finetune.
+`src/memory/vault.ts` implements:
 
-```
-ingest      pi sessions → memory.sqlite (watermark-gated, upsert)
-chunk       segment each new conversation into topic chunks   (model call)
-cluster     assign chunks to existing buckets / propose new    (model call)
-synthesize  per bucket: create-or-update its article           (model call)
-wikify      deterministic gate + Wikipedia-style cleanup (non-fatal)
-normalize   canonical [[wikilinks]], footnote repair
-changelog   per-run digest → Meta/Changelog.md, git commit
+- `vaultRoot()` → `~/.mlx-bun/wiki` or `MLX_BUN_WIKI`.
+- `setupVault()` — idempotent directory/Meta/README/git setup plus `Reference/` symlinks.
+- `listArticles()` / `listReferenceDocs()` / `listMemoryDocuments()`.
+- `readArticle()` / `searchArticles()` over both user articles and read-only references.
+- `parseToc()` / `extractSection()`.
+- `getArticleLinks()` with resolved inbound/outbound wikilinks.
+- `importArticlesFrom()` for seeding from an existing vault such as `~/Dreaming`.
+- `vaultStatus()` with article/reference counts and recent user article mtimes.
+
+The read/search/link logic is ported from Lucien's pure filesystem helpers, not from MCP.
+
+### CLI
+
+`mlx-bun memory` supports:
+
+```bash
+mlx-bun memory init              # create vault, optionally import + schedule
+mlx-bun memory setup             # alias for init
+mlx-bun memory status            # path, article count, git, schedule, recent articles
+mlx-bun memory open              # open the vault in Obsidian, fallback Finder
+mlx-bun memory open <article>    # open one article in Obsidian/default Markdown app
+mlx-bun memory list              # list article stems + Reference/* docs
+mlx-bun memory search <query>    # search articles
+mlx-bun memory toc <article>     # headings + anchors
+mlx-bun memory section <article> <anchor>
+mlx-bun memory links <article>   # inbound/outbound wikilinks
+mlx-bun memory read <article>    # print full article
+mlx-bun memory synthesize [--dry-run] [--model <q>] [--since <cursor>]
+mlx-bun memory schedule [--at HH:MM]
+mlx-bun memory unschedule
 ```
 
-Idempotency is inherited from lucien's design: watermarks (don't re-ingest),
-`chunked_at > updated_at` (re-chunk only changed convs), and a
-`synthesized_bucket_chunks` table (never feed the same chunk into an article
-twice). A killed run resumes cleanly.
+`memory synthesize` currently reports the M1 stub and writes nothing. Scheduling is real; the scheduled job is harmless until synthesis lands. In the full flow, `memory init` should also offer to download the memory pipeline adapter pack from Hugging Face so synthesis is ready to run locally.
 
-### Synthesis runs on the local model
+### pi/web tools
 
-`mlx-bun memory synthesize` loads a model in-process (or reuses a running
-server) and drives every model call through it — **no cloud, local-only**
-(decided 2026-06-14). Default synthesis model: **e4b** (decided 2026-06-14) —
-lighter, lower memory, and mlx-bun's strongest h2h result; a weak pass is gated
-to no-op rather than allowed to corrupt the vault. Overridable per-run
-(`--model 12B` when you want the parity model's quality). This is the seam where
-the future memory
-LoRAs plug in: today base-model + prompt per stage; later `adapterScoped` per
-stage ([[lora-routing-architecture]]).
+`src/memory/tools.ts` defines native pi custom tools:
 
-### Read tools — how the assistant consults memory
+- `memory_search`
+- `memory_read`
+- `memory_toc`
+- `memory_section`
+- `memory_links`
+- `memory_list` (user articles plus read-only `Reference/*` docs)
+- `memory_status` (vault path, setup state, article count, git state, last-synthesis placeholder, schedule state, recent changed articles)
 
-Add native read tools in `src/web-tools.ts` (the `defineTool` pattern), name
-them in `WEB_TOOL_NAMES`, list them in `ALL_TOOLS` in both
-[pi-web.ts](../../src/pi-web.ts) and [pi-terminal.ts](../../src/pi-terminal.ts);
-read-only, so they auto-allow (no approval gate). Mirror lucien's read surface:
+These tools are registered by the shared pi surface in `src/pi-session.ts` and consumed by both front doors:
 
-- `memory_search` — keyword/substring search across articles → ranked hits with
-  section anchors (the discovery entry point)
-- `memory_read` — full article body
-- `memory_list` — list article titles (fallback / overview)
-- (later) `memory_section`, `memory_links` for focused reads + wikilink graph
+- `src/pi-terminal.ts`
+- `src/pi-web.ts`
 
-**Knowing *when* to consult** uses three reinforcing mechanisms:
-1. Tool descriptions that strongly cue "search the user's memory when they
-   mention their own projects, people, tools, preferences, or history."
-2. A bundled **memory skill** (below) that says "consult quietly before
-   answering personal-context questions" — same trigger surface as the
-   `anthropic-skills:lucien` skill.
-3. Cheap **index injection**: at session start, put the *list of article titles*
-   into the system prompt (a few hundred tokens) so the model knows what exists
-   and can fetch the right page. (Not the article bodies — that's what the tools
-   are for.)
+If the vault does not exist, memory tools and the memory skill are not exposed at all, avoiding prompt fixation on unavailable memory. If the vault exists, `memoryIndexHint()` adds a short scoped presence hint. Detailed usage lives in the memory skill rather than dumping article titles into the prompt.
 
-## Two front doors: CLI + assistant
+Read-only mlx-bun docs are now a separate lookup surface (`reference_search`, `reference_read`, `reference_list`) rather than being mixed into personal-memory search. Personal memory answers “what does the user think / what did we decide”; reference tools answer “what do mlx-bun docs say.”
 
-One implementation (the CLI / `src/memory` module), two ways to reach it.
+### Bundled memory skill
 
-### CLI — `mlx-bun memory <subcommand>`
+`src/web/skills/memory/SKILL.md` is bundled through `src/web/skills.ts` and materialized under `~/.mlx-bun/skills` for pi to load.
 
-New `case "memory"` in [cli.ts](../../src/cli.ts) + a `HELP.memory` entry.
+The skill tells the agent to:
 
-```
-mlx-bun memory setup            create the vault + Meta pages + git (idempotent)
-mlx-bun memory status           vault path, #articles, last synthesis, schedule state
-mlx-bun memory synthesize       run the pipeline now (--model, --since, --dry-run)
-mlx-bun memory schedule         install the nightly launchd agent (--at 03:00)
-mlx-bun memory unschedule       remove the launchd agent
-mlx-bun memory search <q>       inspect: search articles from the terminal
-mlx-bun memory read <title>     inspect: print an article
-mlx-bun memory forget <title>   delete an article (git-tracked, recoverable)
+- consult memory quietly for personal context;
+- use `memory_search` then `memory_read`, or `memory_toc` + `memory_section` for focused reads;
+- use `memory_links` for nearby concepts/backlinks;
+- use `memory_status` for setup/location/schedule questions;
+- never announce retrieval as the user's experience;
+- never edit memory articles during chat;
+- open the vault/article for the user if asked via `mlx-bun memory open [article]`.
+
+### Session capture substrate
+
+The synthesis pipeline will ingest mlx-bun's own pi sessions only:
+
+- web sessions: `~/.mlx-bun/sessions/...` / `~/.mlx-bun/pi-sessions/...` depending on the web `SessionManager` path;
+- terminal sessions: `~/.mlx-bun/pi/sessions/...`.
+
+No Playwright, scraping, cloud transcript import, or Lucien runtime dependency is planned for mlx-bun memory.
+
+### Scheduling
+
+`src/memory/schedule.ts` installs a launchd agent:
+
+- label: `com.mlx-bun.memory`
+- plist: `~/Library/LaunchAgents/com.mlx-bun.memory.plist`
+- command: `mlx-bun memory synthesize`
+- logs: `~/.mlx-bun/logs/memory-synthesis.{out,err}.log`
+
+`mlx-bun memory schedule --at 03:00` installs/loads it. `unschedule` unloads/removes it. `status` and `memory_status` report it.
+
+## Synthesis pipeline design (M1/M2)
+
+The M1/M2 pipeline mirrors Lucien's proven stages while running on the same local model stack the user chats with. The difference is execution context and authority: chat uses the model conversationally and read-only; synthesis runs the model as a constrained editor behind deterministic gates.
+
+```text
+ingest      deterministic: pi session JSONL -> memory.sqlite conversations
+chunk       base model + memory-chunk LoRA -> topic chunks
+cluster     base model + memory-bucket LoRA -> buckets / target articles
+synthesize  base model + memory-synthesis LoRA -> candidate article updates
+wikify      base model + memory-editor LoRA + deterministic normalization
+changelog   deterministic: run summary + git commit
 ```
 
-### Assistant-driven setup (a bundled skill)
+The chunking LoRA has already been trained as a WIP artifact. The remaining stage adapters are planned product assets trained by Josh from the existing Lucien/pipeline data and published to Hugging Face. mlx-bun should download/cache them during `memory init` (with user consent) or lazily on first `memory synthesize`.
 
-Add `src/web/skills/memory/SKILL.md` to `BUNDLED_SKILLS`
-([web/skills.ts](../../src/web/skills.ts)). This is the best vehicle because:
+Important nouns must stay distinct:
 
-- The skill's **description** makes "manage memory" appear under "what can you
-  do?" for free.
-- The skill's **body** encodes the explain → consent → setup script: explain in
-  plain terms (it's a folder of markdown articles; it saves your conversations;
-  it synthesizes them into articles nightly; it uses them to remember you), then
-  on agreement run the `mlx-bun memory setup` and `mlx-bun memory schedule`
-  commands via the bash tool.
+- **Conversation** — source transcript/evidence.
+- **Chunk** — topical segment from a conversation.
+- **Bucket** — staging group that collects related chunks.
+- **Article** — durable Markdown memory page.
 
-So the assistant can both **offer/explain** memory and **set it up** for you,
-and the single CLI implementation stays the source of truth.
+### Bookkeeping DB
 
-### Consent model
+`src/memory/db.ts` defines `MemoryStore` over `~/.cache/mlx-bun/memory.sqlite` with tables for:
 
-Drawn deliberately, two tiers:
+- `conversations`
+- `chunks`
+- `buckets`
+- `watermarks`
+- `synthesized_bucket_chunks`
 
-- **Capture is default-on, not a consent gate.** Saving sessions locally is what
-  pi already does (like shell history) and it never leaves the machine. But
-  default-on ≠ silent: state it once ("I keep our chats at `~/.mlx-bun/sessions`
-  so I can remember — clear them anytime"). The recent-chats sidebar doubles as
-  the "where's my data / delete it" surface.
-- **Synthesis into a persistent wiki + a scheduled job IS the consent gate** —
-  that's what changes the privacy posture. The skill asks in prose, and the
-  per-action approval gate confirms each persistent change (writing the vault,
-  installing the launchd agent). Two natural confirmations, the right amount for
-  installing a recurring job.
+The DB is derived and rebuildable from sessions + articles. It should not be committed to the vault.
 
-## Scheduling (launchd)
+### Stage-specific adapters
 
-mlx-bun has no scheduler today. Recommendation: **launchd**, not crontab.
-mlx-bun is macOS/Apple-Silicon-only by identity, so launchd is the native
-choice — it survives reboot, runs missed jobs, and gives clean
-install/uninstall. `mlx-bun memory schedule` writes
-`~/Library/LaunchAgents/com.mlx-bun.memory.plist` (a `StartCalendarInterval` at
-~03:00 running `mlx-bun memory synthesize`) and `launchctl load`s it;
-`unschedule` unloads + removes it. Schedule state surfaces in
-`mlx-bun memory status`.
+The pipeline deliberately uses **four specialized LoRA adapters** rather than one generic memory adapter:
 
-## Privacy (the throughline)
+| Stage | Adapter | Job | Primary quality signal |
+| --- | --- | --- | --- |
+| Chunk | memory-chunk | Conversation transcript → topic-coherent chunks | Boundary/label accuracy vs gold chunks |
+| Bucket | memory-bucket | Chunk → existing bucket or new bucket proposal | Stable routing, low duplicate topics, good article target |
+| Synthesize | memory-synthesis | New chunks + existing article → conservative candidate update | Citation preservation, faithful integration, no hallucinated claims |
+| Edit/wikify | memory-editor | Candidate article → normalized wiki Markdown | Style, wikilinks, references, deterministic format compliance |
 
-Local model + local vault + local cron + local transcripts — **nothing leaves
-the machine**. This is consistent with the web-chat privacy framing where the
-web tools are the only outbound path. It is also the differentiator: not the
-best AI, the best at knowing *you* ([[local-to-you-thesis]]).
+These adapters do **not** get direct write authority. They produce candidates. The pipeline validates, normalizes, and decides whether to write/commit.
 
-## Module layout (reusable)
+Implementation notes:
 
-A self-contained `src/memory/` module so the feature is reusable by both the CLI
-and the server, and is testable in isolation:
+- Adapters should be cached with the rest of mlx-bun's HF/downloaded artifacts, not stored in the wiki vault.
+- `memory init` should describe the adapter pack plainly and ask before downloading multi-MB/GB assets.
+- `memory synthesize --dry-run` must work without downloading adapters when it only builds the deterministic manifest.
+- If a required adapter is missing for a real synthesis stage, the command should offer the download or fail cleanly; it should not silently fall back to an untrained prompt path unless explicitly requested.
+- Future routing should use the existing adapter-scoped generation path: one resident base model, stage LoRA mounted per pipeline call.
 
-```
-src/memory/
-  vault.ts        paths, setup (dirs + Meta + git), article read/write/list
-  db.ts           memory.sqlite: conversations, chunks, buckets, watermarks
-  ingest.ts       pi sessions → db  (uses SessionManager.list/listAll)
-  chunk.ts        conversation → topic chunks  (local-model call)
-  cluster.ts      chunks → buckets             (local-model call)
-  synthesize.ts   bucket → article create/update + gate  (local-model call)
-  wikify.ts       deterministic editorial gate + normalize
-  pipeline.ts     registerRunner("memory-synthesis", …) — ties stages together
-  tools.ts        memory_search / memory_read / memory_list (web-tools)
-  schedule.ts     launchd plist install/remove
-  prompts/        per-stage prompts (the future LoRA seams)
-```
+### Current stage modules
+
+These files exist as typed stubs and currently throw if called directly:
+
+- `src/memory/ingest.ts`
+- `src/memory/chunk.ts`
+- `src/memory/cluster.ts`
+- `src/memory/synthesize.ts`
+- `src/memory/wikify.ts`
+- `src/memory/prompts.ts`
+
+`src/memory/pipeline.ts` intentionally does **not** call them yet. It emits a no-op M1-stub plan so manual and scheduled synthesis cannot accidentally corrupt the vault before gates exist.
+
+## Synthesis quality gates to port from Lucien
+
+M1 should preserve these Lucien properties:
+
+- deterministic manifest / dry-run before writes;
+- per-bucket/per-article workers so failures are isolated;
+- idempotency via `synthesized_bucket_chunks`;
+- backfill behavior for existing imported articles;
+- orphan detection when ledger and files disagree;
+- conservative update prompt: integrate, do not rewrite wholesale;
+- minimum word-count/truncation guard, e.g. Lucien's 70% floor;
+- citation preservation: every original `conv:` reference survives;
+- deterministic footnote repair;
+- deterministic wikilink normalization;
+- git commit for article changes;
+- changelog / run summary.
+
+A weak local-model synthesis pass should no-op, not damage the vault. This remains true even with trained adapters: LoRAs propose; deterministic gates dispose. Synthesis must operate only on `articles/` and must treat `Reference/` as immutable read-only context.
+
+## Consent and safety
+
+- Local session capture is local-only and part of pi/web session persistence.
+- Creating the vault is explicit: `mlx-bun memory init`.
+- Scheduling is explicit: `mlx-bun memory schedule` or the init wizard prompt.
+- Chat-time memory tools are read-only and auto-allowed.
+- Opening Obsidian/Finder is user-directed browsing/editing; it is not agent mutation.
+- Article mutation by the agent during chat is out of bounds.
+
+## Next implementation steps
 
-## Phasing
-
-- **M0 — vault + read path.** `src/memory/{vault,db}.ts`, `memory setup`, the
-  read tools + index injection, the bundled skill (offer/explain/setup). No
-  synthesis yet — the assistant can already read a hand-written or imported
-  vault. Lets us validate the *consult* half against a real vault (even one
-  copied from `~/Dreaming`) before building the pipeline.
-- **M1 — ingest + pipeline (manual).** `ingest/chunk/cluster/synthesize/wikify`
-  as a `memory-synthesis` job, run on demand via `memory synthesize`. Validate
-  synthesis quality on the local model against every model/quant we care about
-  ([[per-model-quant-specialization]]).
-- **M2 — schedule.** launchd install/remove; status surfacing; transparency
-  line on first capture.
-- **M3 — quality + LoRAs.** Per-stage memory LoRAs via `adapterScoped`; KL /
-  quality benchmarks for the synthesis output ([[lora-routing-architecture]]).
-
-## Help text + docs changes (don't forget)
-
-- `HELP.memory` in [cli.ts](../../src/cli.ts) (+ the top-of-file command list).
-- README: a "Memory" section under the pitch (the flywheel + privacy).
-- `docs/reference/`: a `memory.md` (vault format, CLI surface, tool list).
-- This design doc; update PLAN.md with a Memory phase + exit criteria.
-- The bundled skill description (so "manage memory" shows in "what can you do?").
-
-## Decisions (settled 2026-06-14)
-
-1. **Storage** — everything (vault + `memory.sqlite`) under `~/.mlx-bun/memory/`,
-   one portable folder; `.gitignore` excludes the DB.
-2. **Synthesis engine** — local-only, no cloud; weak output is gated to no-op.
-3. **Default synthesis model** — e4b, overridable per-run (`--model 12B`).
-
-## Open decisions
-
-1. **Retroactive capture.** On first setup, synthesize past sessions too, or
-   future-only? Recommendation: offer "synthesize what's already here" as a
-   one-time step right after setup, surfaced by the skill.
+1. Implement `memory synthesize --dry-run` manifest generation without model calls.
+2. Add memory adapter-pack metadata and `memory init` download prompt (HF-hosted LoRAs; chunk adapter exists as WIP, others follow).
+3. Implement ingest from saved pi session JSONL into `MemoryStore`.
+4. Wire the chunk stage to the memory-chunk LoRA and evaluate with chunk boundary/label accuracy.
+5. Implement bucket assignment with the memory-bucket LoRA and article manifest create/update/backfill/orphan modes.
+6. Implement per-article synthesis workers with the memory-synthesis LoRA and conservative gates.
+7. Implement editor/wikify adapter pass plus deterministic footnote/wikilink normalization and git commits.
+8. Move synthesis execution onto the jobs runner with SSE progress once manual synthesis is reliable.

package/docs/design/optimization_plan.md

@@ -1,8 +1,10 @@
 # mlx-bun: gemma4.ts decompose & optimize — action plan
 
 > This is the canonical **design/rationale** for the optimization work.
-> Live **status/tracking** (checkboxes, findings) lives in PLAN.md under
-> "Optimization plan Phase A–E".
+> Live **status/tracking** (checkboxes, findings) for Phases A–D lives in
+> PLAN-archive.md; Phase E remains in PLAN.md.
+
+**Status (verified 2026-06-17):** Phases A–D are **DONE** (`[x]` in PLAN-archive.md, all landed 2026-06-10/11). Phase E — per-config fused decode-SDPA kernel — is **in progress** (`[~]` in PLAN.md, started 2026-06-11; fused kernel exists and is quality-gated per PLAN.md Step 4). The body of this doc remains the durable design rationale; completed phase checkboxes and findings live in PLAN-archive.md; Phase E tracking lives in PLAN.md.
 
 **Premise (confirmed from the source read):** the current engine sits at
 parity-to-ahead vs mlx-lm on the 12B *while still interpreting the decode