mlx-bun 0.2.0 → 0.4.0

package/docs/design/memory-synthesis.md

@@ -1,287 +0,0 @@
-# Memory synthesis — the local Morpheus (M1 design)
-
-> **SUPERSEDED (2026-07-01)** by
-> [dreaming-nightly-pipeline.md](./dreaming-nightly-pipeline.md). Kept for
-> history. Known-dead parts: embedding-based section-routing and silhouette
-> signals (**no embeddings anywhere in the project**), bucket-as-optional
-> framing, and the periodic-only wikify (RECONCILE/LINK/infobox are nightly
-> stages now).
-
-Status: **design**, 2026-06-26. Companion to [memory-system.md](./memory-system.md)
-(the M0 substrate: vault, read tools, CLI, scheduling — all landed). This doc is
-the M1 design for the **write side**: the synthesis pipeline that turns raw
-conversations into well-formed wiki articles, **on local models**.
-
-The inspiration and proof-of-concept is Lucien / The Dreaming / Morpheus
-(`../lucien`) — it works, but every stage runs on a cloud model (`claude --model
-opus`, `pi`). The whole point of M1 is to keep every learning from Lucien and
-**solve the cost by running the pipeline locally**, using one local base model +
-a per-stage LoRA adapter trained from Lucien's corpus (the e4b chunk adapter is
-the first of these — designed components of the appliance, each distilled from
-the cloud pipeline's gold output).
-
-Guiding principle, inherited verbatim from Lucien: **do whatever Wikipedia does.**
-Wikipedia turned a crowd of strangers into the largest organized, continuously-
-updated knowledge base ever built. We are not going to out-design it — we copy
-its conventions (articles, sections, talk pages, citations, wikilinks, edit
-history) and supply the per-edit labor with local models instead of volunteers.
-
-And the division of labor is fixed: **AI does the judgment; deterministic code
-enforces the invariants we can't trust AI on** (footnote bijection, wikilink
-canonicalization, the conservative write gate).
-
-## Current state: ZERO articles
-
-The vault starts empty. So the pipeline has to handle the **creation flow**, not
-just steady-state updates — most of the early design discussion assumed an
-existing structured article to patch; from zero there is nothing to patch yet.
-The two flows are genuinely different operations and the doc treats them
-separately below.
-
-## The pipeline (DAG)
-
-```
-WRITE · the local Morpheus  (what `mlx-bun memory synthesize` runs)
-══════════════════════════════════════════════════════════════════
-
-  conversation transcripts  (ingest, by cursor)
-        │
-        ▼  CHUNK ································· local model + chunk LoRA   [e4b ✓]
-  single-topic (segment + label)
-        │
-        ▼  FILTER — worth keeping? ──no──▶ drop
-        │ yes
-        ▼  CLASSIFY → bucket(s)   many-to-many ──no fit──▶ drop
-        │   bucket ≈ article; from zero, buckets are PROPOSED (taxonomy emerges)
-        ▼  SYNTHESIZE   ······························ local model + synth LoRA
-        │   ├─ article does NOT exist  →  CREATE  (sub-cluster → outline → per-section draft)
-        │   └─ article EXISTS          →  UPDATE  (SECTION-ROUTE → PATCH one section)
-        │        + conservative write gate (≥70% prose floor, every citation survives, weak → NO-OP)
-        ▼
-  article written / one section changed (rest byte-identical)
-        │
-        ▼  NORMALIZE (deterministic code, not a model)
-        │   footnote renumber/repair · wikilink canonicalize
-        ▼
-        ▼  WIKIFY (periodic, local model + wikify LoRA)        ◀── distinct LLM node
-        │   "make this a BETTER version" — per-section improve + structural pass
-        └──▶ Talk-page signals (embedding/silhouette): SPLIT · MERGE · CONTRADICTION
-
-
-READ · any chat agent  (M0, landed; read-only via memory_* tools)
-══════════════════════════════════════════════════════════════════
-  question ─▶ search / toc ─▶ read section · follow [[links]] ─▶ reason ─▶ answer
-```
-
-```mermaid
-flowchart TD
-    T[conversation transcripts] --> C["CHUNK · local+LoRA"]
-    C -->|segment+label| F{worth keeping?}
-    F -->|no| D1((drop))
-    F -->|yes| CL["CLASSIFY → bucket(s)"]
-    CL -->|no fit| D2((drop))
-    CL --> EX{article exists?}
-    EX -->|no| CR["CREATE: sub-cluster → outline → per-section draft"]
-    EX -->|yes| UP["UPDATE: section-route → patch one section"]
-    CR --> GATE["conservative write gate"]
-    UP --> GATE
-    GATE --> NORM["NORMALIZE footnotes + wikilinks · deterministic"]
-    NORM --> WK["WIKIFY periodic · per-section improve + structural pass"]
-    WK -->|talk-page| SIG["SPLIT · MERGE · CONTRADICTION"]
-```
-
-## The two synthesis flows
-
-### CREATE (cold-start / bootstrap — the zero-article case)
-
-Trigger: a bucket has accumulated enough chunks (≥N, conservative) and has **no
-article file yet**. Naively this is whole-article generation from a pile of
-chunks — exactly the open-ended task a small local model is *worst* at. So we
-decompose it the same way we decompose updates — **section-granular**:
-
-1. **Sub-cluster** the bucket's chunks into proto-sections (cluster the chunks
-   *within* the bucket — the cohesion/embedding machinery applies one level down;
-   silhouette is a strong cohesion signal, ρ≈0.68 vs the judges).
-2. **Outline** — name each section from its sub-cluster, write the lead. A small
-   structural task, not long-form prose.
-3. **Draft each section** by synthesizing only its sub-cluster's chunks — the
-   *same bounded per-section operation* as a patch, just starting from empty.
-4. **Assemble** → article with `# Title`, lead, sections, `## References`, TOC.
-
-So even creation never asks the model to hold a whole article in its head: it's
-sub-cluster → outline → N bounded section drafts. Conservative bootstrap bias:
-**under-cover rather than over-cover** — thin buckets stay `{{stub}}` or wait for
-more material (Lucien's lesson: twenty good articles beat a sprawling mediocre
-set).
-
-### UPDATE (steady state — article exists)
-
-1. **SECTION-ROUTE** the new chunk to the section(s) it affects, via the TOC —
-   "this topic affects THESE sections" (e.g. a Reflecting-Pool-renovation chunk →
-   the *Renovation* section, and possibly an *Abuse-of-Power* section of a Trump
-   article). Many-to-many at section granularity. If it fits no existing section,
-   the route is "new section."
-2. **PATCH** just that section: hand the model the section text + the chunk, ask
-   it to weave the chunk in with a `[^N]` citation. Bounded in, bounded out.
-   Facet extraction is implicit — the same chunk becomes the renovation facts in
-   one section and the no-bid-contract framing in another, because the model
-   writes what's relevant to the section it's editing.
-3. **Everything else stays byte-identical** — only the routed section is rewritten.
-
-Both flows share the **conservative write gate** (already specced in
-`synthesize.ts`): preserve prose, ≥70% word-count floor, every original `conv:`
-citation survives, integrate-not-overwrite; a weak local pass is gated to NO-OP
-rather than allowed to corrupt the vault.
-
-## Why WIKIFY is its own (necessary) node
-
-Section-patches are greedy and local. Over many of them an article accumulates
-cruft: a section sprawls, sentences go redundant, the lead goes stale, section
-order drifts. **Wikify is the periodic global pass that keeps the article *good*
-— "how do I turn this into a better version of this article" — and it is
-necessary; without it you just keep bolting things on.** This is the LLM
-editorial-restructure (Lucien's "Editorial restructure" commits), *distinct from*
-the deterministic footnote/wikilink normalize that the current `wikify.ts` stub
-conflates it with.
-
-Structure makes even this global pass decompose toward local: wikify can run
-**per-section "improve this section"** + a **cheap structural pass over the TOC**
-(reorder, fix the lead, re-section, emit split/merge signals) instead of one
-whole-article regeneration. So wikify is local-tractable for the same reason
-synthesis is: sections.
-
-## Data model — pointers, not copies
-
-The SQLite store (`~/.cache/mlx-bun/memory.sqlite`) is **derived, rebuildable**
-state — not the source of truth. The principle that keeps it lean and consistent:
-**materialize text in exactly one place; everything else points.**
-
-- **Messages hold the text, once.** A `messages` table (text per message), keyed
-  by `(conv, position)`.
-- **A chunk is a pointer range, not a copy.** `chunks(id, conv, start, end, label)`
-  — a segmentation is *where the boundaries are*, so storing it as a message range
-  + label is the natural representation. Synthesis *dereferences* to message text
-  only when it reads a chunk. Re-chunking rewrites pointer rows, never text.
-- **Assignments are join tables (many-to-many).** `chunk_buckets(chunk_id,
-  bucket_name)` and, for the section design, `chunk_sections(chunk_id, article,
-  section_anchor)` — "this chunk affects THESE sections of THESE articles."
-- **The article markdown is the only materialized synthesis.** Its `[^N]:
-  conv:HASH` citations are themselves *pointers back to source conversations*, not
-  copies. The vault is the precious git-tracked truth for *articles*; the DB is the
-  rebuildable index for *pipeline state*.
-
-**The current `db.ts` stub violates this and must be fixed for M1** (it mirrors a
-naive copy design, not Lucien's pointer design):
-- `conversations.transcript` stores the whole transcript blob AND `chunks.text`
-  copies each slice → the same text twice. Replace with a `messages` table +
-  pointer `chunks` (drop `chunks.text`).
-- `chunks.bucket TEXT` is one-bucket-per-chunk → breaks many-to-many. Replace with
-  `chunk_buckets`. Add `chunk_sections` for the section-route output.
-- `synthesized_bucket_chunks` → `synthesized_chunk_sections(article, section_anchor,
-  chunk_id, synthesized_at)` so the patch ledger is per-section (fold each chunk
-  into each section exactly once).
-
-Target tables: `messages` · `chunks` (pointers) · `conversations` · `buckets` ·
-`chunk_buckets` · `chunk_sections` · `synthesized_chunk_sections` · `watermarks`.
-
-### Engine: bun:sqlite now, Turso swappable (for vectors)
-
-Build M1 on **`bun:sqlite`** — native to Bun, zero-dependency, synchronous, the
-most battle-tested DB there is, and already the project's pattern (`JobStore`,
-`EvalDB`, `registry`, `MemoryStore`). No reason to pay friction on day one.
-
-**Turso** (the current product — a from-scratch Rust rewrite of SQLite that is
-*embedded / on-device / offline / SQLite-compatible*, NOT the old cloud-sync
-product) is the evaluation path. Its one concrete payoff for us is **native vector
-search** — and this design is embedding-heavy (section-routing, semantic article
-search, the silhouette signals). Reasons to defer rather than adopt now: it's a
-young engine vs C SQLite, and the Bun-integration story (native binding vs
-FFI/overhead) needs checking.
-
-The decision is cheap to defer because (a) Turso is SQLite-*compatible* so the
-schema/SQL above is identical either way, (b) `MemoryStore` already wraps the DB
-behind a small class, so it's a driver swap not a rewrite, and (c) this DB is a
-**rebuildable cache** (truth = vault + git), so it's a low-risk place to adopt a
-younger engine. Plan: keep the SQL standard, keep the driver swappable, and switch
-to Turso *if/when* its vector search beats in-process cosine (or `sqlite-vec`) at a
-corpus size that actually needs ANN.
-
-## The cascade (why local is possible at all)
-
-```
-clean single-topic chunks → clean buckets → clean section-routing → bounded section-patch
-        └──── purity at the root ────┘
-```
-
-Each stage's locality depends on the prior stage's cleanliness. If a chunk fuses
-two topics, it pollutes every section it's routed into. So **chunk purity is the
-foundation of the whole local system** — not an abstract metric.
-
-Caution from the chunk research (research-journal, reports/orpo-chunk-paper):
-embeddings measure **cohesion/dispersion well** (silhouette ρ≈0.68 per-conv vs
-the LLM judges) but **purity/contamination poorly** (−intraSpread ρ≈0.32, ns
-within variants). So a clean-bucket guarantee cannot lean on a cheap embedding
-signal — it rides on the local **chunker** actually being good at single-topic
-splitting (the e4b adapter's real job), backed where needed by a non-embedding
-purity check, not by cosine geometry.
-
-## Stage → model → adapter
-
-| stage | file | model | adapter |
-|---|---|---|---|
-| chunk | `src/memory/chunk.ts` | local | **chunk LoRA (e4b chunk-300, exists)** |
-| classify | `src/memory/cluster.ts` | local + embeddings | classify LoRA (TODO) |
-| synthesize (create/update) | `src/memory/synthesize.ts` | local | synth LoRA (TODO) |
-| normalize | `src/memory/wikify.ts` (rename) | **deterministic code** | — |
-| wikify (LLM) | **new node** | local | wikify LoRA (TODO) |
-
-Section-routing (the new intra-article step) is embeddings-shaped: embed the
-chunk (or its label) and the article's section headings/lead (from `parseToc` /
-`extractSection`, already in `vault.ts`), route to the nearest section(s) over a
-threshold. The local Qwen3-Embedding backbone (bit-exact, served, `/v1/embeddings`)
-is exactly the "vector search over articles, not conversations" the vision calls
-for — now applied at section granularity.
-
-## What changes in the scaffold
-
-The M1 stubs are wired (`pipeline.ts` orchestrates ingest→chunk→cluster→
-synthesize→wikify→changelog; stage bodies throw "not implemented"). Two
-corrections vs the stubs:
-
-1. **`synthesize.ts` is whole-article today.** Make it `create-or-update`, with
-   update = section-route + section-patch (not whole-article regeneration).
-2. **The LLM wikify node is missing.** The current `wikify.ts` is *only*
-   deterministic normalize. Keep that (rename to `normalize.ts`), and add a
-   distinct **LLM wikify** node for the periodic "better version" pass.
-
-## Build sequence
-
-Cold-start first, because the vault is empty and you can't patch what doesn't exist:
-
-1. **Wire CHUNK** to the e4b chunk adapter (`chunk.ts`) — the one stage with an
-   adapter in hand. Verify single-topic chunks + labels on real conversations.
-2. **CLASSIFY from zero** (`cluster.ts`) — emergent bucket taxonomy
-   (`Meta/Buckets.md`); assign chunks to buckets, propose new ones; filter no-fit.
-3. **CREATE flow** (`synthesize.ts`) — sub-cluster → outline → per-section draft
-   for buckets crossing the chunk threshold. Behind the conservative gate. This
-   is what gets us from zero to a first set of articles.
-4. **NORMALIZE** (`normalize.ts`) — wire the deterministic footnote/wikilink
-   repair (`vault.ts` already has the resolver).
-5. **UPDATE flow** — section-route + patch, once articles exist to update.
-6. **WIKIFY (LLM)** + Talk-page split/merge signals — once articles have
-   accumulated enough cruft/drift to need them.
-
-Steps 1–4 are the cold-start path (zero → first articles); 5–6 are steady state.
-
-## Open questions
-
-- **Article granularity / when to CREATE** — chunk threshold per bucket; when a
-  bucket should become one article vs several (the vision's "soft size caps").
-- **Section-route confidence** — threshold for "fits an existing section" vs
-  "new section needed" vs "route to a different article."
-- **Provisional facts** — representing "work set to begin July" so the next pass
-  cleanly updates it to "began / completed" (temporal/trajectory tracking).
-- **Create vs strong-model** — is first-creation good enough on the local model,
-  or is bootstrap the one place a stronger model is worth it (vision: bootstrap is
-  heavier but the same code path)?
-- **The purity check** — what non-embedding signal guarantees clean buckets.

package/docs/design/memory-system.md

@@ -1,237 +0,0 @@
-# Memory system
-
-Status: **M0/M0.5 landed (2026-06-19)**. The vault, read tools, CLI, Obsidian open flow, and launchd scheduling are implemented. The actual synthesis stage bodies are still **M1 TODO**; `mlx-bun memory synthesize` is currently a safe no-op stub. The planned M1/M2 synthesis engine uses the same local models the user chats with, plus stage-specific LoRA adapters trained by Josh from the existing Lucien-scale memory corpus and distributed from Hugging Face.
-
-## Contract
-
-The memory contract is ported from `../lucien`:
-
-> **Chat-time agents read memory. The synthesis pipeline writes memory.**
-
-Memory articles are durable priors. Conversations are evidence. Synthesis is the deliberate, auditable process that turns evidence into updated articles. Normal chat must not mutate articles.
-
-This gives both terminal and web the full pi agent experience without turning memory into an unsafe scratchpad:
-
-- Terminal and web expose the same read-only memory tools through one shared pi surface factory.
-- Memory tools/skill are exposed only when the user has enabled the vault.
-- When enabled, memory is scoped to user-specific continuity: explicit recall, named personal projects/people/preferences/decisions/history, or missing personal context.
-- Memory is **not** the default first step for weather, current public facts, generic web research, or ordinary coding/file tasks where the current files are sufficient.
-- Mutating/coding tools remain full-power and approval-gated by pi/web as usual.
-- Memory article writes happen only through `mlx-bun memory synthesize` once M1 lands.
-- Users can always browse/edit their own Markdown vault directly, especially via Obsidian.
-
-## User-facing model
-
-- User-facing capability: **memory**.
-- Store on disk: `~/.mlx-bun/wiki/` (override: `MLX_BUN_WIKI`).
-- User articles: `~/.mlx-bun/wiki/articles/*.md`.
-- Read-only reference residents: `~/.mlx-bun/wiki/Reference/*.md` symlinked to mlx-bun's own docs.
-- Operational pages: `~/.mlx-bun/wiki/Meta/*.md`.
-- Talk pages: `~/.mlx-bun/wiki/Talk/`.
-- Synthesis DB: `~/.cache/mlx-bun/memory.sqlite`.
-
-The vault is plain Markdown and git-tracked. It is intentionally usable as an Obsidian vault. `Reference/` is a separate read-only namespace: chat tools may search/read it, but synthesis must skip it and never write through those symlinks. The SQLite DB is derived state and lives in the rebuildable cache, not in the user-owned Markdown vault.
-
-Stage-specific memory LoRAs are product assets, not user-authored vault content. The creator trains them from the existing memory-pipeline data, publishes them to Hugging Face, and `memory init` can offer to download/cache them for local synthesis.
-
-## What is implemented now
-
-### Vault and article helpers
-
-`src/memory/vault.ts` implements:
-
-- `vaultRoot()` → `~/.mlx-bun/wiki` or `MLX_BUN_WIKI`.
-- `setupVault()` — idempotent directory/Meta/README/git setup plus `Reference/` symlinks.
-- `listArticles()` / `listReferenceDocs()` / `listMemoryDocuments()`.
-- `readArticle()` / `searchArticles()` over both user articles and read-only references.
-- `parseToc()` / `extractSection()`.
-- `getArticleLinks()` with resolved inbound/outbound wikilinks.
-- `importArticlesFrom()` for seeding from an existing vault such as `~/Dreaming`.
-- `vaultStatus()` with article/reference counts and recent user article mtimes.
-
-The read/search/link logic is ported from Lucien's pure filesystem helpers, not from MCP.
-
-### CLI
-
-`mlx-bun memory` supports:
-
-```bash
-mlx-bun memory init              # create vault, optionally import + schedule
-mlx-bun memory setup             # alias for init
-mlx-bun memory status            # path, article count, git, schedule, recent articles
-mlx-bun memory open              # open the vault in Obsidian, fallback Finder
-mlx-bun memory open <article>    # open one article in Obsidian/default Markdown app
-mlx-bun memory list              # list article stems + Reference/* docs
-mlx-bun memory search <query>    # search articles
-mlx-bun memory toc <article>     # headings + anchors
-mlx-bun memory section <article> <anchor>
-mlx-bun memory links <article>   # inbound/outbound wikilinks
-mlx-bun memory read <article>    # print full article
-mlx-bun memory synthesize [--dry-run] [--model <q>] [--since <cursor>]
-mlx-bun memory schedule [--at HH:MM]
-mlx-bun memory unschedule
-```
-
-`memory synthesize` currently reports the M1 stub and writes nothing. Scheduling is real; the scheduled job is harmless until synthesis lands. In the full flow, `memory init` should also offer to download the memory pipeline adapter pack from Hugging Face so synthesis is ready to run locally.
-
-### pi/web tools
-
-`src/memory/tools.ts` defines native pi custom tools:
-
-- `memory_search`
-- `memory_read`
-- `memory_toc`
-- `memory_section`
-- `memory_links`
-- `memory_list` (user articles plus read-only `Reference/*` docs)
-- `memory_status` (vault path, setup state, article count, git state, last-synthesis placeholder, schedule state, recent changed articles)
-
-These tools are registered by the shared pi surface in `src/pi-session.ts` and consumed by both front doors:
-
-- `src/pi-terminal.ts`
-- `src/pi-web.ts`
-
-If the vault does not exist, memory tools and the memory skill are not exposed at all, avoiding prompt fixation on unavailable memory. If the vault exists, `memoryIndexHint()` adds a short scoped presence hint. Detailed usage lives in the memory skill rather than dumping article titles into the prompt.
-
-Read-only mlx-bun docs are now a separate lookup surface (`reference_search`, `reference_read`, `reference_list`) rather than being mixed into personal-memory search. Personal memory answers “what does the user think / what did we decide”; reference tools answer “what do mlx-bun docs say.”
-
-### Bundled memory skill
-
-`src/web/skills/memory/SKILL.md` is bundled through `src/web/skills.ts` and materialized under `~/.mlx-bun/skills` for pi to load.
-
-The skill tells the agent to:
-
-- consult memory quietly for personal context;
-- use `memory_search` then `memory_read`, or `memory_toc` + `memory_section` for focused reads;
-- use `memory_links` for nearby concepts/backlinks;
-- use `memory_status` for setup/location/schedule questions;
-- never announce retrieval as the user's experience;
-- never edit memory articles during chat;
-- open the vault/article for the user if asked via `mlx-bun memory open [article]`.
-
-### Session capture substrate
-
-The synthesis pipeline will ingest mlx-bun's own pi sessions only:
-
-- web sessions: `~/.mlx-bun/sessions/...` / `~/.mlx-bun/pi-sessions/...` depending on the web `SessionManager` path;
-- terminal sessions: `~/.mlx-bun/pi/sessions/...`.
-
-No Playwright, scraping, cloud transcript import, or Lucien runtime dependency is planned for mlx-bun memory.
-
-### Scheduling
-
-`src/memory/schedule.ts` installs a launchd agent:
-
-- label: `com.mlx-bun.memory`
-- plist: `~/Library/LaunchAgents/com.mlx-bun.memory.plist`
-- command: `mlx-bun memory synthesize`
-- logs: `~/.mlx-bun/logs/memory-synthesis.{out,err}.log`
-
-`mlx-bun memory schedule --at 03:00` installs/loads it. `unschedule` unloads/removes it. `status` and `memory_status` report it.
-
-## Synthesis pipeline design (M1/M2)
-
-The M1/M2 pipeline mirrors Lucien's proven stages while running on the same local model stack the user chats with. The difference is execution context and authority: chat uses the model conversationally and read-only; synthesis runs the model as a constrained editor behind deterministic gates.
-
-```text
-ingest      deterministic: pi session JSONL -> memory.sqlite conversations
-chunk       base model + memory-chunk LoRA -> topic chunks
-cluster     base model + memory-bucket LoRA -> buckets / target articles
-synthesize  base model + memory-synthesis LoRA -> candidate article updates
-wikify      base model + memory-editor LoRA + deterministic normalization
-changelog   deterministic: run summary + git commit
-```
-
-The chunking LoRA has already been trained as a WIP artifact. The remaining stage adapters are planned product assets trained by Josh from the existing Lucien/pipeline data and published to Hugging Face. mlx-bun should download/cache them during `memory init` (with user consent) or lazily on first `memory synthesize`.
-
-Important nouns must stay distinct:
-
-- **Conversation** — source transcript/evidence.
-- **Chunk** — topical segment from a conversation.
-- **Bucket** — staging group that collects related chunks.
-- **Article** — durable Markdown memory page.
-
-### Bookkeeping DB
-
-`src/memory/db.ts` defines `MemoryStore` over `~/.cache/mlx-bun/memory.sqlite` with tables for:
-
-- `conversations`
-- `chunks`
-- `buckets`
-- `watermarks`
-- `synthesized_bucket_chunks`
-
-The DB is derived and rebuildable from sessions + articles. It should not be committed to the vault.
-
-### Stage-specific adapters
-
-The pipeline deliberately uses **four specialized LoRA adapters** rather than one generic memory adapter:
-
-| Stage | Adapter | Job | Primary quality signal |
-| --- | --- | --- | --- |
-| Chunk | memory-chunk | Conversation transcript → topic-coherent chunks | Boundary/label accuracy vs gold chunks |
-| Bucket | memory-bucket | Chunk → existing bucket or new bucket proposal | Stable routing, low duplicate topics, good article target |
-| Synthesize | memory-synthesis | New chunks + existing article → conservative candidate update | Citation preservation, faithful integration, no hallucinated claims |
-| Edit/wikify | memory-editor | Candidate article → normalized wiki Markdown | Style, wikilinks, references, deterministic format compliance |
-
-These adapters do **not** get direct write authority. They produce candidates. The pipeline validates, normalizes, and decides whether to write/commit.
-
-Implementation notes:
-
-- Adapters should be cached with the rest of mlx-bun's HF/downloaded artifacts, not stored in the wiki vault.
-- `memory init` should describe the adapter pack plainly and ask before downloading multi-MB/GB assets.
-- `memory synthesize --dry-run` must work without downloading adapters when it only builds the deterministic manifest.
-- If a required adapter is missing for a real synthesis stage, the command should offer the download or fail cleanly; it should not silently fall back to an untrained prompt path unless explicitly requested.
-- Future routing should use the existing adapter-scoped generation path: one resident base model, stage LoRA mounted per pipeline call.
-
-### Current stage modules
-
-These files exist as typed stubs and currently throw if called directly:
-
-- `src/memory/ingest.ts`
-- `src/memory/chunk.ts`
-- `src/memory/cluster.ts`
-- `src/memory/synthesize.ts`
-- `src/memory/wikify.ts`
-- `src/memory/prompts.ts`
-
-`src/memory/pipeline.ts` intentionally does **not** call them yet. It emits a no-op M1-stub plan so manual and scheduled synthesis cannot accidentally corrupt the vault before gates exist.
-
-## Synthesis quality gates to port from Lucien
-
-M1 should preserve these Lucien properties:
-
-- deterministic manifest / dry-run before writes;
-- per-bucket/per-article workers so failures are isolated;
-- idempotency via `synthesized_bucket_chunks`;
-- backfill behavior for existing imported articles;
-- orphan detection when ledger and files disagree;
-- conservative update prompt: integrate, do not rewrite wholesale;
-- minimum word-count/truncation guard, e.g. Lucien's 70% floor;
-- citation preservation: every original `conv:` reference survives;
-- deterministic footnote repair;
-- deterministic wikilink normalization;
-- git commit for article changes;
-- changelog / run summary.
-
-A weak local-model synthesis pass should no-op, not damage the vault. This remains true even with trained adapters: LoRAs propose; deterministic gates dispose. Synthesis must operate only on `articles/` and must treat `Reference/` as immutable read-only context.
-
-## Consent and safety
-
-- Local session capture is local-only and part of pi/web session persistence.
-- Creating the vault is explicit: `mlx-bun memory init`.
-- Scheduling is explicit: `mlx-bun memory schedule` or the init wizard prompt.
-- Chat-time memory tools are read-only and auto-allowed.
-- Opening Obsidian/Finder is user-directed browsing/editing; it is not agent mutation.
-- Article mutation by the agent during chat is out of bounds.
-
-## Next implementation steps
-
-1. Implement `memory synthesize --dry-run` manifest generation without model calls.
-2. Add memory adapter-pack metadata and `memory init` download prompt (HF-hosted LoRAs; chunk adapter exists as WIP, others follow).
-3. Implement ingest from saved pi session JSONL into `MemoryStore`.
-4. Wire the chunk stage to the memory-chunk LoRA and evaluate with chunk boundary/label accuracy.
-5. Implement bucket assignment with the memory-bucket LoRA and article manifest create/update/backfill/orphan modes.
-6. Implement per-article synthesis workers with the memory-synthesis LoRA and conservative gates.
-7. Implement editor/wikify adapter pass plus deterministic footnote/wikilink normalization and git commits.
-8. Move synthesis execution onto the jobs runner with SSE progress once manual synthesis is reliable.

package/docs/design/mlx-lm-serving-execution-seam.md

@@ -1,81 +0,0 @@
-# mlx-lm serving execution seam
-
-## Question
-
-What should mlx-bun borrow from the pinned mlx-lm oracle when deepening Completion execution around this flow?
-
-```text
-prepared prompt -> semantic events -> terminal summary
-```
-
-This note reads the local mlx-lm 0.31.3 source installed in the reference venv. The comparison uses the serving term defined in `CONTEXT.md`: a **Completion execution** includes output, resource lifetime, usage, and scheduling placement.
-
-mlx-lm is the correctness oracle only for behavior and numeric schemes it implements. It is not mlx-bun's architecture oracle or capability ceiling. mlx-bun also implements explicitly selected features such as mixed-precision KV, TurboQuant, and native draft methods. The execution path must preserve those resolved settings exactly.
-
-## Implementation status
-
-The first consolidation slice landed on 2026-08-21. `prepareCompletion()` performs admission before a streaming response starts and returns an opaque, single-use `PreparedCompletion`. `CompletionExecutor.execute()` then owns placement reporting, `CompletionSink`, logprobs, semantic event delivery, finish reason, usage, and pre-generation cleanup. [`completion-executor.ts:75`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:75) [`completion-executor.ts:219`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:219)
-
-Chat and raw-text adapters still declare their semantic components and format their own wire responses. Their streaming and non-streaming branches now call the same executor. `GenerationGateway` still owns placement and model execution. The change does not add or remove flags, alter defaults, or substitute one inference method for another.
-
-S3 completed the scheduling declaration at the same seam. A
-`GenerationPlacement` now names `serial` or `continuous`; it no longer calls a
-lone scheduler request "batched." The continuous scheduler chooses its B=1
-fast path or B=N step from active rows. The strict serial executor remains for
-`--batch 1` and for declared compositions the continuous scheduler does not yet
-implement. Existing parity evidence covers both the default scheduler's B=1
-path and strict serial. The recorded scheduler/serial decode ratios are
-0.992–0.996, and a fresh main-vs-branch serial gate after the sampler
-consolidation measured a 1.000 median-best ratio (about 265.0 tok/s on both
-sides, MiniCPM5, two interleaved 256-token repeats).
-
-Final acceptance after the GPQA pause exercised the seam through the complete
-two-shard suite and a real two-turn browser conversation. The suite passed
-2,064 tests with zero failures. The browser run verified streaming, tool
-events, context carryover, metrics, and turn recovery. A mainline-only queued
-DOM render race found by that run was fixed at the UI lifecycle boundary; no
-decode-loop work was added. The architecture changes likewise keep accounting
-and capability decisions outside numerical kernels. The fresh serial A/B plus
-the established scheduler B=1 parity and latency evidence are the performance
-gate.
-
-## What the oracle actually does
-
-The HTTP adapter creates a small `CompletionRequest` that still contains protocol-shaped chat messages or a raw text prompt. It separately builds `GenerationArguments`. Prompt preparation has not happened yet. [`server.py:182`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:182) [`server.py:198`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:198) [`server.py:1368`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1368)
-
-`ResponseGenerator.generate()` submits that pair to one worker thread through a queue. The worker applies the chat template, tokenizes, finds cache segments, chooses the initial reasoning state, constructs stop/tool/reasoning transitions, selects a lane, owns cache work, and emits per-token `Response` records. [`server.py:516`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:516) [`server.py:626`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:626) [`server.py:688`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:688) [`server.py:1026`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1026)
-
-That organization can be summarized as:
-
-```text
-HTTP request
-  -> CompletionRequest + GenerationArguments
-  -> ResponseGenerator queue
-  -> tokenize + state machine + lane + cache
-  -> token Response { text, token, state, match, logprob, finish_reason }
-  -> HTTP handler accumulates semantics and formats the protocol response
-```
-
-The model loader is separate. `utils.load()` returns the model and a tokenizer wrapper, while tokenizer loading chooses the streaming detokenizer and tool parser. This keeps model construction out of per-request execution. [`utils.py:453`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/utils.py:453) [`tokenizer_utils.py:578`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/tokenizer_utils.py:578)
-
-## Comparison by concern
-
-| Concern | Oracle organization | Consequence for the mlx-bun seam |
-| --- | --- | --- |
-| Prepared prompt | The worker owns chat templating, tokenization, initial reasoning state, and cache-oriented prompt segmentation. The HTTP adapter does not know those details. [`server.py:530`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:530) [`server.py:565`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:565) | A prepared-prompt value is a sound seam, but it should be richer than token ids. It must carry the generation inputs and owned media or grammar state needed by either lane. Protocol adapters should not assemble `RequestShape` or transfer resources piecemeal. |
-| Semantic events | The state machine classifies tokens as `normal`, `reasoning`, or `tool`. `_process_control_tokens` withholds matched control sequences. The HTTP handler then accumulates reasoning, tool text, visible text, and parses completed tool calls. [`server.py:236`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:236) [`server.py:654`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:654) [`server.py:1442`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1442) | Do not copy this split. mlx-bun's `CompletionSink` already emits content, reasoning, and tool-call events through one interface. [`completion-sink.ts:7`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-sink.ts:7) [`completion-sink.ts:90`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-sink.ts:90) The deepened module should own that semantic work so all protocol adapters consume the same events. |
-| Terminal summary | `generate.stream_generate()` puts timing, token counts, memory, and finish reason on every low-level response, but the server discards most of it. The HTTP handler counts emitted tokens, reads prompt and cache counts from `GenerationContext`, derives tool-call finish, and builds usage. [`generate.py:269`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/generate.py:269) [`generate.py:728`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/generate.py:728) [`server.py:1495`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1495) [`server.py:1517`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1517) | The oracle has no single terminal summary at the server seam. mlx-bun should add one. It should be the sole source of finish reason, prompt/generated/cached counts, final lane, speculation facts, and collected result. |
-| Streaming and non-streaming | Both modes consume the same response iterator and semantic accumulation loop. Streaming flushes deltas and optionally emits a final usage packet; non-streaming sends one accumulated response. [`server.py:1454`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1454) [`server.py:1507`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1507) [`server.py:1529`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1529) | Keep one Completion execution interface. Streaming and non-streaming are protocol adapters over the same semantic events and terminal summary, not separate execution paths. |
-| Scheduling placement | Model loading marks a model batchable only when no draft model exists and every cache supports merge. Per-request placement also excludes a fixed seed. The worker joins the active batch, creates a batch, runs serially, or drains an incompatible batch. [`server.py:370`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:370) [`server.py:685`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:685) [`server.py:731`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:731) [`server.py:801`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:801) [`server.py:831`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:831) | `RequestShape` describes settings already resolved from flags, request fields, model metadata, and defaults. `place()` asks only whether the continuous scheduler implements that exact composition; it never replaces MTP, a KV scheme, or another requested method. The frozen declaration names the `serial` or `continuous` mechanism, and both reporting and `run()` consume it. Active-row count selects B=1 or B=N inside the continuous mechanism. |
-| Cancellation and flow | Cancellation is cooperative. The handler's `finally` sets a flag; serial execution checks it after tokens, while batched execution checks it during prompt and generation work and removes the row. [`server.py:1008`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1008) [`server.py:861`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:861) [`server.py:911`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:911) [`server.py:1551`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1551) The response queue is unbounded, and socket writes block only the HTTP thread, so there is no end-to-end demand signal from the client to generation. [`server.py:444`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:444) [`server.py:1032`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1032) [`server.py:1489`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1489) | Do not regress to the oracle here. mlx-bun's token callback accepts a promise or `false`, and `GenerationGateway.run()` accepts an `AbortSignal`. [`generation-gateway.ts:94`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/generation-gateway.ts:94) [`generation-gateway.ts:354`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/generation-gateway.ts:354) Cancellation and async flow control are part of the interface, not HTTP details. |
-| Cache and resource lifetime | The worker fetches or creates the serial cache, generates, then inserts it. Batched execution fetches before admission, snapshots segment caches, inserts the final cache, and removes cancelled rows. [`server.py:963`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:963) [`server.py:1018`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1018) [`server.py:752`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:752) [`server.py:864`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:864) [`server.py:901`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:901) `LRUPromptCache` hides prefix matching, copying, trimming, and count/byte eviction behind a small interface. [`cache.py:1623`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/models/cache.py:1623) [`cache.py:1674`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/models/cache.py:1674) [`cache.py:1696`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/models/cache.py:1696) | `prepareCompletion()` now disposes owned media and grammar state on admission rejection. `execute()` disposes the same plan on placement or pre-generation failure, then transfers ownership exactly once before `GenerationGateway.run()`. Cache and in-generation disposal remain in the gateway and generation path. [`completion-executor.ts:75`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:75) [`completion-executor.ts:219`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:219) |
-| Errors | Errors before the response iterator exists become a JSON error, although the oracle uses status 404 for that whole class. Errors delivered later through the worker queue are raised by the iterator. The handler guarantees `ctx.stop()` but has no matching mid-stream error packet in this loop. [`server.py:1417`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1417) [`server.py:1424`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1424) [`server.py:1035`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1035) [`server.py:1454`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1454) | Preserve a typed distinction between rejection before execution and failure after events begin. Both must run the same ownership cleanup. Protocol adapters decide how to encode the failure after headers have been sent. |
-
-## Strongest conclusions
-
-1. **Keep resolved inference settings authoritative.** Serial and continuous execution are scheduling mechanisms for the already-selected composition. Consolidating placement may shrink the HTTP adapter, but it must not turn explicit features or opinionated defaults into inferred choices.
-2. **Do not borrow the oracle's semantic split.** Its worker emits state-tagged token records, then the HTTP handler performs semantic accumulation and tool parsing. mlx-bun already has a better starting point in `CompletionSink`. Move that module behind the Completion execution seam so protocol adapters see semantic events only.
-3. **Keep the terminal summary as the only completion result.** It now settles finish reason, usage, cached-token count, final lane, speculation, logprobs, and collected output once. The four HTTP branches only format that summary. [`completion-executor.ts:103`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:103) [`server.ts:3104`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/server.ts:3104)
-4. **Treat cancellation, async flow control, and ownership as interface requirements.** The oracle's queue makes cancellation and socket pressure indirect. mlx-bun has stronger mechanisms already. The deepening should hide their implementation without weakening their behavior.
-
-The deletion test is positive. If this Completion execution module disappeared, scheduling policy, cache participation, semantic parsing, finish rules, usage, and cleanup would spread back across four protocol branches. Concentrating those rules gives callers leverage and gives maintainers locality. Feature selection and fidelity policy remain outside this seam as resolved inputs.