mlx-bun 0.0.8 → 0.0.10

package/docs/design/bucketing-stage.md

@@ -0,0 +1,196 @@
+# Bucketing / Classify stage — local build plan
+
+Status: **design**, 2026-06-26. Companion to [memory-synthesis.md](./memory-synthesis.md).
+Grounded in a research+critique workflow over the real Lucien classify + the local
+foundation; re-centered on the governing principle below.
+
+## Governing principle — capacity by decomposition
+
+**A small local model's effective capacity scales inversely with task size. You don't
+make the model smarter; you make each task narrower.** Every stage is a *ladder of
+atomic, single-purpose tasks*, each one a single clear question. The full pipeline:
+
+```
+"what buckets should we have?"        ← taxonomy (cold-start)
+"which buckets need this chunk?"      ← classify / assignment
+"which section needs this info?"      ← section routing
+"integrate THIS chunk into THIS section"  ← synthesis patch (most atomic)
+```
+
+…and each rung decomposes *further* into the smallest decisive question. We never ask
+the local model to "assign a chunk among 12 buckets and emit JSON" — that is still too
+big. We ask **"does this chunk belong in bucket B? yes/no,"** one bucket at a time.
+
+**Retrieval (embeddings, lexical) exists only to NARROW the candidate set so the model
+makes the fewest, smallest, clearest calls. It never decides — the model decides, one
+tiny question at a time.** This is the production home for the embedding work: a
+candidate-narrower, not a classifier.
+
+**Why brute-force decomposition is affordable: the model is LOCAL, so inference is ~free.**
+A cloud model forces you to minimize calls (which is exactly why Lucien made one big
+"reason over everything" call). A local model flips the economics — you can run **M×N tiny
+binary checks** ("does chunk *i* belong in bucket *j*?" / "should chunk *i* update section
+*j*?") where a cloud model would have to make one expensive combined decision. We trade one
+big unreliable call for many tiny reliable ones. Local-ness is what makes radical
+decomposition viable; retrieval just keeps M×N from getting silly.
+
+This is adapting + improving Lucien, not duplicating it. Lucien put all 379 buckets in
+Opus's context and let it reason globally in one shot — a capability a 1B model does not
+have. We replace "one global decision" with "many tiny local decisions."
+
+## What we keep from Lucien
+
+- Classify on chunk **labels** (~49-char topic strings), not text or vectors.
+- The `Bucketing.md` policy governs behavior, inlined into the task prompt — editing it
+  re-tunes routing with no code change.
+- **Many-to-many** (a chunk → 0..3 buckets; empty is legal = the substance filter).
+- **Create buckets by judgment** when nothing fits and the topic is substantive.
+- **Existing-buckets-win** (re-scoped to global below — fix E).
+- Idempotent set-difference resumption (`NOT IN chunk_buckets`).
+
+## The bucketing stage, decomposed
+
+### "which buckets need this chunk?" — assignment
+
+For one chunk (its label):
+1. **Narrow (no model):** embedding shortlist — cosine(label, bucket centroid) → top-K
+   candidates; PLUS a **lexical floor** (any bucket whose name token-overlaps the label)
+   so an embedding miss can't drop an obvious match [fix C].
+2. **Decide (atomic model task, one per candidate):** *"Does this chunk — `<label>` —
+   belong in bucket `<name>: <description>`? Answer yes or no."* Run over the (few)
+   shortlisted candidates. No multi-item JSON, no batch indices → the JSON-fidelity and
+   batching failure modes (critique D) simply don't exist. A chunk can get "yes" on
+   several candidates (many-to-many falls out naturally) or none.
+3. **New bucket (atomic task, only if every candidate was "no" AND substantive):**
+   *"This topic fits no existing bucket. Propose one Wikipedia-style bucket name (2–6
+   words) + one-line description, or answer NONE if too thin."* Then the **global gate
+   [fix E — the #1 correctness fix]:** re-embed the proposed name against the FULL bucket
+   centroid cache; if the nearest existing bucket clears a similarity threshold, downgrade
+   to it. This restores Lucien's "none of 379 fit" discipline that a shortlist-only view
+   would break (→ taxonomy explosion).
+
+Each model call is a tiny yes/no or a single short proposal — the clear single tasks a
+1B model can do reliably. Cost is more calls, but each is fast and reliable; embeddings
+keep K small so it's bounded.
+
+### "what buckets should we have?" — cold-start taxonomy (from zero)
+
+There are **7308 distinct labels of 7316** — dedup collapses nothing, and a 1B model
+cannot globally compress thousands of labels into a coherent 30–60-bucket taxonomy
+(that is exactly Opus's global-reasoning ability). Decompose:
+1. **Group (geometry, no model):** embed all labels → k-means (k∈[30,60], pick by
+   silhouette). The geometry does the grouping the model can't do globally.
+2. **Purity-gate each group (atomic task) [fix B]:** *"Do these labels form ONE topic or
+   more than one?"* k-means optimizes cohesion and is blind to fusion (the ρ≈0.32 purity
+   blind spot we measured) — split any group the model flags before it becomes a bucket.
+3. **Name each group (atomic task):** *"Here are ~15 representative labels. Give ONE
+   Wikipedia-style bucket name (2–6 words) + a one-line description. Be specific, not
+   generic."*
+4. **Seed centroids from members [fix H]:** each new bucket's centroid = mean of the
+   cluster labels that birthed it (those vectors already exist) — so the first assignment
+   pass runs on strong member-based centroids, not weak name-only ones.
+
+### later rungs (designed here, built after classify)
+
+- **"which section needs this info?"** — the **M×N binary** pattern: for an article's
+  assigned bucket of *M* chunks and its *N* TOC sections, ask *"should chunk i update
+  section j? yes/no"* — M×N times. The "yes" pairs become atomic patches. A chunk that gets
+  **all "no" and is substantive → an atomic CREATE-NEW-SECTION task** (mirrors new-bucket).
+  (Embeddings can prune N, but with a local model you can also just run all M×N.)
+- **"integrate THIS chunk into THIS section"** — the atomic synthesis patch (memory-synthesis.md),
+  applied to each (chunk, section) "yes" pair.
+
+### The entities (distinct — do not conflate)
+
+- **chunk** — one topical segment of a conversation (pointer + label).
+- **bucket** — a single-topic GROUP OF CHUNKS (the classify output; the "Cameras and
+  Camera Gear" container you drop camera chunks into). **A bucket is NOT an article.**
+- **article** — a synthesized wiki page. A bucket RELATES TO **one or more** articles
+  (many-to-many): the reflecting-pool-renovation bucket relates to the *Lincoln Memorial
+  Reflecting Pool* article AND a *Trump* article.
+- **section** — a part of an article (its TOC entries).
+
+### Hierarchical pruning (coarse gates fine)
+
+Decompose top-down and **prune the subtree at every "no."** The levels are distinct:
+chunk → bucket → article(s) → section(s). The coarse gate is **"does this bucket relate to
+this article? yes/no"** — if no, skip *every* section of that article (a bucket can't
+relate to a section of an article it doesn't relate to). So the M×N section checks run
+*only inside the few articles a bucket relates to*, never "× every section in the wiki."
+Same idea recursively: each coarser binary check prunes the finer ones beneath it.
+Retrieval narrows candidates *within* a level; pruning eliminates whole levels.
+
+### The uniform growth rule (no-match → create)
+
+At *every* rung, "no existing candidate matched" branches on substance, it does not just
+drop: **substantive → CREATE, trivial → drop.** A chunk that fits no bucket → new bucket;
+a bucket that relates to no existing article → new article; a chunk that fits no section of
+a related article → new section. Only non-substantive trivia is filtered. So substantive
+content always lands *somewhere* — this is how the taxonomy grows new buckets, the wiki
+grows new articles, and articles grow new sections (the Wikipedia editor adds a page or a
+section for a fact that fits nowhere). Each create is its own atomic task, gated against
+the existing set (the new-bucket global gate, fix E).
+
+## Data model (`src/memory/db.ts`)
+
+Rewrite the copy-schema to the pointer design (memory-synthesis.md; no live data):
+`messages` (text once, `(conv, position)`), `chunks` (pointers + `label`, no text),
+`buckets` (writable), `chunk_buckets` (M:N — the classify output). `PRAGMA foreign_keys
+= ON` + `ON DELETE CASCADE` (fixes Lucien's 368 orphan edges / 570 dangling pointers).
+**Keep `synthesized_bucket_chunks` per-(bucket,chunk) [fix G]** — do not rename the
+working ledger for the unbuilt section feature; `chunk_sections` may exist as a
+future-proof empty table only.
+
+## Eval — embeddings as the test instrument
+
+`scripts/experiments/bucket-cohesion-signal.ts` (mirrors `chunk-embedding-signal.ts`):
+- **[measure FIRST — the #1 risk] shortlist recall ceiling:** is each gold bucket in its
+  chunk's embedding top-K? Report overall **and per bucket-size-bin** (122/379 buckets
+  have <5 chunks — the global number hides tail collapse [fix C]). If low, the shortlist
+  caps recall and must be re-tuned before building on it.
+- **bucket cohesion / separation / subcluster-gap** baseline over the 379 cloud buckets.
+- **purity (LLM judge)** on lowest-subcluster-gap buckets — embeddings can't grade purity
+  (ρ≈0.32); confirm the asymmetry so the judge stays load-bearing.
+- **decision accuracy:** since each task is binary, eval is per-(chunk,candidate-bucket)
+  yes/no accuracy + per-chunk Jaccard vs the 8441 valid gold edges. Report **oracle-shortlist**
+  F1 (force gold into candidates → pure model judgment) vs **real-shortlist** F1 (end-to-end);
+  the gap is the retrieval tax. **The LoRA decision keys off the oracle F1 [fix F].**
+
+## LoRA decision (staged)
+
+- **Stage 0 — embedding-only:** top-K cosine + threshold vs the 8809 gold edges → F1/Jaccard.
+  Might end it (classify could be embedding-shaped).
+- **Stage 1 — base MiniCPM5-1B, prompt-only, binary tasks + shortlist:** measure oracle-F1,
+  yes/no consistency. The binary framing is *much* easier for a small base model than JSON
+  multi-assignment — Stage 1 may clear the bar with no LoRA precisely because the tasks are atomic.
+- **Stage 2 — train `memory-bucket` LoRA, only if Stage 0/1 miss.** Data from lucien.db, one
+  example per (chunk, candidate-bucket) binary decision (gold = belongs?), shortlist regime =
+  serve regime. Recipe = `chunk-finetune.ts` (sft, rank 16, lr 1e-5, seq 8192,
+  `MLX_BUN_PERF_KERNEL=0 MLX_BUN_FUSED_GELU=0`, no train-flash); adapter →
+  `~/.cache/mlx-bun/adapters/memory-bucket`; pick checkpoints by the F1 judge, not val loss.
+  **GPU-gated.**
+
+## Build sequence
+
+Runnable now (no GPU training, no server):
+1. **[FIRST]** `bucket-cohesion-signal.ts` → shortlist recall ceiling (per size-bin) +
+   cohesion/sep/subcluster baseline. Verifies the shortlist foundation before any build.
+2. Rewrite `db.ts` to the pointer schema + `foreign_keys` + helpers; in-memory test.
+3. Stage-0 embedding-only baseline (oracle + real F1).
+4. Purity judge on lowest-subcluster-gap buckets; confirm the asymmetry.
+5. Implement `cluster.ts` as **atomic tasks**: shortlist → per-candidate yes/no → new-bucket
+   proposal + global gate; cold-start = cluster → purity-gate → name → member-seed.
+6. Stage-1 eval (base, no adapter) → apply LoRA criteria.
+
+GPU-gated (only if needed): 7. build binary SFT data · 8. train `memory-bucket` · 9. re-eval.
+Integration: 10. wire `clusterChunks` into `pipeline.ts`.
+
+**First command:** `bun scripts/experiments/bucket-cohesion-signal.ts`
+
+## Open risks
+
+- Shortlist recall ceiling (measure per size-bin first — step 1).
+- More model calls per chunk (K binary vs 1 JSON) — bounded by K; each call is tiny/fast.
+- Cold-start naming going generic — seed with distinctive labels + "specific not generic".
+- Distilled labels ≠ ground truth — don't over-trust exact-match F1; use reference-free
+  silhouette + purity on divergences.

package/docs/design/compat-cli-surface-design.md

@@ -55,14 +55,14 @@ shared subset; optiq-only features are additive (marked ⊕).
 | `generate` | mlx_lm.generate | ✅ `generate.ts` (no CLI verb) | wire verb + full flag set |
 | `chat` | mlx_lm.chat | engine ✅, web chat ✅; no terminal REPL (`tui.ts` is help-formatting) | build REPL frontend over `generate` |
 | `server` | mlx_lm.server / optiq serve | ✅ `serve` | rename flags; ⊕ `--kv-config`, `--anthropic` honored (default on) |
-| `convert` | mlx_lm.convert + optiq convert | ❌ | quant pipeline; ⊕ `--target-bpw/--candidate-bits/--reference` |
+| `convert` | mlx_lm.convert + optiq convert | ✅ `convert` (wraps `quantize/quantizer.ts`) | -q/--q-bits/--q-group-size/--mlx-path/--upload-repo parity (upload rides `hf-push.ts`); ⊕ `--target-bpw/--candidate-bits/--calibration-mix/--n-calibration` (mixed precision); no `--dtype`/`--dequantize`/plain non-quantizing convert |
 | `lora` | mlx_lm.lora + optiq lora | serve/hot-swap ✅, train ✅ | `--train/--test`; ⊕ `--rank-scaling by_bits\|by_kl`; `lora info` |
-| `fuse` | mlx_lm.fuse | ❌ | fuse adapters, dequant, GGUF export, upload |
+| `fuse` | mlx_lm.fuse | ✅ `fuse` (`train/fuse.ts`) | --adapter-path/--save-path parity; preserves per-module quant layout; dequant/GGUF/upload not supported (says so, exits 1) |
 | `cache_prompt` | mlx_lm.cache_prompt | runtime ✅, CLI ❌ | precompute + save reusable KV cache file |
 | `benchmark` | mlx_lm.benchmark / optiq benchmark | ✅ `benchmark` | flag-name parity (`-p/-g/-b/-n`); ppl half ❌ |
 | `evaluate` | mlx_lm.evaluate / optiq eval | ❌ (`evals` = our bench viewer) | task harness; ⊕ optiq task names |
-| `perplexity` | mlx_lm.perplexity | ❌ | ppl on HF dataset |
-| `upload` | mlx_lm.upload | ❌ | push MLX dir to HF |
+| `perplexity` | mlx_lm.perplexity | ✅ `perplexity` (`eval/perplexity.ts`) | reference methodology (pack → non-overlapping rows → f32 CE, delta-method SE); data source is a LOCAL .txt/.jsonl, not an HF dataset |
+| `upload` | mlx_lm.upload | ✅ `upload` (wraps `hf-push.ts`, the web push engine) | --path/--upload-repo parity; ⊕ `--private`; token from hf.json/$HF_TOKEN/hf auth login |
 | `manage` | mlx_lm.manage | ✅ `get`/`scan`/`ls` | add `--delete/--pattern/--scan` parity |
 | `kv-cache` | optiq kv-cache | runtime ✅, profiler ❌ | per-layer sensitivity → `kv_config.json` |
 | `latency` | optiq latency | ✅ `fit` | alias + `--calibrate` |
@@ -146,19 +146,19 @@ column; the 🟥 column is the capability matrix that fills in behind it.
 | Precompute prompt cache | `cache_prompt` | 🔌 | Phase 17 |
 | LoRA serve / hot-swap | `lora` (serve) | ✅ | done (Phase 8) |
 | `lora info` reporting | `lora info` | 🔌 | Phase 17 |
-| **HF→MLX convert → new model artifact** | `convert` | 🟥 | Model-quant (new phase) |
-| **Sensitivity / mixed-precision weights** | `convert`, `dynamic_quant` | 🟥 | Model-quant |
+| **HF→MLX convert → new model artifact** | `convert` | ✅ | done (CLI verb over the native quantize pipeline) |
+| **Sensitivity / mixed-precision weights** | `convert`, `dynamic_quant` | ✅ (`convert --target-bpw`) | done via convert; `dynamic_quant` alias 🟥 |
 | **AWQ / DWQ / GPTQ (weights)** | `awq`/`dwq`/`gptq` | 🟥 | Model-quant |
 | KV-cache quant: bf16 / uniform / **mixed per-layer** (kv_config.json) | `server` (`off`/`N`/`config`) | ✅ | done (Phase 9/10; config.ts reads, generate.ts applies per-layer) |
 | KV sensitivity *profiler* (authors a NEW kv_config.json) | `kv-cache` | 🟥 | model-prep, low pri (shipped artifacts already include one) |
 | TurboQuant — optional extra quant *method* (rotation VQ) | scheme, not a verb | 🟥 | Phase 13 (KV plumbing already exists) |
 | **LoRA / DoRA / full training** | `lora --train` | ✅ | Training (done: Phase A MiniCPM5 + Phase B e4b) |
 | **Sensitivity-aware rank scaling** | `lora --rank-scaling` | 🟥 | Training |
-| **Fuse adapters (+GGUF/upload)** | `fuse` | 🟥 | Training |
+| **Fuse adapters (+GGUF/upload)** | `fuse` | ✅ fuse; GGUF/dequant/upload 🟥 | done (fold-into-base, quant layout preserved) |
 | **lm-eval-harness tasks** | `evaluate` | 🟥 | Eval (new phase) |
 | **optiq eval suite (kl/gsm8k/ifeval/bfcl/hashhop…)** | `evaluate` | 🟥 | Eval |
-| **Perplexity** | `perplexity` | 🟥 | Eval |
-| **HF upload** | `upload` | 🟥 | Distribution |
+| **Perplexity** | `perplexity` | ✅ | done (local text/jsonl datasets) |
+| **HF upload** | `upload` | ✅ | done (native `hf-push.ts`; also `convert --upload-repo`) |
 | **Distributed share** | `share` | 🟥 | Distribution (lowest pri) |
 | **Web UI: quantize** | `lab`/UI | 🟥 | gated on Quantize |
 | **Web UI: fine-tune** | `lab`/UI | ✅ | done (SFT/DPO workflow, loss curve, merge, export) |
@@ -181,7 +181,7 @@ does. The real capability work splits along two axes:
   (Phase 13).
 - **Training**: LoRA/DoRA/full training ✅ (shipped); `fuse` verb (adapter merge, GGUF export, upload) 🟥 remaining.
 - **Eval**: harness + `perplexity`.
-- **Distribution**: `upload`/`share`.
+- **Distribution**: `upload` ✅ (CLI verb + `convert --upload-repo`); `share` 🟥.
 
 The OptIQ-Lab web-UI tiles gate on these: quantize ← model-quant,
 fine-tune ← training, training-data ← independent, chat ✅.
@@ -225,5 +225,5 @@ correctness diffs need no clean machine.
    to author new configs (low pri) + the TurboQuant method (Phase 13).
 4. Training → ✅ `lora --train` + web-UI fine-tune tile (both shipped); remaining: `fuse` verb (adapter merge, GGUF export, upload).
 5. Eval → `evaluate`/`perplexity`.
-6. Distribution → `upload`/`share`.
+6. Distribution → `upload` ✅ (2026-07-01); `share` remaining.
 7. Web UI training-data tile (independent of engine work).

package/docs/design/docs-reorg-plan.md

@@ -12,7 +12,7 @@ and need a test run, so deferred to their own pass). Authored 2026-06-14.
 - P1 #5 — **DONE**: `STATUS.md` exists at root.
 - P1 #6 — **DONE**: `docs/planning/` exists with `IDEAS.md`, `PRODUCT_ROADMAP.md`, `ResearchTopics.md`, `journal.md`.
 - P1 #7 — open (optimization_plan.md still lives in docs/design/ as a standalone).
-- P1 #8 — status unknown (archive/ not verified in this pass).
+- P1 #8 — **RESOLVED (2026-07-01)**: the duplicate `archive/pi-builtin-investigation.html` is already gone; `archive/` deliberately keeps 2 small demo artifacts (`mlx-bun-lab-report.html`, `wwdc-mlx-bun.html`, 76K total), tracked.
 - P2 #9 — **NOT DONE**: `harness-pi.ts`, `pi-provider.ts`, `pi-terminal.ts`, `pi-web.ts` still live flat in `src/` root (no `src/pi/` subdirectory).
 - P2 #10 — **PARTIALLY DONE**: ~57 one-off research/debug scripts moved to `scripts/experiments/`; production + tooling + bench-* + regen-* + parity-* + gen-model + eval/serve remain at `scripts/` root (the full bench/regen/build split was not applied).
 - P2 #11 — **DONE**: CLI vocabulary reconciled.

package/docs/design/dreaming-nightly-pipeline.md

@@ -0,0 +1,291 @@
+# The Dreaming — nightly memory pipeline (CANONICAL design)
+
+Status: **canonical**, 2026-07-01. This document states the system design as
+settled with Josh and **supersedes any conflicting statement** in
+[the-dreaming-master-plan.md](./the-dreaming-master-plan.md) (task breakdown still
+useful; several decisions overridden below),
+[memory-synthesis.md](./memory-synthesis.md) (superseded), and
+[the-dreaming-handoff.md](./the-dreaming-handoff.md) (session log, mostly aligned).
+
+The system is the second iteration of Lucien's memory pipeline, rebuilt to run
+**entirely on local models** inside mlx-bun (`src/memory/`). Same methodology and
+pipeline at the core; the local constraint drives two design consequences:
+
+1. **Small tasks.** Every AI step is decomposed to the smallest single-purpose
+   unit a small local model can do well — one chunk, one bucket, one section,
+   ideally a binary judgment. Never a whole-article blob in or out.
+   **And every AI step is pure text-in/text-out — no tool calls, no agent
+   loop.** The model never edits a file, never searches, never decides what to
+   do next; the pipeline hands it exactly the context it needs and applies its
+   output deterministically. This lowers the capability bar dramatically: a
+   small model that could never drive an agentic edit session can absolutely
+   rewrite one section given the section and the new information — and pure
+   completion tasks are exactly what per-stage LoRA SFT trains well.
+2. **Deterministic gates between AI steps.** AI does judgment; deterministic
+   code enforces invariants (citations, footnotes, wikilinks, infobox shape,
+   word floors). A weak AI output is gated to NO-OP, never allowed to corrupt
+   the vault.
+
+## Fixed principles (do not re-derive, do not contradict)
+
+- **Conversations are the base.** The pipeline's only input is conversation
+  transcripts (pi sessions daily; historical imports are a one-time onboarding
+  path through the same code).
+- **No notability filter.** *"If I talk about it, it's notable."* Everything
+  surfaces; nothing is dropped or parked as trivia. Any prior "owned/decided/≥3
+  chunks else PARK" gate is dead — the catch-all bucket exists only so that
+  low-volume topics remain searchable until they warrant their own article.
+- **No embeddings, anywhere in the project.** Not for routing, not for
+  clustering, not as "offline instruments." Grouping and routing are done by
+  deterministic name/alias/category lookup plus small local-model judgments.
+- **Articles are real single-topic wiki articles with sections** — one subject
+  per article, a lead, an infobox, `##` sections, See also, References. All
+  subject mentions are `[[wikilinks]]`.
+- **Vault = truth, DB = derived cache.** Markdown in the vault is canonical and
+  git-tracked; the SQLite store is pipeline state + rebuildable index
+  (`reindex`).
+- **Self-healing through ordinary use.** Corrections arrive as ordinary chunks;
+  chronological processing (`updated_at ASC`) resolves claims toward the user's
+  latest position.
+- **Prompts and Meta pages are SCHEMATIC** (no real names/specs/dates) — the
+  small model parrots concrete example values into unrelated articles.
+- **Runs nightly**, locally, unattended (launchd, `src/memory/schedule.ts`).
+- **Model evals are not part of this design.** The eval harnesses (chunk-eval,
+  judges, F1/Jaccard gates, cloud-judge sampling) are tooling for *choosing and
+  training the models used by* the pipeline. They live beside the system, never
+  inside it, and belong in their own docs.
+
+## The pipeline
+
+```
+conversations                                    (ingest, watermark/cursor-gated)
+     │
+     ▼  SEGMENT ──────────────── AI: find the topics in each conversation
+  chunks: single-topic segments with labels (pointer ranges, no text copies)
+     │
+     ▼  BUCKET ───────────────── AI + deterministic: group similar topics
+  buckets: one per STORY/TOPIC; everything discussed about that topic since the
+  last run accumulates here (a bucket usually has NO article of its own)
+     │
+     ▼  RESOLVE ──────────────── which articles does each story affect?
+  bucket delta → the subject articles it touches (update · create · sections)
+     │
+     ▼  SYNTHESIZE ───────────── AI: per-section create/patch, bounded in and out
+  create: outline → per-section draft · update: patch one section, rest byte-identical
+     │
+     ▼  ── deterministic gates ── citation survival · footnote bijection ·
+     │     word floor · wikilink resolution · weak output → NO-OP
+     │
+     ▼  RECONCILE ────────────── infobox = ground truth; lead/body/infobox agree
+     ▼  LINK ─────────────────── cross-link: mentions → [[links]], co-occurrence → See also
+     ▼  NORMALIZE ────────────── deterministic: footnote renumber, wikilink canonicalize,
+     │                           structure guards (References last, stub marking)
+     ▼  EDITORIAL (wikify) ───── AI: remove redundancy, organize, make it read well —
+     │                           per-section improve + structural pass over the TOC
+     ▼
+  commit to vault (git) + run changelog
+```
+
+Every stage is an independent, resumable, DB-state-keyed worker
+(`src/memory/stages.ts`), runnable separately or as the full DAG
+(`mlx-bun memory synthesize`).
+
+### SEGMENT — conversations → topics
+
+The e4b chunk adapter (`e4b-chunk-300`, symlinked `memory-chunk`) segments each
+conversation into single-topic chunks with short labels. One chunk = one
+coherent topic; a multi-stage process is ONE topic; long single-topic
+conversations stay whole. Chunks are pointer ranges over `messages` — text is
+materialized exactly once. Deterministic anchor repair validates every boundary.
+
+### BUCKET — group similar topics (first-class stage)
+
+**The Wikipedia-editor analogy (this is the model — Josh's framing).** A news
+story gets reported seven times in a day — Fox says one thing, MSNBC another,
+CBS a third, each report carrying a bit of different information. A Wikipedia
+editor who heard ALL of the reports incorporates everything they learned into
+every article the story affects. In our system: the news shows are
+**conversations**, the stories are **topics (chunks)**, and the **bucket is the
+editor's brain** — the place where everything reported TODAY about THAT TOPIC
+accumulates before any article is touched.
+
+**A bucket is keyed by the STORY/TOPIC, not by a subject.** The worked example:
+the topic is *"2026 Lincoln Memorial Reflecting Pool controversy"* (the algae
+issue in the pool that Trump renovated). Every report about it that day goes in
+that one bucket. There is **no article for that topic** — and there may never
+be — but the story affects THREE articles: *Lincoln Memorial Reflecting Pool*,
+*Second presidency of Donald Trump*, and *John J. Cafaro*. Each article
+receives the facet of the delta relevant to it (renovation/algae facts vs.
+abuse-of-power framing vs. the contract angle). This is why bucket ≠ article is
+**structural**: articles are THINGS (subjects); buckets are STORIES (what got
+discussed); the story→things relationship is inherently many-to-many.
+
+Assignment mechanics: group chunks by topic — normalize/compare chunk labels
+deterministically first, with a small-model binary judgment ("are these two
+chunks about the same story? yes/no") for near-misses, and a new-bucket
+proposal for genuinely new topics. The taxonomy is emergent: buckets come into
+existence because the user talked about something, never from a predefined
+list. Buckets are naturally short-lived relative to articles — a story
+accumulates while it's live and goes quiet when the user stops discussing it;
+the articles it fed persist.
+
+`_captured` is the reserved catch-all bucket: chunks whose subject doesn't yet
+justify assignment stay there, searchable, and are revisited on later runs —
+**nothing is dropped**.
+
+### RESOLVE — bucket deltas → articles/sections
+
+The unit of work from here on is the **bucket delta**: all of a bucket's new
+(un-synthesized) chunks since the last run, taken TOGETHER. For each non-empty
+delta, RESOLVE answers the editor's question: **"which articles does this story
+affect?"** (the Reflecting Pool story → the pool article, the presidency
+article, the contractor article). For each affected subject:
+
+- **The article exists** → UPDATE. Section-route the delta's *facet for that
+  article* via binary per-section judgments over the TOC ("does this new
+  information update THIS section? yes/no"); all-no → propose a new section.
+- **The article doesn't exist yet** → CREATE the subject article (a THING —
+  never an article named after the story itself). Thin is fine — 2–3 chunks
+  make a legitimate stub (stubs are a feature, not a failure).
+
+The per-section ledger (`synthesized_chunk_sections`) records which chunks each
+fold incorporated, making every fold idempotent and every claim traceable to
+its source chunks.
+
+### SYNTHESIZE — one edit per (bucket, article), never chunk-at-a-time
+
+**The cardinal rule:** synthesis is the editor sitting down ONCE per article
+with everything the bucket learned. Never patch a section from one chunk and
+then re-patch it from the next chunk five minutes later — sequential
+single-chunk edits overwrite each other, churn citations, and produce
+patchwork prose. The whole point of the bucket layer is that the delta arrives
+at the article **already aggregated**.
+
+- **CREATE**: group the bucket's chunks into proto-sections (by label/subject
+  affinity — small-model judgment, no embeddings) → outline (title, lead,
+  section names) → draft each section from only its chunks. The model never
+  holds a whole article in its head.
+- **UPDATE (patch)**: hand the model one section + the delta's facet relevant
+  to that article and section (all of it, with per-conversation `[^N]` citation
+  material); it rewrites that one section once, weaving everything in. Facet
+  extraction is implicit: the same story becomes algae/renovation facts in the
+  Reflecting Pool article and framing in the presidency article, because the
+  model writes what's relevant to the section it's editing.
+  Contradiction = correction: resolve toward the latest statement, including
+  contradictions *within* the delta (latest chunk wins — the editor heard all
+  seven reports and writes the reconciled version, not seven amendments).
+  User's evolving positions keep a `## History` trace; AI factual errors are
+  silently overwritten.
+- **Edit mechanics (SETTLED 2026-07-01, Josh-confirmed — and how Lucien always
+  worked): the model never edits the article; it is handed the current section
+  + the new information and outputs a complete REPLACEMENT SECTION.** Not an
+  in-place file edit (the
+  model would have to hold the whole article — the exact blob failure this
+  design avoids) and not a diff (small local models produce brittle patch
+  formats, and a diff is hard to gate). A full replacement section is bounded
+  in and out, trivially appliable (`extractSection` swap — the rest of the file
+  byte-identical by construction), and directly gateable: the gate compares old
+  section vs. new section for word floor and citation survival. Deterministic
+  code performs the actual write. CREATE works the same way — the model emits
+  one section at a time; code assembles the article.
+- **Multiple buckets converging on one article** (the algae-controversy bucket
+  AND the olympian-arrest bucket both affect *Reflecting Pool*): each bucket's
+  facet folds as its own section-level edit within the run — different stories
+  usually land in different sections. If two buckets touch the SAME section,
+  fold chronologically (latest wins, per self-healing) and let the nightly
+  EDITORIAL pass smooth the seam; the editorial pass runs after all folds, so
+  the article the user reads next morning is one coherent account.
+- **Conservative write gate** on every output: ≥70% word floor on the touched
+  section, every pre-existing `conv:` citation survives, H1/References survive,
+  footnote bijection. Weak output → NO-OP; the delta retries next run.
+
+### RECONCILE · LINK · NORMALIZE — nightly deterministic/structural stages
+
+These run **every night**, not periodically:
+
+- **RECONCILE** (`reconcile.ts`): the infobox is the article's ground truth;
+  lead, body sections, and infobox must agree; date-aware (latest wins).
+  Infobox extraction/refresh is part of this — every article carries a current,
+  parseable infobox whose entity-valued fields are `[[wikilinks]]`.
+- **LINK** (`crosslink.ts`): cross-linking is a dedicated stage, NOT
+  synthesis's job (synthesis sees one article and cannot know the others).
+  Mention-based prose links + co-occurrence See-also. Idempotent. Ensures **all
+  subjects are wikilinked** across the vault.
+- **NORMALIZE** (`normalize.ts`): pure code — footnote renumber/merge, wikilink
+  canonicalization, orphan repair, structure guards (`## References` last,
+  `{{stub}}`/`{{Main}}` recognition, title == H1 == stem).
+
+### EDITORIAL (wikify) — make the article good
+
+The final AI pass, and the reason articles stay readable instead of accreting
+patches: per-section "tighten this — remove redundancy, keep every citation and
+every specific detail" plus a structural pass over the TOC (reorder sections,
+refresh the lead, split a section that outgrew the article into its own article
+with a `{{Main}}` summary, merge duplicate stubs). Behind the same conservative
+gate; a weak edit is rejected and the original kept.
+
+## Nightly run contract
+
+- Trigger: launchd (`com.mlx-bun.memory`), logs under `~/.mlx-bun/logs/`.
+- Scope: only conversations past the per-source watermark — a normal night is
+  minutes of work, not hours.
+- Inference: one resident base model (Gemma-4-e4b), per-stage adapter mounted
+  iff `~/.cache/mlx-bun/adapters/memory-<stage>` exists, else base+policy.
+  Route generation through the persistent server (load-once) when available;
+  no-server fallback works but pays model load per invocation.
+- Every article change is a git commit; the run ends with a changelog summary.
+- Failure isolation: stages are resumable; a failed bucket/article never blocks
+  the rest of the run.
+
+## Decisions overridden from the master plan (explicit, so no agent re-derives them)
+
+| Master-plan statement | Superseded by |
+| --- | --- |
+| Personal-notability checklist; CREATE gate "≥3 chunks OR owned/decided, else PARK as trivia"; ≤40-article first-run cap (P1-T5, P5-T4, R5, P9-T3) | **No notability filter.** Everything surfaces; `_captured` keeps the tail searchable; thin buckets → stubs. |
+| Buckets demoted to "optional staging, may ship empty" (L4) | **Buckets are first-class**: the per-subject accumulation layer between chunks and articles; synthesis consumes buckets. |
+| Embeddings allowed as offline instruments (P4 recall tripwire experiments, P8-T4 silhouette triage) | **No embeddings anywhere in the project.** The tripwire (`getEmbedCounter()==0`) remains as a guard, permanently. |
+| Routing/section-route acceptance gated on Lucien bucket-F1 / Jaccard (P4-T2, P7-T1) | Model-selection evals live **outside** the system design; pipeline health is judged on real output quality, never Lucien's bin taxonomy. |
+| Cross-linking + infobox work assigned to periodic WIKIFY (P8) | RECONCILE + LINK are **nightly stages** in the DAG; editorial wikify is the readability pass. |
+
+## Work plan (docs-only reconciliation done; implementation next)
+
+Ordered; none of it touches the running full-corpus import
+(`~/.mlx-bun/wiki-full`, cursor file `~/.mlx-bun/full-run-cursor.txt`) or the
+uncommitted `src/model/universal` / trainer work.
+
+1. **Bucket layer made explicit** — a topic-keyed BUCKET stage between SEGMENT
+   and the article fan-out (`buckets` + `chunk_buckets` tables already exist):
+   group chunks by story (label normalization + binary "same story?"
+   judgments), `_captured` catch-all, and **delta-at-once synthesis** — the run
+   collects each bucket's new chunks, then makes ONE edit per (bucket, article).
+   The as-built entity-keyed ROUTE becomes RESOLVE (bucket → affected subject
+   articles); the per-chunk patch loop is replaced, not wrapped. (Settled
+   2026-07-01 — the Wikipedia-editor / Reflecting-Pool framing above; buckets
+   are STORIES, articles are THINGS, many-to-many by nature.)
+2. **Nightly DAG order** — ensure `mlx-bun memory synthesize` runs
+   segment → bucket → resolve → synthesize → reconcile → link → normalize →
+   editorial every run (reconcile/link/infobox currently periodic/partial).
+3. **Editorial stage** — wire the per-section improve + structural pass as the
+   closing stage behind the conservative gate.
+4. **Persistent-server inference routing** — load-once for all memory
+   generation, no-server fallback (the in-progress inference-path rework;
+   finish, don't fork).
+5. **Promote + schedule** — after the import is judged good: point
+   `~/.mlx-bun/wiki` at the bootstrapped content, enable the nightly launchd
+   fold-in from pi sessions.
+6. **Known quality fixes** — schematic-prompt leaks (<1% of articles), run-on
+   leads, `owned:` framing on abstract concepts (reframe the lead spec per
+   subject kind).
+
+## Open questions (for discussion with Josh)
+
+- ~~Bucket granularity vs article granularity~~ — **SETTLED 2026-07-01**:
+  buckets are a separate layer from articles (many-to-many both ways); the
+  bucket is the editor's brain that aggregates the day's information about a
+  subject, and RESOLVE fans the delta out to every affected article. Bucket is
+  never a 1:1 alias for an article.
+- **When does a `_captured` chunk graduate?** Re-scan `_captured` every night
+  for subjects that have since accumulated enough related chunks, or only when
+  a new bucket's name matches?
+- **Editorial cadence.** Every night over every touched article, or touched
+  articles nightly + a slower full-vault sweep?