mlx-bun 0.0.13 → 0.3.0

package/docs/design/the-dreaming-master-plan.md

@@ -1,445 +0,0 @@
-# THE DREAMING (LOCAL) — MASTER BUILD PLAN
-
-> **PARTIALLY SUPERSEDED (2026-07-01).** The canonical system design is now
-> [dreaming-nightly-pipeline.md](./dreaming-nightly-pipeline.md); its
-> "Decisions overridden" table lists exactly which statements here are dead.
-> Do NOT execute from this doc alone. In particular: there is **NO notability
-> filter / CREATE gate / trivia PARK** (everything surfaces; `_captured` holds
-> the tail), **buckets are first-class** (not "optional staging, may ship
-> empty"), **no embeddings anywhere** (not even as offline instruments), and
-> the Lucien bucket-F1/Jaccard acceptance gates are model-eval tooling, not
-> system design. The per-task file/port references below remain useful.
-
-*Turn a user's conversations into Wikipedia-style entity articles using local models, cheaply and privately. This document is the complete execution spec: a fresh agent reading ONLY this document can execute every task end-to-end without asking a question. Every decision is made; where a real choice existed it is PICKED with a one-line rationale + fallback. All paths are absolute. All model/adapter/schema strings are exact and verified against the live repo and disk on the build machine (M4 Pro / M1 Max).*
-
----
-
-## PREAMBLE
-
-### Goal
-Build a local "personal Wikipedia" (codename **the Dreaming**) inside `mlx-bun` (`src/memory/`) that synthesizes a user's (Josh's) chat history into **entity articles** and lets a local chat agent answer questions by **small, focused reads** — never a context dump. The cloud original is **Lucien** (`/Users/joshrossi/Code/lucien`, live articles in `~/Dreaming`, ground-truth DB `/Users/joshrossi/Code/lucien/.lucien/lucien.db`). The bar: read one article and get up to speed on a thing, the way Wikipedia works.
-
-### Settled principles (FINAL — do not re-derive or contradict)
-- **Articles are THINGS (nouns/entities):** `Panasonic_Lumix_S5IIX`, `L-Mount`, `Sigma_150-600`, `Photography`. Every article = "<the thing> AND how the user relates to it." Write the RELATIONSHIP (why chosen, how used, opinions, plans); LINK OUT to real Wikipedia for general world-facts.
-- **Notability is personal** (the ONE deviation from Wikipedia): a thing is article-worthy because the user engaged with it — owned / used / decided-on / recurring (≥2 conversations) / holds a substantive opinion. This is the only deviation; otherwise **do whatever Wikipedia does**.
-- **NO VECTOR SEARCH on the routing/read hot path.** "About lenses" = a CATEGORY / TYPE / TITLE lookup (deterministic, zero embeddings). Embeddings are at most an OFFLINE TEST INSTRUMENT, never the router. The recall experiment's job was to PROVE topic-bins are wrong (~67% top-12 recall on broad bins — an *indicative* prior-session figure with no committed source in `reports/`/`scripts/`; it is RE-MEASURED live in P4-T2, so nothing gates on the literal), which it did.
-- **Capacity by decomposition:** a small local model's capacity scales inversely with task size. Decompose every stage to the smallest single-purpose task, ideally BINARY ("does this chunk update this section? yes/no"), run M×N. Affordable because local inference is ~free. Coarse gates first (hierarchical pruning: "does this relate to this article?" gates the section checks).
-- **Small focused units, never a dump — read AND write.** The read path hands the model one short focused thing per hop (one article, one section, one infobox). The write path emits small focused units. No giant blobs in either direction.
-- **Self-healing through ordinary use.** The conversation IS the input — the user's in-conversation CORRECTIONS are normal chunks like any other. When a new chunk CONTRADICTS an existing article claim, synthesis updates the article TOWARD the user's latest/corrected position (Bayesian-style — resolve toward the most recent statement; not append-both, not ignore). So the wiki heals itself with no manual editing. Example: the AI reads an article and says *"you love 35mm for photography"*; the user replies *"it's fine but nowhere near as good as 50mm"*; that correction is an ordinary chunk that folds in via the nightly synthesis; the next day the AI reads the updated article and says *"how great is that 50mm."*
-- **AI does JUDGMENT; deterministic CODE enforces invariants** (footnote bijection, wikilink canonicalization, infobox-traces-to-cited-claim, ≥70% word floor, weak output → NO-OP). "LoRAs propose, gates dispose."
-- **Entities (do not conflate):** **chunk** (a conversation segment) → **bucket** (optional staging group, may ship empty) → **article** (a THING) → **section**. Many-to-many at every layer. bucket ≠ article.
-  - **L4 stated deviation (intentional, not drift):** this plan demotes *bucket* from Lucien's "a group of chunks about one topic" to "OPTIONAL staging only, may ship empty." Articles, not buckets, are the synthesis target here. A future reader should treat this redefinition as a deliberate design choice recorded in this doc, not an inconsistency to repair.
-- **Vault = truth, DB = derived index.** Markdown in the vault is canonical and git-tracked; every DB table is either *pipeline state* or a *derived cache* rebuildable from the markdown by a `reindex` pass.
-
-### The entity data model
-- **chunk**: a pointer range `(conv, start, end)` + a 4–10-word `label`. No text copy.
-- **entity**: a canonical name (== article H1 == filename stem with `_`→space), with aliases and a `kind ∈ {thing, person, domain, project, standard}`.
-- **article**: one markdown file per entity; fixed skeleton (H1 → series banner → infobox → lead → sections → See also → Notes → References).
-- **infobox**: typed key-value facts that the AI READS as part of the article's content (like a Wikipedia infobox), never a query/filter engine. **world-facts** (`type`, `mount`, `focal_length`, …) trace to citable external facts; **relationship-facts** (`owned`, `acquired`, `used_for`, `chosen_over`, `opinion`, …) are the personal-notability layer and must trace to a cited body claim. A value naming another entity is a `[[wikilink]]` whose only structural role is to feed the link graph for NAVIGATION — there is no numeric-facet query over infobox fields.
-- **category**: a Wikipedia-style grouping of things, **emergent** (created by judgment as articles are written, governed by the editable `Category_Definitions.md`; the wiki starts with zero). Membership is **explicit** — the article declares `[[Category:Name]]`. "About lenses" = `Category:Lenses` membership (deterministic lookup over whatever categories exist).
-
-### North-star acceptance test (design backward from this)
-User asks the chat AI: *"what's the best lens for really long reach that isn't crazy expensive?"* The local agent SILENTLY (like a colleague — no "per the wiki…") FINDS the one relevant article — by category ("articles about lenses/telephoto"), name, or a `[[link]]` from Josh's camera article — then READS it (its prose AND its infobox, as content) and answers from what it read. The article already holds the user's CURRENT POSITION on the topic — the synthesized conclusion he reached before (e.g. "you already shoot the 75-300, so for real reach the Sigma 150-600 is the sane pick") — so the answer is just reporting back what the article says. It is **find → read → answer** in SMALL reads: NOT a faceted filter that computes "Sigma 150-600" by ordering lenses on a price field, NOT a 90-article substring dump, NOT a 671-line blob.
-
-**Where the work is.** The read path is DELIBERATELY TRIVIAL — find the right article (category / name / link), read it, answer — exactly like reading one Wikipedia page. All the complexity, machinery, and acceptance pressure live on the WRITE side: synthesis keeping each article a good, current, well-shaped account of the user's relationship to the thing. The thesis is proved by the WRITE→READ round-trip: fold a new chunk (e.g. "I bought the 70-300") → it patches the affected article(s) → the same simple read now returns the evolved answer.
-
-### Trace evidence — the bar to beat (from `/Users/joshrossi/Downloads/lucien_examples/*.md` and `.lucien/lucien.db`)
-- Context burn **33–150 KB PER QUERY** ("list my lenses" = 150 KB). Fatal locally.
-- Substring search is junk: "3D print" → ~40 articles incl. `Mass_Driver_Space_Launch`, `Toyota_Production_System`, `npm_CLI`; "lens" → 90 incl. "the Toyota lens". Root cause: no categories/types.
-- The mega-blob `Camera_Gear_and_Lenses.md` (671 lines) is read/sectioned up to **42×** in ONE query.
-- Links are rarely traversed even on multi-hop queries; search+blob "works" only because the blob pre-contains everything at max context.
-- Taxonomy diagnosis: of 385 `~/Dreaming` articles, **0** are named for a thing Josh owns; **245** are `X_and_Y` topic-bins; **0** infoboxes / 0 series / 0 `{{Main}}`. The lone entity-shaped cluster (Archie + summary-style children) PROVES the entity pattern emerges when a topic is engaged deeply.
-- Corpus size (DESCRIPTIVE, not a pass/fail constant — see H5): ~2063 conv / ~45179 msg / ~7316 chunks / ~8809 chunk_buckets edges / ~379 buckets; ~83.4% single-edge chunks (many-to-many is real but thin → score routing by **Jaccard**, not exact-match). Bucket size bins: 1→25, 2-4→97, 5-20→167, 21-100→78, 100+→12 (122 buckets <5 = the tail where shortlist recall collapses). **Every count below that names a corpus number is illustrative; all acceptance gates compare against `SELECT COUNT(*)` from the live source, never the literal.**
-
-**Acceptance budget to beat:** answer the north-star query locally, silently, in **< 5 KB mean context** (8 KB hard ceiling), **zero embedding calls**, no single read > 2 KB.
-
-### Conventions used below
-Each TASK: **id** · one-line description · **DELIVERABLE** (file/function/artifact) · **ACCEPTANCE** (executable pass/fail) · **DEPENDS-ON** (task ids) · **TAG** (NOW / USER-ACTION / GPU-GATED). All paths absolute. Production code → `src/memory/`; one-off probes → `scripts/experiments/`; eval/bootstrap scripts → `scripts/memory/` (**create this directory in P0-T7 — it does not exist yet**). The whole repo (incl. `scripts/experiments`) must `tsc` to 0 — no tolerated baseline.
-
-### Lucien port-source map (load-bearing — verified on disk)
-Every "port Lucien X" target lives under **`/Users/joshrossi/Code/lucien/scripts/`** (NOT `src/`; `lucien/src/` holds only `index.ts` + `tools/`). Cite these exact files/symbols:
-- `scripts/meta-inline.ts` → `loadMetaPolicyBlock(...)` (P0-T4)
-- `scripts/chunk-recent.ts` → `CHUNK_PROMPT`; `scripts/chunk-validation.ts` → anchor repair (P5-T2)
-- `scripts/wikify.ts` → `verifyEditorialResult` / `checkFootnoteIntegrity` / `extractConvHashes` (P6-T1) and `buildEditorialPrompt(articleText)` (an exported FUNCTION wrapping the module-private `EDITORIAL_PROMPT` const; P8-T1)
-- `scripts/normalize-footnotes.ts` + `scripts/normalize-wikilinks.ts` (P6-T2)
-- `scripts/synthesize.ts` → `SYNTHESIS_PROMPT_BOOTSTRAP` (P6-T3 / P7-T2)
-- `scripts/cluster-assign-recent.ts` → exports `ASSIGN_OR_PROPOSE_PROMPT` + `buildPrompt(...)` — repurposed for section-routing (P7-T1; there is NO `bucket-assign` file)
-- `lucien/src/index.ts:28-38` → `LUCIEN_INSTRUCTIONS` (P3-T5; the one source that IS under `src/`)
-
-### THE LOCAL-MODEL FACTS (load-bearing — verified against source, do not guess)
-- **Base model (every stage).** There is **ONE** base model in this pipeline: **`mlx-community/gemma-4-e4b-it-OptiQ-4bit`** ("Gemma-4-e4b"). It was trained overnight via the segmented-backward path and found surprisingly strong. The earlier **MiniCPM5-1B pivot is ABANDONED** (poor even after training) — there is **NO MiniCPM5 model anywhere in the Dreaming pipeline**; any `cpm-chunk-*` / `orpo-cpm5-*` adapter on disk is unrelated legacy and MUST NOT be referenced.
-- **`MODEL_ID` resolution rule — reuse the repo's `SNAPSHOT_E4B`.** Resolve the base snapshot exactly as `tests/paths.ts hfSnapshot()` / the `SNAPSHOT_E4B` export (`tests/paths.ts:59`) does: iterate the snapshots and return the first one CONTAINING `config.json`. (Note: TWO e4b snapshots exist on disk and BOTH carry `config.json`, so this loop returns `readdirSync`-order-first — it does NOT uniquely disambiguate; that is acceptable here only because both snapshots share bit-identical text weights, so either resolves to the same base. Don't claim this is "exact/machine-independent" — just defer to `SNAPSHOT_E4B` so the whole plan uses one resolver.)
-  ```ts
-  import { readdirSync, existsSync } from "node:fs";
-  const base = `${process.env.HOME}/.cache/huggingface/hub/models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots`;
-  let MODEL_ID = "";
-  for (const snap of readdirSync(base)) { if (existsSync(`${base}/${snap}/config.json`)) { MODEL_ID = `${base}/${snap}`; break; } }
-  ```
-- **The model seam.** `loadTaskModel(query, adapterDir)` from `src/eval/runner.ts` takes a **model selector as its FIRST arg** (a dir or registry id → `resolveModelDir`), NOT a prompt. The prompt goes to **`generateText(taskModel, body, opts)`**. So `callLocal` calls `loadTaskModel(MODEL_ID, adapterDir)` once (cached), then `generateText(tm, prompt, {maxTokens, useChat:true})`. `generateText` already does the bit-exact render→encode→trim — **never re-implement prompt→ids**. Adapter activation: `loadTaskModel` mounts AND activates the adapter (`model.loraState.active`); `generateText` re-passes `activeAdapters` each call.
-- **The SEGMENT (chunking) adapter — real on-disk path.** The validated Gemma-4-e4b chunk SFT adapter is **`~/.cache/mlx-bun/adapters/e4b-chunk-300`** (the sweet-spot checkpoint). `orpo-e4b-chunk` is the parent run and `e4b-chunk-400` over-merged — use neither. P5-T2 symlinks `e4b-chunk-300` to `~/.cache/mlx-bun/adapters/memory-chunk` so `callLocal("chunk", …)` finds it by the `memory-<stage>` convention. The `cpm-chunk-*` / `orpo-cpm5-chunk` dirs are the abandoned MiniCPM5 runs — ignore them.
-- **LoRA recipe (all stages).** `scripts/chunk-finetune.ts` (sft, rank 16, lr 1e-5, seq 8192, `ops.sdpa` not flash; no env-flag sanitization needed — the old no-vjp kernels were deleted 2026-07-05). Checkpoints picked by the task judge, **NOT val loss**.
-- **`callLocal` stage→adapter rule.** `callLocal(stage, prompt)` selects `~/.cache/mlx-bun/adapters/memory-<stage>` **iff that dir exists**, else falls back to base `MODEL_ID` (no adapter). So a stage runs base+policy until its LoRA is trained and symlinked in — the whole plan is executable base-only through M4.
-
-### MODEL/DATA PRECHECK (USER-ACTION fallbacks — agent must NOT download multi-GB or start GPU jobs)
-- **Gemma-4-e4b base present?** `test -d ~/.cache/huggingface/hub/models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots && ls ~/.cache/huggingface/hub/models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots`. If MISSING → **USER-ACTION**: `HF_HUB_DISABLE_XET=1 hf download mlx-community/gemma-4-e4b-it-OptiQ-4bit`.
-- **Embedding model present (P4 instrument only)?** `ls ~/.cache/huggingface/hub | grep Qwen3-Embedding`. Expected: `models--mlx-community--Qwen3-Embedding-4B-4bit-DWQ`. If MISSING → **USER-ACTION**: `HF_HUB_DISABLE_XET=1 hf download mlx-community/Qwen3-Embedding-4B-4bit-DWQ`.
-- **`resolveModelDir`/`resolveModel` THROW if the snapshot is absent** — every task that calls a model must run the precheck first and, on miss, STOP with the USER-ACTION command (never auto-download).
-
-### Output-parsing contracts (H1 — bake once, used by every model stage)
-In `src/memory/parse.ts`:
-- `parseBinary(out: string): boolean` — `return /^\s*y/i.test(out);` (anything not starting with y/Y → `false` → NO-OP). Binary stages call `generateText(tm, prompt, {maxTokens: 4})`.
-- `parseLines(out: string): string[]` — split on `\n`, `trim`, drop empties and any line equal (casefold) to `NONE`. List stages call `generateText(tm, prompt, {maxTokens: 128})`.
-- A malformed/over-long generation is truncated by `maxTokens`; `parseBinary` defaults it to `false`, `parseLines` drops it. No stage ever blocks on bad output.
-
-### Vault path decision
-The vault lives at `~/.mlx-bun/wiki/` (articles in `~/.mlx-bun/wiki/articles/`, policy in `~/.mlx-bun/wiki/Meta/`). Rationale: matches the landed M0 `setupVault` store; `~/Dreaming` stays read-only as eval oracle. Fallback if a clean fixture is needed: `tests/fixtures/wiki/`. **Fixture/test redirect:** the read tools resolve the vault via `vaultRoot()`, which honors the **`MLX_BUN_WIKI` env override** (`vault.ts:26`); P3-T1/P3-T5/P9-T2 point the read path at their fixture vault by setting `MLX_BUN_WIKI=<fixture dir>` (no code change, no writing the real `~/.mlx-bun/wiki`).
-
-### Migration decision (made, not deferred): REBUILD FRESH from the chunk corpus; do NOT convert `~/Dreaming` markdown.
-Rationale: 245/385 articles are wrong-shaped topic-bins with 0 infoboxes — converting = re-deriving entity granularity/infoboxes/links from prose anyway, with the bin taxonomy fighting you; the clean source (~7316 pointer chunks + ~8809 edges in `.lucien/lucien.db`) is the pipeline's native input. `~/Dreaming` is preserved untouched as the eval oracle and mined only for the gold entity vocabulary. Fallback: none needed — the DB is a rebuildable cache, `~/Dreaming` is never written.
-
-### Lucien source schema (B9 — inlined; verified via `sqlite3 .lucien/lucien.db .schema`)
-The bootstrap reads these tables READ-ONLY. **Note the column names differ from mlx-bun's** (mapping baked into P5-T1):
-```
-conversations(uuid PK, name, summary, created_at TEXT, updated_at TEXT, message_count INT, source TEXT)
-messages(uuid PK, conversation_uuid, position INT, sender, text, timestamp TEXT, parent_message_uuid)   -- INDEX(conversation_uuid, position)
-chunks(id INTEGER PK AUTOINCREMENT, conversation_uuid, start_message_uuid, end_message_uuid, label)       -- chunks point at MESSAGE UUIDs, not positions
-buckets(name PK, description)
-chunk_buckets(chunk_id, bucket_name, PK(chunk_id,bucket_name))    -- the "~8809 edges"; reinterpreted as entity edges in P4-T2
-chunked_conversations(conversation_uuid PK, chunked_at TEXT, status)
-synthesized_bucket_chunks(bucket_name, chunk_id, synthesized_at, PK(bucket_name,chunk_id))
-```
-There is **no `edges` table and no `entities` table** in Lucien — entity gold is MINED from `~/Dreaming` markdown (P0-T5), and "edges" means `chunk_buckets`. mlx-bun `chunks` use integer `(start,end)` POSITIONS; Lucien `chunks` use `(start_message_uuid,end_message_uuid)` — P5-T1 maps UUID→position via `messages(conversation_uuid, position)`.
-
-**`(chunk, section_before, section_after)` triple-recovery rule (B9 — explicit, used by P7-T2 / P10-T1).** Recover the chunk's identity and its article from Lucien's STRUCTURED tables, NOT by parsing footnote hashes — a `conv:HASH` carries only the conversation (see the citation format in P1-T1), never a chunk range, so it cannot identify one chunk of the many in a conversation. Source of record (verified via `sqlite3 /Users/joshrossi/Code/lucien/.lucien/lucien.db .schema`):
-- **(chunk → article):** `chunks(id, conversation_uuid, start_message_uuid, end_message_uuid, label)` ⋈ `chunk_buckets(chunk_id, bucket_name)` ⋈ the bucket→article-stem mapping (the bucket that became `articles/<stem>.md`); `synthesized_bucket_chunks(bucket_name, chunk_id, synthesized_at)` is the ledger of which chunks were actually integrated, and when.
-- **(chunk text/span):** the chunk's `start_message_uuid`/`end_message_uuid` resolved over `messages` (the P5-T1 UUID→position map) — never copied from the article prose.
-- **(section_before, section_after):** from the article's GIT HISTORY. Run in a clone/worktree of `~/Dreaming` at its frozen SHA; for each commit in `git log --reverse` touching `articles/<stem>.md` (or `<stem>.md`) that ADDED a `conv:` footnote-def (the presence of an added `[^N]:` footnote merely flags an *integration* commit vs a pure normalize/rename commit, which is skipped), the diff hunk's enclosing `##`/`###` heading is the target **section**, `section_before` = that section's body at the commit's parent (`git show <parent>:<path>` → `extractSection`), `section_after` = its body at the commit.
-- **(pairing):** align the article's integration commits (in `git log` order) with its `synthesized_bucket_chunks` rows for that bucket (ordered by `synthesized_at`) — each integration commit ↔ one integrated chunk row. The footnote only marks *that* a commit integrated a chunk; the chunk's IDENTITY and span come from the structured `chunks` row, never from the hash.
-Each surviving `(chunk, section_before, section_after)` triple is one SFT example.
-
-### Training-data schema (B4 — exact, verified against `src/train/dataset.ts sftRowFormat` + Lucien `train.jsonl`)
-The trainer (`finetuneRunner`) reads `${DATA}/train.jsonl` + `${DATA}/valid.jsonl`, one JSON object per line, format auto-probed by `probeFormat`. Accepted shapes: `{messages:[…]}`, `{prompt,completion}`, or `{text}`. **Use the `messages` shape** (matches the existing chunk dataset):
-```json
-{"messages":[{"role":"system","content":"<stage system prompt + inlined Meta policy>"},{"role":"user","content":"<stage input>"},{"role":"assistant","content":"<gold output>"}]}
-```
-Loss is masked to the assistant turn (`promptLen`). Per-stage field mapping:
-- **synthesis (P10-T1):** user = current section markdown + the integrating chunk text (`chunkText`) + footnote contract; assistant = the gold `section_after` (the integrated section + new `[^N]:` lines), from the triple.
-- **section (P10-T2):** user = article title + section heading + section gist + chunk label/gist; assistant = `yes`/`no` (one token) from the gold `chunk_sections` edge.
-- **entity (P10-T3):** user = chunk text; assistant = newline-listed gold entity names for that chunk.
-- **editor (P10-T4):** user = section markdown; assistant = the tightened/infobox-extracted gold output.
-
----
-
-## P0 — FOUNDATION: pointer+entity data model, model seam, parser, instruments
-**Goal:** Replace the M0 copy-schema with the pointer+entity schema and the shared deterministic parser everything indexes through, plus the cross-cutting instruments (gold vocab, embedding tripwire, parse contracts, `scripts/memory/`). Nothing else builds until the data model is right.
-
-**P0-T1 · Local-model call seam** — TAG: NOW · DEPENDS-ON: none
-- DELIVERABLE: `src/memory/model.ts` exporting `callLocal(stage: string, prompt: string, opts?: {maxTokens?: number}): Promise<string>`. It resolves `MODEL_ID` by the rule in PREAMBLE (iterate e4b snapshots, pick the one with `config.json`), runs the MODEL precheck (STOP with the USER-ACTION `hf download` line if absent), lazily `loadTaskModel(MODEL_ID, adapterDir)` (cached per `(stage, adapterPresent)` — so symlinking an adapter in/out yields a fresh mount rather than a stale cache hit, which the adapter-flip acceptance below requires), where `adapterDir = ~/.cache/mlx-bun/adapters/memory-${stage}` **iff it exists**, else `undefined` (base). Calls `generateText(tm, prompt, {maxTokens: opts?.maxTokens ?? 256, useChat: true})`. Do NOT re-implement prompt→ids.
-- ACCEPTANCE: `callLocal("entity", "List the things: I bought a Lumix S5IIX")` returns base-model text in-process; symlinking `~/.cache/mlx-bun/adapters/memory-entity → ~/.cache/mlx-bun/adapters/e4b-chunk-300` and re-calling flips `tm.activeAdapters`/`loraState.active` and changes output on a fixed probe prompt; removing the symlink reverts to base.
-
-**P0-T2 · Shared fence-aware parser/serializer** — TAG: NOW · DEPENDS-ON: none
-- DELIVERABLE: `src/memory/article.ts` with pure functions: `parseInfobox(md)→{type, fields:{key,value,isEntityLink,kind}[]}`, `serializeInfobox(...)`, `parseSeriesBanner(md)`, `parseLead(md)`, `parseFootnotes(md)→{markers,defs}`, `articleStructure(md)→ordered skeleton with line ranges`. Factor the fence scanner out of `src/memory/vault.ts` so this module and `vault.ts` (`parseToc`/`extractSection`/`searchArticles`) share one fence implementation. Also export the ONE fixed structural constant `ENTITY_KINDS` (the literal `["thing","person","domain","project","standard"] as const`; full rationale in P1-T4c) and read the entity kind from the explicit infobox `kind:` field (validate against `ENTITY_KINDS`, default `thing`) — there is NO `kindOf(type)` map and NO category constant; both `type:` labels and categories are emergent (declared per-article).
-- ACCEPTANCE: `tests/memory-article.test.ts` builds **synthetic inline fixtures** (NOT the P1-T1 golden — this task has no P1 dependency): parse→serialize a hand-written infobox → byte-identical; extracts a declared `aliases:` list exactly; classifies each value's `isEntityLink` correctly; `parseFootnotes` yields a bijection on a well-formed fixture and flags an injected dangling `[^9]`. `bun test tests/memory-article.test.ts` green. *(The golden `Panasonic_Lumix_S5IIX.md` round-trip is asserted in P1-T1, which depends on this parser.)*
-
-**P0-T3 · Pointer+entity DB rewrite** — TAG: NOW · DEPENDS-ON: none
-- DELIVERABLE: Rewrite `src/memory/db.ts` `MemoryStore` SCHEMA + `migrate()`. **Legacy-DB rule (H8): on open, if a legacy column exists (`PRAGMA table_info(chunks)` contains `text`, or table `synthesized_bucket_chunks` exists), DROP ALL tables and recreate from scratch — there is no production data to preserve (the DB is a rebuildable cache; the vault is truth).** Delete `conversations.transcript`, `chunks.text`, `chunks.bucket`, `synthesized_bucket_chunks`. Add `PRAGMA foreign_keys = ON` + `ON DELETE CASCADE` everywhere. Tables (exact DDL):
-  - `conversations(conv TEXT PK, source TEXT, title TEXT DEFAULT '', updated_at INT, chunked_at INT)`
-  - `messages(conv TEXT REFERENCES conversations(conv) ON DELETE CASCADE, position INT, role TEXT, uuid TEXT, text TEXT, PRIMARY KEY(conv,position))` — text materialized here exactly once.
-  - `chunks(id TEXT PK, conv TEXT REFERENCES conversations(conv) ON DELETE CASCADE, start INT, end INT, label TEXT)` + `idx_chunks_conv`. **`id` derivation (fixed, used by P5-T1 and every downstream idempotency table):** `id = \`${conv}:${start}-${end}\`` (e.g. `abc123:4-9`) — deterministic from the pointer range, so re-ingesting the same chunk yields the same id (idempotent).
-  - `entities(name TEXT PK, article_stem TEXT, kind TEXT, notable INT DEFAULT 0)`
-  - `entity_aliases(alias TEXT, entity_name TEXT REFERENCES entities(name) ON DELETE CASCADE, PRIMARY KEY(alias,entity_name))` + `idx_aliases_alias`.
-  - `chunk_entities(chunk_id TEXT REFERENCES chunks(id) ON DELETE CASCADE, entity_name TEXT REFERENCES entities(name) ON DELETE CASCADE, surface_form TEXT, PRIMARY KEY(chunk_id,entity_name))`
-  - `categories(name TEXT PK, definition TEXT DEFAULT '', parent TEXT REFERENCES categories(name) ON DELETE SET NULL)`
-  - `article_categories(article_stem TEXT, category TEXT REFERENCES categories(name) ON DELETE CASCADE, source TEXT, PRIMARY KEY(article_stem,category))` + `idx_artcat_category`.
-  - `infobox_facts(article_stem TEXT, key TEXT, value TEXT, kind TEXT, entity_link TEXT, PRIMARY KEY(article_stem,key,value))` + `idx_infobox_key`, `idx_infobox_link`.
-  - `links(src_stem TEXT, dst_stem TEXT, via TEXT, PRIMARY KEY(src_stem,dst_stem,via))` + `idx_links_dst`. `via ∈ {prose,infobox,seealso,series,main}`.
-  - `chunk_sections(chunk_id TEXT REFERENCES chunks(id) ON DELETE CASCADE, article_stem TEXT, section_anchor TEXT, PRIMARY KEY(chunk_id,article_stem,section_anchor))`
-  - `synthesized_chunk_sections(chunk_id TEXT REFERENCES chunks(id) ON DELETE CASCADE, article_stem TEXT, section_anchor TEXT, synthesized_at INT, PRIMARY KEY(chunk_id,article_stem,section_anchor))`
-  - `watermarks(source TEXT PK, cursor TEXT, updated_at INT)`
-  - `buckets(name TEXT PK, article_stem TEXT, description TEXT DEFAULT '')` + `chunk_buckets(chunk_id TEXT REFERENCES chunks(id) ON DELETE CASCADE, bucket TEXT REFERENCES buckets(name) ON DELETE CASCADE, PRIMARY KEY(chunk_id,bucket))` — optional staging, may ship empty (L4).
-  Add helper `chunkText(id)` joining `messages` on `(conv, position BETWEEN start AND end)`. Update `ChunkRow`/`ConversationRow`/add `EntityRow`. Document the contract at the top of `db.ts`: *derived tables (entities, entity_aliases, categories, article_categories, infobox_facts, links) are a cache of the markdown; pipeline tables (messages, chunks, chunk_*, synthesized_*, watermarks) are the synthesis ledger; `reindex` makes the former match the vault, never the reverse.*
-- ACCEPTANCE: `tests/memory-db.test.ts` (in-memory): opening a DB carrying a legacy `chunks.text` column drops+recreates cleanly; insert a 6-message conv + 2 chunks → total text bytes == sum(message bytes) (no duplication); `chunkText` reassembles the right slice; `DELETE FROM conversations` cascades messages/chunks/edges to zero rows; `PRAGMA foreign_keys` reads 1. `bun test tests/memory-db.test.ts` green.
-
-**P0-T4 · Meta-policy inliner** — TAG: NOW · DEPENDS-ON: none
-- DELIVERABLE: Port Lucien `scripts/meta-inline.ts loadMetaPolicyBlock()` into `src/memory/prompts.ts` as `loadMetaPolicy(names: string[]): string` reading `~/.mlx-bun/wiki/Meta/*.md` and concatenating them for prompt inlining. Policy lives in the vault, not in code. If a requested Meta page is missing on disk, throw a clear error naming the missing file (so a stage never silently loses its policy — guards B10).
-- ACCEPTANCE: a unit test writes a Meta page, edits one line, and asserts the rendered policy block reflects the edit with no code change; requesting a non-existent Meta page throws naming it.
-
-**P0-T5 · Gold-entity vocabulary miner** — TAG: NOW · DEPENDS-ON: none *(promoted early per B5 — no model, reads `~/Dreaming` markdown directly; consumed by P4-T1, P4-T2, P5-T3)*
-- DELIVERABLE: `scripts/memory/mine-gold-entities.ts` → writes `goldens/entities.json`. Mines `~/Dreaming/articles/*.md` titles (the 385 article files live there, NOT at `~/Dreaming/*.md` which holds only a handful of top-level files) — stem `_`→space — + bolded lead phrases (`**X**` in sentence 1) + `~/Dreaming/Meta/Buckets.md` bucket names, normalizing (casefold, ws-collapse, strip leading articles). Output schema: `{entities: [{name, aliases: string[], sourceTitle}], builtFromSha}` where `builtFromSha` = `git -C ~/Dreaming rev-parse HEAD` (freezes the oracle).
-- ACCEPTANCE: `goldens/entities.json` exists with **≥1 title-derived entry per `~/Dreaming/articles/*.md` file** (RELATIVE gate: `entities.filter(e=>e.sourceTitle).length == $(ls ~/Dreaming/articles/*.md | wc -l)`, currently ~385 — never a hardcoded literal), every entry has a non-empty `name`; re-running over the same SHA is byte-identical; `S5IIX`/`S5 IIX`/`Lumix S5IIX` collapse into one entry's `aliases`.
-
-**P0-T6 · Embedding tripwire instrument** — TAG: NOW · DEPENDS-ON: none *(B7 — the counter R8/P9-T2 assert ==0)*
-- DELIVERABLE: In `src/embed.ts`, add a module-level `let embedCallCount = 0`, increment it once per call at the top of `embedOne` (so `embedMany` accrues via `embedOne`), and export `resetEmbedCounter(): void` (`embedCallCount = 0`) and `getEmbedCounter(): number`. No behavior change to vectors (parity test must still pass).
-- ACCEPTANCE: `tests/embed-tripwire.test.ts`: `resetEmbedCounter(); embedMany(model,tok,["a","b","c"]); getEmbedCounter() === 3`; `tests/qwen3-embed-parity.test.ts` still green (no vector change).
-
-**P0-T7 · `scripts/memory/` directory + parse contracts** — TAG: NOW · DEPENDS-ON: none *(L3 + H1)*
-- DELIVERABLE: `mkdir -p scripts/memory` with a `.gitkeep`; `src/memory/parse.ts` exporting `parseBinary`/`parseLines` per the PREAMBLE contracts.
-- ACCEPTANCE: `tests/memory-parse.test.ts`: `parseBinary("Yes.")===true`, `parseBinary("no")===false`, `parseBinary("maybe")===false`; `parseLines("A\n\nNONE\n B ")` → `["A","B"]`. `scripts/memory/` exists in git.
-
----
-
-## P1 — TAXONOMY & MARKDOWN SHAPE: the on-disk grammar + categories + notability
-**Goal:** Define the article grammar, the EMERGENT infobox-schema grammar (governed like categories — no closed type list), the emergent category mechanism, and the personal-notability checklist — the foundation the parser, indexer, and writer all agree on. (Meta-page authoring is parallelizable with P0.) **Every task here that authors a NEW Meta page MUST also register it in the `META_PAGES` seed map in `src/memory/vault.ts` (currently at `vault.ts:537`, keyed `"<File>.md": <CONST>`), so `memory init`/`setupVault` actually writes it and `loadMetaPolicy` can read it (B10).**
-
-**P1-T1 · Canonical article skeleton spec + golden fixture** — TAG: NOW · DEPENDS-ON: P0-T2
-- DELIVERABLE: Extend `~/.mlx-bun/wiki/Meta/Article_Conventions.md` (and update the existing `vault.ts` `ARTICLE_CONVENTIONS` seed constant — already in `META_PAGES`) with the fixed top-to-bottom grammar: (1) `# H1` == filename stem with `_`→space; (2) optional series banner `*Part of a series on [[Lenses]].*`; (3) optional infobox fenced block; (4) lead = 2–4 sentence relationship abstract, present tense, names the thing in bold in sentence 1; (5) `##`/`###` sections in reading order; (6) `## See also` wikilink bullets; (7) optional `## Notes` (`[^N]` commentary); (8) `## References` (`[^N]:` defs), the LAST section. Hand-author the golden `~/.mlx-bun/wiki/articles/Panasonic_Lumix_S5IIX.md` to this spec. Footnote def format (fixed — the Lucien standard, verified against `lucien/scripts/synthesize.ts` `SYNTHESIS_PROMPT_BOOTSTRAP` and `lucien/scripts/wikify.ts` `extractConvHashes`/`checkFootnoteIntegrity`): `` [^N]: `conv:HASH` (YYYY-MM-DD, source) — desc ``, where **`HASH` = the first 8 lowercase hex digits of the conversation UUID with hyphens stripped** (e.g. `00000000-0000-0000-0000-000000000000` → `00000000`; `extractConvHashes` matches `conv:[0-9a-z]{8}`). The hash identifies the CONVERSATION only — it deliberately carries NO chunk range — so chunk→section/article provenance is recovered from STRUCTURED state, never by parsing footnotes: the local pipeline reads its own `synthesized_chunk_sections` ledger (P0-T3), and the B9 gold-mining rule reads Lucien's `chunks`/`chunk_buckets`/`synthesized_bucket_chunks` tables.
-- ACCEPTANCE: the golden parses cleanly with `parseToc` (References last, See also before it); the P0-T2 parser round-trips the golden's infobox byte-identical and extracts its `aliases`; the parser rejects 3 malformed variants (References not last, two H1s, infobox after lead). `bun test` green.
-
-**P1-T2 · Infobox fenced-block format + field-kind taxonomy** — TAG: NOW · DEPENDS-ON: P1-T1
-- DELIVERABLE: In `Article_Conventions.md`, spec the infobox as a fenced ` ```info ` block of `key: value` lines (fenced so `parseToc`/`extractSection`/`searchArticles` skip it). Rules: `snake_case` keys, stable-sorted within field-groups by NORMALIZE; values naming another entity are `[[wikilinks]]` (load-bearing for NAVIGATION); `aliases:` is a comma list feeding the entity index; multi-value fields are `;`-separated. Each field is classed **world-fact** (`type`,`kind`,`mount`,`focal_length`,`aperture`,`sensor`,`format`,`compatible_with`) or **relationship-fact** (`owned` [enum `yes`|`no`|`wishlist`, where `wishlist` = wants-but-doesn't-yet-own],`acquired`,`used_for`,`chosen_over`,`status`,`opinion`). Every field is CONTENT the AI reads — there is no numeric facet or `gte/lte` query over any field. Example block:
-  ```info
-  type: Mirrorless camera
-  mount: [[L-Mount]]
-  sensor: full-frame 24.2MP
-  kind: thing
-  owned: yes
-  acquired: 2024-12 ($1,645 used)
-  used_for: video-conference camera; anamorphic
-  aliases: S5IIX, S5 IIX, Lumix S5IIX, LUMIX S5IIX
-  ```
-- ACCEPTANCE: the Meta page contains a table `field | kind(world/relationship) | value-type(scalar/entity-link/enum/list/date) | example` covering every field in the P1-T3 schemas; each world-fact is spec-sheet verifiable, each relationship-fact is conversation-established.
-
-**P1-T3 · Infobox-schema GRAMMAR — emergent, not a closed type list** — TAG: NOW · DEPENDS-ON: P1-T2
-- **PRINCIPLE (same as categories — do not violate):** infobox schemas are **EMERGENT**, created by judgment from the user's content and governed by an editable policy page that starts with rules + a tiny illustrative seed, NOT a hardcoded closed set of domain-tilted `type`s. A fixed seven-type table (five of them photography-specific) is the exact impose-don't-emerge failure this plan caught for categories; it would be wrong for any other user. There is **NO code-level enum of infobox `type`s** and **NO `type→category` / `type→schema` map**.
-- DELIVERABLE: `~/.mlx-bun/wiki/Meta/Infobox_Schemas.md` (**NEW Meta page — add `"Infobox_Schemas.md": INFOBOX_SCHEMAS` to `META_PAGES` in `vault.ts` and define the `INFOBOX_SCHEMAS` const**), shaped like `Category_Definitions.md`: a GENERIC GRAMMAR + a GROWING REGISTRY the model extends by judgment. The grammar (the only fixed part): (i) every fact is either a **world-fact** (spec-sheet/citable) or a **relationship-fact** (conversation-established, the personal-notability layer) — both are CONTENT the AI reads, neither is a query/filter index; (ii) keys are `snake_case`; (iii) a value naming another entity is a `[[wikilink]]` (feeds the link graph for navigation); (iv) value-types are `scalar | entity-link | enum | list | date`. A `kind:` field (H2/P1-T4c) tags the closed entity-kind; a `type:` field carries the emergent label. The registry then lists per-`type` field sets that the model has actually used, each entry **explicitly labelled "illustrative seed, not a closed set — extend by judgment when a new kind of thing appears."** Ship 1–2 illustrative examples only, e.g.:
-  - **lens** (world: `type,mount,focal_length,aperture,mfr,coverage` / relationship: `owned,acquired,used_for,chosen_over,opinion`)
-  - **camera-body** (world: `type,mount,sensor,resolution,mfr,format` / relationship: `owned,acquired,used_for,status,chosen_over`)
-  - *(further seeds the model may add by judgment — mount, adapter, person, project, tool, material, … — are illustrative, NOT a closed enum; reconcile any new `type` like `material` into the registry the same way a new category name is added.)*
-  When a newly-emergent `type` has no registry entry, the model proposes one (world/relationship split) and appends it to the page — editing the page re-tunes infobox extraction with **no code change** (same contract as the category page).
-- ACCEPTANCE (the gate): trace the north-star on paper using only the grammar's named fields — from Josh's camera article you reach `mount: [[L-Mount]]` → `L-Mount.adaptable_from ⊇ {M42,EF}` and `.native_lenses`, i.e. the wikilinks let you NAVIGATE to the lens/telephoto articles; the answer to "isn't crazy expensive / long reach" is then READ from the relevant article's prose+infobox (the user's recorded conclusion), not computed by filtering a field. If any hop needs a fact with no field, ADD the field to the grammar/registry (the page is editable by design). Assert the page ships rules + ≤2 illustrative `type` entries (not a closed seven) and the registry grows by reindex, never from a TS constant.
-
-**P1-T4 · Category mechanism — EMERGENT, not a hardcoded list** — TAG: NOW · DEPENDS-ON: P1-T3
-- **PRINCIPLE (do not violate):** categories are **emergent**, derived from the user's actual content, NOT imposed in advance — exactly as Lucien's `Buckets.md` states ("emergent — derived from the actual content of conversations, not imposed in advance; this list will be populated by the first synthesis run"). There is **NO hardcoded category vocabulary** and **NO `type→category` map**. Baking in a fixed list (especially one tilted to one user's domains) is forbidden — it would make the system wrong for any other user and contradicts the whole taxonomy-emerges principle.
-- DELIVERABLE: Three parts.
-  - **(a) Articles DECLARE their categories** (Wikipedia-style). An article carries a `categories:` line listing `[[Category:Name]]` memberships, assigned by the model **by judgment** when the article is created/wikified (governed by the policy page below). The deriver (P2-T1) reads these declarations into `article_categories`. Categories come into existence the first time an article declares them — the `categories` table simply accumulates the declared names. So a fresh wiki starts with **zero categories** and grows them as entity articles are written.
-  - **(b) `~/.mlx-bun/wiki/Meta/Category_Definitions.md`** (**NEW Meta page — `META_PAGES` (`vault.ts:537`) has only 4 keys today and does NOT include this; add `"Category_Definitions.md": CATEGORY_DEFINITIONS` to `META_PAGES` and define the `CATEGORY_DEFINITIONS` const = naming rules + an EMPTY registry, exactly like the other new pages in P1-T3/T5/T6/T7**) is the editable **policy + growing registry**, NOT a fixed list: it ships with (i) what a category is (a Wikipedia-style grouping of things), (ii) naming rules (title-case plural noun; specific; create a new category only when ≥2 articles would genuinely share it, else reuse/omit — "existing categories win"), (iii) an empty registry section that accumulates one `Name — definition — parent?` line as each category is first created — so a fresh `memory init` writes this page with rules + zero registry entries (`SELECT COUNT(*) FROM categories == 0`). The router reads this page via `loadMetaPolicy(["Category_Definitions"])`; because P0-T4's guard THROWS on a missing Meta page, this registration is load-bearing. Editing it re-tunes categorization with **no code change** (same contract as Lucien's policy pages).
-  - **(c) The ONLY fixed structural constant** is `ENTITY_KINDS` in `src/memory/article.ts` — the closed set of entity KINDS `{thing, person, domain, project, standard}` (a schema, not content): `export const ENTITY_KINDS = ["thing","person","domain","project","standard"] as const;`. Infobox `type:` values (`camera-body`, `lens`, …) are themselves **emergent labels the model assigns** (P1-T3), NOT a code-level enum. **`kind` is an explicit infobox field** (`kind:`) the model tags by policy from `ENTITY_KINDS`, defaulting to `thing` when untagged — there is NO deterministic `kindOf(type)` table over the open `type` domain (that would be another imposed map). The parser reads the declared `kind:` straight into `entities.kind` (validating it against `ENTITY_KINDS`); `person`/`domain`/`project`/`standard` are set when the model tags them, else `thing`.
-- ACCEPTANCE: "about lenses" = `SELECT article_stem FROM article_categories WHERE category='Lenses'` — exactly the articles that DECLARED `[[Category:Lenses]]`, with **zero** false positives like "the Toyota lens" (the 90-hit substring failure this replaces). Fixture: 3 lens articles declare `[[Category:Lenses]]`, a Toyota article does not → query returns exactly the 3. A fresh `memory init` has `Category_Definitions.md` = rules + EMPTY registry and `SELECT COUNT(*) FROM categories == 0`; after reindexing the fixture, `categories` holds exactly the declared names — none invented.
-
-**P1-T5 · `Entities.md` policy page + personal-notability checklist** — TAG: NOW · DEPENDS-ON: P1-T1
-- DELIVERABLE: `~/.mlx-bun/wiki/Meta/Entities.md` (**NEW Meta page — add `"Entities.md": ENTITIES_POLICY` to `META_PAGES`**; governs ENTITY-EXTRACT/ROUTE, inlined into stage prompts via P0-T4). Contents: (a) the five kinds {thing,person,domain,project,standard}; (b) **personal-notability as a binary checklist** — engagement-worthy if ANY of {owned/used it? · made a decision (chose/rejected)? · recurring across ≥2 conversations? · substantive opinion/conclusion?}, else DROP as trivia; (c) **the single CREATE rule (B8, baked but TUNABLE):** CREATE an entity article when it has **≥3 routed chunks** (the `3` is a deliberately-chosen tunable constant, not a derived value — tune it against P9-T3's ≤40-article first-run gate, raising it if the bootstrap over-creates), OR **≥1 chunk whose substantive-binary returns yes AND that chunk carries an ownership/decision fact** (`owned`/`used`/`chose`/`rejected`); otherwise PARK in the reserved staging bucket **`_unrouted`** (a fixed `buckets.name` for un-promoted chunks; the leading underscore keeps it out of the emergent bucket namespace — `memory init` seeds this one reserved row). "Recurring ≥2 conversations" = `COUNT(DISTINCT conv) ≥ 2` over `chunk_entities ⋈ chunks`; (d) canonical naming (Wikipedia title casing, prefer specific product name, `aliases` capture variants); (e) emit-both-granularities (specific + broad domain). Add a one-line redirect note to the seeded **`Buckets.md`** (the `BUCKETS` const at `vault.ts:540` — NOT `Bucketing.md`, which is a Lucien-only page absent from mlx-bun's `META_PAGES`) marking it **superseded by Entities.md** for synthesis targeting; do not delete — buckets survive as optional staging (L4).
-- ACCEPTANCE: a 20-mention gold fixture (`goldens/notability.json`: 10 notable — S5IIX, L-Mount, Sigma 150-600, Sankor 16C…; 10 trivia) classified by the checklist on paper matches ≥18/20, and **every** owned/decided-on item is KEPT. This gold set is reused by P5-T4 and P9.
-
-**P1-T6 · Summary-style split/merge rules** — TAG: NOW · DEPENDS-ON: P1-T1, P1-T3
-- DELIVERABLE: `~/.mlx-bun/wiki/Meta/Summary_Style.md` (**NEW Meta page — add `"Summary_Style.md": SUMMARY_STYLE` to `META_PAGES`**): **SPLIT** (thing in a section crosses notability + size ≥ a screen / its own infobox-worth → move to `Specific_Thing.md`, leave a 2–3 sentence summary + `*Main article: [[Specific_Thing]]*`; this is how S5IIX escapes `Camera_Gear_and_Lenses`); **MERGE** (two stubs, same thing, different names → merge to canonical title, loser becomes `alias` + redirect note); `{{Main}}`/`{{stub}}`/redirect template syntax defined so NORMALIZE and the parser recognize them. **`{{stub}}` threshold (L1, baked): an article is a stub when it has <2 body sections OR <40 words of body prose** (excludes infobox/References).
-- ACCEPTANCE: a worked example takes the real `Camera_Gear_and_Lenses` "Panasonic Lumix S5IIx" subsection and shows the post-split parent summary + new `Panasonic_Lumix_S5IIX.md` with infobox + `{{Main}}` link; reviewer confirms every `[^N]` in the moved section travels to the child (no facts lost).
-
-**P1-T7 · Port `Chunking.md` + confirm `Topics_to_Ignore.md` seeds** — TAG: NOW · DEPENDS-ON: none *(B10 — SEGMENT inlines these via P0-T4; they must be seeded into the mlx-bun vault)*
-- DELIVERABLE: Copy `~/Dreaming/Meta/Chunking.md` content into a `CHUNKING` const and add `"Chunking.md": CHUNKING` to `META_PAGES` in `vault.ts`. Verify `"Topics_to_Ignore.md"` is already in `META_PAGES` (it is — `vault.ts:541`) and its content matches `~/Dreaming/Meta/Topics_to_Ignore.md`; if drifted, update the `TOPICS_TO_IGNORE` const.
-- ACCEPTANCE: after `memory init` into a temp vault, both `~/.mlx-bun/wiki/Meta/Chunking.md` and `Topics_to_Ignore.md` exist and `loadMetaPolicy(["Chunking","Topics_to_Ignore"])` returns non-empty for both.
-
----
-
-## P2 — DERIVED INDEX + REINDEX: make the DB a rebuildable cache of the vault
-**Goal:** Populate categories/infobox/links/entities tables purely from markdown, deterministically and idempotently.
-
-**P2-T1 · Category membership deriver** — TAG: NOW · DEPENDS-ON: P0-T2, P0-T3, P1-T4
-- DELIVERABLE: In `src/memory/article.ts`, a deriver emitting one `article_categories` row per explicit `[[Category:Name]]` the article declares (the sole mechanism — emergent, no type-derived map); also upsert each newly-seen category name into the `categories` table.
-- ACCEPTANCE: against a ~10-article fixture vault, `SELECT article_stem FROM article_categories WHERE category='Lenses'` returns exactly the articles that DECLARED `[[Category:Lenses]]`. Membership moves ONLY by editing that declaration: **adding** a `[[Category:Lenses]]` line to an article adds its `article_categories` row on reindex and **removing** it deletes the row — whereas changing that same article's `infobox.type` from `lens`→`camera-body` and reindexing leaves `article_categories` UNCHANGED (there is no `type→category` derivation; that would re-impose the forbidden map). A Toyota/manufacturing article that declares no `[[Category:Lenses]]` never appears under `Lenses`.
-
-**P2-T2 · `reindex` — rebuild all derived tables from markdown** — TAG: NOW · DEPENDS-ON: P0-T2, P0-T3, P2-T1
-- DELIVERABLE: `src/memory/reindex.ts` — drops & rebuilds `entities`, `entity_aliases`, `categories` (accumulated from the `[[Category:…]]` declarations found in articles — NOT seeded from any constant), `article_categories`, `infobox_facts`, `links` purely from vault markdown via the P0-T2 parser. Pipeline-state tables untouched. `entities` gets one row per article (`kind` read from the explicit infobox `kind:` field, validated against `ENTITY_KINDS`, default `thing` — not derived from `type`); `entity_aliases` from titles + infobox `aliases:` (normalized: casefold, ws-collapse). `infobox_facts` one row per key-value with `kind` + resolved `entity_link` (via `resolveWikilinkToStem`). `links` walks prose + infobox entity-values + series/main banners, tagging `via`.
-- ACCEPTANCE: round-trip — fixture vault (5 entity articles incl. post-split S5IIX) → `reindex` → snapshot derived tables → `rm memory.sqlite` → `reindex` again → byte-identical (determinism). Mutate one article's infobox `type`/fields on disk → `reindex` → only that article's `infobox_facts`/`links`/`entity_aliases` rows change while its `article_categories` rows stay put (category membership tracks the explicit `[[Category:…]]` declaration, not `type`); separately, add/remove a `[[Category:…]]` declaration → `reindex` → only that article's `article_categories` rows change. `resolveName('S5 IIX')` and `resolveName('the Lumix S5IIX')` both resolve to `Panasonic_Lumix_S5IIX`. `mount: [[L-Mount]]` produces a `via='infobox'` edge; a `{{Main}}` split produces a `via='main'` parent→child edge; deleting an entity cascades its aliases/edges.
-
----
-
-## P3 — READ PATH PROVEN ON HAND-BUILT ARTICLES (the thesis, harder-to-fake half)
-**Goal:** With 5 hand-authored entity articles, answer the north-star query by **find article → read its TOC → read just the one section it needs → answer** — locate the right article (category / name / link), open its TOC (`parseToc`), then `memory_section` the single relevant section, answering from it — in < 5 KB, before any synthesis exists. **TOC→section is the DEFAULT read; a whole-article `memory_read` is the exception** (this is the concrete "small focused reads, never a dump" mechanism). The read path is deliberately trivial; there is NO facet/filter query anywhere in it.
-
-**P3-T1 · Hand-author the lens fixture set** — TAG: NOW · DEPENDS-ON: P1-T1, P1-T2, P1-T3
-- DELIVERABLE: Entity articles with infoboxes + wikilinks in `tests/fixtures/wiki/`: `Panasonic_Lumix_S5IIX` (`mount:[[L-Mount]]`, owned), `L-Mount` (`adaptable_from: [[M42]];[[EF]]`, `native_lenses` list), `Sigma_150-600` (`focal_length:150-600mm`, `mount:[[L-Mount]]`, prose verdict naming it the sane long-reach pick at ~$600 — the user's recorded conclusion, in the body), `Lumix_75-300` (`focal_length:75-300mm`, owned), `Lens_Mount_Adaptation` (section "adaptable-mounts" stating Josh adapts M42 + EF). Plus a `PETG` material article (`type: material`, `kind: standard` declared explicitly in the infobox) with a `print-settings` section for the second query. The "isn't crazy expensive / long reach" judgment lives in the `Sigma_150-600` article's PROSE (synthesized content the AI reads), not in a queryable numeric field. **Each lens fixture (`Sigma_150-600`, `Lumix_75-300`) MUST DECLARE `[[Category:Lenses]]`, and `PETG` MUST DECLARE `[[Category:Materials]]`** — categories exist ONLY by explicit per-article declaration (P1-T4), and P3-T3 (`articlesInCategory('Lenses')`) / P3-T4 (`memory_category(category="Lenses")`) use them to FIND the articles; without the declaration the "articles about X" hop returns empty.
-- ACCEPTANCE: all parse via P0-T2; `reindex` populates `infobox_facts`/`links`/`article_categories` with the navigation edges present (`articlesInCategory('Lenses')` returns exactly the two declared lens fixtures).
-
-**P3-T2 · Read-side index (rebuildable cache)** — TAG: NOW · DEPENDS-ON: P2-T2
-- DELIVERABLE: `buildMemoryIndex(root)` (in `vault.ts` or `db.ts`) producing mtime-invalidated in-memory maps mirrored into the derived tables: `aliasToStem`, `categoryToStems`, `seriesToStems`, `kindToStems`, `typeToStems`, `infoboxByStem`, `leadByStem`. No embeddings.
-- ACCEPTANCE: rebuild over the fixture vault is correct and < 50 ms for ~500 articles; deleting the cache and rebuilding yields byte-identical maps; touching one file re-indexes only that file.
-
-**P3-T3 · Navigation primitives (find + resolve + links)** — TAG: NOW · DEPENDS-ON: P3-T2
-- DELIVERABLE: `src/memory/query.ts` — the small set of deterministic lookups that FIND an article: `articlesInCategory(cat)` ("articles about X" = `Category:X` membership), `resolveName(nameOrAlias)→stem`, `neighbors(stem)→{outbound,inbound}` (wraps `getArticleLinks`, the `[[link]]` graph). No `byInfoboxField`, no numeric facets, no `gte/lte` — the infobox is content the AI READS (via the read tools below), not a filter target. All over indexed names/categories/links — no embeddings.
-- ACCEPTANCE: `articlesInCategory('Lenses')` returns exactly the two declared lens fixtures; `resolveName('S5 IIX')`/`resolveName('the Lumix S5IIX')` both resolve to `Panasonic_Lumix_S5IIX`; `neighbors('Panasonic_Lumix_S5IIX')` includes the `[[L-Mount]]` edge. Plain SQL/map joins, zero embeddings.
-
-**P3-T4 · Navigation read tools (extend `tools.ts` + `vault.ts`)** — TAG: NOW · DEPENDS-ON: P3-T2, P3-T3
-- DELIVERABLE: The read tools are exactly **find-article-by-category, resolve-by-name/alias, read article, read section, follow [[links]]** — nothing that filters or queries infobox fields. New/updated pi tools in `src/memory/tools.ts`, registered in `MEMORY_TOOL_NAMES` (`tools.ts:31`) + `createMemoryTools()` (`tools.ts:357`) (read-only/auto-allow):
-  - `memory_resolve(surface)` → `{stem,title,kind,lead}` or ≤3 disambig candidates via `aliasToStem` (no model, no substring). ≤0.5 KB.
-  - `memory_category({category|type|series})` → member rows `stem · lead`, paginated (cap 40 rows / ≤4 KB) — this is the "articles about X" finder.
-  - `memory_read(stem)` → the article's TOC + lead by default (via `parseToc`), so the agent picks the section it needs; the FULL prose+infobox is the EXCEPTION, returned only on `force:true`. The >8 KB size guard returns TOC + lead + "read a section with memory_section" steer regardless.
-  - `memory_section(stem, anchor)` → one section's body. **This is the DEFAULT read** — find article → TOC → this one section → answer; whole-article reads are the exception.
-  - `memory_links(stem)` — keep `getArticleLinks` but group by origin (`infobox`/`series`/`seealso`/`prose`), each capped at 25 with "+N more". ≤1 KB. (`memory_infobox(stem)` may surface the infobox key:value lines as READABLE CONTENT, but it is NOT a query/filter — there is no `memory_infobox_query` tool.)
-  - Demote `memory_search`: reword description to "fallback only when name/category lookup has failed across a couple phrasings"; cap returned summaries to top ~8.
-  Update the **`memory` skill body** (the bundled SKILL.md the description references at `tools.ts:10-12`) and **`memoryIndexHint()` (`tools.ts:384`)** to teach the order **resolve / category (find the article) → TOC → memory_section (read the ONE relevant section — the default) → links**, with whole-article read and search LAST.
-- ACCEPTANCE: `memory_category(category="Lenses")` returns the declared lens articles in < 2 KB; `memory_resolve("S5 IIX")` returns the S5IIX stem; `memory_read("Sigma_150-600")` returns the article whose prose holds the long-reach verdict in < 2 KB; `memory_read` on the 671-line blob fixture returns TOC+lead (< 1 KB) not the blob; `"3D print"` search returns ≤8 capped rows with a steer; there is NO infobox-field-query tool registered; `tsc` clean.
-
-**P3-T5 · North-star traversal proof + silent-colleague contract** — TAG: NOW · DEPENDS-ON: P3-T4
-- DELIVERABLE: `scripts/experiments/north-star-traversal.ts` running the literal lens query over the fixture vault and printing the **find → read → answer** trace: FIND the relevant lens article (via `memory_category("Lenses")`, or a `[[link]]` hop from the camera article's `mount:[[L-Mount]]`), open its TOC (`memory_read` returns TOC+lead), READ just the one relevant section (`memory_section` on `Sigma_150-600`'s verdict section), ANSWER from the user's recorded conclusion there — no field filter, no ordering by price, no whole-article dump. Plus port Lucien's `LUCIEN_INSTRUCTIONS` spirit (`/Users/joshrossi/Code/lucien/src/index.ts:28-38`) into the read-tool descriptions + the `memory` skill body + the MCP server `instructions` string: (1) look up BEFORE answering; (2) each article is the consolidated position — don't reconstruct from raw history; (3) speak as continuation, never "per the wiki…" (cite by name only if asked or to distinguish wiki-fact from own-opinion).
-- ACCEPTANCE: the answer is grounded in the article that was FOUND and READ (category + name + wikilink navigation only, no facet/filter query); with `resetEmbedCounter()` at start, assert `getEmbedCounter()===0` (zero embedding calls), no single read > 2 KB, **total bytes < ~8 KB** (vs 33–150 KB). Second query "PETG print settings" resolves via `memory_resolve("PETG") → memory_read/memory_section("PETG","print-settings")` in < 1.5 KB, never consulting substring search.
-
----
-
-## P4 — MEASURE-FIRST INSTRUMENTS (the #1 risk, gated before building ROUTE)
-**Goal:** Prove deterministic name/category lookup beats vector routing, and decide whether any embedding fallback is even needed — by measurement, not assumption.
-
-**P4-T1 · Lexical-shortlist recall ceiling over names** — TAG: NOW · DEPENDS-ON: P0-T3, P0-T5
-- DELIVERABLE: Extend `scripts/experiments/bucket-cohesion-signal.ts` methodology to NAMES: build the alias vocabulary from `goldens/entities.json` (P0-T5), compute trigram/token-overlap top-K recall of the gold entity per chunk, reported overall + per bucket-size-bin (1, 2-4, 5-20, 21-100, 100+).
-- ACCEPTANCE: report printed. Decision rule (made here): if lexical recall@K ≥ 0.90 overall AND ≥ 0.80 in every size-bin → no embedding fallback. Else add a narrow embedding-fallback shortlister (`src/embed.ts` as instrument only, never the router — the tripwire counter still applies) scoped to the failing bin.
-
-**P4-T2 · No-vector routing proof** — TAG: NOW · DEPENDS-ON: P4-T1, P0-T5
-- DELIVERABLE: `scripts/memory/eval-route.ts` — read the Lucien `chunk_buckets` edges (the "~8809 edges") and reinterpret each chunk's gold buckets as the things it's about (gold from `goldens/entities.json` alias-matched to bucket names). Run normalize → alias-lookup → trigram-shortlist → binary "same thing?" disambiguation. Compare real-shortlist routing F1 against a Stage-0 embedding-cosine baseline (the indicative ~67% top-12 broad-bin number — re-measured live here, not taken as a literal; embeddings via `src/embed.ts` as instrument). Report Jaccard per chunk, and **oracle-shortlist F1** (gold forced into candidates) vs **real-shortlist F1** (gap = retrieval tax).
-- ACCEPTANCE (gates): name-match real-F1 > embedding-cosine F1, AND oracle-F1 ≥ 0.85. **Lexical recall@K is reported as a MEASUREMENT, not a hard gate** — it feeds the P4-T1/R1 fallback DECISION: if recall@K < 0.90 overall or < 0.80 in any size-bin (the 122-bucket <5 tail is expected to miss, per line 39's 67%-broad-bin evidence), add the narrow embedding-fallback shortlister scoped to the failing bin (offline instrument, never the router — tripwire still applies). A recall miss is the planned trigger, NOT a phase failure. The `memory-section`/disambig LoRA decision keys off oracle-F1. *(All cited corpus counts are descriptive; the script reads live `SELECT COUNT(*)` from `.lucien/lucien.db`, never the literal 8809.)*
-
----
-
-## P5 — INGEST + SEGMENT + ENTITY-EXTRACT + ROUTE (the write front-half)
-**Goal:** Conversations → chunks → entities → routed `(chunk, article)` pairs, deterministically, base+policy (no new LoRA).
-
-**P5-T1 · Ingest the corpus** — TAG: NOW · DEPENDS-ON: P0-T3
-- DELIVERABLE: `src/memory/ingest.ts` — load pi `SessionManager.listAll()` (and, for bootstrap, `.lucien/lucien.db`) → `conversations`/`messages`, watermark-gated per source. **Lucien column mapping (B9):** `conversations.uuid→conv`, `name→title`, `source→source`, `updated_at→updated_at`; `messages.conversation_uuid→conv`, `position→position`, `sender→role`, `uuid→uuid`, `text→text`. Lucien `chunks` carry `start_message_uuid`/`end_message_uuid` → resolve each to its `messages.position` (via `conversation_uuid`) to fill mlx-bun `chunks.start`/`.end`, then set `chunks.id = \`${conv}:${start}-${end}\`` (the P0-T3 derivation — discard Lucien's INTEGER chunk id).
-- ACCEPTANCE: ingesting from `.lucien/lucien.db` yields `COUNT(*)` of `conversations`/`messages` EQUAL to the source's `SELECT COUNT(*)` (relative gate, H5 — not a hardcoded 2063/45179); re-run is a watermark no-op.
-
-**P5-T2 · SEGMENT (wire the Gemma-4-e4b chunk adapter)** — TAG: NOW · DEPENDS-ON: P5-T1, P0-T1, P0-T4, P1-T7
-- DELIVERABLE: `chunkConversations` in `src/memory/chunk.ts` — port Lucien `chunk-recent.ts CHUNK_PROMPT` onto `callLocal("chunk", …)`. Wire the adapter: `ln -s ~/.cache/mlx-bun/adapters/e4b-chunk-300 ~/.cache/mlx-bun/adapters/memory-chunk` (the validated Gemma-4-e4b chunk SFT sweet-spot checkpoint — **NOT `e4b-chunk-400` (over-merged) nor any `cpm-*` MiniCPM5 adapter**). Inline `Chunking.md` + `Topics_to_Ignore.md` via `loadMetaPolicy`; format `[sender] (uuid) text`; out `{"chunks":[{start_message_uuid,end_message_uuid,label}]}`. Port `chunk-validation.ts` anchor repair; write pointer rows; re-chunk only `updated_at > chunked_at`.
-- ACCEPTANCE: on 10 real pi sessions ≥95% produce valid JSON with in-range anchors; chunks stored as pointers; re-run is a no-op. `scripts/chunk-eval.ts` (with `ADAPTER=~/.cache/mlx-bun/adapters/e4b-chunk-300`) reproduces the recorded `e4b-chunk-300` chunk-eval numbers. **For e4b the recorded metric is `purity × cohesion`, NOT boundary/label (boundary/label was the abandoned MiniCPM5 metric).** Gate: **≥ the recorded `e4b-chunk-300` scores — purity 92.2 / cohesion 87.8** (source: `reports/orpo-chunk-paper/research-journal.md`, § "e4b experiment + results"; base-e4b for reference is 99.5 / 58.8). Keep the purity sub-gate concrete: sample 50 chunks, LLM-judge single-topic, target ≥90% (consistent with the recorded 92.2).
-
-**P5-T3 · ENTITY-EXTRACT (base+policy)** — TAG: NOW · DEPENDS-ON: P5-T2, P1-T5, P0-T5
-- DELIVERABLE: `extractEntities(chunk)` in `src/memory/entity.ts` on `callLocal("entity", …, {maxTokens:128})` base + `Entities.md` inlined. Prompt: *"What things is this chunk about? List canonical wiki-title names, one per line; include specific things AND the broad domain; NONE if nothing nameable/substantive."* Parse via `parseLines` (NO multi-item JSON, no batch indices — avoids the batch-index-drift that sank Lucien's batched assign). Write `chunk_entities(chunk_id, entity_name, surface_form)`. Plus deterministic canonicalization (casefold, ws-collapse, strip leading articles/possessives) building `entity_aliases` from titles + infobox `aliases:`.
-- ACCEPTANCE: `scripts/memory/eval-entity-extract.ts` (gold = `goldens/entities.json` from P0-T5) reports precision/recall/F1 + canonicalization rate; **recall ≥ 0.80** on entities with a gold article, **precision ≥ 0.70**; "S5IIX"/"S5 IIX"/"the Panasonic" all normalize to one stem. Report over/under-extract split to drive the `memory-entity` LoRA decision (P10-T3).
-
-**P5-T4 · ROUTE (deterministic name-match + binary disambiguation)** — TAG: NOW · DEPENDS-ON: P5-T3, P4-T2, P1-T5
-- DELIVERABLE: `src/memory/route.ts` — deterministic join of `chunk_entities` against `entities`: exact/alias hit → routed; miss → trigram shortlist + per-candidate binary *"Is `{surface_form}` (context: `{label}`) the same thing as `{candidate}` — `{lead}`? yes/no"* via `callLocal("route", …, {maxTokens:4})` parsed by `parseBinary`; no-match + passes CREATE rule → CREATE candidate; trivial → drop. A chunk routes to MULTIPLE articles (entity fan-out — one chunk can be about several things). The **CREATE gate is the baked B8 rule from P1-T5**: create iff ≥3 routed chunks for the entity OR (≥1 substantive-yes chunk AND it carries an ownership/decision fact); else PARK in the reserved `_unrouted` staging bucket (P1-T5). The substantive-binary: *"Did the user engage with `{entity}` substantively (own/use/decide-on/recurring/strong opinion)? yes/no."*
-- ACCEPTANCE: `Panasonic S5IIX` and `Lumix S5 IIX` resolve to ONE article (no duplicate page); trivia ("npm", incidental names) drop or park while engaged things (≥3 chunks or owned/decided) create; the S5IIX-anamorphic chunk fans out to `Panasonic_Lumix_S5IIX`, `Anamorphic_Lens_Adapters`, AND `Lens_Mount_Adaptation` (all three `(chunk,article)` pairs enqueued). Re-run the P1-T5 `goldens/notability.json`: every owned/decided item creates, every trivia parks/drops.
-
----
-
-## P6 — COLD-START CREATE + NORMALIZE (first articles from real conversations)
-**Goal:** From an empty vault, manufacture entity articles with infoboxes — the first end-to-end win.
-
-**P6-T1 · Conservative write gate** — TAG: NOW · DEPENDS-ON: P0-T2
-- DELIVERABLE: `src/memory/gate.ts` porting Lucien `wikify.ts` `verifyEditorialResult` + `checkFootnoteIntegrity` + `extractConvHashes`: preserve prose, ≥70% word-count floor on the touched section, every pre-existing `conv:` citation survives, H1/References survive, footnote bijection. A weak pass → NO-OP (never corrupt the vault).
-- ACCEPTANCE: ported Lucien gate tests pass (dropped-citation, word-floor, footnote-bijection, H1/References-survival).
-
-**P6-T2 · NORMALIZE (deterministic, no model)** — TAG: NOW · DEPENDS-ON: P0-T2, P1-T6
-- DELIVERABLE: `src/memory/normalize.ts` porting Lucien `normalize-footnotes.ts` + `normalize-wikilinks.ts`: footnote bijection (every `[^N]`⇔one `[^N]:`, renumber 1..k in first-appearance order, merge duplicate `conv:HASH` defs); wikilink canonicalization (`resolveWikilinkToStem`, repair/flag orphans, infobox entity-values normalized the same); infobox key sort; structural guards (`## References` last, `## See also` before it, `{{stub}}` for thin articles per the L1 threshold <2 sections/<40 words, title=H1=stem); recognize `{{Main}}`/`{{stub}}`/redirect. Non-fatal (logs + continues), runs after every write before `commitVault`.
-- ACCEPTANCE: ported Lucien `.test.ts` suites pass; a patched article with appended out-of-order footnotes renumbers contiguous; every run leaves footnote-bijective, orphan-free articles; an article with 1 section / 20 words gets `{{stub}}`.
-
-**P6-T3 · CREATE flow** — TAG: NOW · DEPENDS-ON: P5-T4, P6-T1, P6-T2, P1-T3
-- DELIVERABLE: Rewrite `src/memory/synthesize.ts` stub to `create-or-update`; CREATE branch (entered only when the P1-T5/P5-T4 CREATE rule fires): for an entity's accumulated chunks → sub-cluster → outline (a clean TOC of single-topic sections — the read path's TOC→section reads ride this, so sharp section structure is the primary quality goal) → per-section draft (each draft = the same bounded per-section op as a patch, from empty, via `callLocal("synthesis", …)` — the `synthesis` stage selects the `memory-synthesis` adapter once trained, P10-T1 — never hold a whole article in the model's head) → seed infobox (entity-valued fields as wikilinks; declare `aliases:`). Port Lucien `scripts/synthesize.ts SYNTHESIS_PROMPT_BOOTSTRAP` citation/wikilink rules; feed a `getOtherArticles`-style stem list so `[[wikilinks]]` resolve. Conservative bias: thin entities stay `{{stub}}` (L1 threshold). Commit via `commitVault`; record `synthesized_chunk_sections`.
-- ACCEPTANCE: from zero, a camera conversation yields a stub-or-better `Panasonic_Lumix_S5IIX.md` with a queryable infobox (`mount: [[L-Mount]]`) and ≥1 cited section, passing gate + NORMALIZE.
-
-**P6-T4 · Cold-start end-to-end smoke (≤10 sessions)** — TAG: NOW · DEPENDS-ON: P6-T3
-- DELIVERABLE: Wire pipeline DAG in `src/memory/pipeline.ts`: `ingest → SEGMENT → ENTITY-EXTRACT → ROUTE → CREATE → NORMALIZE → commitVault`. Run a **handful (≤10) of real pi sessions** through it (fixture scale — the FULL-corpus bootstrap is USER-ACTION, see P6-T5).
-- ACCEPTANCE: a small set of entity-granular articles with infoboxes and resolvable wikilinks committed to vault git; **the north-star query (P3-T5) now answers against the synthesized `Panasonic_Lumix_S5IIX` whose infobox carries `mount: [[L-Mount]]`** — the write path manufactures the read-path fixture.
-
-**P6-T5 · FULL-corpus bootstrap** — TAG: **USER-ACTION** · DEPENDS-ON: P6-T4 *(H6 — segmenting ~2063 convs / ~7316 chunks is a long GPU job; the agent must NOT start it)*
-- DELIVERABLE: Document the exact command for Josh to run the whole corpus through the DAG: `mlx-bun memory synthesize` (CLI in `src/cli.ts`, the `runSynthesis` path). Precondition: Gemma-4-e4b base present (precheck), `memory-chunk` adapter symlinked (P5-T2). The agent prepares everything and STOPS here; Josh launches.
-- ACCEPTANCE: (after Josh's run) the vault contains the bootstrapped articles; P9 evals run against it. The agent never executes this task itself.
-
----
-
-## P7 — SECTION-ROUTE + PATCH (steady state, once articles exist)
-**Goal:** Fold new chunks into existing articles via M×N binary section routing + atomic section patches.
-
-**P7-T1 · SECTION-ROUTE (M×N binary)** — TAG: NOW · DEPENDS-ON: P6-T4
-- DELIVERABLE: `routeSections(chunk, article)` in `src/memory/cluster.ts` (repurposed from Lucien `scripts/cluster-assign-recent.ts`, which exports `ASSIGN_OR_PROPOSE_PROMPT` + `buildPrompt`) on `callLocal("section", …, {maxTokens:4})`: for each of the article's N TOC sections (via `parseToc`/`extractSection`), prompt *"article title + section heading + section's first ~2 sentences + chunk label + 1-line gist → Should this chunk update THIS section? yes/no."* parsed by `parseBinary`. Hierarchical pruning: only run sections of articles ROUTE already matched. All-no + substantive → CREATE-NEW-SECTION sub-task *"Name a new section (2–5 words) or NONE."* (parsed by `parseLines`). Write `chunk_sections`.
-- ACCEPTANCE: `scripts/memory/eval-section-route.ts` (proxy gold from the dozen large buckets that became sectioned `~/Dreaming` articles via `synthesized_bucket_chunks` + `parseToc`, LLM-labeled once + frozen) reports per-pair accuracy + per-chunk section-set **Jaccard ≥ 0.6**; a chunk fitting no section produces a named new section, not a drop.
-
-**P7-T2 · PATCH (atomic section integration)** — TAG: NOW · DEPENDS-ON: P7-T1, P6-T1, P6-T2
-- DELIVERABLE: In `synthesize.ts`, the UPDATE branch via `callLocal("synthesis", …)` (same `synthesis` stage as CREATE → the `memory-synthesis` adapter, P10-T1): *"Integrate THIS chunk into THIS section. Add a `[^N]` citation. Change nothing else. **If the chunk CONTRADICTS an existing claim, this is a CORRECTION — rewrite the claim TOWARD the new statement (resolve toward the latest); do NOT keep both and do NOT ignore the new one.**"* Prompt = current section markdown + chunk (conv UUID/date/slug for the footnote) + the footnote contract inlined; out = rewritten section only + new `[^N]:` lines. This is the self-healing mechanism: a user's in-conversation correction is an ordinary chunk that updates the article toward their latest position. Swap just that section via `extractSection`; rest byte-identical. Run through gate (weak → NO-OP, chunk retries next run) + NORMALIZE. Idempotency via `synthesized_chunk_sections` (PK blocks double-fold).
-- ACCEPTANCE: integrating a chunk rewrites only that section (rest byte-identical), adds a correct `[^N]`/`[^N]:`, passes the gate; a deliberately weak output is gated to NO-OP; a second run writes nothing new. **Self-healing case:** a section asserting "loves 35mm" + a later correction chunk ("35mm is fine but nowhere near as good as 50mm") yields a section whose claim now favors 50mm (toward the latest), citing the new chunk — old claim resolved, not appended alongside. `scripts/memory/eval-patch.ts` (gold = `(chunk, section_before, section_after)` triples mined from `~/Dreaming` via the B9 structured-state recovery rule — chunk identity from Lucien's `chunks`/`chunk_buckets`/`synthesized_bucket_chunks` tables, section diffs from git history; NOT footnote parsing) reports 100% on deterministic gates and faithfulness judge ≥ 0.8. The LOCAL pipeline's own (chunk→section) provenance for idempotency/eval is its `synthesized_chunk_sections` ledger (P0-T3), never footnote parsing.
-
----
-
-## P8 — WIKIFY structural pass (periodic editorial sweep)
-**Goal:** Build/maintain the traversable structure (infoboxes, `{{Main}}` splits, series) the read path rides; retrofit the S5IIX out of the gear blob. **Primary article-quality objective: well-formatted SECTIONS + a clean TOC** — because that is exactly what makes the read path's targeted TOC→section reads (P3) possible. The structural pass keeps section headings sharp, single-topic, and `parseToc`-clean; a well-sectioned article IS a high-quality article.
-
-**P8-T1 · LLM wikify node (per-section improve)** — TAG: NOW · DEPENDS-ON: P6-T1, P6-T2
-- DELIVERABLE: `src/memory/wikify.ts` (distinct from deterministic `normalize.ts`): per-section *"Tighten this section: remove redundancy, keep every citation and every specific detail, don't generalize away particulars."* (port Lucien `scripts/wikify.ts buildEditorialPrompt(articleText)` — the exported function wrapping the private `EDITORIAL_PROMPT` const — applied per-section) via `callLocal("editor", …)`, behind the gate. Runs periodically, not per-chunk.
-- ACCEPTANCE: tightens a redundant section while passing the citation/word-floor gate; a weak edit is rejected, original kept.
-
-**P8-T2 · Infobox extractor/refresher** — TAG: NOW · DEPENDS-ON: P8-T1, P1-T3
-- DELIVERABLE: *"From this entity article, extract/refresh the structured key-value facts"* per the kind's P1-T3 schema; entity-valued fields → wikilinks; declare `aliases:` (feeds `entities` index, closing the loop). Corpus today: 0 infoboxes — this creates them.
-- ACCEPTANCE: ingesting a camera conversation yields `mount: [[L-Mount]]` as a parseable key-value and the declared `aliases:` feed back into `entities` on `reindex`.
-
-**P8-T3 · Summary-style split + series banner** — TAG: NOW · DEPENDS-ON: P8-T2, P1-T6
-- DELIVERABLE: *"Has a section outgrown the article?"* → split a child entity article (e.g. `Panasonic_Lumix_S5IIX` born from `Camera_Gear_and_Lenses`) leaving a summary + `{{Main}}`; *"Does this article belong to a series?"* → maintain `*Part of a series on [[Lenses]].*` banner.
-- ACCEPTANCE: `scripts/memory/eval-wikify-split.ts` (gold = the Archie cluster children + the Camera_Gear→S5IIX retrofit) splits Camera_Gear→S5IIX correctly with a `{{Main}}` edge in `links`, and does NOT split a cohesive article (no false splits on the 78 mid-size buckets); two near-duplicate stubs emit a MERGE signal.
-
-**P8-T4 · Talk-page restructure signals** — TAG: NOW · DEPENDS-ON: P8-T3
-- DELIVERABLE: Split/merge/contradiction triage writing `Talk/<stem>.md` suggestions (Lucien `<<<TALK>>>` style). Embedding silhouette permitted here as the one sanctioned offline triage instrument — never on the routing hot path (tripwire still 0 on the read path).
-- ACCEPTANCE: writes a `Talk/<stem>.md` suggestion; never silently restructures across articles.
-
-**P8-T5 · Wire periodic synthesis to scheduler** — TAG: NOW · DEPENDS-ON: P7-T2, P8-T3
-- DELIVERABLE: Wire `runSynthesis` (CLI `memory synthesize`, `src/cli.ts`) onto the existing launchd nightly scheduler (`src/memory/schedule.ts`) + SSE progress.
-- ACCEPTANCE: `mlx-bun memory synthesize` runs the full DAG; scheduled run fires and reports progress. *(The full-corpus run itself is USER-ACTION per P6-T5.)*
-
----
-
-## P9 — EVAL & ACCEPTANCE (system-level + migration parity)
-**Goal:** Freeze the acceptance suite and the migration parity check; treat `~/Dreaming` as read-only oracle.
-
-**P9-T1 · Offline answer grader** — TAG: NOW · DEPENDS-ON: P0-T1
-- DELIVERABLE: `scripts/memory/judge-answer.ts` — an LLM grader (a stronger model offline is acceptable as the GRADER, never in the hot path) returning binary correct/incorrect + a silent-violation flag (regex + judge for "per the wiki / according to your notes").
-- ACCEPTANCE: grades a labeled set of 5 answers correctly. *(The 5-answer self-test set and the per-query gold answers for P9-T2 are AUTHORED when `scripts/memory/northstar-queries.json` is frozen — normal test authoring, derived from the Q1–Q6 intents + the P3-T1 fixtures.)*
-
-**P9-T2 · North-star acceptance harness** — TAG: NOW · DEPENDS-ON: P3-T4, P9-T1, P6-T4, P0-T6
-- DELIVERABLE: `scripts/memory/eval-northstar.ts` + frozen `scripts/memory/northstar-queries.json` (Q1 lens-reach, Q2 camera-why, Q3 M42-adaptable, Q4 anamorphic-best, Q5 list-every-lens [the 150 KB killer], Q6 negative control astrophotography — must return "no personal opinion on file", no hallucination). Each query's gold/expected answer is authored into `northstar-queries.json` when it is frozen (derivable from the stated Q1–Q6 intents + the P3-T1 fixtures; normal test authoring, not a separate gold-mining task). Runs the read path only via `loadTaskModel(MODEL_ID,…)`/`generateText`, instruments `memory_read`/`memory_section`/`memory_search` byte counts, asserts the hot-path embedding tripwire (`resetEmbedCounter()` before, `getEmbedCounter()===0` after — P0-T6), and a no-network interceptor. Five gates per query: correctness (judge), context budget, no-vector, local-only, silent.
-- ACCEPTANCE: **≥5/6 pass (Q6 must pass)**, `mean_context_kb < 5`, `max_read_kb < 2`, `hot_path_embed_calls == 0`. Q5 ("list every lens") must resolve by FINDING the `Category:Lenses` members and reading them, not by dumping the blob (fail if any single `memory_read` > 2 KB) — category membership, never a field query.
-
-**P9-T3 · Bootstrap audit** — TAG: NOW · DEPENDS-ON: P6-T4
-- DELIVERABLE: `scripts/memory/audit-bootstrap.ts` — per new article: (i) title is a thing/noun not `X_and_Y` (reject `_and_` unless a genuine compound entity); (ii) infobox present + every entity-value resolves to a stem or redlink; (iii) footnote bijection; (iv) every claim traces to a cited chunk (infobox-traces-to-cited-claim).
-- ACCEPTANCE: 100% structural pass; first run produces **≤ 40 entity articles** each with a populated infobox + ≥1 citation (under-cover is success); manual spot-read of the top-10 confirms the "read one article, get up to speed" bar.
-
-**P9-T4 · Migration parity** — TAG: NOW · DEPENDS-ON: P5-T3, P6-T4, P0-T5
-- DELIVERABLE: `scripts/memory/eval-coverage.ts` (every engagement-worthy thing in the old vault's 12 big bins has a corresponding entity article in the new vault; tail allowed to stay stubs), keyed off `goldens/entities.json` (P0-T5, already frozen to the `~/Dreaming` SHA). `~/Dreaming` stays read-only; the SHA snapshot is `goldens/entities.json:builtFromSha`.
-- ACCEPTANCE: **0 lost engaged-with things** even though article count drops (385 bins → fewer, sharper entities); P9-T2 north-star passes on the new vault, provably impossible on the old (150 KB / 42× blob).
-
----
-
-## P10 — LoRA TRAINING (GATED, LATE, only on measured miss)
-**Goal:** Train per-stage adapters ONLY where P5–P8 base+policy missed the F1/quality bar. Priority: `memory-synthesis` (PATCH/CREATE) > `memory-section` > `memory-entity` > `memory-editor`. Checkpoints by judge F1, not val loss. **All training is USER-ACTION (Josh launches; the agent never starts GPU work).**
-
-**e4b training-recipe caveat (applies to EVERY P10 stage — Gemma-4-e4b is the base).** Beyond using `ops.sdpa` (not flash), e4b long-context training needs the **segmented-backward path** — the validated chunk adapter (`e4b-chunk-300`) was trained that way. Plain `chunk-finetune.ts` SFT at `SEQ=8192` may OOM e4b WITHOUT it. So every P10 command sets `SEG=2` (`chunk-finetune.ts` `SEG` env → `segment_size` = layers per segment; **2** is the value the validated `e4b-chunk-300` adapter was trained with — `reports/orpo-chunk-paper/research-journal.md` records `seg2`; background in `docs/design/segmented-backward-training.md`). `SEG=0` (default) disables segmented backward and is the OOM footgun for e4b.
-
-**P10-T0 · Generalize `chunk-finetune.ts` with a `TASK` env selector** — TAG: NOW · DEPENDS-ON: none *(B3 — the trainer is env-driven and single-task today; this is the ONLY trainer change needed)*
-- DELIVERABLE: Two changes to `scripts/chunk-finetune.ts`. **(a) Re-default the base MODEL off the ABANDONED MiniCPM5.** `resolveModel()` (`chunk-finetune.ts:19-23`) currently defaults `MODEL` (when unset) to `models--mlx-community--MiniCPM5-1B-OptiQ-4bit` — and that snapshot is on disk, so it loads SILENTLY rather than erroring. Change the unset-`MODEL` branch to resolve the Gemma-4-e4b snapshot via the PREAMBLE `config.json`-iterating rule (iterate `…models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots/*`, pick the one containing `config.json`), and update the stale header comment `// MODEL defaults to MiniCPM5-1B-OptiQ-4bit` (`chunk-finetune.ts:11`) to name the e4b default. Also re-point the stale default `ADAPTER` (`chunk-finetune.ts:31`, currently `…/mlx-bun-finetunes/minicpm5-chunk-seq${SEQ}`) off the MiniCPM5 name — fold it into the `TASK`-driven default below (when `TASK` is unset, default to a neutral `…/mlx-bun-finetunes/e4b-chunk-seq${SEQ}`) so no path string still advertises MiniCPM5. **(b) Add a `TASK` env var** that ONLY changes the DEFAULT `DATA` and `ADAPTER` (the SFT path is identical across stages). Mapping: `TASK=synthesis|section|entity|editor` → `DATA` defaults to `~/.cache/mlx-bun/memory-train/<TASK>` and `ADAPTER` defaults to `~/.cache/mlx-bun/adapters/memory-<TASK>`. Explicit `MODEL`/`DATA`/`ADAPTER` env still override. No task-routing inside the trainer, no new CLI flags (Bun ignores unknown argv — the existing interface is env-only). After this, NO P10 run can silently load MiniCPM5.
-- ACCEPTANCE: `MODEL="$(for d in ~/.cache/huggingface/hub/models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots/*/; do [ -f "$d/config.json" ] && echo "$d" && break; done)" TASK=entity ITERS=0 bun -e 'import("./scripts/chunk-finetune.ts")'` prints `model=gemma-4-e4b-it-OptiQ-4bit`, `data=…/memory-train/entity`, `adapter=…/adapters/memory-entity`; running the same with `MODEL` UNSET resolves the e4b snapshot (NEVER MiniCPM5) — assert the printed `model=` is `gemma-4-e4b-it-OptiQ-4bit`; omitting `TASK` keeps the existing chunk defaults; `tsc` clean.
-
-**P10-T1 · Train `memory-synthesis` (highest value)** — TAG: USER-ACTION/GPU-GATED · DEPENDS-ON: P7-T2, P10-T0
-- DELIVERABLE: Build section-granular SFT data into `~/.cache/mlx-bun/memory-train/synthesis/{train,valid}.jsonl` using the B4 `messages` schema and the B9 triples (chunk identity/span recovered from Lucien's structured `chunks`/`chunk_buckets`/`synthesized_bucket_chunks` tables, section before/after from git history — per the B9 rule; each `(chunk, section_before, section_after)` = one example; user = section_before + chunkText + footnote contract; assistant = section_after). **USER-ACTION command (env-driven — verified against `chunk-finetune.ts`):**
-  ```
-  MODEL="$(for d in ~/.cache/huggingface/hub/models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots/*/; do [ -f "$d/config.json" ] && echo "$d" && break; done)" \
-  TASK=synthesis DATA=~/.cache/mlx-bun/memory-train/synthesis \
-  ADAPTER=~/.cache/mlx-bun/adapters/memory-synthesis \
-  SEQ=8192 ITERS=300 RANK=16 LR=1e-5 SEG=2 \
-  bun scripts/chunk-finetune.ts
-  ```
-  Then `callLocal("synthesis", …)` auto-loads it. (Josh launches; the agent must not start GPU work.)
-- ACCEPTANCE: `scripts/memory/eval-patch.ts` gate pass-rate + citation-preservation beat base by a margin justifying the adapter; checkpoint selected by the integration-quality judge.
-
-**P10-T2 · Train `memory-section` (conditional on P7-T1 miss)** — TAG: USER-ACTION/GPU-GATED · DEPENDS-ON: P7-T1, P4-T2, P10-T0
-- DELIVERABLE: One `messages` example per `(chunk, section)` binary (assistant = `yes`/`no`) from the Lucien gold `chunk_buckets`→section edges, into `~/.cache/mlx-bun/memory-train/section/`. USER-ACTION: same command shape with `TASK=section ADAPTER=~/.cache/mlx-bun/adapters/memory-section`. Trigger only if oracle-F1 (P4-T2) or Jaccard (P7-T1) misses.
-- ACCEPTANCE: oracle-F1 improves enough to justify; checkpoints by F1.
-
-**P10-T3 · Train `memory-entity` (conditional on P5-T3 miss)** — TAG: USER-ACTION/GPU-GATED · DEPENDS-ON: P5-T3, P10-T0
-- DELIVERABLE: One `messages` example per `(chunk → gold entity list)` (assistant = newline list) into `~/.cache/mlx-bun/memory-train/entity/`. USER-ACTION: same command shape with `TASK=entity ADAPTER=~/.cache/mlx-bun/adapters/memory-entity`. Trigger only if P5-T3 recall < ~0.80 or canonicalization fails.
-- ACCEPTANCE: entity F1 beats base by a justifying margin; checkpoints by F1.
-
-**P10-T4 · Train `memory-editor` (conditional on P8 miss)** — TAG: USER-ACTION/GPU-GATED · DEPENDS-ON: P8-T1, P8-T2, P10-T0
-- DELIVERABLE: Shared improve/infobox/structure adapter `messages` data into `~/.cache/mlx-bun/memory-train/editor/`, only if base misfires. USER-ACTION: same command shape with `TASK=editor ADAPTER=~/.cache/mlx-bun/adapters/memory-editor`.
-- ACCEPTANCE: editorial gate pass-rate / infobox-field accuracy beats base.
-
----
-
-## MILESTONES (thesis-proving checkpoints)
-- **M1 — Foundation solid (after P2):** `reindex` rebuilds all derived tables deterministically from markdown; `resolveName` redirects work; infobox `mount: [[L-Mount]]` produces a `via='infobox'` edge. Proves vault=truth / DB=cache.
-- **M2 — READ thesis proven (after P3):** the north-star lens query is answered locally, silently, by **find → read → answer** (find the right article via category/name/link, read it, answer from its synthesized content) in **< 5 KB, zero embeddings (tripwire==0), no read > 2 KB** — against hand-built articles. The measurable inverse of the 150 KB / 90-hit-substring / 42×-blob failure. (The read path is the easy half; the BIG ONE is M3 — the write path keeping the article current.)
-- **M3 — WRITE thesis proven (after P6):** a real pi conversation is synthesized into a `Panasonic_Lumix_S5IIX` article whose infobox carries `mount: [[L-Mount]]`, and M2's north-star query then answers against the *synthesized* article. The write path manufactures the read-path fixture.
-- **M4 — Full acceptance + migration (after P9):** `eval-northstar.ts` passes ≥5/6 on the bootstrapped vault; `eval-coverage.ts` shows 0 lost engaged-with things; ≤40 sharp entity articles replace 385 bins.
-
-## RISKS LEDGER
-- **R1 — Lexical name-shortlist recall collapses in the 122-bucket tail (<5 chunks).** Mitigation: P4-T1 measures recall per size-bin BEFORE ROUTE is built; if a bin < 0.80, add a narrow embedding-fallback shortlister (offline instrument, never the router). Gate: P4-T2.
-- **R2 — Base model can't extract/canonicalize entities (P5-T3 misses).** Mitigation: over-extraction is cheap (Stage-disambiguation catches it); train `memory-entity` (P10-T3) only on a measured recall miss.
-- **R3 — PATCH corrupts the vault (drops citations / over-rewrites).** Mitigation: the conservative gate (P6-T1) makes any weak output a NO-OP; deterministic gates are the eval metric (P7-T2 requires 100%). The vault is git-committed and reindexable.
-- **R4 — The model dumps whole articles (regresses to the blob pathology).** Mitigation: `memory_read` size guard (P3-T4) + per-tool byte caps + P9-T2's `max_read_kb < 2` hard fail.
-- **R5 — Cold-start over-creates (sprawling mediocre articles).** Mitigation: the single baked CREATE rule (P1-T5/P5-T4: ≥3 chunks OR owned/decided) + tail parks in staging; P9-T3 caps the first run at ≤40 articles — under-cover is success.
-- **R6 — Section-route has no native ground truth.** Mitigation: P7-T1 builds a frozen LLM-labeled proxy gold from the dozen large `~/Dreaming` articles' TOC + `synthesized_bucket_chunks`; scored by Jaccard not exact-match (83.4% single-edge → many-to-many is thin).
-- **R7 — GPU training blocks progress.** Mitigation: P10 is gated and LATE; everything through M4 is base+policy NOW work. Training and the full-corpus bootstrap (P6-T5) are USER-ACTION (Josh launches; the agent never starts GPU work or long-running servers).
-- **R8 — Embeddings creep onto the hot path.** Mitigation: the `embedCallCount` tripwire in `src/embed.ts` (P0-T6) that P9-T2 asserts == 0; embeddings are confined to P4 (recall instrument) and P8-T4 (Talk-page triage).