mlx-bun 0.0.7 → 0.0.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +112 -20
- package/docs/design/batching-v2-plan.md +515 -0
- package/docs/design/bucketing-stage.md +196 -0
- package/docs/design/compat-cli-surface-design.md +11 -11
- package/docs/design/diffusion-gemma-port.md +602 -0
- package/docs/design/docs-reorg-plan.md +1 -1
- package/docs/design/dreaming-nightly-pipeline.md +291 -0
- package/docs/design/dspark-speculative-decoding.md +106 -0
- package/docs/design/generic-model-support.md +381 -0
- package/docs/design/memory-inference-path.md +422 -0
- package/docs/design/memory-synthesis.md +287 -0
- package/docs/design/minicpm5-decode-megakernel.md +484 -0
- package/docs/design/mlx-lm-tool-parity-plan.md +535 -0
- package/docs/design/orpo-dynamic-lambda.md +41 -0
- package/docs/design/orpo-training.md +35 -5
- package/docs/design/parallel-slots.md +12 -1
- package/docs/design/parity-tier-dag.md +13 -5
- package/docs/design/spec-decode-larger-targets.md +229 -0
- package/docs/design/tauri-desktop-app.md +2 -2
- package/docs/design/the-dreaming-handoff.md +141 -0
- package/docs/design/the-dreaming-master-plan.md +446 -0
- package/docs/design/write-pipeline-entity.md +468 -0
- package/docs/investigations/curve-bisector-routing.md +1 -1
- package/docs/investigations/decode-roofline-lookagain.md +262 -0
- package/docs/investigations/dspark-handoff.md +58 -0
- package/docs/investigations/kernel-perf-review-2026-07.md +92 -0
- package/docs/planning/PRODUCT_ROADMAP.md +2 -2
- package/docs/planning/curve-sampler-research-plan.md +331 -0
- package/docs/planning/memory-docs-and-dag-plan.md +185 -0
- package/docs/planning/pi-integration-review.md +178 -0
- package/docs/planning/release-notes-v0.0.9.md +122 -0
- package/docs/planning/web-ui-pass-plan.md +213 -0
- package/docs/planning/website-readme-pass-plan.md +224 -0
- package/docs/reference/cli.md +255 -0
- package/docs/reference/embedding.md +7 -2
- package/docs/reference/library-api.md +22 -3
- package/docs/reference/memory.md +5 -5
- package/docs/reference/models.md +137 -0
- package/docs/reference/orpo-quickstart.md +6 -0
- package/docs/reference/server-api.md +187 -9
- package/docs/reference/server-config.md +40 -9
- package/docs/reference/training.md +25 -11
- package/package.json +1 -1
- package/src/anthropic.ts +22 -0
- package/src/assets/curve-designer.html +1 -1
- package/src/chat-template.ts +25 -7
- package/src/cli.ts +1029 -86
- package/src/config.ts +54 -3
- package/src/diffusion/diffusion-generate.ts +436 -0
- package/src/download.ts +99 -8
- package/src/embed.ts +77 -0
- package/src/eval/perplexity.ts +164 -0
- package/src/eval/runner.ts +19 -7
- package/src/eval/tasks/bfcl.ts +1 -1
- package/src/generate.ts +196 -19
- package/src/harness-pi.ts +43 -17
- package/src/index.ts +2 -0
- package/src/lora.ts +2 -2
- package/src/memory/article.ts +558 -0
- package/src/memory/chunk-validation.ts +150 -0
- package/src/memory/chunk.ts +305 -12
- package/src/memory/cluster.ts +344 -17
- package/src/memory/coverage.ts +134 -0
- package/src/memory/crosslink.ts +0 -0
- package/src/memory/db.ts +184 -45
- package/src/memory/entity.ts +203 -0
- package/src/memory/gate.ts +187 -0
- package/src/memory/history.ts +152 -0
- package/src/memory/ingest.ts +324 -11
- package/src/memory/model.ts +321 -0
- package/src/memory/normalize.ts +671 -0
- package/src/memory/parse.ts +22 -0
- package/src/memory/pipeline.ts +242 -30
- package/src/memory/prompts.ts +42 -0
- package/src/memory/query.ts +334 -0
- package/src/memory/reconcile.ts +734 -0
- package/src/memory/redirect.ts +108 -0
- package/src/memory/reindex.ts +325 -0
- package/src/memory/resolve.ts +307 -0
- package/src/memory/route.ts +272 -0
- package/src/memory/schedule.ts +6 -4
- package/src/memory/stages.ts +774 -0
- package/src/memory/synthesize.ts +1629 -20
- package/src/memory/tools.ts +446 -96
- package/src/memory/vault.ts +462 -35
- package/src/memory/wikify.ts +1057 -16
- package/src/mlx/ffi.ts +11 -0
- package/src/mlx/ops.ts +113 -0
- package/src/model/compiled-decode.ts +43 -8
- package/src/model/diffusion-gemma.ts +1026 -0
- package/src/model/factory.ts +35 -8
- package/src/model/flash-attention.ts +4 -2
- package/src/model/fused-decode-kernel.ts +14 -7
- package/src/model/fused-mlp-kernel.ts +169 -0
- package/src/model/fused-swiglu-kernel.ts +240 -0
- package/src/model/gemma4-base.ts +62 -6
- package/src/model/gemma4.ts +33 -2
- package/src/model/minicpm5.ts +106 -8
- package/src/model/qwen3.ts +276 -0
- package/src/model/steel-linear-kernel.ts +96 -0
- package/src/model/support.ts +45 -7
- package/src/model/universal/archs.ts +332 -0
- package/src/model/universal/dense.ts +576 -0
- package/src/model/universal/modules.ts +163 -0
- package/src/model/universal/rope.ts +366 -0
- package/src/pi-provider.ts +1 -1
- package/src/pi-terminal.ts +29 -8
- package/src/pi-web.ts +19 -2
- package/src/quantize/config-writer.ts +1 -1
- package/src/registry.ts +215 -7
- package/src/responses.ts +23 -0
- package/src/sampler.ts +220 -9
- package/src/serve/batch-scheduler.ts +272 -80
- package/src/serve/generation-gateway.ts +90 -18
- package/src/server.ts +643 -50
- package/src/spec/drafter.ts +81 -49
- package/src/spec/dspark/data-dflash.ts +156 -0
- package/src/spec/dspark/data.ts +201 -0
- package/src/spec/dspark/generate-dflash.ts +193 -0
- package/src/spec/dspark/generate.ts +290 -0
- package/src/spec/dspark/loss.ts +213 -0
- package/src/spec/dspark/module-dflash.ts +361 -0
- package/src/spec/dspark/module.ts +540 -0
- package/src/spec/dspark/sample.ts +112 -0
- package/src/spec/generate.ts +40 -19
- package/src/status-page.html +1 -1
- package/src/tokenizer.ts +7 -0
- package/src/train/diffusion-lora.ts +216 -0
- package/src/train/flash-cce.ts +70 -15
- package/src/train/fuse.ts +243 -0
- package/src/train/index.ts +3 -0
- package/src/train/job.ts +8 -0
- package/src/train/loss.ts +289 -30
- package/src/train/prefix-shared.ts +33 -5
- package/src/train/segmented.ts +67 -11
- package/src/train/trainer.ts +92 -28
- package/src/vision/diffusion-vision.ts +620 -0
- package/src/web/skills/memory/SKILL.md +40 -17
|
@@ -0,0 +1,291 @@
|
|
|
1
|
+
# The Dreaming — nightly memory pipeline (CANONICAL design)
|
|
2
|
+
|
|
3
|
+
Status: **canonical**, 2026-07-01. This document states the system design as
|
|
4
|
+
settled with Josh and **supersedes any conflicting statement** in
|
|
5
|
+
[the-dreaming-master-plan.md](./the-dreaming-master-plan.md) (task breakdown still
|
|
6
|
+
useful; several decisions overridden below),
|
|
7
|
+
[memory-synthesis.md](./memory-synthesis.md) (superseded), and
|
|
8
|
+
[the-dreaming-handoff.md](./the-dreaming-handoff.md) (session log, mostly aligned).
|
|
9
|
+
|
|
10
|
+
The system is the second iteration of Lucien's memory pipeline, rebuilt to run
|
|
11
|
+
**entirely on local models** inside mlx-bun (`src/memory/`). Same methodology and
|
|
12
|
+
pipeline at the core; the local constraint drives two design consequences:
|
|
13
|
+
|
|
14
|
+
1. **Small tasks.** Every AI step is decomposed to the smallest single-purpose
|
|
15
|
+
unit a small local model can do well — one chunk, one bucket, one section,
|
|
16
|
+
ideally a binary judgment. Never a whole-article blob in or out.
|
|
17
|
+
**And every AI step is pure text-in/text-out — no tool calls, no agent
|
|
18
|
+
loop.** The model never edits a file, never searches, never decides what to
|
|
19
|
+
do next; the pipeline hands it exactly the context it needs and applies its
|
|
20
|
+
output deterministically. This lowers the capability bar dramatically: a
|
|
21
|
+
small model that could never drive an agentic edit session can absolutely
|
|
22
|
+
rewrite one section given the section and the new information — and pure
|
|
23
|
+
completion tasks are exactly what per-stage LoRA SFT trains well.
|
|
24
|
+
2. **Deterministic gates between AI steps.** AI does judgment; deterministic
|
|
25
|
+
code enforces invariants (citations, footnotes, wikilinks, infobox shape,
|
|
26
|
+
word floors). A weak AI output is gated to NO-OP, never allowed to corrupt
|
|
27
|
+
the vault.
|
|
28
|
+
|
|
29
|
+
## Fixed principles (do not re-derive, do not contradict)
|
|
30
|
+
|
|
31
|
+
- **Conversations are the base.** The pipeline's only input is conversation
|
|
32
|
+
transcripts (pi sessions daily; historical imports are a one-time onboarding
|
|
33
|
+
path through the same code).
|
|
34
|
+
- **No notability filter.** *"If I talk about it, it's notable."* Everything
|
|
35
|
+
surfaces; nothing is dropped or parked as trivia. Any prior "owned/decided/≥3
|
|
36
|
+
chunks else PARK" gate is dead — the catch-all bucket exists only so that
|
|
37
|
+
low-volume topics remain searchable until they warrant their own article.
|
|
38
|
+
- **No embeddings, anywhere in the project.** Not for routing, not for
|
|
39
|
+
clustering, not as "offline instruments." Grouping and routing are done by
|
|
40
|
+
deterministic name/alias/category lookup plus small local-model judgments.
|
|
41
|
+
- **Articles are real single-topic wiki articles with sections** — one subject
|
|
42
|
+
per article, a lead, an infobox, `##` sections, See also, References. All
|
|
43
|
+
subject mentions are `[[wikilinks]]`.
|
|
44
|
+
- **Vault = truth, DB = derived cache.** Markdown in the vault is canonical and
|
|
45
|
+
git-tracked; the SQLite store is pipeline state + rebuildable index
|
|
46
|
+
(`reindex`).
|
|
47
|
+
- **Self-healing through ordinary use.** Corrections arrive as ordinary chunks;
|
|
48
|
+
chronological processing (`updated_at ASC`) resolves claims toward the user's
|
|
49
|
+
latest position.
|
|
50
|
+
- **Prompts and Meta pages are SCHEMATIC** (no real names/specs/dates) — the
|
|
51
|
+
small model parrots concrete example values into unrelated articles.
|
|
52
|
+
- **Runs nightly**, locally, unattended (launchd, `src/memory/schedule.ts`).
|
|
53
|
+
- **Model evals are not part of this design.** The eval harnesses (chunk-eval,
|
|
54
|
+
judges, F1/Jaccard gates, cloud-judge sampling) are tooling for *choosing and
|
|
55
|
+
training the models used by* the pipeline. They live beside the system, never
|
|
56
|
+
inside it, and belong in their own docs.
|
|
57
|
+
|
|
58
|
+
## The pipeline
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
conversations (ingest, watermark/cursor-gated)
|
|
62
|
+
│
|
|
63
|
+
▼ SEGMENT ──────────────── AI: find the topics in each conversation
|
|
64
|
+
chunks: single-topic segments with labels (pointer ranges, no text copies)
|
|
65
|
+
│
|
|
66
|
+
▼ BUCKET ───────────────── AI + deterministic: group similar topics
|
|
67
|
+
buckets: one per STORY/TOPIC; everything discussed about that topic since the
|
|
68
|
+
last run accumulates here (a bucket usually has NO article of its own)
|
|
69
|
+
│
|
|
70
|
+
▼ RESOLVE ──────────────── which articles does each story affect?
|
|
71
|
+
bucket delta → the subject articles it touches (update · create · sections)
|
|
72
|
+
│
|
|
73
|
+
▼ SYNTHESIZE ───────────── AI: per-section create/patch, bounded in and out
|
|
74
|
+
create: outline → per-section draft · update: patch one section, rest byte-identical
|
|
75
|
+
│
|
|
76
|
+
▼ ── deterministic gates ── citation survival · footnote bijection ·
|
|
77
|
+
│ word floor · wikilink resolution · weak output → NO-OP
|
|
78
|
+
│
|
|
79
|
+
▼ RECONCILE ────────────── infobox = ground truth; lead/body/infobox agree
|
|
80
|
+
▼ LINK ─────────────────── cross-link: mentions → [[links]], co-occurrence → See also
|
|
81
|
+
▼ NORMALIZE ────────────── deterministic: footnote renumber, wikilink canonicalize,
|
|
82
|
+
│ structure guards (References last, stub marking)
|
|
83
|
+
▼ EDITORIAL (wikify) ───── AI: remove redundancy, organize, make it read well —
|
|
84
|
+
│ per-section improve + structural pass over the TOC
|
|
85
|
+
▼
|
|
86
|
+
commit to vault (git) + run changelog
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Every stage is an independent, resumable, DB-state-keyed worker
|
|
90
|
+
(`src/memory/stages.ts`), runnable separately or as the full DAG
|
|
91
|
+
(`mlx-bun memory synthesize`).
|
|
92
|
+
|
|
93
|
+
### SEGMENT — conversations → topics
|
|
94
|
+
|
|
95
|
+
The e4b chunk adapter (`e4b-chunk-300`, symlinked `memory-chunk`) segments each
|
|
96
|
+
conversation into single-topic chunks with short labels. One chunk = one
|
|
97
|
+
coherent topic; a multi-stage process is ONE topic; long single-topic
|
|
98
|
+
conversations stay whole. Chunks are pointer ranges over `messages` — text is
|
|
99
|
+
materialized exactly once. Deterministic anchor repair validates every boundary.
|
|
100
|
+
|
|
101
|
+
### BUCKET — group similar topics (first-class stage)
|
|
102
|
+
|
|
103
|
+
**The Wikipedia-editor analogy (this is the model — Josh's framing).** A news
|
|
104
|
+
story gets reported seven times in a day — Fox says one thing, MSNBC another,
|
|
105
|
+
CBS a third, each report carrying a bit of different information. A Wikipedia
|
|
106
|
+
editor who heard ALL of the reports incorporates everything they learned into
|
|
107
|
+
every article the story affects. In our system: the news shows are
|
|
108
|
+
**conversations**, the stories are **topics (chunks)**, and the **bucket is the
|
|
109
|
+
editor's brain** — the place where everything reported TODAY about THAT TOPIC
|
|
110
|
+
accumulates before any article is touched.
|
|
111
|
+
|
|
112
|
+
**A bucket is keyed by the STORY/TOPIC, not by a subject.** The worked example:
|
|
113
|
+
the topic is *"2026 Lincoln Memorial Reflecting Pool controversy"* (the algae
|
|
114
|
+
issue in the pool that Trump renovated). Every report about it that day goes in
|
|
115
|
+
that one bucket. There is **no article for that topic** — and there may never
|
|
116
|
+
be — but the story affects THREE articles: *Lincoln Memorial Reflecting Pool*,
|
|
117
|
+
*Second presidency of Donald Trump*, and *John J. Cafaro*. Each article
|
|
118
|
+
receives the facet of the delta relevant to it (renovation/algae facts vs.
|
|
119
|
+
abuse-of-power framing vs. the contract angle). This is why bucket ≠ article is
|
|
120
|
+
**structural**: articles are THINGS (subjects); buckets are STORIES (what got
|
|
121
|
+
discussed); the story→things relationship is inherently many-to-many.
|
|
122
|
+
|
|
123
|
+
Assignment mechanics: group chunks by topic — normalize/compare chunk labels
|
|
124
|
+
deterministically first, with a small-model binary judgment ("are these two
|
|
125
|
+
chunks about the same story? yes/no") for near-misses, and a new-bucket
|
|
126
|
+
proposal for genuinely new topics. The taxonomy is emergent: buckets come into
|
|
127
|
+
existence because the user talked about something, never from a predefined
|
|
128
|
+
list. Buckets are naturally short-lived relative to articles — a story
|
|
129
|
+
accumulates while it's live and goes quiet when the user stops discussing it;
|
|
130
|
+
the articles it fed persist.
|
|
131
|
+
|
|
132
|
+
`_captured` is the reserved catch-all bucket: chunks whose subject doesn't yet
|
|
133
|
+
justify assignment stay there, searchable, and are revisited on later runs —
|
|
134
|
+
**nothing is dropped**.
|
|
135
|
+
|
|
136
|
+
### RESOLVE — bucket deltas → articles/sections
|
|
137
|
+
|
|
138
|
+
The unit of work from here on is the **bucket delta**: all of a bucket's new
|
|
139
|
+
(un-synthesized) chunks since the last run, taken TOGETHER. For each non-empty
|
|
140
|
+
delta, RESOLVE answers the editor's question: **"which articles does this story
|
|
141
|
+
affect?"** (the Reflecting Pool story → the pool article, the presidency
|
|
142
|
+
article, the contractor article). For each affected subject:
|
|
143
|
+
|
|
144
|
+
- **The article exists** → UPDATE. Section-route the delta's *facet for that
|
|
145
|
+
article* via binary per-section judgments over the TOC ("does this new
|
|
146
|
+
information update THIS section? yes/no"); all-no → propose a new section.
|
|
147
|
+
- **The article doesn't exist yet** → CREATE the subject article (a THING —
|
|
148
|
+
never an article named after the story itself). Thin is fine — 2–3 chunks
|
|
149
|
+
make a legitimate stub (stubs are a feature, not a failure).
|
|
150
|
+
|
|
151
|
+
The per-section ledger (`synthesized_chunk_sections`) records which chunks each
|
|
152
|
+
fold incorporated, making every fold idempotent and every claim traceable to
|
|
153
|
+
its source chunks.
|
|
154
|
+
|
|
155
|
+
### SYNTHESIZE — one edit per (bucket, article), never chunk-at-a-time
|
|
156
|
+
|
|
157
|
+
**The cardinal rule:** synthesis is the editor sitting down ONCE per article
|
|
158
|
+
with everything the bucket learned. Never patch a section from one chunk and
|
|
159
|
+
then re-patch it from the next chunk five minutes later — sequential
|
|
160
|
+
single-chunk edits overwrite each other, churn citations, and produce
|
|
161
|
+
patchwork prose. The whole point of the bucket layer is that the delta arrives
|
|
162
|
+
at the article **already aggregated**.
|
|
163
|
+
|
|
164
|
+
- **CREATE**: group the bucket's chunks into proto-sections (by label/subject
|
|
165
|
+
affinity — small-model judgment, no embeddings) → outline (title, lead,
|
|
166
|
+
section names) → draft each section from only its chunks. The model never
|
|
167
|
+
holds a whole article in its head.
|
|
168
|
+
- **UPDATE (patch)**: hand the model one section + the delta's facet relevant
|
|
169
|
+
to that article and section (all of it, with per-conversation `[^N]` citation
|
|
170
|
+
material); it rewrites that one section once, weaving everything in. Facet
|
|
171
|
+
extraction is implicit: the same story becomes algae/renovation facts in the
|
|
172
|
+
Reflecting Pool article and framing in the presidency article, because the
|
|
173
|
+
model writes what's relevant to the section it's editing.
|
|
174
|
+
Contradiction = correction: resolve toward the latest statement, including
|
|
175
|
+
contradictions *within* the delta (latest chunk wins — the editor heard all
|
|
176
|
+
seven reports and writes the reconciled version, not seven amendments).
|
|
177
|
+
User's evolving positions keep a `## History` trace; AI factual errors are
|
|
178
|
+
silently overwritten.
|
|
179
|
+
- **Edit mechanics (SETTLED 2026-07-01, Josh-confirmed — and how Lucien always
|
|
180
|
+
worked): the model never edits the article; it is handed the current section
|
|
181
|
+
+ the new information and outputs a complete REPLACEMENT SECTION.** Not an
|
|
182
|
+
in-place file edit (the
|
|
183
|
+
model would have to hold the whole article — the exact blob failure this
|
|
184
|
+
design avoids) and not a diff (small local models produce brittle patch
|
|
185
|
+
formats, and a diff is hard to gate). A full replacement section is bounded
|
|
186
|
+
in and out, trivially appliable (`extractSection` swap — the rest of the file
|
|
187
|
+
byte-identical by construction), and directly gateable: the gate compares old
|
|
188
|
+
section vs. new section for word floor and citation survival. Deterministic
|
|
189
|
+
code performs the actual write. CREATE works the same way — the model emits
|
|
190
|
+
one section at a time; code assembles the article.
|
|
191
|
+
- **Multiple buckets converging on one article** (the algae-controversy bucket
|
|
192
|
+
AND the olympian-arrest bucket both affect *Reflecting Pool*): each bucket's
|
|
193
|
+
facet folds as its own section-level edit within the run — different stories
|
|
194
|
+
usually land in different sections. If two buckets touch the SAME section,
|
|
195
|
+
fold chronologically (latest wins, per self-healing) and let the nightly
|
|
196
|
+
EDITORIAL pass smooth the seam; the editorial pass runs after all folds, so
|
|
197
|
+
the article the user reads next morning is one coherent account.
|
|
198
|
+
- **Conservative write gate** on every output: ≥70% word floor on the touched
|
|
199
|
+
section, every pre-existing `conv:` citation survives, H1/References survive,
|
|
200
|
+
footnote bijection. Weak output → NO-OP; the delta retries next run.
|
|
201
|
+
|
|
202
|
+
### RECONCILE · LINK · NORMALIZE — nightly deterministic/structural stages
|
|
203
|
+
|
|
204
|
+
These run **every night**, not periodically:
|
|
205
|
+
|
|
206
|
+
- **RECONCILE** (`reconcile.ts`): the infobox is the article's ground truth;
|
|
207
|
+
lead, body sections, and infobox must agree; date-aware (latest wins).
|
|
208
|
+
Infobox extraction/refresh is part of this — every article carries a current,
|
|
209
|
+
parseable infobox whose entity-valued fields are `[[wikilinks]]`.
|
|
210
|
+
- **LINK** (`crosslink.ts`): cross-linking is a dedicated stage, NOT
|
|
211
|
+
synthesis's job (synthesis sees one article and cannot know the others).
|
|
212
|
+
Mention-based prose links + co-occurrence See-also. Idempotent. Ensures **all
|
|
213
|
+
subjects are wikilinked** across the vault.
|
|
214
|
+
- **NORMALIZE** (`normalize.ts`): pure code — footnote renumber/merge, wikilink
|
|
215
|
+
canonicalization, orphan repair, structure guards (`## References` last,
|
|
216
|
+
`{{stub}}`/`{{Main}}` recognition, title == H1 == stem).
|
|
217
|
+
|
|
218
|
+
### EDITORIAL (wikify) — make the article good
|
|
219
|
+
|
|
220
|
+
The final AI pass, and the reason articles stay readable instead of accreting
|
|
221
|
+
patches: per-section "tighten this — remove redundancy, keep every citation and
|
|
222
|
+
every specific detail" plus a structural pass over the TOC (reorder sections,
|
|
223
|
+
refresh the lead, split a section that outgrew the article into its own article
|
|
224
|
+
with a `{{Main}}` summary, merge duplicate stubs). Behind the same conservative
|
|
225
|
+
gate; a weak edit is rejected and the original kept.
|
|
226
|
+
|
|
227
|
+
## Nightly run contract
|
|
228
|
+
|
|
229
|
+
- Trigger: launchd (`com.mlx-bun.memory`), logs under `~/.mlx-bun/logs/`.
|
|
230
|
+
- Scope: only conversations past the per-source watermark — a normal night is
|
|
231
|
+
minutes of work, not hours.
|
|
232
|
+
- Inference: one resident base model (Gemma-4-e4b), per-stage adapter mounted
|
|
233
|
+
iff `~/.cache/mlx-bun/adapters/memory-<stage>` exists, else base+policy.
|
|
234
|
+
Route generation through the persistent server (load-once) when available;
|
|
235
|
+
no-server fallback works but pays model load per invocation.
|
|
236
|
+
- Every article change is a git commit; the run ends with a changelog summary.
|
|
237
|
+
- Failure isolation: stages are resumable; a failed bucket/article never blocks
|
|
238
|
+
the rest of the run.
|
|
239
|
+
|
|
240
|
+
## Decisions overridden from the master plan (explicit, so no agent re-derives them)
|
|
241
|
+
|
|
242
|
+
| Master-plan statement | Superseded by |
|
|
243
|
+
| --- | --- |
|
|
244
|
+
| Personal-notability checklist; CREATE gate "≥3 chunks OR owned/decided, else PARK as trivia"; ≤40-article first-run cap (P1-T5, P5-T4, R5, P9-T3) | **No notability filter.** Everything surfaces; `_captured` keeps the tail searchable; thin buckets → stubs. |
|
|
245
|
+
| Buckets demoted to "optional staging, may ship empty" (L4) | **Buckets are first-class**: the per-subject accumulation layer between chunks and articles; synthesis consumes buckets. |
|
|
246
|
+
| Embeddings allowed as offline instruments (P4 recall tripwire experiments, P8-T4 silhouette triage) | **No embeddings anywhere in the project.** The tripwire (`getEmbedCounter()==0`) remains as a guard, permanently. |
|
|
247
|
+
| Routing/section-route acceptance gated on Lucien bucket-F1 / Jaccard (P4-T2, P7-T1) | Model-selection evals live **outside** the system design; pipeline health is judged on real output quality, never Lucien's bin taxonomy. |
|
|
248
|
+
| Cross-linking + infobox work assigned to periodic WIKIFY (P8) | RECONCILE + LINK are **nightly stages** in the DAG; editorial wikify is the readability pass. |
|
|
249
|
+
|
|
250
|
+
## Work plan (docs-only reconciliation done; implementation next)
|
|
251
|
+
|
|
252
|
+
Ordered; none of it touches the running full-corpus import
|
|
253
|
+
(`~/.mlx-bun/wiki-full`, cursor file `~/.mlx-bun/full-run-cursor.txt`) or the
|
|
254
|
+
uncommitted `src/model/universal` / trainer work.
|
|
255
|
+
|
|
256
|
+
1. **Bucket layer made explicit** — a topic-keyed BUCKET stage between SEGMENT
|
|
257
|
+
and the article fan-out (`buckets` + `chunk_buckets` tables already exist):
|
|
258
|
+
group chunks by story (label normalization + binary "same story?"
|
|
259
|
+
judgments), `_captured` catch-all, and **delta-at-once synthesis** — the run
|
|
260
|
+
collects each bucket's new chunks, then makes ONE edit per (bucket, article).
|
|
261
|
+
The as-built entity-keyed ROUTE becomes RESOLVE (bucket → affected subject
|
|
262
|
+
articles); the per-chunk patch loop is replaced, not wrapped. (Settled
|
|
263
|
+
2026-07-01 — the Wikipedia-editor / Reflecting-Pool framing above; buckets
|
|
264
|
+
are STORIES, articles are THINGS, many-to-many by nature.)
|
|
265
|
+
2. **Nightly DAG order** — ensure `mlx-bun memory synthesize` runs
|
|
266
|
+
segment → bucket → resolve → synthesize → reconcile → link → normalize →
|
|
267
|
+
editorial every run (reconcile/link/infobox currently periodic/partial).
|
|
268
|
+
3. **Editorial stage** — wire the per-section improve + structural pass as the
|
|
269
|
+
closing stage behind the conservative gate.
|
|
270
|
+
4. **Persistent-server inference routing** — load-once for all memory
|
|
271
|
+
generation, no-server fallback (the in-progress inference-path rework;
|
|
272
|
+
finish, don't fork).
|
|
273
|
+
5. **Promote + schedule** — after the import is judged good: point
|
|
274
|
+
`~/.mlx-bun/wiki` at the bootstrapped content, enable the nightly launchd
|
|
275
|
+
fold-in from pi sessions.
|
|
276
|
+
6. **Known quality fixes** — schematic-prompt leaks (<1% of articles), run-on
|
|
277
|
+
leads, `owned:` framing on abstract concepts (reframe the lead spec per
|
|
278
|
+
subject kind).
|
|
279
|
+
|
|
280
|
+
## Open questions (for discussion with Josh)
|
|
281
|
+
|
|
282
|
+
- ~~Bucket granularity vs article granularity~~ — **SETTLED 2026-07-01**:
|
|
283
|
+
buckets are a separate layer from articles (many-to-many both ways); the
|
|
284
|
+
bucket is the editor's brain that aggregates the day's information about a
|
|
285
|
+
subject, and RESOLVE fans the delta out to every affected article. Bucket is
|
|
286
|
+
never a 1:1 alias for an article.
|
|
287
|
+
- **When does a `_captured` chunk graduate?** Re-scan `_captured` every night
|
|
288
|
+
for subjects that have since accumulated enough related chunks, or only when
|
|
289
|
+
a new bucket's name matches?
|
|
290
|
+
- **Editorial cadence.** Every night over every touched article, or touched
|
|
291
|
+
articles nightly + a slower full-vault sweep?
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# DSpark speculative decoding for mlx-bun
|
|
2
|
+
|
|
3
|
+
Implementation of DeepSeek's **DSpark** ("Confidence-Scheduled Speculative
|
|
4
|
+
Decoding with Semi-Autoregressive Generation", `DSpark_paper.pdf`) against this
|
|
5
|
+
repo's models. DSpark = a **DFlash** parallel backbone (KV-injected, all γ draft
|
|
6
|
+
tokens in one pass) **+ a lightweight sequential head** (Markov, Eq 5, or RNN,
|
|
7
|
+
Eq 6) that adds intra-block dependency (the *semi-autoregressive* part) **+ a
|
|
8
|
+
confidence head → STS calibration → hardware-aware prefix scheduler**. Decode
|
|
9
|
+
becomes draft → verify → accept; output is lossless (bit-identical to the target
|
|
10
|
+
at any temperature); only decode speed (τ) changes.
|
|
11
|
+
|
|
12
|
+
> **DFlash vs DSpark:** DFlash is *only* the parallel backbone (prior work) and
|
|
13
|
+
> suffers suffix decay. DSpark adds the sequential head + confidence-scheduled
|
|
14
|
+
> verification. The sequential Markov head is what makes this DSpark.
|
|
15
|
+
|
|
16
|
+
## Two implementations (flag-selected, both in `src/spec/dspark/`)
|
|
17
|
+
|
|
18
|
+
| | `module.ts` (v1) | `module-dflash.ts` (v2, **the real one**) |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| context | **single vector** (final hidden at the anchor) fused as one token | faithful **multi-layer KV injection** (Eq 2–3): H_ctx = RMSNorm(W_c·[H^{l1};…;H^{lm}]) over the FULL context, re-projected into every draft layer's K/V |
|
|
21
|
+
| layers | 2 | 5 |
|
|
22
|
+
| status | **superseded** — dropped the paper's core idea (kept as baseline) | the actual DSpark model (Markov variant) |
|
|
23
|
+
|
|
24
|
+
v1 was a shortcut that removed the load-bearing mechanism; it never tested the
|
|
25
|
+
paper. v2 is faithful. Checkpoints stamp `variant` in `dspark.json`; the runtime
|
|
26
|
+
loads the matching module. (`variant:"dflash"` should be renamed to `"dspark"` —
|
|
27
|
+
see Open items.)
|
|
28
|
+
|
|
29
|
+
## The mechanism (v2, paper Eq 2–3)
|
|
30
|
+
|
|
31
|
+
- **Eq 2 (context):** tap the target's hidden states at layers `{l1…lm}` (e4b
|
|
32
|
+
default {20,31,41,42}, the last = post-finalNorm), concat on the feature axis,
|
|
33
|
+
project to the draft width: `H_ctx ∈ [L_ctx, d]`. Extraction via a parity-safe
|
|
34
|
+
`model.hiddenTap` (no-op when off) added to `gemma4.ts` `forwardLayers`.
|
|
35
|
+
- **Eq 3 (injection):** every draft layer forms `K_i=[W_i^K H_ctx; W_i^K H_d]`,
|
|
36
|
+
`V_i` likewise, `Q_i=W_i^Q H_d`; the γ-block queries attend bidirectionally
|
|
37
|
+
over `[context ++ block]`. Context is read-only memory re-projected fresh each
|
|
38
|
+
layer. The draft projects the target's HIDDEN STATES with its OWN Wk/Wv (not
|
|
39
|
+
the target's K/V — that's the separate `GemmaAssistantDrafter` baseline).
|
|
40
|
+
- **Markov head (Eq 5):** `B_k = W1[x_{k-1}]·W2`, r=256, W2=0 init → starts as
|
|
41
|
+
pure DFlash, τ climbs as intra-block dependency is learned.
|
|
42
|
+
- **Confidence head (Eq 7):** `c_k = σ(wᵀ[h_k; W1[x_{k-1}]])`.
|
|
43
|
+
- **Loss (Eq 9–12):** `0.1·CE + 0.9·TV + 1.0·BCE`, position-weighted.
|
|
44
|
+
- **Lossless verify** (`generate-dflash.ts`): greedy = longest-prefix match;
|
|
45
|
+
temp>0 = `min(1,p/q)` accept + residual `norm(relu(p−q))` resample. Growing
|
|
46
|
+
multi-layer H_ctx buffer; rollback via `trim(n, bypass=true)` (physically
|
|
47
|
+
slices the rejected tips past the sliding window — see `gemma4-base.ts`).
|
|
48
|
+
|
|
49
|
+
## Results & findings (measured, M1 Max 32GB, e4b OptiQ-4bit)
|
|
50
|
+
|
|
51
|
+
**The architecture is CORRECT — proven by an overfit test.** Trained on 3
|
|
52
|
+
articles, eval on the same 3: per-position acceptance **~0.75**, τ **3.24** —
|
|
53
|
+
squarely paper-range (0.6–0.9 / τ 3–4). So the faithful DFlash, correctly
|
|
54
|
+
implemented, reaches the paper's numbers.
|
|
55
|
+
|
|
56
|
+
**Getting there required finding a real bug — an off-by-one in the p^t target
|
|
57
|
+
alignment** (the TV loss, weight 0.9). Block position k predicts `x_{t+1+k}`,
|
|
58
|
+
whose target distribution is `softmax(LM_head(h_{t+k}))`; the data gathered the
|
|
59
|
+
hidden at `t+k+1` (shifted forward one). Present in **both v1 and v2** — so every
|
|
60
|
+
τ number before the fix was trained against a shifted target. Fixed in
|
|
61
|
+
`data-dflash.ts` (`blockIdx = g+k`, k=0..γ-1). Before fix: per-pos ~0.08. After:
|
|
62
|
+
~0.17 generalizing, ~0.75 overfit.
|
|
63
|
+
|
|
64
|
+
**Two things stand between "correct architecture" and "net speedup," now
|
|
65
|
+
separated:**
|
|
66
|
+
1. **Data.** 160 articles → generalizes only to per-pos ~0.17 (plateaus, doesn't
|
|
67
|
+
climb); overfit on 3 proves capacity. The paper uses 1.3M samples × 10 epochs.
|
|
68
|
+
→ needs real data scale.
|
|
69
|
+
2. **Target speed (the decisive one).** Spec decode amortizes the fixed draft
|
|
70
|
+
overhead only when the *target forward* is expensive. Measured decode:
|
|
71
|
+
**e4b 45.9 tok/s, 12b 27.5 tok/s, 27B ~15 tok/s.** On fast e4b the draft
|
|
72
|
+
overhead dominates → even τ≈3 nets ~0.4–0.5× (SLOWER). On a slow target (27B,
|
|
73
|
+
~67 ms/forward) that overhead is a small fraction → the same τ≈3 could land
|
|
74
|
+
~2–3× (15 → ~35–45 tok/s). **e4b is close to the worst case for spec decode;
|
|
75
|
+
the 27B agentic workload is the ideal target and the real motivation.**
|
|
76
|
+
|
|
77
|
+
Also confirmed: width is NOT the ceiling (`dDraft=1024` and `2560` both plateau
|
|
78
|
+
~0.17 pre-scale); the parity-safe `hiddenTap` doesn't change target numerics;
|
|
79
|
+
long-context spec rollback works via the `trim(n, bypass)` physical-slice.
|
|
80
|
+
|
|
81
|
+
## Files
|
|
82
|
+
- `src/model/gemma4.ts` — `hiddenTap`/`captureLayer` (parity-safe multi-layer tap).
|
|
83
|
+
- `src/model/gemma4-base.ts` — `trim(n, bypass)` on rotating caches (spec rollback past the window).
|
|
84
|
+
- `src/spec/dspark/module-dflash.ts` — faithful DFlash+Markov+confidence module.
|
|
85
|
+
- `src/spec/dspark/data-dflash.ts` — multi-layer shards, variable-length prefix context, left-pad masking.
|
|
86
|
+
- `src/spec/dspark/generate-dflash.ts` — growing-H_ctx lossless decode loop.
|
|
87
|
+
- `src/spec/dspark/{loss,sample}.ts` — loss (Eq 9–12) + temp>0 sampling (shared with v1).
|
|
88
|
+
- `src/spec/dspark/{module,data,generate}.ts` — v1 single-vector (superseded baseline).
|
|
89
|
+
- `scripts/dspark-regen-dflash.ts` / `dspark-train-dflash.ts` (`--resume`) / `dspark-measure-dflash.ts`.
|
|
90
|
+
- `scripts/dspark-dflash-smoke.ts` — CPU smoke (16/16). `scripts/dspark-smoke.ts` — v1 smoke (33/33).
|
|
91
|
+
|
|
92
|
+
## Run sequence (faithful DFlash; GPU = Josh runs)
|
|
93
|
+
```
|
|
94
|
+
bun scripts/dspark-regen-dflash.ts --topics <topics.txt> --out <data> --max-resp 320
|
|
95
|
+
bun scripts/dspark-train-dflash.ts --data <data> --out <ckpt> --iters 8000 --batch 8 [--resume] [--ddraft 2560]
|
|
96
|
+
bun scripts/dspark-measure-dflash.ts --drafter <ckpt> --data <prompts.jsonl> # τ + tok/s vs vanilla
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## Open items / next
|
|
100
|
+
- **Real data scale** (thousands of on-distribution generations, not 160) to lift generalizing τ toward the overfit ~0.75.
|
|
101
|
+
- **Retarget to a slow model (27B/12b)** where the τ≈3 architecture actually nets a speedup — the whole point. Needs the drafter sized to that model's H/layers + regen+train there; 27B on 32GB is memory-tight (17.75GB weights + KV budget).
|
|
102
|
+
- **Tighten the draft inference loop** (per-position host syncs, double 262K LM-head) so τ translates to wall-clock.
|
|
103
|
+
- **Remaining paper components:** RNN head (Eq 6), STS calibration (§3.2.1), hardware-aware prefix scheduler (Alg 1 — single-user form = confidence-scheduled draft-length pruning).
|
|
104
|
+
- **Rename** `dflash`→`dspark` (the faithful module IS DSpark; v1 is the legacy single-vector variant).
|
|
105
|
+
|
|
106
|
+
Full session handoff: `docs/investigations/dspark-handoff.md`.
|