mlx-bun 0.0.7 → 0.0.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (138) hide show
  1. package/README.md +112 -20
  2. package/docs/design/batching-v2-plan.md +515 -0
  3. package/docs/design/bucketing-stage.md +196 -0
  4. package/docs/design/compat-cli-surface-design.md +11 -11
  5. package/docs/design/diffusion-gemma-port.md +602 -0
  6. package/docs/design/docs-reorg-plan.md +1 -1
  7. package/docs/design/dreaming-nightly-pipeline.md +291 -0
  8. package/docs/design/dspark-speculative-decoding.md +106 -0
  9. package/docs/design/generic-model-support.md +381 -0
  10. package/docs/design/memory-inference-path.md +422 -0
  11. package/docs/design/memory-synthesis.md +287 -0
  12. package/docs/design/minicpm5-decode-megakernel.md +484 -0
  13. package/docs/design/mlx-lm-tool-parity-plan.md +535 -0
  14. package/docs/design/orpo-dynamic-lambda.md +41 -0
  15. package/docs/design/orpo-training.md +35 -5
  16. package/docs/design/parallel-slots.md +12 -1
  17. package/docs/design/parity-tier-dag.md +13 -5
  18. package/docs/design/spec-decode-larger-targets.md +229 -0
  19. package/docs/design/tauri-desktop-app.md +2 -2
  20. package/docs/design/the-dreaming-handoff.md +141 -0
  21. package/docs/design/the-dreaming-master-plan.md +446 -0
  22. package/docs/design/write-pipeline-entity.md +468 -0
  23. package/docs/investigations/curve-bisector-routing.md +1 -1
  24. package/docs/investigations/decode-roofline-lookagain.md +262 -0
  25. package/docs/investigations/dspark-handoff.md +58 -0
  26. package/docs/investigations/kernel-perf-review-2026-07.md +92 -0
  27. package/docs/planning/PRODUCT_ROADMAP.md +2 -2
  28. package/docs/planning/curve-sampler-research-plan.md +331 -0
  29. package/docs/planning/memory-docs-and-dag-plan.md +185 -0
  30. package/docs/planning/pi-integration-review.md +178 -0
  31. package/docs/planning/release-notes-v0.0.9.md +122 -0
  32. package/docs/planning/web-ui-pass-plan.md +213 -0
  33. package/docs/planning/website-readme-pass-plan.md +224 -0
  34. package/docs/reference/cli.md +255 -0
  35. package/docs/reference/embedding.md +7 -2
  36. package/docs/reference/library-api.md +22 -3
  37. package/docs/reference/memory.md +5 -5
  38. package/docs/reference/models.md +137 -0
  39. package/docs/reference/orpo-quickstart.md +6 -0
  40. package/docs/reference/server-api.md +187 -9
  41. package/docs/reference/server-config.md +40 -9
  42. package/docs/reference/training.md +25 -11
  43. package/package.json +1 -1
  44. package/src/anthropic.ts +22 -0
  45. package/src/assets/curve-designer.html +1 -1
  46. package/src/chat-template.ts +25 -7
  47. package/src/cli.ts +1029 -86
  48. package/src/config.ts +54 -3
  49. package/src/diffusion/diffusion-generate.ts +436 -0
  50. package/src/download.ts +99 -8
  51. package/src/embed.ts +77 -0
  52. package/src/eval/perplexity.ts +164 -0
  53. package/src/eval/runner.ts +19 -7
  54. package/src/eval/tasks/bfcl.ts +1 -1
  55. package/src/generate.ts +196 -19
  56. package/src/harness-pi.ts +43 -17
  57. package/src/index.ts +2 -0
  58. package/src/lora.ts +2 -2
  59. package/src/memory/article.ts +558 -0
  60. package/src/memory/chunk-validation.ts +150 -0
  61. package/src/memory/chunk.ts +305 -12
  62. package/src/memory/cluster.ts +344 -17
  63. package/src/memory/coverage.ts +134 -0
  64. package/src/memory/crosslink.ts +0 -0
  65. package/src/memory/db.ts +184 -45
  66. package/src/memory/entity.ts +203 -0
  67. package/src/memory/gate.ts +187 -0
  68. package/src/memory/history.ts +152 -0
  69. package/src/memory/ingest.ts +324 -11
  70. package/src/memory/model.ts +321 -0
  71. package/src/memory/normalize.ts +671 -0
  72. package/src/memory/parse.ts +22 -0
  73. package/src/memory/pipeline.ts +242 -30
  74. package/src/memory/prompts.ts +42 -0
  75. package/src/memory/query.ts +334 -0
  76. package/src/memory/reconcile.ts +734 -0
  77. package/src/memory/redirect.ts +108 -0
  78. package/src/memory/reindex.ts +325 -0
  79. package/src/memory/resolve.ts +307 -0
  80. package/src/memory/route.ts +272 -0
  81. package/src/memory/schedule.ts +6 -4
  82. package/src/memory/stages.ts +774 -0
  83. package/src/memory/synthesize.ts +1629 -20
  84. package/src/memory/tools.ts +446 -96
  85. package/src/memory/vault.ts +462 -35
  86. package/src/memory/wikify.ts +1057 -16
  87. package/src/mlx/ffi.ts +11 -0
  88. package/src/mlx/ops.ts +113 -0
  89. package/src/model/compiled-decode.ts +43 -8
  90. package/src/model/diffusion-gemma.ts +1026 -0
  91. package/src/model/factory.ts +35 -8
  92. package/src/model/flash-attention.ts +4 -2
  93. package/src/model/fused-decode-kernel.ts +14 -7
  94. package/src/model/fused-mlp-kernel.ts +169 -0
  95. package/src/model/fused-swiglu-kernel.ts +240 -0
  96. package/src/model/gemma4-base.ts +62 -6
  97. package/src/model/gemma4.ts +33 -2
  98. package/src/model/minicpm5.ts +106 -8
  99. package/src/model/qwen3.ts +276 -0
  100. package/src/model/steel-linear-kernel.ts +96 -0
  101. package/src/model/support.ts +45 -7
  102. package/src/model/universal/archs.ts +332 -0
  103. package/src/model/universal/dense.ts +576 -0
  104. package/src/model/universal/modules.ts +163 -0
  105. package/src/model/universal/rope.ts +366 -0
  106. package/src/pi-provider.ts +1 -1
  107. package/src/pi-terminal.ts +29 -8
  108. package/src/pi-web.ts +19 -2
  109. package/src/quantize/config-writer.ts +1 -1
  110. package/src/registry.ts +215 -7
  111. package/src/responses.ts +23 -0
  112. package/src/sampler.ts +220 -9
  113. package/src/serve/batch-scheduler.ts +272 -80
  114. package/src/serve/generation-gateway.ts +90 -18
  115. package/src/server.ts +643 -50
  116. package/src/spec/drafter.ts +81 -49
  117. package/src/spec/dspark/data-dflash.ts +156 -0
  118. package/src/spec/dspark/data.ts +201 -0
  119. package/src/spec/dspark/generate-dflash.ts +193 -0
  120. package/src/spec/dspark/generate.ts +290 -0
  121. package/src/spec/dspark/loss.ts +213 -0
  122. package/src/spec/dspark/module-dflash.ts +361 -0
  123. package/src/spec/dspark/module.ts +540 -0
  124. package/src/spec/dspark/sample.ts +112 -0
  125. package/src/spec/generate.ts +40 -19
  126. package/src/status-page.html +1 -1
  127. package/src/tokenizer.ts +7 -0
  128. package/src/train/diffusion-lora.ts +216 -0
  129. package/src/train/flash-cce.ts +70 -15
  130. package/src/train/fuse.ts +243 -0
  131. package/src/train/index.ts +3 -0
  132. package/src/train/job.ts +8 -0
  133. package/src/train/loss.ts +289 -30
  134. package/src/train/prefix-shared.ts +33 -5
  135. package/src/train/segmented.ts +67 -11
  136. package/src/train/trainer.ts +92 -28
  137. package/src/vision/diffusion-vision.ts +620 -0
  138. package/src/web/skills/memory/SKILL.md +40 -17
@@ -0,0 +1,291 @@
1
+ # The Dreaming — nightly memory pipeline (CANONICAL design)
2
+
3
+ Status: **canonical**, 2026-07-01. This document states the system design as
4
+ settled with Josh and **supersedes any conflicting statement** in
5
+ [the-dreaming-master-plan.md](./the-dreaming-master-plan.md) (task breakdown still
6
+ useful; several decisions overridden below),
7
+ [memory-synthesis.md](./memory-synthesis.md) (superseded), and
8
+ [the-dreaming-handoff.md](./the-dreaming-handoff.md) (session log, mostly aligned).
9
+
10
+ The system is the second iteration of Lucien's memory pipeline, rebuilt to run
11
+ **entirely on local models** inside mlx-bun (`src/memory/`). Same methodology and
12
+ pipeline at the core; the local constraint drives two design consequences:
13
+
14
+ 1. **Small tasks.** Every AI step is decomposed to the smallest single-purpose
15
+ unit a small local model can do well — one chunk, one bucket, one section,
16
+ ideally a binary judgment. Never a whole-article blob in or out.
17
+ **And every AI step is pure text-in/text-out — no tool calls, no agent
18
+ loop.** The model never edits a file, never searches, never decides what to
19
+ do next; the pipeline hands it exactly the context it needs and applies its
20
+ output deterministically. This lowers the capability bar dramatically: a
21
+ small model that could never drive an agentic edit session can absolutely
22
+ rewrite one section given the section and the new information — and pure
23
+ completion tasks are exactly what per-stage LoRA SFT trains well.
24
+ 2. **Deterministic gates between AI steps.** AI does judgment; deterministic
25
+ code enforces invariants (citations, footnotes, wikilinks, infobox shape,
26
+ word floors). A weak AI output is gated to NO-OP, never allowed to corrupt
27
+ the vault.
28
+
29
+ ## Fixed principles (do not re-derive, do not contradict)
30
+
31
+ - **Conversations are the base.** The pipeline's only input is conversation
32
+ transcripts (pi sessions daily; historical imports are a one-time onboarding
33
+ path through the same code).
34
+ - **No notability filter.** *"If I talk about it, it's notable."* Everything
35
+ surfaces; nothing is dropped or parked as trivia. Any prior "owned/decided/≥3
36
+ chunks else PARK" gate is dead — the catch-all bucket exists only so that
37
+ low-volume topics remain searchable until they warrant their own article.
38
+ - **No embeddings, anywhere in the project.** Not for routing, not for
39
+ clustering, not as "offline instruments." Grouping and routing are done by
40
+ deterministic name/alias/category lookup plus small local-model judgments.
41
+ - **Articles are real single-topic wiki articles with sections** — one subject
42
+ per article, a lead, an infobox, `##` sections, See also, References. All
43
+ subject mentions are `[[wikilinks]]`.
44
+ - **Vault = truth, DB = derived cache.** Markdown in the vault is canonical and
45
+ git-tracked; the SQLite store is pipeline state + rebuildable index
46
+ (`reindex`).
47
+ - **Self-healing through ordinary use.** Corrections arrive as ordinary chunks;
48
+ chronological processing (`updated_at ASC`) resolves claims toward the user's
49
+ latest position.
50
+ - **Prompts and Meta pages are SCHEMATIC** (no real names/specs/dates) — the
51
+ small model parrots concrete example values into unrelated articles.
52
+ - **Runs nightly**, locally, unattended (launchd, `src/memory/schedule.ts`).
53
+ - **Model evals are not part of this design.** The eval harnesses (chunk-eval,
54
+ judges, F1/Jaccard gates, cloud-judge sampling) are tooling for *choosing and
55
+ training the models used by* the pipeline. They live beside the system, never
56
+ inside it, and belong in their own docs.
57
+
58
+ ## The pipeline
59
+
60
+ ```
61
+ conversations (ingest, watermark/cursor-gated)
62
+ │
63
+ ▼ SEGMENT ──────────────── AI: find the topics in each conversation
64
+ chunks: single-topic segments with labels (pointer ranges, no text copies)
65
+ │
66
+ ▼ BUCKET ───────────────── AI + deterministic: group similar topics
67
+ buckets: one per STORY/TOPIC; everything discussed about that topic since the
68
+ last run accumulates here (a bucket usually has NO article of its own)
69
+ │
70
+ ▼ RESOLVE ──────────────── which articles does each story affect?
71
+ bucket delta → the subject articles it touches (update · create · sections)
72
+ │
73
+ ▼ SYNTHESIZE ───────────── AI: per-section create/patch, bounded in and out
74
+ create: outline → per-section draft · update: patch one section, rest byte-identical
75
+ │
76
+ ▼ ── deterministic gates ── citation survival · footnote bijection ·
77
+ │ word floor · wikilink resolution · weak output → NO-OP
78
+ │
79
+ ▼ RECONCILE ────────────── infobox = ground truth; lead/body/infobox agree
80
+ ▼ LINK ─────────────────── cross-link: mentions → [[links]], co-occurrence → See also
81
+ ▼ NORMALIZE ────────────── deterministic: footnote renumber, wikilink canonicalize,
82
+ │ structure guards (References last, stub marking)
83
+ ▼ EDITORIAL (wikify) ───── AI: remove redundancy, organize, make it read well —
84
+ │ per-section improve + structural pass over the TOC
85
+ ▼
86
+ commit to vault (git) + run changelog
87
+ ```
88
+
89
+ Every stage is an independent, resumable, DB-state-keyed worker
90
+ (`src/memory/stages.ts`), runnable separately or as the full DAG
91
+ (`mlx-bun memory synthesize`).
92
+
93
+ ### SEGMENT — conversations → topics
94
+
95
+ The e4b chunk adapter (`e4b-chunk-300`, symlinked `memory-chunk`) segments each
96
+ conversation into single-topic chunks with short labels. One chunk = one
97
+ coherent topic; a multi-stage process is ONE topic; long single-topic
98
+ conversations stay whole. Chunks are pointer ranges over `messages` — text is
99
+ materialized exactly once. Deterministic anchor repair validates every boundary.
100
+
101
+ ### BUCKET — group similar topics (first-class stage)
102
+
103
+ **The Wikipedia-editor analogy (this is the model — Josh's framing).** A news
104
+ story gets reported seven times in a day — Fox says one thing, MSNBC another,
105
+ CBS a third, each report carrying a bit of different information. A Wikipedia
106
+ editor who heard ALL of the reports incorporates everything they learned into
107
+ every article the story affects. In our system: the news shows are
108
+ **conversations**, the stories are **topics (chunks)**, and the **bucket is the
109
+ editor's brain** — the place where everything reported TODAY about THAT TOPIC
110
+ accumulates before any article is touched.
111
+
112
+ **A bucket is keyed by the STORY/TOPIC, not by a subject.** The worked example:
113
+ the topic is *"2026 Lincoln Memorial Reflecting Pool controversy"* (the algae
114
+ issue in the pool that Trump renovated). Every report about it that day goes in
115
+ that one bucket. There is **no article for that topic** — and there may never
116
+ be — but the story affects THREE articles: *Lincoln Memorial Reflecting Pool*,
117
+ *Second presidency of Donald Trump*, and *John J. Cafaro*. Each article
118
+ receives the facet of the delta relevant to it (renovation/algae facts vs.
119
+ abuse-of-power framing vs. the contract angle). This is why bucket ≠ article is
120
+ **structural**: articles are THINGS (subjects); buckets are STORIES (what got
121
+ discussed); the story→things relationship is inherently many-to-many.
122
+
123
+ Assignment mechanics: group chunks by topic — normalize/compare chunk labels
124
+ deterministically first, with a small-model binary judgment ("are these two
125
+ chunks about the same story? yes/no") for near-misses, and a new-bucket
126
+ proposal for genuinely new topics. The taxonomy is emergent: buckets come into
127
+ existence because the user talked about something, never from a predefined
128
+ list. Buckets are naturally short-lived relative to articles — a story
129
+ accumulates while it's live and goes quiet when the user stops discussing it;
130
+ the articles it fed persist.
131
+
132
+ `_captured` is the reserved catch-all bucket: chunks whose subject doesn't yet
133
+ justify assignment stay there, searchable, and are revisited on later runs —
134
+ **nothing is dropped**.
135
+
136
+ ### RESOLVE — bucket deltas → articles/sections
137
+
138
+ The unit of work from here on is the **bucket delta**: all of a bucket's new
139
+ (un-synthesized) chunks since the last run, taken TOGETHER. For each non-empty
140
+ delta, RESOLVE answers the editor's question: **"which articles does this story
141
+ affect?"** (the Reflecting Pool story → the pool article, the presidency
142
+ article, the contractor article). For each affected subject:
143
+
144
+ - **The article exists** → UPDATE. Section-route the delta's *facet for that
145
+ article* via binary per-section judgments over the TOC ("does this new
146
+ information update THIS section? yes/no"); all-no → propose a new section.
147
+ - **The article doesn't exist yet** → CREATE the subject article (a THING —
148
+ never an article named after the story itself). Thin is fine — 2–3 chunks
149
+ make a legitimate stub (stubs are a feature, not a failure).
150
+
151
+ The per-section ledger (`synthesized_chunk_sections`) records which chunks each
152
+ fold incorporated, making every fold idempotent and every claim traceable to
153
+ its source chunks.
154
+
155
+ ### SYNTHESIZE — one edit per (bucket, article), never chunk-at-a-time
156
+
157
+ **The cardinal rule:** synthesis is the editor sitting down ONCE per article
158
+ with everything the bucket learned. Never patch a section from one chunk and
159
+ then re-patch it from the next chunk five minutes later — sequential
160
+ single-chunk edits overwrite each other, churn citations, and produce
161
+ patchwork prose. The whole point of the bucket layer is that the delta arrives
162
+ at the article **already aggregated**.
163
+
164
+ - **CREATE**: group the bucket's chunks into proto-sections (by label/subject
165
+ affinity — small-model judgment, no embeddings) → outline (title, lead,
166
+ section names) → draft each section from only its chunks. The model never
167
+ holds a whole article in its head.
168
+ - **UPDATE (patch)**: hand the model one section + the delta's facet relevant
169
+ to that article and section (all of it, with per-conversation `[^N]` citation
170
+ material); it rewrites that one section once, weaving everything in. Facet
171
+ extraction is implicit: the same story becomes algae/renovation facts in the
172
+ Reflecting Pool article and framing in the presidency article, because the
173
+ model writes what's relevant to the section it's editing.
174
+ Contradiction = correction: resolve toward the latest statement, including
175
+ contradictions *within* the delta (latest chunk wins — the editor heard all
176
+ seven reports and writes the reconciled version, not seven amendments).
177
+ User's evolving positions keep a `## History` trace; AI factual errors are
178
+ silently overwritten.
179
+ - **Edit mechanics (SETTLED 2026-07-01, Josh-confirmed — and how Lucien always
180
+ worked): the model never edits the article; it is handed the current section
181
+ + the new information and outputs a complete REPLACEMENT SECTION.** Not an
182
+ in-place file edit (the
183
+ model would have to hold the whole article — the exact blob failure this
184
+ design avoids) and not a diff (small local models produce brittle patch
185
+ formats, and a diff is hard to gate). A full replacement section is bounded
186
+ in and out, trivially appliable (`extractSection` swap — the rest of the file
187
+ byte-identical by construction), and directly gateable: the gate compares old
188
+ section vs. new section for word floor and citation survival. Deterministic
189
+ code performs the actual write. CREATE works the same way — the model emits
190
+ one section at a time; code assembles the article.
191
+ - **Multiple buckets converging on one article** (the algae-controversy bucket
192
+ AND the olympian-arrest bucket both affect *Reflecting Pool*): each bucket's
193
+ facet folds as its own section-level edit within the run — different stories
194
+ usually land in different sections. If two buckets touch the SAME section,
195
+ fold chronologically (latest wins, per self-healing) and let the nightly
196
+ EDITORIAL pass smooth the seam; the editorial pass runs after all folds, so
197
+ the article the user reads next morning is one coherent account.
198
+ - **Conservative write gate** on every output: ≥70% word floor on the touched
199
+ section, every pre-existing `conv:` citation survives, H1/References survive,
200
+ footnote bijection. Weak output → NO-OP; the delta retries next run.
201
+
202
+ ### RECONCILE · LINK · NORMALIZE — nightly deterministic/structural stages
203
+
204
+ These run **every night**, not periodically:
205
+
206
+ - **RECONCILE** (`reconcile.ts`): the infobox is the article's ground truth;
207
+ lead, body sections, and infobox must agree; date-aware (latest wins).
208
+ Infobox extraction/refresh is part of this — every article carries a current,
209
+ parseable infobox whose entity-valued fields are `[[wikilinks]]`.
210
+ - **LINK** (`crosslink.ts`): cross-linking is a dedicated stage, NOT
211
+ synthesis's job (synthesis sees one article and cannot know the others).
212
+ Mention-based prose links + co-occurrence See-also. Idempotent. Ensures **all
213
+ subjects are wikilinked** across the vault.
214
+ - **NORMALIZE** (`normalize.ts`): pure code — footnote renumber/merge, wikilink
215
+ canonicalization, orphan repair, structure guards (`## References` last,
216
+ `{{stub}}`/`{{Main}}` recognition, title == H1 == stem).
217
+
218
+ ### EDITORIAL (wikify) — make the article good
219
+
220
+ The final AI pass, and the reason articles stay readable instead of accreting
221
+ patches: per-section "tighten this — remove redundancy, keep every citation and
222
+ every specific detail" plus a structural pass over the TOC (reorder sections,
223
+ refresh the lead, split a section that outgrew the article into its own article
224
+ with a `{{Main}}` summary, merge duplicate stubs). Behind the same conservative
225
+ gate; a weak edit is rejected and the original kept.
226
+
227
+ ## Nightly run contract
228
+
229
+ - Trigger: launchd (`com.mlx-bun.memory`), logs under `~/.mlx-bun/logs/`.
230
+ - Scope: only conversations past the per-source watermark — a normal night is
231
+ minutes of work, not hours.
232
+ - Inference: one resident base model (Gemma-4-e4b), per-stage adapter mounted
233
+ iff `~/.cache/mlx-bun/adapters/memory-<stage>` exists, else base+policy.
234
+ Route generation through the persistent server (load-once) when available;
235
+ no-server fallback works but pays model load per invocation.
236
+ - Every article change is a git commit; the run ends with a changelog summary.
237
+ - Failure isolation: stages are resumable; a failed bucket/article never blocks
238
+ the rest of the run.
239
+
240
+ ## Decisions overridden from the master plan (explicit, so no agent re-derives them)
241
+
242
+ | Master-plan statement | Superseded by |
243
+ | --- | --- |
244
+ | Personal-notability checklist; CREATE gate "≥3 chunks OR owned/decided, else PARK as trivia"; ≤40-article first-run cap (P1-T5, P5-T4, R5, P9-T3) | **No notability filter.** Everything surfaces; `_captured` keeps the tail searchable; thin buckets → stubs. |
245
+ | Buckets demoted to "optional staging, may ship empty" (L4) | **Buckets are first-class**: the per-subject accumulation layer between chunks and articles; synthesis consumes buckets. |
246
+ | Embeddings allowed as offline instruments (P4 recall tripwire experiments, P8-T4 silhouette triage) | **No embeddings anywhere in the project.** The tripwire (`getEmbedCounter()==0`) remains as a guard, permanently. |
247
+ | Routing/section-route acceptance gated on Lucien bucket-F1 / Jaccard (P4-T2, P7-T1) | Model-selection evals live **outside** the system design; pipeline health is judged on real output quality, never Lucien's bin taxonomy. |
248
+ | Cross-linking + infobox work assigned to periodic WIKIFY (P8) | RECONCILE + LINK are **nightly stages** in the DAG; editorial wikify is the readability pass. |
249
+
250
+ ## Work plan (docs-only reconciliation done; implementation next)
251
+
252
+ Ordered; none of it touches the running full-corpus import
253
+ (`~/.mlx-bun/wiki-full`, cursor file `~/.mlx-bun/full-run-cursor.txt`) or the
254
+ uncommitted `src/model/universal` / trainer work.
255
+
256
+ 1. **Bucket layer made explicit** — a topic-keyed BUCKET stage between SEGMENT
257
+ and the article fan-out (`buckets` + `chunk_buckets` tables already exist):
258
+ group chunks by story (label normalization + binary "same story?"
259
+ judgments), `_captured` catch-all, and **delta-at-once synthesis** — the run
260
+ collects each bucket's new chunks, then makes ONE edit per (bucket, article).
261
+ The as-built entity-keyed ROUTE becomes RESOLVE (bucket → affected subject
262
+ articles); the per-chunk patch loop is replaced, not wrapped. (Settled
263
+ 2026-07-01 — the Wikipedia-editor / Reflecting-Pool framing above; buckets
264
+ are STORIES, articles are THINGS, many-to-many by nature.)
265
+ 2. **Nightly DAG order** — ensure `mlx-bun memory synthesize` runs
266
+ segment → bucket → resolve → synthesize → reconcile → link → normalize →
267
+ editorial every run (reconcile/link/infobox currently periodic/partial).
268
+ 3. **Editorial stage** — wire the per-section improve + structural pass as the
269
+ closing stage behind the conservative gate.
270
+ 4. **Persistent-server inference routing** — load-once for all memory
271
+ generation, no-server fallback (the in-progress inference-path rework;
272
+ finish, don't fork).
273
+ 5. **Promote + schedule** — after the import is judged good: point
274
+ `~/.mlx-bun/wiki` at the bootstrapped content, enable the nightly launchd
275
+ fold-in from pi sessions.
276
+ 6. **Known quality fixes** — schematic-prompt leaks (<1% of articles), run-on
277
+ leads, `owned:` framing on abstract concepts (reframe the lead spec per
278
+ subject kind).
279
+
280
+ ## Open questions (for discussion with Josh)
281
+
282
+ - ~~Bucket granularity vs article granularity~~ — **SETTLED 2026-07-01**:
283
+ buckets are a separate layer from articles (many-to-many both ways); the
284
+ bucket is the editor's brain that aggregates the day's information about a
285
+ subject, and RESOLVE fans the delta out to every affected article. Bucket is
286
+ never a 1:1 alias for an article.
287
+ - **When does a `_captured` chunk graduate?** Re-scan `_captured` every night
288
+ for subjects that have since accumulated enough related chunks, or only when
289
+ a new bucket's name matches?
290
+ - **Editorial cadence.** Every night over every touched article, or touched
291
+ articles nightly + a slower full-vault sweep?
@@ -0,0 +1,106 @@
1
+ # DSpark speculative decoding for mlx-bun
2
+
3
+ Implementation of DeepSeek's **DSpark** ("Confidence-Scheduled Speculative
4
+ Decoding with Semi-Autoregressive Generation", `DSpark_paper.pdf`) against this
5
+ repo's models. DSpark = a **DFlash** parallel backbone (KV-injected, all γ draft
6
+ tokens in one pass) **+ a lightweight sequential head** (Markov, Eq 5, or RNN,
7
+ Eq 6) that adds intra-block dependency (the *semi-autoregressive* part) **+ a
8
+ confidence head → STS calibration → hardware-aware prefix scheduler**. Decode
9
+ becomes draft → verify → accept; output is lossless (bit-identical to the target
10
+ at any temperature); only decode speed (τ) changes.
11
+
12
+ > **DFlash vs DSpark:** DFlash is *only* the parallel backbone (prior work) and
13
+ > suffers suffix decay. DSpark adds the sequential head + confidence-scheduled
14
+ > verification. The sequential Markov head is what makes this DSpark.
15
+
16
+ ## Two implementations (flag-selected, both in `src/spec/dspark/`)
17
+
18
+ | | `module.ts` (v1) | `module-dflash.ts` (v2, **the real one**) |
19
+ |---|---|---|
20
+ | context | **single vector** (final hidden at the anchor) fused as one token | faithful **multi-layer KV injection** (Eq 2–3): H_ctx = RMSNorm(W_c·[H^{l1};…;H^{lm}]) over the FULL context, re-projected into every draft layer's K/V |
21
+ | layers | 2 | 5 |
22
+ | status | **superseded** — dropped the paper's core idea (kept as baseline) | the actual DSpark model (Markov variant) |
23
+
24
+ v1 was a shortcut that removed the load-bearing mechanism; it never tested the
25
+ paper. v2 is faithful. Checkpoints stamp `variant` in `dspark.json`; the runtime
26
+ loads the matching module. (`variant:"dflash"` should be renamed to `"dspark"` —
27
+ see Open items.)
28
+
29
+ ## The mechanism (v2, paper Eq 2–3)
30
+
31
+ - **Eq 2 (context):** tap the target's hidden states at layers `{l1…lm}` (e4b
32
+ default {20,31,41,42}, the last = post-finalNorm), concat on the feature axis,
33
+ project to the draft width: `H_ctx ∈ [L_ctx, d]`. Extraction via a parity-safe
34
+ `model.hiddenTap` (no-op when off) added to `gemma4.ts` `forwardLayers`.
35
+ - **Eq 3 (injection):** every draft layer forms `K_i=[W_i^K H_ctx; W_i^K H_d]`,
36
+ `V_i` likewise, `Q_i=W_i^Q H_d`; the γ-block queries attend bidirectionally
37
+ over `[context ++ block]`. Context is read-only memory re-projected fresh each
38
+ layer. The draft projects the target's HIDDEN STATES with its OWN Wk/Wv (not
39
+ the target's K/V — that's the separate `GemmaAssistantDrafter` baseline).
40
+ - **Markov head (Eq 5):** `B_k = W1[x_{k-1}]·W2`, r=256, W2=0 init → starts as
41
+ pure DFlash, τ climbs as intra-block dependency is learned.
42
+ - **Confidence head (Eq 7):** `c_k = σ(wᵀ[h_k; W1[x_{k-1}]])`.
43
+ - **Loss (Eq 9–12):** `0.1·CE + 0.9·TV + 1.0·BCE`, position-weighted.
44
+ - **Lossless verify** (`generate-dflash.ts`): greedy = longest-prefix match;
45
+ temp>0 = `min(1,p/q)` accept + residual `norm(relu(p−q))` resample. Growing
46
+ multi-layer H_ctx buffer; rollback via `trim(n, bypass=true)` (physically
47
+ slices the rejected tips past the sliding window — see `gemma4-base.ts`).
48
+
49
+ ## Results & findings (measured, M1 Max 32GB, e4b OptiQ-4bit)
50
+
51
+ **The architecture is CORRECT — proven by an overfit test.** Trained on 3
52
+ articles, eval on the same 3: per-position acceptance **~0.75**, τ **3.24** —
53
+ squarely paper-range (0.6–0.9 / τ 3–4). So the faithful DFlash, correctly
54
+ implemented, reaches the paper's numbers.
55
+
56
+ **Getting there required finding a real bug — an off-by-one in the p^t target
57
+ alignment** (the TV loss, weight 0.9). Block position k predicts `x_{t+1+k}`,
58
+ whose target distribution is `softmax(LM_head(h_{t+k}))`; the data gathered the
59
+ hidden at `t+k+1` (shifted forward one). Present in **both v1 and v2** — so every
60
+ τ number before the fix was trained against a shifted target. Fixed in
61
+ `data-dflash.ts` (`blockIdx = g+k`, k=0..γ-1). Before fix: per-pos ~0.08. After:
62
+ ~0.17 generalizing, ~0.75 overfit.
63
+
64
+ **Two things stand between "correct architecture" and "net speedup," now
65
+ separated:**
66
+ 1. **Data.** 160 articles → generalizes only to per-pos ~0.17 (plateaus, doesn't
67
+ climb); overfit on 3 proves capacity. The paper uses 1.3M samples × 10 epochs.
68
+ → needs real data scale.
69
+ 2. **Target speed (the decisive one).** Spec decode amortizes the fixed draft
70
+ overhead only when the *target forward* is expensive. Measured decode:
71
+ **e4b 45.9 tok/s, 12b 27.5 tok/s, 27B ~15 tok/s.** On fast e4b the draft
72
+ overhead dominates → even τ≈3 nets ~0.4–0.5× (SLOWER). On a slow target (27B,
73
+ ~67 ms/forward) that overhead is a small fraction → the same τ≈3 could land
74
+ ~2–3× (15 → ~35–45 tok/s). **e4b is close to the worst case for spec decode;
75
+ the 27B agentic workload is the ideal target and the real motivation.**
76
+
77
+ Also confirmed: width is NOT the ceiling (`dDraft=1024` and `2560` both plateau
78
+ ~0.17 pre-scale); the parity-safe `hiddenTap` doesn't change target numerics;
79
+ long-context spec rollback works via the `trim(n, bypass)` physical-slice.
80
+
81
+ ## Files
82
+ - `src/model/gemma4.ts` — `hiddenTap`/`captureLayer` (parity-safe multi-layer tap).
83
+ - `src/model/gemma4-base.ts` — `trim(n, bypass)` on rotating caches (spec rollback past the window).
84
+ - `src/spec/dspark/module-dflash.ts` — faithful DFlash+Markov+confidence module.
85
+ - `src/spec/dspark/data-dflash.ts` — multi-layer shards, variable-length prefix context, left-pad masking.
86
+ - `src/spec/dspark/generate-dflash.ts` — growing-H_ctx lossless decode loop.
87
+ - `src/spec/dspark/{loss,sample}.ts` — loss (Eq 9–12) + temp>0 sampling (shared with v1).
88
+ - `src/spec/dspark/{module,data,generate}.ts` — v1 single-vector (superseded baseline).
89
+ - `scripts/dspark-regen-dflash.ts` / `dspark-train-dflash.ts` (`--resume`) / `dspark-measure-dflash.ts`.
90
+ - `scripts/dspark-dflash-smoke.ts` — CPU smoke (16/16). `scripts/dspark-smoke.ts` — v1 smoke (33/33).
91
+
92
+ ## Run sequence (faithful DFlash; GPU = Josh runs)
93
+ ```
94
+ bun scripts/dspark-regen-dflash.ts --topics <topics.txt> --out <data> --max-resp 320
95
+ bun scripts/dspark-train-dflash.ts --data <data> --out <ckpt> --iters 8000 --batch 8 [--resume] [--ddraft 2560]
96
+ bun scripts/dspark-measure-dflash.ts --drafter <ckpt> --data <prompts.jsonl> # τ + tok/s vs vanilla
97
+ ```
98
+
99
+ ## Open items / next
100
+ - **Real data scale** (thousands of on-distribution generations, not 160) to lift generalizing τ toward the overfit ~0.75.
101
+ - **Retarget to a slow model (27B/12b)** where the τ≈3 architecture actually nets a speedup — the whole point. Needs the drafter sized to that model's H/layers + regen+train there; 27B on 32GB is memory-tight (17.75GB weights + KV budget).
102
+ - **Tighten the draft inference loop** (per-position host syncs, double 262K LM-head) so τ translates to wall-clock.
103
+ - **Remaining paper components:** RNN head (Eq 6), STS calibration (§3.2.1), hardware-aware prefix scheduler (Alg 1 — single-user form = confidence-scheduled draft-length pruning).
104
+ - **Rename** `dflash`→`dspark` (the faithful module IS DSpark; v1 is the legacy single-vector variant).
105
+
106
+ Full session handoff: `docs/investigations/dspark-handoff.md`.