mlx-bun 0.0.8 → 0.0.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (133) hide show
  1. package/README.md +112 -20
  2. package/docs/design/batching-v2-plan.md +515 -0
  3. package/docs/design/bucketing-stage.md +196 -0
  4. package/docs/design/compat-cli-surface-design.md +11 -11
  5. package/docs/design/docs-reorg-plan.md +1 -1
  6. package/docs/design/dreaming-nightly-pipeline.md +291 -0
  7. package/docs/design/dspark-speculative-decoding.md +106 -0
  8. package/docs/design/generic-model-support.md +381 -0
  9. package/docs/design/memory-inference-path.md +422 -0
  10. package/docs/design/memory-synthesis.md +287 -0
  11. package/docs/design/minicpm5-decode-megakernel.md +484 -0
  12. package/docs/design/mlx-lm-tool-parity-plan.md +535 -0
  13. package/docs/design/orpo-dynamic-lambda.md +41 -0
  14. package/docs/design/orpo-training.md +35 -5
  15. package/docs/design/parallel-slots.md +12 -1
  16. package/docs/design/parity-tier-dag.md +13 -5
  17. package/docs/design/spec-decode-larger-targets.md +229 -0
  18. package/docs/design/tauri-desktop-app.md +2 -2
  19. package/docs/design/the-dreaming-handoff.md +141 -0
  20. package/docs/design/the-dreaming-master-plan.md +446 -0
  21. package/docs/design/write-pipeline-entity.md +468 -0
  22. package/docs/investigations/curve-bisector-routing.md +1 -1
  23. package/docs/investigations/decode-roofline-lookagain.md +262 -0
  24. package/docs/investigations/dspark-handoff.md +58 -0
  25. package/docs/investigations/kernel-perf-review-2026-07.md +92 -0
  26. package/docs/planning/PRODUCT_ROADMAP.md +2 -2
  27. package/docs/planning/curve-sampler-research-plan.md +331 -0
  28. package/docs/planning/memory-docs-and-dag-plan.md +185 -0
  29. package/docs/planning/pi-integration-review.md +178 -0
  30. package/docs/planning/release-notes-v0.0.9.md +122 -0
  31. package/docs/planning/web-ui-pass-plan.md +213 -0
  32. package/docs/planning/website-readme-pass-plan.md +224 -0
  33. package/docs/reference/cli.md +255 -0
  34. package/docs/reference/embedding.md +7 -2
  35. package/docs/reference/library-api.md +22 -3
  36. package/docs/reference/memory.md +5 -5
  37. package/docs/reference/models.md +137 -0
  38. package/docs/reference/orpo-quickstart.md +6 -0
  39. package/docs/reference/server-api.md +187 -9
  40. package/docs/reference/server-config.md +40 -9
  41. package/docs/reference/training.md +25 -11
  42. package/package.json +1 -1
  43. package/src/anthropic.ts +22 -0
  44. package/src/assets/curve-designer.html +1 -1
  45. package/src/chat-template.ts +25 -7
  46. package/src/cli.ts +1029 -86
  47. package/src/config.ts +4 -1
  48. package/src/download.ts +99 -8
  49. package/src/embed.ts +77 -0
  50. package/src/eval/perplexity.ts +164 -0
  51. package/src/eval/runner.ts +15 -6
  52. package/src/eval/tasks/bfcl.ts +1 -1
  53. package/src/generate.ts +141 -18
  54. package/src/harness-pi.ts +43 -17
  55. package/src/index.ts +2 -0
  56. package/src/lora.ts +2 -2
  57. package/src/memory/article.ts +558 -0
  58. package/src/memory/chunk-validation.ts +150 -0
  59. package/src/memory/chunk.ts +305 -12
  60. package/src/memory/cluster.ts +344 -17
  61. package/src/memory/coverage.ts +134 -0
  62. package/src/memory/crosslink.ts +0 -0
  63. package/src/memory/db.ts +184 -45
  64. package/src/memory/entity.ts +203 -0
  65. package/src/memory/gate.ts +187 -0
  66. package/src/memory/history.ts +152 -0
  67. package/src/memory/ingest.ts +324 -11
  68. package/src/memory/model.ts +321 -0
  69. package/src/memory/normalize.ts +671 -0
  70. package/src/memory/parse.ts +22 -0
  71. package/src/memory/pipeline.ts +242 -30
  72. package/src/memory/prompts.ts +42 -0
  73. package/src/memory/query.ts +334 -0
  74. package/src/memory/reconcile.ts +734 -0
  75. package/src/memory/redirect.ts +108 -0
  76. package/src/memory/reindex.ts +325 -0
  77. package/src/memory/resolve.ts +307 -0
  78. package/src/memory/route.ts +272 -0
  79. package/src/memory/schedule.ts +6 -4
  80. package/src/memory/stages.ts +774 -0
  81. package/src/memory/synthesize.ts +1629 -20
  82. package/src/memory/tools.ts +446 -96
  83. package/src/memory/vault.ts +462 -35
  84. package/src/memory/wikify.ts +1057 -16
  85. package/src/mlx/ffi.ts +3 -0
  86. package/src/mlx/ops.ts +46 -0
  87. package/src/model/compiled-decode.ts +43 -8
  88. package/src/model/factory.ts +29 -8
  89. package/src/model/flash-attention.ts +4 -2
  90. package/src/model/fused-decode-kernel.ts +14 -7
  91. package/src/model/fused-mlp-kernel.ts +169 -0
  92. package/src/model/fused-swiglu-kernel.ts +240 -0
  93. package/src/model/gemma4-base.ts +62 -6
  94. package/src/model/gemma4.ts +33 -2
  95. package/src/model/minicpm5.ts +106 -8
  96. package/src/model/qwen3.ts +276 -0
  97. package/src/model/steel-linear-kernel.ts +96 -0
  98. package/src/model/support.ts +38 -8
  99. package/src/model/universal/archs.ts +332 -0
  100. package/src/model/universal/dense.ts +576 -0
  101. package/src/model/universal/modules.ts +163 -0
  102. package/src/model/universal/rope.ts +366 -0
  103. package/src/pi-provider.ts +1 -1
  104. package/src/pi-terminal.ts +29 -8
  105. package/src/pi-web.ts +19 -2
  106. package/src/quantize/config-writer.ts +1 -1
  107. package/src/registry.ts +215 -7
  108. package/src/responses.ts +23 -0
  109. package/src/sampler.ts +220 -9
  110. package/src/serve/batch-scheduler.ts +272 -80
  111. package/src/serve/generation-gateway.ts +86 -18
  112. package/src/server.ts +612 -49
  113. package/src/spec/drafter.ts +81 -49
  114. package/src/spec/dspark/data-dflash.ts +156 -0
  115. package/src/spec/dspark/data.ts +201 -0
  116. package/src/spec/dspark/generate-dflash.ts +193 -0
  117. package/src/spec/dspark/generate.ts +290 -0
  118. package/src/spec/dspark/loss.ts +213 -0
  119. package/src/spec/dspark/module-dflash.ts +361 -0
  120. package/src/spec/dspark/module.ts +540 -0
  121. package/src/spec/dspark/sample.ts +112 -0
  122. package/src/spec/generate.ts +40 -19
  123. package/src/status-page.html +1 -1
  124. package/src/tokenizer.ts +7 -0
  125. package/src/train/flash-cce.ts +70 -15
  126. package/src/train/fuse.ts +243 -0
  127. package/src/train/index.ts +3 -0
  128. package/src/train/job.ts +8 -0
  129. package/src/train/loss.ts +289 -30
  130. package/src/train/prefix-shared.ts +33 -5
  131. package/src/train/segmented.ts +67 -11
  132. package/src/train/trainer.ts +92 -28
  133. package/src/web/skills/memory/SKILL.md +40 -17
@@ -61,16 +61,24 @@ Worked examples:
61
61
  | lever | parity | perf | placement |
62
62
  |---|---|---|---|
63
63
  | `compiled-decode` | bit-exact w/ uncompiled (`compiled-decode.test`) | faster | **default-on, every level** |
64
- | `perf-kernel` | matches optiq, not mlx-lm (`perf-kernel-oracle.test`) | faster | **default-on from L2** (the L2 bridge) |
65
- | `fused-sdpa` | matches optiq, not mlx-lm (`fused-sdpa.test`) | faster | **default-on from L2** (the L2 bridge) |
64
+ | `perf-kernel` | **envelope-gated, no bit-exact oracle** — mlx-bun-original kernel; gate is ≥56/64 teacher-forced argmax vs *our own* frozen compat trajectory (`perf-kernel-oracle.test`, `freeze-perf-oracle.ts`), and the optiq kv-quant goldens track the **stock unfused L=1 decode** (`regen-kvq-goldens.ts`) | faster | **L3 / explicit opt-in only** (off in bare `--l2`; on in `--l3` and the no-tier default) |
65
+ | `fused-sdpa` | matches optiq bit-for-bit, not mlx-lm (`fused-sdpa.test` tier-a goldens) | faster | **default-on from L2** (the L2 bridge) |
66
66
  | `fused-decode` | (would be L3) | ~3% slower (`optimization_plan.md`) | **kept, off everywhere** |
67
67
 
68
68
  So `--l1/--l2/--l3` is the **parity** axis (the guarantee), and the defaults within
69
- each tier are the **performance** axis (the fastest kernel that still holds it). For
70
- the *decode path* L2 == L3 today (every decode kernel matches mlx-lm or optiq); L3's
71
- real territory is the no-oracle **features** — HLG sampler, expert offload, batched
69
+ each tier are the **performance** axis (the fastest kernel that still holds it). On
70
+ the *decode path* L2 and L3 now differ by exactly one node — the perf kernel (bare
71
+ `--l2` = the optiq-golden composition: fused N-tiled prefill + stock unfused L=1
72
+ decode; `--l3` adds the envelope-gated fused decode kernel). L3's remaining
73
+ territory is the no-oracle **features** — HLG sampler, expert offload, batched
72
74
  mixed-precision — which live off the decode axis and are gated by KL + quality.
73
75
 
76
+ > History: commit `f1bf5cc` put the perf kernel in the L2 preset claiming
77
+ > "perf-kernel-oracle.test asserts the fused kernel == the frozen optiq goldens".
78
+ > Both halves were wrong — that golden is frozen from mlx-bun's own compat engine,
79
+ > and the gate is argmax-agreement, not equality — so 2026-07-01 restored the tier
80
+ > rule: the bare tier is the guarantee; perf kernels are opt-ins within it.
81
+
74
82
  ## Why this matters for CLI flags
75
83
 
76
84
  We are accumulating a **lot** of flags (`--grad-accum`, `--grad-clip`, `--seg`,
@@ -0,0 +1,229 @@
1
+ # Spec decode on larger targets — does the Gemma drafter pay on 12B / 26B?
2
+
3
+ Status: **12B γ=1 is a WIN after the verify fix; matches optiq bit-exactly** (2026-06-14; 26B not run)
4
+ Owner: decode / spec
5
+ Default: **off** (speculation is opt-in; non-spec greedy is untouched)
6
+
7
+ ## Result (2026-06-14, v2) — fixed the wrong-oracle bug; γ=1 now wins
8
+
9
+ The first 12B measurement (v1, below) was made through a verify path that
10
+ targeted the **wrong oracle**. Fixing it changed the conclusion.
11
+
12
+ ### The wrong-oracle bug
13
+
14
+ Assistant-drafter speculation is an **optiq** feature — mlx-lm has only
15
+ generic standalone-`draft_model` two-model spec and structurally cannot
16
+ drive a KV-borrowing drafter. So the correctness oracle is optiq's
17
+ `spec_generate`. **Both optiq and mlx-lm BATCH the verify lm-head** (one
18
+ matmul over the γ+1 window, then argmax per position); neither is bit-
19
+ exact to stock token-at-a-time decode. Our port instead verified
20
+ **per-position** to be bit-exact to *stock decode* — a target no real
21
+ implementation has, and one that re-reads the ~1 GB tied lm-head matrix
22
+ γ+1× per iteration instead of once. So we were exact to the wrong thing
23
+ *and* paying a γ-scaling tax for it. Fix: `picksBatched` in
24
+ `src/spec/generate.ts` — one batched verify lm-head, matching optiq.
25
+
26
+ ### Do we match optiq? YES — bit-exact (e4b)
27
+
28
+ Cross-checked our `specGenerate` against optiq's `spec_generate` on the
29
+ identical prompt ids (`scripts/oracle-spec.py` + `scripts/spec-dump.ts`),
30
+ e4b γ=2: **identical 48-token output AND identical accept/reject trace**
31
+ (drafted 60, accepted 17, target calls 31 in both). Before the fix we
32
+ matched *stock*, not optiq, at knife-edges; now we match optiq.
33
+
34
+ ### Is it still a loss? Not at γ=1 (12B)
35
+
36
+ 12B pair, batched verify (loaded machine — ratios paired/trustworthy,
37
+ absolutes depressed & not quotable):
38
+
39
+ | γ | v2 speedup (batched) | v1 (per-position) | acceptance |
40
+ |---|----------------------|-------------------|-----------|
41
+ | 1 | **1.09× (WIN)** | 0.96× | 42% |
42
+ | 2 | 0.91× (near break-even) | 0.77× | 29% |
43
+ | 3 | 0.72× | 0.67× | 23% |
44
+ | 4 | 0.56× | 0.52× | 17% |
45
+
46
+ Acceptance is unchanged (the fix only touches verify cost, not which
47
+ tokens are accepted). The per-position verify was a large self-inflicted
48
+ tax: removing it flips γ=1 from loss to **win** and moves γ=2 to the edge.
49
+ γ≥2 still loses because the *drafter's own* cost dominates there — the
50
+ 12B drafter is heavy (hidden 1024 vs e4b 256, 16 heads, and a full 262k
51
+ tied-embedding head **every draft step**) and acceptance falls with depth.
52
+
53
+ **So the v1 "hypothesis refuted / net loss at every γ" was itself partly
54
+ an artifact of the wrong-oracle verify.** Corrected conclusion: γ=1
55
+ speculation on 12B is a real ~9% decode win (pending clean-machine
56
+ confirmation before quoting); γ≥2 doesn't pay with the shipped heavy
57
+ drafter.
58
+
59
+ ### Bonus finding: we're ahead of optiq on the unified drafters
60
+
61
+ optiq's *installed* runtime can't actually drive the 12B/26B
62
+ `gemma4_unified_assistant` drafters: its head detection is config-based
63
+ (`use_ordered_embeddings = num_centroids and top_k`), but those artifacts
64
+ declare `num_centroids` while shipping **no** centroid tensors and **no**
65
+ `lm_head` (tied `embed_tokens` instead). optiq loads the centroid path
66
+ with zero-init weights → garbage drafts → **0% acceptance** (measured:
67
+ drafted 46, accepted 0 on 12B). Our `drafter.ts` detects the head by
68
+ **tensor presence** and uses the tied-embedding path → 29% acceptance on
69
+ the same artifact.
70
+
71
+ ### Remaining levers / next
72
+
73
+ - **Clean-machine rerun** before promoting any number to RESULTS.md
74
+ (baseline jumped 15.4→24.4 between runs under memory pressure — exactly
75
+ why only paired ratios are trusted here).
76
+ - **Cheaper drafter head**: the 262k tied-embed argmax every draft step is
77
+ the dominant remaining γ≥2 cost; capping/approximating it (e4b-style)
78
+ could extend the win past γ=1.
79
+ - **26B-MoE**: still unlikely — MoE decode reads only top-8/128 experts
80
+ (~2.4 GB active), so per-token target cost is *lower* than its size →
81
+ harder to beat. Measure if curious.
82
+ - **Optional strict mode**: the old per-position verify gave "spec output
83
+ identical to spec-off" — a product nicety no oracle provides. Could be
84
+ kept as a default-off `strictVerify` flag rather than deleted
85
+ ([[dont-delete-optionality-on-one-measurement]]) — not implemented; ask.
86
+
87
+ ## v1 (superseded) — per-position verify, "net loss at every γ"
88
+
89
+ > Kept for the record. The numbers (0.96/0.77/0.67/0.52×) were real but
90
+ > measured through the wrong-oracle per-position verify; v2 above is the
91
+ > optiq-faithful measurement.
92
+
93
+ ## TL;DR
94
+
95
+ The Gemma `-assistant-bf16` drafters are Gemma's substitute for an MTP
96
+ head: a 4-layer Q-only module that reuses the target's KV + hidden state
97
+ to draft tokens (`src/spec/drafter.ts`), verified by the target's own
98
+ greedy pick (`src/spec/generate.ts`). True MTP (a trained
99
+ `mtp.safetensors` head + an `optiq/runtime/mtp` backend) **does not exist
100
+ for Gemma** and never will — that's the Qwen path (Phase 14). The
101
+ assistant drafter is the functional equivalent we already have.
102
+
103
+ It was measured a **net loss on e4b** (Phase 6): γ=2 ran at 0.78× the
104
+ 54 tok/s baseline, ~23% acceptance — because e4b decode is already too
105
+ fast for a low-acceptance drafter to beat. **This doc is the experiment
106
+ to find out whether the same drafter wins on a slower target**, where
107
+ each skipped target-forward is worth more.
108
+
109
+ ## The hypothesis
110
+
111
+ The drafter's per-step cost is roughly fixed. The *value* of each
112
+ accepted draft = one skipped target forward, which grows as the target
113
+ gets slower per token. e4b is the worst case for speculation (fastest
114
+ target). So the SAME acceptance rate can flip net-positive as the target
115
+ slows:
116
+
117
+ | target | non-spec decode | per-accepted-draft value | drafter | status |
118
+ |--------|-----------------|--------------------------|---------|--------|
119
+ | e4b | ~54 tok/s | low | 160 MB (cached) | **measured net loss** (0.91/0.78/0.63/0.51× @ γ=1–4) |
120
+ | 12B | ~25 tok/s | ~2× e4b | 846 MB (cached) | **measured net loss** — see Result above (hypothesis refuted) |
121
+ | 26B-A4B (MoE) | ~32 tok/s | high (MoE forward) | 839 MB (HF, **not downloaded**) | untested (follow-on; expect loss) |
122
+
123
+ > The reasoning below is the *pre-measurement* hypothesis — kept for the
124
+ > record. It predicted the crossover assuming a fixed-cost drafter; the
125
+ > 12B drafter is not fixed-cost (it scales with the target), which is
126
+ > exactly why the Result section above refutes it.
127
+
128
+ Crossover intuition: a drafter is a net win when
129
+ `acceptance × (target_step_cost − draft_step_cost) > verify_overhead`.
130
+ On e4b the target step is cheap, so the bar isn't cleared at 23%. On 12B
131
+ the target step is ~2× more expensive while the drafter cost barely
132
+ moves — same acceptance, very different sign. **Measure, don't assume**
133
+ (it could still lose; this is exactly the kind of thing the e4b result
134
+ warned against assuming).
135
+
136
+ Acceptance itself may also differ on 12B (a stronger target predicts
137
+ itself differently). We don't know the direction — that's why we run it.
138
+
139
+ ## What is already wired (this change)
140
+
141
+ - **`scripts/spec-bench.ts`** — now multi-pair, **one pair per process**
142
+ (memory-safe; 12B+drafter ≈ 8 GB, 26B ≈ 17 GB, never co-resident):
143
+ - `bun scripts/spec-bench.ts e4b` (default) — reproduces the baseline.
144
+ - `bun scripts/spec-bench.ts 12B` — the experiment.
145
+ - `bun scripts/spec-bench.ts 26B` — follow-on.
146
+ - Auto-detects whether each `(target, drafter)` snapshot pair is
147
+ present; exits with a clear "download with: hf download …" message
148
+ otherwise. Records baseline + γ-sweep (acceptance + tok/s) to the
149
+ eval DB tagged `spec-bench[<id>]`.
150
+ - **`tests/spec-decode.test.ts`** — added a `(12B)` parity block,
151
+ auto-skipped until the drafter is downloaded. Gate = long-prefix
152
+ agreement (greedy spec vs greedy non-spec). It guards the shared
153
+ accept/reject/**rollback + cache-trim** machinery against 12B's
154
+ distinct layer/cache shapes — NOT drafter correctness (see the
155
+ important nuance below). The calibrated e4b block (exact `toEqual` +
156
+ knife-edge) is unchanged.
157
+
158
+ The port (`drafter.ts` + `generate.ts`) is already generic over the
159
+ target's `backbone_hidden_size` — drafters are target-bonded artifacts
160
+ (e4b 2560, 12B 3840, 26B 2816) but there's no per-pair code, just the
161
+ matching weights. Expect to *validate* each cell per the project's
162
+ per-model-specialization reality, not assume it's free.
163
+
164
+ ## Important nuance: a broken drafter still produces correct tokens
165
+
166
+ Speculation can't corrupt output. The target verifies every position and
167
+ replaces every rejected draft with its own greedy pick; drafts only ever
168
+ *skip* target forwards. So a mis-wired drafter (wrong KV borrow, wrong
169
+ projection) shows up as **low/zero acceptance — a perf signal**, not bad
170
+ tokens. This splits the validation cleanly:
171
+
172
+ - **Correctness** (parity test): does the shared accept/reject/rollback
173
+ + cache-trim logic still hold on 12B's cache shapes? A trim bug *would*
174
+ corrupt output, shortly after the first rejection — that's what the
175
+ long-prefix gate catches. (Shape-incompatible KV borrow throws, it
176
+ doesn't silently corrupt.)
177
+ - **Payoff** (bench): is acceptance high enough, and the target slow
178
+ enough, that γ>0 beats the non-spec baseline? This is the actual
179
+ question.
180
+
181
+ ## Procedure (once the 12B drafter is downloaded)
182
+
183
+ 1. `hf download mlx-community/gemma-4-12B-it-assistant-bf16`
184
+ (846 MB; `HF_HUB_DISABLE_XET=1`). The 12B OptiQ-4bit target is
185
+ already cached.
186
+ 2. **Correctness first** — `bun test tests/spec-decode.test.ts`
187
+ (the `(12B)` block now runs). Long-prefix gate must pass for γ=1,2,3.
188
+ If it fails early → rollback/trim bug on 12B caches, fix before
189
+ trusting any number.
190
+ 3. **Calibrate + promote (optional)** — find a tie-free prompt
191
+ on-device, promote the 12B gate from long-prefix to exact `toEqual`
192
+ like e4b, for a stronger standing guard.
193
+ 4. **Measure** — `bun scripts/spec-bench.ts 12B` on a clean machine
194
+ (preflight matters: dirty-machine numbers are garbage). Read
195
+ acceptance + speedup-vs-baseline per γ.
196
+ 5. **Record + decide** — numbers land in the eval DB. Promote into
197
+ `benchmarks/RESULTS.md` only if quotable (clean machine).
198
+ 6. **26B follow-on** — repeat with `26B` if 12B is encouraging; MoE
199
+ target adds a validation cell (donor KV borrow through the MoE
200
+ decoder), and a slower/heavier forward that should favor speculation
201
+ even more — but the MoE forward also reads only top-8/128 experts at
202
+ decode, so the per-token cost advantage is smaller than weight size
203
+ suggests. Measure.
204
+
205
+ ## Exit criteria
206
+
207
+ - 12B spec parity gate green (rollback/trim correct on 12B caches).
208
+ - 12B acceptance + tok/s-vs-baseline recorded in the eval DB across
209
+ γ=1–4.
210
+ - A decision written down: ship speculation as the **default** only for
211
+ the (model, context) cells where it measurably wins; otherwise record
212
+ it as a characterized default-off lever (same disposition as the e4b
213
+ net-loss result — a documented option, never a deletion).
214
+
215
+ ## Caveats / interactions
216
+
217
+ - **Batched serving (Phase 18).** Speculation needs trimmable caches for
218
+ partial-accept rollback — `generate.ts` throws once a `RotatingKVCache`
219
+ ring wraps past the sliding window — and variable accept-lengths per
220
+ slot break a uniform-B batch. Speculation is naturally a **B=1 latency**
221
+ optimization; batched-throughput serving is a different regime. Layering
222
+ both is real work, not free composition.
223
+ - **One serve path, two backends.** The assistant-drafter loop and the
224
+ Qwen MTP runtime are two backends of the *same* "draft γ → verify in one
225
+ forward → accept-longest-prefix → rollback" abstraction. As Phase 14
226
+ (MTP) and Phase 18 (slots) land, build one verify/accept executor, not
227
+ two — the drafter vs MTP-head difference is just what fills the draft.
228
+ - **Denominator discipline.** Speedups are only ever vs the *same model's*
229
+ non-spec baseline. e4b's 54 tok/s is the wrong denominator for 12B.
@@ -62,7 +62,7 @@ The window renders the **same page** `mlx-bun serve` serves today
62
62
  app exit. One generation runs at a time server-side, so no client
63
63
  concurrency to manage.
64
64
  3. **Port.** Bind an ephemeral port in Rust, pass it to both the spawn
65
- (`--port`) and the webview URL, to avoid 8090 collisions with a
65
+ (`--port`) and the webview URL, to avoid 8080 collisions with a
66
66
  user's own `mlx-bun serve`.
67
67
  4. **Readiness gate.** Show a lightweight "starting…/downloading model…"
68
68
  splash until `/v1/models` returns 200. Warm start ~0.4s; first-ever
@@ -99,7 +99,7 @@ then `xcrun stapler staple`.
99
99
 
100
100
  ## Milestones
101
101
 
102
- 1. **Spike** — `tauri init` in a new repo; hardcode port 8090; spawn a
102
+ 1. **Spike** — `tauri init` in a new repo; hardcode port 8080; spawn a
103
103
  locally-built `mlx-bun serve --no-open`; load `/#/chat` in the window;
104
104
  `cargo tauri dev`. Proves the wrap end-to-end.
105
105
  2. **Robust runtime** — ephemeral port; readiness poll + splash;
@@ -0,0 +1,141 @@
1
+ # THE DREAMING — HANDOFF (2026-06-28)
2
+
3
+ Live handoff for the local personal-wiki memory system (`src/memory/`). A fresh
4
+ session can resume from this doc. **Companion docs:** `research-journal.md` (dated
5
+ findings), `docs/design/memory-inference-path.md` (the in-progress templating/batching
6
+ rework), and the per-phase `p*-judge.md` reports.
7
+
8
+ ---
9
+
10
+ ## TL;DR — where we are
11
+
12
+ The Dreaming turns conversations → chunks → entities → **subject articles** →
13
+ cross-linked, self-healing personal wiki. It **works end-to-end on real data.**
14
+
15
+ - **Full-corpus import RESUMED from cursor 720/2096** into `~/.mlx-bun/wiki-full`, on the
16
+ corrected templating, batching off (serial, ~50h).
17
+ - **Committed + merged to `main`** (the Dreaming pipeline + inference-path rework). tsc 0.
18
+ - Templating fix **VERIFIED live** on real convs (bare prompt + a 40-msg conv → valid chunks).
19
+
20
+ ## Resume the import
21
+ ```
22
+ # resumable driver; continues from cursor 720. Writes to ~/.mlx-bun/wiki-full (NOT the
23
+ # real vault ~/.mlx-bun/wiki). ~25 min / 20-conv batch, ~50h total, SERIAL.
24
+ bun scripts/experiments/dreaming-full-run.ts # resume (clean-slate guarded by an init flag)
25
+ # state files: ~/.mlx-bun/full-run-cursor.txt, ~/.mlx-bun/full-run-initialized, ~/.mlx-bun/full-run.log
26
+ # after the import finishes: reindex, then connect the graph:
27
+ mlx-bun memory link
28
+ ```
29
+ **Gotcha:** background bash + detached processes get **reaped across turns** in this
30
+ harness. Drive long GPU runs as foreground bounded batches, or `nohup … & disown` (still
31
+ iffy). The pipeline is resumable, so just re-launch to continue.
32
+
33
+ ---
34
+
35
+ ## Architecture (as-built, after this session's corrections)
36
+
37
+ **Staged, chronological, resumable pipeline** (`src/memory/stages.ts`). Each stage is an
38
+ independent DB-state-keyed worker, ordered `updated_at ASC` (chronological), resumable
39
+ (skips done work). CLI: `mlx-bun memory segment|extract|route|synthesize-stage|link` (run
40
+ separately/concurrently) — and `mlx-bun memory synthesize` runs the full DAG.
41
+
42
+ ```
43
+ ingest → SEGMENT (our e4b-chunk-300 chunker) → EXTRACT (entities) → ROUTE (subject-based,
44
+ surface-everything) → CREATE/PATCH (per-section bounded synthesis; self-healing) →
45
+ RECONCILE (infobox-ground-truth consistency) → LINK (cross-link) → WIKIFY (editorial) → commit
46
+ ```
47
+
48
+ - **Vault = truth** (markdown files in `articles/`); **DB = derived cache** (`reindex`
49
+ rebuilds it). The article FILES are the real artifacts. The `entities.article_stem`
50
+ column being unset mid-run is cosmetic (file-presence is the source of truth).
51
+ - **Vaults:** `~/.mlx-bun/wiki` (the REAL vault — untouched), `~/.mlx-bun/wiki-full`
52
+ (import target), `wiki-smoke|slice|pattern|pattern2|link-test` (scratch).
53
+ - **Model:** ONE base **Gemma-4-e4b** (`SNAPSHOT_E4B`). **Only ONE adapter trained so far:
54
+ `e4b-chunk-300`** (the chunker), symlinked `~/.cache/mlx-bun/adapters/memory-chunk`.
55
+ `callLocal(stage,…)` picks `memory-<stage>` adapter iff it exists, else base.
56
+ - **Lucien is a READ-ONLY oracle/data-source** (`~/Code/lucien/.lucien/lucien.db` opened
57
+ readonly; `~/Dreaming`). We reuse NONE of its derived chunks/buckets — only the raw
58
+ conversation text (≡ a JSONL export / scraper). Never write there.
59
+
60
+ ---
61
+
62
+ ## What we LEARNED / CORRECTED this session (load-bearing — don't re-derive)
63
+
64
+ 1. **NO notability/ownership filter.** Surface everything; notable = "anything that recurs
65
+ in your thinking" (Lucien `Editorial_Guidelines.md`). The plan's "owned/decided/≥3
66
+ chunks → PARK as trivia" was **fabricated by a prior agent and ripped out.** ROUTE now:
67
+ surface everything, strongly prefer an existing article, mint a new one only for a
68
+ substantive/recurring subject, `_captured` bucket (searchable) for the rest — **nothing
69
+ dropped.**
70
+ 2. **Subject-based articles = the one deliberate improvement over Lucien** (which drifted
71
+ into `X_and_Y` topic-bins). Verified: frameworks (Kaizen, TPS, LessWrong, Free energy
72
+ principle) get good articles; coverage resolves (`resolve("Kaizen")` → article).
73
+ 3. **Our chunker must be used.** The pipeline's `segment` was a no-op reusing Lucien's
74
+ chunks; fixed — `e4b-chunk-300` segments the raw conversations (better, topic-coherent).
75
+ 4. **Chronological ordering.** Was `ORDER BY id` (= conv UUID, random); fixed to
76
+ `updated_at ASC` everywhere. Load-bearing for self-healing.
77
+ 5. **Self-healing.** A later-dated correction flips the LEAD + INFOBOX + every body section
78
+ to the latest value; KEEPS the trajectory in a `## History` section **only for the
79
+ USER's evolving positions** — an AI *factual error* is silently overwritten (no
80
+ history). Date-aware; `reconcile.ts` is the consistency authority (infobox = ground
81
+ truth). Proven via a timestamp-swap demo (`scripts/experiments/dreaming-selfheal-demo.ts`).
82
+ 6. **Parroting (recurred 5×).** The small model copies ANY concrete example value from
83
+ prompts/Meta pages into unrelated articles (phantom S5IIX, "Anamorphic Video", etc.).
84
+ **ALL prompts + Meta pages must be SCHEMATIC** (no real names/specs/dates). Purged;
85
+ defensive strips added. Keep this rule for any new prompt.
86
+ 7. **Cross-linking is WIKIFY's job, NOT synthesis** (synthesis updates ONE article, can't
87
+ know others). Built a dedicated `link` stage (`crosslink.ts`): mention-based (prose
88
+ names another article → `[[link]]`) + co-occurrence (shared folded chunks → See also).
89
+ Result on a 587-article snapshot: See-also 18%→96%, resolved edges 309→**7026 (22.7×)**.
90
+ Idempotent; fence-fix handles malformed input.
91
+ 8. **Metrics.** Do NOT measure articles-per-conversation (there is no conv→article link).
92
+ Measure **topic aggregation** (e.g. `El_Salvador` ← 30 conversations → 1 article).
93
+ Quality is judged by a **CLOUD JUDGE on real output**, never Lucien bucket-F1 (that's the
94
+ bin taxonomy we replace).
95
+ 9. **Inference path (IN PROGRESS — see `docs/design/memory-inference-path.md`):**
96
+ - **Template mismatch (fixed in working tree):** the chunk adapter was SFT'd with a
97
+ **system role** (Gemma-4 renders `<|turn>system…<turn|>` distinctly), but inference
98
+ dropped it (one user message). `callLocal` now passes `{system, user}` matching
99
+ training; the chunk stage uses `CHUNK_SYSTEM` (its exact trained system prompt). Also
100
+ noted: the live `CHUNK_PROMPT` drifted from training (only 2 of 8 Meta pages; abstracted
101
+ examples) — adapter still generalizes, so quality delta is **modest**.
102
+ - **Batching = DEAD END.** Measured ~**1.7× SLOWER** (warm, uniform: 0.607 vs 0.357
103
+ s/chunk). The `BatchScheduler` pads heterogeneous prefills and doesn't amortize
104
+ weight-loads. Default OFF (`MLX_BUN_MEMORY_BATCH=1`). Import is genuinely ~50h serial;
105
+ batching is NOT the lever (a *better* batched-decode that amortizes weights could help
106
+ in principle, but that's a real perf project, not worth it for a one-time import).
107
+ - **Load-once / persistent server = the REAL efficiency win, for DAILY use.** Each CLI
108
+ invocation today reloads the model + pays warmup; routing memory generation through the
109
+ **resident server** (model loaded once) kills that. Within the single-process import
110
+ the model is already loaded once, so it doesn't speed the import.
111
+
112
+ ## Product framing (from the user)
113
+ - **Daily local conversations (pi sessions) folding in is the product** — watermark-gated
114
+ ingest, minutes/day, self-healing. The 4-year Claude/ChatGPT import is a one-time
115
+ **onboarding feature + stress test**. Throughput only matters for the import.
116
+
117
+ ---
118
+
119
+ ## NEXT STEPS (prioritized)
120
+
121
+ 1. **Finish the inference-path rework.** Templating fix is in the working tree (verify it
122
+ end-to-end). **Wire memory generation through the persistent server (load-once)** — the
123
+ user's actual ask — keeping a no-server fallback; default batching OFF. Verify the chunk
124
+ stage decodes on-distribution (system role present).
125
+ 2. **Import is RUNNING** (resumed from 720, serial, corrected templating). Just let it finish.
126
+ Re-segmenting the first 720 is **low-urgency** (old chunks were valid).
127
+ 3. **After import:** `reindex` → `mlx-bun memory link` → browse `wiki-full` in Obsidian;
128
+ cloud-judge a broad, diverse sample (not just frameworks/gear).
129
+ 4. **Promote to the real vault** when satisfied: point `~/.mlx-bun/wiki` at the bootstrapped
130
+ content; wire the **nightly daily fold-in** from pi sessions (`schedule.ts` exists).
131
+ 5. **Open quality items** (low priority): ~<1% articles leak a prompt fragment; some leads
132
+ run on; the `owned/used/chosen` framing in `Article_Conventions` lead spec can put
133
+ `owned: yes` on abstract concepts (reframe).
134
+ 6. **Commit when ready** — everything is uncommitted; the user has not asked to commit.
135
+
136
+ ## Gotchas / facts
137
+ - `bunx tsc` sometimes can't resolve typescript in this checkout; use global `tsc`. Repo = 0 errors.
138
+ - The `mlx-bun train-watch orpo-e4b-chunk` process is a **stale idle daemon, not training** — GPU is free.
139
+ - e4b training still OOMs on this 32GB M1 Max; P10 adapter training stays USER-ACTION (segmented-backward).
140
+ - The `Reference/` folder in vaults symlinks the mlx-bun repo docs (`setupVault`) — non-portable; fine for now, drop for the real personal vault.
141
+ - Methodology memories: cloud-judge not bucket-F1; Lucien read-only; surface-everything notability.