mlx-bun 0.0.8 → 0.0.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +112 -20
- package/docs/design/batching-v2-plan.md +515 -0
- package/docs/design/bucketing-stage.md +196 -0
- package/docs/design/compat-cli-surface-design.md +11 -11
- package/docs/design/docs-reorg-plan.md +1 -1
- package/docs/design/dreaming-nightly-pipeline.md +291 -0
- package/docs/design/dspark-speculative-decoding.md +106 -0
- package/docs/design/generic-model-support.md +381 -0
- package/docs/design/memory-inference-path.md +422 -0
- package/docs/design/memory-synthesis.md +287 -0
- package/docs/design/minicpm5-decode-megakernel.md +484 -0
- package/docs/design/mlx-lm-tool-parity-plan.md +535 -0
- package/docs/design/orpo-dynamic-lambda.md +41 -0
- package/docs/design/orpo-training.md +35 -5
- package/docs/design/parallel-slots.md +12 -1
- package/docs/design/parity-tier-dag.md +13 -5
- package/docs/design/spec-decode-larger-targets.md +229 -0
- package/docs/design/tauri-desktop-app.md +2 -2
- package/docs/design/the-dreaming-handoff.md +141 -0
- package/docs/design/the-dreaming-master-plan.md +446 -0
- package/docs/design/write-pipeline-entity.md +468 -0
- package/docs/investigations/curve-bisector-routing.md +1 -1
- package/docs/investigations/decode-roofline-lookagain.md +262 -0
- package/docs/investigations/dspark-handoff.md +58 -0
- package/docs/investigations/kernel-perf-review-2026-07.md +92 -0
- package/docs/planning/PRODUCT_ROADMAP.md +2 -2
- package/docs/planning/curve-sampler-research-plan.md +331 -0
- package/docs/planning/memory-docs-and-dag-plan.md +185 -0
- package/docs/planning/pi-integration-review.md +178 -0
- package/docs/planning/release-notes-v0.0.9.md +122 -0
- package/docs/planning/web-ui-pass-plan.md +213 -0
- package/docs/planning/website-readme-pass-plan.md +224 -0
- package/docs/reference/cli.md +255 -0
- package/docs/reference/embedding.md +7 -2
- package/docs/reference/library-api.md +22 -3
- package/docs/reference/memory.md +5 -5
- package/docs/reference/models.md +137 -0
- package/docs/reference/orpo-quickstart.md +6 -0
- package/docs/reference/server-api.md +187 -9
- package/docs/reference/server-config.md +40 -9
- package/docs/reference/training.md +25 -11
- package/package.json +1 -1
- package/src/anthropic.ts +22 -0
- package/src/assets/curve-designer.html +1 -1
- package/src/chat-template.ts +25 -7
- package/src/cli.ts +1029 -86
- package/src/config.ts +4 -1
- package/src/download.ts +99 -8
- package/src/embed.ts +77 -0
- package/src/eval/perplexity.ts +164 -0
- package/src/eval/runner.ts +15 -6
- package/src/eval/tasks/bfcl.ts +1 -1
- package/src/generate.ts +141 -18
- package/src/harness-pi.ts +43 -17
- package/src/index.ts +2 -0
- package/src/lora.ts +2 -2
- package/src/memory/article.ts +558 -0
- package/src/memory/chunk-validation.ts +150 -0
- package/src/memory/chunk.ts +305 -12
- package/src/memory/cluster.ts +344 -17
- package/src/memory/coverage.ts +134 -0
- package/src/memory/crosslink.ts +0 -0
- package/src/memory/db.ts +184 -45
- package/src/memory/entity.ts +203 -0
- package/src/memory/gate.ts +187 -0
- package/src/memory/history.ts +152 -0
- package/src/memory/ingest.ts +324 -11
- package/src/memory/model.ts +321 -0
- package/src/memory/normalize.ts +671 -0
- package/src/memory/parse.ts +22 -0
- package/src/memory/pipeline.ts +242 -30
- package/src/memory/prompts.ts +42 -0
- package/src/memory/query.ts +334 -0
- package/src/memory/reconcile.ts +734 -0
- package/src/memory/redirect.ts +108 -0
- package/src/memory/reindex.ts +325 -0
- package/src/memory/resolve.ts +307 -0
- package/src/memory/route.ts +272 -0
- package/src/memory/schedule.ts +6 -4
- package/src/memory/stages.ts +774 -0
- package/src/memory/synthesize.ts +1629 -20
- package/src/memory/tools.ts +446 -96
- package/src/memory/vault.ts +462 -35
- package/src/memory/wikify.ts +1057 -16
- package/src/mlx/ffi.ts +3 -0
- package/src/mlx/ops.ts +46 -0
- package/src/model/compiled-decode.ts +43 -8
- package/src/model/factory.ts +29 -8
- package/src/model/flash-attention.ts +4 -2
- package/src/model/fused-decode-kernel.ts +14 -7
- package/src/model/fused-mlp-kernel.ts +169 -0
- package/src/model/fused-swiglu-kernel.ts +240 -0
- package/src/model/gemma4-base.ts +62 -6
- package/src/model/gemma4.ts +33 -2
- package/src/model/minicpm5.ts +106 -8
- package/src/model/qwen3.ts +276 -0
- package/src/model/steel-linear-kernel.ts +96 -0
- package/src/model/support.ts +38 -8
- package/src/model/universal/archs.ts +332 -0
- package/src/model/universal/dense.ts +576 -0
- package/src/model/universal/modules.ts +163 -0
- package/src/model/universal/rope.ts +366 -0
- package/src/pi-provider.ts +1 -1
- package/src/pi-terminal.ts +29 -8
- package/src/pi-web.ts +19 -2
- package/src/quantize/config-writer.ts +1 -1
- package/src/registry.ts +215 -7
- package/src/responses.ts +23 -0
- package/src/sampler.ts +220 -9
- package/src/serve/batch-scheduler.ts +272 -80
- package/src/serve/generation-gateway.ts +86 -18
- package/src/server.ts +612 -49
- package/src/spec/drafter.ts +81 -49
- package/src/spec/dspark/data-dflash.ts +156 -0
- package/src/spec/dspark/data.ts +201 -0
- package/src/spec/dspark/generate-dflash.ts +193 -0
- package/src/spec/dspark/generate.ts +290 -0
- package/src/spec/dspark/loss.ts +213 -0
- package/src/spec/dspark/module-dflash.ts +361 -0
- package/src/spec/dspark/module.ts +540 -0
- package/src/spec/dspark/sample.ts +112 -0
- package/src/spec/generate.ts +40 -19
- package/src/status-page.html +1 -1
- package/src/tokenizer.ts +7 -0
- package/src/train/flash-cce.ts +70 -15
- package/src/train/fuse.ts +243 -0
- package/src/train/index.ts +3 -0
- package/src/train/job.ts +8 -0
- package/src/train/loss.ts +289 -30
- package/src/train/prefix-shared.ts +33 -5
- package/src/train/segmented.ts +67 -11
- package/src/train/trainer.ts +92 -28
- package/src/web/skills/memory/SKILL.md +40 -17
|
@@ -61,16 +61,24 @@ Worked examples:
|
|
|
61
61
|
| lever | parity | perf | placement |
|
|
62
62
|
|---|---|---|---|
|
|
63
63
|
| `compiled-decode` | bit-exact w/ uncompiled (`compiled-decode.test`) | faster | **default-on, every level** |
|
|
64
|
-
| `perf-kernel` |
|
|
65
|
-
| `fused-sdpa` | matches optiq, not mlx-lm (`fused-sdpa.test`) | faster | **default-on from L2** (the L2 bridge) |
|
|
64
|
+
| `perf-kernel` | **envelope-gated, no bit-exact oracle** — mlx-bun-original kernel; gate is ≥56/64 teacher-forced argmax vs *our own* frozen compat trajectory (`perf-kernel-oracle.test`, `freeze-perf-oracle.ts`), and the optiq kv-quant goldens track the **stock unfused L=1 decode** (`regen-kvq-goldens.ts`) | faster | **L3 / explicit opt-in only** (off in bare `--l2`; on in `--l3` and the no-tier default) |
|
|
65
|
+
| `fused-sdpa` | matches optiq bit-for-bit, not mlx-lm (`fused-sdpa.test` tier-a goldens) | faster | **default-on from L2** (the L2 bridge) |
|
|
66
66
|
| `fused-decode` | (would be L3) | ~3% slower (`optimization_plan.md`) | **kept, off everywhere** |
|
|
67
67
|
|
|
68
68
|
So `--l1/--l2/--l3` is the **parity** axis (the guarantee), and the defaults within
|
|
69
|
-
each tier are the **performance** axis (the fastest kernel that still holds it).
|
|
70
|
-
the *decode path* L2
|
|
71
|
-
|
|
69
|
+
each tier are the **performance** axis (the fastest kernel that still holds it). On
|
|
70
|
+
the *decode path* L2 and L3 now differ by exactly one node — the perf kernel (bare
|
|
71
|
+
`--l2` = the optiq-golden composition: fused N-tiled prefill + stock unfused L=1
|
|
72
|
+
decode; `--l3` adds the envelope-gated fused decode kernel). L3's remaining
|
|
73
|
+
territory is the no-oracle **features** — HLG sampler, expert offload, batched
|
|
72
74
|
mixed-precision — which live off the decode axis and are gated by KL + quality.
|
|
73
75
|
|
|
76
|
+
> History: commit `f1bf5cc` put the perf kernel in the L2 preset claiming
|
|
77
|
+
> "perf-kernel-oracle.test asserts the fused kernel == the frozen optiq goldens".
|
|
78
|
+
> Both halves were wrong — that golden is frozen from mlx-bun's own compat engine,
|
|
79
|
+
> and the gate is argmax-agreement, not equality — so 2026-07-01 restored the tier
|
|
80
|
+
> rule: the bare tier is the guarantee; perf kernels are opt-ins within it.
|
|
81
|
+
|
|
74
82
|
## Why this matters for CLI flags
|
|
75
83
|
|
|
76
84
|
We are accumulating a **lot** of flags (`--grad-accum`, `--grad-clip`, `--seg`,
|
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
# Spec decode on larger targets — does the Gemma drafter pay on 12B / 26B?
|
|
2
|
+
|
|
3
|
+
Status: **12B γ=1 is a WIN after the verify fix; matches optiq bit-exactly** (2026-06-14; 26B not run)
|
|
4
|
+
Owner: decode / spec
|
|
5
|
+
Default: **off** (speculation is opt-in; non-spec greedy is untouched)
|
|
6
|
+
|
|
7
|
+
## Result (2026-06-14, v2) — fixed the wrong-oracle bug; γ=1 now wins
|
|
8
|
+
|
|
9
|
+
The first 12B measurement (v1, below) was made through a verify path that
|
|
10
|
+
targeted the **wrong oracle**. Fixing it changed the conclusion.
|
|
11
|
+
|
|
12
|
+
### The wrong-oracle bug
|
|
13
|
+
|
|
14
|
+
Assistant-drafter speculation is an **optiq** feature — mlx-lm has only
|
|
15
|
+
generic standalone-`draft_model` two-model spec and structurally cannot
|
|
16
|
+
drive a KV-borrowing drafter. So the correctness oracle is optiq's
|
|
17
|
+
`spec_generate`. **Both optiq and mlx-lm BATCH the verify lm-head** (one
|
|
18
|
+
matmul over the γ+1 window, then argmax per position); neither is bit-
|
|
19
|
+
exact to stock token-at-a-time decode. Our port instead verified
|
|
20
|
+
**per-position** to be bit-exact to *stock decode* — a target no real
|
|
21
|
+
implementation has, and one that re-reads the ~1 GB tied lm-head matrix
|
|
22
|
+
γ+1× per iteration instead of once. So we were exact to the wrong thing
|
|
23
|
+
*and* paying a γ-scaling tax for it. Fix: `picksBatched` in
|
|
24
|
+
`src/spec/generate.ts` — one batched verify lm-head, matching optiq.
|
|
25
|
+
|
|
26
|
+
### Do we match optiq? YES — bit-exact (e4b)
|
|
27
|
+
|
|
28
|
+
Cross-checked our `specGenerate` against optiq's `spec_generate` on the
|
|
29
|
+
identical prompt ids (`scripts/oracle-spec.py` + `scripts/spec-dump.ts`),
|
|
30
|
+
e4b γ=2: **identical 48-token output AND identical accept/reject trace**
|
|
31
|
+
(drafted 60, accepted 17, target calls 31 in both). Before the fix we
|
|
32
|
+
matched *stock*, not optiq, at knife-edges; now we match optiq.
|
|
33
|
+
|
|
34
|
+
### Is it still a loss? Not at γ=1 (12B)
|
|
35
|
+
|
|
36
|
+
12B pair, batched verify (loaded machine — ratios paired/trustworthy,
|
|
37
|
+
absolutes depressed & not quotable):
|
|
38
|
+
|
|
39
|
+
| γ | v2 speedup (batched) | v1 (per-position) | acceptance |
|
|
40
|
+
|---|----------------------|-------------------|-----------|
|
|
41
|
+
| 1 | **1.09× (WIN)** | 0.96× | 42% |
|
|
42
|
+
| 2 | 0.91× (near break-even) | 0.77× | 29% |
|
|
43
|
+
| 3 | 0.72× | 0.67× | 23% |
|
|
44
|
+
| 4 | 0.56× | 0.52× | 17% |
|
|
45
|
+
|
|
46
|
+
Acceptance is unchanged (the fix only touches verify cost, not which
|
|
47
|
+
tokens are accepted). The per-position verify was a large self-inflicted
|
|
48
|
+
tax: removing it flips γ=1 from loss to **win** and moves γ=2 to the edge.
|
|
49
|
+
γ≥2 still loses because the *drafter's own* cost dominates there — the
|
|
50
|
+
12B drafter is heavy (hidden 1024 vs e4b 256, 16 heads, and a full 262k
|
|
51
|
+
tied-embedding head **every draft step**) and acceptance falls with depth.
|
|
52
|
+
|
|
53
|
+
**So the v1 "hypothesis refuted / net loss at every γ" was itself partly
|
|
54
|
+
an artifact of the wrong-oracle verify.** Corrected conclusion: γ=1
|
|
55
|
+
speculation on 12B is a real ~9% decode win (pending clean-machine
|
|
56
|
+
confirmation before quoting); γ≥2 doesn't pay with the shipped heavy
|
|
57
|
+
drafter.
|
|
58
|
+
|
|
59
|
+
### Bonus finding: we're ahead of optiq on the unified drafters
|
|
60
|
+
|
|
61
|
+
optiq's *installed* runtime can't actually drive the 12B/26B
|
|
62
|
+
`gemma4_unified_assistant` drafters: its head detection is config-based
|
|
63
|
+
(`use_ordered_embeddings = num_centroids and top_k`), but those artifacts
|
|
64
|
+
declare `num_centroids` while shipping **no** centroid tensors and **no**
|
|
65
|
+
`lm_head` (tied `embed_tokens` instead). optiq loads the centroid path
|
|
66
|
+
with zero-init weights → garbage drafts → **0% acceptance** (measured:
|
|
67
|
+
drafted 46, accepted 0 on 12B). Our `drafter.ts` detects the head by
|
|
68
|
+
**tensor presence** and uses the tied-embedding path → 29% acceptance on
|
|
69
|
+
the same artifact.
|
|
70
|
+
|
|
71
|
+
### Remaining levers / next
|
|
72
|
+
|
|
73
|
+
- **Clean-machine rerun** before promoting any number to RESULTS.md
|
|
74
|
+
(baseline jumped 15.4→24.4 between runs under memory pressure — exactly
|
|
75
|
+
why only paired ratios are trusted here).
|
|
76
|
+
- **Cheaper drafter head**: the 262k tied-embed argmax every draft step is
|
|
77
|
+
the dominant remaining γ≥2 cost; capping/approximating it (e4b-style)
|
|
78
|
+
could extend the win past γ=1.
|
|
79
|
+
- **26B-MoE**: still unlikely — MoE decode reads only top-8/128 experts
|
|
80
|
+
(~2.4 GB active), so per-token target cost is *lower* than its size →
|
|
81
|
+
harder to beat. Measure if curious.
|
|
82
|
+
- **Optional strict mode**: the old per-position verify gave "spec output
|
|
83
|
+
identical to spec-off" — a product nicety no oracle provides. Could be
|
|
84
|
+
kept as a default-off `strictVerify` flag rather than deleted
|
|
85
|
+
([[dont-delete-optionality-on-one-measurement]]) — not implemented; ask.
|
|
86
|
+
|
|
87
|
+
## v1 (superseded) — per-position verify, "net loss at every γ"
|
|
88
|
+
|
|
89
|
+
> Kept for the record. The numbers (0.96/0.77/0.67/0.52×) were real but
|
|
90
|
+
> measured through the wrong-oracle per-position verify; v2 above is the
|
|
91
|
+
> optiq-faithful measurement.
|
|
92
|
+
|
|
93
|
+
## TL;DR
|
|
94
|
+
|
|
95
|
+
The Gemma `-assistant-bf16` drafters are Gemma's substitute for an MTP
|
|
96
|
+
head: a 4-layer Q-only module that reuses the target's KV + hidden state
|
|
97
|
+
to draft tokens (`src/spec/drafter.ts`), verified by the target's own
|
|
98
|
+
greedy pick (`src/spec/generate.ts`). True MTP (a trained
|
|
99
|
+
`mtp.safetensors` head + an `optiq/runtime/mtp` backend) **does not exist
|
|
100
|
+
for Gemma** and never will — that's the Qwen path (Phase 14). The
|
|
101
|
+
assistant drafter is the functional equivalent we already have.
|
|
102
|
+
|
|
103
|
+
It was measured a **net loss on e4b** (Phase 6): γ=2 ran at 0.78× the
|
|
104
|
+
54 tok/s baseline, ~23% acceptance — because e4b decode is already too
|
|
105
|
+
fast for a low-acceptance drafter to beat. **This doc is the experiment
|
|
106
|
+
to find out whether the same drafter wins on a slower target**, where
|
|
107
|
+
each skipped target-forward is worth more.
|
|
108
|
+
|
|
109
|
+
## The hypothesis
|
|
110
|
+
|
|
111
|
+
The drafter's per-step cost is roughly fixed. The *value* of each
|
|
112
|
+
accepted draft = one skipped target forward, which grows as the target
|
|
113
|
+
gets slower per token. e4b is the worst case for speculation (fastest
|
|
114
|
+
target). So the SAME acceptance rate can flip net-positive as the target
|
|
115
|
+
slows:
|
|
116
|
+
|
|
117
|
+
| target | non-spec decode | per-accepted-draft value | drafter | status |
|
|
118
|
+
|--------|-----------------|--------------------------|---------|--------|
|
|
119
|
+
| e4b | ~54 tok/s | low | 160 MB (cached) | **measured net loss** (0.91/0.78/0.63/0.51× @ γ=1–4) |
|
|
120
|
+
| 12B | ~25 tok/s | ~2× e4b | 846 MB (cached) | **measured net loss** — see Result above (hypothesis refuted) |
|
|
121
|
+
| 26B-A4B (MoE) | ~32 tok/s | high (MoE forward) | 839 MB (HF, **not downloaded**) | untested (follow-on; expect loss) |
|
|
122
|
+
|
|
123
|
+
> The reasoning below is the *pre-measurement* hypothesis — kept for the
|
|
124
|
+
> record. It predicted the crossover assuming a fixed-cost drafter; the
|
|
125
|
+
> 12B drafter is not fixed-cost (it scales with the target), which is
|
|
126
|
+
> exactly why the Result section above refutes it.
|
|
127
|
+
|
|
128
|
+
Crossover intuition: a drafter is a net win when
|
|
129
|
+
`acceptance × (target_step_cost − draft_step_cost) > verify_overhead`.
|
|
130
|
+
On e4b the target step is cheap, so the bar isn't cleared at 23%. On 12B
|
|
131
|
+
the target step is ~2× more expensive while the drafter cost barely
|
|
132
|
+
moves — same acceptance, very different sign. **Measure, don't assume**
|
|
133
|
+
(it could still lose; this is exactly the kind of thing the e4b result
|
|
134
|
+
warned against assuming).
|
|
135
|
+
|
|
136
|
+
Acceptance itself may also differ on 12B (a stronger target predicts
|
|
137
|
+
itself differently). We don't know the direction — that's why we run it.
|
|
138
|
+
|
|
139
|
+
## What is already wired (this change)
|
|
140
|
+
|
|
141
|
+
- **`scripts/spec-bench.ts`** — now multi-pair, **one pair per process**
|
|
142
|
+
(memory-safe; 12B+drafter ≈ 8 GB, 26B ≈ 17 GB, never co-resident):
|
|
143
|
+
- `bun scripts/spec-bench.ts e4b` (default) — reproduces the baseline.
|
|
144
|
+
- `bun scripts/spec-bench.ts 12B` — the experiment.
|
|
145
|
+
- `bun scripts/spec-bench.ts 26B` — follow-on.
|
|
146
|
+
- Auto-detects whether each `(target, drafter)` snapshot pair is
|
|
147
|
+
present; exits with a clear "download with: hf download …" message
|
|
148
|
+
otherwise. Records baseline + γ-sweep (acceptance + tok/s) to the
|
|
149
|
+
eval DB tagged `spec-bench[<id>]`.
|
|
150
|
+
- **`tests/spec-decode.test.ts`** — added a `(12B)` parity block,
|
|
151
|
+
auto-skipped until the drafter is downloaded. Gate = long-prefix
|
|
152
|
+
agreement (greedy spec vs greedy non-spec). It guards the shared
|
|
153
|
+
accept/reject/**rollback + cache-trim** machinery against 12B's
|
|
154
|
+
distinct layer/cache shapes — NOT drafter correctness (see the
|
|
155
|
+
important nuance below). The calibrated e4b block (exact `toEqual` +
|
|
156
|
+
knife-edge) is unchanged.
|
|
157
|
+
|
|
158
|
+
The port (`drafter.ts` + `generate.ts`) is already generic over the
|
|
159
|
+
target's `backbone_hidden_size` — drafters are target-bonded artifacts
|
|
160
|
+
(e4b 2560, 12B 3840, 26B 2816) but there's no per-pair code, just the
|
|
161
|
+
matching weights. Expect to *validate* each cell per the project's
|
|
162
|
+
per-model-specialization reality, not assume it's free.
|
|
163
|
+
|
|
164
|
+
## Important nuance: a broken drafter still produces correct tokens
|
|
165
|
+
|
|
166
|
+
Speculation can't corrupt output. The target verifies every position and
|
|
167
|
+
replaces every rejected draft with its own greedy pick; drafts only ever
|
|
168
|
+
*skip* target forwards. So a mis-wired drafter (wrong KV borrow, wrong
|
|
169
|
+
projection) shows up as **low/zero acceptance — a perf signal**, not bad
|
|
170
|
+
tokens. This splits the validation cleanly:
|
|
171
|
+
|
|
172
|
+
- **Correctness** (parity test): does the shared accept/reject/rollback
|
|
173
|
+
+ cache-trim logic still hold on 12B's cache shapes? A trim bug *would*
|
|
174
|
+
corrupt output, shortly after the first rejection — that's what the
|
|
175
|
+
long-prefix gate catches. (Shape-incompatible KV borrow throws, it
|
|
176
|
+
doesn't silently corrupt.)
|
|
177
|
+
- **Payoff** (bench): is acceptance high enough, and the target slow
|
|
178
|
+
enough, that γ>0 beats the non-spec baseline? This is the actual
|
|
179
|
+
question.
|
|
180
|
+
|
|
181
|
+
## Procedure (once the 12B drafter is downloaded)
|
|
182
|
+
|
|
183
|
+
1. `hf download mlx-community/gemma-4-12B-it-assistant-bf16`
|
|
184
|
+
(846 MB; `HF_HUB_DISABLE_XET=1`). The 12B OptiQ-4bit target is
|
|
185
|
+
already cached.
|
|
186
|
+
2. **Correctness first** — `bun test tests/spec-decode.test.ts`
|
|
187
|
+
(the `(12B)` block now runs). Long-prefix gate must pass for γ=1,2,3.
|
|
188
|
+
If it fails early → rollback/trim bug on 12B caches, fix before
|
|
189
|
+
trusting any number.
|
|
190
|
+
3. **Calibrate + promote (optional)** — find a tie-free prompt
|
|
191
|
+
on-device, promote the 12B gate from long-prefix to exact `toEqual`
|
|
192
|
+
like e4b, for a stronger standing guard.
|
|
193
|
+
4. **Measure** — `bun scripts/spec-bench.ts 12B` on a clean machine
|
|
194
|
+
(preflight matters: dirty-machine numbers are garbage). Read
|
|
195
|
+
acceptance + speedup-vs-baseline per γ.
|
|
196
|
+
5. **Record + decide** — numbers land in the eval DB. Promote into
|
|
197
|
+
`benchmarks/RESULTS.md` only if quotable (clean machine).
|
|
198
|
+
6. **26B follow-on** — repeat with `26B` if 12B is encouraging; MoE
|
|
199
|
+
target adds a validation cell (donor KV borrow through the MoE
|
|
200
|
+
decoder), and a slower/heavier forward that should favor speculation
|
|
201
|
+
even more — but the MoE forward also reads only top-8/128 experts at
|
|
202
|
+
decode, so the per-token cost advantage is smaller than weight size
|
|
203
|
+
suggests. Measure.
|
|
204
|
+
|
|
205
|
+
## Exit criteria
|
|
206
|
+
|
|
207
|
+
- 12B spec parity gate green (rollback/trim correct on 12B caches).
|
|
208
|
+
- 12B acceptance + tok/s-vs-baseline recorded in the eval DB across
|
|
209
|
+
γ=1–4.
|
|
210
|
+
- A decision written down: ship speculation as the **default** only for
|
|
211
|
+
the (model, context) cells where it measurably wins; otherwise record
|
|
212
|
+
it as a characterized default-off lever (same disposition as the e4b
|
|
213
|
+
net-loss result — a documented option, never a deletion).
|
|
214
|
+
|
|
215
|
+
## Caveats / interactions
|
|
216
|
+
|
|
217
|
+
- **Batched serving (Phase 18).** Speculation needs trimmable caches for
|
|
218
|
+
partial-accept rollback — `generate.ts` throws once a `RotatingKVCache`
|
|
219
|
+
ring wraps past the sliding window — and variable accept-lengths per
|
|
220
|
+
slot break a uniform-B batch. Speculation is naturally a **B=1 latency**
|
|
221
|
+
optimization; batched-throughput serving is a different regime. Layering
|
|
222
|
+
both is real work, not free composition.
|
|
223
|
+
- **One serve path, two backends.** The assistant-drafter loop and the
|
|
224
|
+
Qwen MTP runtime are two backends of the *same* "draft γ → verify in one
|
|
225
|
+
forward → accept-longest-prefix → rollback" abstraction. As Phase 14
|
|
226
|
+
(MTP) and Phase 18 (slots) land, build one verify/accept executor, not
|
|
227
|
+
two — the drafter vs MTP-head difference is just what fills the draft.
|
|
228
|
+
- **Denominator discipline.** Speedups are only ever vs the *same model's*
|
|
229
|
+
non-spec baseline. e4b's 54 tok/s is the wrong denominator for 12B.
|
|
@@ -62,7 +62,7 @@ The window renders the **same page** `mlx-bun serve` serves today
|
|
|
62
62
|
app exit. One generation runs at a time server-side, so no client
|
|
63
63
|
concurrency to manage.
|
|
64
64
|
3. **Port.** Bind an ephemeral port in Rust, pass it to both the spawn
|
|
65
|
-
(`--port`) and the webview URL, to avoid
|
|
65
|
+
(`--port`) and the webview URL, to avoid 8080 collisions with a
|
|
66
66
|
user's own `mlx-bun serve`.
|
|
67
67
|
4. **Readiness gate.** Show a lightweight "starting…/downloading model…"
|
|
68
68
|
splash until `/v1/models` returns 200. Warm start ~0.4s; first-ever
|
|
@@ -99,7 +99,7 @@ then `xcrun stapler staple`.
|
|
|
99
99
|
|
|
100
100
|
## Milestones
|
|
101
101
|
|
|
102
|
-
1. **Spike** — `tauri init` in a new repo; hardcode port
|
|
102
|
+
1. **Spike** — `tauri init` in a new repo; hardcode port 8080; spawn a
|
|
103
103
|
locally-built `mlx-bun serve --no-open`; load `/#/chat` in the window;
|
|
104
104
|
`cargo tauri dev`. Proves the wrap end-to-end.
|
|
105
105
|
2. **Robust runtime** — ephemeral port; readiness poll + splash;
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# THE DREAMING — HANDOFF (2026-06-28)
|
|
2
|
+
|
|
3
|
+
Live handoff for the local personal-wiki memory system (`src/memory/`). A fresh
|
|
4
|
+
session can resume from this doc. **Companion docs:** `research-journal.md` (dated
|
|
5
|
+
findings), `docs/design/memory-inference-path.md` (the in-progress templating/batching
|
|
6
|
+
rework), and the per-phase `p*-judge.md` reports.
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## TL;DR — where we are
|
|
11
|
+
|
|
12
|
+
The Dreaming turns conversations → chunks → entities → **subject articles** →
|
|
13
|
+
cross-linked, self-healing personal wiki. It **works end-to-end on real data.**
|
|
14
|
+
|
|
15
|
+
- **Full-corpus import RESUMED from cursor 720/2096** into `~/.mlx-bun/wiki-full`, on the
|
|
16
|
+
corrected templating, batching off (serial, ~50h).
|
|
17
|
+
- **Committed + merged to `main`** (the Dreaming pipeline + inference-path rework). tsc 0.
|
|
18
|
+
- Templating fix **VERIFIED live** on real convs (bare prompt + a 40-msg conv → valid chunks).
|
|
19
|
+
|
|
20
|
+
## Resume the import
|
|
21
|
+
```
|
|
22
|
+
# resumable driver; continues from cursor 720. Writes to ~/.mlx-bun/wiki-full (NOT the
|
|
23
|
+
# real vault ~/.mlx-bun/wiki). ~25 min / 20-conv batch, ~50h total, SERIAL.
|
|
24
|
+
bun scripts/experiments/dreaming-full-run.ts # resume (clean-slate guarded by an init flag)
|
|
25
|
+
# state files: ~/.mlx-bun/full-run-cursor.txt, ~/.mlx-bun/full-run-initialized, ~/.mlx-bun/full-run.log
|
|
26
|
+
# after the import finishes: reindex, then connect the graph:
|
|
27
|
+
mlx-bun memory link
|
|
28
|
+
```
|
|
29
|
+
**Gotcha:** background bash + detached processes get **reaped across turns** in this
|
|
30
|
+
harness. Drive long GPU runs as foreground bounded batches, or `nohup … & disown` (still
|
|
31
|
+
iffy). The pipeline is resumable, so just re-launch to continue.
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## Architecture (as-built, after this session's corrections)
|
|
36
|
+
|
|
37
|
+
**Staged, chronological, resumable pipeline** (`src/memory/stages.ts`). Each stage is an
|
|
38
|
+
independent DB-state-keyed worker, ordered `updated_at ASC` (chronological), resumable
|
|
39
|
+
(skips done work). CLI: `mlx-bun memory segment|extract|route|synthesize-stage|link` (run
|
|
40
|
+
separately/concurrently) — and `mlx-bun memory synthesize` runs the full DAG.
|
|
41
|
+
|
|
42
|
+
```
|
|
43
|
+
ingest → SEGMENT (our e4b-chunk-300 chunker) → EXTRACT (entities) → ROUTE (subject-based,
|
|
44
|
+
surface-everything) → CREATE/PATCH (per-section bounded synthesis; self-healing) →
|
|
45
|
+
RECONCILE (infobox-ground-truth consistency) → LINK (cross-link) → WIKIFY (editorial) → commit
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
- **Vault = truth** (markdown files in `articles/`); **DB = derived cache** (`reindex`
|
|
49
|
+
rebuilds it). The article FILES are the real artifacts. The `entities.article_stem`
|
|
50
|
+
column being unset mid-run is cosmetic (file-presence is the source of truth).
|
|
51
|
+
- **Vaults:** `~/.mlx-bun/wiki` (the REAL vault — untouched), `~/.mlx-bun/wiki-full`
|
|
52
|
+
(import target), `wiki-smoke|slice|pattern|pattern2|link-test` (scratch).
|
|
53
|
+
- **Model:** ONE base **Gemma-4-e4b** (`SNAPSHOT_E4B`). **Only ONE adapter trained so far:
|
|
54
|
+
`e4b-chunk-300`** (the chunker), symlinked `~/.cache/mlx-bun/adapters/memory-chunk`.
|
|
55
|
+
`callLocal(stage,…)` picks `memory-<stage>` adapter iff it exists, else base.
|
|
56
|
+
- **Lucien is a READ-ONLY oracle/data-source** (`~/Code/lucien/.lucien/lucien.db` opened
|
|
57
|
+
readonly; `~/Dreaming`). We reuse NONE of its derived chunks/buckets — only the raw
|
|
58
|
+
conversation text (≡ a JSONL export / scraper). Never write there.
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## What we LEARNED / CORRECTED this session (load-bearing — don't re-derive)
|
|
63
|
+
|
|
64
|
+
1. **NO notability/ownership filter.** Surface everything; notable = "anything that recurs
|
|
65
|
+
in your thinking" (Lucien `Editorial_Guidelines.md`). The plan's "owned/decided/≥3
|
|
66
|
+
chunks → PARK as trivia" was **fabricated by a prior agent and ripped out.** ROUTE now:
|
|
67
|
+
surface everything, strongly prefer an existing article, mint a new one only for a
|
|
68
|
+
substantive/recurring subject, `_captured` bucket (searchable) for the rest — **nothing
|
|
69
|
+
dropped.**
|
|
70
|
+
2. **Subject-based articles = the one deliberate improvement over Lucien** (which drifted
|
|
71
|
+
into `X_and_Y` topic-bins). Verified: frameworks (Kaizen, TPS, LessWrong, Free energy
|
|
72
|
+
principle) get good articles; coverage resolves (`resolve("Kaizen")` → article).
|
|
73
|
+
3. **Our chunker must be used.** The pipeline's `segment` was a no-op reusing Lucien's
|
|
74
|
+
chunks; fixed — `e4b-chunk-300` segments the raw conversations (better, topic-coherent).
|
|
75
|
+
4. **Chronological ordering.** Was `ORDER BY id` (= conv UUID, random); fixed to
|
|
76
|
+
`updated_at ASC` everywhere. Load-bearing for self-healing.
|
|
77
|
+
5. **Self-healing.** A later-dated correction flips the LEAD + INFOBOX + every body section
|
|
78
|
+
to the latest value; KEEPS the trajectory in a `## History` section **only for the
|
|
79
|
+
USER's evolving positions** — an AI *factual error* is silently overwritten (no
|
|
80
|
+
history). Date-aware; `reconcile.ts` is the consistency authority (infobox = ground
|
|
81
|
+
truth). Proven via a timestamp-swap demo (`scripts/experiments/dreaming-selfheal-demo.ts`).
|
|
82
|
+
6. **Parroting (recurred 5×).** The small model copies ANY concrete example value from
|
|
83
|
+
prompts/Meta pages into unrelated articles (phantom S5IIX, "Anamorphic Video", etc.).
|
|
84
|
+
**ALL prompts + Meta pages must be SCHEMATIC** (no real names/specs/dates). Purged;
|
|
85
|
+
defensive strips added. Keep this rule for any new prompt.
|
|
86
|
+
7. **Cross-linking is WIKIFY's job, NOT synthesis** (synthesis updates ONE article, can't
|
|
87
|
+
know others). Built a dedicated `link` stage (`crosslink.ts`): mention-based (prose
|
|
88
|
+
names another article → `[[link]]`) + co-occurrence (shared folded chunks → See also).
|
|
89
|
+
Result on a 587-article snapshot: See-also 18%→96%, resolved edges 309→**7026 (22.7×)**.
|
|
90
|
+
Idempotent; fence-fix handles malformed input.
|
|
91
|
+
8. **Metrics.** Do NOT measure articles-per-conversation (there is no conv→article link).
|
|
92
|
+
Measure **topic aggregation** (e.g. `El_Salvador` ← 30 conversations → 1 article).
|
|
93
|
+
Quality is judged by a **CLOUD JUDGE on real output**, never Lucien bucket-F1 (that's the
|
|
94
|
+
bin taxonomy we replace).
|
|
95
|
+
9. **Inference path (IN PROGRESS — see `docs/design/memory-inference-path.md`):**
|
|
96
|
+
- **Template mismatch (fixed in working tree):** the chunk adapter was SFT'd with a
|
|
97
|
+
**system role** (Gemma-4 renders `<|turn>system…<turn|>` distinctly), but inference
|
|
98
|
+
dropped it (one user message). `callLocal` now passes `{system, user}` matching
|
|
99
|
+
training; the chunk stage uses `CHUNK_SYSTEM` (its exact trained system prompt). Also
|
|
100
|
+
noted: the live `CHUNK_PROMPT` drifted from training (only 2 of 8 Meta pages; abstracted
|
|
101
|
+
examples) — adapter still generalizes, so quality delta is **modest**.
|
|
102
|
+
- **Batching = DEAD END.** Measured ~**1.7× SLOWER** (warm, uniform: 0.607 vs 0.357
|
|
103
|
+
s/chunk). The `BatchScheduler` pads heterogeneous prefills and doesn't amortize
|
|
104
|
+
weight-loads. Default OFF (`MLX_BUN_MEMORY_BATCH=1`). Import is genuinely ~50h serial;
|
|
105
|
+
batching is NOT the lever (a *better* batched-decode that amortizes weights could help
|
|
106
|
+
in principle, but that's a real perf project, not worth it for a one-time import).
|
|
107
|
+
- **Load-once / persistent server = the REAL efficiency win, for DAILY use.** Each CLI
|
|
108
|
+
invocation today reloads the model + pays warmup; routing memory generation through the
|
|
109
|
+
**resident server** (model loaded once) kills that. Within the single-process import
|
|
110
|
+
the model is already loaded once, so it doesn't speed the import.
|
|
111
|
+
|
|
112
|
+
## Product framing (from the user)
|
|
113
|
+
- **Daily local conversations (pi sessions) folding in is the product** — watermark-gated
|
|
114
|
+
ingest, minutes/day, self-healing. The 4-year Claude/ChatGPT import is a one-time
|
|
115
|
+
**onboarding feature + stress test**. Throughput only matters for the import.
|
|
116
|
+
|
|
117
|
+
---
|
|
118
|
+
|
|
119
|
+
## NEXT STEPS (prioritized)
|
|
120
|
+
|
|
121
|
+
1. **Finish the inference-path rework.** Templating fix is in the working tree (verify it
|
|
122
|
+
end-to-end). **Wire memory generation through the persistent server (load-once)** — the
|
|
123
|
+
user's actual ask — keeping a no-server fallback; default batching OFF. Verify the chunk
|
|
124
|
+
stage decodes on-distribution (system role present).
|
|
125
|
+
2. **Import is RUNNING** (resumed from 720, serial, corrected templating). Just let it finish.
|
|
126
|
+
Re-segmenting the first 720 is **low-urgency** (old chunks were valid).
|
|
127
|
+
3. **After import:** `reindex` → `mlx-bun memory link` → browse `wiki-full` in Obsidian;
|
|
128
|
+
cloud-judge a broad, diverse sample (not just frameworks/gear).
|
|
129
|
+
4. **Promote to the real vault** when satisfied: point `~/.mlx-bun/wiki` at the bootstrapped
|
|
130
|
+
content; wire the **nightly daily fold-in** from pi sessions (`schedule.ts` exists).
|
|
131
|
+
5. **Open quality items** (low priority): ~<1% articles leak a prompt fragment; some leads
|
|
132
|
+
run on; the `owned/used/chosen` framing in `Article_Conventions` lead spec can put
|
|
133
|
+
`owned: yes` on abstract concepts (reframe).
|
|
134
|
+
6. **Commit when ready** — everything is uncommitted; the user has not asked to commit.
|
|
135
|
+
|
|
136
|
+
## Gotchas / facts
|
|
137
|
+
- `bunx tsc` sometimes can't resolve typescript in this checkout; use global `tsc`. Repo = 0 errors.
|
|
138
|
+
- The `mlx-bun train-watch orpo-e4b-chunk` process is a **stale idle daemon, not training** — GPU is free.
|
|
139
|
+
- e4b training still OOMs on this 32GB M1 Max; P10 adapter training stays USER-ACTION (segmented-backward).
|
|
140
|
+
- The `Reference/` folder in vaults symlinks the mlx-bun repo docs (`setupVault`) — non-portable; fine for now, drop for the real personal vault.
|
|
141
|
+
- Methodology memories: cloud-judge not bucket-F1; Lucien read-only; surface-everything notability.
|