@threadedstack/wmd-web 0.0.0-stage → 0.1.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (92) hide show
  1. package/README.md +2601 -2
  2. package/RELEASE.json +123 -0
  3. package/export-page/coi-sw.js +20 -0
  4. package/export-page/index.html +17 -0
  5. package/export-page/run.css +10 -0
  6. package/export-page/run.js +236 -0
  7. package/guest/arm64/Image +4 -0
  8. package/guest/arm64/Image-4k +4 -0
  9. package/guest/arm64/initramfs.cpio.gz +0 -0
  10. package/guest/arm64/vm.wasm +0 -0
  11. package/package.json +36 -4
  12. package/page/sw.js +381 -0
  13. package/relay/control.ts +139 -0
  14. package/relay/engine.ts +787 -0
  15. package/relay/hosted.ts +660 -0
  16. package/relay/keys.ts +53 -0
  17. package/relay/local-tls.ts +121 -0
  18. package/relay/lock.ts +81 -0
  19. package/relay/logfile.ts +138 -0
  20. package/relay/main.ts +642 -0
  21. package/relay/policy.ts +241 -0
  22. package/relay/relay.ts +266 -0
  23. package/relay/token.ts +116 -0
  24. package/relay/v1.ts +107 -0
  25. package/relay/wisp.ts +615 -0
  26. package/src/base.js +325 -0
  27. package/src/boot.js +2037 -0
  28. package/src/calib-worker.js +40 -0
  29. package/src/fs/9p/codec.js +303 -0
  30. package/src/fs/9p/server.js +521 -0
  31. package/src/fs/errno.js +43 -0
  32. package/src/fs/graph.js +380 -0
  33. package/src/fs/overlay.js +1018 -0
  34. package/src/fs/pack-write.js +165 -0
  35. package/src/fs/pack.js +427 -0
  36. package/src/fs/seed.js +22 -0
  37. package/src/fs/svc-snapshot.js +65 -0
  38. package/src/fs/upper-pack.js +112 -0
  39. package/src/host/base-cache.js +195 -0
  40. package/src/host/base-file.js +328 -0
  41. package/src/host/base-io.js +93 -0
  42. package/src/host/boot-events.js +209 -0
  43. package/src/host/chan-service.js +300 -0
  44. package/src/host/chan-services.js +689 -0
  45. package/src/host/chan.js +655 -0
  46. package/src/host/ckpt-file.js +395 -0
  47. package/src/host/ckpt-io.js +207 -0
  48. package/src/host/ckpt-point.js +160 -0
  49. package/src/host/ckpt-save.js +64 -0
  50. package/src/host/ckpt.js +434 -0
  51. package/src/host/clock.js +111 -0
  52. package/src/host/ctl.js +778 -0
  53. package/src/host/errno.js +18 -0
  54. package/src/host/fs.js +167 -0
  55. package/src/host/gcnudge.js +83 -0
  56. package/src/host/i64shim.js +81 -0
  57. package/src/host/jit-cache.js +341 -0
  58. package/src/host/jit-hub-core.js +208 -0
  59. package/src/host/jit-hub.js +93 -0
  60. package/src/host/jit-snap.js +226 -0
  61. package/src/host/jit.js +1903 -0
  62. package/src/host/layout.js +100 -0
  63. package/src/host/monitor.js +110 -0
  64. package/src/host/p9mount.js +27 -0
  65. package/src/host/ram-mem.js +125 -0
  66. package/src/host/stdio.js +143 -0
  67. package/src/host/threads.js +109 -0
  68. package/src/host/wasi.js +335 -0
  69. package/src/host/wmd-imports.js +71 -0
  70. package/src/host/yamux.js +387 -0
  71. package/src/image-files.js +154 -0
  72. package/src/net/dhcp-dns.js +126 -0
  73. package/src/net/net-worker.js +172 -0
  74. package/src/net/packet.js +227 -0
  75. package/src/net/peer.js +230 -0
  76. package/src/net/proto.js +189 -0
  77. package/src/net/relay-client.js +226 -0
  78. package/src/net/stack.js +867 -0
  79. package/src/net/switch.js +160 -0
  80. package/src/net/wisp-client.js +570 -0
  81. package/src/net/wisp-transport.js +196 -0
  82. package/src/net/wisp.js +511 -0
  83. package/src/registry/client.js +176 -0
  84. package/src/registry/converter.js +68 -0
  85. package/src/registry/hub.js +221 -0
  86. package/src/registry/oci.js +76 -0
  87. package/src/registry/pack-worker.js +70 -0
  88. package/src/registry/pull.js +258 -0
  89. package/src/registry/ref.js +110 -0
  90. package/src/registry/wasi-mem.js +355 -0
  91. package/src/worker.js +315 -0
  92. package/wmdpack.wasm +0 -0
package/README.md CHANGED
@@ -1,3 +1,2602 @@
1
- # Temporary Holding Version
1
+ # wmd-web
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ The browser host for wmd's guest `vm.wasm`. It is the same `vm.wasm` that `wmd-run`
4
+ runs (sha1 `5ac0e005134bfd198f01a92721324fe37bb93758`), running in Web Workers on one
5
+ shared `WebAssembly.Memory`, with a JavaScript WASI preview1 subset, `wasi.thread-spawn`
6
+ over a Worker pool, and the `wmd.*` imports stubbed for interpreter mode.
7
+
8
+ Status: **P-B1 passes** in Chromium, Firefox and WebKit (Playwright 1.63). The arm64
9
+ kernel and the proof initramfs boot in interpreter mode, and every boot-gate regex of
10
+ `guest/vm/test/expected/boot-arm64.txt` matches the page console, with exit 0. The
11
+ amd64 kernel also passes in Chromium.
12
+
13
+ **P-B2 passes** in all three: the guest's own JIT (`--mode jit`) runs in the browser
14
+ through the JIT door (`src/host/jit.js`). The same proof boot reaches `PROOF-DONE` 6.7x
15
+ (Chromium), 8.1x (Firefox) and 7.3x (WebKit) sooner than P-B1, and the total run is within 3–11% of
16
+ native `wmd-run --mode jit` wall time. See [P-B2: the JIT door](#p-b2-the-jit-door).
17
+
18
+ **P-B3 passes** in all three: `wmd.chan_*` run on shared-memory rings with a service
19
+ Worker. A 9P round trip as a guest process sees it (`fstat` on a 9P mount) takes 89–187 µs
20
+ (native `wmd-run` with wmd's Go 9P server over TCP: 104–207 µs), and the guest never waited
21
+ for the host while writing 50 MiB into the control channel. See [P-B3](#p-b3-channels-on-shared-memory-rings).
22
+
23
+ **P-B4, first integration:** alpine:3.20 runs from its pack over the JS 9P server in all
24
+ three browsers, both from the proof initramfs (chroot) and **as a container through wmd's
25
+ Go init and crun**, as the engine boots it: `ls -R /` finishes, `apk info` lists its 14
26
+ packages, and the init writes `bundle/exitcode` back over 9P. See [P-B4](#p-b4-alpine-from-packs).
27
+
28
+ **P-B11, the wmn agent in the browser's guest (core):** the container kernel and Go init boot
29
+ with the engine's own command line (`wmd.control=vport`); the page drives the init over the
30
+ control channel with a JS yamux client (ROADMAP 2.2b); python:3.12-slim and node:22-slim run
31
+ from one pack per layer, with the agent as PID 1 (`/.wmn/agent init`) and `/.wmn/agent serve`
32
+ exec'd beside it; `hello_reply` and the 13 P-K2 checks pass in Chromium, Firefox and WebKit,
33
+ with failures on a loaded host (Firefox, two runs). See [P-B11](#p-b11-the-wmn-agent-in-the-browsers-guest).
34
+
35
+ **P-B10 passes** in all three: the guest has a network. Its virtio-net frames go to a TCP/IP
36
+ stack in a network Worker, which carries its TCP connections and DNS over one WebSocket to a
37
+ small relay (`relay/relay.ts`). `apk add curl` in alpine:3.20 and `pip install requests` in
38
+ python:3.12-slim complete, and curl and requests then fetch over HTTPS. The relay refuses
39
+ private, loopback, link-local, metadata, reserved and its own host's addresses after its own
40
+ DNS resolution, needs a per-session token, and caps connections and bandwidth. See
41
+ [P-B10](#p-b10-the-guests-network-through-a-relay).
42
+
43
+ **P-B7, packs from a static host and the registry proxy:** packs are content addressed
44
+ (`/blobs/sha256/<hex>`, immutable) with one small index per image@platform; a service worker
45
+ caches them after a hash check, so a second visit boots with the server gone; a Bun proxy
46
+ (`registry/`) pulls any public Docker Hub image (anonymous token, index -> linux/arm64,
47
+ sha256 checked while streaming) and converts its layers with wmd's own `union.PackLayer`.
48
+ See [P-B7](#p-b7-packs-from-a-static-host-and-the-registry-proxy).
49
+
50
+ The interface this host implements is in [docs/host-interface.md](docs/host-interface.md).
51
+
52
+ ## Layout
53
+
54
+ | File | What it does |
55
+ |---|---|
56
+ | `src/boot.js` | Page API `bootWmd({vmUrl, files, args, env, ...})`. Downloads the files, compiles vm.wasm, creates the shared memory, the control block and the Worker pool, then runs `_start` in a Worker. Reports stdout/stderr and timings. |
57
+ | `src/worker.js` | One Worker = one wasm thread. Role `main` runs `_start` (tid 1). Role `pool` parks on a slot and, for each spawn, instantiates vm.wasm again (same memory, a new 155-entry table) and calls `wasi_thread_start(tid, arg)`. |
58
+ | `src/host/wasi.js` | The 18 `wasi_snapshot_preview1` imports, modelled on `runner/src/wasi.rs`. |
59
+ | `src/host/fs.js` | A read-only, in-memory preopen (`/g`) over SharedArrayBuffers. Its fd table is process-wide: it lives in the control block, so any thread can read a file another thread opened. |
60
+ | `src/host/threads.js` | `thread-spawn`: claims an idle pool slot with a compare-exchange, writes tid and arg, notifies, and returns the tid synchronously. |
61
+ | `src/host/clock.js` | One clock for all Workers: `timeOrigin + now()` minus a shared start time, with a shared high-water mark so CLOCK_MONOTONIC never goes back across Workers. CLOCK_REALTIME is the shared start time plus CLOCK_MONOTONIC, so it never goes back either. |
62
+ | `src/host/wmd-imports.js` | The `wmd.*` imports. `ram_map`→1, `blk_rw`→-38 (no swap disk), `win_init`→1, other `win_*`→0, `snapshot`→-1, `chan_open`→-111 without channel services; without a JIT door `jit_*`→-1 or no-op. `createWmdImports({jit, chan, counters})`: the worker passes the JIT door as `jit` and, when the boot has channel services, the P-B3 channel imports as `chan`. Calls to the stubs are counted in the control block (`counters`), not posted. |
63
+ | `src/host/chan.js` | P-B3: the chan block (one SharedArrayBuffer: doorbell, port registry, a duplex ring pair per channel, the console pipe to the page, counters and the 9P round-trip histogram), the ring primitives, `markServiceDead` / `markGuestGone`, and the five guest imports (`createChanImports`). |
64
+ | `src/host/chan-services.js` | P-B3: `ChanHost`, the host side of every channel, synchronous (`step()`): the 9P server per connection over one Overlay per port, the console (to the page's pipe, and page input), and the control and net stubs (P-B10: the net channel goes to the network Worker when a relay is configured). |
65
+ | `src/host/chan-service.js` | P-B3: the service Worker around `ChanHost`. Waits on the doorbell with `Atomics.waitAsync` (spinning 200 µs after work first), holds a Web Lock for its life so the page learns when it ends, answers `stats`/`collect`, and has test hooks (`die`, `crash`, `stall`). |
66
+ | `src/host/jit-hub.js` | The JIT module hub: one Worker that forwards each compiled `WebAssembly.Module` to every vCPU Worker, so a module is compiled once per process (finding F1). |
67
+ | `src/host/jit.js` | P-B2, the JIT door: `jit_load`, `jit_compile`, `jit_sync`, `jit_flush`, `jit_known`, `jit_note`. One global slot list in SharedArrayBuffers, per-thread tables, counters and per-module timings. Without a module cache `jit_known` is -1 and `jit_note` a no-op; with one (W77) they read and feed `jit-cache.js`. |
68
+ | `src/host/jit-cache.js` | W77, the JIT module cache across visits: `bootWmd({jit: {cache}})` with a `{get, put}` byte store keyed by the vm.wasm's sha256. The page loads one index (block key → module, function, span) into a SharedArrayBuffer table at boot; `jit_known` finds a key there, gets the module's bytes from the page once (hash and length checked), and returns the function's slot; `jit_note` names the modules to put. See D-486. |
69
+ | `src/host/stdio.js` | stdout/stderr rings in a SharedArrayBuffer. Workers write into them under a spin lock, and the page drains them every 16 ms. |
70
+ | `src/host/layout.js` | Cell layout of the shared control block: tid counter, exit flag and code, pool slots, fd table, clock base, stub host-call counters. |
71
+ | `src/host/monitor.js` | Page-side monitors: the kernel-panic scanner, the progress watchdog and the throttle meter. |
72
+ | `src/calib-worker.js` | The throttle detector's Worker: times a fixed integer loop and a 10 ms `Atomics.wait` every 2 s. |
73
+ | `page/` | A small test page. `?case=arm64\|amd64\|hello\|wfi\|prog\|loop\|flush\|smpflush\|smc&mode=interp\|jit&auto=1` boots on load (other options: `arg=`, `env=`, `prog=`, `jitc=`, `brk=`, `share=`, `door=0`, `watchdog=`; see the head of `page/main.js`). State is on `window.wmd`, and the JIT counters are in a table on the page. `page/smc-initrd.js` builds the self-modifying-code initramfs. `page/jit-stress.html` is the multi-Worker JIT door stress test, and `page/wasm-mini.js` encodes its block-shaped test modules. `page/idb-bytes.js` is the IndexedDB byte store behind `?jitcache=1` (`=fresh` empties it first), on this page and the wmn test page. |
74
+ | `src/fs/` | P-B4, the 9P2000.L file server (no browser wiring yet): the pack reader (`pack.js`), the overlay of pack lowers under an in-memory upper (`overlay.js`), the codec and server (`9p/codec.js`, `9p/server.js`) and Linux errno values (`errno.js`). See [docs/9p.md](docs/9p.md). |
75
+ | `page/chan.html`, `page/chan.js`, `page/b3-initrd.js` | P-B3/P-B4 test page: `?case=p9lat\|console\|alpine\|container\|p9load&auto=1` (options at the head of `chan.js`). `b3-initrd.js` appends a newc archive to the proof initramfs with `/p9stat` (a hand-assembled arm64 ELF that times N `fstat` calls) and the case's `/proof.sh`. |
76
+ | `test/serve.ts` | A Bun static server: COOP `same-origin`, COEP `require-corp`, CORP, and a CSP with `'wasm-unsafe-eval'`. It serves read-only from fixed roots. |
77
+ | `test/unit/` | `bun test` for the WASI layer, the spawn pool, the JIT door (`jit.test.js`, `jit-share.test.js`, `b2-*.test.js`, which include real Workers flushing under load), the stdio rings, the P-B1 monitors, counters, clocks and gate (`monitor.test.js`, `gate.test.js`) and the 9P server (`9p-*.test.js`; fixtures made on first run by `test/fixtures-gen`, see docs/9p.md). |
78
+ | `test/e2e/chan.spec.js`, `test/unit/chan*.js`, `test/unit/clock-strict.test.js` | P-B3/P-B4: browser boots per browser (9P round trip, console and control throughput, alpine, the container init, the adversarial cases) and the ring/import/service unit tests with real Workers as guest threads. |
79
+ | `src/host/yamux.js`, `src/host/ctl.js` | P-B11 (ROADMAP 2.2b): the hashicorp/yamux client (run by the channel service, `ChanHost` `ControlSession`, so pings are answered off the page) and wmd's control protocol on the page (`ControlClient`: run, startExec, attach, kill, procs; frames; the engine's OCI bundle, `containerBundle`). `vm.control` when a service is `{kind: 'control', yamux: true}`. |
80
+ | `page/wmn.html`, `page/wmn.js`, `page/k2.js` | P-B11: an image's packs + the agent, the engine's command line, run + exec over the control session, the P-K2 checks (ported from the wmn repo's `spikes/P-K2/k2.ts`). `?image=python\|node&auto=1` (options at the head of `wmn.js`). |
81
+ | `src/net/` | P-B10: the guest's network. `stack.js` (ARP, DHCP, DNS, ICMP, TCP for the guest's frames), `packet.js` (framing and checksums), `dhcp-dns.js`, `relay-client.js` (the relay WebSocket), `proto.js` (the relay protocol, shared with the relay), `net-worker.js` (the network Worker). |
82
+ | `relay/` | P-B10: the relay server (`relay.ts`, Bun), its address policy (`policy.ts`) and session tokens (`token.ts`). |
83
+ | `page/net.html`, `page/net.js` | P-B10: an image with a network (`?image=alpine\|python&auto=1`); `window.net.sh(cmd)` runs a command in the container. |
84
+ | `test/e2e/` | Playwright: the boots (`boot.spec.js`), the P-B1 adversarial checks (`adversarial.spec.js`), the tests of the P-B1 adversarial fixes (`fixes.spec.js`, `throttle.spec.js`) and the P-B1 review suites (`review-*.spec.js`), the wasm feature probes (`features.spec.js`), the P-B2 JIT boots (`jit.spec.js`), the JIT door stress in each browser (`jit-stress.spec.js`) an opt-in benchmark of host variants (`jit-bench.spec.js`, `WMD_WEB_BENCH=1`) and the JIT module cache across two visits (`w77-jit-cache.spec.js`; its python test needs `WMD_WEB_IMAGE_FIXTURES`). |
85
+
86
+ ## Design notes
87
+
88
+ - **Workers only.** `memory.atomic.wait32` traps on the main thread, so the page thread
89
+ never runs wasm.
90
+ - **Thread spawn is synchronous.** A Worker blocked inside wasm cannot wait for a new
91
+ Worker to load. The page therefore starts 4 pool Workers up front, each holding the
92
+ compiled `Module`, and keeps 2 spare: after each spawn the spawner posts `need` and the
93
+ page tops up the pool asynchronously.
94
+ - If no Worker is idle, the spawner asks for more and waits up to 10 s on the pool
95
+ generation cell.
96
+ - If none appears in that time, it returns -11 (EAGAIN) and the page prints `thread
97
+ pool exhausted` or `thread-spawn found no idle Worker`.
98
+ - Up to 128 slots are allowed.
99
+ - A P-B1 boot spawns 2 threads (tids 2 and 3), one of which is the timer thread.
100
+ Which code spawns the other was not traced.
101
+ - **Exit.** `proc_exit` sets the shared exit flag (the first code wins), tells the page,
102
+ and throws to unwind its own Worker. The page then terminates every Worker. A trap on
103
+ any thread ends the machine with code 4, as in the runner.
104
+ - **Memory.** One `WebAssembly.Memory({initial: 6144, maximum, shared: true})`.
105
+ `maximum` is 6144 + RAM pages + 1 GiB of slack (10,240 + 16,384 pages for
106
+ `--ram 256`). Views over it are rebuilt whenever `memory.buffer` changes after a grow.
107
+ - **Output.** stdout and stderr go into a ring per fd in a SharedArrayBuffer
108
+ (`src/host/stdio.js`), 32 MiB each for a boot (`bootWmd({stdioRingBytes})`, a power of
109
+ two). The page drains the rings every 16 ms and decodes them with a streaming
110
+ `TextDecoder`. P-B1 posted one message per `fd_write`; P-B2 replaced that, because a
111
+ guest that writes megabytes a second flooded the page's event loop (see P-B2 problem
112
+ 3). If the page falls a whole ring behind, it skips ahead, counts the skipped bytes
113
+ (`timings.outputDropped`, `timings.outputDropMarkers`) and puts a line
114
+ `[wmd-web: N bytes of stdout were dropped]` into the delivered text where the bytes
115
+ are missing. The boot gate fails on that line (finding F4 below).
116
+ - **Endings.** `vm.exited` resolves `{code, reason, timings}`. Guest exit codes are
117
+ >= 0. The host's own endings are negative: -1 `stopped` (`vm.stop()`) or `aborted`
118
+ (the AbortSignal), -2 `no progress: ...` (the watchdog), -3 `kernel panic: <line>`
119
+ (the guest's own exit code is then in `timings.guestExitCode`). An AbortSignal that
120
+ fires before the Workers start makes `bootWmd` reject with an `AbortError` instead;
121
+ the signal is checked after every await, so this is immediate even while vm.wasm
122
+ compiles.
123
+ - **Kernel panic.** The host scans the console for `Kernel panic - not syncing`. The
124
+ arm64 kernel has `CONFIG_PANIC_TIMEOUT=-1`, so it reboots at once and the guest
125
+ reports exit 0 (PSCI SYSTEM_RESET); the host reports -3. A kernel that hangs after a
126
+ panic instead is stopped 5 s after the panic line.
127
+ - **Progress watchdog.** After the main Worker starts `_start`, the boot ends with -2
128
+ and the line `guest made no progress` when nothing is written to stdout or stderr
129
+ and no thread is spawned for `watchdogMs`. Default 60 s for an arm64 `--machine`,
130
+ 300 s for x86 (the amd64 decompressor is silent for about 75 s in Chromium), off for
131
+ user-mode programs. The limit is multiplied by the throttle ratio while the host
132
+ reports throttling, and a page whose own thread was stalled restarts the count.
133
+ - **Throttle detector.** A small Worker (`calib-worker.js`) times a fixed integer loop
134
+ every 2 s (about 1% of one core). The page compares the median of the last three
135
+ runs with the fastest run seen (this boot, or the `throttle.baselineMs` the caller
136
+ passes; the test page calibrates once at load and keeps the best in `localStorage`).
137
+ At 2x or more it emits `{t:'throttle', throttled:true, ratio, ...}`, writes
138
+ `[wmd-web] the guest is running about Nx slower ...` to stderr, and
139
+ `vm.stats().throttle` / `timings.throttle` carry `{ratio, maxRatio, throttledSamples}`.
140
+ See "Background throttling" below for what it catches and why.
141
+ - **Stub host calls.** Calls to the stubbed `snapshot` and `chan_open`, and to the
142
+ stubbed `jit_*` when a boot runs without the JIT door (`bootWmd({jit: false})`, test
143
+ page `?door=0`), are counted in shared cells of the control block. `vm.stats().hostCalls` reads them on
144
+ demand; the page gets one `{t:'host-calls', counts}` event at most every 250 ms, and
145
+ only when a count changed. The test page keeps the latest counts, not a list, and caps
146
+ its diagnostic event list at 1000 entries.
147
+ - **Releasing a finished boot.** When a boot ends (exit, stop, abort, trap, watchdog,
148
+ panic) the host terminates every Worker (the calibration Worker too), clears their
149
+ handlers, its timers and the abort listener, terminates the JIT hub Worker, and drops
150
+ the compiled module and the file buffers. Only `vm.memory` stays, for post-mortem reads, until `vm.dispose()`;
151
+ the test page calls `dispose()` and keeps no handle.
152
+ stdout is line-buffered by the guest's libc. The test page keeps about 8 MB of each
153
+ stream in `window.wmd` (the start and the most recent output, with a
154
+ `[wmd-web: N bytes of stdout were dropped]` line between, so a gate run on it fails
155
+ too) and shows the last 128 KB. P-B1's merged branch had the same rings at 4 MiB
156
+ (cherry-picked as `17e9ce50`); the merge keeps P-B2's 32 MiB rings and drop markers.
157
+
158
+ ## Running the tests
159
+
160
+ ```sh
161
+ cd web
162
+ bun install # @playwright/test 1.63.0 (browsers already in ~/Library/Caches/ms-playwright)
163
+ bun test test/unit # WASI layer, spawn pool, JIT door, stdio rings, monitors, counters, clocks, gate, 9P; no browser
164
+
165
+ bunx playwright test --project=chromium # boots + adversarial + features
166
+ WMD_WEB_AMD64=1 bunx playwright test boot --project=chromium -g amd64 # ~6.6 min
167
+ bunx playwright test boot --project=firefox
168
+ bunx playwright test boot --project=webkit
169
+ bunx playwright test features # all three browsers, no guest boot
170
+ bunx playwright test jit-stress # JIT door stress, all three browsers, no guest boot
171
+
172
+ # P-B2 (each under the lock, one browser per run, about 3 min each):
173
+ bunx playwright test jit.spec boot.spec jit-stress --project=chromium
174
+ bunx playwright test jit.spec boot.spec jit-stress --project=firefox
175
+ bunx playwright test jit.spec boot.spec jit-stress --project=webkit
176
+ WMD_WEB_BENCH=1 bunx playwright test jit-bench --project=chromium # host variants, one boot each
177
+
178
+ # P-B2 adversarial fixes (each under the lock, per browser): F1 multi-vCPU --jit-batch 1
179
+ # boots and F4 drop-marker boots; B2F_SHARE=0 reproduces F1 in Firefox
180
+ bunx playwright test b2-fixes --project=firefox
181
+ WMD_WEB_B2ADV=1 bunx playwright test b2-adversarial --project=firefox # the review's own suite
182
+
183
+ # P-B3/P-B4 adversarial fixes (each under the lock, per browser; WMD_WEB_B3ADV=1 opts in):
184
+ WMD_WEB_B3ADV=1 bunx playwright test b3-fixes --project=webkit -g "default|record|F2/F3" # ~3 min
185
+ WMD_WEB_B3ADV=1 bunx playwright test b3-fixes --project=webkit -g "measure cache" # F1 numbers, ~8-12 min
186
+ WMD_WEB_B3ADV=1 bunx playwright test b3-adversarial --project=webkit -g "rings" # the review's own suite, in parts
187
+
188
+ # P-B1 adversarial fixes and the P-B1 review's suites (each under the lock, per browser):
189
+ bunx playwright test fixes --project=firefox # (under the lock) the adversarial-fix tests, any browser
190
+ REVIEW_SEQ_BOOTS=5 bunx playwright test review-lifecycle -g sequential --project=webkit # (under the lock) memory over 5 boots
191
+
192
+ # Real background throttling: opens a headed Chromium window (under the lock).
193
+ REVIEW_HEADED=minimized bunx playwright test throttle --project=chromium # must report throttling
194
+ REVIEW_HEADED=visible bunx playwright test throttle --project=chromium # must not; full strict gate
195
+ REVIEW_HEADED=minimized bunx playwright test review-lifecycle -g headed --project=chromium # the adversary's version
196
+ ```
197
+
198
+ Environment variables:
199
+
200
+ - `WMD_REPO`: the wmd checkout the boot files are served from. Default
201
+ `~/keg-hub/external/apps/wmd`. The files used are `guest/linux/dist/{arm64,amd64}/*`
202
+ and `guest/vm/build/progs/arm64/hello`.
203
+ - `WMD_VM_WASM`: the vm.wasm to serve. Default `~/.wmd/bin/guest/arm64/vm.wasm`.
204
+ - `PORT`: the server port. Default 8791 for Playwright, 8787 for `bun test/serve.ts`.
205
+ The P-B2 runs used `PORT=8793`, so they would not collide with another worktree's
206
+ server.
207
+ - `WMD_WEB_BOOT_TIMEOUT_MS`: the per-boot bound. Default 420000 (300000 in `chan.spec.js`).
208
+ - `WMD_GUEST_DIR`: where `/files/guest/arm64/` (the container kernel and the Go init's
209
+ initramfs, P-B4) is served from. Default `~/.wmd/bin/guest/arm64`, read only.
210
+ - `WMD_WEB_9P_FIXTURES`: the packs served at `/fixtures/9p/` (default `test/fixtures/9p`,
211
+ made by the unit tests' `ensureFixtures()`).
212
+ - `chan.spec.js`: `WMD_WEB_CONSOLE_BYTES` (default 50 MiB), `WMD_WEB_UART=BYTES` (runs the
213
+ UART comparison), `WMD_WEB_HVC0_TIMEOUT_MS` (120000), `WMD_WEB_RUNCLOG=1` (crun's debug log
214
+ into `bundle/runc.log`), `WMD_WEB_P9LAT_QUERY` (extra page options, e.g. `&spin=0`).
215
+ The P-B3/P-B4 runs used `PORT=8796`.
216
+
217
+ The logs and timings of each boot are written to
218
+ `test-results/<browser>-<case>.{stdout.log,stderr.log,timings.json}`.
219
+
220
+ ## Numbers (2026-10-04, Apple Silicon, one run each, interpreter, 1 vCPU, --ram 256)
221
+
222
+ All times are in ms. Browser times start when `bootWmd` is called on the page, so they
223
+ include the download. Native times start at process start. The canary read
224
+ 617–649 ms around these runs, so the numbers are reliable.
225
+
226
+ | arm64 proof boot | native `wmd-run` | Chromium 153 (headless shell) | Firefox 155 | WebKit 26.6 |
227
+ |---|---|---|---|---|
228
+ | download (vm.wasm + Image + initramfs, 43.2 MB, localhost) | n/a | 62 | 165 | 61 |
229
+ | `WebAssembly.compile(vm.wasm)` | n/a (Cranelift, cached) | 9 | 25 | 11 |
230
+ | main Worker instantiated | n/a | 107 | 230 | 89 |
231
+ | first console byte | 222 (`Linux version`) | 345 | 654 | 334 |
232
+ | `wmd-init: up` | 19,977 | 17,342 | 21,653 | 19,169 |
233
+ | `PROOF-DONE` | 51,554 | 44,042 | 57,534 | 49,166 |
234
+ | `reboot: Power down` | 51,771 | 44,218 | 57,752 | 49,374 |
235
+ | total (exit) | 52,024 (wall 51.7 s) | 44,240 | 57,772 | 49,431 |
236
+
237
+ - **Native reference.** It was run once:
238
+ `wmd-run --wmd-stats --dir guest/linux/dist guest/vm/build/vm.wasm -- --machine --arch a64 --board arm64 --mode interp --cpus 1 --ram 256 --kernel .../arm64/Image --initrd .../arm64/initramfs.cpio.gz --append "console=ttyAMA0 earlycon=pl011,0x9000000 rdinit=/init loglevel=8 wmd.proof wmd.exit" --stats`.
239
+ - **Compile time.** The compile times are what `WebAssembly.compile` returns. That is
240
+ the engines' baseline tier (Liftoff, or equivalent); optimizing tier-up happens later,
241
+ in the background.
242
+ - **Browsers vs native.** In Chromium and WebKit the interpreter boot is *faster* than
243
+ native wasmtime, by 15% and 5%. Firefox is 11% slower. This is one run each, and the
244
+ cause was not investigated.
245
+
246
+ | amd64 proof boot (Chromium) | ms |
247
+ |---|---|
248
+ | download (vm.wasm + bzImage + initramfs, 11.8 MB) | 37 |
249
+ | compile | 7 |
250
+ | first console byte (decompressor) | 403 |
251
+ | `Linux version` | 74,841 |
252
+ | `wmd-init: up` | 238,001 |
253
+ | total (exit 0, all boot-gate patterns met) | 392,857 |
254
+
255
+ - **amd64 marks.** The `PROOF-DONE` and power-down marks were not captured on this run:
256
+ the x86 console sends CRLF and the mark regexes were anchored. The boot gate itself
257
+ strips CR and passed. Mark matching now strips CR (commit `6e4735e5`).
258
+ - **No native amd64 reference.** The brief allowed one native run, and it was spent on
259
+ arm64.
260
+
261
+ After the adversarial fixes (same day, every boot under the lock, canary 600-650 ms):
262
+ `PROOF-DONE` at 40.4-42.5 s in Chromium, 52.5-56.4 s in Firefox and 45.9-47.3 s in
263
+ WebKit, TIMER-DELTA 1.1 everywhere. The review measured 40.3-45.8 s (Chromium), 55.6-58.6 s
264
+ (Firefox) and 48.0-50.5 s (WebKit) before the fixes, so the throttle detector's Worker
265
+ and the 250 ms tick cost no measurable time.
266
+
267
+ ## Browser wasm feature support (Playwright 1.63 browsers, `test/e2e/features.spec.js`)
268
+
269
+ Each feature was checked by compiling, instantiating and running a tiny module in a
270
+ cross-origin isolated page.
271
+
272
+ | Feature | Chromium 153 | Firefox 155 | WebKit 26.6 |
273
+ |---|---|---|---|
274
+ | cross-origin isolated, SharedArrayBuffer | yes | yes | yes |
275
+ | shared memory 384 MiB initial / 1 GiB max | yes | yes | yes |
276
+ | threads (shared memory + atomics) | yes | yes | yes |
277
+ | SIMD128 | yes | yes | yes |
278
+ | bulk memory | yes | yes | yes |
279
+ | multi-memory | yes | yes | yes |
280
+ | two *shared* imported memories + atomic load from memory 1 (the JIT block-module shape) | yes | yes | yes |
281
+ | tail calls (`return_call`) | yes | yes | yes |
282
+ | memory64 | yes | yes | yes |
283
+
284
+ These are Playwright's engine builds. Shipping Safari and Firefox releases may differ.
285
+ P-B5 has to re-check them on real browsers.
286
+
287
+ ## Firefox and WebKit
288
+
289
+ Both pass steps 1–3, the same tests as Chromium. Each prints `hello from aarch64` and
290
+ `board init ok`, and the arm64 proof boot passes the full gate with exit 0. The timings
291
+ are in the table above. Nothing needed per-browser code.
292
+
293
+ ## P-B3: channels on shared-memory rings
294
+
295
+ `wmd.chan_*` as `runner/src/chan.rs` has them, for Workers (`src/host/chan.js`):
296
+
297
+ - **One chan block per machine.** A SharedArrayBuffer holds a header (doorbell, service
298
+ state, guest-gone flag, the ports the services listen on), 8 channel slots, and per
299
+ slot two byte rings: guest→host and host→guest, 1 MiB each by default
300
+ (`bootWmd({channels: {ringBytes}})`, any power of two from 64 bytes), plus an 8 MiB pipe
301
+ for the console's bytes to the page: about 25 MiB of SharedArrayBuffer per boot, touched
302
+ only as it is used. Positions are 32-bit counters that wrap. Boots without `channels`
303
+ get the old stubs and no service Worker.
304
+ - **Guest side: blocking imports.** `chan_send` copies straight from guest memory into
305
+ the ring, waiting (`Atomics.wait` on the ring's "space" sequence cell) while it is full;
306
+ it holds the ring's producer lock for the whole message, so two vCPUs' 9P requests never
307
+ interleave, whatever their size. `chan_recv` returns what is there (a partial read is
308
+ normal), waits up to the timeout (rounded up to ms, as chan.rs) on the ring's "data"
309
+ sequence cell, returns 0 on timeout and -1 at EOF. Every wait is at most 1 s at a time,
310
+ and re-checks the machine's exit flag, the service state and the channel's own state, so
311
+ nothing in the guest can hang on a peer that is gone.
312
+ - **Return values as chan.rs.** `chan_open`: -22 for a port outside 1..65535, -111 when no
313
+ service listens on the port (or the service is dead), else the lowest free handle. The
314
+ port decides the service; `kind` is only recorded. -14 bad range, -9 bad handle, -32 when
315
+ the peer is gone or after our own `chan_shutdown`. `chan_shutdown` is a half-close: the
316
+ host reads to the end, sees EOF, and closes its side once its replies are out.
317
+ `chan_close` wakes any thread blocked on the handle (-1 / -32) and frees the slot only
318
+ after the service has dropped the session, so a reply in flight can never land in a new
319
+ channel that reused the handle.
320
+ - **Host side: a service Worker** (`src/host/chan-service.js`), never the page. It never
321
+ blocks: it waits on the doorbell cell with `Atomics.waitAsync` and spins on it for 200 µs
322
+ after any work (`channels.spinUs`), so its event loop stays free for the page's messages.
323
+ Services by the kind registered for the port: **9p** (`src/fs/9p/server.js`, a
324
+ `NinePServer` per connection over one Overlay per port: packs as lowers, an in-memory
325
+ upper), **console** (to the page through a pipe ring the page drains every 16 ms; no loss:
326
+ when the page falls behind, the service stops reading and the guest blocks; page input
327
+ through `vm.consoleWrite`, which takes at most 4 MiB the guest has not read and says how
328
+ much it took), **control** (a stub: reads, counts and CRC-32s), **net** (a stub: counts
329
+ the 4-byte-length frames and drops them; P-B10 adds the relay).
330
+ - **Backpressure.** The 9P service answers at most `OUT_LIMIT` (4 MiB) of replies the guest
331
+ has not read; after that it stops reading requests, so tiny requests cannot make it queue
332
+ unbounded replies, and the guest's sends block on a full ring.
333
+ - **A dead service.** The service holds a Web Lock for its whole life; the page asks for the
334
+ same lock and is granted it when the Worker ends, however it ends (terminated, closed,
335
+ crashed, killed by the browser). The page then calls `markServiceDead`: every guest
336
+ waiter wakes, sends fail (-32), receives drain what is left and then get -1, `chan_open`
337
+ refuses. An exception in the service's loop marks it dead from inside too. The guest's 9P
338
+ pump powers the machine off on -1 (`chan_pump.c` `host_gone`) with exit 0; `bootWmd`
339
+ reports that as **-4** `channel service died: ...` (the guest's code in
340
+ `timings.guestExitCode`), as a panic is -3.
341
+ - **The guest ends.** `finish()` sets the guest-gone flag, which also wakes every guest
342
+ thread still blocked on a channel (-32 / -1 at once); the service drops every session
343
+ (counting any message it held half-read in `sessionsDropped`, never handing it to the
344
+ server) and stops; then the page terminates it with the other Workers.
345
+ - **Counters.** `vm.chanStats()` / `timings.chan`: per channel bytes each way and how often
346
+ and how long a guest send waited for room (`sendWaits`, `sendWaitMs`: the "host fell
347
+ behind" signal); the 9P round trip measured by the guest's threads (send of a T-message
348
+ to its R-message delivered by the pump: a histogram, p50/p90/p99) and the service's
349
+ per-request time; per-kind bytes, frames, EOFs, protocol errors.
350
+ - **Files out.** `channels.collect: [{port, path}]` reads files out of the 9P trees when the
351
+ machine ends (`timings.chanFiles`), e.g. `bundle/exitcode`.
352
+
353
+ ### Which console the guest uses
354
+
355
+ The proof boot's console is the PL011 UART, which vm.wasm writes to its stdout (the stdout
356
+ ring), and so is the container kernel's (`console=ttyAMA0`, `internal/engine/run.go`).
357
+ Containers' stdio does not go through either: the Go init runs its yamux session, which
358
+ carries the container's stdin/stdout/stderr, on virtio-console **port 1** (`wmd.control`,
359
+ `/dev/vportNp1`), i.e. the **control channel** (kind 3). Nothing in wmd uses port 0 (`hvc0`,
360
+ kind 4). The throughput test therefore writes 50 MiB into each of: the control channel (the
361
+ path container output will take), `hvc0` (the console channel), the UART (the stdout ring)
362
+ and `/dev/null` (what the guest's `yes | head` costs by itself).
363
+
364
+ ### Numbers (2026-10-05, Apple Silicon, Playwright 1.63, `--mode jit`, 1 vCPU, --ram 256)
365
+
366
+ One run each per browser (`test/e2e/chan.spec.js`, the final runs; canary 605–634 ms around
367
+ the Chromium and Firefox runs; the WebKit run started contended, canary 1,378 ms before and
368
+ 604 after, which shows in its first p9lat run). Native: `~/.wmd/bin/wmd-run` with wmd's Go 9P
369
+ server (`guest/linux/testhelper`) over loopback TCP, the same initramfs, canary 657–680 ms.
370
+
371
+ **9P round trip.** `/p9stat FILE 2000` on the 9P mount (no cache: every `fstat` is one
372
+ Tgetattr, which the service's counts confirm: 6,050+ Tgetattr), three runs after a warm-up of
373
+ 50, µs per call; on tmpfs for the syscall's own cost:
374
+
375
+ | | Chromium | Firefox | WebKit | native, Go server over TCP |
376
+ |---|---|---|---|---|
377
+ | `fstat` on the 9P mount (3 runs) | 137.4, 109.5, 101.4 | 153.1, 141.0, 138.4 | 187.2, 138.9, 88.9 | 113.2, 104.1, 108.0 (2nd boot; 1st: 207, 127, 133) |
378
+ | `fstat` on tmpfs | 14.9, 13.8, 14.1 | 19.3, 19.0, 18.6 | 21.0, 19.7, 17.5 | 7.5, 6.9, 7.3 |
379
+ | channel round trip, guest threads (send of the T-message → R-message in the pump), p50 / p90 / p99 | 30.5 / 40.5 / 100.5 | 20.5 / 40.5 / 139.5 | 20.5 / 100.5 / 559.5 | – |
380
+ | service time per request, p50 / p99 | 0.5 / 25.5 | 0.5 / 59.5 | 0.5 / 99.5 | – |
381
+
382
+ - **P-B3's bar (under 500 µs) is met** by a wide margin in all three, and the browser is
383
+ as fast as native wmd over TCP: almost all of the ~100 µs is the guest (syscall, v9fs,
384
+ virtio, the MMIO kick, the interrupt and the wake-up of the waiting process in the
385
+ emulated kernel), not the channel (20–40 µs) or the server (under 1 µs typical).
386
+ - The histograms are quantised by each browser's coarsened clock (`performance.now()`:
387
+ 5 µs in Chromium, 20 µs in Firefox and WebKit), hence the x0.5 values.
388
+ - **The spin is worth about 20 µs.** Chromium, two boots each: with `spin=0` the channel's
389
+ p50 was 40.5 µs and `fstat` 135–205 µs; with the default 200 µs spin, 19.5–20.5 µs and
390
+ 104–157 µs.
391
+
392
+ **Console throughput.** `yes | head -c 52428800` (50 MiB; since W53 piped through `cat`, see "hvc0 after W17"), timed by the guest
393
+ (`/proc/uptime`) and by the service (first to last byte read):
394
+
395
+ | | Chromium | Firefox | WebKit |
396
+ |---|---|---|---|
397
+ | into the **control channel** (vport, container stdio path) | 14.05 s, **3.7 MB/s** (service 13.95 s) | 15.86 s, **3.3 MB/s** | 10.69 s, **4.9 MB/s** |
398
+ | times the guest waited for the host | **0** | **0** | **0** |
399
+ | into `/dev/null` (the guest alone) | 11.83 s, 4.4 MB/s | 13.81 s, 3.8 MB/s | 12.33 s, 4.3 MB/s |
400
+ | through the UART (stdout ring) | 368.4 s, 0.14 MB/s, 0 bytes dropped (`WMD_WEB_UART`, Chromium only) | – | – |
401
+ | into `hvc0` (console channel) | **fails**: the guest's IRQ storm, see problem 1 | fails | fails |
402
+
403
+ The control channel keeps up: every byte arrived, in order, and no `chan_send` ever found
404
+ the ring full. In Chromium and Firefox the run took 15–19% longer than into `/dev/null`,
405
+ which is the guest's own virtio-console write path (copying into 32 KiB buffers, the MMIO
406
+ kick, the interrupt); in WebKit the control run was faster than its `/dev/null` run, so
407
+ one run each is noisy at that level. The UART is 26x slower: one
408
+ MMIO trap per byte, and the tty turns each `\n` into `\r\n`.
409
+
410
+ ### Tests
411
+
412
+ ```sh
413
+ bun test test/unit # 241 tests; chan.test.js: rings, imports, ChanHost, the service Worker;
414
+ # b3-fixes.test.js: the fixes of the P-B3/P-B4 review
415
+ # each under the boot lock, one browser at a time (about 4 min each):
416
+ PORT=8796 bunx playwright test chan.spec --project=chromium
417
+ WMD_WEB_UART=52428800 PORT=8796 bunx playwright test chan.spec -g UART --project=chromium # 6 min
418
+ ```
419
+
420
+ Unit (`test/unit/chan.test.js`, Workers play the guest's threads and the real service
421
+ Worker serves): a 64-byte ring carrying 1 MB of odd-sized chunks; full and empty; 32-bit
422
+ counter wrap; open/handle/range errors; timeouts (0 polls, 20 ms waits ≥ 19 ms, 1 ns rounds
423
+ up to 1 ms); partial reads; shutdown both ways; close freeing the slot only after the host
424
+ drops it; a dead service; 9P frames through a 64-byte ring giving the same replies as the
425
+ server itself; a bad frame size closing the channel; **4 producers x 200 messages up to 3.5x
426
+ the ring never interleaving**; **512 KiB Twrite/Tread over 4 KiB rings**; **rings full on
427
+ both sides** (the sender blocks after the host's 4 MiB of unread replies, nothing is lost,
428
+ everything arrives in order once the guest reads); **small requests for big replies** (the
429
+ host answers no more than OUT_LIMIT + a ring ahead); **the service terminated while a
430
+ sender is blocked mid-message** (-32) and a receiver blocked forever (-1), within 500 ms;
431
+ **an exception in the service** (-1, service dead); **the guest gone with half a frame sent**
432
+ (dropped, never given to the server); a sender blocked on a full ring returning when the
433
+ machine exits; console input in order through a 64-byte ring; net and control stubs.
434
+
435
+ Browser (`test/e2e/chan.spec.js`), each in Chromium, Firefox and WebKit:
436
+
437
+ | | Chromium | Firefox | WebKit |
438
+ |---|---|---|---|
439
+ | 9P round trip (P-B3 bar) | pass | pass | pass |
440
+ | control channel 50 MiB, every byte, no waits | pass | pass | pass |
441
+ | `hvc0` 50 MiB | **fail** (guest, problem 1) | **fail** | **fail** |
442
+ | `/dev/null` baseline | pass | pass | pass |
443
+ | alpine from packs (proof initramfs) | pass | pass | pass |
444
+ | alpine as a container (Go init, crun) | pass | pass | pass |
445
+ | service terminated by the page mid-traffic → -4, ended 262 / 267 / 258 ms after | pass | pass | pass |
446
+ | service `self.close()` mid-traffic (only the lock tells) → -4, 261 / 259 / 257 ms | pass | pass | pass |
447
+ | service throws mid-traffic → -4, 257 / 259 / 257 ms | pass | pass | pass |
448
+ | guest powers off with 9P reads in flight → exit 0, no service errors | pass | pass | pass |
449
+ | 4 KiB rings: alpine with 256 KiB messages | pass | pass | pass |
450
+
451
+ The P-B1 and P-B2 suites still pass after the clock change (problem 2): `boot.spec` and
452
+ `jit.spec` in Chromium (14 passed, 1 skipped), and the P-B1 and P-B2 proof boots in Firefox
453
+ and WebKit.
454
+
455
+ ### Problems found, and their causes
456
+
457
+ 1. **`hvc0` (the console channel) crawls and the kernel disables its interrupt.** With
458
+ `yes | head -c N > /dev/hvc0` the guest prints `irq 14: nobody cared` after a few seconds
459
+ and `Disabling IRQ #14`, and goes on at about 60 KB/s. The cause is in the guest, not the
460
+ host: native `wmd-run --console PORT` (a `nc -l` behind the port) does exactly the same
461
+ (4 MiB in 70 s, the same two lines). Linux's hvc path (`put_chars` → `__send_to_port`)
462
+ kicks the queue and then **polls** for the used buffer with interrupts off, while
463
+ vm.wasm's virtio-console (`guest/vm/src/dev/console.c` `console_notify`) raises an
464
+ interrupt for every transmit buffer. By the time interrupts are back on the buffer has
465
+ been taken, `vring_interrupt` finds nothing and returns IRQ_NONE, and after 100,000 of
466
+ those Linux disables the line. A raw tty (`stty raw -echo -opost`) does not help. A guest
467
+ fix would be to honour `VRING_AVAIL_F_NO_INTERRUPT` on the console's transmit queues
468
+ (`virtio_used_irq_wanted`, as `p9proxy.c` does since D-324) or not to interrupt for
469
+ transmit completions the driver polls; not built here. Nothing in wmd uses port 0, and
470
+ the container stdio path (port 1) is not affected: its driver path takes the interrupts.
471
+ **Update (wmd main, W17 2ec1cf9d):** vm.wasm now raises no interrupt for port 0's transmits;
472
+ the storm and the disabled IRQ are gone. hvc0 is still about 33-72 KB/s in the browsers and
473
+ 69 KB/s natively, so the 50 MiB test still times out: a vm.wasm/guest cost, not the host
474
+ (P-B11 section "hvc0 after W17").
475
+ 2. **The container's `crun` hung: no entropy in the browser.** The first container boot
476
+ stopped after `wmd-init: writing runc bundle`, the vCPU idle in WFI, and no `random: crng
477
+ init done` in the log. Linux's jitter entropy (`try_to_generate_entropy`, run when
478
+ `getrandom()` blocks) reads the cycle counter 8,192 times and gives up when too few reads
479
+ differ. The guest's counter is CLOCK_MONOTONIC / 16 ns, and browsers coarsen
480
+ `performance.now()` (5–20 µs), so consecutive reads were equal; natively the clock has
481
+ ns resolution. Without a seed (the direct-run path gets none; the control path's run
482
+ request carries one) `crun` blocked in `getrandom()` forever. Fix (`src/host/clock.js`):
483
+ CLOCK_MONOTONIC is strictly increasing, every read at least 16 ns (one counter tick) after
484
+ the previous one on any Worker; it can run ahead of real time by no more than 16 ns per
485
+ read made within one coarsened step. After it, `crng init done` came at 22 s and the
486
+ container ran (`test/unit/clock-strict.test.js` fails on the old clock). The entropy this
487
+ gives is weak; real entropy should come from the host (`crypto.getRandomValues`): the
488
+ control channel's `seed`, or a `/chosen/rng-seed` in vm.wasm's device tree (a guest change).
489
+ **Update (wmd main, W18 70d03426):** vm.wasm now puts 64 host bytes in `/chosen/rng-seed`
490
+ (WASI `random_get`, which `src/host/wasi.js` serves from `crypto.getRandomValues`); browser
491
+ boots log `crng init done` at 0.000000 (P-B11 section "Review of 70d03426").
492
+ 3. **9P reply amplification.** The first version read a whole chunk of requests and answered
493
+ all of them before checking its output limit: 400 23-byte Treads made 26 MB of queued
494
+ replies, and the "rings full on both sides" test passed or failed by timing (1 in 6
495
+ runs). The service now answers only while less than OUT_LIMIT of replies wait unread and
496
+ holds the rest of the requests back.
497
+ 4. **The guest's `udhcpc` waits 15 s** in the container boot: eth0 exists and nothing answers
498
+ (the net stub drops frames). That is 15.5 of the container boot's 25.7 s (Chromium).
499
+ P-B10's relay, or a DHCP answer from the stub, removes it. **Update (P-B10):** with a net
500
+ service the network Worker answers DHCP; udhcpc takes 0.4-0.5 s.
501
+ 5. **busybox `stty -F`** said "Not a tty" on `/dev/hvc0`; the test sets the tty up through a
502
+ descriptor instead (`exec 3<>/dev/hvc0; stty ... <&3`).
503
+
504
+ ## P-B4: alpine from packs
505
+
506
+ Two paths, both from `alpine.pack` (alpine:3.20 linux/arm64, made by `test/fixtures-gen`
507
+ from the layer in `~/.wmd/data`, read only) served by the JS 9P server under `at: 'rootfs'`
508
+ with `bundle/config/config.json` beside it, i.e. the tree the engine exports
509
+ (`internal/engine/ninep`):
510
+
511
+ 1. **Proof kernel and initramfs** (`?case=alpine`). `/proof.sh` mounts the share at `/mnt`
512
+ (`p9MountOptions()`: `trans=virtio,version=9p2000.L,msize=262144,cache=mmap`, see the
513
+ review's F1 below; it was `cache=loose` for the numbers in this table), runs `ls -R /mnt/rootfs`,
514
+ mounts proc, sys and devtmpfs in it and runs `chroot /mnt/rootfs /sbin/apk info`, a
515
+ shell, `ls -R /` inside the chroot, and writes a file into the upper.
516
+ 2. **The container path, as the engine boots it** (`?case=container`): the container
517
+ kernel and the Go init's initramfs from `~/.wmd/bin/guest/arm64`, the engine's kernel
518
+ command line without `wmd.control` (`wmd.init wmd.p9tag=wmdroot wmd.exit`, plus
519
+ `p9InitCmdline()`, `wmd.p9msize=262144 wmd.p9cache=mmap`, since the review; the init
520
+ then runs the bundle's process directly, as `guest/linux/test-init-boot.sh` does
521
+ without `control`). The init mounts the 9P share at `/run/wmd9p`, binds `rootfs/`,
522
+ writes runc's config from `bundle/config/config.json` and runs `crun`; the process is
523
+ `/bin/sh -c '...; ls -R /; apk info; ...'` in alpine. The init then writes
524
+ `bundle/exitcode` over 9P, which the page reads back (`channels.collect`).
525
+
526
+ | | Chromium | Firefox | WebKit |
527
+ |---|---|---|---|
528
+ | proof initramfs: boot to `PROOF-DONE` (ms) | 9,943 | 12,162 | 9,627 |
529
+ | `ls -R /mnt/rootfs` (710 lines, 0 errors) | 0.88 s | 0.69 s | 0.84 s |
530
+ | `chroot ... apk info` (14 packages) | 0.43 s | 0.38 s | 0.40 s |
531
+ | `ls -R /` in the chroot (proc and sys mounted) | 3.56 s | 5.84 s | 3.42 s |
532
+ | 9P requests / channel round trip p50 | 2,774 / 35.5 µs | 2,774 / 40.5 µs | 2,774 / 80.5 µs |
533
+ | container: bundle mounted (ms) | 19,654 | 19,631 | 19,667 |
534
+ | container: `ls -R /` (24,334 lines, 0 errors), `apk info` (14), `B4-DONE` (ms) | 25,400 | 27,649 | 25,406 |
535
+ | container: exit code; `bundle/exitcode` read back | 0; `0` | 0; `0` | 0; `0` |
536
+
537
+ Of the 19.7 s before the bundle mount, 4 s are the kernel and the init's start and 15.5 s
538
+ are `udhcpc` waiting with no network (problem 4).
539
+
540
+ ### What the Go init needs over the control channel (not built here)
541
+
542
+ With `wmd.control=vport` (the engine's command line) the init (`guest/container2wasm/cmd/init`)
543
+ opens the virtio-console port named `wmd.control` (`/dev/vportNp1`; kind 3 here) and runs a
544
+ hashicorp/yamux **server** on it; the host is the yamux **client** (keepalive every 5 s) and
545
+ opens one stream per operation (`internal/engine/control.go`, `ctlproto.go`):
546
+
547
+ - **run**: one JSON line `{"op":"run","now":<unix s>,"seed":<base64, 32 bytes>,"p9tag":"wmdroot","p9opts":"trans=virtio,version=9p2000.L,msize=65536,cache=loose"}`.
548
+ The init refuses a first `run` without `p9opts`. `now` sets the guest clock and `seed` is
549
+ credited to the kernel's pool (`RNDADDENTROPY`), which is the proper fix for problem 2
550
+ (`crypto.getRandomValues` on the host). Then frames both ways, `id[1] | len[4] BE |
551
+ payload`: 0 control (`{"resize":[r,c]}`, `{"close_stdin":true}`), 1 stdin, 2 stdout, 3
552
+ stderr, 4 exit (4-byte BE code, optional flags, bit 0 "synced"). The host sends
553
+ `close_stdin` at once when there is no stdin.
554
+ - **exec / kill / pause / resume / ps / stats**: the same line with their fields;
555
+ non-streaming ops answer one JSON line.
556
+ - After the main process exits the init syncs, closes the session, writes `bundle/exitcode`
557
+ and powers off.
558
+
559
+ So P-B4 with the engine's own command line needs a yamux client in the service Worker
560
+ (kind 3), the run request above, and the frame demux to the page; the transport underneath
561
+ (the control channel) is the one measured above.
562
+
563
+ ### What remains for P-B4
564
+
565
+ - The control channel's yamux client and run request (above), instead of the direct-run
566
+ command line.
567
+ - Packs from OPFS (`syncHandleSource` is written, untested in a browser; the tests fetch the
568
+ pack into memory), several layers per image, and writing the upper back out.
569
+ - Network (P-B10): the 15 s `udhcpc` wait, and `apk add`.
570
+ - Entropy from the host (problem 2) once the control channel carries the seed.
571
+ - The browser engine's run request must send `p9opts` from `p9MountOptions()` (cache=mmap),
572
+ not the native engine's cache=loose (review F1 below).
573
+
574
+ ## P-B3/P-B4 adversarial review: findings and fixes
575
+
576
+ The independent review (`test/b3-adversarial-results.txt`, commit `d6d41e41`) found F1-F5.
577
+ Fixes on `wmd-web-b3`; each has a test that failed before and passes after (unit: the first
578
+ 13 tests of the new `test/unit/b3-fixes.test.js` run against the pre-fix `src/` give 2 pass,
579
+ 11 fail; the two that pass are F4a's gap tests, which fail under the review's mutation M8
580
+ instead; the O_APPEND and step-budget tests, added later, each failed before their fix).
581
+ Browser tests: `test/e2e/b3-fixes.spec.js` (opt-in, `WMD_WEB_B3ADV=1`, under the boot lock).
582
+
583
+ | | Status | Fix | Test |
584
+ |---|---|---|---|
585
+ | F1 full upper + cache=loose hangs the writer | **fixed for the browser's mount options**: the guest mounts with `cache=mmap`; cache=loose cannot be fixed host-side (below) | `src/host/p9mount.js` (`p9MountOptions()`, `p9InitCmdline()`), used by `page/chan.js`, `page/b3-initrd.js`, `test/features/b3r-page.js` | b3-fixes "the upper memory cap with the default mount options" (C/F/W pass); the review's cache=loose test still fails, as it must |
586
+ | F1 follow-up: concurrent O_APPEND under cache=mmap lost records | fixed | `src/fs/9p/server.js` `Twrite` appends for O_APPEND fids (as QEMU) | unit "two fids opened with O_APPEND ..."; the review's container torture (`conc.append`) and b3-fixes "torture on the default mount options" |
587
+ | F2 no default upper cap | fixed | `src/boot.js` `upperCapDefault()`, `channels.maxUpperBytes`, `timings.chanUppers`; `src/fs/overlay.js` `full()`/`fullHits`/`onFull`; `chan-service.js` `upper-full` message; `chan-services.js` `snapshot().uppers` | unit F2 (3); b3-fixes "channels.maxUpperBytes ..." and "the default cap ..." |
588
+ | F3 console input unbounded | fixed | `src/host/chan.js` `consoleInputGate`, `H_CIN_TAKEN`; `chan-services.js` `consoleInput()` bound; `boot.js` `vm.consoleWrite()` returns the bytes taken, `vm.consoleSpace()` | unit F3 (3); the review's unit finding test now passes; b3-fixes 512 MiB `vm.consoleWrite` to a guest that reads nothing |
589
+ | F4a recv vs the exit flag untested | test added | - | unit F4a (2): fail under M8 (drop `\|\| exiting()`), pass otherwise |
590
+ | F4b chan.spec leak check vacuous | made real | - | chan.spec "powers off ... nothing leaks": every Worker the page made was terminated (Worker wrapper), Chromium: no worker target left (CDP) |
591
+ | F4c control order/content unchecked | fixed | `chan-services.js` sink CRC-32 (`kinds.control.crc32`) | unit F4c; chan.spec control 50 MiB: CRC equals that of `yes \| head -c 50M` |
592
+ | F4d "the guest never waited" not asserted | asserted | - | chan.spec control: `sendWaits` 0 |
593
+ | F5 blocked guest threads wake only at the next 1 s slice | fixed | `chan.js` `markGuestGone` kicks every ring; recv/send also stop on the guest-gone flag | unit F5 (3): 0.1-0.3 ms after (700 ms before); the review's ring exit test bound 2,000 -> 600 ms |
594
+
595
+ ### F1: why cache=mmap, with numbers
596
+
597
+ **Why loose cannot be fixed on the host side.** Linux 6.1's v9fs with cache=loose writes
598
+ into the page cache; `write(2)` never talks to the server. Writeback sends one 4 KiB page per
599
+ Twrite (`fs/9p/vfs_addr.c` `v9fs_vfs_writepage`, v6.1.187). On any error but EAGAIN the page
600
+ is dropped and the error only recorded (`mapping_set_error`) for a later fsync/close; EAGAIN
601
+ redirties the page (an endless retry). So no errno (ENOSPC, EIO, ...) the server can return
602
+ makes `write(2)` fail, and no headroom the server could reserve helps either: whatever it
603
+ accepts past the cap is still accepted. Measured (b3-fixes "cache=loose, for the record", a
604
+ 48 MiB cap): `dd bs=1M count=64 conv=fsync` writes all 64 MiB and fails only at its fsync
605
+ (ENOSPC); a further 16 MiB `dd` without fsync returns 0, `sync` returns 0, and after
606
+ `drop_caches` the server holds at most the 48 MiB: the rest was lost without the writer
607
+ knowing. A writer past the dirty limit sits in `balance_dirty_pages` while writeback drains
608
+ failing pages one round at a time (the review's hang). With cache=mmap and cache=none,
609
+ `write(2)` is a Twrite of up to msize (`v9fs_file_write_iter`) and ENOSPC reaches it at once.
610
+
611
+ **cache=mmap over cache=none**: the same write path, and shared writable mmap works (none
612
+ refuses it with EINVAL, wmd D-062; the review's torture has an `mmap MAP_SHARED` check).
613
+
614
+ **What cache=mmap broke, and its fix: O_APPEND.** Without the page cache, v9fs takes an
615
+ O_APPEND write's offset from its i_size without a lock (`v9fs_file_write_iter` takes no
616
+ inode lock), so processes appending to one file through their own opens overwrite each
617
+ other's records: the review's torture inside a container (crun, cache=mmap from the new
618
+ default) failed `conc.append`: 565 records, 191 out of order. QEMU's server does not have
619
+ this problem (its pwrite on an O_APPEND host file appends) and neither does Linux's pwrite(2);
620
+ wmd's Go server ignores O_APPEND, as this one did. The JS server now writes every Twrite of
621
+ a fid opened or created with O_APPEND at the end of the file (`src/fs/9p/server.js`
622
+ `Twrite`; unit test in `b3-fixes.test.js`, fails without it). Under cache=loose writes go
623
+ through the writeback fid, opened without O_APPEND, so nothing changes there.
624
+
625
+ **Cost.** `b3-fixes.spec.js` "measure cache=X", one boot each (proof initramfs, 2 vCPUs, 512
626
+ MiB, msize 262144), then a container boot (Go init + crun) with the same mode; seconds from
627
+ the guest's `/proc/uptime`, one run each (they vary by about ±30% run to run, e.g. WebKit
628
+ mmap `ls -R` 1.22 and 2.63 s); canary 0.62-0.68 s around every run except where noted.
629
+
630
+ | | Chromium none / mmap / loose | Firefox none / mmap / loose | WebKit none / mmap / loose |
631
+ |---|---|---|---|
632
+ | `ls -R /mnt/rootfs` (710 lines) | 1.08 / 1.15 / **0.70** | 1.49 / 1.24 / **0.99** | 2.47 / 2.63 / 1.75 |
633
+ | the same again (warm) | 0.70 / 0.68 / **0.30** | 1.68 / 1.01 / **0.33** | 1.06 / 1.39 / 0.63 |
634
+ | `chroot apk info` | 0.57 / 0.52 / 0.40 | 0.85 / 0.48 / 0.42 | 0.68 / 0.54 / 0.53 |
635
+ | 100 x exec `/bin/true` in the chroot | 3.18 / 3.39 / 2.99 | 5.79 / 2.92 / 2.73 | 3.67 / 3.38 / 3.41 |
636
+ | `ls -R /` in the chroot | 4.35 / 4.27 / 3.84 | 12.27 / 6.47 / 6.13 | 6.05 / 4.78 / 5.01 |
637
+ | 100 MiB write / copy / read, MB/s (sha256 equal) | 37.0/57.0/95.4 / 37.9/54.8/82.9 / 33.5/40.3/93.5 | 34.5/19.5/45.8 / 40.1/26.4/40.9 / 29.2/21.8/55.2 | 43.3/34.8/67.1 / 43.3/28.7/84.9 / 38.0/27.0/88.5 |
638
+ | 100,000-file create, s (µs/file) | 78.9 (789) / 81.6 (816) / 134.5 (1345) | 158.5 (1585) / 159.2 (1592) / 191.7 (1917) | 146.3 (1463) / 102.4 (1024) / see below |
639
+ | container boot: `ls -R /` + `apk info` done (ms after start) | 25,676 / 25,659 / 25,557 | 26,736 / 26,551 / 25,965 | 28,443 / 26,885 / 26,829 |
640
+
641
+ So cache=mmap costs metadata-heavy reads (no dentry/attribute cache): a cold `ls -R` of the
642
+ rootfs +0.25-0.9 s, a warm one +0.4-0.8 s, `apk info` +0.0-0.1 s, and sequential reads of a
643
+ big file 4-26% (no readahead); it is faster for writes (100k creates 1.2-1.65x faster than
644
+ loose, writes 13-37% and copies 6-36% faster), and the container boot is the same (within
645
+ 1.6 s of 25.6-28.4 s, most of it `udhcpc`'s 15 s). cache=none and cache=mmap are within
646
+ run-to-run noise of each other.
647
+
648
+ WebKit, cache=loose, 100,000 files: WebKit ended the page (Playwright saw a navigation, the
649
+ page came back empty) 25-40k files into the create, 3 times of 3 at 2 vCPUs (with and without
650
+ the 100 MiB copy before; the review's run at 1 vCPU and msize 65536 passed in 122.5 s), and
651
+ once of two under cache=none in a run where the canary read 1.28 s afterwards (the retry
652
+ passed). Not investigated further: the browser gives no reason (no `crash` event, no report
653
+ in `~/Library/Logs/DiagnosticReports`). The WebKit loose row comes from a boot without the
654
+ 100k-file step (`B3FIX_SKIP=bigdir`).
655
+
656
+ ### F2: the default cap
657
+
658
+ `bootWmd({channels})` caps each 9P service's upper at `channels.maxUpperBytes` or, by
659
+ default, `upperCapDefault()`: an eighth of `navigator.deviceMemory` from 256 MiB to 1 GiB
660
+ (Playwright's Chromium reports 16 GiB here: 1 GiB), 512 MiB where the browser does not say
661
+ (Firefox, WebKit). A service's own `maxUpperBytes` wins; `Infinity` turns it off.
662
+ `timings.chanUppers` says what each port got and why, `vm.chanStats().service.uppers` the
663
+ bytes used, the cap and the refusals. The first refused write (then at most one every 10 s)
664
+ puts on `onStderr`: `[wmd-web] the guest's files on 9P port 564 are full: the upper holds
665
+ 32 MiB of its 32 MiB cap (channels.maxUpperBytes); the guest's writes get ENOSPC (No space
666
+ left on device)`, and an `{t:'chan', ev:'upper-full'}` event. Only file contents count (as
667
+ before); names, inodes and xattrs do not.
668
+
669
+ ### F3: console input
670
+
671
+ `vm.consoleWrite(data)` returns the bytes it took (it returned `true`), at most
672
+ `vm.consoleSpace()`: the page counts what it posted, the service counts in the chan block
673
+ what it has put into the guest's ring (or had to drop), and the difference never exceeds
674
+ `channels.consoleInputBytes` (4 MiB). When it takes less than it was given, `onStderr` says
675
+ once `[wmd-web] console input is full: the guest has not read the 4 MiB already queued ...`
676
+ (again after a write that fits), plus a `console-full` event. The service also bounds its
677
+ own queue (anything posted past the bound by other means is dropped and counted:
678
+ `service.console.inputDropped`). For a string the count is of its UTF-8 bytes. Measured
679
+ (b3-fixes, a guest that reads nothing, 512 x 1 MiB writes): the page took 6,295,552 bytes
680
+ (4 MiB + the 1 MiB ring + 1 MiB the guest's virtio-console driver had buffered), refused
681
+ the rest in 0.2 s, nothing dropped; the service Worker's heap in Chromium stayed at 18 MiB
682
+ (540 MiB before the fix).
683
+
684
+ ### Test runs after the fixes
685
+
686
+ Each under the boot lock, one browser per run, `PORT=8796`, canary 0.61-0.73 s around every run.
687
+
688
+ | | Chromium | Firefox | WebKit |
689
+ |---|---|---|---|
690
+ | `bun test test/unit` | 241 pass, 2 skip (the review's opt-in file), 0 fail; with `WMD_WEB_B3ADV=1` 243 pass, 0 fail | | |
691
+ | `b3-fixes` (torture on the default options, ENOSPC on the default options, the loose record, F2 cap, F3 flood, default) | 6 passed | 6 passed | 6 passed (F3 at the second run: the first failed my own bound of > 500 refused writes with 498; now > 480) |
692
+ | `chan.spec` | 10 passed, 1 failed (`hvc0`: problem 1, the guest's IRQ storm), 1 skipped (UART) | same | same |
693
+ | the review's rings (`b3-adversarial -g rings`, 7 tests) | 7 passed | 7 passed | 7 passed |
694
+ | the review's guests (10 tests) | 9 passed, 1 failed: ENOSPC with cache=loose (F1: cannot pass, above) | same | same |
695
+ | `boot.spec` + `jit.spec` | 14 passed, 1 skipped (amd64) | same | same |
696
+
697
+ - The control channel's 50 MiB: CRC-32 equal, `sendWaits` 0 in all three (F4c, F4d). The
698
+ leak check: 10 Workers made, all terminated, and no worker target left in Chromium; with
699
+ `chanSvc?.terminate()` removed from `finish()` it fails (`not terminated:
700
+ [{"name":"wmd-chan"}]`) (F4b).
701
+ - The review's ring tests after the fixes: the guest exits at random moments, last thread
702
+ back 207 / 207 / 209 ms after (was up to 1,010 ms); the service killed, 206 / 218 / 211 ms
703
+ (was 208 / 238 / 216); the 512 MiB console flood, service Worker heap 23 MiB in Chromium
704
+ (was 540 MiB).
705
+ - `alpine` and `container` in `chan.spec` now mount with cache=mmap (the default): proof
706
+ `ls -R` 1.2 s and `ls -R /` in the chroot 4.4 s in Chromium (3.56 s with cache=loose in
707
+ the P-B4 table), container `B4-DONE` at 26.8 s (25.4 s).
708
+
709
+ Problems the re-runs found, all fixed and re-run:
710
+
711
+ 1. **The service could be held in one step** (`chan-services.js` `#pump`): it read a channel
712
+ for as long as the guest kept its ring full. With the control stub now computing a
713
+ CRC-32, a guest writing 1 GiB to the control channel through a 1 MiB ring kept one
714
+ `step()` reading 180 MB (572 ms in the unit test), so the service took its messages late:
715
+ the review's kill test saw the page learn of a `die`/`crash` 1.7-3.8 s late (3 of 12, all
716
+ on 1 MiB rings). Now one channel reads at most `STEP_READ_BUDGET` (1 MiB) per step (unit:
717
+ 1 MiB, 6 ms).
718
+ 2. **The review's ring harness posted console input past the new bound** (it plays the page
719
+ and posted straight to the service, 4 MiB ahead of the guest plus one chunk): the
720
+ service dropped the excess and the 8-channel test's console reader got 5.5 of 16 MiB. The
721
+ harness now takes input through `consoleInputGate`, as `vm.consoleWrite` does (the flood
722
+ test still posts past it, which is what tests the service's own bound).
723
+ 3. **cache=mmap and O_APPEND** (F1 above): the container torture failed `conc.append` until
724
+ the server honoured O_APPEND.
725
+ 4. **WebKit ended the page** in the F1 measurement's cache=loose 100,000-file step at 2 vCPUs
726
+ (F1 above); the review's own 100,000-file test (1 vCPU) passed in all three.
727
+
728
+ What is left:
729
+
730
+ - The review's `cache=loose` ENOSPC test still fails, by design: under cache=loose the cap
731
+ cannot reach `write(2)`. A host that mounts with cache=loose (the native engine's run
732
+ containers) must not set a cap it relies on, or must accept that writes past it are lost
733
+ with only fsync/close reporting it.
734
+ - cache=mmap gives up the dentry/attribute cache: metadata-heavy work is slower (the F1
735
+ table); an attribute cache with revalidation (cache=loose for metadata only) is not
736
+ something Linux 6.1's v9fs offers.
737
+ - The cap counts file contents only (names, inodes, xattrs are not charged).
738
+ - `hvc0` (problem 1) and the clock / entropy notes (problem 2) are unchanged.
739
+
740
+ ## P-B4: the 9P server (core)
741
+
742
+ `src/fs/` is a transport-agnostic 9P2000.L server over an overlay of wmd packs (the
743
+ read-only lowers) under an in-memory upper: `new NinePServer(new Overlay({layers,
744
+ at: 'rootfs'})).handle(frame) → reply`, synchronous, so it can sit behind P-B3's
745
+ shared-memory channel. The pack reader is checked entry by entry against wmd's Go
746
+ `pack.Reader`, and the server frame by frame against wmd's Go 9P server
747
+ (`internal/engine/ninep` over memfs) on the same packs, alpine:3.20 included. It has
748
+ not run in a browser or under a guest yet. Format coverage, layering rules, messages,
749
+ the known differences from the Go server, the independent review (F1–F6) and the
750
+ numbers are in [docs/9p.md](docs/9p.md).
751
+
752
+ ## Integration of P-B1, P-B2 and P-B4 (branch `wmd-web-int`, 2026-10-05)
753
+
754
+ `wmd-web` (P-B1 adversarial fixes) and `wmd-web-9p` (P-B4 9P core) merged into the
755
+ P-B2 branch. Everything from both sides is kept: B1's abort, release, watchdog,
756
+ throttle detector, panic exit, host-call counters, realtime clock and `--ram` check;
757
+ B2's JIT door, hub sharing under JSPI, 32 MiB stdio rings with drop markers.
758
+
759
+ Two things broke in the merge and were fixed:
760
+
761
+ - **Synthetic guests trapped.** The worker attached the JIT door to every module, and
762
+ the P-B1 review's synthetic guests (`test/e2e/review-synth/*.wat`) have no
763
+ `vm_helper_*` exports (`vm.wasm has no export vm_helper_0`). A module now gets a
764
+ door only if it imports `wmd.jit_load`.
765
+ - **Firefox kept a reloaded page's boot memory.** B2's test page held the running
766
+ handle on `window.wmd.vm`; after three mid-boot reloads RSS stayed 1.3 GiB higher
767
+ even after GC (B1's page: back to base). The page now holds the handle in a WeakRef
768
+ (`window.wmd.vm` still works while a boot runs).
769
+
770
+ The stubbed `jit_*` imports that B1's counters were built for only exist without a
771
+ door now: `bootWmd({jit: false})` / `?door=0`. B1's three stubbed-door tests ask for
772
+ it explicitly.
773
+
774
+ Suites on the merged branch (one run each, under the lock, canary 597–655 ms except one
775
+ contended stretch that was only used for A/B memory runs):
776
+
777
+ | | Chromium | Firefox | WebKit |
778
+ |---|---|---|---|
779
+ | `bun test test/unit` | 197 pass (bun) | | |
780
+ | `boot.spec adversarial.spec` (the filter also picks the 3 no-guest storm tests of `b2-adversarial`) | 11/11; rerun 10/11\* | 11/11 | 10/11 (reload x3 memory: WebKit, same on B1) |
781
+ | `fixes.spec b2-fixes.spec throttle.spec` (skips: headed throttle, opt-in 3b, Chromium-only GC test) | 16/16 | 15/15 | 15/15 |
782
+ | `review-errors` | 27/28 (`--cpus 64` interp) | 27/28 (same) | 27/28 (same) |
783
+ | `review-lifecycle review-smp` | 11/11 | 8/9 ("close a tab": no Worker close events) | 7/9 ("close a tab"; sequential boots: WebKit's lazy return, also fails on B1) |
784
+ | `jit.spec jit-stress` | 20/20 | 18/20 (per-Worker stress: ~16,300-copy limit, share=0) | 20/20 |
785
+ | `b2-adversarial` (`WMD_WEB_B2ADV=1`) | 12/12 | 12/12 | 12/12 |
786
+
787
+ \* "reload during kernel load" failed 2 of 12 runs on Chromium: the test's
788
+ `page.evaluate` after the page's own `location.reload()` lands in the destroyed context.
789
+ It passed 4 of 4 on B1 alone and 6 of 6 in the next merged run.
790
+
791
+ Proof boots on the merged branch (ms, `PROOF-DONE` from the `bootWmd` call):
792
+
793
+ | | Chromium | Firefox | WebKit |
794
+ |---|---|---|---|
795
+ | interp (P-B1 boot) | 42,207 / 39,999 | 53,798 / 52,039 | 44,761 |
796
+ | jit, 1 vCPU | 5,842 | 6,092 | 5,743 |
797
+ | jit, 8 vCPUs: copies / RSS peak | 232 / 1,224 MiB (5,523) | 234 / 1,853 MiB (5,867) | 232 / 1,400 MiB (5,580) |
798
+
799
+ ## Adversarial checks (PROOFS.md P-B1)
800
+
801
+ ### Independent review (test/adversarial-results.txt, commit `b819865e`) and fixes
802
+
803
+ The review's specs are `test/e2e/review-*.spec.js`; the tests for the fixes are
804
+ `test/e2e/fixes.spec.js`, the reload tests in `adversarial.spec.js`,
805
+ `test/e2e/throttle.spec.js` and `test/unit/monitor.test.js`, `gate.test.js`. Each fix's
806
+ test was run against the pre-fix host (Chromium, commit `b819865e` with the new specs)
807
+ and failed there, except where noted.
808
+
809
+ | Finding | Fix | Status |
810
+ |---|---|---|
811
+ | AbortSignal fired while vm.wasm compiles is never seen; boot keeps running | `src/boot.js` `bootWmd`: listener registered at the start; every await is raced against the abort; `signal.aborted` checked after the last await. Before the Workers exist the boot rejects with `AbortError` (no Worker is created); after, `exited` gives `{code:-1, reason:'aborted'}` and every Worker is terminated. | fixed. Ends 2.5 ms after the abort (Chromium); passes in C/F/W, and the review's own abort test passes in C/F/W. (The two guard tests, abort before the call and abort after the Workers started, also passed before the fix.) |
812
+ | `--mode jit`: one postMessage per stubbed `jit_*` call (477k in 30 s), unbounded `events` | `src/host/layout.js` `noteHostCall`/`hostCallCounts`, `src/host/wmd-imports.js` `counters`: Int32 counters in the control block. The page gets one `host-calls` event at most every 250 ms (`onTick`), `vm.stats().hostCalls` on demand. The test page keeps the latest counts and caps `events` at 1000. stdout/stderr through shared rings (see Output). | fixed. 395k `jit_compile` calls in 10 s → 39 events; page lag ≤ 10 ms and `evaluate` 1-4 ms over 30 s (the review's jit test) in C/F/W |
813
+ | Memory kept across sequential boots (WebKit to ~2.5 GiB over 5 boots) | `src/boot.js` `release()`: Workers terminated and their handlers cleared, timers, abort listener, module and file buffers dropped; `vm.memory` is a getter nulled by `vm.dispose()`; the test page disposes and keeps no handle. | host fixed: no boot's memory survives (Chromium forced GC: WeakRef cleared; WebKit: 0 of 8 memories alive after each boot). WebKit RSS still varies, see "Memory" below |
814
+ | Minimised Chromium window: boot 6.2x slower, page still "visible", nobody told | throttle detector (`src/calib-worker.js`, `src/host/monitor.js` `createThrottleMeter`, wired in `bootWmd`) | detected within 2.5 s in a minimised window (ratio 2.4-3.2), never in visible/headless boots. See "Background throttling" |
815
+ | Truncated kernel: silent hang | progress watchdog (`createWatchdog`, `watchdogMs`, default 60 s arm64): code -2, `guest made no progress` | fixed. The review's 120 s-bounded test now ends at the watchdog (about 60 s) in C/F/W; with `watchdogMs: 5000` it ends in 6-8 s. Guest fix needed (below) |
816
+ | Kernel panic reported as exit 0 | `createPanicScanner` on stdout; the exit becomes -3 `kernel panic: Kernel panic - not syncing: ...`, `timings.guestExitCode` keeps the guest's 0 | fixed in C/F/W. Guest fix needed (below) |
817
+ | `--ram abc` gives an engine message | `ramMiB()`: `--ram "abc": expected a whole number of MiB, e.g. --ram 256`, before any download | fixed in C/F/W |
818
+ | Firefox/WebKit `reason` is a stack without the message | page: `reason = "<name>: <message>"`, the stack separately in `stack` | fixed (the test passed before in Chromium, whose stacks include the message) |
819
+ | CLOCK_REALTIME steps back across Workers (620k of 1e6 reads, up to 20 us) | `src/host/clock.js` `realNs` = shared start time + CLOCK_MONOTONIC | fixed: 0 of 1e6 backwards in C/F/W (before: 623,846, max 5.1 us, Chromium) |
820
+ | "reload during kernel load" reloaded before the download | the page reloads itself from inside the window (phase `running`, no console byte yet) and records the state in sessionStorage | fixed: at reload `{"phase":"running","stdoutBytes":0,"stderr":""}` |
821
+ | No reload test checks that the old page's Workers and memory go | `adversarial.spec.js` "reload mid-boot x3 releases ..." | added. Passes in Chromium also before the fix (Chromium frees a reloaded page's memory). Firefox and WebKit: see "Memory" |
822
+ | TIMER-DELTA gate accepts 1.0-1.9 | `test/e2e/helpers.js` `bootGate`: 1.0-1.3 unless `{throttled: true}` | tightened. Every unthrottled boot in this round printed 1.1 |
823
+ | `--cpus 64` still booting at 60 s (INFO) | none | not a host fault: 64 interpreted vCPUs on 10 cores. The review's 60 s bound still fails it in C/F (left as is) |
824
+
825
+ ### Background throttling
826
+
827
+ The review found that a really minimised headed Chromium window boots 6.2x slower
828
+ (`PROOF-DONE` at 267.6 s) while `document.visibilityState` stays `"visible"`, so the
829
+ page cannot tell from the Page Visibility API. The host now measures it directly: the
830
+ calibration Worker's fixed loop takes about 9.5 ms on this machine in the foreground.
831
+
832
+ `test/e2e/throttle.spec.js`, headed Chromium, 2026-10-04 (`REVIEW_HEADED=...`;
833
+ progress at 90 s of wall time unless the boot finished):
834
+
835
+ | Window | Chromium flags / page | throttle report | guest progress |
836
+ |---|---|---|---|
837
+ | visible | none | none (21 samples, max ratio 1.05) | `PROOF-DONE` at 40.5 s, TIMER-DELTA 1.1, strict gate passes |
838
+ | minimised | none | **throttled after 2.5 s, ratio 2.4** | at 90 s: no `wmd-init: up` yet (visible: 15.4 s), 14.4 KB of console |
839
+ | minimised | `--disable-renderer-backgrounding --disable-background-timer-throttling --disable-backgrounding-occluded-windows` | throttled after 2.5 s, ratio 3.15 | the same: no init at 90 s, 14.4 KB |
840
+ | minimised | the page holds a Web Lock (`navigator.locks.request`) for the boot | throttled after 2.5 s, ratio 2.76 | the same: no init at 90 s, 14.4 KB |
841
+
842
+ What this shows:
843
+
844
+ - **Detection works.** In each minimised run the host reported throttling on its
845
+ second sample (2.5 s) and wrote `[wmd-web] the guest is running about 2.4x slower ...`
846
+ to the emulator log; the test page shows it in its status line. No visible or headless
847
+ boot in this round reported it (boot.spec requires 0 throttled samples), and the
848
+ watchdog stretches its limit by the ratio.
849
+ - **The ratio understates the slowdown.** The calibration loop runs 2.4-3.2x slower,
850
+ the guest about 6x (init not reached at 90 s against 15.4 s). A short burst every
851
+ 2 s is scheduled better than a vCPU thread that never sleeps. Treat the ratio as "at
852
+ least this much".
853
+ - **Nothing a page can do helped.** Chromium's own background flags made no difference,
854
+ so this is not Chromium's renderer backgrounding or timer throttling; the slowdown is
855
+ below the browser (macOS lowering the priority of a minimised app's threads, App Nap
856
+ or QoS; not investigated further). A held Web Lock made no difference either.
857
+ `Atomics.wait(10 ms)` overslept by only 0.5-3 ms, so Worker timers are not the
858
+ problem; CPU time is. Audio playback or similar tricks were not tried: they are hacks
859
+ the browser may break, and they would not touch OS-level scheduling anyway. Only
860
+ minimising was measured; an occluded window or a background tab was not.
861
+ - **False positives.** A synthetic guest that parks 128 threads (the review's spawn
862
+ storm) made the calibration loop 2.1-2.6x slower while 128 Workers started, and the
863
+ host reported throttling for a few seconds (Chromium, Firefox). That is CPU
864
+ contention, which also slows the guest, so the report is not wrong, only its wording.
865
+ - Firefox and WebKit were not checked headed.
866
+
867
+ ### Memory
868
+
869
+ 5 sequential arm64 boots in one page (`review-lifecycle.spec.js`, `REVIEW_SEQ_BOOTS=5`),
870
+ RSS of the browser's processes after each boot, MiB:
871
+
872
+ | Browser | before the fixes (review) | after |
873
+ |---|---|---|
874
+ | Chromium (forced GC) | 880, 898, 342, 503, 254 | 404, 358, 407, 422, 432 |
875
+ | Firefox (no GC command) | 1578, 1545 (2 boots) | 1578, 1136, 1104, 1630, 1064 |
876
+ | WebKit (no GC command) | 1074, 1655, 2195, 2591, 2501 | 578, 955, 1248, 1782, 2114 (fails the review's 300 MiB bound) |
877
+
878
+ WebKit, 8 boots with a GC forced by allocation pressure after each boot (allocate and
879
+ drop 2 GiB of never-touched ArrayBuffers; `fixes.spec.js` 3b, `WMD_WEB_SEQ=8`):
880
+ RSS 1073, 1137, 1200, 1655, 750, 1155, 1164, 913 MiB, then flat at 912 for 60 s idle.
881
+ After every boot **0** of the boots' `WebAssembly.Memory` objects were still alive
882
+ (WeakRefs held by the page were all cleared) and the WebContent process was back to
883
+ 17-18 threads (15 before the first boot), so neither the host nor a lingering Worker
884
+ holds any boot's memory. What stays is WebKit returning freed wasm memory to the
885
+ system lazily: bounded (750-1655 MiB over 8 boots, no upward trend), not freed while
886
+ idle. Without allocation pressure WebKit's GC does not run between boots at all (the
887
+ page is idle), which is why the review's test, which forces GC only in Chromium, still
888
+ sees growth. The host does not allocate garbage on purpose to provoke a GC.
889
+
890
+ Reload mid-boot three times (`adversarial.spec.js`), RSS MiB:
891
+
892
+ | Browser | base | after the 3rd reload, no GC | after GC |
893
+ |---|---|---|---|
894
+ | Chromium | 293 | | 434 (forced) |
895
+ | Firefox | 1004 | 1934 | 1099 (allocation pressure) |
896
+ | WebKit | 357 | 2481 | 2481 (allocation pressure did not help) |
897
+
898
+ Every old page's Workers closed within 3 s of each reload in all three browsers.
899
+
900
+ The review's "close a tab mid-boot x3" fails in Firefox both before the fixes (RSS
901
+ 993 → 1390, 1582, 1607 after each close; this round, on the pre-fix commit) and after
902
+ (1000 → 1568, 1581, 1547): Firefox keeps about one boot's memory after a tab closes and
903
+ Playwright reports no Worker `close` events for it, so the test's Worker count cannot
904
+ pass there. It does not grow from round to round. Chromium passes it (280 → 273). WebKit releases
905
+ the memory (351 → 352 after each close) but fails the same Worker-count check. In
906
+ WebKit the reloaded documents' memory was not returned within 10 s even after
907
+ allocation pressure; a reload replaces the whole JS realm, so no host reference can
908
+ survive it. That is left as a WebKit behaviour (the review's reload x5 measured 2567 MiB
909
+ too; it plateaus).
910
+
911
+ ### Needed in the guest
912
+
913
+ Not built here (the guest is out of scope for wmd-web):
914
+
915
+ - **Truncated kernel.** `guest/vm/src/board_arm64.c:527-540` loads `ksz` bytes of the
916
+ Image without comparing them with the header's `image_size` (offset 16). A 1 MiB
917
+ prefix of the 38.8 MB Image boots into garbage and spins silently. `image_size` is
918
+ the in-memory size including BSS (39,321,600 for the current Image, file 38,760,456),
919
+ so the check is `image_size != 0 && (ksz > image_size || ksz < image_size - BSS
920
+ allowance)`, with an allowance of a few MiB, failing with "kernel file truncated".
921
+ The host watchdog only bounds the hang.
922
+ - **Kernel panic exits 0.** `CONFIG_PANIC_TIMEOUT=-1`
923
+ (`guest/linux/fragment-common.config:38`) reboots at once, and PSCI `SYSTEM_RESET`
924
+ sets `exit_code = 0` (`board_arm64.c:702-705`). Reset should not be a clean exit
925
+ under `wmd.exit`: map SYSTEM_RESET to a non-zero code (or set panic_timeout=0 and let
926
+ the watchdog catch it). The host now reports -3 from the console line, but a panic
927
+ whose message is lost would still look like success.
928
+
929
+ `test/e2e/adversarial.spec.js`, Chromium. Every check boots the arm64 proof guest and
930
+ requires the full boot gate to pass.
931
+
932
+ | Check | What the test does | Result |
933
+ |---|---|---|
934
+ | reload mid-boot | Waits for the kernel banner and 3 more seconds, then navigates the same tab to a fresh boot. | pass (the new boot reaches `PROOF-DONE` at about 44 s) |
935
+ | reload during download | Re-navigates as soon as the page reports `booting`. | pass |
936
+ | two tabs at once | Two pages in one context boot concurrently. | both pass. `PROOF-DONE` came at about 43.6 s on one run and 47 s on another, for both tabs |
937
+ | tab backgrounded | After the banner, marks the page hidden and **blocks the page's main thread for 10 s**. | pass. The guest kept running: about 11 KB of console output queued up during the stall, and `PROOF-DONE` time was unchanged at about 44.5 s. |
938
+
939
+ The backgrounded-tab check simulates a background tab; it does not really background one.
940
+ Under Playwright 1.63 the following were measured:
941
+
942
+ - A page behind another tab, or in a minimized window, stays `visibilityState ===
943
+ "visible"`. This holds for headless shell, new headless and headed Chromium, and with
944
+ focus emulation off.
945
+ - CDP `Page.setWebLifecycleState({state: "frozen"})` is a no-op: page timers keep
946
+ firing and no `freeze` event fires.
947
+
948
+ The check therefore reproduces what a background tab does to this host: the main thread
949
+ stops serving events. A 10 s stall is harsher than background-tab throttling. The page
950
+ also falls back from `requestAnimationFrame` to timers when it is hidden. Real
951
+ background-tab behaviour in shipping browsers is still to be checked by hand.
952
+
953
+ ## P-B2: the JIT door
954
+
955
+ `src/host/jit.js` implements `wmd.jit_*` as `runner/src/jit.rs` does, for Workers.
956
+
957
+ - **One slot list for all threads, readable synchronously.** A Worker inside `jit_sync`
958
+ cannot receive `postMessage`, so the list lives in three SharedArrayBuffers:
959
+ - a header: lock, flush generation, record count, first record of this generation,
960
+ live slots, log bytes, largest table;
961
+ - a record per loaded module (growable): log offset, length, first slot, function
962
+ count, content hash and timings;
963
+ - an append-only log of the module bytes (growable, up to 1 GiB). A flush never
964
+ rewrites the log, so a thread reading bytes during a flush still reads valid bytes.
965
+ - **Table indices.** Slot k is at table index 155+k in every thread's table.
966
+ `jit_load` returns 155 + the slot count before the load. Slot i of a module is its
967
+ i-th function export (`b0..bN-1` for batch modules, `dispatch` for the chaining
968
+ dispatcher).
969
+ - **Sync.** A thread compiles each module it has not seen with the synchronous
970
+ `new WebAssembly.Module(bytes)` and keeps the Module by log offset. It instantiates the
971
+ module once per generation with `env.mem0` (and `env.mem1`, which browser blocks no
972
+ longer import: W50, Safari 26.5 refuses two memories) set to the one shared
973
+ memory, `env.tbl` set to its own table, and `hst.h0..h16` set to its own
974
+ `vm_helper_N`. Then it grows its table **exactly** to 155 + installed slots and sets
975
+ the entries. A new thread (pool Worker) syncs before `wasi_thread_start`, as
976
+ `spawn.rs` does.
977
+ - **The five imports.**
978
+ - `jit_load`: copy the bytes, compile, register, sync the caller. It returns -1 if the
979
+ module does not compile.
980
+ - `jit_compile` (the compile-worker thread): register without syncing. Without module
981
+ sharing it runs `WebAssembly.validate` first, so a bad module still returns -1. With
982
+ sharing (the default where JSPI exists, finding F1) it compiles, once for the
983
+ whole process, and posts the Module to the hub.
984
+ - `jit_sync`: as above. With sharing, a vCPU waits (JSPI) for Modules it has not
985
+ received yet instead of compiling them. A failure that is left after one retry
986
+ throws, which traps the thread and ends the machine with exit 4, as in the runner.
987
+ - `jit_flush`: bump the generation. Every thread clears `[155, size)` at its next sync,
988
+ and indices restart at 155.
989
+ - `jit_known` and `jit_note`: this session's block keys (D-513). A block whose code was
990
+ compiled before, wherever in memory it is now, gets that function back. With a module
991
+ cache across visits (W77) they also read and feed it.
992
+ - `jit_evict` (D-514): over the module budget, the door names the modules no block
993
+ entered lately, and the guest drops only those.
994
+ - **Unchanged from P-B1.** Address windows stay off (`win_init` → 1). The compile-worker
995
+ thread (`WMD_JIT_WORKERS`, default 1) runs through `thread-spawn` like any other
996
+ thread. Tail calls are used (the guest's default).
997
+ - **What is recorded.** Per module: compile time in the loading thread and in each
998
+ syncing thread, and instantiate time. Counters cover loads, compiles, syncs, sync time,
999
+ table sets and grows, bytes, dedup hits, flushes and failures. They are in
1000
+ `timings.jit`, in a table on the test page (refreshed every second), and in the test
1001
+ logs (`[<browser> <case>] jit=...`).
1002
+
1003
+ ### Code budgets and big builds (wmn#57, D-511 to D-518)
1004
+
1005
+ A Go build compiles a lot of guest code. Each browser has a limit on wasm machine code:
1006
+
1007
+ - **Firefox**: one 2 GiB pool per process. `jit.maxLiveBytes` defaults to 320 MiB of live
1008
+ modules (D-511). Doors pull Modules from the hub instead of being sent them, so a Worker
1009
+ that never reads its messages holds no released code (D-512).
1010
+ - **WebKit**: the pool fills with no error, and code made then runs interpreted for good.
1011
+ The door times a tiny module (`jit.codePool`, default on in WebKit) and, while the pool
1012
+ is full, answers `JIT_LATER`: the guest interprets and asks again a second later (D-515).
1013
+ - **All**: over the budget, only cold modules go (D-514). Blocks compiled before a release
1014
+ compile again after 2 runs (D-517). A region's function starts at any of its blocks, and
1015
+ no block is compiled into a second function (D-518).
1016
+
1017
+ golang:1.23-alpine with wmn's agent (`test/e2e/go-warm.spec.js`, `WMD_WEB_GO=1`; a fmt program,
1018
+ `encoding/xml`, the cold set), code written by the guest's JIT:
1019
+
1020
+ | | Chromium | Firefox | WebKit |
1021
+ |---|---|---|---|
1022
+ | before D-517, D-518 | 1.07 GB | 1.29-1.53 GB | 7.6 GB, fmt program not built in 600 s |
1023
+ | now | 0.52 GB, 7 s / 12 s / 93 s | 0.46-0.56 GB, 16-18 s / 16 s / 124-126 s | 0.60-0.62 GB, 16-33 s / 22-31 s / 564-574 s |
1024
+
1025
+ ### Numbers (2026-10-04, Apple Silicon, one run each, 1 vCPU, --ram 256; canary 620–654 ms around every run)
1026
+
1027
+ The arm64 proof boot is the P-B1 boot with `--mode jit`. Browser times start at the
1028
+ `bootWmd` call, so they include the download. Native times are wall-clock from process
1029
+ start (`wmd-run --wmd-jit-cache off`, so it has no on-disk cache either, as in the
1030
+ browser).
1031
+
1032
+ | arm64 proof boot | native interp | native JIT | Chromium interp (P-B1) | Chromium JIT | Firefox interp (P-B1) | Firefox JIT | WebKit interp (P-B1) | WebKit JIT |
1033
+ |---|---|---|---|---|---|---|---|---|
1034
+ | `wmd-init: up` (ms) | 19,977 | kernel 3.93 s* | 17,342 | 4,711 | 21,653 | 4,711 | 19,169 | 4,773 |
1035
+ | `PROOF-DONE` (ms) | 51,554 | n/a* | 44,042 | **6,584** | 57,534 | **7,135** | 49,166 | **6,752** |
1036
+ | total (exit) (ms) | 52,024 | 6,526 (wall) | 44,240 | 6,687 | 57,772 | 7,222 | 49,431 | 6,847 |
1037
+ | speed-up of `PROOF-DONE` over P-B1 | – | – | – | **6.7x** | – | **8.1x** | – | **7.3x** |
1038
+ | speed-up of `wmd-init: up` | – | – | – | 3.7x | – | 4.6x | – | 4.0x |
1039
+
1040
+ \* The native console has no wall-clock marks. `Run /init` is at kernel time 3.93 s
1041
+ native, 4.17 s in Chromium and 4.02 s in WebKit. The native run powered off at kernel
1042
+ time 6.25 s, with 6.53 s wall in total.
1043
+
1044
+ Interp boots in the same sessions as the JIT boots gave `PROOF-DONE` at 43,307 ms
1045
+ (Chromium), 56,542 ms (Firefox) and 48,543 ms (WebKit), all within 2% of the P-B1 table.
1046
+
1047
+ | JIT door, proof boot | Chromium | Firefox | WebKit |
1048
+ |---|---|---|---|
1049
+ | modules / functions | 196 / 8,927 | 198 / 8,935 | 198 / 8,933 |
1050
+ | module bytes (all loads) | 26.5 MB | 26.5 MB | 26.5 MB |
1051
+ | compile per module in the vCPU's sync: total / p50 / p99 (ms) | 49.9 / 0.27 / 0.54 | 304.9 / 1.48 / 7.26 | 85.8 / 0.42 / 1.34 |
1052
+ | `WebAssembly.validate` in `jit_compile`, total (ms) | 48.1 | 118.7 | 100.3 |
1053
+ | instantiate: total / p50 / p99 (ms) | 10.5 / 0.055 / 0.13 | 6.7 / 0.04 / 0.08 | 16.7 / 0.08 / 0.14 |
1054
+ | time inside `jit_sync` on all threads (ms) | 96.9 | 330.2 | 124.9 |
1055
+ | modules per second of boot | 30 | 28 | 29 |
1056
+ | table size at the end | 9,082 (= 155 + live slots) | 9,090 | 9,088 |
1057
+ | shared wasm memory / module log | 640 MiB / 26.5 MB | 640 MiB / 26.5 MB | 640 MiB / 26.5 MB |
1058
+
1059
+ The per-module compile is cheap because these engines compile lazily or in tiers: V8
1060
+ validates and compiles each function with Liftoff on first call. The host side of the
1061
+ door costs 0.1–0.3 s of a 6.6–7.1 s boot. Nearly all of the rest is guest execution,
1062
+ the same as native, so the remaining gap to native JIT is engine code quality, not the
1063
+ door. Native wasmtime/Cranelift reports `compile_ms=2999` for the same 194 modules:
1064
+ it compiles everything eagerly, with an optimizing compiler, on its compile thread.
1065
+
1066
+ The shared memory size is the same as in interp mode (640 MiB). JS heap and engine
1067
+ code memory were not measured (there is no cross-browser API for them from a page).
1068
+
1069
+ ### What each step bought
1070
+
1071
+ | Step | Effect |
1072
+ |---|---|
1073
+ | 1. The door as specified (compile in `jit_compile` and again in each syncing thread, chunked table growth) | Proof boot passed in Chromium at 6.84 s (6.4x). Spawned threads trapped `null function` in `threads` (see problem 1). |
1074
+ | 2. Grow each table exactly to its installed slots | Fixed the trap. No speed change. |
1075
+ | 3. `jit_compile` validates instead of compiling | Chromium, one boot each: compile 6,551 ms, validate 6,550 ms, parse-only 6,599 ms. No speed change. It halves the compiled copies per module, which matters for Firefox's code limit (problem 4). Kept as the default. |
1076
+ | `WMD_NO_KNOWN=1` (44,800 `jit_known` calls avoided) | 6,518 ms vs 6,551: noise. Not set. |
1077
+ | `WMD_JIT_WORKERS=0` (the vCPU compiles) | 6,407 ms: noise at one run. Kept the guest default. |
1078
+ | `--no-tail` | 6,326 ms: noise at one run. Kept the guest default (tail calls). |
1079
+
1080
+ The 5x target was met at step 1, so no batching, async compile or cross-run caching was
1081
+ needed. A content-hash cache is in place anyway: identical bytes are logged once and
1082
+ compiled once per Worker (765 hits in the `--jit-batch 1` boot).
1083
+
1084
+ ### Tests (all run; output in the P-B2 report)
1085
+
1086
+ | Test | Chromium | Firefox | WebKit |
1087
+ |---|---|---|---|
1088
+ | unit: slot list, sync, flush, dedup, exact table size, 3,000 modules, mutation switch, 8 real Workers with and without `jit_flush` under load (`bun test test/unit`, 46 tests) | pass (bun/JSC) | | |
1089
+ | P-B1 boots (`boot.spec.js`) with the new output path | pass | pass | pass |
1090
+ | P-B1 adversarial (reload mid-boot, during download, two tabs, stalled tab) | pass | – | – |
1091
+ | synthetic images from `runner/tests/vm.rs`: hot loop JITs; dispatch survives IC IALLU; 2 vCPUs, 32 flushes while one chains | pass | pass | pass |
1092
+ | user-mode sha256 (output, `interp_fallbacks=0`, jit > 1M, chain > 1.5); threads x5 | pass | pass | pass |
1093
+ | self-modifying code: `smc.c` and `smcloop` x2 inside the Linux guest, jit and interp | pass | pass | pass |
1094
+ | **P-B2 proof boot: boot gate + JIT gate + `PROOF-DONE` ≤ P-B1/5** | pass | pass | pass |
1095
+ | thousands of modules: proof boot with `--jit-batch 1` (8,677 / 8,662 / 8,716 modules) | pass, 7.9 s | pass, 12.2 s | pass, 12.8 s |
1096
+ | mutation `brk=load` (`jit_load` → -1) fails the JIT gate | fails it: dispatch_calls=0, chain_len=0 | same | same |
1097
+ | mutation `brk=all` (both → -1) fails the JIT gate | fails it: no exit within P-B1's time | same | same |
1098
+ | stress: 8 Workers x 300 modules, no flush | pass | **fail: out of memory** | pass |
1099
+ | stress: `jit_flush` every 25 loads under load, then flush + load; tables identical, nothing stale | pass | pass | pass |
1100
+ | stress: 4 Workers x 1,500 modules | pass | **fail: out of memory** | pass |
1101
+
1102
+ The JIT gate (`jitGate` in `test/e2e/jit.spec.js`) requires all of these:
1103
+
1104
+ - at least one `jit_load`, no door failures, at least 2 modules, instantiations and
1105
+ table sets;
1106
+ - the guest's own `--stats`: `jit_loads > 0`, `compiled > 0`, more blocks run compiled
1107
+ than interpreted, `dispatch_calls > 0` and `chain_len > 1.5`;
1108
+ - for the proof boot, `PROOF-DONE` ≤ P-B1/5 for that browser.
1109
+
1110
+ ### Problems found, and their causes
1111
+
1112
+ 1. **`null function` trap in a spawned guest thread** (user-mode `threads` in JIT mode).
1113
+ The guest's chaining dispatcher reads `idx >= table.size` as "not synced yet" and
1114
+ returns `IR_EXIT_SYNC_MISS` so the vCPU calls `jit_sync` (`guest/vm/src/wasm_emit.c`
1115
+ around line 5716). The first version grew tables in chunks, which left null entries
1116
+ below `table.size`. The fix grows exactly, as `jit.rs` does, and
1117
+ `test/unit/jit.test.js` now checks it.
1118
+ 2. **`smc.c` is not a user-mode test.** It printed `SMC STALE` in the browser in user
1119
+ mode, and native `wmd-run` prints the same for it in user mode, in interp *and* JIT.
1120
+ The store check (D-159) is in the machine-mode MMU. The test now runs `smc.c` as a
1121
+ process inside the arm64 Linux guest: a newc archive is appended to the proof
1122
+ initramfs at boot (`page/smc-initrd.js`). Next to it runs `smcloop`, a hand-assembled
1123
+ static ELF that calls its function 50,000 times, so that function is a compiled
1124
+ wasm function when it is overwritten, then 50,000 more times. Native
1125
+ `wmd-run --mode jit` passes the same initramfs.
1126
+ 3. **The page froze when the guest flooded stderr.** With both JIT imports broken,
1127
+ vm.wasm retries compiles in a tight loop and dumps each failed block's IR, about
1128
+ 127,000 dumps (25 MB) in 3 s. One `postMessage` per `fd_write` drowned the page's
1129
+ event loop, and `page.evaluate` stopped answering. Output now goes through
1130
+ SharedArrayBuffer rings (`src/host/stdio.js`), and the page keeps only the tail of
1131
+ each stream.
1132
+ 4. **Firefox runs out of wasm code memory at roughly 16,000–24,000 compiled modules
1133
+ per process.** `new WebAssembly.Module` throws `InternalError: out of memory`.
1134
+ Single-Worker probes in Firefox:
1135
+ - 8,000 modules: pass;
1136
+ - 2 Workers x 8,000 compiled copies: pass;
1137
+ - 4 Workers x 6,000: fail.
1138
+ Each Worker compiles its own copy, because a compiled Module can only be handed
1139
+ between Workers through `postMessage`, which a busy vCPU never reads. Each copy
1140
+ costs at least one code allocation. The real guest stays far below the limit: the
1141
+ proof boot has about 200 modules, and `--jit-batch 1` about 8,700 copies on the one
1142
+ vCPU, which passes. A many-vCPU guest with thousands of modules per vCPU would hit
1143
+ it in Firefox. Chromium and WebKit pass the same stress tests. Step 3 halves the
1144
+ copies. **Superseded:** the adversarial review measured the threshold at 2 vCPUs with
1145
+ `--jit-batch 1`, and the fix shares one compiled Module per process. See F1 below.
1146
+ 5. **Firefox and WebKit spend more in sync with very many tiny modules.**
1147
+ `--jit-batch 1` costs 4.3 s (Firefox) and 5.7 s (WebKit) inside `jit_sync`, against
1148
+ 1.7 s and 1.1 s of compile time. A microbenchmark (3,000 modules, no guest) puts the
1149
+ door's own bookkeeping at 0.03–0.09 ms per module, and `table.grow` is not the cost.
1150
+ The rest is unprofiled; GC of about 8,000 Modules and Instances is the likely cause.
1151
+ It does not affect the default batching (about 200 modules).
1152
+
1153
+ ### P-B2 adversarial review: findings and fixes
1154
+
1155
+ The independent review is `test/b2-adversarial-results.txt` (commit `e4a3dfe5`), with its
1156
+ tests in `test/e2e/b2-adversarial.spec.js` and `test/unit/b2-adversarial.test.js`. None
1157
+ of its tests was changed. Every fix has a test that fails without it.
1158
+
1159
+ | Finding | Status | Where |
1160
+ |---|---|---|
1161
+ | F1 Firefox runs out of code memory (2–4 vCPUs, `--jit-batch 1`, exit 4) | **fixed**: one compiled Module per process, shared through a hub Worker; resource failures no longer trap in the shared path | `src/host/jit.js`, `src/host/jit-hub.js`, `src/worker.js`, `src/boot.js` |
1162
+ | F2 "jit_flush under load" flaky (3/51) | **fixed**: a flush now bumps the generation before it rewinds the slot count; 100/100 runs pass | `src/host/jit.js` `flush()` |
1163
+ | F3 no guest run calls `jit_flush` | **documented** (below); the no-clear mutation is now a door switch, `brk=noclear`, and the unit and browser door tests fail on it | `src/host/jit.js` `clearTable()`, `test/unit/b2-fixes.test.js`, `test/e2e/jit-stress.spec.js` |
1164
+ | F4 console output silently dropped after a long stall | **fixed**: a marker line in the delivered text, counted; rings configurable, 32 MiB per fd for boots; the boot gate fails on a marker | `src/host/stdio.js`, `src/boot.js`, `page/main.js`, `test/e2e/helpers.js` |
1165
+ | F5 memory (Firefox 2x Chromium, WebKit RSS growth) | improved by F1 for Firefox (8 vCPUs: 2,866 → 1,917 MiB peak). The WebKit growth was not attributed | – |
1166
+ | F6 native comparison range | not re-measured | – |
1167
+ | F7 failure semantics | `jit_load` after a resource failure now refuses new modules (-1) instead of failing later syncs; the rest is unchanged | `src/host/jit.js` |
1168
+
1169
+ #### F1: one compiled copy per process, not per Worker
1170
+
1171
+ **Mechanism.** Before, every vCPU Worker compiled its own copy of every module and kept
1172
+ it forever. Firefox counts every copy against one per-process wasm code limit, so the
1173
+ total ran out at about 16,200 copies. When the compile threw inside `jit_sync`, the
1174
+ thread trapped and the machine exited 4.
1175
+
1176
+ **What the guest can and cannot take.** A host cannot flush the door on its own to make
1177
+ room. The guest keeps the table index of every compiled block
1178
+ (`guest/vm/src/cache.c` `cp_publish`: `ir->jit_fn = base + i + 1`). Only
1179
+ `vm_cache_reset` gives them up, and it calls `jit_flush` itself (`cache.c:3545-3548`;
1180
+ `include/vm/cache.h:223-230`: "the full flush retires its compiled blocks and their
1181
+ table indices stay live, so nothing may drop the modules there"). What the guest does
1182
+ support is `jit_load` / `jit_compile` returning -1. `cp_publish` then retries each block
1183
+ of the batch in its own module. If that also fails, it marks the block `VM_BLK_INTERP`
1184
+ (the interpreter runs it from then on), counts `interp_fallbacks` and prints the block's
1185
+ IR once (`JITFAIL`). A failure inside `jit_sync` has no such path: the index is already
1186
+ published, and a table entry below `table.size` that is null traps the dispatcher.
1187
+
1188
+ **Sharing is feasible, and it is the fix.** A `WebAssembly.Module` posted to another
1189
+ Worker shares its compiled code, in Firefox too. The probe (`test/features/share-probe*`,
1190
+ `WMD_WEB_SHAREPROBE=1 bunx playwright test b2-share-probe`) gave these results in
1191
+ Firefox 155:
1192
+
1193
+ | probe (block-shaped modules, an instance of each kept in each Worker's table) | result | browser RSS peak |
1194
+ |---|---|---|
1195
+ | 4 Workers each compile their own 6,000 | out of memory at 4,060 + 4,131 + 4,115 + 4,036 = **16,342** copies | 1,853 MiB |
1196
+ | 2 Workers each compile their own 9,000 | out of memory at 8,176 + 8,169 = 16,345 | 1,943 MiB |
1197
+ | 1 Worker compiles 6,000, posts them, 3 others instantiate them | all 24,000 instances | 1,393 MiB |
1198
+ | the same with 8 Workers | all 48,000 instances | 1,462 MiB |
1199
+ | 1 Worker compiles 20,000, 1 (or 3) others instantiate | the compile fails at 16,345 / 16,343 distinct modules. Every received Module instantiates | 1,862 / 1,947 MiB |
1200
+
1201
+ So the limit counts compiled modules in the process, about 16,340, and instances of a
1202
+ shared Module cost no code space. The obstacle was delivery. A vCPU Worker never
1203
+ returns to its event loop, so it cannot receive a message. JSPI solves that: Chromium
1204
+ 153, Firefox 155 and WebKit 26.6 (Playwright) all have `WebAssembly.Suspending` and
1205
+ `WebAssembly.promising`, in Workers too. The design:
1206
+
1207
+ - **Compile once.** The thread that calls `jit_load` or `jit_compile` compiles the
1208
+ module. With sharing on, `jit_compile` compiles instead of only validating. If that
1209
+ compile fails, the import returns -1, which the guest supports. The Worker then posts
1210
+ the Module to the **hub** (`src/host/jit-hub.js`), a Worker that runs no guest code.
1211
+ The hub forwards it to every other Worker's `MessagePort` and keeps the current
1212
+ generation's Modules for Workers that start later. It forwards flushes, and on a flush
1213
+ it drops the older Modules.
1214
+ - **Wait instead of compiling.** In `jit_sync` (and in the sync inside `jit_load` and
1215
+ `jit_flush`), a module this Worker has not received makes the import return a Promise.
1216
+ The worker wraps those three imports in `WebAssembly.Suspending` and calls `_start` /
1217
+ `wasi_thread_start` through `WebAssembly.promising`. The vCPU's wasm stack suspends,
1218
+ the Worker's event loop delivers the queued Modules, and the sync resumes where it
1219
+ stopped. A sync that finds everything returns at once, without suspending. A new
1220
+ thread syncs with `await door.syncAsync()`.
1221
+ - **Fallbacks.** A Module that has not arrived after 60 s (`jit.shareWaitMs`) is
1222
+ compiled locally. That happens only if the hub stops answering, and it never happened
1223
+ in any guest run here (`shareTimeouts=0`). If that local copy fails, the sync waits
1224
+ for the shared Module again instead of trapping. A Module is posted to the hub
1225
+ *before* its record is published, so a post that fails (Firefox past its limit failed
1226
+ to serialize one) returns -1 with nothing published. If the hub cannot forward a
1227
+ Module to some Worker, it sends `{t:'lost'}`, and that Worker compiles a copy at once. A resource failure (anything but
1228
+ `CompileError` / `LinkError`) is retried once. If the retry fails too, the door sets
1229
+ `H_FULL`, and from then on `jit_load` and `jit_compile` refuse new modules with -1, so
1230
+ the guest interprets new blocks and keeps running. The browser test
1231
+ "20,000 distinct shared modules" (4 Workers) drives Firefox past its limit to check
1232
+ this. Firefox: 16,339 modules, then 3,661 loads refused, every table consistent, no
1233
+ trap, 134 s. Chromium (1.6 s) and WebKit (25 s) compiled all 20,000.
1234
+ - **Release on flush (F1a).** The per-Worker `modules` map now holds only the current
1235
+ generation. The first sync (or received Module) of a newer generation drops it, so
1236
+ after a flush nothing in a Worker references an old Module. The old unit test "is not
1237
+ recompiled by a thread that has it" asserted the opposite. It is now "a flush releases
1238
+ the Worker's compiled Modules", with a second test covering every Worker.
1239
+ - **Default.** `bootWmd({jit: {share: 'auto'}})` turns sharing on wherever JSPI exists
1240
+ (`jitShareDefault`). Pass `share: false` (page `?share=0`) for the old per-Worker
1241
+ copies. Without JSPI (for example a shipping Safari that lacks it), the door compiles
1242
+ per Worker as before. There the limit and the trap on a failed sync remain. Chromium
1243
+ and WebKit never hit a limit in these tests.
1244
+
1245
+ **Firefox, before and after** (`test/e2e/b2-fixes.spec.js`, proof boot, one run each,
1246
+ canary 601–627 ms around every run; "copies" = modules compiled anywhere in the
1247
+ process):
1248
+
1249
+ | proof boot | before: `share=0` | after: sharing (default) |
1250
+ |---|---|---|
1251
+ | `--cpus 2 --jit-batch 1` | **exit 4**, out of memory at 16,150 copies (8,495 modules), RSS 2,415 MiB | pass, PROOF-DONE 8,968 ms, 9,340 modules / **8,924 copies**, RSS 2,395 MiB |
1252
+ | `--cpus 3 --jit-batch 1` | **exit 4** at 16,149 copies (5,981 modules), RSS 2,536 MiB | pass, 9,158 ms, 9,509 / 9,072 copies, RSS 2,451 MiB |
1253
+ | `--cpus 4 --jit-batch 1` | **exit 4** at 16,214 copies (4,625 modules), RSS 2,367 MiB | pass, 9,404 ms, 9,595 / 9,097 copies, RSS 2,518 MiB |
1254
+ | `--cpus 1`, default batching | PROOF-DONE 7,943 ms, 206 copies, `jit_sync` 562 ms, RSS 1,768 MiB | 6,234 ms, 199 copies, 17 ms, 1,743 MiB |
1255
+ | `--cpus 4` | 6,770 ms, 859 copies, 1,938 ms, 1,859 MiB | 5,769 ms, 216 copies, 58 ms, 1,905 MiB |
1256
+ | `--cpus 8` | 7,834 ms, 1,883 copies, 5,809 ms, **2,866 MiB** | 5,908 ms, **236 copies**, 124 ms, **1,917 MiB** |
1257
+
1258
+ With `share=0`, the retry did not help (`compileRetries` 1–3, still exit 4). That is the
1259
+ residual of the per-Worker mode.
1260
+
1261
+ **Chromium and WebKit, the same boots** (default batching; `share=0` → sharing):
1262
+
1263
+ | | Chromium copies | Chromium PROOF-DONE ms | Chromium `jit_sync` ms | Chromium RSS MiB | WebKit copies | WebKit PROOF-DONE ms | WebKit `jit_sync` ms | WebKit RSS MiB |
1264
+ |---|---|---|---|---|---|---|---|---|
1265
+ | `--cpus 1` | 200 → 197 | 5,955 → 5,840 | 83 → 14 | 1,100 → 1,071 | 201 → 197 | 5,803 → 5,959 | 96 → 24 | 1,265 → 1,221 |
1266
+ | `--cpus 4` | 875 → 215 | 5,455 → 5,316 | 349 → 48 | 1,146 → 1,162 | 855 → 212 | 6,039 → 5,540 | 526 → 92 | 1,658 → 1,127 |
1267
+ | `--cpus 8` | 1,867 → 233 | 5,728 → 5,495 | 836 → 97 | 1,248 → 1,216 | 1,859 → 232 | 6,711 → 5,846 | 1,370 → 194 | 1,411 → 1,297 |
1268
+
1269
+ The time a vCPU spends waiting for a Module is counted separately (`shareWaitMs`): 31 ms
1270
+ (Firefox), 16 ms (Chromium) and 106 ms (WebKit) at 1 vCPU, and 0.8–1.4 s across all
1271
+ threads in the Firefox `--jit-batch 1` boots. Each number is from one run.
1272
+
1273
+ **Hub throughput (a limit of the fix).** In Firefox, a flood of tiny modules from 8
1274
+ loaders at once overruns the hub. 8 Workers x 1,500 modules took 17–67 s, and once over
1275
+ 170 s, with syncs waiting more than 10 s; Chromium did the same in 1.5 s. At 8 x 1,000
1276
+ it is 3.7 s. The guest boots are far from this: the `--jit-batch 1` boots at 4 vCPUs
1277
+ waited 1.4 s in total, over 36,000 waits. The browser stress test now uses 8 x 1,000.
1278
+ Batching the forwards in the hub was not tried.
1279
+
1280
+ **Suites after the fixes** (one run each per browser, under the lock, canary 595–633 ms):
1281
+
1282
+ | | Chromium | Firefox | WebKit |
1283
+ |---|---|---|---|
1284
+ | `bun test test/unit` (79 tests) | 79 pass (bun) | | |
1285
+ | `jit.spec boot.spec jit-stress` | 23/23 | 20/23 (below) | 23/23 |
1286
+ | `b2-fixes` | 5/5 | 5/5 | 5/5 (F4 marker test after the 4 KiB change) |
1287
+ | `b2-adversarial` (`WMD_WEB_B2ADV=1`), 12 tests | 12/12 | 12/12 | 12/12 |
1288
+
1289
+ The three Firefox failures:
1290
+
1291
+ - "no flush: 8 threads x 300" and "4 threads x 1500" are the original **per-Worker**
1292
+ stress tests. Their pages do not share Modules, so Firefox still runs out of code
1293
+ memory at about 16,300 copies, exactly as before the fixes. They are kept unchanged
1294
+ as the evidence for the limit. Their shared twins pass.
1295
+ - "shared Modules: 8 threads x 1500" timed out on that run (hub throughput, above). It
1296
+ is now 8 x 1000, which passed in all three browsers (Firefox 3.7 s).
1297
+
1298
+ The 20,000-module refusal test timed out in Firefox once at 8 x 3,000 (over 290 s). At
1299
+ 4 x 5,000 it passed in 134 s.
1300
+
1301
+ #### F2: the flaky flush test
1302
+
1303
+ `flush()` used to store `H_NSLOTS = 0` before bumping `H_GEN`. A reader that does not
1304
+ take the lock, such as the stress check in `page/jit-stress-worker.js`, could then see 0
1305
+ live slots in the old generation. The door's own readers were not affected (`sync`
1306
+ does not read `H_NSLOTS`). The order is now `H_MOD_START`, `H_GEN`, `H_NSLOTS`: a sync
1307
+ still sees where a new generation starts, and a rewound slot count always comes with
1308
+ the new generation. `test/unit/b2-fixes.test.js` records the header writes of a flush,
1309
+ which fails deterministically with the old order. It also races the stress check
1310
+ against a Worker doing 20,000 flushes; that check caught the old order in 1 of 3 runs.
1311
+ "jit_flush under load", 100 runs in a row: `jit_flush under load: 100 pass, 0 fail of
1312
+ 100`.
1313
+
1314
+ #### F3: no guest run calls `jit_flush`
1315
+
1316
+ The guest calls `jit_flush` only from `vm_cache_reset`
1317
+ (`guest/vm/src/cache.c:3545-3548`), and `guest/vm/src` and `guest/vm/include` contain no
1318
+ call to `vm_cache_reset` (only its definition, the declaration at
1319
+ `include/vm/cache.h:510` and comments). The guest's full flush, `vm_cache_flush`, keeps
1320
+ the table indices on purpose (D-166, `include/vm/cache.h:223-230`). `jit_wasm.c:64-70`
1321
+ says the same: "only vm_cache_reset does, and nothing in a machine run calls that, so
1322
+ this is not reached in production." The tests "dispatch survives an IC IALLU full
1323
+ flush" and "2 vCPUs, 32 flushes" exercise the guest's own block-cache flush, not the
1324
+ door's. The door's flush is therefore tested only at the door level:
1325
+
1326
+ - unit: `jit.test.js` (`jit_flush` describe, and 8 Workers flushing under load),
1327
+ `b2-adversarial.test.js` (the storm), `b2-fixes.test.js`, `jit-share.test.js` (flush
1328
+ with sharing);
1329
+ - browser: `jit-stress.spec.js` (flush under load, with and without sharing) and the
1330
+ storm in `b2-adversarial.spec.js`;
1331
+ - the adversary's mutation, a flush that does not clear the tables, is the door switch
1332
+ `brk: 'noclear'`. Its tests assert that the checks *fail* under it: the unit test
1333
+ reports "still set after the flush", and the 8-Worker stress reports "table[158] set
1334
+ above the 3 live slots" under bun and "table[161] set above the 6 live slots" in Firefox.
1335
+
1336
+ #### F4: dropped output is visible
1337
+
1338
+ - The writer still never blocks; a guest must not stall on a page that does not drain.
1339
+ - `stdioText()` (used by `bootWmd`) inserts `\n[wmd-web: N bytes of stdout were
1340
+ dropped]\n` into the text where the reader skipped bytes. It ends any UTF-8 sequence
1341
+ the skip cut, and counts the markers (`timings.outputDropMarkers`).
1342
+ - Ring size: `createStdio(ringBytes)` (a power of two, 4 KiB–1 GiB, stored in the
1343
+ header). `bootWmd` uses **32 MiB per fd** (`BOOT_RING`, override with
1344
+ `stdioRingBytes`). The guest console runs at about 0.33 MB/s in JIT mode, so 32 MiB
1345
+ covers about 100 s of a page that serves no events. That is more than the 25 s stall
1346
+ that lost 5.4 MB in the review, and more than Chromium's intensive throttling of
1347
+ hidden tabs (timers once a minute). It costs 64 MiB of SharedArrayBuffer, which is
1348
+ touched only as it fills, next to the guest's 640 MiB memory. `RING` (4 MiB) stays
1349
+ the library default, so the review's 30 MB ring flood test still overruns the ring.
1350
+ - `page/main.js` keeps about 8 MB of each stream (P-B1's head-and-tail cut). When it
1351
+ cuts, it writes the same kind of marker between the head and the tail it keeps.
1352
+ - `bootGate` (`test/e2e/helpers.js`) fails with `OUTPUT DROPPED (a gate line may be
1353
+ missing): ...` when a marker is in the log.
1354
+ - Tests: `stdio.test.js` (marker, count, position, UTF-8, ring size),
1355
+ `b2-fixes.test.js` (gate), and `b2-fixes.spec.js` in the browser. A 4 KiB ring with a
1356
+ 6 s stall must produce markers whose byte counts add up to `outputDropped`, and the
1357
+ gate must fail. (A 256 KiB ring with a 6 s stall overran it in Chromium and Firefox but
1358
+ not in WebKit. There the guest stalled together with the page's main thread, and the
1359
+ kernel printed RCU stall warnings.) The review's own 1.5M-line / 25 s stall case must
1360
+ lose nothing with the default ring. Results: Chromium 1 marker / 1,984,818 bytes,
1361
+ Firefox 124 markers / 1,243,495 bytes, WebKit 237 markers / 997,626 bytes, the counts
1362
+ equal in each. The 25 s stall lost 0 of 1,500,000 lines in all three.
1363
+
1364
+ ### Reproducing the native references
1365
+
1366
+ ```sh
1367
+ cd ~/keg-hub/external/apps/wmd # read-only use of the main checkout's guest files
1368
+ # (under the boot lock)
1369
+ ~/.wmd/bin/wmd-run --wmd-stats --wmd-jit-cache off --dir guest/linux/dist ~/.wmd/bin/guest/arm64/vm.wasm -- \
1370
+ --machine --arch a64 --board arm64 --mode jit --cpus 1 --ram 256 \
1371
+ --kernel guest/linux/dist/arm64/Image --initrd guest/linux/dist/arm64/initramfs.cpio.gz \
1372
+ --append "console=ttyAMA0 earlycon=pl011,0x9000000 rdinit=/init loglevel=8 wmd.proof wmd.exit" --stats
1373
+ ```
1374
+
1375
+ The guest's `--stats` line for that run: `compiled=8782 jit_loads=194 chain_len=99.2 compile_ms=2999`, `wall: 6.526s`.
1376
+
1377
+ ## P-B11: the wmn agent in the browser's guest
1378
+
1379
+ The proof: `hello_reply` and the P-K2 Python block checks pass with the wmn agent running inside
1380
+ wmd's guest in the browser (`test/e2e/wmn.spec.js`, `page/wmn.js`).
1381
+
1382
+ ### What runs
1383
+
1384
+ 1. **Image.** `fixtures-gen image -ref docker.io/library/python:3.12-slim -out test/fixtures/images/python`
1385
+ reads wmd's image store (`~/.wmd/data/images`, read only), picks the linux/arm64 manifest and turns
1386
+ each layer into a pack with wmd's `union.PackLayer` over the packs below it, as the engine's rootfs
1387
+ cache does (`internal/container/chaincache.go`). python:3.12-slim is 4 packs (101 MB, 3.9 MB,
1388
+ 40 MB, 531 B), node:22-slim is 5 (98 MB, 9.9 KB, 143 MB, 7.2 MB, 657 B). The JS overlay stacks them
1389
+ top first under `rootfs/`. `test/unit/image-layers.test.js` compares the merged tree with what
1390
+ `internal/union` shows for the same packs (`fixtures-gen tree`). All 5,574 paths are equal, with
1391
+ type and size.
1392
+ 2. **Bundle.** `containerBundle()` (`src/host/ctl.js`) writes the engine's spec (`internal/engine/ocispec.go`
1393
+ `buildSpec`: docker's capabilities, masked and read-only paths, `RLIMIT_NOFILE`, `oomScoreAdj` 0, the
1394
+ six default mounts, `/etc/{hosts,resolv.conf,hostname}` bound from `bundle/etc`), plus
1395
+ `imageconfig.json`. Each mount is a share `vol<i>` beside `bundle/` and `rootfs/`, as the engine's
1396
+ `layout()` does. The agent (`wmn-agent-linux-arm64` from the wmn repo's `agent/build.sh`, served at
1397
+ `/files/wmn-agent/`) is the file `vol0/wmn-agent` (mode 0755), bound read-only at `/.wmn/agent`.
1398
+ 3. **Boot.** This uses the engine's arm64 command line (`run.go` `guestCmdline`, without `wmd.warmup`)
1399
+ with `wmd.control=vport`: `--p9 564 --p9-tag wmdroot --control 8`, 2 vCPUs, 1 GiB, `--mode jit`.
1400
+ Services are the 9P server over the packs plus the bundle files, and `{port: 8, kind: 'control', yamux: true}`.
1401
+ 4. **Control session (ROADMAP 2.2b).** When the init opens `/dev/vport2p1`, the channel service starts
1402
+ a yamux client on it (`ControlSession`). The page's `vm.control` (`ControlClient`) opens streams
1403
+ through it:
1404
+ - `run` comes first, as `driveMainProcess` sends it: `{"op":"run","id":"main","args":["/.wmn/agent","init"],"env":...,"now":...,"seed":<32 bytes of crypto.getRandomValues, base64>,"p9tag":"wmdroot","p9opts":"trans=virtio,version=9p2000.L,msize=262144,cache=mmap"}`.
1405
+ - Then `exec` of `/.wmn/agent serve` (id `wmn-serve`, stdin open). The init holds it until the
1406
+ container runs.
1407
+ - The wmn protocol runs as JSON lines in stdin/stdout frames: `hello`, `languages_request`, then
1408
+ `page/k2.js` `runChecks`.
1409
+ - At the end the agent's stdin is closed, then `kill {id: main, sig: SIGTERM}`. The agent's init
1410
+ exits, the Go init writes `bundle/exitcode` (read back) and powers off.
1411
+ 5. **Checks.** `page/k2.js` is k2.ts's python/node/ruby cases and its 13 checks with their bounds
1412
+ unchanged (copied: k2.ts imports Bun-only modules, so a page cannot load it), and agent-conn.ts.
1413
+ One harness-only change: a block whose agent stream ended fails at once instead of at its
1414
+ 120 s timeout.
1415
+
1416
+ ### The control client
1417
+
1418
+ - **Wire format.** It speaks hashicorp/yamux v0.1.2's wire format: 12-byte headers; odd stream ids
1419
+ for the client; the SYN window update on open; 256 KiB windows each way; a window update once half
1420
+ the window is consumed; FIN after the queued data; RST; Go Away on a protocol error; pings answered
1421
+ with the same value. A keepalive ping goes out every 5 s, as the engine does; a missing answer is
1422
+ only counted. Streams the guest opens (op `edge`) are refused with RST.
1423
+ - **Where it runs.** The yamux client runs in the channel service Worker, not on the page. The
1424
+ guest's yamux server pings every 30 s and drops the session when an answer takes more than 10 s;
1425
+ the page's main thread can stall that long (P-B1's stalled-tab check), the service does not.
1426
+ `ctl.test.js` shows the session dropping when the host stops answering.
1427
+ - **Backpressure.** The page acknowledges stream bytes after it has delivered them, and only then does
1428
+ the service give the window back. A page that stops reading stops the guest at 256 KiB.
1429
+ - **Protocol.** It matches `ctlproto.go` and the init: Go's `omitempty` for the request line, frames
1430
+ `id|len BE|payload` (at most 10 MiB), the exit frame with the synced flag, `close_stdin` at once when
1431
+ there is no stdin, the attach reply line before the replayed frames, and one JSON line for the other
1432
+ ops. A stream that ends without an exit frame gives 125 with the reason, as `runProcessExit` does.
1433
+ - **Unit tests.** `test/unit/ctl.test.js` has 22 tests. 10 drive the browser's path
1434
+ (`ControlClient` -> `ChanHost.controlInput` -> yamux -> the control channel's rings) against
1435
+ **hashicorp/yamux itself**: `test/ctl-guest` is a Go server with the init's stream protocol, running
1436
+ local processes. They cover:
1437
+ - the request as Go parses it;
1438
+ - stdin, stdout and stderr apart, exit codes, the synced flag;
1439
+ - 5 MiB of output in order and 3 MiB of stdin (cksum equal);
1440
+ - 16 processes at once;
1441
+ - kill, procs and attach;
1442
+ - a page that does not ack stopping the guest at one window;
1443
+ - 50 ms guest pings answered, and the session lost when the host stops answering;
1444
+ - the guest going away: every stream gets 125 and later opens are refused.
1445
+
1446
+ The yamux unit tests cover window accounting, split frames and the protocol errors.
1447
+
1448
+ ### udhcpc
1449
+
1450
+ The run request cannot skip it. The init runs `udhcpc` (up to 15 s with no network, `-t 5 -T 3`)
1451
+ before it opens the control channel, whenever `/sys/class/net/eth0` exists, and the board always has a
1452
+ virtio-net device. The engine does not skip it for `--network none` either: it always passes `--net`,
1453
+ and its netstack answers. In the browser the page passes `initcall_blacklist=virtio_net_driver_init`
1454
+ on the kernel command line (`?nonet=1`, the spec's default). With no eth0 the init skips udhcpc:
1455
+ Chromium python got `hello_reply` at 5.8–6.1 s with it against 21.8 s without it (control session
1456
+ up at 4.4–4.7 s against 20.4 s). P-B10's relay, or a DHCP answer from the net stub, makes it moot. **Update (P-B10):** `page/net.html` boots
1457
+ with eth0 and a net service; udhcpc gets its lease from the network Worker in 0.4-0.5 s.
1458
+
1459
+ ### Numbers (2026-10-05, Apple Silicon, Playwright 1.63, `--mode jit`, 2 vCPUs, 1 GiB, `nonet=1`)
1460
+
1461
+ ms from the `bootWmd` call (download included), one run each. `run2` and `run6` are the runs below;
1462
+ the canary was 640–656 ms before each Chromium and Firefox run and 1,358 ms before the WebKit
1463
+ part of run6. The host was shared with other sessions throughout: load average 24–32 on 10 cores.
1464
+
1465
+ | python:3.12-slim | Chromium (run2 / run6) | Firefox (run3) | WebKit (run6) |
1466
+ |---|---|---|---|
1467
+ | control session up | 4,382 / 4,653 | 9,167 | 6,840 |
1468
+ | `hello_reply` | **5,799 / 6,095** | 11,570 | 10,021 |
1469
+ | `languages_reply` | 7,234 / 7,881 | 17,437 | 13,330 |
1470
+ | `session_start` python (cold) | 8,068 / 8,197 | 18,110 | 12,168 |
1471
+ | define / use block | 52 / 49, 56 / 63 | 75 / 67 | 86 / 57 |
1472
+ | interrupt a loop | 41 | 229 | 69 |
1473
+ | infinite output, stopped in | 42 (771 KB), 20 (243 KB) | 33 (175 KB) | 44 (154 KB) |
1474
+ | all 13 checks done | 20,782 / 21,577 | 43,867 | 32,133 |
1475
+ | 9P requests | 8,372 | | 8,372 |
1476
+
1477
+ | node:22-slim | Chromium (run5 / run6) | Firefox (run6, throttled 7.8x) | WebKit (run6) |
1478
+ |---|---|---|---|
1479
+ | `hello_reply` | 6,978 / 6,571 | 50,007 | 9,867 |
1480
+ | `session_start` node | 4,262 / 4,568 | 4,448 | 5,229 |
1481
+ | define / use | 321 / 100, 297 / 117 | 243 / 81 | 266 / 144 |
1482
+ | interrupt a loop | 102 / 86 | 114 | 80 |
1483
+ | checks | 13/13, 13/13 | 12/13 | 13/13 |
1484
+
1485
+ Native wmd for comparison, from P-K2 (native wmd under the daemon, with its edge fast path):
1486
+ python session start 7.0 s, node 4.0 s, interrupt 8 ms (python) and 24 ms (node). So the browser's
1487
+ session start is about the same. The browser's checks show no network wait, because nothing in them
1488
+ needs the network.
1489
+
1490
+ ### Results, every run (all under the boot lock, `PORT=8798`)
1491
+
1492
+ | run | Chromium | Firefox | WebKit |
1493
+ |---|---|---|---|
1494
+ | run2 python | 13/13 | | |
1495
+ | run3 python / node (256 MiB slack) | 13/13 / 5/7 (trap) | 13/13 / 5/7 (trap) | 0 (`not a pack`) / 5/7 (trap) |
1496
+ | run3 python with udhcpc | 13/13 | | |
1497
+ | run4 (canary 2.5 s: contended) | node 5/7 (trap) | | python 11/13, 13/13, 13/13 |
1498
+ | run5 node, 1 GiB slack | 13/13 | | |
1499
+ | run6 python / node | 13/13 / 13/13 | 0/1 / 12/13 (throttled 7.4-7.8x) | 13/13 / 13/13 |
1500
+ | run7 python / node, on wmd main (ee1886e8) | 13/13 / 13/13 | 13/13 / **13/13** | 13/13 / 13/13 |
1501
+ | run8 python / node, the finished branch (2026-10-06) | 13/13 / 13/13 | 13/13 / 13/13 | 13/13 / 13/13 (and 13/13 / 13/13 in a first run at canary 6.2 s) |
1502
+
1503
+ ### Problems found, and their causes
1504
+
1505
+ 1. **node trapped the machine: vm.wasm's heap hit the memory maximum** (fixed for the page).
1506
+ - What happened: in all three browsers, node's interrupt check never got a reply, and the machine
1507
+ ended with exit 4: `RuntimeError: unreachable` at `abort` <- `interp_run_block`.
1508
+ - Why: when it trapped, the shared memory was 1,664 MiB, exactly `memoryMaxPages()`'s maximum
1509
+ (384 MiB vm.wasm + 1 GiB RAM + 256 MiB slack). node's V8 is a JIT inside the emulated guest
1510
+ (460 block modules, 21,263 functions); it pushed the translator's heap, which shares the memory,
1511
+ to the maximum. `interp_run_block`'s `calloc` then returned NULL, and the guest calls `abort()`
1512
+ (`guest/vm/src/interp.c:848`).
1513
+ - Fix: the wmn page passes `maxPages` with 1 GiB of slack (`?slack=`). node then peaks at
1514
+ 1,784–1,799 MiB and passes 13/13 in all three browsers.
1515
+ - Still open: `bootWmd`'s library default (256 MiB) is unchanged for the older suites. The guest
1516
+ aborting on an allocation failure, instead of reporting it, is a guest finding: native `wmd-run`
1517
+ reserves 4 GiB, so it does not see it.
1518
+ 2. **WebKit: `layer 1: not a pack`, once in run3** (cause not found).
1519
+ - What happened: the channel service could not open python's `layer-2.pack`, and the boot ended
1520
+ with -4 before the control session.
1521
+ - What was ruled out:
1522
+ - `wmn-fetch.spec.js` fetched every pack 3 times as files, and 15 more rounds along the boot's
1523
+ own path (fetchShared next to the kernel's download, then one postMessage of the
1524
+ SharedArrayBuffers to a Worker). Every size and both magics were right.
1525
+ - It did not recur in 7 later WebKit boots of python.
1526
+ - Next time: the service now says what arrived (bytes, head, tail, non-zero pages), so a repeat
1527
+ tells a bad transfer from a bad file.
1528
+ 3. **WebKit python: interrupt took 1,319 ms, so the agent ended the session (once in 4; run4).**
1529
+ - Why: the agent's `InterruptGrace` (1 s, `agent/session.go`) SIGKILLs a block that has not
1530
+ stopped by then. The other WebKit runs stopped in 69–182 ms (Chromium 41 ms, native 8 ms).
1531
+ The run's canary was 2.5 s, against 0.64 s on a quiet host.
1532
+ - Effect: under emulation on a loaded host, the 1 s grace is close to the latency of signal
1533
+ delivery plus KeyboardInterrupt.
1534
+ - This is wmn's setting, so it is left for the agent's step (ROADMAP 4.16); the check's bound
1535
+ was not changed.
1536
+ 4. **Firefox (run6), both boots on a host that the throttle detector measured 7.4–7.8x slow.**
1537
+ - **python: the guest dropped the control session.**
1538
+ - Sequence: the init logged `control: accept error: keepalive timeout`, then `session closed
1539
+ before main process ran`, then exit 125 and a power-off. Its yamux server's 10 s ping timeout
1540
+ expired in guest time.
1541
+ - In the same boot a vCPU waited 60 s for a JIT module from the hub. The door logged
1542
+ `module ... did not arrive from the hub in 60000 ms; compiling a copy`; `shareTimeouts` was 1
1543
+ and `shareWaitMs` 61,772.
1544
+ - The service answers pings at once, so the late answer was the guest's side not reading:
1545
+ its vCPU was stalled.
1546
+ - The hub's forwarding is plain postMessage. Why one module took over 60 s to arrive was not
1547
+ found: P-B2 measured up to 1.4 s of waits per boot; this boot ran at up to 7.8x slowdown.
1548
+ - It is a finding for P-B2's sharing (a lost or starved forward), with this log as the evidence.
1549
+ - **node: `infinite output` printed 7,676 bytes in 1.5 s.** The check wants more than 10,000. The
1550
+ guest ran slowly: control session up at 36 s instead of about 6 s.
1551
+ 5. **Harness problems found on the way.**
1552
+ - The first run never booted: the disk was under 75 GiB when the lock was taken, and
1553
+ `node_modules` was missing. The runner now re-checks the disk under the lock.
1554
+ - Playwright empties `test-results/` at the start of every run, so the logs now go to
1555
+ `wmn-results/`.
1556
+ - `FrameReader` was O(n^2) on byte-sized input: a 70 KB frame fed one byte at a time timed out.
1557
+ It now keeps chunks.
1558
+
1559
+ ### What remains
1560
+
1561
+ - **Ruby.** ruby:3.3-slim is not in wmd's store; only ruby:3.3-alpine is. A pull through wmd's image
1562
+ code into a scratch store was not done. The node case passes; ruby is not checked.
1563
+ - **The adversarial set from P-K2 (robustness.ts) in the browser.** Not ported here; that is the
1564
+ reviewer's step.
1565
+ - **Open causes.** The cause of problem 2 and the stalled hub forward in problem 4.
1566
+ - **Defaults and options.** `bootWmd`'s default slack. Packs from OPFS (here they are fetched into
1567
+ memory: 145 MB for python, 250 MB for node). Network (P-B10). Snapshots and `wmd.warmup`.
1568
+ - **Reruns.** No timed comparison runs on a quiet host: the host was loaded throughout (load average
1569
+ 24–32).
1570
+ - **W21 (branch w21-trap, under review).** Not merged here. Its two commits cherry-pick onto this
1571
+ branch with one README conflict (section "Out-of-memory detection with W21's heap reserve"); after
1572
+ it, a real out-of-memory trap is reported only with a W21-built vm.wasm.
1573
+ - **hvc0 at 33-72 KB/s** (section "hvc0 after W17"): resolved by W53 (D-433). It was the test's
1574
+ 2-byte tty writes, not hvc0; the 50 MiB `chan.spec` hvc0 test now passes at 2.1-3.1 MB/s.
1575
+
1576
+ ### Running
1577
+
1578
+ ```sh
1579
+ (cd ~/keg-hub/external/apps/wmn && agent/build.sh) # the agent, into the wmn repo's agent/dist
1580
+ go run ./web/test/fixtures-gen image -ref docker.io/library/python:3.12-slim -out web/test/fixtures/images/python
1581
+ go run ./web/test/fixtures-gen image -ref docker.io/library/node:22-slim -out web/test/fixtures/images/node
1582
+ cd web && bun test test/unit/ctl.test.js test/unit/image-layers.test.js
1583
+ # under the boot lock, one browser per run (about 45 s per image):
1584
+ WMD_WEB_WMN_IMAGES=python,node PORT=8798 bunx playwright test wmn.spec --project=chromium
1585
+ WMD_WEB_WMN_IMAGES=python WMD_WEB_WMN_QUERY= PORT=8798 bunx playwright test wmn.spec --project=chromium # with udhcpc
1586
+ PORT=8798 bunx playwright test wmn-fetch --project=webkit # no boot: the pack transfer probe
1587
+ ```
1588
+
1589
+ ### P-B11 adversarial review: findings and fixes
1590
+
1591
+ The review is `test/adversarial-b11/RESULTS.md` (branch `wmd-web-b11-adv`, 2b7f2e6d, merged). None of its
1592
+ tests were changed. Each fix has a test that fails on the pre-fix code (347f80ed) and passes after.
1593
+
1594
+ | Finding | Status | Fix | Test (before -> after) |
1595
+ |---|---|---|---|
1596
+ | 1 30 concurrent execs trap the machine in Chromium (`calloc` <- `interp_run_block`) | **not here**: W21, debugged separately | - | - |
1597
+ | 2 python driver interrupt race (c09c) | fixed in wmn's phase1 branch | - | - |
1598
+ | 3 `wmn.vm` collected in a hold (WeakRef) | fixed | `page/wmn.js` holds the vm strongly while it runs, drops it when it ends | b11-fixes "the vm handle survives forced garbage collection" (Chromium, 5 CDP `collectGarbage`): before `{vm: false}`, after `{vm: true, up: true}` |
1599
+ | 4 OOM reported as `trap: unreachable`, `wmn.error` null, a second vCPU on garbage IR | fixed | `boot.js` `finish()`: a trap with the memory within 64 MiB of its maximum becomes `out of memory: the emulator used N of its M memory maximum ...` (`timings.memoryMaxBytes`, `timings.trap` keep the raw facts) and every Worker is terminated at once; `worker.js`: every wasi/wmd/thread-spawn import (not `jit_*`, not `proc_exit`) throws once the machine's exit flag is set, so other threads stop at their next host call; `wmn.js` sets `wmn.error` on any end it did not ask for | b11-fixes synthetic `oom_trap.wat` (grows to the maximum, traps; a second thread writes once a millisecond): before, reason `trap: unreachable` and 810-6,601 lines after the trap (C/F/W); after, the out-of-memory reason and 0-2 lines, 3 rounds x 3 browsers. Real guest, node at `slack=256`: before, `trap: unreachable` and `wmn.error` null; after, `wmn.error` "the machine ended: 4 (out of memory: the emulator used 1664 MiB of its 1664 MiB memory maximum ...)", no "vreg out of range" in stderr |
1600
+ | 5 AgentConn quadratic (20 MB line: 16.5 s Firefox) | fixed | `page/k2.js`: each chunk searched once, line pieces joined once | `test/unit/agent-conn.test.js` (old: 5 MB 62 ms, 20 MB 957 ms, fails); b11-fixes in each browser: before 5/20 MB = C 82/1,142, F 767/16,213, W 70/477 ms; after C 3/11, F 15-16/121-152, W 13-16/19-23 ms |
1601
+ | 6 yamux.js buffers a frame before checking the window; unknown-stream data held | fixed | `feed()` parses incrementally: the header in 12 bytes, the window checked when the header arrives (Go Away before any body), data for a gone stream discarded as it comes; a body is kept as its pieces (at most one window) and delivered once whole | the review's A1, A2 (failed before, pass) |
1602
+ | 8 `feed()` quadratic on small pieces | fixed (same change) | - | the review's A3 (10.7 s before -> 76 ms), A3b (79 ms) |
1603
+ | 9 lenient: window credit past 2^32, `consume(NaN)` | fixed | `sendWindow` capped at 0xffffffff (Go's uint32); `consume` ignores non-finite | the review's A11, A13 (failed before, pass) |
1604
+ | 7 no stdin backpressure | fixed | the service reports bytes it sent on (`{ev:'sent'}`, from the stream's queue as the guest's window allows); each page stream has a credit (`STREAM_CREDIT`, 4 MiB + 64 KiB) of bytes posted and not yet sent; `CtlProcess.write()` returns false and sends nothing past it; `writeAsync()` queues in order and resolves when posted; `closeStdin()` waits for queued data; AgentConn sends through `writeAsync` | the review's G6 (service held 62.8 MiB before -> 2.8 MiB); ctl.test "stdin backpressure": 64 MiB through `cksum` equal, the service never held more than 4.00 MiB |
1605
+ | 9 over 256 streams: "reset by the guest" | fixed | the page keeps at most 128 streams open (`MAX_OPEN_STREAMS`, under the guest's AcceptBacklog 256); more wait their turn; a guest RST before any reply names the backlog | ctl.test: 300 at once all answer, never more than 128 open, 172 queued; the review's G1 passes (it failed 1 in ~5 runs before) |
1606
+ | 9 exec errors | fixed | `checkProcessRequest`: a NUL in an argument, env entry or cwd is refused on the page with the reason (the init gave a bare 125); an id in use by a live run/exec of this page is refused; exec exit 1 with crun's "executable file ... not found" -> 127, "... not executable" -> 126 (the init's `crunExecStatus`, which it applies to `run` only); crun's messages lose the internal container name `foo` | ctl.test (4 tests, all fail on the old code) |
1607
+ | 9 "has not opened the control port" after the session ended | fixed | the service remembers the session and why it ended: "the guest's control session N has ended (...) and none is open" | ctl.test "an open after the session ended" |
1608
+
1609
+ Not changed: c03 late output (wmn's known limit, ROADMAP 4.3b); `non-UTF-8 in args` (JSON); a duplicate FIN is
1610
+ still ignored (the review's A6 is INFO, it passes).
1611
+
1612
+ **Unit:** `bun test test/unit test/adversarial-b11/yamux-adv.test.js`: 302 pass, 2 skip, 0 fail (the review's
1613
+ file: 28/28; before: 22/28).
1614
+
1615
+ **Browser runs after the fixes** (one lock, canary 2.5 s before, 0.81 s after: a loaded host):
1616
+
1617
+ | | Chromium | Firefox | WebKit |
1618
+ |---|---|---|---|
1619
+ | b11-fixes, no guest, 3 rounds | 3/3 x 3 | 3/3 x 3 (two rounds failed on bounds that were too tight for a few-ms measurement, fixed: see below) | 3/3 x 3 |
1620
+ | the vm through 5 forced GCs in a hold | `{vm: true, up: true}`, then SIGTERM through it: exit 0 | - | - |
1621
+ | node, `slack=256` | exit 4, `wmn.error` out of memory, no garbage-IR line | - | - |
1622
+ | wmn python / node | 13/13 / 13/13 | 13/13 / **12/13** | 13/13 / 13/13 |
1623
+
1624
+ Two of my own bounds were tightened against flakiness, not loosened against a regression: the OOM test's
1625
+ second thread now writes once a millisecond (it spun before, so "lines after the trap" measured CPU speed:
1626
+ 11 in one Chromium run), and the AgentConn ratio uses a 50 ms floor (Firefox 5 MB at 15 ms made 20 MB at
1627
+ 121 ms a ratio of 8 while the old code took 16 s).
1628
+
1629
+ **Firefox node `infinite output`, 12/13 (run6 and this run):** 4,545 and 7,676 bytes in 1.5 s against the
1630
+ check's 10,000. This is the guest's JIT in Firefox, not the transport: node's flood block is new V8 code, so the
1631
+ guest translates it; the boot's JIT door spent 2,249 ms compiling modules in Firefox against 270 ms in
1632
+ Chromium (535 / 548 modules) and 2,634 ms waiting for shared modules against 199 ms (P-B2's per-browser
1633
+ compile cost). The same boot's python flood streamed 254 KB. Left as is; the check's bound is unchanged.
1634
+
1635
+ ### On wmd main (branch rebased onto 9833319c, HEAD ee1886e8 and after)
1636
+
1637
+ wmd main brought W14/W15 (9P), W17 (virtio-console interrupts) and W18 (guest entropy from the host
1638
+ at boot, 70d03426). Every run below is under the boot lock, `PORT=8798`, with the installed
1639
+ `vm.wasm` (sha256 30db07f5..., built after W17/W18).
1640
+
1641
+ **wmn suite, run7** (2026-10-05 15:48-15:50, one boot per image and browser; canary 648/704/717 ms
1642
+ before and 662/662/717 ms after the Chromium/Firefox/WebKit runs; load average 12-23):
1643
+
1644
+ | ms from `bootWmd` | Chromium python / node | Firefox python / node | WebKit python / node |
1645
+ |---|---|---|---|
1646
+ | control session up | 4,345 / 4,383 | 4,858 / 5,184 | 5,202 / 5,041 |
1647
+ | `hello_reply` | 5,707 / 5,707 | 6,157 / 6,416 | 7,795 / 7,785 |
1648
+ | `languages_reply` | 7,038 / 6,902 | 8,199 / 8,410 | 9,566 / 9,979 |
1649
+ | `session_start` | 7,484 / 3,706 | 11,606 / 2,860 | 9,054 / 4,352 |
1650
+ | define / use | 63 / 48, 263 / 101 | 49 / 43, 151 / 81 | 46 / 66, 244 / 114 |
1651
+ | interrupt a loop | 35 / 95 | 29 / 68 | 35 / 91 |
1652
+ | infinite output (bytes in 1.5 s, stopped in ms) | 1,077,165 (32) / 60,802 (33) | 328,350 (24) / **29,795** (46) | 248,661 (30) / 33,229 (46) |
1653
+ | all 13 checks done | 19,866 / 16,152 | 26,441 / 16,281 | 24,127 / 19,814 |
1654
+ | checks | 13/13 / 13/13 | 13/13 / **13/13** | 13/13 / 13/13 |
1655
+
1656
+ - **Firefox node passes all 13 for the first time.** Its flood streamed 29,795 bytes against the
1657
+ check's 10,000 (4,545 and 7,676 in the two runs before). The control session came up at 5.2 s, as in
1658
+ the other browsers. The cause of the change was not isolated; this run's host was quieter than run6's
1659
+ (no throttling), and W18 removed the guest's wait for entropy.
1660
+ - **The guest's entropy is seeded at boot in the browser.** Every boot log of run7 and after has
1661
+ `random: crng init done` at `[0.000000]` (13 of 13 logs in `wmn-results/`). Before W18 the same boots
1662
+ logged it at 4.1-19.6 s (`wmn-results/r6`), from jitter entropy over the strictly increasing clock of
1663
+ P-B3 problem 2.
1664
+ - Logs: `wmn-results/main-{chromium,firefox,webkit}/`.
1665
+
1666
+ **wmn suite, run8** (2026-10-06 08:00-08:05, the finished branch: the out-of-memory change, the
1667
+ random_get test; one boot per image and browser; canary 693/637, 924/631, 700/1,012 ms before/after
1668
+ the Chromium/Firefox/WebKit runs; load average 13-23):
1669
+
1670
+ | ms from `bootWmd` | Chromium python / node | Firefox python / node | WebKit python / node |
1671
+ |---|---|---|---|
1672
+ | control session up | 5,217 / 4,953 | 6,220 / 6,091 | 6,709 / 7,336 |
1673
+ | `hello_reply` | 6,668 / 6,623 | 8,099 / 7,654 | 10,186 / 10,961 |
1674
+ | `languages_reply` | 7,864 / 7,621 | 10,210 / 9,969 | 12,717 / 13,606 |
1675
+ | `session_start` | 8,183 / 4,091 | 13,157 / 3,885 | 12,631 / 5,456 |
1676
+ | define / use | 68 / 77, 287 / 86 | 66 / 77, 187 / 109 | 134 / 73, 303 / 202 |
1677
+ | interrupt a loop | 105 / 214 | 210 / 203 | 178 / 107 |
1678
+ | infinite output (bytes in 1.5 s, stopped in ms) | 836,785 (64) / 42,824 (89) | 223,008 (226) / 31,916 (233) | 718,514 (229) / 29,391 (93) |
1679
+ | all 13 checks done | 22,306 / 17,776 | 31,531 / 19,780 | 33,064 / 25,372 |
1680
+ | checks | 13/13 / 13/13 | 13/13 / 13/13 | 13/13 / 13/13 |
1681
+
1682
+ - Firefox node passes all 13 again (31,916 bytes in its flood against the check's 10,000).
1683
+ - `random: crng init done` at `[0.000000]` in all six boot logs.
1684
+ - A first WebKit run (07:25, canary **6.2 s** before it: another session's load) also gave 13/13 and
1685
+ 13/13; its times are not comparable and are not in the table.
1686
+ - Logs: `wmn-results/run8-{chromium,firefox,webkit}/` (the contended WebKit run: `run8-webkit-first/`).
1687
+
1688
+ **b11-fixes guest tests on the finished branch** (Chromium, 08:05, canary 881/634 ms): 2/2.
1689
+ - The vm handle after 5 forced GCs: `{vm: true, up: true}`.
1690
+ - node at `slack=256`: exit 4, "out of memory: the emulator used 1664 MiB of its 1664 MiB memory
1691
+ maximum ...", through the size rule (the installed vm.wasm has no heap reserve).
1692
+
1693
+ **b11-fixes on the rebased branch.**
1694
+ - No-guest tests, 3 browsers, 16:12 (canary 673/690 ms): 9/9.
1695
+ - Guest tests, Chromium (canary 692/640 ms): 2/2.
1696
+ - The vm handle after 5 forced GCs: `{vm: true, up: true}`.
1697
+ - node at `slack=256`: exit 4 with `wmn.error` "the machine ended: 4 (out of memory: the emulator
1698
+ used 1664 MiB of its 1664 MiB memory maximum ...)".
1699
+ - The same no-guest tests on the pre-fix code (347f80ed's `boot.js`/`k2.js`): 6 failed, 3 passed. The
1700
+ 3 that pass are "a trap far below the maximum keeps its plain reason", which is meant to pass before
1701
+ and after.
1702
+
1703
+ **Regression run, Chromium, the whole suite** (16:16-16:49; canary 873 ms before, 641 after; load
1704
+ average 23 falling to 7): 102 passed, 51 skipped (opt-in suites), 3 failed.
1705
+
1706
+ | failure | cause | re-run |
1707
+ |---|---|---|
1708
+ | `chan.spec` hvc0, 50 MiB in 120 s | the guest's hvc0 path is slow; vm.wasm, not the browser (below) | 4 MiB in each browser: pass |
1709
+ | `jit.spec` P-B2 proof boot "5x faster than P-B1" | load: `PROOF-DONE` at 9,002 ms against the bound of 8,808, on a host whose canary read 873 ms | alone, canary 623 ms: pass (7.3 s test) |
1710
+ | `review-errors` `argv --cpus 64` bounded at 60 s | known, left as is (P-B1 review): 64 vCPUs on 10 cores are still booting at 60 s | - |
1711
+
1712
+ **Regression run on the finished branch, Chromium, the whole suite** (2026-10-06 08:06-08:37; canary
1713
+ 638 ms before, 613 after; load average 12 falling to 5): 108 passed, 51 skipped (opt-in suites), 2
1714
+ failed, both known:
1715
+ - `chan.spec` hvc0, 50 MiB in 120 s: "no exit in 120000 ms" (below). The guest's log of that boot
1716
+ (`run8-logs/timeout-console.stdout.log`, `CONSOLE-START` at 4.04 s) has no `nobody cared` and no
1717
+ `Disabling IRQ`: the storm is gone, the console is just slow;
1718
+ - `review-errors` `argv --cpus 64` bounded at 60 s: still booting at 60 s, as before.
1719
+
1720
+ The P-B2 proof boot passed this time (8.0 s; the load failure of the run before did not recur).
1721
+ Logs of every run of 2026-10-06: `wmn-results/run8-logs/`.
1722
+
1723
+ ### hvc0 after W17: slow in vm.wasm, not in the browser host
1724
+
1725
+ **Superseded by W53 (D-433, below):** the slowness was the test's 2-byte tty writes. The
1726
+ conclusions below ("in vm.wasm or the guest's hvc driver", the 2 KiB `put_chars` guess) were wrong.
1727
+
1728
+ W17 (2ec1cf9d) stopped vm.wasm raising an interrupt for console-port transmits. The IRQ storm of
1729
+ P-B3 problem 1 is gone: no `nobody cared` and no `Disabling IRQ` in any run. But `yes | head -c N >
1730
+ /dev/hvc0` on a raw tty is still slow, and the 50 MiB test cannot finish in its 120 s.
1731
+
1732
+ The same initramfs (`b3Initrd` + `consoleScript`), the same kernel and the installed vm.wasm, one boot
1733
+ each, 1 vCPU, `--mode jit`:
1734
+
1735
+ | | bytes | time | rate |
1736
+ |---|---|---|---|
1737
+ | browser hvc0, Chromium (canary 623 ms) | 4 MiB | 74.5 s | 56 KB/s |
1738
+ | browser hvc0, Firefox | 4 MiB | 126.6 s | 33 KB/s |
1739
+ | browser hvc0, WebKit | 4 MiB | 58.4 s | 72 KB/s |
1740
+ | **native `wmd-run`** hvc0, a Unix-socket sink (canary 727 ms) | 4 MiB | 61.1 s | **69 KB/s** |
1741
+ | native `wmd-run` control port (vport1p1), the same sink | 8 MiB | 2.65 s | 3.2 MB/s |
1742
+ | browser control port (P-B3 numbers) | 50 MiB | 10.7-15.9 s | 3.3-4.9 MB/s |
1743
+
1744
+ - **Native is as slow as the browser**, so the cost is in vm.wasm or the guest's hvc driver, not in the
1745
+ browser host.
1746
+ - In every browser run the host kept up:
1747
+ - every byte reached the page, in order;
1748
+ - the guest's `chan_send` never found the ring full (`sendWaits` 0);
1749
+ - the service's first-to-last-byte time was the guest's own time to within 0.1-1.1 s (74.42 against
1750
+ 74.51 s, 126.52 against 126.60, 57.30 against 58.37).
1751
+ - Likely cause, not proven: Linux's hvc path sends each tty write of at most 2 KiB (`put_chars` ->
1752
+ `__send_to_port`, one device exit each) and then polls. At 69 KB/s that is about 34 sends a second,
1753
+ far fewer than an exit costs. D-391 (W17) already notes that line-by-line hvc output is still slow,
1754
+ at about 22 KB/s.
1755
+ - **This is a vm.wasm/guest finding, for the wmd side (a follow-up of W17).** Nothing in wmd uses
1756
+ port 0: container stdio goes over the control port, which runs at MB/s.
1757
+
1758
+ **W53 (D-433): the cause was the test, not hvc0.** busybox `head` writes through stdio, which is
1759
+ line-buffered on a tty: into `/dev/hvc0` it made one 2-byte write per `y` line, each a full hvc
1760
+ write (about 26-36 us in the guest). Natively, on the same vm.wasm and kernel: `dd` into hvc0 with
1761
+ 512 B-64 KiB blocks moved 256 KiB in 0.03-0.07 s; `yes | head -c 1M | cat > /dev/hvc0` 0.33 s
1762
+ (3.2 MB/s); `yes | head -c 256K > /dev/hvc0` 3.42 s (77 KB/s). The control port never showed it
1763
+ because a vport is not a tty. The console script now writes through `cat` for every target, and
1764
+ chan.spec `-g console` passes in all three browsers:
1765
+
1766
+ | 50 MiB | Chromium | Firefox | WebKit |
1767
+ |---|---|---|---|
1768
+ | hvc0 | 16.81 s, 3.1 MB/s | 24.78 s, 2.1 MB/s | 19.39 s, 2.7 MB/s |
1769
+ | control port | 15.71 s, 3.3 MB/s | 20.02 s, 2.6 MB/s | 17.44 s, 3.0 MB/s |
1770
+ | /dev/null (guest alone) | 14.86 s, 3.5 MB/s | 17.64 s, 3.0 MB/s | 17.82 s, 2.9 MB/s |
1771
+
1772
+ A program that writes a tty line by line still pays about 30 us per line.
1773
+ - (Before W53) the test was unchanged and its 50 MiB in 120 s failed; it passes since W53.
1774
+ - Re-run: `WMD_WEB_CONSOLE_BYTES=4194304 WMD_WEB_HVC0_TIMEOUT_MS=300000 bunx playwright test chan.spec -g hvc0`.
1775
+ The native comparison boots `wmd-run vm.wasm -- --machine ... --console 7 --control 8` with
1776
+ `WMD_CHAN_UNIX="7=/tmp/x.sock,8=/tmp/y.sock"`. Use a short socket path: macOS caps `sun_path` at 104
1777
+ bytes, and a longer path makes `wmd-run` report "host not listening".
1778
+
1779
+ ### Review of 70d03426 (W18: WASI random_get in `src/host/wasi.js`)
1780
+
1781
+ - **Correct.**
1782
+ - `random_get(ptr, len)` checks the range (EFAULT) against the current memory size, after
1783
+ `views()` has picked up any grow.
1784
+ - It fills at most 64 KiB per `getRandomValues` call, through a private buffer, then copies into the
1785
+ shared memory. Browsers refuse a view of a SharedArrayBuffer, and the call is limited to 64 KiB.
1786
+ - `len` 0 does nothing.
1787
+ - The `worker.js` guard wraps it like every other WASI import, so once the machine has ended it
1788
+ throws instead of running.
1789
+ - The installed vm.wasm imports `random_get`, so a host without it could not instantiate vm.wasm at
1790
+ all.
1791
+ - **Gap, now covered.** The unit test (`wasi.test.js`) runs in bun, and bun's `getRandomValues`
1792
+ accepts views of shared memory. So the unit test cannot show why the copy is needed; before this
1793
+ commit only a guest boot would. New: b11-fixes "WASI random_get fills shared wasm memory in this
1794
+ browser". It passes in all three browsers (2026-10-06 07:08). With the copy removed
1795
+ (getRandomValues on `u8.subarray`; 07:22, canary 760/698 ms) it fails in each, with the browser's
1796
+ own error:
1797
+ - Chromium: "The provided ArrayBufferView value must not be shared";
1798
+ - Firefox: "can't be a SharedArrayBuffer or an ArrayBufferView backed by a SharedArrayBuffer";
1799
+ - WebKit: "must be an instance of ArrayBufferView".
1800
+ - **Effect.** See run7: `crng init done` at 0.000000 in every browser boot.
1801
+ - **Unchanged.** The strictly increasing clock (P-B3 problem 2), now a fallback for a host that gives
1802
+ no seed. The run request's own 32-byte `seed` (`fillRunRequest`), which the native engine sends as
1803
+ well.
1804
+
1805
+ ### Out-of-memory detection with W21's heap reserve (branch `w21-trap`, not merged here)
1806
+
1807
+ W21's fix (w21-trap, a64e0c1a + 02a281d3) has vm.wasm grow its memory to the maximum before any thread
1808
+ starts (`WMD_HEAP_RESERVE=1`, which `bootWmd` sets by default there). With it, review finding 4's rule
1809
+ ("a trap with the memory within 64 MiB of its maximum is out of memory") would call every trap out of
1810
+ memory. This branch's `finish()` now uses `isOutOfMemory()` (`src/boot.js`):
1811
+
1812
+ - vm.wasm's stderr line `vm.wasm: heap reserve used up (W21): out of memory` means out of memory,
1813
+ whatever the size says;
1814
+ - without a reserve, the old size rule;
1815
+ - with a reserve (below) and no "used up", a plain `trap:` reason.
1816
+
1817
+ `finish()` drains stderr before it decides, so vm.wasm's last line counts. `timings.oom` records
1818
+ which signal it was.
1819
+
1820
+ The reserve counts as in use when either says so:
1821
+ - vm.wasm's environment asks for it (`heapReserveAsked(env)`: `WMD_HEAP_RESERVE` set and not `0`, as
1822
+ `heap_reserve.c` reads it). On w21-trap `bootWmd` adds `WMD_HEAP_RESERVE=1` unless `heapReserve` is
1823
+ false, so after the merge the size rule applies only with `heapReserve: false`, and a reserve line
1824
+ lost on the way cannot turn every trap into out of memory.
1825
+ - vm.wasm printed its reserve line, `vm.wasm: heap reserve N MiB at ...`. That also covers a caller that sets the variable some other way.
1826
+
1827
+ Without either (this branch as it is, or `heapReserve: false`), the size rule of finding 4 is unchanged.
1828
+ One case is not covered: the reserve asked for, but vm.wasm could not grow its memory at all (memory
1829
+ already at its maximum, so `heap_reserve.c` prints nothing and keeps no reserve). Its out-of-memory trap
1830
+ would then read as a plain trap; vm.wasm's own `sbrk` prints nothing in that case either.
1831
+
1832
+ Merging: w21-trap was cut from this branch before the rebase onto wmd main and the review fixes, so a
1833
+ plain merge conflicts in many files. Its two commits (a64e0c1a, 02a281d3) cherry-pick onto this branch
1834
+ with one conflict, the README's last section (both append one); `boot.js` merges cleanly: w21-trap turns
1835
+ `const env = o.env || []` into `vmEnv(o.env, o.heapReserve)`, and `finish()` here reads that `env`.
1836
+ After the merge:
1837
+ - the synthetic out-of-memory test (`oom_trap.wat`) passes `WMD_HEAP_RESERVE=0`, so it still tests the
1838
+ size rule (vmEnv lets the caller's entry win);
1839
+ - a real out-of-memory trap counts only through vm.wasm's "used up" line, so vm.wasm must be the W21
1840
+ build. With the installed vm.wasm (30db07f5, no reserve, prints neither line) and w21-trap's default
1841
+ `heapReserve`, b11-fixes "node out of its memory maximum" would read a plain `trap:`. That is the price
1842
+ of the rule asked for (size check only with `heapReserve: false`); the opposite mistake, every trap
1843
+ called out of memory, is what it prevents.
1844
+
1845
+ Tests (each new one fails on the code before it):
1846
+ - `test/unit/oom.test.js`: 5 tests (`isOutOfMemory`, `heapReserveAsked`, the two regexes against
1847
+ `heap_reserve.c`'s format strings).
1848
+ - b11-fixes, four synthetic guests that print vm.wasm-style lines on stderr and trap (`review-helpers.js`
1849
+ `startBoot` now passes `env`):
1850
+
1851
+ | case | ee1886e8 `boot.js` | first version (keyed on the line only) | now |
1852
+ |---|---|---|---|
1853
+ | reserve line, memory at the maximum, no "used up" | `out of memory: ...` (fail) | `trap: unreachable` | `trap: unreachable` |
1854
+ | `WMD_HEAP_RESERVE=1`, no line, memory at the maximum | `out of memory: ...` (fail) | `out of memory: ...` (fail) | `trap: unreachable` |
1855
+ | `WMD_HEAP_RESERVE=0`, memory at the maximum | out of memory, no `timings.oom` (fail) | out of memory, "memory at its maximum" | the same |
1856
+ | "used up" line, memory 384 of 640 MiB | `trap: unreachable` (fail) | out of memory, "heap reserve used up" | the same |
1857
+
1858
+ Runs (2026-10-06, under the boot lock, canary 625-634 ms, load average 15-23): ee1886e8's `boot.js` 12
1859
+ of 12 failed (3 browsers); the first version 3 failed (the second case, each browser); now all of
1860
+ b11-fixes' no-guest tests pass, 24 of 24 in the three browsers (07:08).
1861
+
1862
+ ### Merged into wmd main (d832d730, W3.1-W3.3), 2026-10-06
1863
+
1864
+ `git merge --no-ff wmd-web-b11` (f45d6352) into main d832d730 had no textual conflicts: main's side
1865
+ (W3.1-W3.3) touches only `guest/vm`, `internal/engine`, `runner` and `docs/research`, and this branch
1866
+ only `web/`. Everything below ran on the merged tree in a scratch worktree, with what the tests serve
1867
+ built from that tree, never into `~/.wmd/bin`:
1868
+ - `vm.wasm`: `make guest-vm` (sha256 f3a7caee..., against the installed 30db07f5...);
1869
+ - the Go init's initramfs: `docker buildx build --target container-rootfs-arm64 --build-context
1870
+ c2w=guest/container2wasm -f guest/linux/Dockerfile --output type=tar,dest=- guest/linux`, the archive's
1871
+ `container-initramfs.cpio.gz`. It is byte-identical to the installed one (0bafbaf9...): neither side
1872
+ changed `guest/container2wasm` or `guest/linux`. The kernel (`Image`) was copied from the installed guest;
1873
+ - the packs: `fixtures-gen image` for python:3.12-slim and node:22-slim, identical to the branch's;
1874
+ - served with `WMD_VM_WASM` and `WMD_GUEST_DIR` pointing at those files, `PORT=8798`.
1875
+
1876
+ **A conflict the merge did not show: W3.3's vm.wasm cannot start in the browser.** Each boot ended at
1877
+ once with exit 4. Nothing in `web/` (on the branch or on main) knew about W3.3's two vm.wasm changes:
1878
+ 1. `trap: WebAssembly.Instance(): table import 1 is smaller than initial 158, got 155`. vm.wasm's table
1879
+ import minimum went from 155 to 158, and `jit.js` had it as the constant `TABLE_MIN = 155`. The
1880
+ runner reads it from the module (`host.rs` `table_min`).
1881
+ 2. With 158: `Import #13 "wmd" "point": function import requires a callable`. W3.3 added the imports
1882
+ `wmd.ctl_attach` and `wmd.point`.
1883
+
1884
+ Fixed in this merge:
1885
+ - `tableMinOf(bytes)` (`src/host/jit.js`) reads the table import minimum from vm.wasm's import section,
1886
+ since the JS API does not expose import limits. `boot.js` passes it to every Worker, which sizes its
1887
+ table with it, and the JIT door starts its slots after it (`createJitDoor({tableMin})`). `TABLE_MIN`
1888
+ (155) is now only the default, for modules with no table import and for the unit tests. The installed
1889
+ vm.wasm (155) and W3.3's (158) both boot.
1890
+ - `wmd-imports.js`: `ctl_attach` returns -1 (no control socket, so the guest starts no ctl thread) and
1891
+ `point` returns -38 (no checkpoint; the guest goes on). These are the weak defaults in
1892
+ `guest/vm/src/dev/chan.c`.
1893
+ - `test/unit/vm-imports.test.js` (6 tests, which fail before the fix): `tableMinOf` on hand-made
1894
+ modules, and on the vm.wasm being served (`WMD_VM_WASM`), the table minimum plus "the host provides
1895
+ every `wmd.*` function it imports".
1896
+
1897
+ Runs on the merged tree with the fix (all under the boot lock, one at a time; canary 610-632 ms
1898
+ before and after each, unless noted):
1899
+
1900
+ | suite | result |
1901
+ |---|---|
1902
+ | `bun test test/unit test/adversarial-b11/yamux-adv.test.js` | 314 pass, 2 skip, 0 fail (308 before `vm-imports.test.js`) |
1903
+ | b11-fixes, no guest, Chromium + Firefox + WebKit | 24/24 (6 guest tests skipped) |
1904
+ | b11-fixes, guest boots, Chromium (`WMD_WEB_B11FIX=1`) | 2/2: `{vm: true, up: true}` after 5 GCs; node at `slack=256`: exit 4, "out of memory: the emulator used 1664 MiB of its 1664 MiB memory maximum ..." |
1905
+ | wmn.spec python / node, Chromium | 13/13 / 13/13 |
1906
+ | wmn.spec python / node, Firefox | 13/13 / 13/13 |
1907
+ | wmn.spec python / node, WebKit | 13/13 / 13/13 |
1908
+ | adversarial-b11 `rob.spec` (`ROB_IMAGES=python,node`), Chromium | python 68 PASS / 4 FAIL, node 66 / 5: the same as before the merge (below) |
1909
+ | native `go test -short ./...` | all pass except `test/compat` TestBuildKitCopyLink and TestBuildKitTarOutputOwnership, which fail on main too |
1910
+ | native `make test` (`go test ./...`, under the lock; canary 995 ms before, 633 after) | all pass except `internal/buildkit` TestBoltCommitsDoNotFlushTheDriveCache, a timing test (commit 8.9 ms against half of 12.1 ms; it passed 3 of 3 times when run alone afterwards), and `test/compat`, which hit `go test`'s default 10 min timeout in TestBuildFailingRunOutput (`make test-compat` gives it 180 min). Neither package is touched by this merge (no Go file outside `web/` changes) |
1911
+
1912
+ Without the fix, the same b11-fixes guest tests and all six wmn boots failed with error 1.
1913
+
1914
+ wmn timings, in ms from `bootWmd` (one run each, load average 5-9):
1915
+
1916
+ | | Chromium python / node | Firefox python / node | WebKit python / node |
1917
+ |---|---|---|---|
1918
+ | control session up | 4,134 / 4,119 | 4,067 / 4,111 | 4,145 / 4,209 |
1919
+ | `hello_reply` | 5,389 / 5,398 | 5,119 / 5,168 | 6,479 / 6,847 |
1920
+ | `session_start` | 6,045 / 3,305 | 7,200 / 2,194 | 6,561 / 3,269 |
1921
+ | interrupt a loop | 28 / 57 | 22 / 35 | 25 / 46 |
1922
+ | infinite output (bytes in 1.5 s) | 1,185,942 / 96,354 | 293,102 / 74,538 | 369,559 / 59,691 |
1923
+ | all 13 checks done | 17,252 / 14,491 | 18,957 / 12,953 | 19,193 / 17,061 |
1924
+
1925
+ **adversarial-b11 on the merged tree: no change from the merge.** The same python run with the
1926
+ installed (pre-W3) vm.wasm and guest gave the same 68 PASS / 4 FAIL, with the same trap (canary 1,320 ms
1927
+ before, 669 after; another session was running). The merged-tree node run ended at canary 1,118 ms. The
1928
+ failures:
1929
+ - c03 "next cell unaffected" (both images): known; late output, wmn's limit (ROADMAP 4.3b).
1930
+ - b04 "65 MiB line refused" and b05 "20 MiB of stdin into an exec" (both images): the review's harness
1931
+ writes 65 MiB and 20 MiB with one synchronous `proc.write()`. Since the finding-7 fix, `write()` past
1932
+ the stream's credit (4 MiB + 64 KiB) returns false and sends nothing, so the agent never sees the line
1933
+ and `wc -c` reads 0. This is the fix's contract, and the review's tests were left unchanged.
1934
+ `writeAsync()` is the call for big data.
1935
+ - b08 "20 MB execute_result" (node): the result came back with 10,030 characters. As in the review's own
1936
+ node run, this is not the transport.
1937
+ - b05 "30 execs at once" (both images): the machine traps, review finding 1. The trap here is
1938
+ `RangeError: Start offset -2147302880 is outside the bounds of the buffer` at `jit.js` `readGuest` <-
1939
+ the door's `compile` <- `vm_wasm_jit_compile` <- `compile_worker`, and it happens with both vm.wasm
1940
+ builds. **New finding, not fixed here:** the JIT door's `load`/`compile` use the guest pointer as a
1941
+ signed i32 (no `>>> 0`, unlike `wasi.js` and `chan.js`). Once vm.wasm's heap passes 2 GiB (the wmn
1942
+ page's 1 GiB slack allows a 2.4 GiB maximum), the first JIT module there kills the machine. This may
1943
+ be part of finding 1 (W21).
1944
+
1945
+ Not run here, as asked: the hvc0 50 MiB `chan.spec` (W22) and the `--cpus 64` boot (over 60 s) are
1946
+ known failures; W3's checkpoint guest tests are W3's.
1947
+
1948
+ ### W23: guest pointers above 2 GiB, and a review of the merge's fix (branch `wmd-web-w23`)
1949
+
1950
+ **W23.** vm.wasm passes guest pointers as wasm i32s; a JS import receives them signed. `jit.js`
1951
+ `readGuest` (`jit_load`, `jit_compile`) used them as is, so once vm.wasm's heap passed 2 GiB (the
1952
+ wmn page's 1 GiB slack gives a 2.4 GiB maximum) the first block translated there trapped the
1953
+ machine (the adversarial-b11 b05 trap above). `readGuest` now reads `ptr` and `len` as unsigned
1954
+ and a range outside the memory makes the call return -1 with a log line, as the runner's
1955
+ `load_slots` does, instead of trapping. The rest of `src/` was audited: `wasi.js` and `chan.js`
1956
+ already converted their pointers; two WASI lengths did not (`path_open`'s path length,
1957
+ `fd_prestat_dir_name`'s buffer length) and now do. `thread-spawn`'s start argument goes back to
1958
+ wasm as the same i32; the hub, the rings, stdio and the 9P server read no guest addresses. Range
1959
+ checks are `ptr + len <= byteLength` in doubles, so the 4 GiB end of a 65536-page memory is exact.
1960
+
1961
+ Tests (each fails before the fix):
1962
+ - `test/unit/w23-pointers.test.js` (9): the door above 2 GiB and ending at 4 GiB, refusals past the
1963
+ end, WASI and channel calls with signed high addresses. 6 fail on 6361975f.
1964
+ - `test/e2e/w23-pointers.spec.js`: a synthetic guest booted through `bootWmd` grows to 4 GiB,
1965
+ `jit_load`s a module at 0x8002c220 (the b05 trap's offset), `jit_compile`s one ending at 4 GiB,
1966
+ calls both through its table and `fd_write`s its result from above 2 GiB. Before: exit 4 in
1967
+ `readGuest`: Chromium "Start offset -2147302880 is outside the bounds of the buffer", Firefox
1968
+ "invalid or out-of-range index", WebKit "byteOffset cannot be negative". After: exit 0,
1969
+ "W23 OK", in all three.
1970
+
1971
+ **Review of `tableMinOf`, the `ctl_attach`/`point` stubs (6361975f).**
1972
+ - `tableMinOf` was right for the vm.wasm shapes built today (155 and 158, one table import),
1973
+ custom sections, non-minimal LEB128, memory64/table64, a table maximum, tags and no table import.
1974
+ It was wrong for: two table imports (it took the first; now `env.__indirect_function_table`,
1975
+ else the last, as `host.rs`), typed references (0x63/0x64 + a heap type) in table and global
1976
+ imports, a custom page size (limits flag 8), a section shorter than its imports, an endless
1977
+ LEB128. Fixed; 16 new cases in `vm-imports.test.js`, 7 failed before.
1978
+ - JIT slots: the runner puts slot k at `table_min + k` (`jit.rs`), and so does the door
1979
+ (`tableMin + slotBase`). The guest uses only the index the door returns. Nothing is hard-coded.
1980
+ - `ctl_attach` -1: `machine_start_ctl` then starts no ctl thread and returns quietly. The only
1981
+ path to `wmd.point` is a request from that thread, so `point` is never called in the browser;
1982
+ -38 would only print "checkpoint not taken" once. The init's redial loop (W3.2) runs only after
1983
+ a session that set `ckpt`, which the page never sends, and it backs off from 50 ms to 1 s.
1984
+ No busy loop, no flood of log lines, no boot delay.
1985
+
1986
+ **With w21-trap.** w21-trap changes neither `jit.js` nor `wasi.js`, and this branch does not touch
1987
+ `boot.js`. W21's reserve grows the memory to its maximum before any thread starts, so heap
1988
+ addresses above 2 GiB come into use sooner. Without this fix, W21's 30-exec storm would hit W23's
1989
+ trap more readily.
1990
+
1991
+ Runs on this branch. Everything is under the boot lock, `PORT=8795`, with vm.wasm built in the
1992
+ worktree (f3a7caee, `make guest-vm`). The disk floor was lowered to 50 GiB by the user, and
1993
+ `locked.sh` now checks 50.
1994
+
1995
+ | suite | result |
1996
+ |---|---|
1997
+ | `bun test test/unit test/adversarial-b11/yamux-adv.test.js` | 339 pass, 2 skip, 0 fail |
1998
+ | b11-fixes (no guest) + `w23-pointers.spec`, Chromium / Firefox / WebKit | 9 passed, 2 skipped, in each |
1999
+ | wmn.spec python / node, Chromium (canary 660 ms before) | 13/13 / 13/13 |
2000
+ | wmn.spec python / node, Firefox (canary 1,450 ms before: a loaded host, load average 17) | 13/13 / 13/13 |
2001
+ | wmn.spec python / node, WebKit (canary 917 ms before, 1,159 after) | 13/13 / 13/13 |
2002
+ | adversarial-b11 b11, 30 execs at once (`ROB_IMAGES=python ROB_QUERY='&cases=b11&conc=30'`), Chromium (canary 826 / 733 ms) | FAIL, but not W23. The machine traps with `memory access out of bounds` at `calloc` <- `interp_run_block` (thread 6), with the memory at 1,562 MiB of its 2,432 MiB maximum. This is review finding 1 / W21 (Chromium's stale memory size after another Worker's grow), not merged here. The heap never reached 2 GiB in this run, so it did not reach W23's path. No `readGuest` or "outside the guest memory" line appears. |
2003
+
2004
+ The 30-exec run cannot show W23 until W21 is in: W21's trap comes first, below 2 GiB. W23 is shown
2005
+ by `w23-pointers.spec.js` (before and after, three browsers) and the unit tests.
2006
+
2007
+ ## W21: 30 concurrent execs trapped the machine in Chromium
2008
+
2009
+ The bug (wmn ROADMAP, "Work in wmd"): 30 `exec`s of `/bin/echo` at once over the control session on
2010
+ python:3.12-slim (`test/adversarial-b11`, `cases=b11&conc=30`) ended the machine with exit 4,
2011
+ `RuntimeError: memory access out of bounds` at `calloc` <- `interp_run_block` <- `vm_run_blocks`, in
2012
+ 5 of 6 Chromium boots. Firefox and WebKit passed, and so did native `wmd-run`. The memory was far
2013
+ below its maximum, so it was not an out-of-memory.
2014
+
2015
+ ### Cause
2016
+
2017
+ vm.wasm's own heap (dlmalloc, through wasi-libc's `sbrk`) grows memory 0 with `memory.grow` while
2018
+ the vCPU threads run. Each thread is a Worker with its own instance of the module. In Chromium, a
2019
+ `memory.fill` or `memory.copy` in one Worker can be checked against a memory size that is older than
2020
+ another Worker's `memory.grow`. A plain load or store sees the new pages at once. When vCPU A grows
2021
+ the heap and dlmalloc hands the new top chunk to vCPU B under its lock, B's `calloc` clears the chunk
2022
+ with `memory.fill` (wasi-libc is built with bulk memory) and traps. 30 execs at once make both vCPUs
2023
+ translate blocks and allocate fast. The heap then grows one page at a time, about every 0.5 ms, so
2024
+ the window comes up again and again.
2025
+
2026
+ Evidence:
2027
+
2028
+ - **The engine probe** (`test/e2e/w21-probe.spec.js`, `page/w21-probe.html`, no guest). Two Workers
2029
+ share one memory. Worker A grows it by one page and sets a flag; worker B spins on the flag, then
2030
+ touches the new page. 300 trials of each:
2031
+
2032
+ | B's access to the new page | Chromium 153 (PW 1.63) | Firefox | WebKit |
2033
+ |---|---|---|---|
2034
+ | `memory.fill` | **3/300 trap** | 0 | 0 |
2035
+ | `memory.copy` | **4/300 trap** | 0 | 0 |
2036
+ | `memory.fill` after `memory.atomic.wait` (not a spin) | 0 | 0 | 0 |
2037
+ | `i32.store8` (plain store) | 0 | 0 | 0 |
2038
+ | `memory.fill` / `memory.copy`, memory grown before the Workers start | 0 / 0 | 0 / 0 | 0 / 0 |
2039
+
2040
+ The trap message reports B's `memory.size` as already including the page. So the stale size is
2041
+ only the one that `memory.fill` and `memory.copy` are checked against. A wait refreshes it.
2042
+ - **In the guest** (a per-grow log in a diagnostic build, kept on branch `w21-trap-wip-backup`).
2043
+ Chromium, without the reserve: thread 6 (vCPU 1, `vcpu_thread_fn`) trapped in `calloc` at
2044
+ 20.027630 s (CLOCK_MONOTONIC), 36 us after vCPU 0's `memory.grow` 24185 -> 24186 pages at
2045
+ 20.027594 s. vCPU 0 had grown the heap 25 times in the 18 ms before, one page at a time. The
2046
+ trapping thread's last own grow was at 20.0095 s. The memory was 24,186 pages (1.5 GiB) of
2047
+ 38,912.
2048
+
2049
+ ### Fix: the heap reserve (`WMD_HEAP_RESERVE=1`)
2050
+
2051
+ `guest/vm/src/heap_reserve.c` replaces wasi-libc's `sbrk`. When `WMD_HEAP_RESERVE` is set, vm.wasm
2052
+ grows memory 0 to its maximum in `run_machine`, after guest RAM and the vCPU arena and before
2053
+ `machine_run` starts any thread. It grows in the largest steps the host accepts. `sbrk` then hands
2054
+ out those pages from an atomic bump pointer. Memory 0 never grows while more than one thread runs,
2055
+ so no Worker can hold a stale size. Without the variable, `sbrk` behaves exactly like wasi-libc's,
2056
+ so native `wmd-run` is unchanged. When the reserve is used up, vm.wasm says
2057
+ `vm.wasm: heap reserve used up (W21): out of memory` once and stops (see the review follow-ups below).
2058
+ `bootWmd` sets the variable unless `heapReserve: false` is passed or the caller's env already has a
2059
+ `WMD_HEAP_RESERVE` entry (`vmEnv`, `src/boot.js`). An older vm.wasm ignores it.
2060
+
2061
+ Cost: the reserve is address space, not RAM. Untouched pages are not resident. Peak RSS of one
2062
+ browser process in a 1-round run with the reserve was 1,210 MiB. The baseline runs, which trapped
2063
+ part-way, peaked at 1,034 and 1,721 MiB. That is not a clean A/B, but there is no extra 1 GiB.
2064
+
2065
+ ### Adversarial review follow-ups (`w21-trap-adv`, 22a71aa9), 2026-10-06
2066
+
2067
+ The branch is rebased onto wmd main 6361975f (P-B11 merged). The review's tests are in
2068
+ `test/adversarial-w21/` (its `RESULTS.md`, `storm.spec.js`, `usermode.spec.js`, `node-repro.mjs` for a V8
2069
+ bug report). `fixq.sh` is this round's run queue, and `out/f*.log` and `out/*.json` are its logs.
2070
+
2071
+ - **A4, done.** vm.wasm takes the reserve only for `WMD_HEAP_RESERVE=1` exactly. Before, `=no` turned it
2072
+ on; `heapReserveAsked` in `src/boot.js` now matches. The native runner no longer passes
2073
+ `WMD_HEAP_RESERVE` to the guest (`runner/src/wasi.rs` `guest_env_from`, cargo unit test). So the
2074
+ native snapshot cost the review measured cannot happen from a shell variable.
2075
+ - **A2, done: the user-mode traps were not W21.** User mode now takes the reserve in `main()` before any
2076
+ thread. rthreads still trapped 4 of 20 times with the reserve in place, and every trap came *after* the
2077
+ program's output. `main()` freed the block cache and guest RAM while the guest's detached threads were
2078
+ still translating (`emit_block_code` on freed memory). It now drains them first
2079
+ (`linux_user_wait_vcpus`, as `runprog.c` does). A thread trap also never ends in exit 0 any more:
2080
+ `worker.js` counts traps in `CTL_TRAPS`, and `finish()` turns an exit 0 into 4 if a thread trapped.
2081
+ - **A3, done (this overlaps W20).** When the reserve is used up, vm.wasm no longer hands NULL to
2082
+ dlmalloc's callers:
2083
+ - `sbrk` calls `vm_out_of_memory`. That prints the exact `vm.wasm: heap reserve used up (W21): out of
2084
+ memory` line, then `vm.wasm: out of memory: <what> of <n> bytes; ...`, then calls `_Exit(4)`.
2085
+ - `interp_run_block`'s calloc and `wasm_enc` call it too. Without a reserve the detail line still
2086
+ comes out. Natively it aborts after the message, as the unit tests expect.
2087
+ - `boot.js` reports that exit as out of memory (`VM_OOM_RE`) and stops every Worker.
2088
+ - Before the heap runs out, the reserve's *low mark* asks the block cache for one reset. The first
2089
+ request comes with a quarter of the reserve left, then one at each halving (`heap_rsv_take`,
2090
+ `heap_reserve_reset_wanted`, unit test `test_low_mark`).
2091
+ - With the reserve, each reset logs `vm.wasm: block cache reset: ...`.
2092
+ - **Why the IR cap was not sized from the reserve.** `ir_bytes` overcounts the heap: 805 MB of IR
2093
+ "bytes" with 546 MiB of a 1023 MiB reserve still free. A cap at 1/2 or 3/4 of the reserve reset the
2094
+ cache in every 30-exec storm and made it 2–3x slower. At 1/2 the heap passed 2 GiB after the reset
2095
+ and hit W23. With the cap at the reserve's full size, the slack-64 run reset again and again until the
2096
+ JIT door's module log filled (1 GiB), and the guest stalled instead of stopping. The low mark
2097
+ measures the heap itself, so D-254's cap is left alone.
2098
+ - **A1 is W23** (`wmd-web-w23`, not fixed here). With its host fix applied, a storm that crossed 2 GiB
2099
+ passed. **A6** (`9p-diff.test.js` dirent-type) fails on main too.
2100
+
2101
+ Runs on the final build (Apple Silicon, `PORT=8797`, every step under the boot lock with a 50 GiB disk
2102
+ floor; canary 609–723 ms around every step):
2103
+
2104
+ | Run | Result |
2105
+ |---|---|
2106
+ | `w21.spec.js` 30-exec storm, Chromium x5 | 5/5 pass, 0 bad, 12.1–15.2 s, 0 cache resets, no grow |
2107
+ | same, Firefox / WebKit | pass / pass (14.7 s / 13.9 s) |
2108
+ | python c45 / c60 / c100, `--ram 512` (Chromium) | 0 bad each; 0 / 2 / 6 cache resets. Of two c100 runs, one passed and one hit "used up" in round 0 and stopped cleanly with the message |
2109
+ | slack 64, c30 mixed, reserve (Chromium) | **stops cleanly**: 4 resets, then `heap reserve used up`, the detail line, exit 4, reported as out of memory. No trap, no garbage IR |
2110
+ | slack 64, no reserve | stops cleanly: `vm.wasm: out of memory: interp_run_block calloc ...`, out of memory (before: abort, or garbage IR) |
2111
+ | rthreads user mode x20 (Chromium) | 0/20 traps (was 8/20) |
2112
+ | `w21-fixes.spec.js` + `b11-fixes.spec.js` | 13/13 in each of C/F/W |
2113
+ | engine probe `w21-probe.spec.js` | C/F/W pass |
2114
+ | web unit tests | 288 pass, 3 fail (all `9p-diff.test.js`, A6) |
2115
+ | native (`runner/target/release/wmd-run`, this worktree) | `smoke-wasm.sh` 13/13 PASS. arm64 `wmd.proof` boot (jit, 4 vCPUs) `PROOF-DONE` with `WMD_HEAP_RESERVE=0` and `=1` (no reserve line: the runner drops the variable). rthreads x20, with and without the variable: 0 bad. `make test` in `guest/vm`: all binaries pass |
2116
+
2117
+ ### Joint W21/W23 review (`w21-w23-review` a597f429), fixes 2026-10-06
2118
+
2119
+ - **R1 (major, native): fixed.** A2's drain in user-mode `main()` waits for the guest's threads, but
2120
+ `clone` (`guest/vm/test/linux_user.c`) kept starting threads after `group_exiting` was set. So for a
2121
+ guest whose threads keep starting threads (`progs/spawnloop.c`), the wait did not settle: 5 of 15
2122
+ native runs took 6.7–8.1 s and printed `vm.wasm: guest threads still running at exit`.
2123
+ - `clone` now returns `-EAGAIN` once the group is exiting. It adds to `live_threads`, then reads
2124
+ `group_exiting`; the drain does the reverse; both seq_cst. So each new thread is either counted by
2125
+ the drain or refused.
2126
+ - A thread that registers after the drain walked the thread list does not enter `vm_run`.
2127
+ - Tests and logs: `test/adversarial-w21/native-spawnloop.sh` and `native-a2.sh`, from the review,
2128
+ with the guest programs in `progs/`.
2129
+ - Native, no variable: spawnloop **0/15** bad, 0.61–1.16 s, no new line.
2130
+ - Against main's vm.wasm, stderr is identical for blockread, blockwrite, pipewait, spawnloop,
2131
+ futexwait, exit3, threxit, rthreads and threads, with the same exit codes and times.
2132
+ - Main itself trapped (exit 4) on spawnloop in **2 of 10** runs: the use-after-free that A2's drain
2133
+ removes. This branch had 0 in 15.
2134
+ - **(4) minor: fixed.** `boot.js` now treats vm.wasm's out-of-memory lines as out of memory only with
2135
+ exit status 4, the status `vm_out_of_memory` uses. Before, a user-mode guest that printed the lines
2136
+ and exited 1 (`oomspoof`) was reported as out of memory. A guest that prints the lines *and* exits
2137
+ 4 cannot be told apart in user mode; in machine mode the guest cannot write vm.wasm's stderr.
2138
+ `w21-fixes.spec.js` has the exit-1 case.
2139
+ - **Stale node test: fixed.** With the reserve, node passes 13/13 at `slack=256`, and also at
2140
+ `slack=64` (C/F/W), so `b11-fixes.spec.js` now boots node at `slack=8`. It ends with exit 4, the used-up
2141
+ line and `wmn.error` "out of memory ... 1416 MiB maximum" (run with `WMD_WEB_B11FIX=1` and
2142
+ `WMD_WEB_IMAGE_FIXTURES` pointing at a tree that has the node packs).
2143
+ - **(3) late trap:** left as is, with a comment in `finish()`. Traps are counted until the 50 ms
2144
+ settle.
2145
+
2146
+ ### Runs (2026-10-05, Apple Silicon, `PORT=8797`, worktree vm.wasm via `WMD_VM_WASM`, each under the boot lock)
2147
+
2148
+ `test/e2e/w21.spec.js` boots python:3.12-slim (`page/wmn.html?checks=0&nonet=1&hold=1`) and runs
2149
+ 3 rounds of 30 control execs at once, then 10 in sequence. It checks every exit code and output, no
2150
+ trap, the reserve line, and that the memory did not grow while the vCPUs ran.
2151
+
2152
+ | Run | Result | Storm time | Canary before / after |
2153
+ |---|---|---|---|
2154
+ | Chromium, reserve, 5 boots | 5/5 pass, 0 bad | 13.3-13.9 s | 631 ms after |
2155
+ | Firefox, reserve | pass | 17.4 s | 673 ms |
2156
+ | WebKit, reserve | pass | 18.5 s | 673 ms |
2157
+ | Chromium, no reserve (`WMD_WEB_W21_BASELINE=1`), 2 boots | **2/2 trap** (77 and 100 execs failed); memory grew 22532 -> 30532 and 22532 -> 23631 pages under the vCPUs | | 657-660 ms |
2158
+ | Earlier session: Chromium with the reserve via `test/adversarial-b11` `cases=b11&conc=30` (build before the used-up message) | 5/5 pass; Firefox and WebKit 1/1 each | 23-28 s per boot | |
2159
+ | Earlier session: Chromium `cases=b11&conc=30` without the reserve (`WMD_HEAP_RESERVE=0`, or the vm.wasm built before it) | 3/3 trap | | |
2160
+
2161
+ Also run: `bun test test/unit` (270 pass, 2 skip, 0 fail). Native, with the installed `~/.wmd/bin/wmd-run`
2162
+ (main 9833319c): `guest/vm/test/smoke-wasm.sh` passed 13/13, and an arm64 `wmd.proof` machine boot (jit, 4 vCPUs)
2163
+ matched all of `test/expected/boot-arm64.txt` both with `WMD_HEAP_RESERVE=0` and with `=1` (canary 624 / 639 ms). `guest/vm`
2164
+ `build/test/test_heap_reserve`. That test covers the grow loop against a fake memory with a smaller
2165
+ maximum and a step limit, the bump (exact fit, refusals that leave the break alone, a reserve ending
2166
+ at 4 GiB), and 8 threads taking disjoint ranges that cover the reserve exactly.
2167
+
2168
+ ### Running
2169
+
2170
+ ```sh
2171
+ cd web
2172
+ # no guest: the engine probe in each browser
2173
+ PORT=8797 bunx playwright test w21-probe
2174
+ # under the boot lock, one browser per run (about 25 s each)
2175
+ PORT=8797 WMD_VM_WASM=../guest/vm/build/vm.wasm bunx playwright test w21.spec --project=chromium --repeat-each=5
2176
+ PORT=8797 WMD_VM_WASM=../guest/vm/build/vm.wasm WMD_WEB_W21_BASELINE=1 bunx playwright test w21.spec --project=chromium # A/B: must trap
2177
+ ```
2178
+
2179
+ ## W21 + W23 merged into wmd main (2026-10-06)
2180
+
2181
+ Branch `w21-w23-review` = main 6361975f + `--no-ff` wmd-web-w23 (9f7b4866) + w21-trap (822f3aaf, then
2182
+ 15945505 with the joint review's fixes) + the joint review's tests (`test/adversarial-w21w23/`, its
2183
+ `STATUS.md`). Conflicts: this README (W23 kept under P-B11, W21 after it) and
2184
+ `adversarial-b11/locked.sh` (w21's wait-and-retry floor of 50 GiB).
2185
+
2186
+ The joint review (STATUS.md) found R1 (major, native: the exit drain never settled while guest threads
2187
+ kept spawning), R2 (a guest's own stderr taken for vm.wasm's out-of-memory lines) and R3 (the node OOM
2188
+ test went stale once W21 stopped node running out at slack 256). All three were fixed on w21-trap
2189
+ 15945505 and rechecked here. W23 (no signed guest pointer or length left in `src/`; 300k-module
2190
+ `tableMinOf` fuzz in bun and node) had no findings.
2191
+
2192
+ Final runs on the combined tree (vm.wasm dfdb0d24 and wmd-run built in the worktree, port 8794, every
2193
+ step under the boot lock, P-core canary 606–748 ms around every step; logs in
2194
+ `test/adversarial-w21w23/out/`, round 1 in `out/r1/`):
2195
+
2196
+ | Run | Result |
2197
+ |---|---|
2198
+ | native spawnloop, no variable (`native-spawnloop.sh 15`) | 0/15 bad (was 5/15 slow with the new stderr line) |
2199
+ | native A2 programs vs main 6361975f, no variable (`native-a2.sh`) | blockread, blockwrite, pipewait, spawnloop, futexwait, exit3, threxit, rthreads, threads: same exit codes and times, stderr identical |
2200
+ | `a2.spec.js` (A2 programs in user mode, oomspoof, 5 arm64 proof boots at 4 vCPUs) | 10/10 in each of Chromium, Firefox, WebKit; no trap after an exit |
2201
+ | `bun test test/unit` | 316 pass, 2 skip, 0 fail (`9p-diff` passes) |
2202
+ | b11-fixes + w21-fixes + w23-pointers, no guest | 15/15 in each of C/F/W |
2203
+ | b11-fixes guest tests (`WMD_WEB_B11FIX=1`, node packs) | 10/10 C, 9/9 F and W (node at slack 8: exit 4, out of memory) |
2204
+ | wmn.spec python + node | 13/13 and 13/13 in each of C/F/W |
2205
+ | w21.spec 30-exec storm | Chromium 5/5 (12.3–13.1 s, 0 bad, no grow, 0 resets); Firefox 16.0 s and WebKit 12.9 s pass |
2206
+ | storm c60, `--ram 1024`, Chromium x3 (the W23 case) | 180/180 each, no grow |
2207
+ | storm c100, `--ram 1024`, Chromium x3 | 1 of 3 clean (300 execs). 2 of 3 used up the 1023 MiB reserve after 3 cache resets: the used-up line, `sbrk of 54460416 bytes; 37289984 ... left`, exit 4, reported as out of memory. No trap, no `readGuest` error (W23 is gone). Heap headroom at 100 execs at once is a limit, not a crash (W20). |
2208
+ | b05 (adversarial-b11, python) | the 30 execs pass (they trapped before W21); the 2 remaining FAILs are the same on main 6361975f |
2209
+ | slack 64, 30 mixed execs, Chromium (round 1) | clean stop: 4 cache resets, used-up line, exit 4, out of memory |
2210
+ | rthreads user mode x20, Chromium (round 1) | 0/20 traps |
2211
+ | native `smoke-wasm.sh` (`WMD_REQUIRE_WASM=1`) | 13/13 PASS |
2212
+ | native arm64 `wmd.proof` (jit, 4 vCPUs) | PROOF-DONE with the variable unset, `=0`, `=1` (no reserve line: the runner drops it) |
2213
+ | `make test` in guest/vm | pass |
2214
+ | `go test -short ./...` (round 1; no Go change since) | only the 2 pre-existing TestBuildKit failures |
2215
+
2216
+ Not fixed, by design or minor: a trap later than the 50 ms settle after an exit 0 is not counted (none
2217
+ seen in 15 proof boots); with the reserve every failed allocation stops the machine (exit 4) instead of
2218
+ cache.c's NULL fallbacks, which never saved it before.
2219
+
2220
+ ## P-B10: the guest's network through a relay
2221
+
2222
+ The proof (wmn PROOFS.md P-B10): outbound TCP from the browser guest through a WebSocket relay;
2223
+ `pip install requests` and `apk add curl` work in the browser. Adversarial: the relay drops
2224
+ mid-download, DNS fails, the guest tries to reach the relay's own host or private ranges (must be
2225
+ refused). Branch `wmd-web-b10`.
2226
+
2227
+ ### What runs
2228
+
2229
+ ```
2230
+ guest virtio-net ──net channel (rings)──▶ channel service ──MessagePort──▶ network Worker ──WebSocket──▶ relay ──TCP──▶ the internet
2231
+ (vm.wasm net.c) 4-byte BE length (NetSession) stack.js + relay-client.js relay.ts
2232
+ + Ethernet frame
2233
+ ```
2234
+
2235
+ - **The device.** vm.wasm's virtio-net already sends each Ethernet frame on a net channel with a
2236
+ 4-byte big-endian length, as natively (`--net PORT`, where gvisor-tap-vsock's netstack takes them,
2237
+ `internal/engine/netstack.go`). The page passes `--net 9 --mac 02:00:00:00:00:01` and a service
2238
+ `{port: 9, kind: 'net', relay: {url, tokenUrl}}`; the kernel command line no longer blacklists
2239
+ virtio-net, so the init runs udhcpc as natively.
2240
+ - **The channel service** (`NetSession`, `src/host/chan-services.js`) moves the bytes between the
2241
+ rings and the network Worker: the guest's bytes go out as they are read (at most 1 MiB the Worker
2242
+ has not processed), the Worker's frames are queued for the guest's ring and the Worker is told how
2243
+ much the ring took (`wrote`), which bounds what it sends (1 MiB). vm.wasm waits rather than drops
2244
+ when the guest has no receive buffer (`chan_pump.c`), so the link is reliable both ways.
2245
+ - **The network Worker** (`src/net/net-worker.js`) runs the stack and the relay connection, off the
2246
+ page and off the channel service. It answers, on the engine's addresses (192.168.127.0/24,
2247
+ gateway .1, guest .2, the netstack's gateway MAC):
2248
+ - ARP for the gateway only; DHCP with one static lease (router and DNS server: the gateway);
2249
+ - DNS on gateway:53: A queries go to the relay, which resolves and filters them; AAAA and other
2250
+ types get an empty answer at once (the guest has no IPv6 route); `gateway.wmd.internal`;
2251
+ - TCP: each connection the guest opens is a relay stream. The SYN-ACK goes back only once the
2252
+ relay has connected, so a refused connect is a RST for the SYN (ECONNREFUSED, curl exit 7);
2253
+ - ICMP echo to the gateway; other ICMP and UDP get "unreachable" (only TCP and DNS leave);
2254
+ - the gateway, other guest-network addresses and loopback/multicast never reach the relay.
2255
+ - **TCP** (`src/net/stack.js`) is the passive end of each connection, without congestion control
2256
+ (the link is reliable): it sends what the guest's window and the ring's room allow, takes back
2257
+ credit from the guest's ACKs, advertises a window of what the relay has not taken (256 KiB, window
2258
+ scale 3), and has retransmission on a timeout and three duplicate ACKs, zero-window probes, and an
2259
+ answer to the guest's own probes. A batch of the guest's segments goes to the relay in messages of
2260
+ up to 64 KiB (problem 2). **A stream ends with a FIN only when the remote closed cleanly; a
2261
+ stream that fails (the remote reset, the relay connection dropped, a relay limit) ends with a
2262
+ RST**, so a program never sees a cut-off body as a complete one.
2263
+ - **The relay** (`relay/relay.ts`, Bun, about 860 lines with its policy and tokens; the browser side, `src/net/`, about 1,600) carries one
2264
+ WebSocket per guest (protocol in `src/net/proto.js`: CONNECT, DATA, END, ABORT, CREDIT and DNS,
2265
+ with per-stream credit both ways, 256 KiB windows).
2266
+
2267
+ ### Security on the relay
2268
+
2269
+ - **Addresses, after resolution** (`relay/policy.ts`). Every connect is checked on the address the
2270
+ relay actually dials. A name is resolved by the relay once and the checked address is dialled (no
2271
+ second lookup), so DNS rebinding cannot swap it. Refused: 0/8, 10/8, 100.64/10, 127/8,
2272
+ 169.254/16 (metadata 169.254.169.254 named as such), 172.16/12, 192.0.0/24, 192.0.2/24,
2273
+ 192.88.99/24, 192.168/16, 198.18/15, 198.51.100/24, 203.0.113/24, 224/4, 240/4,
2274
+ 100.100.100.200; IPv6 ::, ::1, ::/96, 64:ff9b:1::/48, 100::/64, 2001::/23 (Teredo),
2275
+ 2001:db8::/32, 3fff::/20, 5f00::/16, fc00::/7 (fd00:ec2::254), fe80::/10, fec0::/10, ff00::/8;
2276
+ ::ffff:0:0:0/96 (IPv4-translated, SIIT); IPv4 inside IPv6 (mapped, NAT64 64:ff9b::/96, 6to4,
2277
+ ISATAP interface ids) is checked as IPv4; and **the relay host's
2278
+ own addresses** (every interface, refreshed every 30 s, plus `WMD_RELAY_SELF` for a public
2279
+ address behind NAT). Only strict dotted-decimal IPv4 is an address: `127.1`, `0x7f.0.0.1`,
2280
+ `2130706433` and `localhost` are refused as names before any lookup.
2281
+ - **DNS answers** leave refused addresses out; a name that resolves only to refused addresses
2282
+ gets REFUSED (curl exit 6).
2283
+ - **Tokens**: HMAC-SHA256 over `{sid, exp, uid, grp}` with the relay's secret, parsed strictly (one
2284
+ spelling verifies), presented as a WebSocket subprotocol (`t.<token>`), not in the URL. `sid` is the
2285
+ token (one live connection per sid), `uid` the guest identity (one guest machine of one page), `grp`
2286
+ the client the issuer saw (its address, or a user). An expired, forged or missing token is refused
2287
+ (401); `WMD_RELAY_ORIGINS` restricts page origins.
2288
+ - **Token issuance** (`relay/token.ts` `TokenIssuer`, used by `test/serve.ts` `/relay-token?instance=ID`;
2289
+ wmn's server would use it with its user id as the client): per client at most 60 tokens a minute
2290
+ and 20 page instances an hour, each token at most 1 h. The network Worker asks for a fresh token at
2291
+ each reconnect, with the same instance id.
2292
+ - **Caps per guest identity** (uid), kept across reconnects and fresh tokens for as long as a token
2293
+ for it is valid: 64 connections at once, 600 connects, 1,200 DNS lookups and 30 WebSocket opens a
2294
+ minute, 16 MiB/s down and 4 MiB/s up (token buckets with a 1 s burst, covering every byte, a new
2295
+ stream's first window too), 4 GiB in all (after that the identity is refused new sessions); one
2296
+ live session per identity (a newer one replaces it, code 4010); 8 live sessions per client (grp),
2297
+ 100 in all; port 25 refused; data past the granted credit ends the stream. All are `WMD_RELAY_*`
2298
+ settings (head of `relay/relay.ts`). The state is in memory: a relay restart resets it.
2299
+ - **An allowlist** (`WMD_RELAY_ALLOW_HOSTS`, `name[:port|port]`, ports 80 and 443 when none are
2300
+ given): the relay then resolves only those names and connects only to (address, port) pairs it gave
2301
+ out for them to this guest. The tests use it, so the relay connects only to dl-cdn.alpinelinux.org,
2302
+ pypi.org and files.pythonhosted.org.
2303
+ - **The Worker's stack never wedges** on abandoned connections: a handshake the guest does not finish
2304
+ is given up after 60 s (as Linux's `tcp_synack_retries`), at most 32 handshakes are pending at once,
2305
+ and a connection the guest closed is ended (RST, relay ABORT) after 120 s with no word from the
2306
+ remote.
2307
+ - **Test routes** (`WMD_RELAY_TEST_ROUTES`, `WMD_RELAY_TEST_HOSTS`) let the tests reach a local
2308
+ server as `bulk.wmd.test` (203.0.113.10:80 → 127.0.0.1:4572). The relay prints a warning when
2309
+ they are set; a public relay must never set them.
2310
+
2311
+ ### Numbers (2026-10-07, Apple Silicon, Playwright 1.63, `--mode jit`, 2 vCPUs, 1 GiB; relay and bulk server on this host)
2312
+
2313
+ ms and MB/s; one run each. Chromium and Firefox: the final run (09:37-09:48, canary 623 ms before,
2314
+ load average 8). WebKit: its last clean runs (run11, 08:56-09:00, canary 630 / 613 ms, the final stack code,
2315
+ for the transfers; run8, 08:17-08:25, before problem 1's fix, which changes only uploads, for the
2316
+ rest). A dash: no clean WebKit number. The WebKit pass of the final run (09:48-09:53) overlapped another session's guest and is
2317
+ not used for numbers (it passed 5/5). "Uncapped" is with the relay's bandwidth caps raised to
2318
+ 1 GiB/s; the others run with the default caps (16 MiB/s down, 4 MiB/s up, each with a 1 s burst).
2319
+
2320
+ | | Chromium | Firefox | WebKit |
2321
+ |---|---|---|---|
2322
+ | udhcpc (the lease from the network Worker; 15 s with no answer before) | 465 | 427 | 419 |
2323
+ | container answers an exec (from `bootWmd`) | 5,290 | 5,256 | 6,155 |
2324
+ | `apk add curl` (index + 9 packages from dl-cdn.alpinelinux.org) | 9,930 | 13,410 | 10,205 |
2325
+ | `curl` APKINDEX over HTTPS (474 KB): DNS / TCP connect / TLS done / total | 40 / 57 / 1,220 / 1,454 | 43 / 63 / 1,195 / 1,542 | – |
2326
+ | `pip install requests` (5 wheels from pypi.org / files.pythonhosted.org) | 57,949 | 90,395 | 52,774 |
2327
+ | `requests.get('https://pypi.org/simple/requests/')` in the guest (s) | 1.94 | 1.20 | – |
2328
+ | connect latency, guest `curl` `time_connect` to the local server (DNS + TCP via the relay), 5 runs | 36-40 | 41-50 | 37-40 (run11) |
2329
+ | relay-side connect (the relay to the local server) | 1.4-17.6 | 1.8-17.0 | 1.8-15.6 |
2330
+ | download 32 MiB, default cap (MB/s; sha256 equal) | 31.0 | 30.6 | 30.5 |
2331
+ | download 128 MiB, uncapped | **90.0** (1.49 s) | **45.6** (2.94 s) | **81.4** (1.65 s) |
2332
+ | upload 8 MiB, default cap (server's sha256 equal) | 8.3 | 8.4 | 8.5 |
2333
+ | upload 32 MiB, uncapped | **47.3** (0.71 s) | **13.4** (2.50 s) | **38.9** (0.86 s) |
2334
+ | download at a 512 KiB/s relay cap (4 MiB, python) | 506 KB/s | 515 KB/s | 517 KB/s |
2335
+ | TCP retransmissions / duplicate-ACK retransmissions (whole alpine run) | 0 / 0 | 0 / 0 | 0 / 0 |
2336
+
2337
+ - **Connect latency.** The guest's `time_connect` (about 40 ms) is mostly the guest itself: the DNS
2338
+ query for `bulk.wmd.test` and the TCP handshake through the emulated kernel. The relay reaches the
2339
+ local server in 1-2 ms once warm (the first connects of a session, 13-18 ms, include its startup).
2340
+ - **Throughput.** The default caps bind on the 32 MiB transfers (16 MiB/s plus a 16 MiB burst gives
2341
+ 32 MiB in about 1 s). Uncapped, the path runs at 45-90 MB/s down and 13-47 MB/s up; Firefox is the
2342
+ slowest both ways. The guest's own work (virtio, its TCP, curl writing the file) is in these numbers.
2343
+ - **Package installs** are dominated by the guest (unpacking, `pip` itself under emulation), not the
2344
+ network: the bytes are under 3 MB for each.
2345
+
2346
+ ### Results
2347
+
2348
+ Every run under the boot lock, `PORT=8790`, relay on 4571, bulk server on 4572.
2349
+
2350
+ | | Chromium | Firefox | WebKit |
2351
+ |---|---|---|---|
2352
+ | pass: alpine `apk add curl`, then curl over HTTPS, throughput | pass | pass | pass |
2353
+ | pass: python `pip install requests`, then `requests.get` over HTTPS | pass | pass | pass |
2354
+ | adversarial: refused addresses, DNS failures | pass | pass | pass |
2355
+ | adversarial: relay drop / kill / restart / upstream reset mid-download, apk across a relay kill | pass | pass | pass |
2356
+ | caps: connections at once, bandwidth | pass | pass | pass |
2357
+ | `bun test test/unit test/adversarial-b11/yamux-adv.test.js` | 386 pass, 2 skip, 0 fail | | |
2358
+
2359
+ **Refused, from the guest** (`curl -m 20 http://ADDR:PORT/`): 169.254.169.254 (metadata),
2360
+ 100.100.100.200 (metadata), 10.0.0.1 and 172.17.0.1:2375 (private), 203.0.113.10:81 (reserved; the
2361
+ test route covers only port 80), and this host's own addresses on the relay's port (its LAN address
2362
+ 192.168.0.9 and its Tailscale address 100.119.72.125: "self"). Each failed at once, in 330-500 ms with
2363
+ curl exit 7, and the relay's log names the reason (`blocked`, "... is metadata"). 0.0.0.0 and
2364
+ 224.0.0.1 never left the guest. Names that resolve to refused addresses (`127.0.0.1.nip.io`,
2365
+ `169.254.169.254.nip.io`, public DNS) got REFUSED from the relay: curl exit 6, about 430 ms. The
2366
+ gateway's port 80 is refused by the Worker; another guest-network address fails after ARP (3.5 s,
2367
+ EHOSTUNREACH).
2368
+
2369
+ **DNS failures.** A real NXDOMAIN (`no-such-host.invalid`): curl exit 6 in under 0.6 s. The relay
2370
+ restarted with a dead resolver (`WMD_RELAY_DNS=127.0.0.1:4579`): SERVFAIL, curl exit 6 in 0.4 s; with
2371
+ the relay restarted normally the next download worked. The guest kept running throughout.
2372
+
2373
+ **Mid-download failures** (a 4 MiB body sent at 256 KiB/s; each case cut at about 5 s):
2374
+
2375
+ | | Chromium | Firefox | WebKit |
2376
+ |---|---|---|---|
2377
+ | the page drops the relay WebSocket | curl 56, 1.25 MB kept, 319 ms after | 56, 331 ms | 56, 1,298 ms |
2378
+ | relay killed (SIGKILL) and started again | 56, 270 ms | 56, 337 ms | 56, 1,358 ms |
2379
+ | relay restarted (SIGTERM) | 56, 287 ms | 56, 384 ms | 56, 1,349 ms |
2380
+ | the remote resets mid-body | 56 | 56 | 56 |
2381
+ | the next download (`curl \| sha256sum`) | equal | equal | equal |
2382
+ | `apk add python3` with the relay (256 KiB/s) killed during the package downloads | apk exit 1: "python3-pycache-pyc0: IO ERROR", "1 error"; the rerun installed that one package, `python3 -c 'print(6*7)'` printed 42 | same | same |
2383
+
2384
+ Never a curl exit 0 with other bytes: a failed stream is a RST to the guest, never a FIN. In WebKit
2385
+ (run8) a failure took about 1.3 s to reach the guest's stream, against about 0.3 s in the other two;
2386
+ in the overlapped final run the relay kills took 0.27-0.38 s there too. The cause was not checked.
2387
+
2388
+ ### Problems found, and their causes
2389
+
2390
+ 1. **Uploads stalled in Firefox and WebKit** (fixed). A 32 MiB upload ran at 26.8 MB/s in Chromium
2391
+ but at 0.59 MB/s in Firefox and 1.05 MB/s in WebKit, with every byte arriving (sha256 equal).
2392
+ Cause: each guest segment (1,460 bytes) became its own WebSocket message, and the relay answered
2393
+ each socket write with its own CREDIT: about 23,000 messages each way for 32 MiB, against a few
2394
+ hundred 64 KiB messages for the same download. Sending a WebSocket message from a Worker costs
2395
+ far more per message in Firefox and WebKit. Fix: the stack sends a batch of segments to the
2396
+ relay in messages of up to 64 KiB, and the relay sends one CREDIT per stream per turn. After it:
2397
+ Firefox 12.6 MB/s, WebKit 38.9 MB/s, Chromium 48 MB/s (test: `net-stack.test.js` "a batch of
2398
+ the guest's segments goes to the relay as few large messages"). Two causes ruled out on the
2399
+ way: `Atomics.waitAsync` woken by the same Worker's notify takes 11-15 ms in all three browsers
2400
+ (a probe, not kept), and answering the guest's zero-window probes (a real gap, now fixed and
2401
+ tested, but not this stall).
2402
+ 2. **Bun's node:net and a remote reset.** In the relay's own tests, a bulk server written with
2403
+ `drain` events sometimes never sent its reset; the relay was not at fault (pausing and resuming
2404
+ a socket in Bun 1.4.2 does report ECONNRESET). The test server writes with write callbacks.
2405
+ 3. **Bandwidth caps trickled one byte at a time.** With the bucket empty, `take()` returned the
2406
+ few bytes refilled since the last call, so the relay sent DATA messages of 1-10 bytes. It now
2407
+ waits until 16 KiB (or the whole request) is there.
2408
+
2409
+ ### What remains
2410
+
2411
+ - **IPv6.** The guest gets no IPv6 route, and the relay resolves A records only.
2412
+ - **UDP** other than DNS (NTP, QUIC, DNS over TCP from the guest) is not carried.
2413
+ - **Inbound connections** (published ports) are not possible in the browser.
2414
+ - **Native comparison.** The relay path was not measured against native wmd's netstack here.
2415
+ - **Deployment** (ROADMAP 14.8): a hosted relay needs TLS (`wss:`), a token service in wmn's
2416
+ server, and its public address in `WMD_RELAY_SELF`.
2417
+
2418
+ ### Running
2419
+
2420
+ ```sh
2421
+ cd web
2422
+ bun test test/unit/net-stack.test.js test/unit/net-chan.test.js test/unit/net-relay.test.js # no guest
2423
+ # under the boot lock, one browser per run (about 5-6 min each); needs the alpine and python fixtures:
2424
+ # go run ./web/test/fixtures-gen image -ref docker.io/library/alpine:3.20 -out web/test/fixtures/images/alpine
2425
+ # go run ./web/test/fixtures-gen image -ref docker.io/library/python:3.12-slim -out web/test/fixtures/images/python
2426
+ export WMD_RELAY_SECRET=$(openssl rand -hex 16) WMD_RELAY_URL=ws://127.0.0.1:4571/relay PORT=8790
2427
+ test/locked.sh bunx playwright test net.spec --project=chromium # WMD_WEB_NET_CASES=pass,adversarial,restart,caps
2428
+ # a relay of your own:
2429
+ WMD_RELAY_SECRET=... bun relay/relay.ts
2430
+ ```
2431
+
2432
+ The spec starts the relay (`bun relay/relay.ts`, port 4571) and a bulk HTTP server (port 4572)
2433
+ itself, and kills them by PID. Results: `net-results/<browser>-<case>.{json,stdout.log,stderr.log}`.
2434
+
2435
+ ## P-B8: two vCPUs on Workers sharing memory (branch `wmd-web-b8`, in progress)
2436
+
2437
+ Status (2026-10-07): **partly done; no passing proof run yet.**
2438
+ - Done: the stall hook (`bootWmd({stallHook})`, `vm.stallThread(tid, ms)`, `vm.stats().stall`; cells 9-13
2439
+ and bytes 112-127 in `layout.js`; checked in `worker.js`'s import guard only when on); case `b8` in
2440
+ `page/chan.js` (alpine:3.20 + build-base from one pack, chroot from the proof initramfs, `test/b8/run.sh`
2441
+ phases); fixtures `test/b8/build.sh` (Docker: build-base pack, static `b8lock` LL/SC and LSE builds,
2442
+ Lua 5.4.7); `test/b8/b8lock.c` (mutex, spin, atomic, cond, futex, work); specs `b8.spec.js` (opt-in
2443
+ `WMD_WEB_B8=1`) and `b8-hook.spec.js` (synthetic `b8-synth/stall.wat`); native reference
2444
+ `test/b8/native.sh`; `test/b8/pw.sh` (waits for the lock with no "playwright test" in its command line).
2445
+ - Evidence so far: one Chromium boot, 1 vCPU, with 1024 MiB of slack (canary 612/624 ms). nproc, getconf
2446
+ and cpuinfo all reported 1, and cpuinfo lists `atomics` (LSE). The first `make` of Lua then **used up
2447
+ vm.wasm's 1023 MiB heap reserve** (4 block-cache resets, then exit 4, reported as out of memory). gcc's
2448
+ cc1 needs more than 1 GiB of translator heap. The spec now defaults to 2560 MiB of slack
2449
+ (`B8_SLACK`); that run never got the lock.
2450
+ - Not started: the 1/2/4-vCPU runs in Chromium/Firefox/WebKit, the speed-ups, the lock contention
2451
+ numbers, the stall runs (RCU stall at >21 s: `CONFIG_RCU_CPU_STALL_TIMEOUT=21`; the kernel has no
2452
+ soft-lockup detector), the native reference, `b8-hook.spec.js` runs, and the unit tests under the lock.
2453
+ - Note for the stall work: `machine_cache_flush_all` (world-stop) waits for every running vCPU to park, so a
2454
+ stopped vCPU Worker would stall any other vCPU that flushes the whole cache.
2455
+ - Run: `PORT=8774 WMD_VM_WASM=../guest/vm/build/vm.wasm WMD_WEB_B8=1 test/locked.sh test/b8/pw.sh b8.spec --project=chromium`.
2456
+
2457
+ ## P-B7: packs from a static host and the registry proxy
2458
+
2459
+ **Status: done and reviewed; results, root causes and limits are in `test/b7/README.md`.** Branch `wmd-web-b7`. (The text below is the design record; where it says "not done", the later runs in `test/b7/README.md` did it.)
2460
+
2461
+ Built: `src/registry/ref.js` (refs, platforms, the layout: `/blobs/sha256/<hex>` packs, immutable;
2462
+ `/idx/<registry>/<repo>/<tag>/<os>-<arch>.json` per image@platform), `page/sw.js` (packs cached only
2463
+ after a sha256 check; everything else network first, cache when offline), `page/b7.js` (index -> packs
2464
+ -> container run over the control channel), `registry/` (Bun host + proxy `server.ts`; Docker Hub client
2465
+ with anonymous tokens, host allowlist, streamed digest/size checks `upstream.ts`; `pull.ts`: index ->
2466
+ linux/arm64, 4 parallel verified downloads, layers converted in order by wmd's `union.PackLayer`
2467
+ (`wmdpack/`), chain cache, gzip-6 copies served with Content-Encoding, disk floor; `export.ts` from
2468
+ `~/.wmd/data` (read only); `testreg.ts` + `testimages.ts`: a Docker-free registry with 50-layer,
2469
+ multi-arch, nested, lying, corrupt, slow, cut and size-lie images).
2470
+
2471
+ Decision: convert on the proxy, not in the browser. Packs are byte-identical to P-B11's fixtures-gen
2472
+ packs (same converter); gzip-6 packs are within 2% of the registry's layer.gz on the wire (48.5 vs
2473
+ 47.6 MB, alpine + python); conversion runs once per layer chain for every visitor. In-browser gunzip +
2474
+ tar walk of python's layers (148.7 MB tar) is 275 / 297 / 231 ms (C/F/W), vs 717 ms for the Go
2475
+ converter, so a browser converter is affordable in CPU, but would be a second implementation of
2476
+ fromtar.go's whiteout, opaque, hard-link copy-up and symlinked-parent rules.
2477
+
2478
+ Evidence so far:
2479
+ - `bun test test/unit/b7-registry.test.ts`: 17 pass (no guest): refs/platforms, allowlist and foreign
2480
+ token realm/redirect refused, 50-layer union checked with wmd's Go union, multi-arch/nested/no-arm64/
2481
+ lying index, corrupt layer refused with no index or chain or tmp left, size lie, cut transfer then
2482
+ retry, disk floor, concurrent pulls joined, static headers.
2483
+ - No-guest browser tests pass in Chromium, Firefox and WebKit: the SW caches a pack and a second visit
2484
+ with the host killed gets it from the cache (`probe`); a tampered pack is refused by the SW and not
2485
+ cached; corrupt layer refused by the proxy (`do not match`); the gunzip probe.
2486
+ - Proxy against Docker Hub (curl, no browser): busybox 1.1 s (1.9 MB), ruby:3.3-slim 2.56 s (69.8 MB,
2487
+ 5 layers; download 1.63 s, convert 0.99 s; packs 176 MB, gz 72.7 MB). Export from wmd's store:
2488
+ python 0.85 s (convert 717 ms, 207 MB/s).
2489
+ - Guest boots (Chromium, one run, canary 617/619 ms): alpine and python ran from the static host
2490
+ through the SW (alpine main exit at 5.2 s, python at 7.6 s from navigation; exit 0, outputs right);
2491
+ the run stopped at a test regex (apk warnings on stderr), fixed in the spec, not rerun.
2492
+
2493
+ Not done: the offline second-visit boot, the Docker Hub images in the browser, the 50-layer,
2494
+ multi-arch and proxy-killed-mid-pull browser tests (all written in `test/b7/b7.spec.js`), Firefox and
2495
+ WebKit guest runs, the numbers table. The boot lock was held by other sessions most of the session
2496
+ (goprof.sh waits for no `pgrep -f "playwright test"` while holding the lock, which a waiting
2497
+ `locked.sh bunx playwright test ...` matches; this suite queues through `test/b7/run.sh`).
2498
+
2499
+ ```sh
2500
+ go build -o web/registry/.bin/wmdpack ./web/registry/wmdpack
2501
+ cd web && bun test test/unit/b7-registry.test.ts
2502
+ test/locked.sh test/b7/run.sh --project=chromium # then firefox, webkit (ports 8780-8783)
2503
+ ```
2504
+
2505
+ ## W24: start-up events for wmn's engine start-up screen
2506
+
2507
+ `bootWmd({onEvent})` also gets the start-up events wmn's start-up screen draws (wmn D-211, ROADMAP 7.14), from
2508
+ `src/host/boot-events.js`. `atMs` is milliseconds from the boot's `t0` (page clock, `timings.t0`), `tookMs` the stage's own
2509
+ duration, sizes are bytes. Progress-like events (download and restore progress, workers progress, `heap`, `layer`) go out at
2510
+ most 4 times a second per kind (a download's kind is its file, a layer's its port): the latest waits, and a `start` or `done`
2511
+ of the same kind replaces it. Nothing is sent after `exit`. The events that were there before (`chan`, `pool`, `panic`,
2512
+ `throttle`, `thread`, `phase`, `host-calls`) are unchanged.
2513
+
2514
+ | event | when | fields |
2515
+ |---|---|---|
2516
+ | `{t:'boot', stage:'download', state:'progress'\|'done', file}` | per file, as bytes come; `done` when it is in | `bytes`, `total` (Content-Length; null when absent or content-encoded), `cached` (null on progress), `port` (packs), `tookMs` (the fetch's own time) on done; a file the caller passed as bytes: `given: true`, `cached: null`; a file vm.restore() reused: `cached: true` |
2517
+ | `{t:'boot', stage:'download', state:'done'}` (no `file`) | everything in | `fileBytes`, `layerBytes` (`timings`), `tookMs` (`timings.allDownloadMs`) |
2518
+ | `{t:'boot', stage:'compile', state:'start'\|'done'}` | around `WebAssembly.compile` | `moduleBytes`, `reused` (vm.restore() reuses the Module), `tookMs` = `timings.compileMs` |
2519
+ | `{t:'boot', stage:'memory', state:'done'}` | shared memory, control block, JIT store, rings made | `initialBytes`, `maxBytes`, `ringBytes` (stdio + channel block), `tookMs` |
2520
+ | `{t:'boot', stage:'workers', state:'progress'\|'done'}` | each Worker change; `done` once every Worker started before the vCPU is ready | `workers: [{id, role, state, atMs, tid?}]`, `vcpus`, `jitShared`, `tookMs` on done; after the end every Worker is `ended` |
2521
+ | `{t:'boot', stage:'kernel', state:'start'\|'done'}` | `start`: vm.wasm's `_start` began; `done`: the first console (stdout) byte. Not sent for a restored machine | `tookMs` |
2522
+ | `{t:'boot', stage:'restore', state:'progress'\|'done'}` | a base snapshot coming in (`restoreLazy`), every 250 ms; `done` when its memory is written | `bytes`, `total`, `source`: `'opfs'`, `'network'` or `'memory'` (vm.restore()), `used` (false with `fallback`: the snapshot was refused and the kernel boots instead), `tookMs` from `t0` to the memory written (`timings.restore.pagesWrittenAt`; a base snapshot's download may have begun before the boot). A base snapshot that falls back from the cache to the network starts its `bytes` again with `source: 'network'` |
2523
+ | `{t:'boot', stage:'control', state:'done'}` | the guest's yamux control session first up | `tookMs` from the vCPU start |
2524
+ | `{t:'heap'}` | the shared memory's size changed (checked every 250 ms) | `bytes`, `maxBytes` |
2525
+ | `{t:'layer'}` | a 9P service's upper first seen, then on each change of 1 MiB or more | `port`, `usedBytes`, `capBytes` (null: no cap), `kind: 'memory'` |
2526
+ | `{t:'exit'}` | the machine ended (the same as `vm.exited`) | `code` (-1 stopped, -2 no progress, -3 panic, -4 service died, else the guest's), `reason`, `oom` |
2527
+
2528
+ Worker ids and roles: `vcpu-0` (`vcpu`, the main Worker), `pool-<n>` (`pool`; `tid` is the guest thread it runs now:
2529
+ the other vCPUs and vm.wasm's own threads, which wmd-web cannot tell apart), `chan-service`, `net`, `jit-hub`.
2530
+
2531
+ The Workers stage has no `done` when a Worker failed (the list says which); every vCPU after vCPU 0 runs on a `pool` Worker.
2532
+
2533
+ What is not measured, so not sent: a `container` stage (the container's init is behind the control session; runtime-web
2534
+ knows when its run answers), the heap vm.wasm
2535
+ uses inside its memory (with W21's heap reserve the memory is at its maximum from the start, so `heap.bytes` is then the
2536
+ maximum). `cached` comes from Resource Timing: `transferSize` 0 with a body is true, bytes transferred is false; Firefox and
2537
+ WebKit give a service worker's cached response zero sizes (`page/w24/`, `test/e2e/w24-cached.spec.js`), so a zero-size
2538
+ entry is true only for a same-origin file that had bytes; null when the browser has no entry (its timing buffer is full).
2539
+
2540
+ A storage failure is an error named `QuotaExceededError`: a checkpoint store's `CheckpointError` with code `quota`, and the
2541
+ base snapshot cache's (`info.cacheWrite.name`).
2542
+
2543
+ ```sh
2544
+ bun test test/unit/w24-boot-events.test.js
2545
+ PORT=8796 bunx playwright test test/e2e/w24-cached.spec.js # no guest, three browsers
2546
+ PORT=8795 ~/.wmd/bootlock/bootlock.sh run --max 1800 -- test/b9/run.sh b9w24 --project=chromium # boots python
2547
+ ```
2548
+
2549
+ ## 2.7: `wmd image export --format web` (D-477)
2550
+
2551
+ wmd image export --format web -o ./site alpine:latest python:3.12-slim
2552
+
2553
+ writes a folder any static file server can host: the images as packs in the static host layout
2554
+ (`blobs/sha256/`, `idx/...`, as `src/registry/ref.js` names them), the guest's `vm.wasm`, kernel
2555
+ and init (`guest/arm64/`), a copy of `src/` (`wmd-web/`, embedded in the wmd binary by
2556
+ `web/embed.go`), and the runner page from `export-page/` (`index.html?ref=...&cmd=...`). The page
2557
+ checks every guest file against `wmd-web.json`'s sha256 and every pack against its name before it
2558
+ boots. `_headers` carries the COOP/COEP/CORP/CSP for hosts that read it; on a host that sends no
2559
+ headers the page registers `coi-sw.js` and reloads once. The packs are byte-identical to
2560
+ `wmd pull`'s (`test/export/byte-identity.py`), and each index equals the registry proxy's
2561
+ (`test/export/compare-proxy.ts`). Browser gate: `test/locked.sh bunx playwright test -c test/export`.
2562
+
2563
+ ## Releases: `make web-release` (2.15, D-482)
2564
+
2565
+ A release of wmd-web is the version in `package.json`, the local git tag `wmd-web-v<version>` on the
2566
+ commit, and `~/.wmd/releases/wmd-web-<version>.tgz` with its `.sha256`, made from a clean main commit
2567
+ by `make web-release` (`scripts/web-release.py`). Nothing is published and nothing is committed by it:
2568
+ bump `version` in a commit first (minor for a change to the public API, patch otherwise).
2569
+
2570
+ The tarball is in `npm pack` layout, so it can go to a registry unchanged later:
2571
+
2572
+ package/package.json package/README.md package/src/** package/export-page/** from git, at the commit
2573
+ package/guest/arm64/vm.wasm Image Image-4k initramfs.cpio.gz VM= (default guest/vm/build/vm.wasm), GUEST= (default ~/.wmd/bin/guest/arm64)
2574
+ package/RELEASE.json version, commit, previous release, the D numbers since it, guest/vm's tree hash, the sha256 of every file
2575
+
2576
+ The same commit and guest files give the same bytes. `python3 -I scripts/web-release.py --verify
2577
+ FILE.tgz` checks the `.sha256` beside the file (required), that the tarball holds only regular files
2578
+ named `package/<clean path>`, once each, mode 0644, owner 0, nothing after the end, and every file
2579
+ against RELEASE.json; an installer should run it, or do the same checks, before it uses a tarball. A released version is never rewritten, and a
2580
+ tag that already marks another commit is refused. `make web-release-test` tests all of this on a
2581
+ throwaway clone.
2582
+
2583
+ ## Guests above 1536 MiB: RAM in a memory64 of its own (D-533)
2584
+
2585
+ vm.wasm's memory 0 is 4 GiB at most and also holds the emulator's heap, so `--ram` stopped at 1536 MiB
2586
+ (`MAX_BROWSER_RAM_MIB`). With a shared memory64 (Chrome 133, Firefox 134; `haveMemory64()`) a bigger `--ram`
2587
+ now boots the **flat layout**: `bootWmd` makes a second shared memory of exactly `--ram` (at most 16 GiB, the
2588
+ engines' memory64 limit), passes `WMD_RAM_FLAT=1` to vm.wasm and hands the memory to every Worker.
2589
+
2590
+ - Translated code imports it as `env.win` and reads guest RAM with 64-bit addresses (`host/jit.js`).
2591
+ - vm.wasm's C code cannot address a second memory. It calls `wmd.ram_rw` and `wmd.ram_cas`, two exports of a
2592
+ 200-byte wasm module made in `host/ram-mem.js` (wasm to wasm: no BigInt, no JS on a page-table walk).
2593
+ - Memory 0 is vm.wasm's 384 MiB + a 512 MiB stand-in the board loads the kernel through + 1536 MiB of heap
2594
+ slack (`flatMaxPages`), whatever the guest's RAM. Leave `maxPages` out for such a boot.
2595
+ - `flatRam: true|false` forces the layout on or off; unset, it is chosen for `--ram` above 1536 when the engine
2596
+ has memory64 and `allowLargeRam` is not set. Safari has no memory64: the old limit and its message stay.
2597
+ - No checkpoint yet: `checkpoint: true` is ignored and `restore` / `restoreLazy` are refused with a message
2598
+ (a checkpoint holds memory 0 only).
2599
+
2600
+ Measured in Chromium (page/b7.js, `ram=`): 4096 shows `Mem: 3976`; 8192 with 4 vCPUs held 5366 MB in six
2601
+ processes with every byte checked; the same work at 1536 (small) and 4096 (flat) took 35 s and 39 s.
2602
+ Tests: `test/unit/ram-mem.test.js` (bun has no memory64: the helper's part runs under node).