@threadedstack/wmd-web 0.0.0-stage → 0.1.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2601 -2
- package/RELEASE.json +123 -0
- package/export-page/coi-sw.js +20 -0
- package/export-page/index.html +17 -0
- package/export-page/run.css +10 -0
- package/export-page/run.js +236 -0
- package/guest/arm64/Image +4 -0
- package/guest/arm64/Image-4k +4 -0
- package/guest/arm64/initramfs.cpio.gz +0 -0
- package/guest/arm64/vm.wasm +0 -0
- package/package.json +36 -4
- package/page/sw.js +381 -0
- package/relay/control.ts +139 -0
- package/relay/engine.ts +787 -0
- package/relay/hosted.ts +660 -0
- package/relay/keys.ts +53 -0
- package/relay/local-tls.ts +121 -0
- package/relay/lock.ts +81 -0
- package/relay/logfile.ts +138 -0
- package/relay/main.ts +642 -0
- package/relay/policy.ts +241 -0
- package/relay/relay.ts +266 -0
- package/relay/token.ts +116 -0
- package/relay/v1.ts +107 -0
- package/relay/wisp.ts +615 -0
- package/src/base.js +325 -0
- package/src/boot.js +2037 -0
- package/src/calib-worker.js +40 -0
- package/src/fs/9p/codec.js +303 -0
- package/src/fs/9p/server.js +521 -0
- package/src/fs/errno.js +43 -0
- package/src/fs/graph.js +380 -0
- package/src/fs/overlay.js +1018 -0
- package/src/fs/pack-write.js +165 -0
- package/src/fs/pack.js +427 -0
- package/src/fs/seed.js +22 -0
- package/src/fs/svc-snapshot.js +65 -0
- package/src/fs/upper-pack.js +112 -0
- package/src/host/base-cache.js +195 -0
- package/src/host/base-file.js +328 -0
- package/src/host/base-io.js +93 -0
- package/src/host/boot-events.js +209 -0
- package/src/host/chan-service.js +300 -0
- package/src/host/chan-services.js +689 -0
- package/src/host/chan.js +655 -0
- package/src/host/ckpt-file.js +395 -0
- package/src/host/ckpt-io.js +207 -0
- package/src/host/ckpt-point.js +160 -0
- package/src/host/ckpt-save.js +64 -0
- package/src/host/ckpt.js +434 -0
- package/src/host/clock.js +111 -0
- package/src/host/ctl.js +778 -0
- package/src/host/errno.js +18 -0
- package/src/host/fs.js +167 -0
- package/src/host/gcnudge.js +83 -0
- package/src/host/i64shim.js +81 -0
- package/src/host/jit-cache.js +341 -0
- package/src/host/jit-hub-core.js +208 -0
- package/src/host/jit-hub.js +93 -0
- package/src/host/jit-snap.js +226 -0
- package/src/host/jit.js +1903 -0
- package/src/host/layout.js +100 -0
- package/src/host/monitor.js +110 -0
- package/src/host/p9mount.js +27 -0
- package/src/host/ram-mem.js +125 -0
- package/src/host/stdio.js +143 -0
- package/src/host/threads.js +109 -0
- package/src/host/wasi.js +335 -0
- package/src/host/wmd-imports.js +71 -0
- package/src/host/yamux.js +387 -0
- package/src/image-files.js +154 -0
- package/src/net/dhcp-dns.js +126 -0
- package/src/net/net-worker.js +172 -0
- package/src/net/packet.js +227 -0
- package/src/net/peer.js +230 -0
- package/src/net/proto.js +189 -0
- package/src/net/relay-client.js +226 -0
- package/src/net/stack.js +867 -0
- package/src/net/switch.js +160 -0
- package/src/net/wisp-client.js +570 -0
- package/src/net/wisp-transport.js +196 -0
- package/src/net/wisp.js +511 -0
- package/src/registry/client.js +176 -0
- package/src/registry/converter.js +68 -0
- package/src/registry/hub.js +221 -0
- package/src/registry/oci.js +76 -0
- package/src/registry/pack-worker.js +70 -0
- package/src/registry/pull.js +258 -0
- package/src/registry/ref.js +110 -0
- package/src/registry/wasi-mem.js +355 -0
- package/src/worker.js +315 -0
- package/wmdpack.wasm +0 -0
package/README.md
CHANGED
|
@@ -1,3 +1,2602 @@
|
|
|
1
|
-
#
|
|
1
|
+
# wmd-web
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
The browser host for wmd's guest `vm.wasm`. It is the same `vm.wasm` that `wmd-run`
|
|
4
|
+
runs (sha1 `5ac0e005134bfd198f01a92721324fe37bb93758`), running in Web Workers on one
|
|
5
|
+
shared `WebAssembly.Memory`, with a JavaScript WASI preview1 subset, `wasi.thread-spawn`
|
|
6
|
+
over a Worker pool, and the `wmd.*` imports stubbed for interpreter mode.
|
|
7
|
+
|
|
8
|
+
Status: **P-B1 passes** in Chromium, Firefox and WebKit (Playwright 1.63). The arm64
|
|
9
|
+
kernel and the proof initramfs boot in interpreter mode, and every boot-gate regex of
|
|
10
|
+
`guest/vm/test/expected/boot-arm64.txt` matches the page console, with exit 0. The
|
|
11
|
+
amd64 kernel also passes in Chromium.
|
|
12
|
+
|
|
13
|
+
**P-B2 passes** in all three: the guest's own JIT (`--mode jit`) runs in the browser
|
|
14
|
+
through the JIT door (`src/host/jit.js`). The same proof boot reaches `PROOF-DONE` 6.7x
|
|
15
|
+
(Chromium), 8.1x (Firefox) and 7.3x (WebKit) sooner than P-B1, and the total run is within 3–11% of
|
|
16
|
+
native `wmd-run --mode jit` wall time. See [P-B2: the JIT door](#p-b2-the-jit-door).
|
|
17
|
+
|
|
18
|
+
**P-B3 passes** in all three: `wmd.chan_*` run on shared-memory rings with a service
|
|
19
|
+
Worker. A 9P round trip as a guest process sees it (`fstat` on a 9P mount) takes 89–187 µs
|
|
20
|
+
(native `wmd-run` with wmd's Go 9P server over TCP: 104–207 µs), and the guest never waited
|
|
21
|
+
for the host while writing 50 MiB into the control channel. See [P-B3](#p-b3-channels-on-shared-memory-rings).
|
|
22
|
+
|
|
23
|
+
**P-B4, first integration:** alpine:3.20 runs from its pack over the JS 9P server in all
|
|
24
|
+
three browsers, both from the proof initramfs (chroot) and **as a container through wmd's
|
|
25
|
+
Go init and crun**, as the engine boots it: `ls -R /` finishes, `apk info` lists its 14
|
|
26
|
+
packages, and the init writes `bundle/exitcode` back over 9P. See [P-B4](#p-b4-alpine-from-packs).
|
|
27
|
+
|
|
28
|
+
**P-B11, the wmn agent in the browser's guest (core):** the container kernel and Go init boot
|
|
29
|
+
with the engine's own command line (`wmd.control=vport`); the page drives the init over the
|
|
30
|
+
control channel with a JS yamux client (ROADMAP 2.2b); python:3.12-slim and node:22-slim run
|
|
31
|
+
from one pack per layer, with the agent as PID 1 (`/.wmn/agent init`) and `/.wmn/agent serve`
|
|
32
|
+
exec'd beside it; `hello_reply` and the 13 P-K2 checks pass in Chromium, Firefox and WebKit,
|
|
33
|
+
with failures on a loaded host (Firefox, two runs). See [P-B11](#p-b11-the-wmn-agent-in-the-browsers-guest).
|
|
34
|
+
|
|
35
|
+
**P-B10 passes** in all three: the guest has a network. Its virtio-net frames go to a TCP/IP
|
|
36
|
+
stack in a network Worker, which carries its TCP connections and DNS over one WebSocket to a
|
|
37
|
+
small relay (`relay/relay.ts`). `apk add curl` in alpine:3.20 and `pip install requests` in
|
|
38
|
+
python:3.12-slim complete, and curl and requests then fetch over HTTPS. The relay refuses
|
|
39
|
+
private, loopback, link-local, metadata, reserved and its own host's addresses after its own
|
|
40
|
+
DNS resolution, needs a per-session token, and caps connections and bandwidth. See
|
|
41
|
+
[P-B10](#p-b10-the-guests-network-through-a-relay).
|
|
42
|
+
|
|
43
|
+
**P-B7, packs from a static host and the registry proxy:** packs are content addressed
|
|
44
|
+
(`/blobs/sha256/<hex>`, immutable) with one small index per image@platform; a service worker
|
|
45
|
+
caches them after a hash check, so a second visit boots with the server gone; a Bun proxy
|
|
46
|
+
(`registry/`) pulls any public Docker Hub image (anonymous token, index -> linux/arm64,
|
|
47
|
+
sha256 checked while streaming) and converts its layers with wmd's own `union.PackLayer`.
|
|
48
|
+
See [P-B7](#p-b7-packs-from-a-static-host-and-the-registry-proxy).
|
|
49
|
+
|
|
50
|
+
The interface this host implements is in [docs/host-interface.md](docs/host-interface.md).
|
|
51
|
+
|
|
52
|
+
## Layout
|
|
53
|
+
|
|
54
|
+
| File | What it does |
|
|
55
|
+
|---|---|
|
|
56
|
+
| `src/boot.js` | Page API `bootWmd({vmUrl, files, args, env, ...})`. Downloads the files, compiles vm.wasm, creates the shared memory, the control block and the Worker pool, then runs `_start` in a Worker. Reports stdout/stderr and timings. |
|
|
57
|
+
| `src/worker.js` | One Worker = one wasm thread. Role `main` runs `_start` (tid 1). Role `pool` parks on a slot and, for each spawn, instantiates vm.wasm again (same memory, a new 155-entry table) and calls `wasi_thread_start(tid, arg)`. |
|
|
58
|
+
| `src/host/wasi.js` | The 18 `wasi_snapshot_preview1` imports, modelled on `runner/src/wasi.rs`. |
|
|
59
|
+
| `src/host/fs.js` | A read-only, in-memory preopen (`/g`) over SharedArrayBuffers. Its fd table is process-wide: it lives in the control block, so any thread can read a file another thread opened. |
|
|
60
|
+
| `src/host/threads.js` | `thread-spawn`: claims an idle pool slot with a compare-exchange, writes tid and arg, notifies, and returns the tid synchronously. |
|
|
61
|
+
| `src/host/clock.js` | One clock for all Workers: `timeOrigin + now()` minus a shared start time, with a shared high-water mark so CLOCK_MONOTONIC never goes back across Workers. CLOCK_REALTIME is the shared start time plus CLOCK_MONOTONIC, so it never goes back either. |
|
|
62
|
+
| `src/host/wmd-imports.js` | The `wmd.*` imports. `ram_map`→1, `blk_rw`→-38 (no swap disk), `win_init`→1, other `win_*`→0, `snapshot`→-1, `chan_open`→-111 without channel services; without a JIT door `jit_*`→-1 or no-op. `createWmdImports({jit, chan, counters})`: the worker passes the JIT door as `jit` and, when the boot has channel services, the P-B3 channel imports as `chan`. Calls to the stubs are counted in the control block (`counters`), not posted. |
|
|
63
|
+
| `src/host/chan.js` | P-B3: the chan block (one SharedArrayBuffer: doorbell, port registry, a duplex ring pair per channel, the console pipe to the page, counters and the 9P round-trip histogram), the ring primitives, `markServiceDead` / `markGuestGone`, and the five guest imports (`createChanImports`). |
|
|
64
|
+
| `src/host/chan-services.js` | P-B3: `ChanHost`, the host side of every channel, synchronous (`step()`): the 9P server per connection over one Overlay per port, the console (to the page's pipe, and page input), and the control and net stubs (P-B10: the net channel goes to the network Worker when a relay is configured). |
|
|
65
|
+
| `src/host/chan-service.js` | P-B3: the service Worker around `ChanHost`. Waits on the doorbell with `Atomics.waitAsync` (spinning 200 µs after work first), holds a Web Lock for its life so the page learns when it ends, answers `stats`/`collect`, and has test hooks (`die`, `crash`, `stall`). |
|
|
66
|
+
| `src/host/jit-hub.js` | The JIT module hub: one Worker that forwards each compiled `WebAssembly.Module` to every vCPU Worker, so a module is compiled once per process (finding F1). |
|
|
67
|
+
| `src/host/jit.js` | P-B2, the JIT door: `jit_load`, `jit_compile`, `jit_sync`, `jit_flush`, `jit_known`, `jit_note`. One global slot list in SharedArrayBuffers, per-thread tables, counters and per-module timings. Without a module cache `jit_known` is -1 and `jit_note` a no-op; with one (W77) they read and feed `jit-cache.js`. |
|
|
68
|
+
| `src/host/jit-cache.js` | W77, the JIT module cache across visits: `bootWmd({jit: {cache}})` with a `{get, put}` byte store keyed by the vm.wasm's sha256. The page loads one index (block key → module, function, span) into a SharedArrayBuffer table at boot; `jit_known` finds a key there, gets the module's bytes from the page once (hash and length checked), and returns the function's slot; `jit_note` names the modules to put. See D-486. |
|
|
69
|
+
| `src/host/stdio.js` | stdout/stderr rings in a SharedArrayBuffer. Workers write into them under a spin lock, and the page drains them every 16 ms. |
|
|
70
|
+
| `src/host/layout.js` | Cell layout of the shared control block: tid counter, exit flag and code, pool slots, fd table, clock base, stub host-call counters. |
|
|
71
|
+
| `src/host/monitor.js` | Page-side monitors: the kernel-panic scanner, the progress watchdog and the throttle meter. |
|
|
72
|
+
| `src/calib-worker.js` | The throttle detector's Worker: times a fixed integer loop and a 10 ms `Atomics.wait` every 2 s. |
|
|
73
|
+
| `page/` | A small test page. `?case=arm64\|amd64\|hello\|wfi\|prog\|loop\|flush\|smpflush\|smc&mode=interp\|jit&auto=1` boots on load (other options: `arg=`, `env=`, `prog=`, `jitc=`, `brk=`, `share=`, `door=0`, `watchdog=`; see the head of `page/main.js`). State is on `window.wmd`, and the JIT counters are in a table on the page. `page/smc-initrd.js` builds the self-modifying-code initramfs. `page/jit-stress.html` is the multi-Worker JIT door stress test, and `page/wasm-mini.js` encodes its block-shaped test modules. `page/idb-bytes.js` is the IndexedDB byte store behind `?jitcache=1` (`=fresh` empties it first), on this page and the wmn test page. |
|
|
74
|
+
| `src/fs/` | P-B4, the 9P2000.L file server (no browser wiring yet): the pack reader (`pack.js`), the overlay of pack lowers under an in-memory upper (`overlay.js`), the codec and server (`9p/codec.js`, `9p/server.js`) and Linux errno values (`errno.js`). See [docs/9p.md](docs/9p.md). |
|
|
75
|
+
| `page/chan.html`, `page/chan.js`, `page/b3-initrd.js` | P-B3/P-B4 test page: `?case=p9lat\|console\|alpine\|container\|p9load&auto=1` (options at the head of `chan.js`). `b3-initrd.js` appends a newc archive to the proof initramfs with `/p9stat` (a hand-assembled arm64 ELF that times N `fstat` calls) and the case's `/proof.sh`. |
|
|
76
|
+
| `test/serve.ts` | A Bun static server: COOP `same-origin`, COEP `require-corp`, CORP, and a CSP with `'wasm-unsafe-eval'`. It serves read-only from fixed roots. |
|
|
77
|
+
| `test/unit/` | `bun test` for the WASI layer, the spawn pool, the JIT door (`jit.test.js`, `jit-share.test.js`, `b2-*.test.js`, which include real Workers flushing under load), the stdio rings, the P-B1 monitors, counters, clocks and gate (`monitor.test.js`, `gate.test.js`) and the 9P server (`9p-*.test.js`; fixtures made on first run by `test/fixtures-gen`, see docs/9p.md). |
|
|
78
|
+
| `test/e2e/chan.spec.js`, `test/unit/chan*.js`, `test/unit/clock-strict.test.js` | P-B3/P-B4: browser boots per browser (9P round trip, console and control throughput, alpine, the container init, the adversarial cases) and the ring/import/service unit tests with real Workers as guest threads. |
|
|
79
|
+
| `src/host/yamux.js`, `src/host/ctl.js` | P-B11 (ROADMAP 2.2b): the hashicorp/yamux client (run by the channel service, `ChanHost` `ControlSession`, so pings are answered off the page) and wmd's control protocol on the page (`ControlClient`: run, startExec, attach, kill, procs; frames; the engine's OCI bundle, `containerBundle`). `vm.control` when a service is `{kind: 'control', yamux: true}`. |
|
|
80
|
+
| `page/wmn.html`, `page/wmn.js`, `page/k2.js` | P-B11: an image's packs + the agent, the engine's command line, run + exec over the control session, the P-K2 checks (ported from the wmn repo's `spikes/P-K2/k2.ts`). `?image=python\|node&auto=1` (options at the head of `wmn.js`). |
|
|
81
|
+
| `src/net/` | P-B10: the guest's network. `stack.js` (ARP, DHCP, DNS, ICMP, TCP for the guest's frames), `packet.js` (framing and checksums), `dhcp-dns.js`, `relay-client.js` (the relay WebSocket), `proto.js` (the relay protocol, shared with the relay), `net-worker.js` (the network Worker). |
|
|
82
|
+
| `relay/` | P-B10: the relay server (`relay.ts`, Bun), its address policy (`policy.ts`) and session tokens (`token.ts`). |
|
|
83
|
+
| `page/net.html`, `page/net.js` | P-B10: an image with a network (`?image=alpine\|python&auto=1`); `window.net.sh(cmd)` runs a command in the container. |
|
|
84
|
+
| `test/e2e/` | Playwright: the boots (`boot.spec.js`), the P-B1 adversarial checks (`adversarial.spec.js`), the tests of the P-B1 adversarial fixes (`fixes.spec.js`, `throttle.spec.js`) and the P-B1 review suites (`review-*.spec.js`), the wasm feature probes (`features.spec.js`), the P-B2 JIT boots (`jit.spec.js`), the JIT door stress in each browser (`jit-stress.spec.js`) an opt-in benchmark of host variants (`jit-bench.spec.js`, `WMD_WEB_BENCH=1`) and the JIT module cache across two visits (`w77-jit-cache.spec.js`; its python test needs `WMD_WEB_IMAGE_FIXTURES`). |
|
|
85
|
+
|
|
86
|
+
## Design notes
|
|
87
|
+
|
|
88
|
+
- **Workers only.** `memory.atomic.wait32` traps on the main thread, so the page thread
|
|
89
|
+
never runs wasm.
|
|
90
|
+
- **Thread spawn is synchronous.** A Worker blocked inside wasm cannot wait for a new
|
|
91
|
+
Worker to load. The page therefore starts 4 pool Workers up front, each holding the
|
|
92
|
+
compiled `Module`, and keeps 2 spare: after each spawn the spawner posts `need` and the
|
|
93
|
+
page tops up the pool asynchronously.
|
|
94
|
+
- If no Worker is idle, the spawner asks for more and waits up to 10 s on the pool
|
|
95
|
+
generation cell.
|
|
96
|
+
- If none appears in that time, it returns -11 (EAGAIN) and the page prints `thread
|
|
97
|
+
pool exhausted` or `thread-spawn found no idle Worker`.
|
|
98
|
+
- Up to 128 slots are allowed.
|
|
99
|
+
- A P-B1 boot spawns 2 threads (tids 2 and 3), one of which is the timer thread.
|
|
100
|
+
Which code spawns the other was not traced.
|
|
101
|
+
- **Exit.** `proc_exit` sets the shared exit flag (the first code wins), tells the page,
|
|
102
|
+
and throws to unwind its own Worker. The page then terminates every Worker. A trap on
|
|
103
|
+
any thread ends the machine with code 4, as in the runner.
|
|
104
|
+
- **Memory.** One `WebAssembly.Memory({initial: 6144, maximum, shared: true})`.
|
|
105
|
+
`maximum` is 6144 + RAM pages + 1 GiB of slack (10,240 + 16,384 pages for
|
|
106
|
+
`--ram 256`). Views over it are rebuilt whenever `memory.buffer` changes after a grow.
|
|
107
|
+
- **Output.** stdout and stderr go into a ring per fd in a SharedArrayBuffer
|
|
108
|
+
(`src/host/stdio.js`), 32 MiB each for a boot (`bootWmd({stdioRingBytes})`, a power of
|
|
109
|
+
two). The page drains the rings every 16 ms and decodes them with a streaming
|
|
110
|
+
`TextDecoder`. P-B1 posted one message per `fd_write`; P-B2 replaced that, because a
|
|
111
|
+
guest that writes megabytes a second flooded the page's event loop (see P-B2 problem
|
|
112
|
+
3). If the page falls a whole ring behind, it skips ahead, counts the skipped bytes
|
|
113
|
+
(`timings.outputDropped`, `timings.outputDropMarkers`) and puts a line
|
|
114
|
+
`[wmd-web: N bytes of stdout were dropped]` into the delivered text where the bytes
|
|
115
|
+
are missing. The boot gate fails on that line (finding F4 below).
|
|
116
|
+
- **Endings.** `vm.exited` resolves `{code, reason, timings}`. Guest exit codes are
|
|
117
|
+
>= 0. The host's own endings are negative: -1 `stopped` (`vm.stop()`) or `aborted`
|
|
118
|
+
(the AbortSignal), -2 `no progress: ...` (the watchdog), -3 `kernel panic: <line>`
|
|
119
|
+
(the guest's own exit code is then in `timings.guestExitCode`). An AbortSignal that
|
|
120
|
+
fires before the Workers start makes `bootWmd` reject with an `AbortError` instead;
|
|
121
|
+
the signal is checked after every await, so this is immediate even while vm.wasm
|
|
122
|
+
compiles.
|
|
123
|
+
- **Kernel panic.** The host scans the console for `Kernel panic - not syncing`. The
|
|
124
|
+
arm64 kernel has `CONFIG_PANIC_TIMEOUT=-1`, so it reboots at once and the guest
|
|
125
|
+
reports exit 0 (PSCI SYSTEM_RESET); the host reports -3. A kernel that hangs after a
|
|
126
|
+
panic instead is stopped 5 s after the panic line.
|
|
127
|
+
- **Progress watchdog.** After the main Worker starts `_start`, the boot ends with -2
|
|
128
|
+
and the line `guest made no progress` when nothing is written to stdout or stderr
|
|
129
|
+
and no thread is spawned for `watchdogMs`. Default 60 s for an arm64 `--machine`,
|
|
130
|
+
300 s for x86 (the amd64 decompressor is silent for about 75 s in Chromium), off for
|
|
131
|
+
user-mode programs. The limit is multiplied by the throttle ratio while the host
|
|
132
|
+
reports throttling, and a page whose own thread was stalled restarts the count.
|
|
133
|
+
- **Throttle detector.** A small Worker (`calib-worker.js`) times a fixed integer loop
|
|
134
|
+
every 2 s (about 1% of one core). The page compares the median of the last three
|
|
135
|
+
runs with the fastest run seen (this boot, or the `throttle.baselineMs` the caller
|
|
136
|
+
passes; the test page calibrates once at load and keeps the best in `localStorage`).
|
|
137
|
+
At 2x or more it emits `{t:'throttle', throttled:true, ratio, ...}`, writes
|
|
138
|
+
`[wmd-web] the guest is running about Nx slower ...` to stderr, and
|
|
139
|
+
`vm.stats().throttle` / `timings.throttle` carry `{ratio, maxRatio, throttledSamples}`.
|
|
140
|
+
See "Background throttling" below for what it catches and why.
|
|
141
|
+
- **Stub host calls.** Calls to the stubbed `snapshot` and `chan_open`, and to the
|
|
142
|
+
stubbed `jit_*` when a boot runs without the JIT door (`bootWmd({jit: false})`, test
|
|
143
|
+
page `?door=0`), are counted in shared cells of the control block. `vm.stats().hostCalls` reads them on
|
|
144
|
+
demand; the page gets one `{t:'host-calls', counts}` event at most every 250 ms, and
|
|
145
|
+
only when a count changed. The test page keeps the latest counts, not a list, and caps
|
|
146
|
+
its diagnostic event list at 1000 entries.
|
|
147
|
+
- **Releasing a finished boot.** When a boot ends (exit, stop, abort, trap, watchdog,
|
|
148
|
+
panic) the host terminates every Worker (the calibration Worker too), clears their
|
|
149
|
+
handlers, its timers and the abort listener, terminates the JIT hub Worker, and drops
|
|
150
|
+
the compiled module and the file buffers. Only `vm.memory` stays, for post-mortem reads, until `vm.dispose()`;
|
|
151
|
+
the test page calls `dispose()` and keeps no handle.
|
|
152
|
+
stdout is line-buffered by the guest's libc. The test page keeps about 8 MB of each
|
|
153
|
+
stream in `window.wmd` (the start and the most recent output, with a
|
|
154
|
+
`[wmd-web: N bytes of stdout were dropped]` line between, so a gate run on it fails
|
|
155
|
+
too) and shows the last 128 KB. P-B1's merged branch had the same rings at 4 MiB
|
|
156
|
+
(cherry-picked as `17e9ce50`); the merge keeps P-B2's 32 MiB rings and drop markers.
|
|
157
|
+
|
|
158
|
+
## Running the tests
|
|
159
|
+
|
|
160
|
+
```sh
|
|
161
|
+
cd web
|
|
162
|
+
bun install # @playwright/test 1.63.0 (browsers already in ~/Library/Caches/ms-playwright)
|
|
163
|
+
bun test test/unit # WASI layer, spawn pool, JIT door, stdio rings, monitors, counters, clocks, gate, 9P; no browser
|
|
164
|
+
|
|
165
|
+
bunx playwright test --project=chromium # boots + adversarial + features
|
|
166
|
+
WMD_WEB_AMD64=1 bunx playwright test boot --project=chromium -g amd64 # ~6.6 min
|
|
167
|
+
bunx playwright test boot --project=firefox
|
|
168
|
+
bunx playwright test boot --project=webkit
|
|
169
|
+
bunx playwright test features # all three browsers, no guest boot
|
|
170
|
+
bunx playwright test jit-stress # JIT door stress, all three browsers, no guest boot
|
|
171
|
+
|
|
172
|
+
# P-B2 (each under the lock, one browser per run, about 3 min each):
|
|
173
|
+
bunx playwright test jit.spec boot.spec jit-stress --project=chromium
|
|
174
|
+
bunx playwright test jit.spec boot.spec jit-stress --project=firefox
|
|
175
|
+
bunx playwright test jit.spec boot.spec jit-stress --project=webkit
|
|
176
|
+
WMD_WEB_BENCH=1 bunx playwright test jit-bench --project=chromium # host variants, one boot each
|
|
177
|
+
|
|
178
|
+
# P-B2 adversarial fixes (each under the lock, per browser): F1 multi-vCPU --jit-batch 1
|
|
179
|
+
# boots and F4 drop-marker boots; B2F_SHARE=0 reproduces F1 in Firefox
|
|
180
|
+
bunx playwright test b2-fixes --project=firefox
|
|
181
|
+
WMD_WEB_B2ADV=1 bunx playwright test b2-adversarial --project=firefox # the review's own suite
|
|
182
|
+
|
|
183
|
+
# P-B3/P-B4 adversarial fixes (each under the lock, per browser; WMD_WEB_B3ADV=1 opts in):
|
|
184
|
+
WMD_WEB_B3ADV=1 bunx playwright test b3-fixes --project=webkit -g "default|record|F2/F3" # ~3 min
|
|
185
|
+
WMD_WEB_B3ADV=1 bunx playwright test b3-fixes --project=webkit -g "measure cache" # F1 numbers, ~8-12 min
|
|
186
|
+
WMD_WEB_B3ADV=1 bunx playwright test b3-adversarial --project=webkit -g "rings" # the review's own suite, in parts
|
|
187
|
+
|
|
188
|
+
# P-B1 adversarial fixes and the P-B1 review's suites (each under the lock, per browser):
|
|
189
|
+
bunx playwright test fixes --project=firefox # (under the lock) the adversarial-fix tests, any browser
|
|
190
|
+
REVIEW_SEQ_BOOTS=5 bunx playwright test review-lifecycle -g sequential --project=webkit # (under the lock) memory over 5 boots
|
|
191
|
+
|
|
192
|
+
# Real background throttling: opens a headed Chromium window (under the lock).
|
|
193
|
+
REVIEW_HEADED=minimized bunx playwright test throttle --project=chromium # must report throttling
|
|
194
|
+
REVIEW_HEADED=visible bunx playwright test throttle --project=chromium # must not; full strict gate
|
|
195
|
+
REVIEW_HEADED=minimized bunx playwright test review-lifecycle -g headed --project=chromium # the adversary's version
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Environment variables:
|
|
199
|
+
|
|
200
|
+
- `WMD_REPO`: the wmd checkout the boot files are served from. Default
|
|
201
|
+
`~/keg-hub/external/apps/wmd`. The files used are `guest/linux/dist/{arm64,amd64}/*`
|
|
202
|
+
and `guest/vm/build/progs/arm64/hello`.
|
|
203
|
+
- `WMD_VM_WASM`: the vm.wasm to serve. Default `~/.wmd/bin/guest/arm64/vm.wasm`.
|
|
204
|
+
- `PORT`: the server port. Default 8791 for Playwright, 8787 for `bun test/serve.ts`.
|
|
205
|
+
The P-B2 runs used `PORT=8793`, so they would not collide with another worktree's
|
|
206
|
+
server.
|
|
207
|
+
- `WMD_WEB_BOOT_TIMEOUT_MS`: the per-boot bound. Default 420000 (300000 in `chan.spec.js`).
|
|
208
|
+
- `WMD_GUEST_DIR`: where `/files/guest/arm64/` (the container kernel and the Go init's
|
|
209
|
+
initramfs, P-B4) is served from. Default `~/.wmd/bin/guest/arm64`, read only.
|
|
210
|
+
- `WMD_WEB_9P_FIXTURES`: the packs served at `/fixtures/9p/` (default `test/fixtures/9p`,
|
|
211
|
+
made by the unit tests' `ensureFixtures()`).
|
|
212
|
+
- `chan.spec.js`: `WMD_WEB_CONSOLE_BYTES` (default 50 MiB), `WMD_WEB_UART=BYTES` (runs the
|
|
213
|
+
UART comparison), `WMD_WEB_HVC0_TIMEOUT_MS` (120000), `WMD_WEB_RUNCLOG=1` (crun's debug log
|
|
214
|
+
into `bundle/runc.log`), `WMD_WEB_P9LAT_QUERY` (extra page options, e.g. `&spin=0`).
|
|
215
|
+
The P-B3/P-B4 runs used `PORT=8796`.
|
|
216
|
+
|
|
217
|
+
The logs and timings of each boot are written to
|
|
218
|
+
`test-results/<browser>-<case>.{stdout.log,stderr.log,timings.json}`.
|
|
219
|
+
|
|
220
|
+
## Numbers (2026-10-04, Apple Silicon, one run each, interpreter, 1 vCPU, --ram 256)
|
|
221
|
+
|
|
222
|
+
All times are in ms. Browser times start when `bootWmd` is called on the page, so they
|
|
223
|
+
include the download. Native times start at process start. The canary read
|
|
224
|
+
617–649 ms around these runs, so the numbers are reliable.
|
|
225
|
+
|
|
226
|
+
| arm64 proof boot | native `wmd-run` | Chromium 153 (headless shell) | Firefox 155 | WebKit 26.6 |
|
|
227
|
+
|---|---|---|---|---|
|
|
228
|
+
| download (vm.wasm + Image + initramfs, 43.2 MB, localhost) | n/a | 62 | 165 | 61 |
|
|
229
|
+
| `WebAssembly.compile(vm.wasm)` | n/a (Cranelift, cached) | 9 | 25 | 11 |
|
|
230
|
+
| main Worker instantiated | n/a | 107 | 230 | 89 |
|
|
231
|
+
| first console byte | 222 (`Linux version`) | 345 | 654 | 334 |
|
|
232
|
+
| `wmd-init: up` | 19,977 | 17,342 | 21,653 | 19,169 |
|
|
233
|
+
| `PROOF-DONE` | 51,554 | 44,042 | 57,534 | 49,166 |
|
|
234
|
+
| `reboot: Power down` | 51,771 | 44,218 | 57,752 | 49,374 |
|
|
235
|
+
| total (exit) | 52,024 (wall 51.7 s) | 44,240 | 57,772 | 49,431 |
|
|
236
|
+
|
|
237
|
+
- **Native reference.** It was run once:
|
|
238
|
+
`wmd-run --wmd-stats --dir guest/linux/dist guest/vm/build/vm.wasm -- --machine --arch a64 --board arm64 --mode interp --cpus 1 --ram 256 --kernel .../arm64/Image --initrd .../arm64/initramfs.cpio.gz --append "console=ttyAMA0 earlycon=pl011,0x9000000 rdinit=/init loglevel=8 wmd.proof wmd.exit" --stats`.
|
|
239
|
+
- **Compile time.** The compile times are what `WebAssembly.compile` returns. That is
|
|
240
|
+
the engines' baseline tier (Liftoff, or equivalent); optimizing tier-up happens later,
|
|
241
|
+
in the background.
|
|
242
|
+
- **Browsers vs native.** In Chromium and WebKit the interpreter boot is *faster* than
|
|
243
|
+
native wasmtime, by 15% and 5%. Firefox is 11% slower. This is one run each, and the
|
|
244
|
+
cause was not investigated.
|
|
245
|
+
|
|
246
|
+
| amd64 proof boot (Chromium) | ms |
|
|
247
|
+
|---|---|
|
|
248
|
+
| download (vm.wasm + bzImage + initramfs, 11.8 MB) | 37 |
|
|
249
|
+
| compile | 7 |
|
|
250
|
+
| first console byte (decompressor) | 403 |
|
|
251
|
+
| `Linux version` | 74,841 |
|
|
252
|
+
| `wmd-init: up` | 238,001 |
|
|
253
|
+
| total (exit 0, all boot-gate patterns met) | 392,857 |
|
|
254
|
+
|
|
255
|
+
- **amd64 marks.** The `PROOF-DONE` and power-down marks were not captured on this run:
|
|
256
|
+
the x86 console sends CRLF and the mark regexes were anchored. The boot gate itself
|
|
257
|
+
strips CR and passed. Mark matching now strips CR (commit `6e4735e5`).
|
|
258
|
+
- **No native amd64 reference.** The brief allowed one native run, and it was spent on
|
|
259
|
+
arm64.
|
|
260
|
+
|
|
261
|
+
After the adversarial fixes (same day, every boot under the lock, canary 600-650 ms):
|
|
262
|
+
`PROOF-DONE` at 40.4-42.5 s in Chromium, 52.5-56.4 s in Firefox and 45.9-47.3 s in
|
|
263
|
+
WebKit, TIMER-DELTA 1.1 everywhere. The review measured 40.3-45.8 s (Chromium), 55.6-58.6 s
|
|
264
|
+
(Firefox) and 48.0-50.5 s (WebKit) before the fixes, so the throttle detector's Worker
|
|
265
|
+
and the 250 ms tick cost no measurable time.
|
|
266
|
+
|
|
267
|
+
## Browser wasm feature support (Playwright 1.63 browsers, `test/e2e/features.spec.js`)
|
|
268
|
+
|
|
269
|
+
Each feature was checked by compiling, instantiating and running a tiny module in a
|
|
270
|
+
cross-origin isolated page.
|
|
271
|
+
|
|
272
|
+
| Feature | Chromium 153 | Firefox 155 | WebKit 26.6 |
|
|
273
|
+
|---|---|---|---|
|
|
274
|
+
| cross-origin isolated, SharedArrayBuffer | yes | yes | yes |
|
|
275
|
+
| shared memory 384 MiB initial / 1 GiB max | yes | yes | yes |
|
|
276
|
+
| threads (shared memory + atomics) | yes | yes | yes |
|
|
277
|
+
| SIMD128 | yes | yes | yes |
|
|
278
|
+
| bulk memory | yes | yes | yes |
|
|
279
|
+
| multi-memory | yes | yes | yes |
|
|
280
|
+
| two *shared* imported memories + atomic load from memory 1 (the JIT block-module shape) | yes | yes | yes |
|
|
281
|
+
| tail calls (`return_call`) | yes | yes | yes |
|
|
282
|
+
| memory64 | yes | yes | yes |
|
|
283
|
+
|
|
284
|
+
These are Playwright's engine builds. Shipping Safari and Firefox releases may differ.
|
|
285
|
+
P-B5 has to re-check them on real browsers.
|
|
286
|
+
|
|
287
|
+
## Firefox and WebKit
|
|
288
|
+
|
|
289
|
+
Both pass steps 1–3, the same tests as Chromium. Each prints `hello from aarch64` and
|
|
290
|
+
`board init ok`, and the arm64 proof boot passes the full gate with exit 0. The timings
|
|
291
|
+
are in the table above. Nothing needed per-browser code.
|
|
292
|
+
|
|
293
|
+
## P-B3: channels on shared-memory rings
|
|
294
|
+
|
|
295
|
+
`wmd.chan_*` as `runner/src/chan.rs` has them, for Workers (`src/host/chan.js`):
|
|
296
|
+
|
|
297
|
+
- **One chan block per machine.** A SharedArrayBuffer holds a header (doorbell, service
|
|
298
|
+
state, guest-gone flag, the ports the services listen on), 8 channel slots, and per
|
|
299
|
+
slot two byte rings: guest→host and host→guest, 1 MiB each by default
|
|
300
|
+
(`bootWmd({channels: {ringBytes}})`, any power of two from 64 bytes), plus an 8 MiB pipe
|
|
301
|
+
for the console's bytes to the page: about 25 MiB of SharedArrayBuffer per boot, touched
|
|
302
|
+
only as it is used. Positions are 32-bit counters that wrap. Boots without `channels`
|
|
303
|
+
get the old stubs and no service Worker.
|
|
304
|
+
- **Guest side: blocking imports.** `chan_send` copies straight from guest memory into
|
|
305
|
+
the ring, waiting (`Atomics.wait` on the ring's "space" sequence cell) while it is full;
|
|
306
|
+
it holds the ring's producer lock for the whole message, so two vCPUs' 9P requests never
|
|
307
|
+
interleave, whatever their size. `chan_recv` returns what is there (a partial read is
|
|
308
|
+
normal), waits up to the timeout (rounded up to ms, as chan.rs) on the ring's "data"
|
|
309
|
+
sequence cell, returns 0 on timeout and -1 at EOF. Every wait is at most 1 s at a time,
|
|
310
|
+
and re-checks the machine's exit flag, the service state and the channel's own state, so
|
|
311
|
+
nothing in the guest can hang on a peer that is gone.
|
|
312
|
+
- **Return values as chan.rs.** `chan_open`: -22 for a port outside 1..65535, -111 when no
|
|
313
|
+
service listens on the port (or the service is dead), else the lowest free handle. The
|
|
314
|
+
port decides the service; `kind` is only recorded. -14 bad range, -9 bad handle, -32 when
|
|
315
|
+
the peer is gone or after our own `chan_shutdown`. `chan_shutdown` is a half-close: the
|
|
316
|
+
host reads to the end, sees EOF, and closes its side once its replies are out.
|
|
317
|
+
`chan_close` wakes any thread blocked on the handle (-1 / -32) and frees the slot only
|
|
318
|
+
after the service has dropped the session, so a reply in flight can never land in a new
|
|
319
|
+
channel that reused the handle.
|
|
320
|
+
- **Host side: a service Worker** (`src/host/chan-service.js`), never the page. It never
|
|
321
|
+
blocks: it waits on the doorbell cell with `Atomics.waitAsync` and spins on it for 200 µs
|
|
322
|
+
after any work (`channels.spinUs`), so its event loop stays free for the page's messages.
|
|
323
|
+
Services by the kind registered for the port: **9p** (`src/fs/9p/server.js`, a
|
|
324
|
+
`NinePServer` per connection over one Overlay per port: packs as lowers, an in-memory
|
|
325
|
+
upper), **console** (to the page through a pipe ring the page drains every 16 ms; no loss:
|
|
326
|
+
when the page falls behind, the service stops reading and the guest blocks; page input
|
|
327
|
+
through `vm.consoleWrite`, which takes at most 4 MiB the guest has not read and says how
|
|
328
|
+
much it took), **control** (a stub: reads, counts and CRC-32s), **net** (a stub: counts
|
|
329
|
+
the 4-byte-length frames and drops them; P-B10 adds the relay).
|
|
330
|
+
- **Backpressure.** The 9P service answers at most `OUT_LIMIT` (4 MiB) of replies the guest
|
|
331
|
+
has not read; after that it stops reading requests, so tiny requests cannot make it queue
|
|
332
|
+
unbounded replies, and the guest's sends block on a full ring.
|
|
333
|
+
- **A dead service.** The service holds a Web Lock for its whole life; the page asks for the
|
|
334
|
+
same lock and is granted it when the Worker ends, however it ends (terminated, closed,
|
|
335
|
+
crashed, killed by the browser). The page then calls `markServiceDead`: every guest
|
|
336
|
+
waiter wakes, sends fail (-32), receives drain what is left and then get -1, `chan_open`
|
|
337
|
+
refuses. An exception in the service's loop marks it dead from inside too. The guest's 9P
|
|
338
|
+
pump powers the machine off on -1 (`chan_pump.c` `host_gone`) with exit 0; `bootWmd`
|
|
339
|
+
reports that as **-4** `channel service died: ...` (the guest's code in
|
|
340
|
+
`timings.guestExitCode`), as a panic is -3.
|
|
341
|
+
- **The guest ends.** `finish()` sets the guest-gone flag, which also wakes every guest
|
|
342
|
+
thread still blocked on a channel (-32 / -1 at once); the service drops every session
|
|
343
|
+
(counting any message it held half-read in `sessionsDropped`, never handing it to the
|
|
344
|
+
server) and stops; then the page terminates it with the other Workers.
|
|
345
|
+
- **Counters.** `vm.chanStats()` / `timings.chan`: per channel bytes each way and how often
|
|
346
|
+
and how long a guest send waited for room (`sendWaits`, `sendWaitMs`: the "host fell
|
|
347
|
+
behind" signal); the 9P round trip measured by the guest's threads (send of a T-message
|
|
348
|
+
to its R-message delivered by the pump: a histogram, p50/p90/p99) and the service's
|
|
349
|
+
per-request time; per-kind bytes, frames, EOFs, protocol errors.
|
|
350
|
+
- **Files out.** `channels.collect: [{port, path}]` reads files out of the 9P trees when the
|
|
351
|
+
machine ends (`timings.chanFiles`), e.g. `bundle/exitcode`.
|
|
352
|
+
|
|
353
|
+
### Which console the guest uses
|
|
354
|
+
|
|
355
|
+
The proof boot's console is the PL011 UART, which vm.wasm writes to its stdout (the stdout
|
|
356
|
+
ring), and so is the container kernel's (`console=ttyAMA0`, `internal/engine/run.go`).
|
|
357
|
+
Containers' stdio does not go through either: the Go init runs its yamux session, which
|
|
358
|
+
carries the container's stdin/stdout/stderr, on virtio-console **port 1** (`wmd.control`,
|
|
359
|
+
`/dev/vportNp1`), i.e. the **control channel** (kind 3). Nothing in wmd uses port 0 (`hvc0`,
|
|
360
|
+
kind 4). The throughput test therefore writes 50 MiB into each of: the control channel (the
|
|
361
|
+
path container output will take), `hvc0` (the console channel), the UART (the stdout ring)
|
|
362
|
+
and `/dev/null` (what the guest's `yes | head` costs by itself).
|
|
363
|
+
|
|
364
|
+
### Numbers (2026-10-05, Apple Silicon, Playwright 1.63, `--mode jit`, 1 vCPU, --ram 256)
|
|
365
|
+
|
|
366
|
+
One run each per browser (`test/e2e/chan.spec.js`, the final runs; canary 605–634 ms around
|
|
367
|
+
the Chromium and Firefox runs; the WebKit run started contended, canary 1,378 ms before and
|
|
368
|
+
604 after, which shows in its first p9lat run). Native: `~/.wmd/bin/wmd-run` with wmd's Go 9P
|
|
369
|
+
server (`guest/linux/testhelper`) over loopback TCP, the same initramfs, canary 657–680 ms.
|
|
370
|
+
|
|
371
|
+
**9P round trip.** `/p9stat FILE 2000` on the 9P mount (no cache: every `fstat` is one
|
|
372
|
+
Tgetattr, which the service's counts confirm: 6,050+ Tgetattr), three runs after a warm-up of
|
|
373
|
+
50, µs per call; on tmpfs for the syscall's own cost:
|
|
374
|
+
|
|
375
|
+
| | Chromium | Firefox | WebKit | native, Go server over TCP |
|
|
376
|
+
|---|---|---|---|---|
|
|
377
|
+
| `fstat` on the 9P mount (3 runs) | 137.4, 109.5, 101.4 | 153.1, 141.0, 138.4 | 187.2, 138.9, 88.9 | 113.2, 104.1, 108.0 (2nd boot; 1st: 207, 127, 133) |
|
|
378
|
+
| `fstat` on tmpfs | 14.9, 13.8, 14.1 | 19.3, 19.0, 18.6 | 21.0, 19.7, 17.5 | 7.5, 6.9, 7.3 |
|
|
379
|
+
| channel round trip, guest threads (send of the T-message → R-message in the pump), p50 / p90 / p99 | 30.5 / 40.5 / 100.5 | 20.5 / 40.5 / 139.5 | 20.5 / 100.5 / 559.5 | – |
|
|
380
|
+
| service time per request, p50 / p99 | 0.5 / 25.5 | 0.5 / 59.5 | 0.5 / 99.5 | – |
|
|
381
|
+
|
|
382
|
+
- **P-B3's bar (under 500 µs) is met** by a wide margin in all three, and the browser is
|
|
383
|
+
as fast as native wmd over TCP: almost all of the ~100 µs is the guest (syscall, v9fs,
|
|
384
|
+
virtio, the MMIO kick, the interrupt and the wake-up of the waiting process in the
|
|
385
|
+
emulated kernel), not the channel (20–40 µs) or the server (under 1 µs typical).
|
|
386
|
+
- The histograms are quantised by each browser's coarsened clock (`performance.now()`:
|
|
387
|
+
5 µs in Chromium, 20 µs in Firefox and WebKit), hence the x0.5 values.
|
|
388
|
+
- **The spin is worth about 20 µs.** Chromium, two boots each: with `spin=0` the channel's
|
|
389
|
+
p50 was 40.5 µs and `fstat` 135–205 µs; with the default 200 µs spin, 19.5–20.5 µs and
|
|
390
|
+
104–157 µs.
|
|
391
|
+
|
|
392
|
+
**Console throughput.** `yes | head -c 52428800` (50 MiB; since W53 piped through `cat`, see "hvc0 after W17"), timed by the guest
|
|
393
|
+
(`/proc/uptime`) and by the service (first to last byte read):
|
|
394
|
+
|
|
395
|
+
| | Chromium | Firefox | WebKit |
|
|
396
|
+
|---|---|---|---|
|
|
397
|
+
| into the **control channel** (vport, container stdio path) | 14.05 s, **3.7 MB/s** (service 13.95 s) | 15.86 s, **3.3 MB/s** | 10.69 s, **4.9 MB/s** |
|
|
398
|
+
| times the guest waited for the host | **0** | **0** | **0** |
|
|
399
|
+
| into `/dev/null` (the guest alone) | 11.83 s, 4.4 MB/s | 13.81 s, 3.8 MB/s | 12.33 s, 4.3 MB/s |
|
|
400
|
+
| through the UART (stdout ring) | 368.4 s, 0.14 MB/s, 0 bytes dropped (`WMD_WEB_UART`, Chromium only) | – | – |
|
|
401
|
+
| into `hvc0` (console channel) | **fails**: the guest's IRQ storm, see problem 1 | fails | fails |
|
|
402
|
+
|
|
403
|
+
The control channel keeps up: every byte arrived, in order, and no `chan_send` ever found
|
|
404
|
+
the ring full. In Chromium and Firefox the run took 15–19% longer than into `/dev/null`,
|
|
405
|
+
which is the guest's own virtio-console write path (copying into 32 KiB buffers, the MMIO
|
|
406
|
+
kick, the interrupt); in WebKit the control run was faster than its `/dev/null` run, so
|
|
407
|
+
one run each is noisy at that level. The UART is 26x slower: one
|
|
408
|
+
MMIO trap per byte, and the tty turns each `\n` into `\r\n`.
|
|
409
|
+
|
|
410
|
+
### Tests
|
|
411
|
+
|
|
412
|
+
```sh
|
|
413
|
+
bun test test/unit # 241 tests; chan.test.js: rings, imports, ChanHost, the service Worker;
|
|
414
|
+
# b3-fixes.test.js: the fixes of the P-B3/P-B4 review
|
|
415
|
+
# each under the boot lock, one browser at a time (about 4 min each):
|
|
416
|
+
PORT=8796 bunx playwright test chan.spec --project=chromium
|
|
417
|
+
WMD_WEB_UART=52428800 PORT=8796 bunx playwright test chan.spec -g UART --project=chromium # 6 min
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
Unit (`test/unit/chan.test.js`, Workers play the guest's threads and the real service
|
|
421
|
+
Worker serves): a 64-byte ring carrying 1 MB of odd-sized chunks; full and empty; 32-bit
|
|
422
|
+
counter wrap; open/handle/range errors; timeouts (0 polls, 20 ms waits ≥ 19 ms, 1 ns rounds
|
|
423
|
+
up to 1 ms); partial reads; shutdown both ways; close freeing the slot only after the host
|
|
424
|
+
drops it; a dead service; 9P frames through a 64-byte ring giving the same replies as the
|
|
425
|
+
server itself; a bad frame size closing the channel; **4 producers x 200 messages up to 3.5x
|
|
426
|
+
the ring never interleaving**; **512 KiB Twrite/Tread over 4 KiB rings**; **rings full on
|
|
427
|
+
both sides** (the sender blocks after the host's 4 MiB of unread replies, nothing is lost,
|
|
428
|
+
everything arrives in order once the guest reads); **small requests for big replies** (the
|
|
429
|
+
host answers no more than OUT_LIMIT + a ring ahead); **the service terminated while a
|
|
430
|
+
sender is blocked mid-message** (-32) and a receiver blocked forever (-1), within 500 ms;
|
|
431
|
+
**an exception in the service** (-1, service dead); **the guest gone with half a frame sent**
|
|
432
|
+
(dropped, never given to the server); a sender blocked on a full ring returning when the
|
|
433
|
+
machine exits; console input in order through a 64-byte ring; net and control stubs.
|
|
434
|
+
|
|
435
|
+
Browser (`test/e2e/chan.spec.js`), each in Chromium, Firefox and WebKit:
|
|
436
|
+
|
|
437
|
+
| | Chromium | Firefox | WebKit |
|
|
438
|
+
|---|---|---|---|
|
|
439
|
+
| 9P round trip (P-B3 bar) | pass | pass | pass |
|
|
440
|
+
| control channel 50 MiB, every byte, no waits | pass | pass | pass |
|
|
441
|
+
| `hvc0` 50 MiB | **fail** (guest, problem 1) | **fail** | **fail** |
|
|
442
|
+
| `/dev/null` baseline | pass | pass | pass |
|
|
443
|
+
| alpine from packs (proof initramfs) | pass | pass | pass |
|
|
444
|
+
| alpine as a container (Go init, crun) | pass | pass | pass |
|
|
445
|
+
| service terminated by the page mid-traffic → -4, ended 262 / 267 / 258 ms after | pass | pass | pass |
|
|
446
|
+
| service `self.close()` mid-traffic (only the lock tells) → -4, 261 / 259 / 257 ms | pass | pass | pass |
|
|
447
|
+
| service throws mid-traffic → -4, 257 / 259 / 257 ms | pass | pass | pass |
|
|
448
|
+
| guest powers off with 9P reads in flight → exit 0, no service errors | pass | pass | pass |
|
|
449
|
+
| 4 KiB rings: alpine with 256 KiB messages | pass | pass | pass |
|
|
450
|
+
|
|
451
|
+
The P-B1 and P-B2 suites still pass after the clock change (problem 2): `boot.spec` and
|
|
452
|
+
`jit.spec` in Chromium (14 passed, 1 skipped), and the P-B1 and P-B2 proof boots in Firefox
|
|
453
|
+
and WebKit.
|
|
454
|
+
|
|
455
|
+
### Problems found, and their causes
|
|
456
|
+
|
|
457
|
+
1. **`hvc0` (the console channel) crawls and the kernel disables its interrupt.** With
|
|
458
|
+
`yes | head -c N > /dev/hvc0` the guest prints `irq 14: nobody cared` after a few seconds
|
|
459
|
+
and `Disabling IRQ #14`, and goes on at about 60 KB/s. The cause is in the guest, not the
|
|
460
|
+
host: native `wmd-run --console PORT` (a `nc -l` behind the port) does exactly the same
|
|
461
|
+
(4 MiB in 70 s, the same two lines). Linux's hvc path (`put_chars` → `__send_to_port`)
|
|
462
|
+
kicks the queue and then **polls** for the used buffer with interrupts off, while
|
|
463
|
+
vm.wasm's virtio-console (`guest/vm/src/dev/console.c` `console_notify`) raises an
|
|
464
|
+
interrupt for every transmit buffer. By the time interrupts are back on the buffer has
|
|
465
|
+
been taken, `vring_interrupt` finds nothing and returns IRQ_NONE, and after 100,000 of
|
|
466
|
+
those Linux disables the line. A raw tty (`stty raw -echo -opost`) does not help. A guest
|
|
467
|
+
fix would be to honour `VRING_AVAIL_F_NO_INTERRUPT` on the console's transmit queues
|
|
468
|
+
(`virtio_used_irq_wanted`, as `p9proxy.c` does since D-324) or not to interrupt for
|
|
469
|
+
transmit completions the driver polls; not built here. Nothing in wmd uses port 0, and
|
|
470
|
+
the container stdio path (port 1) is not affected: its driver path takes the interrupts.
|
|
471
|
+
**Update (wmd main, W17 2ec1cf9d):** vm.wasm now raises no interrupt for port 0's transmits;
|
|
472
|
+
the storm and the disabled IRQ are gone. hvc0 is still about 33-72 KB/s in the browsers and
|
|
473
|
+
69 KB/s natively, so the 50 MiB test still times out: a vm.wasm/guest cost, not the host
|
|
474
|
+
(P-B11 section "hvc0 after W17").
|
|
475
|
+
2. **The container's `crun` hung: no entropy in the browser.** The first container boot
|
|
476
|
+
stopped after `wmd-init: writing runc bundle`, the vCPU idle in WFI, and no `random: crng
|
|
477
|
+
init done` in the log. Linux's jitter entropy (`try_to_generate_entropy`, run when
|
|
478
|
+
`getrandom()` blocks) reads the cycle counter 8,192 times and gives up when too few reads
|
|
479
|
+
differ. The guest's counter is CLOCK_MONOTONIC / 16 ns, and browsers coarsen
|
|
480
|
+
`performance.now()` (5–20 µs), so consecutive reads were equal; natively the clock has
|
|
481
|
+
ns resolution. Without a seed (the direct-run path gets none; the control path's run
|
|
482
|
+
request carries one) `crun` blocked in `getrandom()` forever. Fix (`src/host/clock.js`):
|
|
483
|
+
CLOCK_MONOTONIC is strictly increasing, every read at least 16 ns (one counter tick) after
|
|
484
|
+
the previous one on any Worker; it can run ahead of real time by no more than 16 ns per
|
|
485
|
+
read made within one coarsened step. After it, `crng init done` came at 22 s and the
|
|
486
|
+
container ran (`test/unit/clock-strict.test.js` fails on the old clock). The entropy this
|
|
487
|
+
gives is weak; real entropy should come from the host (`crypto.getRandomValues`): the
|
|
488
|
+
control channel's `seed`, or a `/chosen/rng-seed` in vm.wasm's device tree (a guest change).
|
|
489
|
+
**Update (wmd main, W18 70d03426):** vm.wasm now puts 64 host bytes in `/chosen/rng-seed`
|
|
490
|
+
(WASI `random_get`, which `src/host/wasi.js` serves from `crypto.getRandomValues`); browser
|
|
491
|
+
boots log `crng init done` at 0.000000 (P-B11 section "Review of 70d03426").
|
|
492
|
+
3. **9P reply amplification.** The first version read a whole chunk of requests and answered
|
|
493
|
+
all of them before checking its output limit: 400 23-byte Treads made 26 MB of queued
|
|
494
|
+
replies, and the "rings full on both sides" test passed or failed by timing (1 in 6
|
|
495
|
+
runs). The service now answers only while less than OUT_LIMIT of replies wait unread and
|
|
496
|
+
holds the rest of the requests back.
|
|
497
|
+
4. **The guest's `udhcpc` waits 15 s** in the container boot: eth0 exists and nothing answers
|
|
498
|
+
(the net stub drops frames). That is 15.5 of the container boot's 25.7 s (Chromium).
|
|
499
|
+
P-B10's relay, or a DHCP answer from the stub, removes it. **Update (P-B10):** with a net
|
|
500
|
+
service the network Worker answers DHCP; udhcpc takes 0.4-0.5 s.
|
|
501
|
+
5. **busybox `stty -F`** said "Not a tty" on `/dev/hvc0`; the test sets the tty up through a
|
|
502
|
+
descriptor instead (`exec 3<>/dev/hvc0; stty ... <&3`).
|
|
503
|
+
|
|
504
|
+
## P-B4: alpine from packs
|
|
505
|
+
|
|
506
|
+
Two paths, both from `alpine.pack` (alpine:3.20 linux/arm64, made by `test/fixtures-gen`
|
|
507
|
+
from the layer in `~/.wmd/data`, read only) served by the JS 9P server under `at: 'rootfs'`
|
|
508
|
+
with `bundle/config/config.json` beside it, i.e. the tree the engine exports
|
|
509
|
+
(`internal/engine/ninep`):
|
|
510
|
+
|
|
511
|
+
1. **Proof kernel and initramfs** (`?case=alpine`). `/proof.sh` mounts the share at `/mnt`
|
|
512
|
+
(`p9MountOptions()`: `trans=virtio,version=9p2000.L,msize=262144,cache=mmap`, see the
|
|
513
|
+
review's F1 below; it was `cache=loose` for the numbers in this table), runs `ls -R /mnt/rootfs`,
|
|
514
|
+
mounts proc, sys and devtmpfs in it and runs `chroot /mnt/rootfs /sbin/apk info`, a
|
|
515
|
+
shell, `ls -R /` inside the chroot, and writes a file into the upper.
|
|
516
|
+
2. **The container path, as the engine boots it** (`?case=container`): the container
|
|
517
|
+
kernel and the Go init's initramfs from `~/.wmd/bin/guest/arm64`, the engine's kernel
|
|
518
|
+
command line without `wmd.control` (`wmd.init wmd.p9tag=wmdroot wmd.exit`, plus
|
|
519
|
+
`p9InitCmdline()`, `wmd.p9msize=262144 wmd.p9cache=mmap`, since the review; the init
|
|
520
|
+
then runs the bundle's process directly, as `guest/linux/test-init-boot.sh` does
|
|
521
|
+
without `control`). The init mounts the 9P share at `/run/wmd9p`, binds `rootfs/`,
|
|
522
|
+
writes runc's config from `bundle/config/config.json` and runs `crun`; the process is
|
|
523
|
+
`/bin/sh -c '...; ls -R /; apk info; ...'` in alpine. The init then writes
|
|
524
|
+
`bundle/exitcode` over 9P, which the page reads back (`channels.collect`).
|
|
525
|
+
|
|
526
|
+
| | Chromium | Firefox | WebKit |
|
|
527
|
+
|---|---|---|---|
|
|
528
|
+
| proof initramfs: boot to `PROOF-DONE` (ms) | 9,943 | 12,162 | 9,627 |
|
|
529
|
+
| `ls -R /mnt/rootfs` (710 lines, 0 errors) | 0.88 s | 0.69 s | 0.84 s |
|
|
530
|
+
| `chroot ... apk info` (14 packages) | 0.43 s | 0.38 s | 0.40 s |
|
|
531
|
+
| `ls -R /` in the chroot (proc and sys mounted) | 3.56 s | 5.84 s | 3.42 s |
|
|
532
|
+
| 9P requests / channel round trip p50 | 2,774 / 35.5 µs | 2,774 / 40.5 µs | 2,774 / 80.5 µs |
|
|
533
|
+
| container: bundle mounted (ms) | 19,654 | 19,631 | 19,667 |
|
|
534
|
+
| container: `ls -R /` (24,334 lines, 0 errors), `apk info` (14), `B4-DONE` (ms) | 25,400 | 27,649 | 25,406 |
|
|
535
|
+
| container: exit code; `bundle/exitcode` read back | 0; `0` | 0; `0` | 0; `0` |
|
|
536
|
+
|
|
537
|
+
Of the 19.7 s before the bundle mount, 4 s are the kernel and the init's start and 15.5 s
|
|
538
|
+
are `udhcpc` waiting with no network (problem 4).
|
|
539
|
+
|
|
540
|
+
### What the Go init needs over the control channel (not built here)
|
|
541
|
+
|
|
542
|
+
With `wmd.control=vport` (the engine's command line) the init (`guest/container2wasm/cmd/init`)
|
|
543
|
+
opens the virtio-console port named `wmd.control` (`/dev/vportNp1`; kind 3 here) and runs a
|
|
544
|
+
hashicorp/yamux **server** on it; the host is the yamux **client** (keepalive every 5 s) and
|
|
545
|
+
opens one stream per operation (`internal/engine/control.go`, `ctlproto.go`):
|
|
546
|
+
|
|
547
|
+
- **run**: one JSON line `{"op":"run","now":<unix s>,"seed":<base64, 32 bytes>,"p9tag":"wmdroot","p9opts":"trans=virtio,version=9p2000.L,msize=65536,cache=loose"}`.
|
|
548
|
+
The init refuses a first `run` without `p9opts`. `now` sets the guest clock and `seed` is
|
|
549
|
+
credited to the kernel's pool (`RNDADDENTROPY`), which is the proper fix for problem 2
|
|
550
|
+
(`crypto.getRandomValues` on the host). Then frames both ways, `id[1] | len[4] BE |
|
|
551
|
+
payload`: 0 control (`{"resize":[r,c]}`, `{"close_stdin":true}`), 1 stdin, 2 stdout, 3
|
|
552
|
+
stderr, 4 exit (4-byte BE code, optional flags, bit 0 "synced"). The host sends
|
|
553
|
+
`close_stdin` at once when there is no stdin.
|
|
554
|
+
- **exec / kill / pause / resume / ps / stats**: the same line with their fields;
|
|
555
|
+
non-streaming ops answer one JSON line.
|
|
556
|
+
- After the main process exits the init syncs, closes the session, writes `bundle/exitcode`
|
|
557
|
+
and powers off.
|
|
558
|
+
|
|
559
|
+
So P-B4 with the engine's own command line needs a yamux client in the service Worker
|
|
560
|
+
(kind 3), the run request above, and the frame demux to the page; the transport underneath
|
|
561
|
+
(the control channel) is the one measured above.
|
|
562
|
+
|
|
563
|
+
### What remains for P-B4
|
|
564
|
+
|
|
565
|
+
- The control channel's yamux client and run request (above), instead of the direct-run
|
|
566
|
+
command line.
|
|
567
|
+
- Packs from OPFS (`syncHandleSource` is written, untested in a browser; the tests fetch the
|
|
568
|
+
pack into memory), several layers per image, and writing the upper back out.
|
|
569
|
+
- Network (P-B10): the 15 s `udhcpc` wait, and `apk add`.
|
|
570
|
+
- Entropy from the host (problem 2) once the control channel carries the seed.
|
|
571
|
+
- The browser engine's run request must send `p9opts` from `p9MountOptions()` (cache=mmap),
|
|
572
|
+
not the native engine's cache=loose (review F1 below).
|
|
573
|
+
|
|
574
|
+
## P-B3/P-B4 adversarial review: findings and fixes
|
|
575
|
+
|
|
576
|
+
The independent review (`test/b3-adversarial-results.txt`, commit `d6d41e41`) found F1-F5.
|
|
577
|
+
Fixes on `wmd-web-b3`; each has a test that failed before and passes after (unit: the first
|
|
578
|
+
13 tests of the new `test/unit/b3-fixes.test.js` run against the pre-fix `src/` give 2 pass,
|
|
579
|
+
11 fail; the two that pass are F4a's gap tests, which fail under the review's mutation M8
|
|
580
|
+
instead; the O_APPEND and step-budget tests, added later, each failed before their fix).
|
|
581
|
+
Browser tests: `test/e2e/b3-fixes.spec.js` (opt-in, `WMD_WEB_B3ADV=1`, under the boot lock).
|
|
582
|
+
|
|
583
|
+
| | Status | Fix | Test |
|
|
584
|
+
|---|---|---|---|
|
|
585
|
+
| F1 full upper + cache=loose hangs the writer | **fixed for the browser's mount options**: the guest mounts with `cache=mmap`; cache=loose cannot be fixed host-side (below) | `src/host/p9mount.js` (`p9MountOptions()`, `p9InitCmdline()`), used by `page/chan.js`, `page/b3-initrd.js`, `test/features/b3r-page.js` | b3-fixes "the upper memory cap with the default mount options" (C/F/W pass); the review's cache=loose test still fails, as it must |
|
|
586
|
+
| F1 follow-up: concurrent O_APPEND under cache=mmap lost records | fixed | `src/fs/9p/server.js` `Twrite` appends for O_APPEND fids (as QEMU) | unit "two fids opened with O_APPEND ..."; the review's container torture (`conc.append`) and b3-fixes "torture on the default mount options" |
|
|
587
|
+
| F2 no default upper cap | fixed | `src/boot.js` `upperCapDefault()`, `channels.maxUpperBytes`, `timings.chanUppers`; `src/fs/overlay.js` `full()`/`fullHits`/`onFull`; `chan-service.js` `upper-full` message; `chan-services.js` `snapshot().uppers` | unit F2 (3); b3-fixes "channels.maxUpperBytes ..." and "the default cap ..." |
|
|
588
|
+
| F3 console input unbounded | fixed | `src/host/chan.js` `consoleInputGate`, `H_CIN_TAKEN`; `chan-services.js` `consoleInput()` bound; `boot.js` `vm.consoleWrite()` returns the bytes taken, `vm.consoleSpace()` | unit F3 (3); the review's unit finding test now passes; b3-fixes 512 MiB `vm.consoleWrite` to a guest that reads nothing |
|
|
589
|
+
| F4a recv vs the exit flag untested | test added | - | unit F4a (2): fail under M8 (drop `\|\| exiting()`), pass otherwise |
|
|
590
|
+
| F4b chan.spec leak check vacuous | made real | - | chan.spec "powers off ... nothing leaks": every Worker the page made was terminated (Worker wrapper), Chromium: no worker target left (CDP) |
|
|
591
|
+
| F4c control order/content unchecked | fixed | `chan-services.js` sink CRC-32 (`kinds.control.crc32`) | unit F4c; chan.spec control 50 MiB: CRC equals that of `yes \| head -c 50M` |
|
|
592
|
+
| F4d "the guest never waited" not asserted | asserted | - | chan.spec control: `sendWaits` 0 |
|
|
593
|
+
| F5 blocked guest threads wake only at the next 1 s slice | fixed | `chan.js` `markGuestGone` kicks every ring; recv/send also stop on the guest-gone flag | unit F5 (3): 0.1-0.3 ms after (700 ms before); the review's ring exit test bound 2,000 -> 600 ms |
|
|
594
|
+
|
|
595
|
+
### F1: why cache=mmap, with numbers
|
|
596
|
+
|
|
597
|
+
**Why loose cannot be fixed on the host side.** Linux 6.1's v9fs with cache=loose writes
|
|
598
|
+
into the page cache; `write(2)` never talks to the server. Writeback sends one 4 KiB page per
|
|
599
|
+
Twrite (`fs/9p/vfs_addr.c` `v9fs_vfs_writepage`, v6.1.187). On any error but EAGAIN the page
|
|
600
|
+
is dropped and the error only recorded (`mapping_set_error`) for a later fsync/close; EAGAIN
|
|
601
|
+
redirties the page (an endless retry). So no errno (ENOSPC, EIO, ...) the server can return
|
|
602
|
+
makes `write(2)` fail, and no headroom the server could reserve helps either: whatever it
|
|
603
|
+
accepts past the cap is still accepted. Measured (b3-fixes "cache=loose, for the record", a
|
|
604
|
+
48 MiB cap): `dd bs=1M count=64 conv=fsync` writes all 64 MiB and fails only at its fsync
|
|
605
|
+
(ENOSPC); a further 16 MiB `dd` without fsync returns 0, `sync` returns 0, and after
|
|
606
|
+
`drop_caches` the server holds at most the 48 MiB: the rest was lost without the writer
|
|
607
|
+
knowing. A writer past the dirty limit sits in `balance_dirty_pages` while writeback drains
|
|
608
|
+
failing pages one round at a time (the review's hang). With cache=mmap and cache=none,
|
|
609
|
+
`write(2)` is a Twrite of up to msize (`v9fs_file_write_iter`) and ENOSPC reaches it at once.
|
|
610
|
+
|
|
611
|
+
**cache=mmap over cache=none**: the same write path, and shared writable mmap works (none
|
|
612
|
+
refuses it with EINVAL, wmd D-062; the review's torture has an `mmap MAP_SHARED` check).
|
|
613
|
+
|
|
614
|
+
**What cache=mmap broke, and its fix: O_APPEND.** Without the page cache, v9fs takes an
|
|
615
|
+
O_APPEND write's offset from its i_size without a lock (`v9fs_file_write_iter` takes no
|
|
616
|
+
inode lock), so processes appending to one file through their own opens overwrite each
|
|
617
|
+
other's records: the review's torture inside a container (crun, cache=mmap from the new
|
|
618
|
+
default) failed `conc.append`: 565 records, 191 out of order. QEMU's server does not have
|
|
619
|
+
this problem (its pwrite on an O_APPEND host file appends) and neither does Linux's pwrite(2);
|
|
620
|
+
wmd's Go server ignores O_APPEND, as this one did. The JS server now writes every Twrite of
|
|
621
|
+
a fid opened or created with O_APPEND at the end of the file (`src/fs/9p/server.js`
|
|
622
|
+
`Twrite`; unit test in `b3-fixes.test.js`, fails without it). Under cache=loose writes go
|
|
623
|
+
through the writeback fid, opened without O_APPEND, so nothing changes there.
|
|
624
|
+
|
|
625
|
+
**Cost.** `b3-fixes.spec.js` "measure cache=X", one boot each (proof initramfs, 2 vCPUs, 512
|
|
626
|
+
MiB, msize 262144), then a container boot (Go init + crun) with the same mode; seconds from
|
|
627
|
+
the guest's `/proc/uptime`, one run each (they vary by about ±30% run to run, e.g. WebKit
|
|
628
|
+
mmap `ls -R` 1.22 and 2.63 s); canary 0.62-0.68 s around every run except where noted.
|
|
629
|
+
|
|
630
|
+
| | Chromium none / mmap / loose | Firefox none / mmap / loose | WebKit none / mmap / loose |
|
|
631
|
+
|---|---|---|---|
|
|
632
|
+
| `ls -R /mnt/rootfs` (710 lines) | 1.08 / 1.15 / **0.70** | 1.49 / 1.24 / **0.99** | 2.47 / 2.63 / 1.75 |
|
|
633
|
+
| the same again (warm) | 0.70 / 0.68 / **0.30** | 1.68 / 1.01 / **0.33** | 1.06 / 1.39 / 0.63 |
|
|
634
|
+
| `chroot apk info` | 0.57 / 0.52 / 0.40 | 0.85 / 0.48 / 0.42 | 0.68 / 0.54 / 0.53 |
|
|
635
|
+
| 100 x exec `/bin/true` in the chroot | 3.18 / 3.39 / 2.99 | 5.79 / 2.92 / 2.73 | 3.67 / 3.38 / 3.41 |
|
|
636
|
+
| `ls -R /` in the chroot | 4.35 / 4.27 / 3.84 | 12.27 / 6.47 / 6.13 | 6.05 / 4.78 / 5.01 |
|
|
637
|
+
| 100 MiB write / copy / read, MB/s (sha256 equal) | 37.0/57.0/95.4 / 37.9/54.8/82.9 / 33.5/40.3/93.5 | 34.5/19.5/45.8 / 40.1/26.4/40.9 / 29.2/21.8/55.2 | 43.3/34.8/67.1 / 43.3/28.7/84.9 / 38.0/27.0/88.5 |
|
|
638
|
+
| 100,000-file create, s (µs/file) | 78.9 (789) / 81.6 (816) / 134.5 (1345) | 158.5 (1585) / 159.2 (1592) / 191.7 (1917) | 146.3 (1463) / 102.4 (1024) / see below |
|
|
639
|
+
| container boot: `ls -R /` + `apk info` done (ms after start) | 25,676 / 25,659 / 25,557 | 26,736 / 26,551 / 25,965 | 28,443 / 26,885 / 26,829 |
|
|
640
|
+
|
|
641
|
+
So cache=mmap costs metadata-heavy reads (no dentry/attribute cache): a cold `ls -R` of the
|
|
642
|
+
rootfs +0.25-0.9 s, a warm one +0.4-0.8 s, `apk info` +0.0-0.1 s, and sequential reads of a
|
|
643
|
+
big file 4-26% (no readahead); it is faster for writes (100k creates 1.2-1.65x faster than
|
|
644
|
+
loose, writes 13-37% and copies 6-36% faster), and the container boot is the same (within
|
|
645
|
+
1.6 s of 25.6-28.4 s, most of it `udhcpc`'s 15 s). cache=none and cache=mmap are within
|
|
646
|
+
run-to-run noise of each other.
|
|
647
|
+
|
|
648
|
+
WebKit, cache=loose, 100,000 files: WebKit ended the page (Playwright saw a navigation, the
|
|
649
|
+
page came back empty) 25-40k files into the create, 3 times of 3 at 2 vCPUs (with and without
|
|
650
|
+
the 100 MiB copy before; the review's run at 1 vCPU and msize 65536 passed in 122.5 s), and
|
|
651
|
+
once of two under cache=none in a run where the canary read 1.28 s afterwards (the retry
|
|
652
|
+
passed). Not investigated further: the browser gives no reason (no `crash` event, no report
|
|
653
|
+
in `~/Library/Logs/DiagnosticReports`). The WebKit loose row comes from a boot without the
|
|
654
|
+
100k-file step (`B3FIX_SKIP=bigdir`).
|
|
655
|
+
|
|
656
|
+
### F2: the default cap
|
|
657
|
+
|
|
658
|
+
`bootWmd({channels})` caps each 9P service's upper at `channels.maxUpperBytes` or, by
|
|
659
|
+
default, `upperCapDefault()`: an eighth of `navigator.deviceMemory` from 256 MiB to 1 GiB
|
|
660
|
+
(Playwright's Chromium reports 16 GiB here: 1 GiB), 512 MiB where the browser does not say
|
|
661
|
+
(Firefox, WebKit). A service's own `maxUpperBytes` wins; `Infinity` turns it off.
|
|
662
|
+
`timings.chanUppers` says what each port got and why, `vm.chanStats().service.uppers` the
|
|
663
|
+
bytes used, the cap and the refusals. The first refused write (then at most one every 10 s)
|
|
664
|
+
puts on `onStderr`: `[wmd-web] the guest's files on 9P port 564 are full: the upper holds
|
|
665
|
+
32 MiB of its 32 MiB cap (channels.maxUpperBytes); the guest's writes get ENOSPC (No space
|
|
666
|
+
left on device)`, and an `{t:'chan', ev:'upper-full'}` event. Only file contents count (as
|
|
667
|
+
before); names, inodes and xattrs do not.
|
|
668
|
+
|
|
669
|
+
### F3: console input
|
|
670
|
+
|
|
671
|
+
`vm.consoleWrite(data)` returns the bytes it took (it returned `true`), at most
|
|
672
|
+
`vm.consoleSpace()`: the page counts what it posted, the service counts in the chan block
|
|
673
|
+
what it has put into the guest's ring (or had to drop), and the difference never exceeds
|
|
674
|
+
`channels.consoleInputBytes` (4 MiB). When it takes less than it was given, `onStderr` says
|
|
675
|
+
once `[wmd-web] console input is full: the guest has not read the 4 MiB already queued ...`
|
|
676
|
+
(again after a write that fits), plus a `console-full` event. The service also bounds its
|
|
677
|
+
own queue (anything posted past the bound by other means is dropped and counted:
|
|
678
|
+
`service.console.inputDropped`). For a string the count is of its UTF-8 bytes. Measured
|
|
679
|
+
(b3-fixes, a guest that reads nothing, 512 x 1 MiB writes): the page took 6,295,552 bytes
|
|
680
|
+
(4 MiB + the 1 MiB ring + 1 MiB the guest's virtio-console driver had buffered), refused
|
|
681
|
+
the rest in 0.2 s, nothing dropped; the service Worker's heap in Chromium stayed at 18 MiB
|
|
682
|
+
(540 MiB before the fix).
|
|
683
|
+
|
|
684
|
+
### Test runs after the fixes
|
|
685
|
+
|
|
686
|
+
Each under the boot lock, one browser per run, `PORT=8796`, canary 0.61-0.73 s around every run.
|
|
687
|
+
|
|
688
|
+
| | Chromium | Firefox | WebKit |
|
|
689
|
+
|---|---|---|---|
|
|
690
|
+
| `bun test test/unit` | 241 pass, 2 skip (the review's opt-in file), 0 fail; with `WMD_WEB_B3ADV=1` 243 pass, 0 fail | | |
|
|
691
|
+
| `b3-fixes` (torture on the default options, ENOSPC on the default options, the loose record, F2 cap, F3 flood, default) | 6 passed | 6 passed | 6 passed (F3 at the second run: the first failed my own bound of > 500 refused writes with 498; now > 480) |
|
|
692
|
+
| `chan.spec` | 10 passed, 1 failed (`hvc0`: problem 1, the guest's IRQ storm), 1 skipped (UART) | same | same |
|
|
693
|
+
| the review's rings (`b3-adversarial -g rings`, 7 tests) | 7 passed | 7 passed | 7 passed |
|
|
694
|
+
| the review's guests (10 tests) | 9 passed, 1 failed: ENOSPC with cache=loose (F1: cannot pass, above) | same | same |
|
|
695
|
+
| `boot.spec` + `jit.spec` | 14 passed, 1 skipped (amd64) | same | same |
|
|
696
|
+
|
|
697
|
+
- The control channel's 50 MiB: CRC-32 equal, `sendWaits` 0 in all three (F4c, F4d). The
|
|
698
|
+
leak check: 10 Workers made, all terminated, and no worker target left in Chromium; with
|
|
699
|
+
`chanSvc?.terminate()` removed from `finish()` it fails (`not terminated:
|
|
700
|
+
[{"name":"wmd-chan"}]`) (F4b).
|
|
701
|
+
- The review's ring tests after the fixes: the guest exits at random moments, last thread
|
|
702
|
+
back 207 / 207 / 209 ms after (was up to 1,010 ms); the service killed, 206 / 218 / 211 ms
|
|
703
|
+
(was 208 / 238 / 216); the 512 MiB console flood, service Worker heap 23 MiB in Chromium
|
|
704
|
+
(was 540 MiB).
|
|
705
|
+
- `alpine` and `container` in `chan.spec` now mount with cache=mmap (the default): proof
|
|
706
|
+
`ls -R` 1.2 s and `ls -R /` in the chroot 4.4 s in Chromium (3.56 s with cache=loose in
|
|
707
|
+
the P-B4 table), container `B4-DONE` at 26.8 s (25.4 s).
|
|
708
|
+
|
|
709
|
+
Problems the re-runs found, all fixed and re-run:
|
|
710
|
+
|
|
711
|
+
1. **The service could be held in one step** (`chan-services.js` `#pump`): it read a channel
|
|
712
|
+
for as long as the guest kept its ring full. With the control stub now computing a
|
|
713
|
+
CRC-32, a guest writing 1 GiB to the control channel through a 1 MiB ring kept one
|
|
714
|
+
`step()` reading 180 MB (572 ms in the unit test), so the service took its messages late:
|
|
715
|
+
the review's kill test saw the page learn of a `die`/`crash` 1.7-3.8 s late (3 of 12, all
|
|
716
|
+
on 1 MiB rings). Now one channel reads at most `STEP_READ_BUDGET` (1 MiB) per step (unit:
|
|
717
|
+
1 MiB, 6 ms).
|
|
718
|
+
2. **The review's ring harness posted console input past the new bound** (it plays the page
|
|
719
|
+
and posted straight to the service, 4 MiB ahead of the guest plus one chunk): the
|
|
720
|
+
service dropped the excess and the 8-channel test's console reader got 5.5 of 16 MiB. The
|
|
721
|
+
harness now takes input through `consoleInputGate`, as `vm.consoleWrite` does (the flood
|
|
722
|
+
test still posts past it, which is what tests the service's own bound).
|
|
723
|
+
3. **cache=mmap and O_APPEND** (F1 above): the container torture failed `conc.append` until
|
|
724
|
+
the server honoured O_APPEND.
|
|
725
|
+
4. **WebKit ended the page** in the F1 measurement's cache=loose 100,000-file step at 2 vCPUs
|
|
726
|
+
(F1 above); the review's own 100,000-file test (1 vCPU) passed in all three.
|
|
727
|
+
|
|
728
|
+
What is left:
|
|
729
|
+
|
|
730
|
+
- The review's `cache=loose` ENOSPC test still fails, by design: under cache=loose the cap
|
|
731
|
+
cannot reach `write(2)`. A host that mounts with cache=loose (the native engine's run
|
|
732
|
+
containers) must not set a cap it relies on, or must accept that writes past it are lost
|
|
733
|
+
with only fsync/close reporting it.
|
|
734
|
+
- cache=mmap gives up the dentry/attribute cache: metadata-heavy work is slower (the F1
|
|
735
|
+
table); an attribute cache with revalidation (cache=loose for metadata only) is not
|
|
736
|
+
something Linux 6.1's v9fs offers.
|
|
737
|
+
- The cap counts file contents only (names, inodes, xattrs are not charged).
|
|
738
|
+
- `hvc0` (problem 1) and the clock / entropy notes (problem 2) are unchanged.
|
|
739
|
+
|
|
740
|
+
## P-B4: the 9P server (core)
|
|
741
|
+
|
|
742
|
+
`src/fs/` is a transport-agnostic 9P2000.L server over an overlay of wmd packs (the
|
|
743
|
+
read-only lowers) under an in-memory upper: `new NinePServer(new Overlay({layers,
|
|
744
|
+
at: 'rootfs'})).handle(frame) → reply`, synchronous, so it can sit behind P-B3's
|
|
745
|
+
shared-memory channel. The pack reader is checked entry by entry against wmd's Go
|
|
746
|
+
`pack.Reader`, and the server frame by frame against wmd's Go 9P server
|
|
747
|
+
(`internal/engine/ninep` over memfs) on the same packs, alpine:3.20 included. It has
|
|
748
|
+
not run in a browser or under a guest yet. Format coverage, layering rules, messages,
|
|
749
|
+
the known differences from the Go server, the independent review (F1–F6) and the
|
|
750
|
+
numbers are in [docs/9p.md](docs/9p.md).
|
|
751
|
+
|
|
752
|
+
## Integration of P-B1, P-B2 and P-B4 (branch `wmd-web-int`, 2026-10-05)
|
|
753
|
+
|
|
754
|
+
`wmd-web` (P-B1 adversarial fixes) and `wmd-web-9p` (P-B4 9P core) merged into the
|
|
755
|
+
P-B2 branch. Everything from both sides is kept: B1's abort, release, watchdog,
|
|
756
|
+
throttle detector, panic exit, host-call counters, realtime clock and `--ram` check;
|
|
757
|
+
B2's JIT door, hub sharing under JSPI, 32 MiB stdio rings with drop markers.
|
|
758
|
+
|
|
759
|
+
Two things broke in the merge and were fixed:
|
|
760
|
+
|
|
761
|
+
- **Synthetic guests trapped.** The worker attached the JIT door to every module, and
|
|
762
|
+
the P-B1 review's synthetic guests (`test/e2e/review-synth/*.wat`) have no
|
|
763
|
+
`vm_helper_*` exports (`vm.wasm has no export vm_helper_0`). A module now gets a
|
|
764
|
+
door only if it imports `wmd.jit_load`.
|
|
765
|
+
- **Firefox kept a reloaded page's boot memory.** B2's test page held the running
|
|
766
|
+
handle on `window.wmd.vm`; after three mid-boot reloads RSS stayed 1.3 GiB higher
|
|
767
|
+
even after GC (B1's page: back to base). The page now holds the handle in a WeakRef
|
|
768
|
+
(`window.wmd.vm` still works while a boot runs).
|
|
769
|
+
|
|
770
|
+
The stubbed `jit_*` imports that B1's counters were built for only exist without a
|
|
771
|
+
door now: `bootWmd({jit: false})` / `?door=0`. B1's three stubbed-door tests ask for
|
|
772
|
+
it explicitly.
|
|
773
|
+
|
|
774
|
+
Suites on the merged branch (one run each, under the lock, canary 597–655 ms except one
|
|
775
|
+
contended stretch that was only used for A/B memory runs):
|
|
776
|
+
|
|
777
|
+
| | Chromium | Firefox | WebKit |
|
|
778
|
+
|---|---|---|---|
|
|
779
|
+
| `bun test test/unit` | 197 pass (bun) | | |
|
|
780
|
+
| `boot.spec adversarial.spec` (the filter also picks the 3 no-guest storm tests of `b2-adversarial`) | 11/11; rerun 10/11\* | 11/11 | 10/11 (reload x3 memory: WebKit, same on B1) |
|
|
781
|
+
| `fixes.spec b2-fixes.spec throttle.spec` (skips: headed throttle, opt-in 3b, Chromium-only GC test) | 16/16 | 15/15 | 15/15 |
|
|
782
|
+
| `review-errors` | 27/28 (`--cpus 64` interp) | 27/28 (same) | 27/28 (same) |
|
|
783
|
+
| `review-lifecycle review-smp` | 11/11 | 8/9 ("close a tab": no Worker close events) | 7/9 ("close a tab"; sequential boots: WebKit's lazy return, also fails on B1) |
|
|
784
|
+
| `jit.spec jit-stress` | 20/20 | 18/20 (per-Worker stress: ~16,300-copy limit, share=0) | 20/20 |
|
|
785
|
+
| `b2-adversarial` (`WMD_WEB_B2ADV=1`) | 12/12 | 12/12 | 12/12 |
|
|
786
|
+
|
|
787
|
+
\* "reload during kernel load" failed 2 of 12 runs on Chromium: the test's
|
|
788
|
+
`page.evaluate` after the page's own `location.reload()` lands in the destroyed context.
|
|
789
|
+
It passed 4 of 4 on B1 alone and 6 of 6 in the next merged run.
|
|
790
|
+
|
|
791
|
+
Proof boots on the merged branch (ms, `PROOF-DONE` from the `bootWmd` call):
|
|
792
|
+
|
|
793
|
+
| | Chromium | Firefox | WebKit |
|
|
794
|
+
|---|---|---|---|
|
|
795
|
+
| interp (P-B1 boot) | 42,207 / 39,999 | 53,798 / 52,039 | 44,761 |
|
|
796
|
+
| jit, 1 vCPU | 5,842 | 6,092 | 5,743 |
|
|
797
|
+
| jit, 8 vCPUs: copies / RSS peak | 232 / 1,224 MiB (5,523) | 234 / 1,853 MiB (5,867) | 232 / 1,400 MiB (5,580) |
|
|
798
|
+
|
|
799
|
+
## Adversarial checks (PROOFS.md P-B1)
|
|
800
|
+
|
|
801
|
+
### Independent review (test/adversarial-results.txt, commit `b819865e`) and fixes
|
|
802
|
+
|
|
803
|
+
The review's specs are `test/e2e/review-*.spec.js`; the tests for the fixes are
|
|
804
|
+
`test/e2e/fixes.spec.js`, the reload tests in `adversarial.spec.js`,
|
|
805
|
+
`test/e2e/throttle.spec.js` and `test/unit/monitor.test.js`, `gate.test.js`. Each fix's
|
|
806
|
+
test was run against the pre-fix host (Chromium, commit `b819865e` with the new specs)
|
|
807
|
+
and failed there, except where noted.
|
|
808
|
+
|
|
809
|
+
| Finding | Fix | Status |
|
|
810
|
+
|---|---|---|
|
|
811
|
+
| AbortSignal fired while vm.wasm compiles is never seen; boot keeps running | `src/boot.js` `bootWmd`: listener registered at the start; every await is raced against the abort; `signal.aborted` checked after the last await. Before the Workers exist the boot rejects with `AbortError` (no Worker is created); after, `exited` gives `{code:-1, reason:'aborted'}` and every Worker is terminated. | fixed. Ends 2.5 ms after the abort (Chromium); passes in C/F/W, and the review's own abort test passes in C/F/W. (The two guard tests, abort before the call and abort after the Workers started, also passed before the fix.) |
|
|
812
|
+
| `--mode jit`: one postMessage per stubbed `jit_*` call (477k in 30 s), unbounded `events` | `src/host/layout.js` `noteHostCall`/`hostCallCounts`, `src/host/wmd-imports.js` `counters`: Int32 counters in the control block. The page gets one `host-calls` event at most every 250 ms (`onTick`), `vm.stats().hostCalls` on demand. The test page keeps the latest counts and caps `events` at 1000. stdout/stderr through shared rings (see Output). | fixed. 395k `jit_compile` calls in 10 s → 39 events; page lag ≤ 10 ms and `evaluate` 1-4 ms over 30 s (the review's jit test) in C/F/W |
|
|
813
|
+
| Memory kept across sequential boots (WebKit to ~2.5 GiB over 5 boots) | `src/boot.js` `release()`: Workers terminated and their handlers cleared, timers, abort listener, module and file buffers dropped; `vm.memory` is a getter nulled by `vm.dispose()`; the test page disposes and keeps no handle. | host fixed: no boot's memory survives (Chromium forced GC: WeakRef cleared; WebKit: 0 of 8 memories alive after each boot). WebKit RSS still varies, see "Memory" below |
|
|
814
|
+
| Minimised Chromium window: boot 6.2x slower, page still "visible", nobody told | throttle detector (`src/calib-worker.js`, `src/host/monitor.js` `createThrottleMeter`, wired in `bootWmd`) | detected within 2.5 s in a minimised window (ratio 2.4-3.2), never in visible/headless boots. See "Background throttling" |
|
|
815
|
+
| Truncated kernel: silent hang | progress watchdog (`createWatchdog`, `watchdogMs`, default 60 s arm64): code -2, `guest made no progress` | fixed. The review's 120 s-bounded test now ends at the watchdog (about 60 s) in C/F/W; with `watchdogMs: 5000` it ends in 6-8 s. Guest fix needed (below) |
|
|
816
|
+
| Kernel panic reported as exit 0 | `createPanicScanner` on stdout; the exit becomes -3 `kernel panic: Kernel panic - not syncing: ...`, `timings.guestExitCode` keeps the guest's 0 | fixed in C/F/W. Guest fix needed (below) |
|
|
817
|
+
| `--ram abc` gives an engine message | `ramMiB()`: `--ram "abc": expected a whole number of MiB, e.g. --ram 256`, before any download | fixed in C/F/W |
|
|
818
|
+
| Firefox/WebKit `reason` is a stack without the message | page: `reason = "<name>: <message>"`, the stack separately in `stack` | fixed (the test passed before in Chromium, whose stacks include the message) |
|
|
819
|
+
| CLOCK_REALTIME steps back across Workers (620k of 1e6 reads, up to 20 us) | `src/host/clock.js` `realNs` = shared start time + CLOCK_MONOTONIC | fixed: 0 of 1e6 backwards in C/F/W (before: 623,846, max 5.1 us, Chromium) |
|
|
820
|
+
| "reload during kernel load" reloaded before the download | the page reloads itself from inside the window (phase `running`, no console byte yet) and records the state in sessionStorage | fixed: at reload `{"phase":"running","stdoutBytes":0,"stderr":""}` |
|
|
821
|
+
| No reload test checks that the old page's Workers and memory go | `adversarial.spec.js` "reload mid-boot x3 releases ..." | added. Passes in Chromium also before the fix (Chromium frees a reloaded page's memory). Firefox and WebKit: see "Memory" |
|
|
822
|
+
| TIMER-DELTA gate accepts 1.0-1.9 | `test/e2e/helpers.js` `bootGate`: 1.0-1.3 unless `{throttled: true}` | tightened. Every unthrottled boot in this round printed 1.1 |
|
|
823
|
+
| `--cpus 64` still booting at 60 s (INFO) | none | not a host fault: 64 interpreted vCPUs on 10 cores. The review's 60 s bound still fails it in C/F (left as is) |
|
|
824
|
+
|
|
825
|
+
### Background throttling
|
|
826
|
+
|
|
827
|
+
The review found that a really minimised headed Chromium window boots 6.2x slower
|
|
828
|
+
(`PROOF-DONE` at 267.6 s) while `document.visibilityState` stays `"visible"`, so the
|
|
829
|
+
page cannot tell from the Page Visibility API. The host now measures it directly: the
|
|
830
|
+
calibration Worker's fixed loop takes about 9.5 ms on this machine in the foreground.
|
|
831
|
+
|
|
832
|
+
`test/e2e/throttle.spec.js`, headed Chromium, 2026-10-04 (`REVIEW_HEADED=...`;
|
|
833
|
+
progress at 90 s of wall time unless the boot finished):
|
|
834
|
+
|
|
835
|
+
| Window | Chromium flags / page | throttle report | guest progress |
|
|
836
|
+
|---|---|---|---|
|
|
837
|
+
| visible | none | none (21 samples, max ratio 1.05) | `PROOF-DONE` at 40.5 s, TIMER-DELTA 1.1, strict gate passes |
|
|
838
|
+
| minimised | none | **throttled after 2.5 s, ratio 2.4** | at 90 s: no `wmd-init: up` yet (visible: 15.4 s), 14.4 KB of console |
|
|
839
|
+
| minimised | `--disable-renderer-backgrounding --disable-background-timer-throttling --disable-backgrounding-occluded-windows` | throttled after 2.5 s, ratio 3.15 | the same: no init at 90 s, 14.4 KB |
|
|
840
|
+
| minimised | the page holds a Web Lock (`navigator.locks.request`) for the boot | throttled after 2.5 s, ratio 2.76 | the same: no init at 90 s, 14.4 KB |
|
|
841
|
+
|
|
842
|
+
What this shows:
|
|
843
|
+
|
|
844
|
+
- **Detection works.** In each minimised run the host reported throttling on its
|
|
845
|
+
second sample (2.5 s) and wrote `[wmd-web] the guest is running about 2.4x slower ...`
|
|
846
|
+
to the emulator log; the test page shows it in its status line. No visible or headless
|
|
847
|
+
boot in this round reported it (boot.spec requires 0 throttled samples), and the
|
|
848
|
+
watchdog stretches its limit by the ratio.
|
|
849
|
+
- **The ratio understates the slowdown.** The calibration loop runs 2.4-3.2x slower,
|
|
850
|
+
the guest about 6x (init not reached at 90 s against 15.4 s). A short burst every
|
|
851
|
+
2 s is scheduled better than a vCPU thread that never sleeps. Treat the ratio as "at
|
|
852
|
+
least this much".
|
|
853
|
+
- **Nothing a page can do helped.** Chromium's own background flags made no difference,
|
|
854
|
+
so this is not Chromium's renderer backgrounding or timer throttling; the slowdown is
|
|
855
|
+
below the browser (macOS lowering the priority of a minimised app's threads, App Nap
|
|
856
|
+
or QoS; not investigated further). A held Web Lock made no difference either.
|
|
857
|
+
`Atomics.wait(10 ms)` overslept by only 0.5-3 ms, so Worker timers are not the
|
|
858
|
+
problem; CPU time is. Audio playback or similar tricks were not tried: they are hacks
|
|
859
|
+
the browser may break, and they would not touch OS-level scheduling anyway. Only
|
|
860
|
+
minimising was measured; an occluded window or a background tab was not.
|
|
861
|
+
- **False positives.** A synthetic guest that parks 128 threads (the review's spawn
|
|
862
|
+
storm) made the calibration loop 2.1-2.6x slower while 128 Workers started, and the
|
|
863
|
+
host reported throttling for a few seconds (Chromium, Firefox). That is CPU
|
|
864
|
+
contention, which also slows the guest, so the report is not wrong, only its wording.
|
|
865
|
+
- Firefox and WebKit were not checked headed.
|
|
866
|
+
|
|
867
|
+
### Memory
|
|
868
|
+
|
|
869
|
+
5 sequential arm64 boots in one page (`review-lifecycle.spec.js`, `REVIEW_SEQ_BOOTS=5`),
|
|
870
|
+
RSS of the browser's processes after each boot, MiB:
|
|
871
|
+
|
|
872
|
+
| Browser | before the fixes (review) | after |
|
|
873
|
+
|---|---|---|
|
|
874
|
+
| Chromium (forced GC) | 880, 898, 342, 503, 254 | 404, 358, 407, 422, 432 |
|
|
875
|
+
| Firefox (no GC command) | 1578, 1545 (2 boots) | 1578, 1136, 1104, 1630, 1064 |
|
|
876
|
+
| WebKit (no GC command) | 1074, 1655, 2195, 2591, 2501 | 578, 955, 1248, 1782, 2114 (fails the review's 300 MiB bound) |
|
|
877
|
+
|
|
878
|
+
WebKit, 8 boots with a GC forced by allocation pressure after each boot (allocate and
|
|
879
|
+
drop 2 GiB of never-touched ArrayBuffers; `fixes.spec.js` 3b, `WMD_WEB_SEQ=8`):
|
|
880
|
+
RSS 1073, 1137, 1200, 1655, 750, 1155, 1164, 913 MiB, then flat at 912 for 60 s idle.
|
|
881
|
+
After every boot **0** of the boots' `WebAssembly.Memory` objects were still alive
|
|
882
|
+
(WeakRefs held by the page were all cleared) and the WebContent process was back to
|
|
883
|
+
17-18 threads (15 before the first boot), so neither the host nor a lingering Worker
|
|
884
|
+
holds any boot's memory. What stays is WebKit returning freed wasm memory to the
|
|
885
|
+
system lazily: bounded (750-1655 MiB over 8 boots, no upward trend), not freed while
|
|
886
|
+
idle. Without allocation pressure WebKit's GC does not run between boots at all (the
|
|
887
|
+
page is idle), which is why the review's test, which forces GC only in Chromium, still
|
|
888
|
+
sees growth. The host does not allocate garbage on purpose to provoke a GC.
|
|
889
|
+
|
|
890
|
+
Reload mid-boot three times (`adversarial.spec.js`), RSS MiB:
|
|
891
|
+
|
|
892
|
+
| Browser | base | after the 3rd reload, no GC | after GC |
|
|
893
|
+
|---|---|---|---|
|
|
894
|
+
| Chromium | 293 | | 434 (forced) |
|
|
895
|
+
| Firefox | 1004 | 1934 | 1099 (allocation pressure) |
|
|
896
|
+
| WebKit | 357 | 2481 | 2481 (allocation pressure did not help) |
|
|
897
|
+
|
|
898
|
+
Every old page's Workers closed within 3 s of each reload in all three browsers.
|
|
899
|
+
|
|
900
|
+
The review's "close a tab mid-boot x3" fails in Firefox both before the fixes (RSS
|
|
901
|
+
993 → 1390, 1582, 1607 after each close; this round, on the pre-fix commit) and after
|
|
902
|
+
(1000 → 1568, 1581, 1547): Firefox keeps about one boot's memory after a tab closes and
|
|
903
|
+
Playwright reports no Worker `close` events for it, so the test's Worker count cannot
|
|
904
|
+
pass there. It does not grow from round to round. Chromium passes it (280 → 273). WebKit releases
|
|
905
|
+
the memory (351 → 352 after each close) but fails the same Worker-count check. In
|
|
906
|
+
WebKit the reloaded documents' memory was not returned within 10 s even after
|
|
907
|
+
allocation pressure; a reload replaces the whole JS realm, so no host reference can
|
|
908
|
+
survive it. That is left as a WebKit behaviour (the review's reload x5 measured 2567 MiB
|
|
909
|
+
too; it plateaus).
|
|
910
|
+
|
|
911
|
+
### Needed in the guest
|
|
912
|
+
|
|
913
|
+
Not built here (the guest is out of scope for wmd-web):
|
|
914
|
+
|
|
915
|
+
- **Truncated kernel.** `guest/vm/src/board_arm64.c:527-540` loads `ksz` bytes of the
|
|
916
|
+
Image without comparing them with the header's `image_size` (offset 16). A 1 MiB
|
|
917
|
+
prefix of the 38.8 MB Image boots into garbage and spins silently. `image_size` is
|
|
918
|
+
the in-memory size including BSS (39,321,600 for the current Image, file 38,760,456),
|
|
919
|
+
so the check is `image_size != 0 && (ksz > image_size || ksz < image_size - BSS
|
|
920
|
+
allowance)`, with an allowance of a few MiB, failing with "kernel file truncated".
|
|
921
|
+
The host watchdog only bounds the hang.
|
|
922
|
+
- **Kernel panic exits 0.** `CONFIG_PANIC_TIMEOUT=-1`
|
|
923
|
+
(`guest/linux/fragment-common.config:38`) reboots at once, and PSCI `SYSTEM_RESET`
|
|
924
|
+
sets `exit_code = 0` (`board_arm64.c:702-705`). Reset should not be a clean exit
|
|
925
|
+
under `wmd.exit`: map SYSTEM_RESET to a non-zero code (or set panic_timeout=0 and let
|
|
926
|
+
the watchdog catch it). The host now reports -3 from the console line, but a panic
|
|
927
|
+
whose message is lost would still look like success.
|
|
928
|
+
|
|
929
|
+
`test/e2e/adversarial.spec.js`, Chromium. Every check boots the arm64 proof guest and
|
|
930
|
+
requires the full boot gate to pass.
|
|
931
|
+
|
|
932
|
+
| Check | What the test does | Result |
|
|
933
|
+
|---|---|---|
|
|
934
|
+
| reload mid-boot | Waits for the kernel banner and 3 more seconds, then navigates the same tab to a fresh boot. | pass (the new boot reaches `PROOF-DONE` at about 44 s) |
|
|
935
|
+
| reload during download | Re-navigates as soon as the page reports `booting`. | pass |
|
|
936
|
+
| two tabs at once | Two pages in one context boot concurrently. | both pass. `PROOF-DONE` came at about 43.6 s on one run and 47 s on another, for both tabs |
|
|
937
|
+
| tab backgrounded | After the banner, marks the page hidden and **blocks the page's main thread for 10 s**. | pass. The guest kept running: about 11 KB of console output queued up during the stall, and `PROOF-DONE` time was unchanged at about 44.5 s. |
|
|
938
|
+
|
|
939
|
+
The backgrounded-tab check simulates a background tab; it does not really background one.
|
|
940
|
+
Under Playwright 1.63 the following were measured:
|
|
941
|
+
|
|
942
|
+
- A page behind another tab, or in a minimized window, stays `visibilityState ===
|
|
943
|
+
"visible"`. This holds for headless shell, new headless and headed Chromium, and with
|
|
944
|
+
focus emulation off.
|
|
945
|
+
- CDP `Page.setWebLifecycleState({state: "frozen"})` is a no-op: page timers keep
|
|
946
|
+
firing and no `freeze` event fires.
|
|
947
|
+
|
|
948
|
+
The check therefore reproduces what a background tab does to this host: the main thread
|
|
949
|
+
stops serving events. A 10 s stall is harsher than background-tab throttling. The page
|
|
950
|
+
also falls back from `requestAnimationFrame` to timers when it is hidden. Real
|
|
951
|
+
background-tab behaviour in shipping browsers is still to be checked by hand.
|
|
952
|
+
|
|
953
|
+
## P-B2: the JIT door
|
|
954
|
+
|
|
955
|
+
`src/host/jit.js` implements `wmd.jit_*` as `runner/src/jit.rs` does, for Workers.
|
|
956
|
+
|
|
957
|
+
- **One slot list for all threads, readable synchronously.** A Worker inside `jit_sync`
|
|
958
|
+
cannot receive `postMessage`, so the list lives in three SharedArrayBuffers:
|
|
959
|
+
- a header: lock, flush generation, record count, first record of this generation,
|
|
960
|
+
live slots, log bytes, largest table;
|
|
961
|
+
- a record per loaded module (growable): log offset, length, first slot, function
|
|
962
|
+
count, content hash and timings;
|
|
963
|
+
- an append-only log of the module bytes (growable, up to 1 GiB). A flush never
|
|
964
|
+
rewrites the log, so a thread reading bytes during a flush still reads valid bytes.
|
|
965
|
+
- **Table indices.** Slot k is at table index 155+k in every thread's table.
|
|
966
|
+
`jit_load` returns 155 + the slot count before the load. Slot i of a module is its
|
|
967
|
+
i-th function export (`b0..bN-1` for batch modules, `dispatch` for the chaining
|
|
968
|
+
dispatcher).
|
|
969
|
+
- **Sync.** A thread compiles each module it has not seen with the synchronous
|
|
970
|
+
`new WebAssembly.Module(bytes)` and keeps the Module by log offset. It instantiates the
|
|
971
|
+
module once per generation with `env.mem0` (and `env.mem1`, which browser blocks no
|
|
972
|
+
longer import: W50, Safari 26.5 refuses two memories) set to the one shared
|
|
973
|
+
memory, `env.tbl` set to its own table, and `hst.h0..h16` set to its own
|
|
974
|
+
`vm_helper_N`. Then it grows its table **exactly** to 155 + installed slots and sets
|
|
975
|
+
the entries. A new thread (pool Worker) syncs before `wasi_thread_start`, as
|
|
976
|
+
`spawn.rs` does.
|
|
977
|
+
- **The five imports.**
|
|
978
|
+
- `jit_load`: copy the bytes, compile, register, sync the caller. It returns -1 if the
|
|
979
|
+
module does not compile.
|
|
980
|
+
- `jit_compile` (the compile-worker thread): register without syncing. Without module
|
|
981
|
+
sharing it runs `WebAssembly.validate` first, so a bad module still returns -1. With
|
|
982
|
+
sharing (the default where JSPI exists, finding F1) it compiles, once for the
|
|
983
|
+
whole process, and posts the Module to the hub.
|
|
984
|
+
- `jit_sync`: as above. With sharing, a vCPU waits (JSPI) for Modules it has not
|
|
985
|
+
received yet instead of compiling them. A failure that is left after one retry
|
|
986
|
+
throws, which traps the thread and ends the machine with exit 4, as in the runner.
|
|
987
|
+
- `jit_flush`: bump the generation. Every thread clears `[155, size)` at its next sync,
|
|
988
|
+
and indices restart at 155.
|
|
989
|
+
- `jit_known` and `jit_note`: this session's block keys (D-513). A block whose code was
|
|
990
|
+
compiled before, wherever in memory it is now, gets that function back. With a module
|
|
991
|
+
cache across visits (W77) they also read and feed it.
|
|
992
|
+
- `jit_evict` (D-514): over the module budget, the door names the modules no block
|
|
993
|
+
entered lately, and the guest drops only those.
|
|
994
|
+
- **Unchanged from P-B1.** Address windows stay off (`win_init` → 1). The compile-worker
|
|
995
|
+
thread (`WMD_JIT_WORKERS`, default 1) runs through `thread-spawn` like any other
|
|
996
|
+
thread. Tail calls are used (the guest's default).
|
|
997
|
+
- **What is recorded.** Per module: compile time in the loading thread and in each
|
|
998
|
+
syncing thread, and instantiate time. Counters cover loads, compiles, syncs, sync time,
|
|
999
|
+
table sets and grows, bytes, dedup hits, flushes and failures. They are in
|
|
1000
|
+
`timings.jit`, in a table on the test page (refreshed every second), and in the test
|
|
1001
|
+
logs (`[<browser> <case>] jit=...`).
|
|
1002
|
+
|
|
1003
|
+
### Code budgets and big builds (wmn#57, D-511 to D-518)
|
|
1004
|
+
|
|
1005
|
+
A Go build compiles a lot of guest code. Each browser has a limit on wasm machine code:
|
|
1006
|
+
|
|
1007
|
+
- **Firefox**: one 2 GiB pool per process. `jit.maxLiveBytes` defaults to 320 MiB of live
|
|
1008
|
+
modules (D-511). Doors pull Modules from the hub instead of being sent them, so a Worker
|
|
1009
|
+
that never reads its messages holds no released code (D-512).
|
|
1010
|
+
- **WebKit**: the pool fills with no error, and code made then runs interpreted for good.
|
|
1011
|
+
The door times a tiny module (`jit.codePool`, default on in WebKit) and, while the pool
|
|
1012
|
+
is full, answers `JIT_LATER`: the guest interprets and asks again a second later (D-515).
|
|
1013
|
+
- **All**: over the budget, only cold modules go (D-514). Blocks compiled before a release
|
|
1014
|
+
compile again after 2 runs (D-517). A region's function starts at any of its blocks, and
|
|
1015
|
+
no block is compiled into a second function (D-518).
|
|
1016
|
+
|
|
1017
|
+
golang:1.23-alpine with wmn's agent (`test/e2e/go-warm.spec.js`, `WMD_WEB_GO=1`; a fmt program,
|
|
1018
|
+
`encoding/xml`, the cold set), code written by the guest's JIT:
|
|
1019
|
+
|
|
1020
|
+
| | Chromium | Firefox | WebKit |
|
|
1021
|
+
|---|---|---|---|
|
|
1022
|
+
| before D-517, D-518 | 1.07 GB | 1.29-1.53 GB | 7.6 GB, fmt program not built in 600 s |
|
|
1023
|
+
| now | 0.52 GB, 7 s / 12 s / 93 s | 0.46-0.56 GB, 16-18 s / 16 s / 124-126 s | 0.60-0.62 GB, 16-33 s / 22-31 s / 564-574 s |
|
|
1024
|
+
|
|
1025
|
+
### Numbers (2026-10-04, Apple Silicon, one run each, 1 vCPU, --ram 256; canary 620–654 ms around every run)
|
|
1026
|
+
|
|
1027
|
+
The arm64 proof boot is the P-B1 boot with `--mode jit`. Browser times start at the
|
|
1028
|
+
`bootWmd` call, so they include the download. Native times are wall-clock from process
|
|
1029
|
+
start (`wmd-run --wmd-jit-cache off`, so it has no on-disk cache either, as in the
|
|
1030
|
+
browser).
|
|
1031
|
+
|
|
1032
|
+
| arm64 proof boot | native interp | native JIT | Chromium interp (P-B1) | Chromium JIT | Firefox interp (P-B1) | Firefox JIT | WebKit interp (P-B1) | WebKit JIT |
|
|
1033
|
+
|---|---|---|---|---|---|---|---|---|
|
|
1034
|
+
| `wmd-init: up` (ms) | 19,977 | kernel 3.93 s* | 17,342 | 4,711 | 21,653 | 4,711 | 19,169 | 4,773 |
|
|
1035
|
+
| `PROOF-DONE` (ms) | 51,554 | n/a* | 44,042 | **6,584** | 57,534 | **7,135** | 49,166 | **6,752** |
|
|
1036
|
+
| total (exit) (ms) | 52,024 | 6,526 (wall) | 44,240 | 6,687 | 57,772 | 7,222 | 49,431 | 6,847 |
|
|
1037
|
+
| speed-up of `PROOF-DONE` over P-B1 | – | – | – | **6.7x** | – | **8.1x** | – | **7.3x** |
|
|
1038
|
+
| speed-up of `wmd-init: up` | – | – | – | 3.7x | – | 4.6x | – | 4.0x |
|
|
1039
|
+
|
|
1040
|
+
\* The native console has no wall-clock marks. `Run /init` is at kernel time 3.93 s
|
|
1041
|
+
native, 4.17 s in Chromium and 4.02 s in WebKit. The native run powered off at kernel
|
|
1042
|
+
time 6.25 s, with 6.53 s wall in total.
|
|
1043
|
+
|
|
1044
|
+
Interp boots in the same sessions as the JIT boots gave `PROOF-DONE` at 43,307 ms
|
|
1045
|
+
(Chromium), 56,542 ms (Firefox) and 48,543 ms (WebKit), all within 2% of the P-B1 table.
|
|
1046
|
+
|
|
1047
|
+
| JIT door, proof boot | Chromium | Firefox | WebKit |
|
|
1048
|
+
|---|---|---|---|
|
|
1049
|
+
| modules / functions | 196 / 8,927 | 198 / 8,935 | 198 / 8,933 |
|
|
1050
|
+
| module bytes (all loads) | 26.5 MB | 26.5 MB | 26.5 MB |
|
|
1051
|
+
| compile per module in the vCPU's sync: total / p50 / p99 (ms) | 49.9 / 0.27 / 0.54 | 304.9 / 1.48 / 7.26 | 85.8 / 0.42 / 1.34 |
|
|
1052
|
+
| `WebAssembly.validate` in `jit_compile`, total (ms) | 48.1 | 118.7 | 100.3 |
|
|
1053
|
+
| instantiate: total / p50 / p99 (ms) | 10.5 / 0.055 / 0.13 | 6.7 / 0.04 / 0.08 | 16.7 / 0.08 / 0.14 |
|
|
1054
|
+
| time inside `jit_sync` on all threads (ms) | 96.9 | 330.2 | 124.9 |
|
|
1055
|
+
| modules per second of boot | 30 | 28 | 29 |
|
|
1056
|
+
| table size at the end | 9,082 (= 155 + live slots) | 9,090 | 9,088 |
|
|
1057
|
+
| shared wasm memory / module log | 640 MiB / 26.5 MB | 640 MiB / 26.5 MB | 640 MiB / 26.5 MB |
|
|
1058
|
+
|
|
1059
|
+
The per-module compile is cheap because these engines compile lazily or in tiers: V8
|
|
1060
|
+
validates and compiles each function with Liftoff on first call. The host side of the
|
|
1061
|
+
door costs 0.1–0.3 s of a 6.6–7.1 s boot. Nearly all of the rest is guest execution,
|
|
1062
|
+
the same as native, so the remaining gap to native JIT is engine code quality, not the
|
|
1063
|
+
door. Native wasmtime/Cranelift reports `compile_ms=2999` for the same 194 modules:
|
|
1064
|
+
it compiles everything eagerly, with an optimizing compiler, on its compile thread.
|
|
1065
|
+
|
|
1066
|
+
The shared memory size is the same as in interp mode (640 MiB). JS heap and engine
|
|
1067
|
+
code memory were not measured (there is no cross-browser API for them from a page).
|
|
1068
|
+
|
|
1069
|
+
### What each step bought
|
|
1070
|
+
|
|
1071
|
+
| Step | Effect |
|
|
1072
|
+
|---|---|
|
|
1073
|
+
| 1. The door as specified (compile in `jit_compile` and again in each syncing thread, chunked table growth) | Proof boot passed in Chromium at 6.84 s (6.4x). Spawned threads trapped `null function` in `threads` (see problem 1). |
|
|
1074
|
+
| 2. Grow each table exactly to its installed slots | Fixed the trap. No speed change. |
|
|
1075
|
+
| 3. `jit_compile` validates instead of compiling | Chromium, one boot each: compile 6,551 ms, validate 6,550 ms, parse-only 6,599 ms. No speed change. It halves the compiled copies per module, which matters for Firefox's code limit (problem 4). Kept as the default. |
|
|
1076
|
+
| `WMD_NO_KNOWN=1` (44,800 `jit_known` calls avoided) | 6,518 ms vs 6,551: noise. Not set. |
|
|
1077
|
+
| `WMD_JIT_WORKERS=0` (the vCPU compiles) | 6,407 ms: noise at one run. Kept the guest default. |
|
|
1078
|
+
| `--no-tail` | 6,326 ms: noise at one run. Kept the guest default (tail calls). |
|
|
1079
|
+
|
|
1080
|
+
The 5x target was met at step 1, so no batching, async compile or cross-run caching was
|
|
1081
|
+
needed. A content-hash cache is in place anyway: identical bytes are logged once and
|
|
1082
|
+
compiled once per Worker (765 hits in the `--jit-batch 1` boot).
|
|
1083
|
+
|
|
1084
|
+
### Tests (all run; output in the P-B2 report)
|
|
1085
|
+
|
|
1086
|
+
| Test | Chromium | Firefox | WebKit |
|
|
1087
|
+
|---|---|---|---|
|
|
1088
|
+
| unit: slot list, sync, flush, dedup, exact table size, 3,000 modules, mutation switch, 8 real Workers with and without `jit_flush` under load (`bun test test/unit`, 46 tests) | pass (bun/JSC) | | |
|
|
1089
|
+
| P-B1 boots (`boot.spec.js`) with the new output path | pass | pass | pass |
|
|
1090
|
+
| P-B1 adversarial (reload mid-boot, during download, two tabs, stalled tab) | pass | – | – |
|
|
1091
|
+
| synthetic images from `runner/tests/vm.rs`: hot loop JITs; dispatch survives IC IALLU; 2 vCPUs, 32 flushes while one chains | pass | pass | pass |
|
|
1092
|
+
| user-mode sha256 (output, `interp_fallbacks=0`, jit > 1M, chain > 1.5); threads x5 | pass | pass | pass |
|
|
1093
|
+
| self-modifying code: `smc.c` and `smcloop` x2 inside the Linux guest, jit and interp | pass | pass | pass |
|
|
1094
|
+
| **P-B2 proof boot: boot gate + JIT gate + `PROOF-DONE` ≤ P-B1/5** | pass | pass | pass |
|
|
1095
|
+
| thousands of modules: proof boot with `--jit-batch 1` (8,677 / 8,662 / 8,716 modules) | pass, 7.9 s | pass, 12.2 s | pass, 12.8 s |
|
|
1096
|
+
| mutation `brk=load` (`jit_load` → -1) fails the JIT gate | fails it: dispatch_calls=0, chain_len=0 | same | same |
|
|
1097
|
+
| mutation `brk=all` (both → -1) fails the JIT gate | fails it: no exit within P-B1's time | same | same |
|
|
1098
|
+
| stress: 8 Workers x 300 modules, no flush | pass | **fail: out of memory** | pass |
|
|
1099
|
+
| stress: `jit_flush` every 25 loads under load, then flush + load; tables identical, nothing stale | pass | pass | pass |
|
|
1100
|
+
| stress: 4 Workers x 1,500 modules | pass | **fail: out of memory** | pass |
|
|
1101
|
+
|
|
1102
|
+
The JIT gate (`jitGate` in `test/e2e/jit.spec.js`) requires all of these:
|
|
1103
|
+
|
|
1104
|
+
- at least one `jit_load`, no door failures, at least 2 modules, instantiations and
|
|
1105
|
+
table sets;
|
|
1106
|
+
- the guest's own `--stats`: `jit_loads > 0`, `compiled > 0`, more blocks run compiled
|
|
1107
|
+
than interpreted, `dispatch_calls > 0` and `chain_len > 1.5`;
|
|
1108
|
+
- for the proof boot, `PROOF-DONE` ≤ P-B1/5 for that browser.
|
|
1109
|
+
|
|
1110
|
+
### Problems found, and their causes
|
|
1111
|
+
|
|
1112
|
+
1. **`null function` trap in a spawned guest thread** (user-mode `threads` in JIT mode).
|
|
1113
|
+
The guest's chaining dispatcher reads `idx >= table.size` as "not synced yet" and
|
|
1114
|
+
returns `IR_EXIT_SYNC_MISS` so the vCPU calls `jit_sync` (`guest/vm/src/wasm_emit.c`
|
|
1115
|
+
around line 5716). The first version grew tables in chunks, which left null entries
|
|
1116
|
+
below `table.size`. The fix grows exactly, as `jit.rs` does, and
|
|
1117
|
+
`test/unit/jit.test.js` now checks it.
|
|
1118
|
+
2. **`smc.c` is not a user-mode test.** It printed `SMC STALE` in the browser in user
|
|
1119
|
+
mode, and native `wmd-run` prints the same for it in user mode, in interp *and* JIT.
|
|
1120
|
+
The store check (D-159) is in the machine-mode MMU. The test now runs `smc.c` as a
|
|
1121
|
+
process inside the arm64 Linux guest: a newc archive is appended to the proof
|
|
1122
|
+
initramfs at boot (`page/smc-initrd.js`). Next to it runs `smcloop`, a hand-assembled
|
|
1123
|
+
static ELF that calls its function 50,000 times, so that function is a compiled
|
|
1124
|
+
wasm function when it is overwritten, then 50,000 more times. Native
|
|
1125
|
+
`wmd-run --mode jit` passes the same initramfs.
|
|
1126
|
+
3. **The page froze when the guest flooded stderr.** With both JIT imports broken,
|
|
1127
|
+
vm.wasm retries compiles in a tight loop and dumps each failed block's IR, about
|
|
1128
|
+
127,000 dumps (25 MB) in 3 s. One `postMessage` per `fd_write` drowned the page's
|
|
1129
|
+
event loop, and `page.evaluate` stopped answering. Output now goes through
|
|
1130
|
+
SharedArrayBuffer rings (`src/host/stdio.js`), and the page keeps only the tail of
|
|
1131
|
+
each stream.
|
|
1132
|
+
4. **Firefox runs out of wasm code memory at roughly 16,000–24,000 compiled modules
|
|
1133
|
+
per process.** `new WebAssembly.Module` throws `InternalError: out of memory`.
|
|
1134
|
+
Single-Worker probes in Firefox:
|
|
1135
|
+
- 8,000 modules: pass;
|
|
1136
|
+
- 2 Workers x 8,000 compiled copies: pass;
|
|
1137
|
+
- 4 Workers x 6,000: fail.
|
|
1138
|
+
Each Worker compiles its own copy, because a compiled Module can only be handed
|
|
1139
|
+
between Workers through `postMessage`, which a busy vCPU never reads. Each copy
|
|
1140
|
+
costs at least one code allocation. The real guest stays far below the limit: the
|
|
1141
|
+
proof boot has about 200 modules, and `--jit-batch 1` about 8,700 copies on the one
|
|
1142
|
+
vCPU, which passes. A many-vCPU guest with thousands of modules per vCPU would hit
|
|
1143
|
+
it in Firefox. Chromium and WebKit pass the same stress tests. Step 3 halves the
|
|
1144
|
+
copies. **Superseded:** the adversarial review measured the threshold at 2 vCPUs with
|
|
1145
|
+
`--jit-batch 1`, and the fix shares one compiled Module per process. See F1 below.
|
|
1146
|
+
5. **Firefox and WebKit spend more in sync with very many tiny modules.**
|
|
1147
|
+
`--jit-batch 1` costs 4.3 s (Firefox) and 5.7 s (WebKit) inside `jit_sync`, against
|
|
1148
|
+
1.7 s and 1.1 s of compile time. A microbenchmark (3,000 modules, no guest) puts the
|
|
1149
|
+
door's own bookkeeping at 0.03–0.09 ms per module, and `table.grow` is not the cost.
|
|
1150
|
+
The rest is unprofiled; GC of about 8,000 Modules and Instances is the likely cause.
|
|
1151
|
+
It does not affect the default batching (about 200 modules).
|
|
1152
|
+
|
|
1153
|
+
### P-B2 adversarial review: findings and fixes
|
|
1154
|
+
|
|
1155
|
+
The independent review is `test/b2-adversarial-results.txt` (commit `e4a3dfe5`), with its
|
|
1156
|
+
tests in `test/e2e/b2-adversarial.spec.js` and `test/unit/b2-adversarial.test.js`. None
|
|
1157
|
+
of its tests was changed. Every fix has a test that fails without it.
|
|
1158
|
+
|
|
1159
|
+
| Finding | Status | Where |
|
|
1160
|
+
|---|---|---|
|
|
1161
|
+
| F1 Firefox runs out of code memory (2–4 vCPUs, `--jit-batch 1`, exit 4) | **fixed**: one compiled Module per process, shared through a hub Worker; resource failures no longer trap in the shared path | `src/host/jit.js`, `src/host/jit-hub.js`, `src/worker.js`, `src/boot.js` |
|
|
1162
|
+
| F2 "jit_flush under load" flaky (3/51) | **fixed**: a flush now bumps the generation before it rewinds the slot count; 100/100 runs pass | `src/host/jit.js` `flush()` |
|
|
1163
|
+
| F3 no guest run calls `jit_flush` | **documented** (below); the no-clear mutation is now a door switch, `brk=noclear`, and the unit and browser door tests fail on it | `src/host/jit.js` `clearTable()`, `test/unit/b2-fixes.test.js`, `test/e2e/jit-stress.spec.js` |
|
|
1164
|
+
| F4 console output silently dropped after a long stall | **fixed**: a marker line in the delivered text, counted; rings configurable, 32 MiB per fd for boots; the boot gate fails on a marker | `src/host/stdio.js`, `src/boot.js`, `page/main.js`, `test/e2e/helpers.js` |
|
|
1165
|
+
| F5 memory (Firefox 2x Chromium, WebKit RSS growth) | improved by F1 for Firefox (8 vCPUs: 2,866 → 1,917 MiB peak). The WebKit growth was not attributed | – |
|
|
1166
|
+
| F6 native comparison range | not re-measured | – |
|
|
1167
|
+
| F7 failure semantics | `jit_load` after a resource failure now refuses new modules (-1) instead of failing later syncs; the rest is unchanged | `src/host/jit.js` |
|
|
1168
|
+
|
|
1169
|
+
#### F1: one compiled copy per process, not per Worker
|
|
1170
|
+
|
|
1171
|
+
**Mechanism.** Before, every vCPU Worker compiled its own copy of every module and kept
|
|
1172
|
+
it forever. Firefox counts every copy against one per-process wasm code limit, so the
|
|
1173
|
+
total ran out at about 16,200 copies. When the compile threw inside `jit_sync`, the
|
|
1174
|
+
thread trapped and the machine exited 4.
|
|
1175
|
+
|
|
1176
|
+
**What the guest can and cannot take.** A host cannot flush the door on its own to make
|
|
1177
|
+
room. The guest keeps the table index of every compiled block
|
|
1178
|
+
(`guest/vm/src/cache.c` `cp_publish`: `ir->jit_fn = base + i + 1`). Only
|
|
1179
|
+
`vm_cache_reset` gives them up, and it calls `jit_flush` itself (`cache.c:3545-3548`;
|
|
1180
|
+
`include/vm/cache.h:223-230`: "the full flush retires its compiled blocks and their
|
|
1181
|
+
table indices stay live, so nothing may drop the modules there"). What the guest does
|
|
1182
|
+
support is `jit_load` / `jit_compile` returning -1. `cp_publish` then retries each block
|
|
1183
|
+
of the batch in its own module. If that also fails, it marks the block `VM_BLK_INTERP`
|
|
1184
|
+
(the interpreter runs it from then on), counts `interp_fallbacks` and prints the block's
|
|
1185
|
+
IR once (`JITFAIL`). A failure inside `jit_sync` has no such path: the index is already
|
|
1186
|
+
published, and a table entry below `table.size` that is null traps the dispatcher.
|
|
1187
|
+
|
|
1188
|
+
**Sharing is feasible, and it is the fix.** A `WebAssembly.Module` posted to another
|
|
1189
|
+
Worker shares its compiled code, in Firefox too. The probe (`test/features/share-probe*`,
|
|
1190
|
+
`WMD_WEB_SHAREPROBE=1 bunx playwright test b2-share-probe`) gave these results in
|
|
1191
|
+
Firefox 155:
|
|
1192
|
+
|
|
1193
|
+
| probe (block-shaped modules, an instance of each kept in each Worker's table) | result | browser RSS peak |
|
|
1194
|
+
|---|---|---|
|
|
1195
|
+
| 4 Workers each compile their own 6,000 | out of memory at 4,060 + 4,131 + 4,115 + 4,036 = **16,342** copies | 1,853 MiB |
|
|
1196
|
+
| 2 Workers each compile their own 9,000 | out of memory at 8,176 + 8,169 = 16,345 | 1,943 MiB |
|
|
1197
|
+
| 1 Worker compiles 6,000, posts them, 3 others instantiate them | all 24,000 instances | 1,393 MiB |
|
|
1198
|
+
| the same with 8 Workers | all 48,000 instances | 1,462 MiB |
|
|
1199
|
+
| 1 Worker compiles 20,000, 1 (or 3) others instantiate | the compile fails at 16,345 / 16,343 distinct modules. Every received Module instantiates | 1,862 / 1,947 MiB |
|
|
1200
|
+
|
|
1201
|
+
So the limit counts compiled modules in the process, about 16,340, and instances of a
|
|
1202
|
+
shared Module cost no code space. The obstacle was delivery. A vCPU Worker never
|
|
1203
|
+
returns to its event loop, so it cannot receive a message. JSPI solves that: Chromium
|
|
1204
|
+
153, Firefox 155 and WebKit 26.6 (Playwright) all have `WebAssembly.Suspending` and
|
|
1205
|
+
`WebAssembly.promising`, in Workers too. The design:
|
|
1206
|
+
|
|
1207
|
+
- **Compile once.** The thread that calls `jit_load` or `jit_compile` compiles the
|
|
1208
|
+
module. With sharing on, `jit_compile` compiles instead of only validating. If that
|
|
1209
|
+
compile fails, the import returns -1, which the guest supports. The Worker then posts
|
|
1210
|
+
the Module to the **hub** (`src/host/jit-hub.js`), a Worker that runs no guest code.
|
|
1211
|
+
The hub forwards it to every other Worker's `MessagePort` and keeps the current
|
|
1212
|
+
generation's Modules for Workers that start later. It forwards flushes, and on a flush
|
|
1213
|
+
it drops the older Modules.
|
|
1214
|
+
- **Wait instead of compiling.** In `jit_sync` (and in the sync inside `jit_load` and
|
|
1215
|
+
`jit_flush`), a module this Worker has not received makes the import return a Promise.
|
|
1216
|
+
The worker wraps those three imports in `WebAssembly.Suspending` and calls `_start` /
|
|
1217
|
+
`wasi_thread_start` through `WebAssembly.promising`. The vCPU's wasm stack suspends,
|
|
1218
|
+
the Worker's event loop delivers the queued Modules, and the sync resumes where it
|
|
1219
|
+
stopped. A sync that finds everything returns at once, without suspending. A new
|
|
1220
|
+
thread syncs with `await door.syncAsync()`.
|
|
1221
|
+
- **Fallbacks.** A Module that has not arrived after 60 s (`jit.shareWaitMs`) is
|
|
1222
|
+
compiled locally. That happens only if the hub stops answering, and it never happened
|
|
1223
|
+
in any guest run here (`shareTimeouts=0`). If that local copy fails, the sync waits
|
|
1224
|
+
for the shared Module again instead of trapping. A Module is posted to the hub
|
|
1225
|
+
*before* its record is published, so a post that fails (Firefox past its limit failed
|
|
1226
|
+
to serialize one) returns -1 with nothing published. If the hub cannot forward a
|
|
1227
|
+
Module to some Worker, it sends `{t:'lost'}`, and that Worker compiles a copy at once. A resource failure (anything but
|
|
1228
|
+
`CompileError` / `LinkError`) is retried once. If the retry fails too, the door sets
|
|
1229
|
+
`H_FULL`, and from then on `jit_load` and `jit_compile` refuse new modules with -1, so
|
|
1230
|
+
the guest interprets new blocks and keeps running. The browser test
|
|
1231
|
+
"20,000 distinct shared modules" (4 Workers) drives Firefox past its limit to check
|
|
1232
|
+
this. Firefox: 16,339 modules, then 3,661 loads refused, every table consistent, no
|
|
1233
|
+
trap, 134 s. Chromium (1.6 s) and WebKit (25 s) compiled all 20,000.
|
|
1234
|
+
- **Release on flush (F1a).** The per-Worker `modules` map now holds only the current
|
|
1235
|
+
generation. The first sync (or received Module) of a newer generation drops it, so
|
|
1236
|
+
after a flush nothing in a Worker references an old Module. The old unit test "is not
|
|
1237
|
+
recompiled by a thread that has it" asserted the opposite. It is now "a flush releases
|
|
1238
|
+
the Worker's compiled Modules", with a second test covering every Worker.
|
|
1239
|
+
- **Default.** `bootWmd({jit: {share: 'auto'}})` turns sharing on wherever JSPI exists
|
|
1240
|
+
(`jitShareDefault`). Pass `share: false` (page `?share=0`) for the old per-Worker
|
|
1241
|
+
copies. Without JSPI (for example a shipping Safari that lacks it), the door compiles
|
|
1242
|
+
per Worker as before. There the limit and the trap on a failed sync remain. Chromium
|
|
1243
|
+
and WebKit never hit a limit in these tests.
|
|
1244
|
+
|
|
1245
|
+
**Firefox, before and after** (`test/e2e/b2-fixes.spec.js`, proof boot, one run each,
|
|
1246
|
+
canary 601–627 ms around every run; "copies" = modules compiled anywhere in the
|
|
1247
|
+
process):
|
|
1248
|
+
|
|
1249
|
+
| proof boot | before: `share=0` | after: sharing (default) |
|
|
1250
|
+
|---|---|---|
|
|
1251
|
+
| `--cpus 2 --jit-batch 1` | **exit 4**, out of memory at 16,150 copies (8,495 modules), RSS 2,415 MiB | pass, PROOF-DONE 8,968 ms, 9,340 modules / **8,924 copies**, RSS 2,395 MiB |
|
|
1252
|
+
| `--cpus 3 --jit-batch 1` | **exit 4** at 16,149 copies (5,981 modules), RSS 2,536 MiB | pass, 9,158 ms, 9,509 / 9,072 copies, RSS 2,451 MiB |
|
|
1253
|
+
| `--cpus 4 --jit-batch 1` | **exit 4** at 16,214 copies (4,625 modules), RSS 2,367 MiB | pass, 9,404 ms, 9,595 / 9,097 copies, RSS 2,518 MiB |
|
|
1254
|
+
| `--cpus 1`, default batching | PROOF-DONE 7,943 ms, 206 copies, `jit_sync` 562 ms, RSS 1,768 MiB | 6,234 ms, 199 copies, 17 ms, 1,743 MiB |
|
|
1255
|
+
| `--cpus 4` | 6,770 ms, 859 copies, 1,938 ms, 1,859 MiB | 5,769 ms, 216 copies, 58 ms, 1,905 MiB |
|
|
1256
|
+
| `--cpus 8` | 7,834 ms, 1,883 copies, 5,809 ms, **2,866 MiB** | 5,908 ms, **236 copies**, 124 ms, **1,917 MiB** |
|
|
1257
|
+
|
|
1258
|
+
With `share=0`, the retry did not help (`compileRetries` 1–3, still exit 4). That is the
|
|
1259
|
+
residual of the per-Worker mode.
|
|
1260
|
+
|
|
1261
|
+
**Chromium and WebKit, the same boots** (default batching; `share=0` → sharing):
|
|
1262
|
+
|
|
1263
|
+
| | Chromium copies | Chromium PROOF-DONE ms | Chromium `jit_sync` ms | Chromium RSS MiB | WebKit copies | WebKit PROOF-DONE ms | WebKit `jit_sync` ms | WebKit RSS MiB |
|
|
1264
|
+
|---|---|---|---|---|---|---|---|---|
|
|
1265
|
+
| `--cpus 1` | 200 → 197 | 5,955 → 5,840 | 83 → 14 | 1,100 → 1,071 | 201 → 197 | 5,803 → 5,959 | 96 → 24 | 1,265 → 1,221 |
|
|
1266
|
+
| `--cpus 4` | 875 → 215 | 5,455 → 5,316 | 349 → 48 | 1,146 → 1,162 | 855 → 212 | 6,039 → 5,540 | 526 → 92 | 1,658 → 1,127 |
|
|
1267
|
+
| `--cpus 8` | 1,867 → 233 | 5,728 → 5,495 | 836 → 97 | 1,248 → 1,216 | 1,859 → 232 | 6,711 → 5,846 | 1,370 → 194 | 1,411 → 1,297 |
|
|
1268
|
+
|
|
1269
|
+
The time a vCPU spends waiting for a Module is counted separately (`shareWaitMs`): 31 ms
|
|
1270
|
+
(Firefox), 16 ms (Chromium) and 106 ms (WebKit) at 1 vCPU, and 0.8–1.4 s across all
|
|
1271
|
+
threads in the Firefox `--jit-batch 1` boots. Each number is from one run.
|
|
1272
|
+
|
|
1273
|
+
**Hub throughput (a limit of the fix).** In Firefox, a flood of tiny modules from 8
|
|
1274
|
+
loaders at once overruns the hub. 8 Workers x 1,500 modules took 17–67 s, and once over
|
|
1275
|
+
170 s, with syncs waiting more than 10 s; Chromium did the same in 1.5 s. At 8 x 1,000
|
|
1276
|
+
it is 3.7 s. The guest boots are far from this: the `--jit-batch 1` boots at 4 vCPUs
|
|
1277
|
+
waited 1.4 s in total, over 36,000 waits. The browser stress test now uses 8 x 1,000.
|
|
1278
|
+
Batching the forwards in the hub was not tried.
|
|
1279
|
+
|
|
1280
|
+
**Suites after the fixes** (one run each per browser, under the lock, canary 595–633 ms):
|
|
1281
|
+
|
|
1282
|
+
| | Chromium | Firefox | WebKit |
|
|
1283
|
+
|---|---|---|---|
|
|
1284
|
+
| `bun test test/unit` (79 tests) | 79 pass (bun) | | |
|
|
1285
|
+
| `jit.spec boot.spec jit-stress` | 23/23 | 20/23 (below) | 23/23 |
|
|
1286
|
+
| `b2-fixes` | 5/5 | 5/5 | 5/5 (F4 marker test after the 4 KiB change) |
|
|
1287
|
+
| `b2-adversarial` (`WMD_WEB_B2ADV=1`), 12 tests | 12/12 | 12/12 | 12/12 |
|
|
1288
|
+
|
|
1289
|
+
The three Firefox failures:
|
|
1290
|
+
|
|
1291
|
+
- "no flush: 8 threads x 300" and "4 threads x 1500" are the original **per-Worker**
|
|
1292
|
+
stress tests. Their pages do not share Modules, so Firefox still runs out of code
|
|
1293
|
+
memory at about 16,300 copies, exactly as before the fixes. They are kept unchanged
|
|
1294
|
+
as the evidence for the limit. Their shared twins pass.
|
|
1295
|
+
- "shared Modules: 8 threads x 1500" timed out on that run (hub throughput, above). It
|
|
1296
|
+
is now 8 x 1000, which passed in all three browsers (Firefox 3.7 s).
|
|
1297
|
+
|
|
1298
|
+
The 20,000-module refusal test timed out in Firefox once at 8 x 3,000 (over 290 s). At
|
|
1299
|
+
4 x 5,000 it passed in 134 s.
|
|
1300
|
+
|
|
1301
|
+
#### F2: the flaky flush test
|
|
1302
|
+
|
|
1303
|
+
`flush()` used to store `H_NSLOTS = 0` before bumping `H_GEN`. A reader that does not
|
|
1304
|
+
take the lock, such as the stress check in `page/jit-stress-worker.js`, could then see 0
|
|
1305
|
+
live slots in the old generation. The door's own readers were not affected (`sync`
|
|
1306
|
+
does not read `H_NSLOTS`). The order is now `H_MOD_START`, `H_GEN`, `H_NSLOTS`: a sync
|
|
1307
|
+
still sees where a new generation starts, and a rewound slot count always comes with
|
|
1308
|
+
the new generation. `test/unit/b2-fixes.test.js` records the header writes of a flush,
|
|
1309
|
+
which fails deterministically with the old order. It also races the stress check
|
|
1310
|
+
against a Worker doing 20,000 flushes; that check caught the old order in 1 of 3 runs.
|
|
1311
|
+
"jit_flush under load", 100 runs in a row: `jit_flush under load: 100 pass, 0 fail of
|
|
1312
|
+
100`.
|
|
1313
|
+
|
|
1314
|
+
#### F3: no guest run calls `jit_flush`
|
|
1315
|
+
|
|
1316
|
+
The guest calls `jit_flush` only from `vm_cache_reset`
|
|
1317
|
+
(`guest/vm/src/cache.c:3545-3548`), and `guest/vm/src` and `guest/vm/include` contain no
|
|
1318
|
+
call to `vm_cache_reset` (only its definition, the declaration at
|
|
1319
|
+
`include/vm/cache.h:510` and comments). The guest's full flush, `vm_cache_flush`, keeps
|
|
1320
|
+
the table indices on purpose (D-166, `include/vm/cache.h:223-230`). `jit_wasm.c:64-70`
|
|
1321
|
+
says the same: "only vm_cache_reset does, and nothing in a machine run calls that, so
|
|
1322
|
+
this is not reached in production." The tests "dispatch survives an IC IALLU full
|
|
1323
|
+
flush" and "2 vCPUs, 32 flushes" exercise the guest's own block-cache flush, not the
|
|
1324
|
+
door's. The door's flush is therefore tested only at the door level:
|
|
1325
|
+
|
|
1326
|
+
- unit: `jit.test.js` (`jit_flush` describe, and 8 Workers flushing under load),
|
|
1327
|
+
`b2-adversarial.test.js` (the storm), `b2-fixes.test.js`, `jit-share.test.js` (flush
|
|
1328
|
+
with sharing);
|
|
1329
|
+
- browser: `jit-stress.spec.js` (flush under load, with and without sharing) and the
|
|
1330
|
+
storm in `b2-adversarial.spec.js`;
|
|
1331
|
+
- the adversary's mutation, a flush that does not clear the tables, is the door switch
|
|
1332
|
+
`brk: 'noclear'`. Its tests assert that the checks *fail* under it: the unit test
|
|
1333
|
+
reports "still set after the flush", and the 8-Worker stress reports "table[158] set
|
|
1334
|
+
above the 3 live slots" under bun and "table[161] set above the 6 live slots" in Firefox.
|
|
1335
|
+
|
|
1336
|
+
#### F4: dropped output is visible
|
|
1337
|
+
|
|
1338
|
+
- The writer still never blocks; a guest must not stall on a page that does not drain.
|
|
1339
|
+
- `stdioText()` (used by `bootWmd`) inserts `\n[wmd-web: N bytes of stdout were
|
|
1340
|
+
dropped]\n` into the text where the reader skipped bytes. It ends any UTF-8 sequence
|
|
1341
|
+
the skip cut, and counts the markers (`timings.outputDropMarkers`).
|
|
1342
|
+
- Ring size: `createStdio(ringBytes)` (a power of two, 4 KiB–1 GiB, stored in the
|
|
1343
|
+
header). `bootWmd` uses **32 MiB per fd** (`BOOT_RING`, override with
|
|
1344
|
+
`stdioRingBytes`). The guest console runs at about 0.33 MB/s in JIT mode, so 32 MiB
|
|
1345
|
+
covers about 100 s of a page that serves no events. That is more than the 25 s stall
|
|
1346
|
+
that lost 5.4 MB in the review, and more than Chromium's intensive throttling of
|
|
1347
|
+
hidden tabs (timers once a minute). It costs 64 MiB of SharedArrayBuffer, which is
|
|
1348
|
+
touched only as it fills, next to the guest's 640 MiB memory. `RING` (4 MiB) stays
|
|
1349
|
+
the library default, so the review's 30 MB ring flood test still overruns the ring.
|
|
1350
|
+
- `page/main.js` keeps about 8 MB of each stream (P-B1's head-and-tail cut). When it
|
|
1351
|
+
cuts, it writes the same kind of marker between the head and the tail it keeps.
|
|
1352
|
+
- `bootGate` (`test/e2e/helpers.js`) fails with `OUTPUT DROPPED (a gate line may be
|
|
1353
|
+
missing): ...` when a marker is in the log.
|
|
1354
|
+
- Tests: `stdio.test.js` (marker, count, position, UTF-8, ring size),
|
|
1355
|
+
`b2-fixes.test.js` (gate), and `b2-fixes.spec.js` in the browser. A 4 KiB ring with a
|
|
1356
|
+
6 s stall must produce markers whose byte counts add up to `outputDropped`, and the
|
|
1357
|
+
gate must fail. (A 256 KiB ring with a 6 s stall overran it in Chromium and Firefox but
|
|
1358
|
+
not in WebKit. There the guest stalled together with the page's main thread, and the
|
|
1359
|
+
kernel printed RCU stall warnings.) The review's own 1.5M-line / 25 s stall case must
|
|
1360
|
+
lose nothing with the default ring. Results: Chromium 1 marker / 1,984,818 bytes,
|
|
1361
|
+
Firefox 124 markers / 1,243,495 bytes, WebKit 237 markers / 997,626 bytes, the counts
|
|
1362
|
+
equal in each. The 25 s stall lost 0 of 1,500,000 lines in all three.
|
|
1363
|
+
|
|
1364
|
+
### Reproducing the native references
|
|
1365
|
+
|
|
1366
|
+
```sh
|
|
1367
|
+
cd ~/keg-hub/external/apps/wmd # read-only use of the main checkout's guest files
|
|
1368
|
+
# (under the boot lock)
|
|
1369
|
+
~/.wmd/bin/wmd-run --wmd-stats --wmd-jit-cache off --dir guest/linux/dist ~/.wmd/bin/guest/arm64/vm.wasm -- \
|
|
1370
|
+
--machine --arch a64 --board arm64 --mode jit --cpus 1 --ram 256 \
|
|
1371
|
+
--kernel guest/linux/dist/arm64/Image --initrd guest/linux/dist/arm64/initramfs.cpio.gz \
|
|
1372
|
+
--append "console=ttyAMA0 earlycon=pl011,0x9000000 rdinit=/init loglevel=8 wmd.proof wmd.exit" --stats
|
|
1373
|
+
```
|
|
1374
|
+
|
|
1375
|
+
The guest's `--stats` line for that run: `compiled=8782 jit_loads=194 chain_len=99.2 compile_ms=2999`, `wall: 6.526s`.
|
|
1376
|
+
|
|
1377
|
+
## P-B11: the wmn agent in the browser's guest
|
|
1378
|
+
|
|
1379
|
+
The proof: `hello_reply` and the P-K2 Python block checks pass with the wmn agent running inside
|
|
1380
|
+
wmd's guest in the browser (`test/e2e/wmn.spec.js`, `page/wmn.js`).
|
|
1381
|
+
|
|
1382
|
+
### What runs
|
|
1383
|
+
|
|
1384
|
+
1. **Image.** `fixtures-gen image -ref docker.io/library/python:3.12-slim -out test/fixtures/images/python`
|
|
1385
|
+
reads wmd's image store (`~/.wmd/data/images`, read only), picks the linux/arm64 manifest and turns
|
|
1386
|
+
each layer into a pack with wmd's `union.PackLayer` over the packs below it, as the engine's rootfs
|
|
1387
|
+
cache does (`internal/container/chaincache.go`). python:3.12-slim is 4 packs (101 MB, 3.9 MB,
|
|
1388
|
+
40 MB, 531 B), node:22-slim is 5 (98 MB, 9.9 KB, 143 MB, 7.2 MB, 657 B). The JS overlay stacks them
|
|
1389
|
+
top first under `rootfs/`. `test/unit/image-layers.test.js` compares the merged tree with what
|
|
1390
|
+
`internal/union` shows for the same packs (`fixtures-gen tree`). All 5,574 paths are equal, with
|
|
1391
|
+
type and size.
|
|
1392
|
+
2. **Bundle.** `containerBundle()` (`src/host/ctl.js`) writes the engine's spec (`internal/engine/ocispec.go`
|
|
1393
|
+
`buildSpec`: docker's capabilities, masked and read-only paths, `RLIMIT_NOFILE`, `oomScoreAdj` 0, the
|
|
1394
|
+
six default mounts, `/etc/{hosts,resolv.conf,hostname}` bound from `bundle/etc`), plus
|
|
1395
|
+
`imageconfig.json`. Each mount is a share `vol<i>` beside `bundle/` and `rootfs/`, as the engine's
|
|
1396
|
+
`layout()` does. The agent (`wmn-agent-linux-arm64` from the wmn repo's `agent/build.sh`, served at
|
|
1397
|
+
`/files/wmn-agent/`) is the file `vol0/wmn-agent` (mode 0755), bound read-only at `/.wmn/agent`.
|
|
1398
|
+
3. **Boot.** This uses the engine's arm64 command line (`run.go` `guestCmdline`, without `wmd.warmup`)
|
|
1399
|
+
with `wmd.control=vport`: `--p9 564 --p9-tag wmdroot --control 8`, 2 vCPUs, 1 GiB, `--mode jit`.
|
|
1400
|
+
Services are the 9P server over the packs plus the bundle files, and `{port: 8, kind: 'control', yamux: true}`.
|
|
1401
|
+
4. **Control session (ROADMAP 2.2b).** When the init opens `/dev/vport2p1`, the channel service starts
|
|
1402
|
+
a yamux client on it (`ControlSession`). The page's `vm.control` (`ControlClient`) opens streams
|
|
1403
|
+
through it:
|
|
1404
|
+
- `run` comes first, as `driveMainProcess` sends it: `{"op":"run","id":"main","args":["/.wmn/agent","init"],"env":...,"now":...,"seed":<32 bytes of crypto.getRandomValues, base64>,"p9tag":"wmdroot","p9opts":"trans=virtio,version=9p2000.L,msize=262144,cache=mmap"}`.
|
|
1405
|
+
- Then `exec` of `/.wmn/agent serve` (id `wmn-serve`, stdin open). The init holds it until the
|
|
1406
|
+
container runs.
|
|
1407
|
+
- The wmn protocol runs as JSON lines in stdin/stdout frames: `hello`, `languages_request`, then
|
|
1408
|
+
`page/k2.js` `runChecks`.
|
|
1409
|
+
- At the end the agent's stdin is closed, then `kill {id: main, sig: SIGTERM}`. The agent's init
|
|
1410
|
+
exits, the Go init writes `bundle/exitcode` (read back) and powers off.
|
|
1411
|
+
5. **Checks.** `page/k2.js` is k2.ts's python/node/ruby cases and its 13 checks with their bounds
|
|
1412
|
+
unchanged (copied: k2.ts imports Bun-only modules, so a page cannot load it), and agent-conn.ts.
|
|
1413
|
+
One harness-only change: a block whose agent stream ended fails at once instead of at its
|
|
1414
|
+
120 s timeout.
|
|
1415
|
+
|
|
1416
|
+
### The control client
|
|
1417
|
+
|
|
1418
|
+
- **Wire format.** It speaks hashicorp/yamux v0.1.2's wire format: 12-byte headers; odd stream ids
|
|
1419
|
+
for the client; the SYN window update on open; 256 KiB windows each way; a window update once half
|
|
1420
|
+
the window is consumed; FIN after the queued data; RST; Go Away on a protocol error; pings answered
|
|
1421
|
+
with the same value. A keepalive ping goes out every 5 s, as the engine does; a missing answer is
|
|
1422
|
+
only counted. Streams the guest opens (op `edge`) are refused with RST.
|
|
1423
|
+
- **Where it runs.** The yamux client runs in the channel service Worker, not on the page. The
|
|
1424
|
+
guest's yamux server pings every 30 s and drops the session when an answer takes more than 10 s;
|
|
1425
|
+
the page's main thread can stall that long (P-B1's stalled-tab check), the service does not.
|
|
1426
|
+
`ctl.test.js` shows the session dropping when the host stops answering.
|
|
1427
|
+
- **Backpressure.** The page acknowledges stream bytes after it has delivered them, and only then does
|
|
1428
|
+
the service give the window back. A page that stops reading stops the guest at 256 KiB.
|
|
1429
|
+
- **Protocol.** It matches `ctlproto.go` and the init: Go's `omitempty` for the request line, frames
|
|
1430
|
+
`id|len BE|payload` (at most 10 MiB), the exit frame with the synced flag, `close_stdin` at once when
|
|
1431
|
+
there is no stdin, the attach reply line before the replayed frames, and one JSON line for the other
|
|
1432
|
+
ops. A stream that ends without an exit frame gives 125 with the reason, as `runProcessExit` does.
|
|
1433
|
+
- **Unit tests.** `test/unit/ctl.test.js` has 22 tests. 10 drive the browser's path
|
|
1434
|
+
(`ControlClient` -> `ChanHost.controlInput` -> yamux -> the control channel's rings) against
|
|
1435
|
+
**hashicorp/yamux itself**: `test/ctl-guest` is a Go server with the init's stream protocol, running
|
|
1436
|
+
local processes. They cover:
|
|
1437
|
+
- the request as Go parses it;
|
|
1438
|
+
- stdin, stdout and stderr apart, exit codes, the synced flag;
|
|
1439
|
+
- 5 MiB of output in order and 3 MiB of stdin (cksum equal);
|
|
1440
|
+
- 16 processes at once;
|
|
1441
|
+
- kill, procs and attach;
|
|
1442
|
+
- a page that does not ack stopping the guest at one window;
|
|
1443
|
+
- 50 ms guest pings answered, and the session lost when the host stops answering;
|
|
1444
|
+
- the guest going away: every stream gets 125 and later opens are refused.
|
|
1445
|
+
|
|
1446
|
+
The yamux unit tests cover window accounting, split frames and the protocol errors.
|
|
1447
|
+
|
|
1448
|
+
### udhcpc
|
|
1449
|
+
|
|
1450
|
+
The run request cannot skip it. The init runs `udhcpc` (up to 15 s with no network, `-t 5 -T 3`)
|
|
1451
|
+
before it opens the control channel, whenever `/sys/class/net/eth0` exists, and the board always has a
|
|
1452
|
+
virtio-net device. The engine does not skip it for `--network none` either: it always passes `--net`,
|
|
1453
|
+
and its netstack answers. In the browser the page passes `initcall_blacklist=virtio_net_driver_init`
|
|
1454
|
+
on the kernel command line (`?nonet=1`, the spec's default). With no eth0 the init skips udhcpc:
|
|
1455
|
+
Chromium python got `hello_reply` at 5.8–6.1 s with it against 21.8 s without it (control session
|
|
1456
|
+
up at 4.4–4.7 s against 20.4 s). P-B10's relay, or a DHCP answer from the net stub, makes it moot. **Update (P-B10):** `page/net.html` boots
|
|
1457
|
+
with eth0 and a net service; udhcpc gets its lease from the network Worker in 0.4-0.5 s.
|
|
1458
|
+
|
|
1459
|
+
### Numbers (2026-10-05, Apple Silicon, Playwright 1.63, `--mode jit`, 2 vCPUs, 1 GiB, `nonet=1`)
|
|
1460
|
+
|
|
1461
|
+
ms from the `bootWmd` call (download included), one run each. `run2` and `run6` are the runs below;
|
|
1462
|
+
the canary was 640–656 ms before each Chromium and Firefox run and 1,358 ms before the WebKit
|
|
1463
|
+
part of run6. The host was shared with other sessions throughout: load average 24–32 on 10 cores.
|
|
1464
|
+
|
|
1465
|
+
| python:3.12-slim | Chromium (run2 / run6) | Firefox (run3) | WebKit (run6) |
|
|
1466
|
+
|---|---|---|---|
|
|
1467
|
+
| control session up | 4,382 / 4,653 | 9,167 | 6,840 |
|
|
1468
|
+
| `hello_reply` | **5,799 / 6,095** | 11,570 | 10,021 |
|
|
1469
|
+
| `languages_reply` | 7,234 / 7,881 | 17,437 | 13,330 |
|
|
1470
|
+
| `session_start` python (cold) | 8,068 / 8,197 | 18,110 | 12,168 |
|
|
1471
|
+
| define / use block | 52 / 49, 56 / 63 | 75 / 67 | 86 / 57 |
|
|
1472
|
+
| interrupt a loop | 41 | 229 | 69 |
|
|
1473
|
+
| infinite output, stopped in | 42 (771 KB), 20 (243 KB) | 33 (175 KB) | 44 (154 KB) |
|
|
1474
|
+
| all 13 checks done | 20,782 / 21,577 | 43,867 | 32,133 |
|
|
1475
|
+
| 9P requests | 8,372 | | 8,372 |
|
|
1476
|
+
|
|
1477
|
+
| node:22-slim | Chromium (run5 / run6) | Firefox (run6, throttled 7.8x) | WebKit (run6) |
|
|
1478
|
+
|---|---|---|---|
|
|
1479
|
+
| `hello_reply` | 6,978 / 6,571 | 50,007 | 9,867 |
|
|
1480
|
+
| `session_start` node | 4,262 / 4,568 | 4,448 | 5,229 |
|
|
1481
|
+
| define / use | 321 / 100, 297 / 117 | 243 / 81 | 266 / 144 |
|
|
1482
|
+
| interrupt a loop | 102 / 86 | 114 | 80 |
|
|
1483
|
+
| checks | 13/13, 13/13 | 12/13 | 13/13 |
|
|
1484
|
+
|
|
1485
|
+
Native wmd for comparison, from P-K2 (native wmd under the daemon, with its edge fast path):
|
|
1486
|
+
python session start 7.0 s, node 4.0 s, interrupt 8 ms (python) and 24 ms (node). So the browser's
|
|
1487
|
+
session start is about the same. The browser's checks show no network wait, because nothing in them
|
|
1488
|
+
needs the network.
|
|
1489
|
+
|
|
1490
|
+
### Results, every run (all under the boot lock, `PORT=8798`)
|
|
1491
|
+
|
|
1492
|
+
| run | Chromium | Firefox | WebKit |
|
|
1493
|
+
|---|---|---|---|
|
|
1494
|
+
| run2 python | 13/13 | | |
|
|
1495
|
+
| run3 python / node (256 MiB slack) | 13/13 / 5/7 (trap) | 13/13 / 5/7 (trap) | 0 (`not a pack`) / 5/7 (trap) |
|
|
1496
|
+
| run3 python with udhcpc | 13/13 | | |
|
|
1497
|
+
| run4 (canary 2.5 s: contended) | node 5/7 (trap) | | python 11/13, 13/13, 13/13 |
|
|
1498
|
+
| run5 node, 1 GiB slack | 13/13 | | |
|
|
1499
|
+
| run6 python / node | 13/13 / 13/13 | 0/1 / 12/13 (throttled 7.4-7.8x) | 13/13 / 13/13 |
|
|
1500
|
+
| run7 python / node, on wmd main (ee1886e8) | 13/13 / 13/13 | 13/13 / **13/13** | 13/13 / 13/13 |
|
|
1501
|
+
| run8 python / node, the finished branch (2026-10-06) | 13/13 / 13/13 | 13/13 / 13/13 | 13/13 / 13/13 (and 13/13 / 13/13 in a first run at canary 6.2 s) |
|
|
1502
|
+
|
|
1503
|
+
### Problems found, and their causes
|
|
1504
|
+
|
|
1505
|
+
1. **node trapped the machine: vm.wasm's heap hit the memory maximum** (fixed for the page).
|
|
1506
|
+
- What happened: in all three browsers, node's interrupt check never got a reply, and the machine
|
|
1507
|
+
ended with exit 4: `RuntimeError: unreachable` at `abort` <- `interp_run_block`.
|
|
1508
|
+
- Why: when it trapped, the shared memory was 1,664 MiB, exactly `memoryMaxPages()`'s maximum
|
|
1509
|
+
(384 MiB vm.wasm + 1 GiB RAM + 256 MiB slack). node's V8 is a JIT inside the emulated guest
|
|
1510
|
+
(460 block modules, 21,263 functions); it pushed the translator's heap, which shares the memory,
|
|
1511
|
+
to the maximum. `interp_run_block`'s `calloc` then returned NULL, and the guest calls `abort()`
|
|
1512
|
+
(`guest/vm/src/interp.c:848`).
|
|
1513
|
+
- Fix: the wmn page passes `maxPages` with 1 GiB of slack (`?slack=`). node then peaks at
|
|
1514
|
+
1,784–1,799 MiB and passes 13/13 in all three browsers.
|
|
1515
|
+
- Still open: `bootWmd`'s library default (256 MiB) is unchanged for the older suites. The guest
|
|
1516
|
+
aborting on an allocation failure, instead of reporting it, is a guest finding: native `wmd-run`
|
|
1517
|
+
reserves 4 GiB, so it does not see it.
|
|
1518
|
+
2. **WebKit: `layer 1: not a pack`, once in run3** (cause not found).
|
|
1519
|
+
- What happened: the channel service could not open python's `layer-2.pack`, and the boot ended
|
|
1520
|
+
with -4 before the control session.
|
|
1521
|
+
- What was ruled out:
|
|
1522
|
+
- `wmn-fetch.spec.js` fetched every pack 3 times as files, and 15 more rounds along the boot's
|
|
1523
|
+
own path (fetchShared next to the kernel's download, then one postMessage of the
|
|
1524
|
+
SharedArrayBuffers to a Worker). Every size and both magics were right.
|
|
1525
|
+
- It did not recur in 7 later WebKit boots of python.
|
|
1526
|
+
- Next time: the service now says what arrived (bytes, head, tail, non-zero pages), so a repeat
|
|
1527
|
+
tells a bad transfer from a bad file.
|
|
1528
|
+
3. **WebKit python: interrupt took 1,319 ms, so the agent ended the session (once in 4; run4).**
|
|
1529
|
+
- Why: the agent's `InterruptGrace` (1 s, `agent/session.go`) SIGKILLs a block that has not
|
|
1530
|
+
stopped by then. The other WebKit runs stopped in 69–182 ms (Chromium 41 ms, native 8 ms).
|
|
1531
|
+
The run's canary was 2.5 s, against 0.64 s on a quiet host.
|
|
1532
|
+
- Effect: under emulation on a loaded host, the 1 s grace is close to the latency of signal
|
|
1533
|
+
delivery plus KeyboardInterrupt.
|
|
1534
|
+
- This is wmn's setting, so it is left for the agent's step (ROADMAP 4.16); the check's bound
|
|
1535
|
+
was not changed.
|
|
1536
|
+
4. **Firefox (run6), both boots on a host that the throttle detector measured 7.4–7.8x slow.**
|
|
1537
|
+
- **python: the guest dropped the control session.**
|
|
1538
|
+
- Sequence: the init logged `control: accept error: keepalive timeout`, then `session closed
|
|
1539
|
+
before main process ran`, then exit 125 and a power-off. Its yamux server's 10 s ping timeout
|
|
1540
|
+
expired in guest time.
|
|
1541
|
+
- In the same boot a vCPU waited 60 s for a JIT module from the hub. The door logged
|
|
1542
|
+
`module ... did not arrive from the hub in 60000 ms; compiling a copy`; `shareTimeouts` was 1
|
|
1543
|
+
and `shareWaitMs` 61,772.
|
|
1544
|
+
- The service answers pings at once, so the late answer was the guest's side not reading:
|
|
1545
|
+
its vCPU was stalled.
|
|
1546
|
+
- The hub's forwarding is plain postMessage. Why one module took over 60 s to arrive was not
|
|
1547
|
+
found: P-B2 measured up to 1.4 s of waits per boot; this boot ran at up to 7.8x slowdown.
|
|
1548
|
+
- It is a finding for P-B2's sharing (a lost or starved forward), with this log as the evidence.
|
|
1549
|
+
- **node: `infinite output` printed 7,676 bytes in 1.5 s.** The check wants more than 10,000. The
|
|
1550
|
+
guest ran slowly: control session up at 36 s instead of about 6 s.
|
|
1551
|
+
5. **Harness problems found on the way.**
|
|
1552
|
+
- The first run never booted: the disk was under 75 GiB when the lock was taken, and
|
|
1553
|
+
`node_modules` was missing. The runner now re-checks the disk under the lock.
|
|
1554
|
+
- Playwright empties `test-results/` at the start of every run, so the logs now go to
|
|
1555
|
+
`wmn-results/`.
|
|
1556
|
+
- `FrameReader` was O(n^2) on byte-sized input: a 70 KB frame fed one byte at a time timed out.
|
|
1557
|
+
It now keeps chunks.
|
|
1558
|
+
|
|
1559
|
+
### What remains
|
|
1560
|
+
|
|
1561
|
+
- **Ruby.** ruby:3.3-slim is not in wmd's store; only ruby:3.3-alpine is. A pull through wmd's image
|
|
1562
|
+
code into a scratch store was not done. The node case passes; ruby is not checked.
|
|
1563
|
+
- **The adversarial set from P-K2 (robustness.ts) in the browser.** Not ported here; that is the
|
|
1564
|
+
reviewer's step.
|
|
1565
|
+
- **Open causes.** The cause of problem 2 and the stalled hub forward in problem 4.
|
|
1566
|
+
- **Defaults and options.** `bootWmd`'s default slack. Packs from OPFS (here they are fetched into
|
|
1567
|
+
memory: 145 MB for python, 250 MB for node). Network (P-B10). Snapshots and `wmd.warmup`.
|
|
1568
|
+
- **Reruns.** No timed comparison runs on a quiet host: the host was loaded throughout (load average
|
|
1569
|
+
24–32).
|
|
1570
|
+
- **W21 (branch w21-trap, under review).** Not merged here. Its two commits cherry-pick onto this
|
|
1571
|
+
branch with one README conflict (section "Out-of-memory detection with W21's heap reserve"); after
|
|
1572
|
+
it, a real out-of-memory trap is reported only with a W21-built vm.wasm.
|
|
1573
|
+
- **hvc0 at 33-72 KB/s** (section "hvc0 after W17"): resolved by W53 (D-433). It was the test's
|
|
1574
|
+
2-byte tty writes, not hvc0; the 50 MiB `chan.spec` hvc0 test now passes at 2.1-3.1 MB/s.
|
|
1575
|
+
|
|
1576
|
+
### Running
|
|
1577
|
+
|
|
1578
|
+
```sh
|
|
1579
|
+
(cd ~/keg-hub/external/apps/wmn && agent/build.sh) # the agent, into the wmn repo's agent/dist
|
|
1580
|
+
go run ./web/test/fixtures-gen image -ref docker.io/library/python:3.12-slim -out web/test/fixtures/images/python
|
|
1581
|
+
go run ./web/test/fixtures-gen image -ref docker.io/library/node:22-slim -out web/test/fixtures/images/node
|
|
1582
|
+
cd web && bun test test/unit/ctl.test.js test/unit/image-layers.test.js
|
|
1583
|
+
# under the boot lock, one browser per run (about 45 s per image):
|
|
1584
|
+
WMD_WEB_WMN_IMAGES=python,node PORT=8798 bunx playwright test wmn.spec --project=chromium
|
|
1585
|
+
WMD_WEB_WMN_IMAGES=python WMD_WEB_WMN_QUERY= PORT=8798 bunx playwright test wmn.spec --project=chromium # with udhcpc
|
|
1586
|
+
PORT=8798 bunx playwright test wmn-fetch --project=webkit # no boot: the pack transfer probe
|
|
1587
|
+
```
|
|
1588
|
+
|
|
1589
|
+
### P-B11 adversarial review: findings and fixes
|
|
1590
|
+
|
|
1591
|
+
The review is `test/adversarial-b11/RESULTS.md` (branch `wmd-web-b11-adv`, 2b7f2e6d, merged). None of its
|
|
1592
|
+
tests were changed. Each fix has a test that fails on the pre-fix code (347f80ed) and passes after.
|
|
1593
|
+
|
|
1594
|
+
| Finding | Status | Fix | Test (before -> after) |
|
|
1595
|
+
|---|---|---|---|
|
|
1596
|
+
| 1 30 concurrent execs trap the machine in Chromium (`calloc` <- `interp_run_block`) | **not here**: W21, debugged separately | - | - |
|
|
1597
|
+
| 2 python driver interrupt race (c09c) | fixed in wmn's phase1 branch | - | - |
|
|
1598
|
+
| 3 `wmn.vm` collected in a hold (WeakRef) | fixed | `page/wmn.js` holds the vm strongly while it runs, drops it when it ends | b11-fixes "the vm handle survives forced garbage collection" (Chromium, 5 CDP `collectGarbage`): before `{vm: false}`, after `{vm: true, up: true}` |
|
|
1599
|
+
| 4 OOM reported as `trap: unreachable`, `wmn.error` null, a second vCPU on garbage IR | fixed | `boot.js` `finish()`: a trap with the memory within 64 MiB of its maximum becomes `out of memory: the emulator used N of its M memory maximum ...` (`timings.memoryMaxBytes`, `timings.trap` keep the raw facts) and every Worker is terminated at once; `worker.js`: every wasi/wmd/thread-spawn import (not `jit_*`, not `proc_exit`) throws once the machine's exit flag is set, so other threads stop at their next host call; `wmn.js` sets `wmn.error` on any end it did not ask for | b11-fixes synthetic `oom_trap.wat` (grows to the maximum, traps; a second thread writes once a millisecond): before, reason `trap: unreachable` and 810-6,601 lines after the trap (C/F/W); after, the out-of-memory reason and 0-2 lines, 3 rounds x 3 browsers. Real guest, node at `slack=256`: before, `trap: unreachable` and `wmn.error` null; after, `wmn.error` "the machine ended: 4 (out of memory: the emulator used 1664 MiB of its 1664 MiB memory maximum ...)", no "vreg out of range" in stderr |
|
|
1600
|
+
| 5 AgentConn quadratic (20 MB line: 16.5 s Firefox) | fixed | `page/k2.js`: each chunk searched once, line pieces joined once | `test/unit/agent-conn.test.js` (old: 5 MB 62 ms, 20 MB 957 ms, fails); b11-fixes in each browser: before 5/20 MB = C 82/1,142, F 767/16,213, W 70/477 ms; after C 3/11, F 15-16/121-152, W 13-16/19-23 ms |
|
|
1601
|
+
| 6 yamux.js buffers a frame before checking the window; unknown-stream data held | fixed | `feed()` parses incrementally: the header in 12 bytes, the window checked when the header arrives (Go Away before any body), data for a gone stream discarded as it comes; a body is kept as its pieces (at most one window) and delivered once whole | the review's A1, A2 (failed before, pass) |
|
|
1602
|
+
| 8 `feed()` quadratic on small pieces | fixed (same change) | - | the review's A3 (10.7 s before -> 76 ms), A3b (79 ms) |
|
|
1603
|
+
| 9 lenient: window credit past 2^32, `consume(NaN)` | fixed | `sendWindow` capped at 0xffffffff (Go's uint32); `consume` ignores non-finite | the review's A11, A13 (failed before, pass) |
|
|
1604
|
+
| 7 no stdin backpressure | fixed | the service reports bytes it sent on (`{ev:'sent'}`, from the stream's queue as the guest's window allows); each page stream has a credit (`STREAM_CREDIT`, 4 MiB + 64 KiB) of bytes posted and not yet sent; `CtlProcess.write()` returns false and sends nothing past it; `writeAsync()` queues in order and resolves when posted; `closeStdin()` waits for queued data; AgentConn sends through `writeAsync` | the review's G6 (service held 62.8 MiB before -> 2.8 MiB); ctl.test "stdin backpressure": 64 MiB through `cksum` equal, the service never held more than 4.00 MiB |
|
|
1605
|
+
| 9 over 256 streams: "reset by the guest" | fixed | the page keeps at most 128 streams open (`MAX_OPEN_STREAMS`, under the guest's AcceptBacklog 256); more wait their turn; a guest RST before any reply names the backlog | ctl.test: 300 at once all answer, never more than 128 open, 172 queued; the review's G1 passes (it failed 1 in ~5 runs before) |
|
|
1606
|
+
| 9 exec errors | fixed | `checkProcessRequest`: a NUL in an argument, env entry or cwd is refused on the page with the reason (the init gave a bare 125); an id in use by a live run/exec of this page is refused; exec exit 1 with crun's "executable file ... not found" -> 127, "... not executable" -> 126 (the init's `crunExecStatus`, which it applies to `run` only); crun's messages lose the internal container name `foo` | ctl.test (4 tests, all fail on the old code) |
|
|
1607
|
+
| 9 "has not opened the control port" after the session ended | fixed | the service remembers the session and why it ended: "the guest's control session N has ended (...) and none is open" | ctl.test "an open after the session ended" |
|
|
1608
|
+
|
|
1609
|
+
Not changed: c03 late output (wmn's known limit, ROADMAP 4.3b); `non-UTF-8 in args` (JSON); a duplicate FIN is
|
|
1610
|
+
still ignored (the review's A6 is INFO, it passes).
|
|
1611
|
+
|
|
1612
|
+
**Unit:** `bun test test/unit test/adversarial-b11/yamux-adv.test.js`: 302 pass, 2 skip, 0 fail (the review's
|
|
1613
|
+
file: 28/28; before: 22/28).
|
|
1614
|
+
|
|
1615
|
+
**Browser runs after the fixes** (one lock, canary 2.5 s before, 0.81 s after: a loaded host):
|
|
1616
|
+
|
|
1617
|
+
| | Chromium | Firefox | WebKit |
|
|
1618
|
+
|---|---|---|---|
|
|
1619
|
+
| b11-fixes, no guest, 3 rounds | 3/3 x 3 | 3/3 x 3 (two rounds failed on bounds that were too tight for a few-ms measurement, fixed: see below) | 3/3 x 3 |
|
|
1620
|
+
| the vm through 5 forced GCs in a hold | `{vm: true, up: true}`, then SIGTERM through it: exit 0 | - | - |
|
|
1621
|
+
| node, `slack=256` | exit 4, `wmn.error` out of memory, no garbage-IR line | - | - |
|
|
1622
|
+
| wmn python / node | 13/13 / 13/13 | 13/13 / **12/13** | 13/13 / 13/13 |
|
|
1623
|
+
|
|
1624
|
+
Two of my own bounds were tightened against flakiness, not loosened against a regression: the OOM test's
|
|
1625
|
+
second thread now writes once a millisecond (it spun before, so "lines after the trap" measured CPU speed:
|
|
1626
|
+
11 in one Chromium run), and the AgentConn ratio uses a 50 ms floor (Firefox 5 MB at 15 ms made 20 MB at
|
|
1627
|
+
121 ms a ratio of 8 while the old code took 16 s).
|
|
1628
|
+
|
|
1629
|
+
**Firefox node `infinite output`, 12/13 (run6 and this run):** 4,545 and 7,676 bytes in 1.5 s against the
|
|
1630
|
+
check's 10,000. This is the guest's JIT in Firefox, not the transport: node's flood block is new V8 code, so the
|
|
1631
|
+
guest translates it; the boot's JIT door spent 2,249 ms compiling modules in Firefox against 270 ms in
|
|
1632
|
+
Chromium (535 / 548 modules) and 2,634 ms waiting for shared modules against 199 ms (P-B2's per-browser
|
|
1633
|
+
compile cost). The same boot's python flood streamed 254 KB. Left as is; the check's bound is unchanged.
|
|
1634
|
+
|
|
1635
|
+
### On wmd main (branch rebased onto 9833319c, HEAD ee1886e8 and after)
|
|
1636
|
+
|
|
1637
|
+
wmd main brought W14/W15 (9P), W17 (virtio-console interrupts) and W18 (guest entropy from the host
|
|
1638
|
+
at boot, 70d03426). Every run below is under the boot lock, `PORT=8798`, with the installed
|
|
1639
|
+
`vm.wasm` (sha256 30db07f5..., built after W17/W18).
|
|
1640
|
+
|
|
1641
|
+
**wmn suite, run7** (2026-10-05 15:48-15:50, one boot per image and browser; canary 648/704/717 ms
|
|
1642
|
+
before and 662/662/717 ms after the Chromium/Firefox/WebKit runs; load average 12-23):
|
|
1643
|
+
|
|
1644
|
+
| ms from `bootWmd` | Chromium python / node | Firefox python / node | WebKit python / node |
|
|
1645
|
+
|---|---|---|---|
|
|
1646
|
+
| control session up | 4,345 / 4,383 | 4,858 / 5,184 | 5,202 / 5,041 |
|
|
1647
|
+
| `hello_reply` | 5,707 / 5,707 | 6,157 / 6,416 | 7,795 / 7,785 |
|
|
1648
|
+
| `languages_reply` | 7,038 / 6,902 | 8,199 / 8,410 | 9,566 / 9,979 |
|
|
1649
|
+
| `session_start` | 7,484 / 3,706 | 11,606 / 2,860 | 9,054 / 4,352 |
|
|
1650
|
+
| define / use | 63 / 48, 263 / 101 | 49 / 43, 151 / 81 | 46 / 66, 244 / 114 |
|
|
1651
|
+
| interrupt a loop | 35 / 95 | 29 / 68 | 35 / 91 |
|
|
1652
|
+
| infinite output (bytes in 1.5 s, stopped in ms) | 1,077,165 (32) / 60,802 (33) | 328,350 (24) / **29,795** (46) | 248,661 (30) / 33,229 (46) |
|
|
1653
|
+
| all 13 checks done | 19,866 / 16,152 | 26,441 / 16,281 | 24,127 / 19,814 |
|
|
1654
|
+
| checks | 13/13 / 13/13 | 13/13 / **13/13** | 13/13 / 13/13 |
|
|
1655
|
+
|
|
1656
|
+
- **Firefox node passes all 13 for the first time.** Its flood streamed 29,795 bytes against the
|
|
1657
|
+
check's 10,000 (4,545 and 7,676 in the two runs before). The control session came up at 5.2 s, as in
|
|
1658
|
+
the other browsers. The cause of the change was not isolated; this run's host was quieter than run6's
|
|
1659
|
+
(no throttling), and W18 removed the guest's wait for entropy.
|
|
1660
|
+
- **The guest's entropy is seeded at boot in the browser.** Every boot log of run7 and after has
|
|
1661
|
+
`random: crng init done` at `[0.000000]` (13 of 13 logs in `wmn-results/`). Before W18 the same boots
|
|
1662
|
+
logged it at 4.1-19.6 s (`wmn-results/r6`), from jitter entropy over the strictly increasing clock of
|
|
1663
|
+
P-B3 problem 2.
|
|
1664
|
+
- Logs: `wmn-results/main-{chromium,firefox,webkit}/`.
|
|
1665
|
+
|
|
1666
|
+
**wmn suite, run8** (2026-10-06 08:00-08:05, the finished branch: the out-of-memory change, the
|
|
1667
|
+
random_get test; one boot per image and browser; canary 693/637, 924/631, 700/1,012 ms before/after
|
|
1668
|
+
the Chromium/Firefox/WebKit runs; load average 13-23):
|
|
1669
|
+
|
|
1670
|
+
| ms from `bootWmd` | Chromium python / node | Firefox python / node | WebKit python / node |
|
|
1671
|
+
|---|---|---|---|
|
|
1672
|
+
| control session up | 5,217 / 4,953 | 6,220 / 6,091 | 6,709 / 7,336 |
|
|
1673
|
+
| `hello_reply` | 6,668 / 6,623 | 8,099 / 7,654 | 10,186 / 10,961 |
|
|
1674
|
+
| `languages_reply` | 7,864 / 7,621 | 10,210 / 9,969 | 12,717 / 13,606 |
|
|
1675
|
+
| `session_start` | 8,183 / 4,091 | 13,157 / 3,885 | 12,631 / 5,456 |
|
|
1676
|
+
| define / use | 68 / 77, 287 / 86 | 66 / 77, 187 / 109 | 134 / 73, 303 / 202 |
|
|
1677
|
+
| interrupt a loop | 105 / 214 | 210 / 203 | 178 / 107 |
|
|
1678
|
+
| infinite output (bytes in 1.5 s, stopped in ms) | 836,785 (64) / 42,824 (89) | 223,008 (226) / 31,916 (233) | 718,514 (229) / 29,391 (93) |
|
|
1679
|
+
| all 13 checks done | 22,306 / 17,776 | 31,531 / 19,780 | 33,064 / 25,372 |
|
|
1680
|
+
| checks | 13/13 / 13/13 | 13/13 / 13/13 | 13/13 / 13/13 |
|
|
1681
|
+
|
|
1682
|
+
- Firefox node passes all 13 again (31,916 bytes in its flood against the check's 10,000).
|
|
1683
|
+
- `random: crng init done` at `[0.000000]` in all six boot logs.
|
|
1684
|
+
- A first WebKit run (07:25, canary **6.2 s** before it: another session's load) also gave 13/13 and
|
|
1685
|
+
13/13; its times are not comparable and are not in the table.
|
|
1686
|
+
- Logs: `wmn-results/run8-{chromium,firefox,webkit}/` (the contended WebKit run: `run8-webkit-first/`).
|
|
1687
|
+
|
|
1688
|
+
**b11-fixes guest tests on the finished branch** (Chromium, 08:05, canary 881/634 ms): 2/2.
|
|
1689
|
+
- The vm handle after 5 forced GCs: `{vm: true, up: true}`.
|
|
1690
|
+
- node at `slack=256`: exit 4, "out of memory: the emulator used 1664 MiB of its 1664 MiB memory
|
|
1691
|
+
maximum ...", through the size rule (the installed vm.wasm has no heap reserve).
|
|
1692
|
+
|
|
1693
|
+
**b11-fixes on the rebased branch.**
|
|
1694
|
+
- No-guest tests, 3 browsers, 16:12 (canary 673/690 ms): 9/9.
|
|
1695
|
+
- Guest tests, Chromium (canary 692/640 ms): 2/2.
|
|
1696
|
+
- The vm handle after 5 forced GCs: `{vm: true, up: true}`.
|
|
1697
|
+
- node at `slack=256`: exit 4 with `wmn.error` "the machine ended: 4 (out of memory: the emulator
|
|
1698
|
+
used 1664 MiB of its 1664 MiB memory maximum ...)".
|
|
1699
|
+
- The same no-guest tests on the pre-fix code (347f80ed's `boot.js`/`k2.js`): 6 failed, 3 passed. The
|
|
1700
|
+
3 that pass are "a trap far below the maximum keeps its plain reason", which is meant to pass before
|
|
1701
|
+
and after.
|
|
1702
|
+
|
|
1703
|
+
**Regression run, Chromium, the whole suite** (16:16-16:49; canary 873 ms before, 641 after; load
|
|
1704
|
+
average 23 falling to 7): 102 passed, 51 skipped (opt-in suites), 3 failed.
|
|
1705
|
+
|
|
1706
|
+
| failure | cause | re-run |
|
|
1707
|
+
|---|---|---|
|
|
1708
|
+
| `chan.spec` hvc0, 50 MiB in 120 s | the guest's hvc0 path is slow; vm.wasm, not the browser (below) | 4 MiB in each browser: pass |
|
|
1709
|
+
| `jit.spec` P-B2 proof boot "5x faster than P-B1" | load: `PROOF-DONE` at 9,002 ms against the bound of 8,808, on a host whose canary read 873 ms | alone, canary 623 ms: pass (7.3 s test) |
|
|
1710
|
+
| `review-errors` `argv --cpus 64` bounded at 60 s | known, left as is (P-B1 review): 64 vCPUs on 10 cores are still booting at 60 s | - |
|
|
1711
|
+
|
|
1712
|
+
**Regression run on the finished branch, Chromium, the whole suite** (2026-10-06 08:06-08:37; canary
|
|
1713
|
+
638 ms before, 613 after; load average 12 falling to 5): 108 passed, 51 skipped (opt-in suites), 2
|
|
1714
|
+
failed, both known:
|
|
1715
|
+
- `chan.spec` hvc0, 50 MiB in 120 s: "no exit in 120000 ms" (below). The guest's log of that boot
|
|
1716
|
+
(`run8-logs/timeout-console.stdout.log`, `CONSOLE-START` at 4.04 s) has no `nobody cared` and no
|
|
1717
|
+
`Disabling IRQ`: the storm is gone, the console is just slow;
|
|
1718
|
+
- `review-errors` `argv --cpus 64` bounded at 60 s: still booting at 60 s, as before.
|
|
1719
|
+
|
|
1720
|
+
The P-B2 proof boot passed this time (8.0 s; the load failure of the run before did not recur).
|
|
1721
|
+
Logs of every run of 2026-10-06: `wmn-results/run8-logs/`.
|
|
1722
|
+
|
|
1723
|
+
### hvc0 after W17: slow in vm.wasm, not in the browser host
|
|
1724
|
+
|
|
1725
|
+
**Superseded by W53 (D-433, below):** the slowness was the test's 2-byte tty writes. The
|
|
1726
|
+
conclusions below ("in vm.wasm or the guest's hvc driver", the 2 KiB `put_chars` guess) were wrong.
|
|
1727
|
+
|
|
1728
|
+
W17 (2ec1cf9d) stopped vm.wasm raising an interrupt for console-port transmits. The IRQ storm of
|
|
1729
|
+
P-B3 problem 1 is gone: no `nobody cared` and no `Disabling IRQ` in any run. But `yes | head -c N >
|
|
1730
|
+
/dev/hvc0` on a raw tty is still slow, and the 50 MiB test cannot finish in its 120 s.
|
|
1731
|
+
|
|
1732
|
+
The same initramfs (`b3Initrd` + `consoleScript`), the same kernel and the installed vm.wasm, one boot
|
|
1733
|
+
each, 1 vCPU, `--mode jit`:
|
|
1734
|
+
|
|
1735
|
+
| | bytes | time | rate |
|
|
1736
|
+
|---|---|---|---|
|
|
1737
|
+
| browser hvc0, Chromium (canary 623 ms) | 4 MiB | 74.5 s | 56 KB/s |
|
|
1738
|
+
| browser hvc0, Firefox | 4 MiB | 126.6 s | 33 KB/s |
|
|
1739
|
+
| browser hvc0, WebKit | 4 MiB | 58.4 s | 72 KB/s |
|
|
1740
|
+
| **native `wmd-run`** hvc0, a Unix-socket sink (canary 727 ms) | 4 MiB | 61.1 s | **69 KB/s** |
|
|
1741
|
+
| native `wmd-run` control port (vport1p1), the same sink | 8 MiB | 2.65 s | 3.2 MB/s |
|
|
1742
|
+
| browser control port (P-B3 numbers) | 50 MiB | 10.7-15.9 s | 3.3-4.9 MB/s |
|
|
1743
|
+
|
|
1744
|
+
- **Native is as slow as the browser**, so the cost is in vm.wasm or the guest's hvc driver, not in the
|
|
1745
|
+
browser host.
|
|
1746
|
+
- In every browser run the host kept up:
|
|
1747
|
+
- every byte reached the page, in order;
|
|
1748
|
+
- the guest's `chan_send` never found the ring full (`sendWaits` 0);
|
|
1749
|
+
- the service's first-to-last-byte time was the guest's own time to within 0.1-1.1 s (74.42 against
|
|
1750
|
+
74.51 s, 126.52 against 126.60, 57.30 against 58.37).
|
|
1751
|
+
- Likely cause, not proven: Linux's hvc path sends each tty write of at most 2 KiB (`put_chars` ->
|
|
1752
|
+
`__send_to_port`, one device exit each) and then polls. At 69 KB/s that is about 34 sends a second,
|
|
1753
|
+
far fewer than an exit costs. D-391 (W17) already notes that line-by-line hvc output is still slow,
|
|
1754
|
+
at about 22 KB/s.
|
|
1755
|
+
- **This is a vm.wasm/guest finding, for the wmd side (a follow-up of W17).** Nothing in wmd uses
|
|
1756
|
+
port 0: container stdio goes over the control port, which runs at MB/s.
|
|
1757
|
+
|
|
1758
|
+
**W53 (D-433): the cause was the test, not hvc0.** busybox `head` writes through stdio, which is
|
|
1759
|
+
line-buffered on a tty: into `/dev/hvc0` it made one 2-byte write per `y` line, each a full hvc
|
|
1760
|
+
write (about 26-36 us in the guest). Natively, on the same vm.wasm and kernel: `dd` into hvc0 with
|
|
1761
|
+
512 B-64 KiB blocks moved 256 KiB in 0.03-0.07 s; `yes | head -c 1M | cat > /dev/hvc0` 0.33 s
|
|
1762
|
+
(3.2 MB/s); `yes | head -c 256K > /dev/hvc0` 3.42 s (77 KB/s). The control port never showed it
|
|
1763
|
+
because a vport is not a tty. The console script now writes through `cat` for every target, and
|
|
1764
|
+
chan.spec `-g console` passes in all three browsers:
|
|
1765
|
+
|
|
1766
|
+
| 50 MiB | Chromium | Firefox | WebKit |
|
|
1767
|
+
|---|---|---|---|
|
|
1768
|
+
| hvc0 | 16.81 s, 3.1 MB/s | 24.78 s, 2.1 MB/s | 19.39 s, 2.7 MB/s |
|
|
1769
|
+
| control port | 15.71 s, 3.3 MB/s | 20.02 s, 2.6 MB/s | 17.44 s, 3.0 MB/s |
|
|
1770
|
+
| /dev/null (guest alone) | 14.86 s, 3.5 MB/s | 17.64 s, 3.0 MB/s | 17.82 s, 2.9 MB/s |
|
|
1771
|
+
|
|
1772
|
+
A program that writes a tty line by line still pays about 30 us per line.
|
|
1773
|
+
- (Before W53) the test was unchanged and its 50 MiB in 120 s failed; it passes since W53.
|
|
1774
|
+
- Re-run: `WMD_WEB_CONSOLE_BYTES=4194304 WMD_WEB_HVC0_TIMEOUT_MS=300000 bunx playwright test chan.spec -g hvc0`.
|
|
1775
|
+
The native comparison boots `wmd-run vm.wasm -- --machine ... --console 7 --control 8` with
|
|
1776
|
+
`WMD_CHAN_UNIX="7=/tmp/x.sock,8=/tmp/y.sock"`. Use a short socket path: macOS caps `sun_path` at 104
|
|
1777
|
+
bytes, and a longer path makes `wmd-run` report "host not listening".
|
|
1778
|
+
|
|
1779
|
+
### Review of 70d03426 (W18: WASI random_get in `src/host/wasi.js`)
|
|
1780
|
+
|
|
1781
|
+
- **Correct.**
|
|
1782
|
+
- `random_get(ptr, len)` checks the range (EFAULT) against the current memory size, after
|
|
1783
|
+
`views()` has picked up any grow.
|
|
1784
|
+
- It fills at most 64 KiB per `getRandomValues` call, through a private buffer, then copies into the
|
|
1785
|
+
shared memory. Browsers refuse a view of a SharedArrayBuffer, and the call is limited to 64 KiB.
|
|
1786
|
+
- `len` 0 does nothing.
|
|
1787
|
+
- The `worker.js` guard wraps it like every other WASI import, so once the machine has ended it
|
|
1788
|
+
throws instead of running.
|
|
1789
|
+
- The installed vm.wasm imports `random_get`, so a host without it could not instantiate vm.wasm at
|
|
1790
|
+
all.
|
|
1791
|
+
- **Gap, now covered.** The unit test (`wasi.test.js`) runs in bun, and bun's `getRandomValues`
|
|
1792
|
+
accepts views of shared memory. So the unit test cannot show why the copy is needed; before this
|
|
1793
|
+
commit only a guest boot would. New: b11-fixes "WASI random_get fills shared wasm memory in this
|
|
1794
|
+
browser". It passes in all three browsers (2026-10-06 07:08). With the copy removed
|
|
1795
|
+
(getRandomValues on `u8.subarray`; 07:22, canary 760/698 ms) it fails in each, with the browser's
|
|
1796
|
+
own error:
|
|
1797
|
+
- Chromium: "The provided ArrayBufferView value must not be shared";
|
|
1798
|
+
- Firefox: "can't be a SharedArrayBuffer or an ArrayBufferView backed by a SharedArrayBuffer";
|
|
1799
|
+
- WebKit: "must be an instance of ArrayBufferView".
|
|
1800
|
+
- **Effect.** See run7: `crng init done` at 0.000000 in every browser boot.
|
|
1801
|
+
- **Unchanged.** The strictly increasing clock (P-B3 problem 2), now a fallback for a host that gives
|
|
1802
|
+
no seed. The run request's own 32-byte `seed` (`fillRunRequest`), which the native engine sends as
|
|
1803
|
+
well.
|
|
1804
|
+
|
|
1805
|
+
### Out-of-memory detection with W21's heap reserve (branch `w21-trap`, not merged here)
|
|
1806
|
+
|
|
1807
|
+
W21's fix (w21-trap, a64e0c1a + 02a281d3) has vm.wasm grow its memory to the maximum before any thread
|
|
1808
|
+
starts (`WMD_HEAP_RESERVE=1`, which `bootWmd` sets by default there). With it, review finding 4's rule
|
|
1809
|
+
("a trap with the memory within 64 MiB of its maximum is out of memory") would call every trap out of
|
|
1810
|
+
memory. This branch's `finish()` now uses `isOutOfMemory()` (`src/boot.js`):
|
|
1811
|
+
|
|
1812
|
+
- vm.wasm's stderr line `vm.wasm: heap reserve used up (W21): out of memory` means out of memory,
|
|
1813
|
+
whatever the size says;
|
|
1814
|
+
- without a reserve, the old size rule;
|
|
1815
|
+
- with a reserve (below) and no "used up", a plain `trap:` reason.
|
|
1816
|
+
|
|
1817
|
+
`finish()` drains stderr before it decides, so vm.wasm's last line counts. `timings.oom` records
|
|
1818
|
+
which signal it was.
|
|
1819
|
+
|
|
1820
|
+
The reserve counts as in use when either says so:
|
|
1821
|
+
- vm.wasm's environment asks for it (`heapReserveAsked(env)`: `WMD_HEAP_RESERVE` set and not `0`, as
|
|
1822
|
+
`heap_reserve.c` reads it). On w21-trap `bootWmd` adds `WMD_HEAP_RESERVE=1` unless `heapReserve` is
|
|
1823
|
+
false, so after the merge the size rule applies only with `heapReserve: false`, and a reserve line
|
|
1824
|
+
lost on the way cannot turn every trap into out of memory.
|
|
1825
|
+
- vm.wasm printed its reserve line, `vm.wasm: heap reserve N MiB at ...`. That also covers a caller that sets the variable some other way.
|
|
1826
|
+
|
|
1827
|
+
Without either (this branch as it is, or `heapReserve: false`), the size rule of finding 4 is unchanged.
|
|
1828
|
+
One case is not covered: the reserve asked for, but vm.wasm could not grow its memory at all (memory
|
|
1829
|
+
already at its maximum, so `heap_reserve.c` prints nothing and keeps no reserve). Its out-of-memory trap
|
|
1830
|
+
would then read as a plain trap; vm.wasm's own `sbrk` prints nothing in that case either.
|
|
1831
|
+
|
|
1832
|
+
Merging: w21-trap was cut from this branch before the rebase onto wmd main and the review fixes, so a
|
|
1833
|
+
plain merge conflicts in many files. Its two commits (a64e0c1a, 02a281d3) cherry-pick onto this branch
|
|
1834
|
+
with one conflict, the README's last section (both append one); `boot.js` merges cleanly: w21-trap turns
|
|
1835
|
+
`const env = o.env || []` into `vmEnv(o.env, o.heapReserve)`, and `finish()` here reads that `env`.
|
|
1836
|
+
After the merge:
|
|
1837
|
+
- the synthetic out-of-memory test (`oom_trap.wat`) passes `WMD_HEAP_RESERVE=0`, so it still tests the
|
|
1838
|
+
size rule (vmEnv lets the caller's entry win);
|
|
1839
|
+
- a real out-of-memory trap counts only through vm.wasm's "used up" line, so vm.wasm must be the W21
|
|
1840
|
+
build. With the installed vm.wasm (30db07f5, no reserve, prints neither line) and w21-trap's default
|
|
1841
|
+
`heapReserve`, b11-fixes "node out of its memory maximum" would read a plain `trap:`. That is the price
|
|
1842
|
+
of the rule asked for (size check only with `heapReserve: false`); the opposite mistake, every trap
|
|
1843
|
+
called out of memory, is what it prevents.
|
|
1844
|
+
|
|
1845
|
+
Tests (each new one fails on the code before it):
|
|
1846
|
+
- `test/unit/oom.test.js`: 5 tests (`isOutOfMemory`, `heapReserveAsked`, the two regexes against
|
|
1847
|
+
`heap_reserve.c`'s format strings).
|
|
1848
|
+
- b11-fixes, four synthetic guests that print vm.wasm-style lines on stderr and trap (`review-helpers.js`
|
|
1849
|
+
`startBoot` now passes `env`):
|
|
1850
|
+
|
|
1851
|
+
| case | ee1886e8 `boot.js` | first version (keyed on the line only) | now |
|
|
1852
|
+
|---|---|---|---|
|
|
1853
|
+
| reserve line, memory at the maximum, no "used up" | `out of memory: ...` (fail) | `trap: unreachable` | `trap: unreachable` |
|
|
1854
|
+
| `WMD_HEAP_RESERVE=1`, no line, memory at the maximum | `out of memory: ...` (fail) | `out of memory: ...` (fail) | `trap: unreachable` |
|
|
1855
|
+
| `WMD_HEAP_RESERVE=0`, memory at the maximum | out of memory, no `timings.oom` (fail) | out of memory, "memory at its maximum" | the same |
|
|
1856
|
+
| "used up" line, memory 384 of 640 MiB | `trap: unreachable` (fail) | out of memory, "heap reserve used up" | the same |
|
|
1857
|
+
|
|
1858
|
+
Runs (2026-10-06, under the boot lock, canary 625-634 ms, load average 15-23): ee1886e8's `boot.js` 12
|
|
1859
|
+
of 12 failed (3 browsers); the first version 3 failed (the second case, each browser); now all of
|
|
1860
|
+
b11-fixes' no-guest tests pass, 24 of 24 in the three browsers (07:08).
|
|
1861
|
+
|
|
1862
|
+
### Merged into wmd main (d832d730, W3.1-W3.3), 2026-10-06
|
|
1863
|
+
|
|
1864
|
+
`git merge --no-ff wmd-web-b11` (f45d6352) into main d832d730 had no textual conflicts: main's side
|
|
1865
|
+
(W3.1-W3.3) touches only `guest/vm`, `internal/engine`, `runner` and `docs/research`, and this branch
|
|
1866
|
+
only `web/`. Everything below ran on the merged tree in a scratch worktree, with what the tests serve
|
|
1867
|
+
built from that tree, never into `~/.wmd/bin`:
|
|
1868
|
+
- `vm.wasm`: `make guest-vm` (sha256 f3a7caee..., against the installed 30db07f5...);
|
|
1869
|
+
- the Go init's initramfs: `docker buildx build --target container-rootfs-arm64 --build-context
|
|
1870
|
+
c2w=guest/container2wasm -f guest/linux/Dockerfile --output type=tar,dest=- guest/linux`, the archive's
|
|
1871
|
+
`container-initramfs.cpio.gz`. It is byte-identical to the installed one (0bafbaf9...): neither side
|
|
1872
|
+
changed `guest/container2wasm` or `guest/linux`. The kernel (`Image`) was copied from the installed guest;
|
|
1873
|
+
- the packs: `fixtures-gen image` for python:3.12-slim and node:22-slim, identical to the branch's;
|
|
1874
|
+
- served with `WMD_VM_WASM` and `WMD_GUEST_DIR` pointing at those files, `PORT=8798`.
|
|
1875
|
+
|
|
1876
|
+
**A conflict the merge did not show: W3.3's vm.wasm cannot start in the browser.** Each boot ended at
|
|
1877
|
+
once with exit 4. Nothing in `web/` (on the branch or on main) knew about W3.3's two vm.wasm changes:
|
|
1878
|
+
1. `trap: WebAssembly.Instance(): table import 1 is smaller than initial 158, got 155`. vm.wasm's table
|
|
1879
|
+
import minimum went from 155 to 158, and `jit.js` had it as the constant `TABLE_MIN = 155`. The
|
|
1880
|
+
runner reads it from the module (`host.rs` `table_min`).
|
|
1881
|
+
2. With 158: `Import #13 "wmd" "point": function import requires a callable`. W3.3 added the imports
|
|
1882
|
+
`wmd.ctl_attach` and `wmd.point`.
|
|
1883
|
+
|
|
1884
|
+
Fixed in this merge:
|
|
1885
|
+
- `tableMinOf(bytes)` (`src/host/jit.js`) reads the table import minimum from vm.wasm's import section,
|
|
1886
|
+
since the JS API does not expose import limits. `boot.js` passes it to every Worker, which sizes its
|
|
1887
|
+
table with it, and the JIT door starts its slots after it (`createJitDoor({tableMin})`). `TABLE_MIN`
|
|
1888
|
+
(155) is now only the default, for modules with no table import and for the unit tests. The installed
|
|
1889
|
+
vm.wasm (155) and W3.3's (158) both boot.
|
|
1890
|
+
- `wmd-imports.js`: `ctl_attach` returns -1 (no control socket, so the guest starts no ctl thread) and
|
|
1891
|
+
`point` returns -38 (no checkpoint; the guest goes on). These are the weak defaults in
|
|
1892
|
+
`guest/vm/src/dev/chan.c`.
|
|
1893
|
+
- `test/unit/vm-imports.test.js` (6 tests, which fail before the fix): `tableMinOf` on hand-made
|
|
1894
|
+
modules, and on the vm.wasm being served (`WMD_VM_WASM`), the table minimum plus "the host provides
|
|
1895
|
+
every `wmd.*` function it imports".
|
|
1896
|
+
|
|
1897
|
+
Runs on the merged tree with the fix (all under the boot lock, one at a time; canary 610-632 ms
|
|
1898
|
+
before and after each, unless noted):
|
|
1899
|
+
|
|
1900
|
+
| suite | result |
|
|
1901
|
+
|---|---|
|
|
1902
|
+
| `bun test test/unit test/adversarial-b11/yamux-adv.test.js` | 314 pass, 2 skip, 0 fail (308 before `vm-imports.test.js`) |
|
|
1903
|
+
| b11-fixes, no guest, Chromium + Firefox + WebKit | 24/24 (6 guest tests skipped) |
|
|
1904
|
+
| b11-fixes, guest boots, Chromium (`WMD_WEB_B11FIX=1`) | 2/2: `{vm: true, up: true}` after 5 GCs; node at `slack=256`: exit 4, "out of memory: the emulator used 1664 MiB of its 1664 MiB memory maximum ..." |
|
|
1905
|
+
| wmn.spec python / node, Chromium | 13/13 / 13/13 |
|
|
1906
|
+
| wmn.spec python / node, Firefox | 13/13 / 13/13 |
|
|
1907
|
+
| wmn.spec python / node, WebKit | 13/13 / 13/13 |
|
|
1908
|
+
| adversarial-b11 `rob.spec` (`ROB_IMAGES=python,node`), Chromium | python 68 PASS / 4 FAIL, node 66 / 5: the same as before the merge (below) |
|
|
1909
|
+
| native `go test -short ./...` | all pass except `test/compat` TestBuildKitCopyLink and TestBuildKitTarOutputOwnership, which fail on main too |
|
|
1910
|
+
| native `make test` (`go test ./...`, under the lock; canary 995 ms before, 633 after) | all pass except `internal/buildkit` TestBoltCommitsDoNotFlushTheDriveCache, a timing test (commit 8.9 ms against half of 12.1 ms; it passed 3 of 3 times when run alone afterwards), and `test/compat`, which hit `go test`'s default 10 min timeout in TestBuildFailingRunOutput (`make test-compat` gives it 180 min). Neither package is touched by this merge (no Go file outside `web/` changes) |
|
|
1911
|
+
|
|
1912
|
+
Without the fix, the same b11-fixes guest tests and all six wmn boots failed with error 1.
|
|
1913
|
+
|
|
1914
|
+
wmn timings, in ms from `bootWmd` (one run each, load average 5-9):
|
|
1915
|
+
|
|
1916
|
+
| | Chromium python / node | Firefox python / node | WebKit python / node |
|
|
1917
|
+
|---|---|---|---|
|
|
1918
|
+
| control session up | 4,134 / 4,119 | 4,067 / 4,111 | 4,145 / 4,209 |
|
|
1919
|
+
| `hello_reply` | 5,389 / 5,398 | 5,119 / 5,168 | 6,479 / 6,847 |
|
|
1920
|
+
| `session_start` | 6,045 / 3,305 | 7,200 / 2,194 | 6,561 / 3,269 |
|
|
1921
|
+
| interrupt a loop | 28 / 57 | 22 / 35 | 25 / 46 |
|
|
1922
|
+
| infinite output (bytes in 1.5 s) | 1,185,942 / 96,354 | 293,102 / 74,538 | 369,559 / 59,691 |
|
|
1923
|
+
| all 13 checks done | 17,252 / 14,491 | 18,957 / 12,953 | 19,193 / 17,061 |
|
|
1924
|
+
|
|
1925
|
+
**adversarial-b11 on the merged tree: no change from the merge.** The same python run with the
|
|
1926
|
+
installed (pre-W3) vm.wasm and guest gave the same 68 PASS / 4 FAIL, with the same trap (canary 1,320 ms
|
|
1927
|
+
before, 669 after; another session was running). The merged-tree node run ended at canary 1,118 ms. The
|
|
1928
|
+
failures:
|
|
1929
|
+
- c03 "next cell unaffected" (both images): known; late output, wmn's limit (ROADMAP 4.3b).
|
|
1930
|
+
- b04 "65 MiB line refused" and b05 "20 MiB of stdin into an exec" (both images): the review's harness
|
|
1931
|
+
writes 65 MiB and 20 MiB with one synchronous `proc.write()`. Since the finding-7 fix, `write()` past
|
|
1932
|
+
the stream's credit (4 MiB + 64 KiB) returns false and sends nothing, so the agent never sees the line
|
|
1933
|
+
and `wc -c` reads 0. This is the fix's contract, and the review's tests were left unchanged.
|
|
1934
|
+
`writeAsync()` is the call for big data.
|
|
1935
|
+
- b08 "20 MB execute_result" (node): the result came back with 10,030 characters. As in the review's own
|
|
1936
|
+
node run, this is not the transport.
|
|
1937
|
+
- b05 "30 execs at once" (both images): the machine traps, review finding 1. The trap here is
|
|
1938
|
+
`RangeError: Start offset -2147302880 is outside the bounds of the buffer` at `jit.js` `readGuest` <-
|
|
1939
|
+
the door's `compile` <- `vm_wasm_jit_compile` <- `compile_worker`, and it happens with both vm.wasm
|
|
1940
|
+
builds. **New finding, not fixed here:** the JIT door's `load`/`compile` use the guest pointer as a
|
|
1941
|
+
signed i32 (no `>>> 0`, unlike `wasi.js` and `chan.js`). Once vm.wasm's heap passes 2 GiB (the wmn
|
|
1942
|
+
page's 1 GiB slack allows a 2.4 GiB maximum), the first JIT module there kills the machine. This may
|
|
1943
|
+
be part of finding 1 (W21).
|
|
1944
|
+
|
|
1945
|
+
Not run here, as asked: the hvc0 50 MiB `chan.spec` (W22) and the `--cpus 64` boot (over 60 s) are
|
|
1946
|
+
known failures; W3's checkpoint guest tests are W3's.
|
|
1947
|
+
|
|
1948
|
+
### W23: guest pointers above 2 GiB, and a review of the merge's fix (branch `wmd-web-w23`)
|
|
1949
|
+
|
|
1950
|
+
**W23.** vm.wasm passes guest pointers as wasm i32s; a JS import receives them signed. `jit.js`
|
|
1951
|
+
`readGuest` (`jit_load`, `jit_compile`) used them as is, so once vm.wasm's heap passed 2 GiB (the
|
|
1952
|
+
wmn page's 1 GiB slack gives a 2.4 GiB maximum) the first block translated there trapped the
|
|
1953
|
+
machine (the adversarial-b11 b05 trap above). `readGuest` now reads `ptr` and `len` as unsigned
|
|
1954
|
+
and a range outside the memory makes the call return -1 with a log line, as the runner's
|
|
1955
|
+
`load_slots` does, instead of trapping. The rest of `src/` was audited: `wasi.js` and `chan.js`
|
|
1956
|
+
already converted their pointers; two WASI lengths did not (`path_open`'s path length,
|
|
1957
|
+
`fd_prestat_dir_name`'s buffer length) and now do. `thread-spawn`'s start argument goes back to
|
|
1958
|
+
wasm as the same i32; the hub, the rings, stdio and the 9P server read no guest addresses. Range
|
|
1959
|
+
checks are `ptr + len <= byteLength` in doubles, so the 4 GiB end of a 65536-page memory is exact.
|
|
1960
|
+
|
|
1961
|
+
Tests (each fails before the fix):
|
|
1962
|
+
- `test/unit/w23-pointers.test.js` (9): the door above 2 GiB and ending at 4 GiB, refusals past the
|
|
1963
|
+
end, WASI and channel calls with signed high addresses. 6 fail on 6361975f.
|
|
1964
|
+
- `test/e2e/w23-pointers.spec.js`: a synthetic guest booted through `bootWmd` grows to 4 GiB,
|
|
1965
|
+
`jit_load`s a module at 0x8002c220 (the b05 trap's offset), `jit_compile`s one ending at 4 GiB,
|
|
1966
|
+
calls both through its table and `fd_write`s its result from above 2 GiB. Before: exit 4 in
|
|
1967
|
+
`readGuest`: Chromium "Start offset -2147302880 is outside the bounds of the buffer", Firefox
|
|
1968
|
+
"invalid or out-of-range index", WebKit "byteOffset cannot be negative". After: exit 0,
|
|
1969
|
+
"W23 OK", in all three.
|
|
1970
|
+
|
|
1971
|
+
**Review of `tableMinOf`, the `ctl_attach`/`point` stubs (6361975f).**
|
|
1972
|
+
- `tableMinOf` was right for the vm.wasm shapes built today (155 and 158, one table import),
|
|
1973
|
+
custom sections, non-minimal LEB128, memory64/table64, a table maximum, tags and no table import.
|
|
1974
|
+
It was wrong for: two table imports (it took the first; now `env.__indirect_function_table`,
|
|
1975
|
+
else the last, as `host.rs`), typed references (0x63/0x64 + a heap type) in table and global
|
|
1976
|
+
imports, a custom page size (limits flag 8), a section shorter than its imports, an endless
|
|
1977
|
+
LEB128. Fixed; 16 new cases in `vm-imports.test.js`, 7 failed before.
|
|
1978
|
+
- JIT slots: the runner puts slot k at `table_min + k` (`jit.rs`), and so does the door
|
|
1979
|
+
(`tableMin + slotBase`). The guest uses only the index the door returns. Nothing is hard-coded.
|
|
1980
|
+
- `ctl_attach` -1: `machine_start_ctl` then starts no ctl thread and returns quietly. The only
|
|
1981
|
+
path to `wmd.point` is a request from that thread, so `point` is never called in the browser;
|
|
1982
|
+
-38 would only print "checkpoint not taken" once. The init's redial loop (W3.2) runs only after
|
|
1983
|
+
a session that set `ckpt`, which the page never sends, and it backs off from 50 ms to 1 s.
|
|
1984
|
+
No busy loop, no flood of log lines, no boot delay.
|
|
1985
|
+
|
|
1986
|
+
**With w21-trap.** w21-trap changes neither `jit.js` nor `wasi.js`, and this branch does not touch
|
|
1987
|
+
`boot.js`. W21's reserve grows the memory to its maximum before any thread starts, so heap
|
|
1988
|
+
addresses above 2 GiB come into use sooner. Without this fix, W21's 30-exec storm would hit W23's
|
|
1989
|
+
trap more readily.
|
|
1990
|
+
|
|
1991
|
+
Runs on this branch. Everything is under the boot lock, `PORT=8795`, with vm.wasm built in the
|
|
1992
|
+
worktree (f3a7caee, `make guest-vm`). The disk floor was lowered to 50 GiB by the user, and
|
|
1993
|
+
`locked.sh` now checks 50.
|
|
1994
|
+
|
|
1995
|
+
| suite | result |
|
|
1996
|
+
|---|---|
|
|
1997
|
+
| `bun test test/unit test/adversarial-b11/yamux-adv.test.js` | 339 pass, 2 skip, 0 fail |
|
|
1998
|
+
| b11-fixes (no guest) + `w23-pointers.spec`, Chromium / Firefox / WebKit | 9 passed, 2 skipped, in each |
|
|
1999
|
+
| wmn.spec python / node, Chromium (canary 660 ms before) | 13/13 / 13/13 |
|
|
2000
|
+
| wmn.spec python / node, Firefox (canary 1,450 ms before: a loaded host, load average 17) | 13/13 / 13/13 |
|
|
2001
|
+
| wmn.spec python / node, WebKit (canary 917 ms before, 1,159 after) | 13/13 / 13/13 |
|
|
2002
|
+
| adversarial-b11 b11, 30 execs at once (`ROB_IMAGES=python ROB_QUERY='&cases=b11&conc=30'`), Chromium (canary 826 / 733 ms) | FAIL, but not W23. The machine traps with `memory access out of bounds` at `calloc` <- `interp_run_block` (thread 6), with the memory at 1,562 MiB of its 2,432 MiB maximum. This is review finding 1 / W21 (Chromium's stale memory size after another Worker's grow), not merged here. The heap never reached 2 GiB in this run, so it did not reach W23's path. No `readGuest` or "outside the guest memory" line appears. |
|
|
2003
|
+
|
|
2004
|
+
The 30-exec run cannot show W23 until W21 is in: W21's trap comes first, below 2 GiB. W23 is shown
|
|
2005
|
+
by `w23-pointers.spec.js` (before and after, three browsers) and the unit tests.
|
|
2006
|
+
|
|
2007
|
+
## W21: 30 concurrent execs trapped the machine in Chromium
|
|
2008
|
+
|
|
2009
|
+
The bug (wmn ROADMAP, "Work in wmd"): 30 `exec`s of `/bin/echo` at once over the control session on
|
|
2010
|
+
python:3.12-slim (`test/adversarial-b11`, `cases=b11&conc=30`) ended the machine with exit 4,
|
|
2011
|
+
`RuntimeError: memory access out of bounds` at `calloc` <- `interp_run_block` <- `vm_run_blocks`, in
|
|
2012
|
+
5 of 6 Chromium boots. Firefox and WebKit passed, and so did native `wmd-run`. The memory was far
|
|
2013
|
+
below its maximum, so it was not an out-of-memory.
|
|
2014
|
+
|
|
2015
|
+
### Cause
|
|
2016
|
+
|
|
2017
|
+
vm.wasm's own heap (dlmalloc, through wasi-libc's `sbrk`) grows memory 0 with `memory.grow` while
|
|
2018
|
+
the vCPU threads run. Each thread is a Worker with its own instance of the module. In Chromium, a
|
|
2019
|
+
`memory.fill` or `memory.copy` in one Worker can be checked against a memory size that is older than
|
|
2020
|
+
another Worker's `memory.grow`. A plain load or store sees the new pages at once. When vCPU A grows
|
|
2021
|
+
the heap and dlmalloc hands the new top chunk to vCPU B under its lock, B's `calloc` clears the chunk
|
|
2022
|
+
with `memory.fill` (wasi-libc is built with bulk memory) and traps. 30 execs at once make both vCPUs
|
|
2023
|
+
translate blocks and allocate fast. The heap then grows one page at a time, about every 0.5 ms, so
|
|
2024
|
+
the window comes up again and again.
|
|
2025
|
+
|
|
2026
|
+
Evidence:
|
|
2027
|
+
|
|
2028
|
+
- **The engine probe** (`test/e2e/w21-probe.spec.js`, `page/w21-probe.html`, no guest). Two Workers
|
|
2029
|
+
share one memory. Worker A grows it by one page and sets a flag; worker B spins on the flag, then
|
|
2030
|
+
touches the new page. 300 trials of each:
|
|
2031
|
+
|
|
2032
|
+
| B's access to the new page | Chromium 153 (PW 1.63) | Firefox | WebKit |
|
|
2033
|
+
|---|---|---|---|
|
|
2034
|
+
| `memory.fill` | **3/300 trap** | 0 | 0 |
|
|
2035
|
+
| `memory.copy` | **4/300 trap** | 0 | 0 |
|
|
2036
|
+
| `memory.fill` after `memory.atomic.wait` (not a spin) | 0 | 0 | 0 |
|
|
2037
|
+
| `i32.store8` (plain store) | 0 | 0 | 0 |
|
|
2038
|
+
| `memory.fill` / `memory.copy`, memory grown before the Workers start | 0 / 0 | 0 / 0 | 0 / 0 |
|
|
2039
|
+
|
|
2040
|
+
The trap message reports B's `memory.size` as already including the page. So the stale size is
|
|
2041
|
+
only the one that `memory.fill` and `memory.copy` are checked against. A wait refreshes it.
|
|
2042
|
+
- **In the guest** (a per-grow log in a diagnostic build, kept on branch `w21-trap-wip-backup`).
|
|
2043
|
+
Chromium, without the reserve: thread 6 (vCPU 1, `vcpu_thread_fn`) trapped in `calloc` at
|
|
2044
|
+
20.027630 s (CLOCK_MONOTONIC), 36 us after vCPU 0's `memory.grow` 24185 -> 24186 pages at
|
|
2045
|
+
20.027594 s. vCPU 0 had grown the heap 25 times in the 18 ms before, one page at a time. The
|
|
2046
|
+
trapping thread's last own grow was at 20.0095 s. The memory was 24,186 pages (1.5 GiB) of
|
|
2047
|
+
38,912.
|
|
2048
|
+
|
|
2049
|
+
### Fix: the heap reserve (`WMD_HEAP_RESERVE=1`)
|
|
2050
|
+
|
|
2051
|
+
`guest/vm/src/heap_reserve.c` replaces wasi-libc's `sbrk`. When `WMD_HEAP_RESERVE` is set, vm.wasm
|
|
2052
|
+
grows memory 0 to its maximum in `run_machine`, after guest RAM and the vCPU arena and before
|
|
2053
|
+
`machine_run` starts any thread. It grows in the largest steps the host accepts. `sbrk` then hands
|
|
2054
|
+
out those pages from an atomic bump pointer. Memory 0 never grows while more than one thread runs,
|
|
2055
|
+
so no Worker can hold a stale size. Without the variable, `sbrk` behaves exactly like wasi-libc's,
|
|
2056
|
+
so native `wmd-run` is unchanged. When the reserve is used up, vm.wasm says
|
|
2057
|
+
`vm.wasm: heap reserve used up (W21): out of memory` once and stops (see the review follow-ups below).
|
|
2058
|
+
`bootWmd` sets the variable unless `heapReserve: false` is passed or the caller's env already has a
|
|
2059
|
+
`WMD_HEAP_RESERVE` entry (`vmEnv`, `src/boot.js`). An older vm.wasm ignores it.
|
|
2060
|
+
|
|
2061
|
+
Cost: the reserve is address space, not RAM. Untouched pages are not resident. Peak RSS of one
|
|
2062
|
+
browser process in a 1-round run with the reserve was 1,210 MiB. The baseline runs, which trapped
|
|
2063
|
+
part-way, peaked at 1,034 and 1,721 MiB. That is not a clean A/B, but there is no extra 1 GiB.
|
|
2064
|
+
|
|
2065
|
+
### Adversarial review follow-ups (`w21-trap-adv`, 22a71aa9), 2026-10-06
|
|
2066
|
+
|
|
2067
|
+
The branch is rebased onto wmd main 6361975f (P-B11 merged). The review's tests are in
|
|
2068
|
+
`test/adversarial-w21/` (its `RESULTS.md`, `storm.spec.js`, `usermode.spec.js`, `node-repro.mjs` for a V8
|
|
2069
|
+
bug report). `fixq.sh` is this round's run queue, and `out/f*.log` and `out/*.json` are its logs.
|
|
2070
|
+
|
|
2071
|
+
- **A4, done.** vm.wasm takes the reserve only for `WMD_HEAP_RESERVE=1` exactly. Before, `=no` turned it
|
|
2072
|
+
on; `heapReserveAsked` in `src/boot.js` now matches. The native runner no longer passes
|
|
2073
|
+
`WMD_HEAP_RESERVE` to the guest (`runner/src/wasi.rs` `guest_env_from`, cargo unit test). So the
|
|
2074
|
+
native snapshot cost the review measured cannot happen from a shell variable.
|
|
2075
|
+
- **A2, done: the user-mode traps were not W21.** User mode now takes the reserve in `main()` before any
|
|
2076
|
+
thread. rthreads still trapped 4 of 20 times with the reserve in place, and every trap came *after* the
|
|
2077
|
+
program's output. `main()` freed the block cache and guest RAM while the guest's detached threads were
|
|
2078
|
+
still translating (`emit_block_code` on freed memory). It now drains them first
|
|
2079
|
+
(`linux_user_wait_vcpus`, as `runprog.c` does). A thread trap also never ends in exit 0 any more:
|
|
2080
|
+
`worker.js` counts traps in `CTL_TRAPS`, and `finish()` turns an exit 0 into 4 if a thread trapped.
|
|
2081
|
+
- **A3, done (this overlaps W20).** When the reserve is used up, vm.wasm no longer hands NULL to
|
|
2082
|
+
dlmalloc's callers:
|
|
2083
|
+
- `sbrk` calls `vm_out_of_memory`. That prints the exact `vm.wasm: heap reserve used up (W21): out of
|
|
2084
|
+
memory` line, then `vm.wasm: out of memory: <what> of <n> bytes; ...`, then calls `_Exit(4)`.
|
|
2085
|
+
- `interp_run_block`'s calloc and `wasm_enc` call it too. Without a reserve the detail line still
|
|
2086
|
+
comes out. Natively it aborts after the message, as the unit tests expect.
|
|
2087
|
+
- `boot.js` reports that exit as out of memory (`VM_OOM_RE`) and stops every Worker.
|
|
2088
|
+
- Before the heap runs out, the reserve's *low mark* asks the block cache for one reset. The first
|
|
2089
|
+
request comes with a quarter of the reserve left, then one at each halving (`heap_rsv_take`,
|
|
2090
|
+
`heap_reserve_reset_wanted`, unit test `test_low_mark`).
|
|
2091
|
+
- With the reserve, each reset logs `vm.wasm: block cache reset: ...`.
|
|
2092
|
+
- **Why the IR cap was not sized from the reserve.** `ir_bytes` overcounts the heap: 805 MB of IR
|
|
2093
|
+
"bytes" with 546 MiB of a 1023 MiB reserve still free. A cap at 1/2 or 3/4 of the reserve reset the
|
|
2094
|
+
cache in every 30-exec storm and made it 2–3x slower. At 1/2 the heap passed 2 GiB after the reset
|
|
2095
|
+
and hit W23. With the cap at the reserve's full size, the slack-64 run reset again and again until the
|
|
2096
|
+
JIT door's module log filled (1 GiB), and the guest stalled instead of stopping. The low mark
|
|
2097
|
+
measures the heap itself, so D-254's cap is left alone.
|
|
2098
|
+
- **A1 is W23** (`wmd-web-w23`, not fixed here). With its host fix applied, a storm that crossed 2 GiB
|
|
2099
|
+
passed. **A6** (`9p-diff.test.js` dirent-type) fails on main too.
|
|
2100
|
+
|
|
2101
|
+
Runs on the final build (Apple Silicon, `PORT=8797`, every step under the boot lock with a 50 GiB disk
|
|
2102
|
+
floor; canary 609–723 ms around every step):
|
|
2103
|
+
|
|
2104
|
+
| Run | Result |
|
|
2105
|
+
|---|---|
|
|
2106
|
+
| `w21.spec.js` 30-exec storm, Chromium x5 | 5/5 pass, 0 bad, 12.1–15.2 s, 0 cache resets, no grow |
|
|
2107
|
+
| same, Firefox / WebKit | pass / pass (14.7 s / 13.9 s) |
|
|
2108
|
+
| python c45 / c60 / c100, `--ram 512` (Chromium) | 0 bad each; 0 / 2 / 6 cache resets. Of two c100 runs, one passed and one hit "used up" in round 0 and stopped cleanly with the message |
|
|
2109
|
+
| slack 64, c30 mixed, reserve (Chromium) | **stops cleanly**: 4 resets, then `heap reserve used up`, the detail line, exit 4, reported as out of memory. No trap, no garbage IR |
|
|
2110
|
+
| slack 64, no reserve | stops cleanly: `vm.wasm: out of memory: interp_run_block calloc ...`, out of memory (before: abort, or garbage IR) |
|
|
2111
|
+
| rthreads user mode x20 (Chromium) | 0/20 traps (was 8/20) |
|
|
2112
|
+
| `w21-fixes.spec.js` + `b11-fixes.spec.js` | 13/13 in each of C/F/W |
|
|
2113
|
+
| engine probe `w21-probe.spec.js` | C/F/W pass |
|
|
2114
|
+
| web unit tests | 288 pass, 3 fail (all `9p-diff.test.js`, A6) |
|
|
2115
|
+
| native (`runner/target/release/wmd-run`, this worktree) | `smoke-wasm.sh` 13/13 PASS. arm64 `wmd.proof` boot (jit, 4 vCPUs) `PROOF-DONE` with `WMD_HEAP_RESERVE=0` and `=1` (no reserve line: the runner drops the variable). rthreads x20, with and without the variable: 0 bad. `make test` in `guest/vm`: all binaries pass |
|
|
2116
|
+
|
|
2117
|
+
### Joint W21/W23 review (`w21-w23-review` a597f429), fixes 2026-10-06
|
|
2118
|
+
|
|
2119
|
+
- **R1 (major, native): fixed.** A2's drain in user-mode `main()` waits for the guest's threads, but
|
|
2120
|
+
`clone` (`guest/vm/test/linux_user.c`) kept starting threads after `group_exiting` was set. So for a
|
|
2121
|
+
guest whose threads keep starting threads (`progs/spawnloop.c`), the wait did not settle: 5 of 15
|
|
2122
|
+
native runs took 6.7–8.1 s and printed `vm.wasm: guest threads still running at exit`.
|
|
2123
|
+
- `clone` now returns `-EAGAIN` once the group is exiting. It adds to `live_threads`, then reads
|
|
2124
|
+
`group_exiting`; the drain does the reverse; both seq_cst. So each new thread is either counted by
|
|
2125
|
+
the drain or refused.
|
|
2126
|
+
- A thread that registers after the drain walked the thread list does not enter `vm_run`.
|
|
2127
|
+
- Tests and logs: `test/adversarial-w21/native-spawnloop.sh` and `native-a2.sh`, from the review,
|
|
2128
|
+
with the guest programs in `progs/`.
|
|
2129
|
+
- Native, no variable: spawnloop **0/15** bad, 0.61–1.16 s, no new line.
|
|
2130
|
+
- Against main's vm.wasm, stderr is identical for blockread, blockwrite, pipewait, spawnloop,
|
|
2131
|
+
futexwait, exit3, threxit, rthreads and threads, with the same exit codes and times.
|
|
2132
|
+
- Main itself trapped (exit 4) on spawnloop in **2 of 10** runs: the use-after-free that A2's drain
|
|
2133
|
+
removes. This branch had 0 in 15.
|
|
2134
|
+
- **(4) minor: fixed.** `boot.js` now treats vm.wasm's out-of-memory lines as out of memory only with
|
|
2135
|
+
exit status 4, the status `vm_out_of_memory` uses. Before, a user-mode guest that printed the lines
|
|
2136
|
+
and exited 1 (`oomspoof`) was reported as out of memory. A guest that prints the lines *and* exits
|
|
2137
|
+
4 cannot be told apart in user mode; in machine mode the guest cannot write vm.wasm's stderr.
|
|
2138
|
+
`w21-fixes.spec.js` has the exit-1 case.
|
|
2139
|
+
- **Stale node test: fixed.** With the reserve, node passes 13/13 at `slack=256`, and also at
|
|
2140
|
+
`slack=64` (C/F/W), so `b11-fixes.spec.js` now boots node at `slack=8`. It ends with exit 4, the used-up
|
|
2141
|
+
line and `wmn.error` "out of memory ... 1416 MiB maximum" (run with `WMD_WEB_B11FIX=1` and
|
|
2142
|
+
`WMD_WEB_IMAGE_FIXTURES` pointing at a tree that has the node packs).
|
|
2143
|
+
- **(3) late trap:** left as is, with a comment in `finish()`. Traps are counted until the 50 ms
|
|
2144
|
+
settle.
|
|
2145
|
+
|
|
2146
|
+
### Runs (2026-10-05, Apple Silicon, `PORT=8797`, worktree vm.wasm via `WMD_VM_WASM`, each under the boot lock)
|
|
2147
|
+
|
|
2148
|
+
`test/e2e/w21.spec.js` boots python:3.12-slim (`page/wmn.html?checks=0&nonet=1&hold=1`) and runs
|
|
2149
|
+
3 rounds of 30 control execs at once, then 10 in sequence. It checks every exit code and output, no
|
|
2150
|
+
trap, the reserve line, and that the memory did not grow while the vCPUs ran.
|
|
2151
|
+
|
|
2152
|
+
| Run | Result | Storm time | Canary before / after |
|
|
2153
|
+
|---|---|---|---|
|
|
2154
|
+
| Chromium, reserve, 5 boots | 5/5 pass, 0 bad | 13.3-13.9 s | 631 ms after |
|
|
2155
|
+
| Firefox, reserve | pass | 17.4 s | 673 ms |
|
|
2156
|
+
| WebKit, reserve | pass | 18.5 s | 673 ms |
|
|
2157
|
+
| Chromium, no reserve (`WMD_WEB_W21_BASELINE=1`), 2 boots | **2/2 trap** (77 and 100 execs failed); memory grew 22532 -> 30532 and 22532 -> 23631 pages under the vCPUs | | 657-660 ms |
|
|
2158
|
+
| Earlier session: Chromium with the reserve via `test/adversarial-b11` `cases=b11&conc=30` (build before the used-up message) | 5/5 pass; Firefox and WebKit 1/1 each | 23-28 s per boot | |
|
|
2159
|
+
| Earlier session: Chromium `cases=b11&conc=30` without the reserve (`WMD_HEAP_RESERVE=0`, or the vm.wasm built before it) | 3/3 trap | | |
|
|
2160
|
+
|
|
2161
|
+
Also run: `bun test test/unit` (270 pass, 2 skip, 0 fail). Native, with the installed `~/.wmd/bin/wmd-run`
|
|
2162
|
+
(main 9833319c): `guest/vm/test/smoke-wasm.sh` passed 13/13, and an arm64 `wmd.proof` machine boot (jit, 4 vCPUs)
|
|
2163
|
+
matched all of `test/expected/boot-arm64.txt` both with `WMD_HEAP_RESERVE=0` and with `=1` (canary 624 / 639 ms). `guest/vm`
|
|
2164
|
+
`build/test/test_heap_reserve`. That test covers the grow loop against a fake memory with a smaller
|
|
2165
|
+
maximum and a step limit, the bump (exact fit, refusals that leave the break alone, a reserve ending
|
|
2166
|
+
at 4 GiB), and 8 threads taking disjoint ranges that cover the reserve exactly.
|
|
2167
|
+
|
|
2168
|
+
### Running
|
|
2169
|
+
|
|
2170
|
+
```sh
|
|
2171
|
+
cd web
|
|
2172
|
+
# no guest: the engine probe in each browser
|
|
2173
|
+
PORT=8797 bunx playwright test w21-probe
|
|
2174
|
+
# under the boot lock, one browser per run (about 25 s each)
|
|
2175
|
+
PORT=8797 WMD_VM_WASM=../guest/vm/build/vm.wasm bunx playwright test w21.spec --project=chromium --repeat-each=5
|
|
2176
|
+
PORT=8797 WMD_VM_WASM=../guest/vm/build/vm.wasm WMD_WEB_W21_BASELINE=1 bunx playwright test w21.spec --project=chromium # A/B: must trap
|
|
2177
|
+
```
|
|
2178
|
+
|
|
2179
|
+
## W21 + W23 merged into wmd main (2026-10-06)
|
|
2180
|
+
|
|
2181
|
+
Branch `w21-w23-review` = main 6361975f + `--no-ff` wmd-web-w23 (9f7b4866) + w21-trap (822f3aaf, then
|
|
2182
|
+
15945505 with the joint review's fixes) + the joint review's tests (`test/adversarial-w21w23/`, its
|
|
2183
|
+
`STATUS.md`). Conflicts: this README (W23 kept under P-B11, W21 after it) and
|
|
2184
|
+
`adversarial-b11/locked.sh` (w21's wait-and-retry floor of 50 GiB).
|
|
2185
|
+
|
|
2186
|
+
The joint review (STATUS.md) found R1 (major, native: the exit drain never settled while guest threads
|
|
2187
|
+
kept spawning), R2 (a guest's own stderr taken for vm.wasm's out-of-memory lines) and R3 (the node OOM
|
|
2188
|
+
test went stale once W21 stopped node running out at slack 256). All three were fixed on w21-trap
|
|
2189
|
+
15945505 and rechecked here. W23 (no signed guest pointer or length left in `src/`; 300k-module
|
|
2190
|
+
`tableMinOf` fuzz in bun and node) had no findings.
|
|
2191
|
+
|
|
2192
|
+
Final runs on the combined tree (vm.wasm dfdb0d24 and wmd-run built in the worktree, port 8794, every
|
|
2193
|
+
step under the boot lock, P-core canary 606–748 ms around every step; logs in
|
|
2194
|
+
`test/adversarial-w21w23/out/`, round 1 in `out/r1/`):
|
|
2195
|
+
|
|
2196
|
+
| Run | Result |
|
|
2197
|
+
|---|---|
|
|
2198
|
+
| native spawnloop, no variable (`native-spawnloop.sh 15`) | 0/15 bad (was 5/15 slow with the new stderr line) |
|
|
2199
|
+
| native A2 programs vs main 6361975f, no variable (`native-a2.sh`) | blockread, blockwrite, pipewait, spawnloop, futexwait, exit3, threxit, rthreads, threads: same exit codes and times, stderr identical |
|
|
2200
|
+
| `a2.spec.js` (A2 programs in user mode, oomspoof, 5 arm64 proof boots at 4 vCPUs) | 10/10 in each of Chromium, Firefox, WebKit; no trap after an exit |
|
|
2201
|
+
| `bun test test/unit` | 316 pass, 2 skip, 0 fail (`9p-diff` passes) |
|
|
2202
|
+
| b11-fixes + w21-fixes + w23-pointers, no guest | 15/15 in each of C/F/W |
|
|
2203
|
+
| b11-fixes guest tests (`WMD_WEB_B11FIX=1`, node packs) | 10/10 C, 9/9 F and W (node at slack 8: exit 4, out of memory) |
|
|
2204
|
+
| wmn.spec python + node | 13/13 and 13/13 in each of C/F/W |
|
|
2205
|
+
| w21.spec 30-exec storm | Chromium 5/5 (12.3–13.1 s, 0 bad, no grow, 0 resets); Firefox 16.0 s and WebKit 12.9 s pass |
|
|
2206
|
+
| storm c60, `--ram 1024`, Chromium x3 (the W23 case) | 180/180 each, no grow |
|
|
2207
|
+
| storm c100, `--ram 1024`, Chromium x3 | 1 of 3 clean (300 execs). 2 of 3 used up the 1023 MiB reserve after 3 cache resets: the used-up line, `sbrk of 54460416 bytes; 37289984 ... left`, exit 4, reported as out of memory. No trap, no `readGuest` error (W23 is gone). Heap headroom at 100 execs at once is a limit, not a crash (W20). |
|
|
2208
|
+
| b05 (adversarial-b11, python) | the 30 execs pass (they trapped before W21); the 2 remaining FAILs are the same on main 6361975f |
|
|
2209
|
+
| slack 64, 30 mixed execs, Chromium (round 1) | clean stop: 4 cache resets, used-up line, exit 4, out of memory |
|
|
2210
|
+
| rthreads user mode x20, Chromium (round 1) | 0/20 traps |
|
|
2211
|
+
| native `smoke-wasm.sh` (`WMD_REQUIRE_WASM=1`) | 13/13 PASS |
|
|
2212
|
+
| native arm64 `wmd.proof` (jit, 4 vCPUs) | PROOF-DONE with the variable unset, `=0`, `=1` (no reserve line: the runner drops it) |
|
|
2213
|
+
| `make test` in guest/vm | pass |
|
|
2214
|
+
| `go test -short ./...` (round 1; no Go change since) | only the 2 pre-existing TestBuildKit failures |
|
|
2215
|
+
|
|
2216
|
+
Not fixed, by design or minor: a trap later than the 50 ms settle after an exit 0 is not counted (none
|
|
2217
|
+
seen in 15 proof boots); with the reserve every failed allocation stops the machine (exit 4) instead of
|
|
2218
|
+
cache.c's NULL fallbacks, which never saved it before.
|
|
2219
|
+
|
|
2220
|
+
## P-B10: the guest's network through a relay
|
|
2221
|
+
|
|
2222
|
+
The proof (wmn PROOFS.md P-B10): outbound TCP from the browser guest through a WebSocket relay;
|
|
2223
|
+
`pip install requests` and `apk add curl` work in the browser. Adversarial: the relay drops
|
|
2224
|
+
mid-download, DNS fails, the guest tries to reach the relay's own host or private ranges (must be
|
|
2225
|
+
refused). Branch `wmd-web-b10`.
|
|
2226
|
+
|
|
2227
|
+
### What runs
|
|
2228
|
+
|
|
2229
|
+
```
|
|
2230
|
+
guest virtio-net ──net channel (rings)──▶ channel service ──MessagePort──▶ network Worker ──WebSocket──▶ relay ──TCP──▶ the internet
|
|
2231
|
+
(vm.wasm net.c) 4-byte BE length (NetSession) stack.js + relay-client.js relay.ts
|
|
2232
|
+
+ Ethernet frame
|
|
2233
|
+
```
|
|
2234
|
+
|
|
2235
|
+
- **The device.** vm.wasm's virtio-net already sends each Ethernet frame on a net channel with a
|
|
2236
|
+
4-byte big-endian length, as natively (`--net PORT`, where gvisor-tap-vsock's netstack takes them,
|
|
2237
|
+
`internal/engine/netstack.go`). The page passes `--net 9 --mac 02:00:00:00:00:01` and a service
|
|
2238
|
+
`{port: 9, kind: 'net', relay: {url, tokenUrl}}`; the kernel command line no longer blacklists
|
|
2239
|
+
virtio-net, so the init runs udhcpc as natively.
|
|
2240
|
+
- **The channel service** (`NetSession`, `src/host/chan-services.js`) moves the bytes between the
|
|
2241
|
+
rings and the network Worker: the guest's bytes go out as they are read (at most 1 MiB the Worker
|
|
2242
|
+
has not processed), the Worker's frames are queued for the guest's ring and the Worker is told how
|
|
2243
|
+
much the ring took (`wrote`), which bounds what it sends (1 MiB). vm.wasm waits rather than drops
|
|
2244
|
+
when the guest has no receive buffer (`chan_pump.c`), so the link is reliable both ways.
|
|
2245
|
+
- **The network Worker** (`src/net/net-worker.js`) runs the stack and the relay connection, off the
|
|
2246
|
+
page and off the channel service. It answers, on the engine's addresses (192.168.127.0/24,
|
|
2247
|
+
gateway .1, guest .2, the netstack's gateway MAC):
|
|
2248
|
+
- ARP for the gateway only; DHCP with one static lease (router and DNS server: the gateway);
|
|
2249
|
+
- DNS on gateway:53: A queries go to the relay, which resolves and filters them; AAAA and other
|
|
2250
|
+
types get an empty answer at once (the guest has no IPv6 route); `gateway.wmd.internal`;
|
|
2251
|
+
- TCP: each connection the guest opens is a relay stream. The SYN-ACK goes back only once the
|
|
2252
|
+
relay has connected, so a refused connect is a RST for the SYN (ECONNREFUSED, curl exit 7);
|
|
2253
|
+
- ICMP echo to the gateway; other ICMP and UDP get "unreachable" (only TCP and DNS leave);
|
|
2254
|
+
- the gateway, other guest-network addresses and loopback/multicast never reach the relay.
|
|
2255
|
+
- **TCP** (`src/net/stack.js`) is the passive end of each connection, without congestion control
|
|
2256
|
+
(the link is reliable): it sends what the guest's window and the ring's room allow, takes back
|
|
2257
|
+
credit from the guest's ACKs, advertises a window of what the relay has not taken (256 KiB, window
|
|
2258
|
+
scale 3), and has retransmission on a timeout and three duplicate ACKs, zero-window probes, and an
|
|
2259
|
+
answer to the guest's own probes. A batch of the guest's segments goes to the relay in messages of
|
|
2260
|
+
up to 64 KiB (problem 2). **A stream ends with a FIN only when the remote closed cleanly; a
|
|
2261
|
+
stream that fails (the remote reset, the relay connection dropped, a relay limit) ends with a
|
|
2262
|
+
RST**, so a program never sees a cut-off body as a complete one.
|
|
2263
|
+
- **The relay** (`relay/relay.ts`, Bun, about 860 lines with its policy and tokens; the browser side, `src/net/`, about 1,600) carries one
|
|
2264
|
+
WebSocket per guest (protocol in `src/net/proto.js`: CONNECT, DATA, END, ABORT, CREDIT and DNS,
|
|
2265
|
+
with per-stream credit both ways, 256 KiB windows).
|
|
2266
|
+
|
|
2267
|
+
### Security on the relay
|
|
2268
|
+
|
|
2269
|
+
- **Addresses, after resolution** (`relay/policy.ts`). Every connect is checked on the address the
|
|
2270
|
+
relay actually dials. A name is resolved by the relay once and the checked address is dialled (no
|
|
2271
|
+
second lookup), so DNS rebinding cannot swap it. Refused: 0/8, 10/8, 100.64/10, 127/8,
|
|
2272
|
+
169.254/16 (metadata 169.254.169.254 named as such), 172.16/12, 192.0.0/24, 192.0.2/24,
|
|
2273
|
+
192.88.99/24, 192.168/16, 198.18/15, 198.51.100/24, 203.0.113/24, 224/4, 240/4,
|
|
2274
|
+
100.100.100.200; IPv6 ::, ::1, ::/96, 64:ff9b:1::/48, 100::/64, 2001::/23 (Teredo),
|
|
2275
|
+
2001:db8::/32, 3fff::/20, 5f00::/16, fc00::/7 (fd00:ec2::254), fe80::/10, fec0::/10, ff00::/8;
|
|
2276
|
+
::ffff:0:0:0/96 (IPv4-translated, SIIT); IPv4 inside IPv6 (mapped, NAT64 64:ff9b::/96, 6to4,
|
|
2277
|
+
ISATAP interface ids) is checked as IPv4; and **the relay host's
|
|
2278
|
+
own addresses** (every interface, refreshed every 30 s, plus `WMD_RELAY_SELF` for a public
|
|
2279
|
+
address behind NAT). Only strict dotted-decimal IPv4 is an address: `127.1`, `0x7f.0.0.1`,
|
|
2280
|
+
`2130706433` and `localhost` are refused as names before any lookup.
|
|
2281
|
+
- **DNS answers** leave refused addresses out; a name that resolves only to refused addresses
|
|
2282
|
+
gets REFUSED (curl exit 6).
|
|
2283
|
+
- **Tokens**: HMAC-SHA256 over `{sid, exp, uid, grp}` with the relay's secret, parsed strictly (one
|
|
2284
|
+
spelling verifies), presented as a WebSocket subprotocol (`t.<token>`), not in the URL. `sid` is the
|
|
2285
|
+
token (one live connection per sid), `uid` the guest identity (one guest machine of one page), `grp`
|
|
2286
|
+
the client the issuer saw (its address, or a user). An expired, forged or missing token is refused
|
|
2287
|
+
(401); `WMD_RELAY_ORIGINS` restricts page origins.
|
|
2288
|
+
- **Token issuance** (`relay/token.ts` `TokenIssuer`, used by `test/serve.ts` `/relay-token?instance=ID`;
|
|
2289
|
+
wmn's server would use it with its user id as the client): per client at most 60 tokens a minute
|
|
2290
|
+
and 20 page instances an hour, each token at most 1 h. The network Worker asks for a fresh token at
|
|
2291
|
+
each reconnect, with the same instance id.
|
|
2292
|
+
- **Caps per guest identity** (uid), kept across reconnects and fresh tokens for as long as a token
|
|
2293
|
+
for it is valid: 64 connections at once, 600 connects, 1,200 DNS lookups and 30 WebSocket opens a
|
|
2294
|
+
minute, 16 MiB/s down and 4 MiB/s up (token buckets with a 1 s burst, covering every byte, a new
|
|
2295
|
+
stream's first window too), 4 GiB in all (after that the identity is refused new sessions); one
|
|
2296
|
+
live session per identity (a newer one replaces it, code 4010); 8 live sessions per client (grp),
|
|
2297
|
+
100 in all; port 25 refused; data past the granted credit ends the stream. All are `WMD_RELAY_*`
|
|
2298
|
+
settings (head of `relay/relay.ts`). The state is in memory: a relay restart resets it.
|
|
2299
|
+
- **An allowlist** (`WMD_RELAY_ALLOW_HOSTS`, `name[:port|port]`, ports 80 and 443 when none are
|
|
2300
|
+
given): the relay then resolves only those names and connects only to (address, port) pairs it gave
|
|
2301
|
+
out for them to this guest. The tests use it, so the relay connects only to dl-cdn.alpinelinux.org,
|
|
2302
|
+
pypi.org and files.pythonhosted.org.
|
|
2303
|
+
- **The Worker's stack never wedges** on abandoned connections: a handshake the guest does not finish
|
|
2304
|
+
is given up after 60 s (as Linux's `tcp_synack_retries`), at most 32 handshakes are pending at once,
|
|
2305
|
+
and a connection the guest closed is ended (RST, relay ABORT) after 120 s with no word from the
|
|
2306
|
+
remote.
|
|
2307
|
+
- **Test routes** (`WMD_RELAY_TEST_ROUTES`, `WMD_RELAY_TEST_HOSTS`) let the tests reach a local
|
|
2308
|
+
server as `bulk.wmd.test` (203.0.113.10:80 → 127.0.0.1:4572). The relay prints a warning when
|
|
2309
|
+
they are set; a public relay must never set them.
|
|
2310
|
+
|
|
2311
|
+
### Numbers (2026-10-07, Apple Silicon, Playwright 1.63, `--mode jit`, 2 vCPUs, 1 GiB; relay and bulk server on this host)
|
|
2312
|
+
|
|
2313
|
+
ms and MB/s; one run each. Chromium and Firefox: the final run (09:37-09:48, canary 623 ms before,
|
|
2314
|
+
load average 8). WebKit: its last clean runs (run11, 08:56-09:00, canary 630 / 613 ms, the final stack code,
|
|
2315
|
+
for the transfers; run8, 08:17-08:25, before problem 1's fix, which changes only uploads, for the
|
|
2316
|
+
rest). A dash: no clean WebKit number. The WebKit pass of the final run (09:48-09:53) overlapped another session's guest and is
|
|
2317
|
+
not used for numbers (it passed 5/5). "Uncapped" is with the relay's bandwidth caps raised to
|
|
2318
|
+
1 GiB/s; the others run with the default caps (16 MiB/s down, 4 MiB/s up, each with a 1 s burst).
|
|
2319
|
+
|
|
2320
|
+
| | Chromium | Firefox | WebKit |
|
|
2321
|
+
|---|---|---|---|
|
|
2322
|
+
| udhcpc (the lease from the network Worker; 15 s with no answer before) | 465 | 427 | 419 |
|
|
2323
|
+
| container answers an exec (from `bootWmd`) | 5,290 | 5,256 | 6,155 |
|
|
2324
|
+
| `apk add curl` (index + 9 packages from dl-cdn.alpinelinux.org) | 9,930 | 13,410 | 10,205 |
|
|
2325
|
+
| `curl` APKINDEX over HTTPS (474 KB): DNS / TCP connect / TLS done / total | 40 / 57 / 1,220 / 1,454 | 43 / 63 / 1,195 / 1,542 | – |
|
|
2326
|
+
| `pip install requests` (5 wheels from pypi.org / files.pythonhosted.org) | 57,949 | 90,395 | 52,774 |
|
|
2327
|
+
| `requests.get('https://pypi.org/simple/requests/')` in the guest (s) | 1.94 | 1.20 | – |
|
|
2328
|
+
| connect latency, guest `curl` `time_connect` to the local server (DNS + TCP via the relay), 5 runs | 36-40 | 41-50 | 37-40 (run11) |
|
|
2329
|
+
| relay-side connect (the relay to the local server) | 1.4-17.6 | 1.8-17.0 | 1.8-15.6 |
|
|
2330
|
+
| download 32 MiB, default cap (MB/s; sha256 equal) | 31.0 | 30.6 | 30.5 |
|
|
2331
|
+
| download 128 MiB, uncapped | **90.0** (1.49 s) | **45.6** (2.94 s) | **81.4** (1.65 s) |
|
|
2332
|
+
| upload 8 MiB, default cap (server's sha256 equal) | 8.3 | 8.4 | 8.5 |
|
|
2333
|
+
| upload 32 MiB, uncapped | **47.3** (0.71 s) | **13.4** (2.50 s) | **38.9** (0.86 s) |
|
|
2334
|
+
| download at a 512 KiB/s relay cap (4 MiB, python) | 506 KB/s | 515 KB/s | 517 KB/s |
|
|
2335
|
+
| TCP retransmissions / duplicate-ACK retransmissions (whole alpine run) | 0 / 0 | 0 / 0 | 0 / 0 |
|
|
2336
|
+
|
|
2337
|
+
- **Connect latency.** The guest's `time_connect` (about 40 ms) is mostly the guest itself: the DNS
|
|
2338
|
+
query for `bulk.wmd.test` and the TCP handshake through the emulated kernel. The relay reaches the
|
|
2339
|
+
local server in 1-2 ms once warm (the first connects of a session, 13-18 ms, include its startup).
|
|
2340
|
+
- **Throughput.** The default caps bind on the 32 MiB transfers (16 MiB/s plus a 16 MiB burst gives
|
|
2341
|
+
32 MiB in about 1 s). Uncapped, the path runs at 45-90 MB/s down and 13-47 MB/s up; Firefox is the
|
|
2342
|
+
slowest both ways. The guest's own work (virtio, its TCP, curl writing the file) is in these numbers.
|
|
2343
|
+
- **Package installs** are dominated by the guest (unpacking, `pip` itself under emulation), not the
|
|
2344
|
+
network: the bytes are under 3 MB for each.
|
|
2345
|
+
|
|
2346
|
+
### Results
|
|
2347
|
+
|
|
2348
|
+
Every run under the boot lock, `PORT=8790`, relay on 4571, bulk server on 4572.
|
|
2349
|
+
|
|
2350
|
+
| | Chromium | Firefox | WebKit |
|
|
2351
|
+
|---|---|---|---|
|
|
2352
|
+
| pass: alpine `apk add curl`, then curl over HTTPS, throughput | pass | pass | pass |
|
|
2353
|
+
| pass: python `pip install requests`, then `requests.get` over HTTPS | pass | pass | pass |
|
|
2354
|
+
| adversarial: refused addresses, DNS failures | pass | pass | pass |
|
|
2355
|
+
| adversarial: relay drop / kill / restart / upstream reset mid-download, apk across a relay kill | pass | pass | pass |
|
|
2356
|
+
| caps: connections at once, bandwidth | pass | pass | pass |
|
|
2357
|
+
| `bun test test/unit test/adversarial-b11/yamux-adv.test.js` | 386 pass, 2 skip, 0 fail | | |
|
|
2358
|
+
|
|
2359
|
+
**Refused, from the guest** (`curl -m 20 http://ADDR:PORT/`): 169.254.169.254 (metadata),
|
|
2360
|
+
100.100.100.200 (metadata), 10.0.0.1 and 172.17.0.1:2375 (private), 203.0.113.10:81 (reserved; the
|
|
2361
|
+
test route covers only port 80), and this host's own addresses on the relay's port (its LAN address
|
|
2362
|
+
192.168.0.9 and its Tailscale address 100.119.72.125: "self"). Each failed at once, in 330-500 ms with
|
|
2363
|
+
curl exit 7, and the relay's log names the reason (`blocked`, "... is metadata"). 0.0.0.0 and
|
|
2364
|
+
224.0.0.1 never left the guest. Names that resolve to refused addresses (`127.0.0.1.nip.io`,
|
|
2365
|
+
`169.254.169.254.nip.io`, public DNS) got REFUSED from the relay: curl exit 6, about 430 ms. The
|
|
2366
|
+
gateway's port 80 is refused by the Worker; another guest-network address fails after ARP (3.5 s,
|
|
2367
|
+
EHOSTUNREACH).
|
|
2368
|
+
|
|
2369
|
+
**DNS failures.** A real NXDOMAIN (`no-such-host.invalid`): curl exit 6 in under 0.6 s. The relay
|
|
2370
|
+
restarted with a dead resolver (`WMD_RELAY_DNS=127.0.0.1:4579`): SERVFAIL, curl exit 6 in 0.4 s; with
|
|
2371
|
+
the relay restarted normally the next download worked. The guest kept running throughout.
|
|
2372
|
+
|
|
2373
|
+
**Mid-download failures** (a 4 MiB body sent at 256 KiB/s; each case cut at about 5 s):
|
|
2374
|
+
|
|
2375
|
+
| | Chromium | Firefox | WebKit |
|
|
2376
|
+
|---|---|---|---|
|
|
2377
|
+
| the page drops the relay WebSocket | curl 56, 1.25 MB kept, 319 ms after | 56, 331 ms | 56, 1,298 ms |
|
|
2378
|
+
| relay killed (SIGKILL) and started again | 56, 270 ms | 56, 337 ms | 56, 1,358 ms |
|
|
2379
|
+
| relay restarted (SIGTERM) | 56, 287 ms | 56, 384 ms | 56, 1,349 ms |
|
|
2380
|
+
| the remote resets mid-body | 56 | 56 | 56 |
|
|
2381
|
+
| the next download (`curl \| sha256sum`) | equal | equal | equal |
|
|
2382
|
+
| `apk add python3` with the relay (256 KiB/s) killed during the package downloads | apk exit 1: "python3-pycache-pyc0: IO ERROR", "1 error"; the rerun installed that one package, `python3 -c 'print(6*7)'` printed 42 | same | same |
|
|
2383
|
+
|
|
2384
|
+
Never a curl exit 0 with other bytes: a failed stream is a RST to the guest, never a FIN. In WebKit
|
|
2385
|
+
(run8) a failure took about 1.3 s to reach the guest's stream, against about 0.3 s in the other two;
|
|
2386
|
+
in the overlapped final run the relay kills took 0.27-0.38 s there too. The cause was not checked.
|
|
2387
|
+
|
|
2388
|
+
### Problems found, and their causes
|
|
2389
|
+
|
|
2390
|
+
1. **Uploads stalled in Firefox and WebKit** (fixed). A 32 MiB upload ran at 26.8 MB/s in Chromium
|
|
2391
|
+
but at 0.59 MB/s in Firefox and 1.05 MB/s in WebKit, with every byte arriving (sha256 equal).
|
|
2392
|
+
Cause: each guest segment (1,460 bytes) became its own WebSocket message, and the relay answered
|
|
2393
|
+
each socket write with its own CREDIT: about 23,000 messages each way for 32 MiB, against a few
|
|
2394
|
+
hundred 64 KiB messages for the same download. Sending a WebSocket message from a Worker costs
|
|
2395
|
+
far more per message in Firefox and WebKit. Fix: the stack sends a batch of segments to the
|
|
2396
|
+
relay in messages of up to 64 KiB, and the relay sends one CREDIT per stream per turn. After it:
|
|
2397
|
+
Firefox 12.6 MB/s, WebKit 38.9 MB/s, Chromium 48 MB/s (test: `net-stack.test.js` "a batch of
|
|
2398
|
+
the guest's segments goes to the relay as few large messages"). Two causes ruled out on the
|
|
2399
|
+
way: `Atomics.waitAsync` woken by the same Worker's notify takes 11-15 ms in all three browsers
|
|
2400
|
+
(a probe, not kept), and answering the guest's zero-window probes (a real gap, now fixed and
|
|
2401
|
+
tested, but not this stall).
|
|
2402
|
+
2. **Bun's node:net and a remote reset.** In the relay's own tests, a bulk server written with
|
|
2403
|
+
`drain` events sometimes never sent its reset; the relay was not at fault (pausing and resuming
|
|
2404
|
+
a socket in Bun 1.4.2 does report ECONNRESET). The test server writes with write callbacks.
|
|
2405
|
+
3. **Bandwidth caps trickled one byte at a time.** With the bucket empty, `take()` returned the
|
|
2406
|
+
few bytes refilled since the last call, so the relay sent DATA messages of 1-10 bytes. It now
|
|
2407
|
+
waits until 16 KiB (or the whole request) is there.
|
|
2408
|
+
|
|
2409
|
+
### What remains
|
|
2410
|
+
|
|
2411
|
+
- **IPv6.** The guest gets no IPv6 route, and the relay resolves A records only.
|
|
2412
|
+
- **UDP** other than DNS (NTP, QUIC, DNS over TCP from the guest) is not carried.
|
|
2413
|
+
- **Inbound connections** (published ports) are not possible in the browser.
|
|
2414
|
+
- **Native comparison.** The relay path was not measured against native wmd's netstack here.
|
|
2415
|
+
- **Deployment** (ROADMAP 14.8): a hosted relay needs TLS (`wss:`), a token service in wmn's
|
|
2416
|
+
server, and its public address in `WMD_RELAY_SELF`.
|
|
2417
|
+
|
|
2418
|
+
### Running
|
|
2419
|
+
|
|
2420
|
+
```sh
|
|
2421
|
+
cd web
|
|
2422
|
+
bun test test/unit/net-stack.test.js test/unit/net-chan.test.js test/unit/net-relay.test.js # no guest
|
|
2423
|
+
# under the boot lock, one browser per run (about 5-6 min each); needs the alpine and python fixtures:
|
|
2424
|
+
# go run ./web/test/fixtures-gen image -ref docker.io/library/alpine:3.20 -out web/test/fixtures/images/alpine
|
|
2425
|
+
# go run ./web/test/fixtures-gen image -ref docker.io/library/python:3.12-slim -out web/test/fixtures/images/python
|
|
2426
|
+
export WMD_RELAY_SECRET=$(openssl rand -hex 16) WMD_RELAY_URL=ws://127.0.0.1:4571/relay PORT=8790
|
|
2427
|
+
test/locked.sh bunx playwright test net.spec --project=chromium # WMD_WEB_NET_CASES=pass,adversarial,restart,caps
|
|
2428
|
+
# a relay of your own:
|
|
2429
|
+
WMD_RELAY_SECRET=... bun relay/relay.ts
|
|
2430
|
+
```
|
|
2431
|
+
|
|
2432
|
+
The spec starts the relay (`bun relay/relay.ts`, port 4571) and a bulk HTTP server (port 4572)
|
|
2433
|
+
itself, and kills them by PID. Results: `net-results/<browser>-<case>.{json,stdout.log,stderr.log}`.
|
|
2434
|
+
|
|
2435
|
+
## P-B8: two vCPUs on Workers sharing memory (branch `wmd-web-b8`, in progress)
|
|
2436
|
+
|
|
2437
|
+
Status (2026-10-07): **partly done; no passing proof run yet.**
|
|
2438
|
+
- Done: the stall hook (`bootWmd({stallHook})`, `vm.stallThread(tid, ms)`, `vm.stats().stall`; cells 9-13
|
|
2439
|
+
and bytes 112-127 in `layout.js`; checked in `worker.js`'s import guard only when on); case `b8` in
|
|
2440
|
+
`page/chan.js` (alpine:3.20 + build-base from one pack, chroot from the proof initramfs, `test/b8/run.sh`
|
|
2441
|
+
phases); fixtures `test/b8/build.sh` (Docker: build-base pack, static `b8lock` LL/SC and LSE builds,
|
|
2442
|
+
Lua 5.4.7); `test/b8/b8lock.c` (mutex, spin, atomic, cond, futex, work); specs `b8.spec.js` (opt-in
|
|
2443
|
+
`WMD_WEB_B8=1`) and `b8-hook.spec.js` (synthetic `b8-synth/stall.wat`); native reference
|
|
2444
|
+
`test/b8/native.sh`; `test/b8/pw.sh` (waits for the lock with no "playwright test" in its command line).
|
|
2445
|
+
- Evidence so far: one Chromium boot, 1 vCPU, with 1024 MiB of slack (canary 612/624 ms). nproc, getconf
|
|
2446
|
+
and cpuinfo all reported 1, and cpuinfo lists `atomics` (LSE). The first `make` of Lua then **used up
|
|
2447
|
+
vm.wasm's 1023 MiB heap reserve** (4 block-cache resets, then exit 4, reported as out of memory). gcc's
|
|
2448
|
+
cc1 needs more than 1 GiB of translator heap. The spec now defaults to 2560 MiB of slack
|
|
2449
|
+
(`B8_SLACK`); that run never got the lock.
|
|
2450
|
+
- Not started: the 1/2/4-vCPU runs in Chromium/Firefox/WebKit, the speed-ups, the lock contention
|
|
2451
|
+
numbers, the stall runs (RCU stall at >21 s: `CONFIG_RCU_CPU_STALL_TIMEOUT=21`; the kernel has no
|
|
2452
|
+
soft-lockup detector), the native reference, `b8-hook.spec.js` runs, and the unit tests under the lock.
|
|
2453
|
+
- Note for the stall work: `machine_cache_flush_all` (world-stop) waits for every running vCPU to park, so a
|
|
2454
|
+
stopped vCPU Worker would stall any other vCPU that flushes the whole cache.
|
|
2455
|
+
- Run: `PORT=8774 WMD_VM_WASM=../guest/vm/build/vm.wasm WMD_WEB_B8=1 test/locked.sh test/b8/pw.sh b8.spec --project=chromium`.
|
|
2456
|
+
|
|
2457
|
+
## P-B7: packs from a static host and the registry proxy
|
|
2458
|
+
|
|
2459
|
+
**Status: done and reviewed; results, root causes and limits are in `test/b7/README.md`.** Branch `wmd-web-b7`. (The text below is the design record; where it says "not done", the later runs in `test/b7/README.md` did it.)
|
|
2460
|
+
|
|
2461
|
+
Built: `src/registry/ref.js` (refs, platforms, the layout: `/blobs/sha256/<hex>` packs, immutable;
|
|
2462
|
+
`/idx/<registry>/<repo>/<tag>/<os>-<arch>.json` per image@platform), `page/sw.js` (packs cached only
|
|
2463
|
+
after a sha256 check; everything else network first, cache when offline), `page/b7.js` (index -> packs
|
|
2464
|
+
-> container run over the control channel), `registry/` (Bun host + proxy `server.ts`; Docker Hub client
|
|
2465
|
+
with anonymous tokens, host allowlist, streamed digest/size checks `upstream.ts`; `pull.ts`: index ->
|
|
2466
|
+
linux/arm64, 4 parallel verified downloads, layers converted in order by wmd's `union.PackLayer`
|
|
2467
|
+
(`wmdpack/`), chain cache, gzip-6 copies served with Content-Encoding, disk floor; `export.ts` from
|
|
2468
|
+
`~/.wmd/data` (read only); `testreg.ts` + `testimages.ts`: a Docker-free registry with 50-layer,
|
|
2469
|
+
multi-arch, nested, lying, corrupt, slow, cut and size-lie images).
|
|
2470
|
+
|
|
2471
|
+
Decision: convert on the proxy, not in the browser. Packs are byte-identical to P-B11's fixtures-gen
|
|
2472
|
+
packs (same converter); gzip-6 packs are within 2% of the registry's layer.gz on the wire (48.5 vs
|
|
2473
|
+
47.6 MB, alpine + python); conversion runs once per layer chain for every visitor. In-browser gunzip +
|
|
2474
|
+
tar walk of python's layers (148.7 MB tar) is 275 / 297 / 231 ms (C/F/W), vs 717 ms for the Go
|
|
2475
|
+
converter, so a browser converter is affordable in CPU, but would be a second implementation of
|
|
2476
|
+
fromtar.go's whiteout, opaque, hard-link copy-up and symlinked-parent rules.
|
|
2477
|
+
|
|
2478
|
+
Evidence so far:
|
|
2479
|
+
- `bun test test/unit/b7-registry.test.ts`: 17 pass (no guest): refs/platforms, allowlist and foreign
|
|
2480
|
+
token realm/redirect refused, 50-layer union checked with wmd's Go union, multi-arch/nested/no-arm64/
|
|
2481
|
+
lying index, corrupt layer refused with no index or chain or tmp left, size lie, cut transfer then
|
|
2482
|
+
retry, disk floor, concurrent pulls joined, static headers.
|
|
2483
|
+
- No-guest browser tests pass in Chromium, Firefox and WebKit: the SW caches a pack and a second visit
|
|
2484
|
+
with the host killed gets it from the cache (`probe`); a tampered pack is refused by the SW and not
|
|
2485
|
+
cached; corrupt layer refused by the proxy (`do not match`); the gunzip probe.
|
|
2486
|
+
- Proxy against Docker Hub (curl, no browser): busybox 1.1 s (1.9 MB), ruby:3.3-slim 2.56 s (69.8 MB,
|
|
2487
|
+
5 layers; download 1.63 s, convert 0.99 s; packs 176 MB, gz 72.7 MB). Export from wmd's store:
|
|
2488
|
+
python 0.85 s (convert 717 ms, 207 MB/s).
|
|
2489
|
+
- Guest boots (Chromium, one run, canary 617/619 ms): alpine and python ran from the static host
|
|
2490
|
+
through the SW (alpine main exit at 5.2 s, python at 7.6 s from navigation; exit 0, outputs right);
|
|
2491
|
+
the run stopped at a test regex (apk warnings on stderr), fixed in the spec, not rerun.
|
|
2492
|
+
|
|
2493
|
+
Not done: the offline second-visit boot, the Docker Hub images in the browser, the 50-layer,
|
|
2494
|
+
multi-arch and proxy-killed-mid-pull browser tests (all written in `test/b7/b7.spec.js`), Firefox and
|
|
2495
|
+
WebKit guest runs, the numbers table. The boot lock was held by other sessions most of the session
|
|
2496
|
+
(goprof.sh waits for no `pgrep -f "playwright test"` while holding the lock, which a waiting
|
|
2497
|
+
`locked.sh bunx playwright test ...` matches; this suite queues through `test/b7/run.sh`).
|
|
2498
|
+
|
|
2499
|
+
```sh
|
|
2500
|
+
go build -o web/registry/.bin/wmdpack ./web/registry/wmdpack
|
|
2501
|
+
cd web && bun test test/unit/b7-registry.test.ts
|
|
2502
|
+
test/locked.sh test/b7/run.sh --project=chromium # then firefox, webkit (ports 8780-8783)
|
|
2503
|
+
```
|
|
2504
|
+
|
|
2505
|
+
## W24: start-up events for wmn's engine start-up screen
|
|
2506
|
+
|
|
2507
|
+
`bootWmd({onEvent})` also gets the start-up events wmn's start-up screen draws (wmn D-211, ROADMAP 7.14), from
|
|
2508
|
+
`src/host/boot-events.js`. `atMs` is milliseconds from the boot's `t0` (page clock, `timings.t0`), `tookMs` the stage's own
|
|
2509
|
+
duration, sizes are bytes. Progress-like events (download and restore progress, workers progress, `heap`, `layer`) go out at
|
|
2510
|
+
most 4 times a second per kind (a download's kind is its file, a layer's its port): the latest waits, and a `start` or `done`
|
|
2511
|
+
of the same kind replaces it. Nothing is sent after `exit`. The events that were there before (`chan`, `pool`, `panic`,
|
|
2512
|
+
`throttle`, `thread`, `phase`, `host-calls`) are unchanged.
|
|
2513
|
+
|
|
2514
|
+
| event | when | fields |
|
|
2515
|
+
|---|---|---|
|
|
2516
|
+
| `{t:'boot', stage:'download', state:'progress'\|'done', file}` | per file, as bytes come; `done` when it is in | `bytes`, `total` (Content-Length; null when absent or content-encoded), `cached` (null on progress), `port` (packs), `tookMs` (the fetch's own time) on done; a file the caller passed as bytes: `given: true`, `cached: null`; a file vm.restore() reused: `cached: true` |
|
|
2517
|
+
| `{t:'boot', stage:'download', state:'done'}` (no `file`) | everything in | `fileBytes`, `layerBytes` (`timings`), `tookMs` (`timings.allDownloadMs`) |
|
|
2518
|
+
| `{t:'boot', stage:'compile', state:'start'\|'done'}` | around `WebAssembly.compile` | `moduleBytes`, `reused` (vm.restore() reuses the Module), `tookMs` = `timings.compileMs` |
|
|
2519
|
+
| `{t:'boot', stage:'memory', state:'done'}` | shared memory, control block, JIT store, rings made | `initialBytes`, `maxBytes`, `ringBytes` (stdio + channel block), `tookMs` |
|
|
2520
|
+
| `{t:'boot', stage:'workers', state:'progress'\|'done'}` | each Worker change; `done` once every Worker started before the vCPU is ready | `workers: [{id, role, state, atMs, tid?}]`, `vcpus`, `jitShared`, `tookMs` on done; after the end every Worker is `ended` |
|
|
2521
|
+
| `{t:'boot', stage:'kernel', state:'start'\|'done'}` | `start`: vm.wasm's `_start` began; `done`: the first console (stdout) byte. Not sent for a restored machine | `tookMs` |
|
|
2522
|
+
| `{t:'boot', stage:'restore', state:'progress'\|'done'}` | a base snapshot coming in (`restoreLazy`), every 250 ms; `done` when its memory is written | `bytes`, `total`, `source`: `'opfs'`, `'network'` or `'memory'` (vm.restore()), `used` (false with `fallback`: the snapshot was refused and the kernel boots instead), `tookMs` from `t0` to the memory written (`timings.restore.pagesWrittenAt`; a base snapshot's download may have begun before the boot). A base snapshot that falls back from the cache to the network starts its `bytes` again with `source: 'network'` |
|
|
2523
|
+
| `{t:'boot', stage:'control', state:'done'}` | the guest's yamux control session first up | `tookMs` from the vCPU start |
|
|
2524
|
+
| `{t:'heap'}` | the shared memory's size changed (checked every 250 ms) | `bytes`, `maxBytes` |
|
|
2525
|
+
| `{t:'layer'}` | a 9P service's upper first seen, then on each change of 1 MiB or more | `port`, `usedBytes`, `capBytes` (null: no cap), `kind: 'memory'` |
|
|
2526
|
+
| `{t:'exit'}` | the machine ended (the same as `vm.exited`) | `code` (-1 stopped, -2 no progress, -3 panic, -4 service died, else the guest's), `reason`, `oom` |
|
|
2527
|
+
|
|
2528
|
+
Worker ids and roles: `vcpu-0` (`vcpu`, the main Worker), `pool-<n>` (`pool`; `tid` is the guest thread it runs now:
|
|
2529
|
+
the other vCPUs and vm.wasm's own threads, which wmd-web cannot tell apart), `chan-service`, `net`, `jit-hub`.
|
|
2530
|
+
|
|
2531
|
+
The Workers stage has no `done` when a Worker failed (the list says which); every vCPU after vCPU 0 runs on a `pool` Worker.
|
|
2532
|
+
|
|
2533
|
+
What is not measured, so not sent: a `container` stage (the container's init is behind the control session; runtime-web
|
|
2534
|
+
knows when its run answers), the heap vm.wasm
|
|
2535
|
+
uses inside its memory (with W21's heap reserve the memory is at its maximum from the start, so `heap.bytes` is then the
|
|
2536
|
+
maximum). `cached` comes from Resource Timing: `transferSize` 0 with a body is true, bytes transferred is false; Firefox and
|
|
2537
|
+
WebKit give a service worker's cached response zero sizes (`page/w24/`, `test/e2e/w24-cached.spec.js`), so a zero-size
|
|
2538
|
+
entry is true only for a same-origin file that had bytes; null when the browser has no entry (its timing buffer is full).
|
|
2539
|
+
|
|
2540
|
+
A storage failure is an error named `QuotaExceededError`: a checkpoint store's `CheckpointError` with code `quota`, and the
|
|
2541
|
+
base snapshot cache's (`info.cacheWrite.name`).
|
|
2542
|
+
|
|
2543
|
+
```sh
|
|
2544
|
+
bun test test/unit/w24-boot-events.test.js
|
|
2545
|
+
PORT=8796 bunx playwright test test/e2e/w24-cached.spec.js # no guest, three browsers
|
|
2546
|
+
PORT=8795 ~/.wmd/bootlock/bootlock.sh run --max 1800 -- test/b9/run.sh b9w24 --project=chromium # boots python
|
|
2547
|
+
```
|
|
2548
|
+
|
|
2549
|
+
## 2.7: `wmd image export --format web` (D-477)
|
|
2550
|
+
|
|
2551
|
+
wmd image export --format web -o ./site alpine:latest python:3.12-slim
|
|
2552
|
+
|
|
2553
|
+
writes a folder any static file server can host: the images as packs in the static host layout
|
|
2554
|
+
(`blobs/sha256/`, `idx/...`, as `src/registry/ref.js` names them), the guest's `vm.wasm`, kernel
|
|
2555
|
+
and init (`guest/arm64/`), a copy of `src/` (`wmd-web/`, embedded in the wmd binary by
|
|
2556
|
+
`web/embed.go`), and the runner page from `export-page/` (`index.html?ref=...&cmd=...`). The page
|
|
2557
|
+
checks every guest file against `wmd-web.json`'s sha256 and every pack against its name before it
|
|
2558
|
+
boots. `_headers` carries the COOP/COEP/CORP/CSP for hosts that read it; on a host that sends no
|
|
2559
|
+
headers the page registers `coi-sw.js` and reloads once. The packs are byte-identical to
|
|
2560
|
+
`wmd pull`'s (`test/export/byte-identity.py`), and each index equals the registry proxy's
|
|
2561
|
+
(`test/export/compare-proxy.ts`). Browser gate: `test/locked.sh bunx playwright test -c test/export`.
|
|
2562
|
+
|
|
2563
|
+
## Releases: `make web-release` (2.15, D-482)
|
|
2564
|
+
|
|
2565
|
+
A release of wmd-web is the version in `package.json`, the local git tag `wmd-web-v<version>` on the
|
|
2566
|
+
commit, and `~/.wmd/releases/wmd-web-<version>.tgz` with its `.sha256`, made from a clean main commit
|
|
2567
|
+
by `make web-release` (`scripts/web-release.py`). Nothing is published and nothing is committed by it:
|
|
2568
|
+
bump `version` in a commit first (minor for a change to the public API, patch otherwise).
|
|
2569
|
+
|
|
2570
|
+
The tarball is in `npm pack` layout, so it can go to a registry unchanged later:
|
|
2571
|
+
|
|
2572
|
+
package/package.json package/README.md package/src/** package/export-page/** from git, at the commit
|
|
2573
|
+
package/guest/arm64/vm.wasm Image Image-4k initramfs.cpio.gz VM= (default guest/vm/build/vm.wasm), GUEST= (default ~/.wmd/bin/guest/arm64)
|
|
2574
|
+
package/RELEASE.json version, commit, previous release, the D numbers since it, guest/vm's tree hash, the sha256 of every file
|
|
2575
|
+
|
|
2576
|
+
The same commit and guest files give the same bytes. `python3 -I scripts/web-release.py --verify
|
|
2577
|
+
FILE.tgz` checks the `.sha256` beside the file (required), that the tarball holds only regular files
|
|
2578
|
+
named `package/<clean path>`, once each, mode 0644, owner 0, nothing after the end, and every file
|
|
2579
|
+
against RELEASE.json; an installer should run it, or do the same checks, before it uses a tarball. A released version is never rewritten, and a
|
|
2580
|
+
tag that already marks another commit is refused. `make web-release-test` tests all of this on a
|
|
2581
|
+
throwaway clone.
|
|
2582
|
+
|
|
2583
|
+
## Guests above 1536 MiB: RAM in a memory64 of its own (D-533)
|
|
2584
|
+
|
|
2585
|
+
vm.wasm's memory 0 is 4 GiB at most and also holds the emulator's heap, so `--ram` stopped at 1536 MiB
|
|
2586
|
+
(`MAX_BROWSER_RAM_MIB`). With a shared memory64 (Chrome 133, Firefox 134; `haveMemory64()`) a bigger `--ram`
|
|
2587
|
+
now boots the **flat layout**: `bootWmd` makes a second shared memory of exactly `--ram` (at most 16 GiB, the
|
|
2588
|
+
engines' memory64 limit), passes `WMD_RAM_FLAT=1` to vm.wasm and hands the memory to every Worker.
|
|
2589
|
+
|
|
2590
|
+
- Translated code imports it as `env.win` and reads guest RAM with 64-bit addresses (`host/jit.js`).
|
|
2591
|
+
- vm.wasm's C code cannot address a second memory. It calls `wmd.ram_rw` and `wmd.ram_cas`, two exports of a
|
|
2592
|
+
200-byte wasm module made in `host/ram-mem.js` (wasm to wasm: no BigInt, no JS on a page-table walk).
|
|
2593
|
+
- Memory 0 is vm.wasm's 384 MiB + a 512 MiB stand-in the board loads the kernel through + 1536 MiB of heap
|
|
2594
|
+
slack (`flatMaxPages`), whatever the guest's RAM. Leave `maxPages` out for such a boot.
|
|
2595
|
+
- `flatRam: true|false` forces the layout on or off; unset, it is chosen for `--ram` above 1536 when the engine
|
|
2596
|
+
has memory64 and `allowLargeRam` is not set. Safari has no memory64: the old limit and its message stay.
|
|
2597
|
+
- No checkpoint yet: `checkpoint: true` is ignored and `restore` / `restoreLazy` are refused with a message
|
|
2598
|
+
(a checkpoint holds memory 0 only).
|
|
2599
|
+
|
|
2600
|
+
Measured in Chromium (page/b7.js, `ram=`): 4096 shows `Mem: 3976`; 8192 with 4 vCPUs held 5366 MB in six
|
|
2601
|
+
processes with every byte checked; the same work at 1536 (small) and 4096 (flat) took 35 s and 39 s.
|
|
2602
|
+
Tests: `test/unit/ram-mem.test.js` (bun has no memory64: the helper's part runs under node).
|