space-data-module-sdk 0.8.20 → 0.8.22
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/isomorphic-pthreads.html +100 -0
- package/docs/isomorphic-pthreads.md +335 -0
- package/package.json +7 -2
- package/src/browser.js +6 -0
- package/src/compiler/invokeGlue.js +38 -0
- package/src/flow/flatsqlLinkShim.js +295 -0
- package/src/flow/index.d.ts +31 -0
- package/src/flow/index.js +8 -0
- package/src/host/browserCapabilityProbe.js +251 -0
- package/src/host/browserModuleHarness.js +24 -5
- package/src/host/flatsqlIo.js +14 -0
- package/src/host/flatsqlIoConformance.js +349 -0
- package/src/host/flatsqlIoContract.js +240 -0
- package/src/host/flatsqlIoImports.js +335 -0
- package/src/host/flatsqlIoMemoryBackend.js +178 -0
- package/src/host/flatsqlIoServer.js +960 -0
- package/src/host/flatsqlIoWorkers.js +323 -0
- package/src/host/hostWorkerBundleSources.js +11 -0
- package/src/host/hostWorkerBundles.js +79 -0
- package/src/host/index.js +9 -0
- package/src/host/isomorphicLoader.js +10 -2
- package/src/host/nodeSyncFsIo.js +633 -0
- package/src/host/opfsIoBackend.js +280 -0
- package/src/host/opfsIoWorker.mjs +223 -0
- package/src/host/sabIoChannel.js +848 -0
- package/src/host/sabIoMirror.js +306 -0
- package/src/host/wasiThreadBrowserWorker.mjs +21 -11
- package/src/host/wasiThreadHost.js +261 -43
- package/src/host/wasiThreadWorker.mjs +46 -8
- package/src/host/wasiThreadWorkerRuntime.js +54 -0
- package/src/index.d.ts +349 -0
- package/src/runtime/constants.js +6 -0
- package/src/testing/native/wasmedge_wasi_threads_runner.c +13 -5
- package/src/testing/parityHarness.js +7 -1
- package/src/testing/parityLanes.js +19 -0
|
@@ -146,6 +146,16 @@
|
|
|
146
146
|
<div class="codeblock"><div class="codeblock-head">sh</div><pre><code>SPACE_DATA_MODULE_SDK_ENABLE_WASMEDGE_PARITY=1 \
|
|
147
147
|
SPACE_DATA_MODULE_SDK_ENABLE_TRI_RUNTIME_PARITY=1 \
|
|
148
148
|
node --test test/wasi-threads-command.test.js</code></pre></div>
|
|
149
|
+
<h3 id="sdk-0821-constructors-on-the-direct-surface"><a class="anchor" href="#sdk-0821-constructors-on-the-direct-surface" aria-hidden="true">#</a>SDK 0.8.21: constructors on the direct surface</h3>
|
|
150
|
+
<p>An artifact built with both the <code>direct</code> and the <code>command</code> surface links the WASI command runtime. Its <code>_start</code> sets up the main thread's pthread descriptor, runs the global constructors, then runs <code>main</code>, which reads stdin. A host that serves the direct surface never enters <code>_start</code>. From 0.8.21 such an artifact also exports <code>__wasm_call_ctors</code>: the same descriptor setup and constructors, without <code>main</code>, at most once per instance. Reactors keep <code>_initialize</code>.</p>
|
|
151
|
+
<p>A direct host runs <code>_initialize</code> if the module exports it, otherwise <code>__wasm_call_ctors</code>, once per instance, before the first direct call. The browser harness does this for <code>surface: "direct"</code>; a command instance only enters <code>_start</code>. The wasi-threads runner serves the direct surface of a command artifact with <code>--sdm-direct</code>, selected by <code>createStandaloneHarness("wasmedge", path, { surface: "direct" })</code> and by <code>runParityHarness({ surface: "direct" })</code>. The SDN node uses the same order.</p>
|
|
152
|
+
<p>Artifacts built with 0.8.20 or earlier have no such export. On their direct surface the constructors never run, in every runtime, and a threaded artifact's main thread has no pthread descriptor, so a recursive mutex held by the main thread does not exclude other threads. Rebuild them with 0.8.21.</p>
|
|
153
|
+
<div class="codeblock"><div class="codeblock-head">sh</div><pre><code>SPACE_DATA_MODULE_SDK_ENABLE_WASMEDGE_PARITY=1 \
|
|
154
|
+
SPACE_DATA_MODULE_SDK_ENABLE_TRI_RUNTIME_PARITY=1 \
|
|
155
|
+
node --test test/direct-call-constructors.test.js</code></pre></div>
|
|
156
|
+
<h3 id="guest-thread-faults"><a class="anchor" href="#guest-thread-faults" aria-hidden="true">#</a>Guest thread faults</h3>
|
|
157
|
+
<p>A guest thread that traps never finishes the pthread exit protocol. WasmEdge cancels the whole command. In the browser and Node harnesses the joining thread stays blocked inside the guest and cannot run the worker's error event, so the worker itself writes <code>[wasi-thread] guest thread N trapped: ...</code> to stderr (Node) or the console (browser). The call still does not return.</p>
|
|
158
|
+
<p>V8 (Node 20 to 25) checks bulk memory operations, and every access when the WebAssembly trap handler is off (Node on Linux arm64), against a per-instance copy of a shared memory's size. That copy is refreshed asynchronously after another thread grows the memory, so a thread that writes into memory another thread has just grown can trap with "memory access out of bounds", even after synchronizing with the growing thread. The guest's allocator uses the heap the artifact was linked with and then grows the memory, so a larger imported initial memory does not prevent it.</p>
|
|
149
159
|
<p>The old source path <code>src/testing/browserModuleHarness.js</code> remains a pure compatibility re-export. New browser consumers should use the public <code>space-data-module-sdk/host/browser-module</code> entry point.</p>
|
|
150
160
|
<h2 id="4-integrators-the-browser-worker-anchor-required-when-you-bundle"><a class="anchor" href="#4-integrators-the-browser-worker-anchor-required-when-you-bundle" aria-hidden="true">#</a>4. Integrators: the browser worker anchor (REQUIRED when you bundle)</h2>
|
|
151
161
|
<p>The browser leg of the wasi-threads host runs each guest pthread on a pooled module <code>Worker</code>. That worker is a <strong>served asset</strong>, and the SDK cannot guess where your build published it.</p>
|
|
@@ -174,6 +184,73 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
174
184
|
<li><strong>An unreachable worker fails LOUD.</strong> When the pooled path was requested (threads enabled + cross-origin isolated + shared memory) and the worker asset at the resolved anchor never loads, <code>createWasiThreadSpawn</code> <strong>throws</strong> <code>WasiThreadWorkerUnreachableError</code> naming the URL it tried. It does not drop quietly to one thread — a silent sequential fallback is how a deployment defect hid behind a passing gate as a pure performance loss.</li>
|
|
175
185
|
</ul>
|
|
176
186
|
<p>Genuine capability negotiation is unaffected and stays soft: a worker that loads and reports it cannot instantiate the module, a probe timeout, a non-isolated context, non-shared memory, or a 1-core host all still disable threading and let the guest run its proven sequential path (<code>wasi.thread-spawn</code> -> <code>-1</code>).</p>
|
|
187
|
+
<h2 id="5-flatsql-partition-store-host-io-sdk-0822"><a class="anchor" href="#5-flatsql-partition-store-host-io-sdk-0822" aria-hidden="true">#</a>5. FlatSQL partition store host I/O (SDK 0.8.22)</h2>
|
|
188
|
+
<p>The FlatSQL partition store (stack design <code>docs/architecture/flatsql-partition-store.md</code>, §5.5, §5.6, §18 T9, A7, A36, A38, A39) runs one wasi-threads artifact as a writer instance and reader instances, each with guest threads that call FlatSQL's seven <code>env.flatsql_io_*</code> imports. This section is the SDK half: the thread pool, the I/O channel and workers, the Node provider, link shim v2, the worker bundles and the browser capability probe.</p>
|
|
189
|
+
<h3 id="explicit-pool-size-partial-spawns-supervision-hooks"><a class="anchor" href="#explicit-pool-size-partial-spawns-supervision-hooks" aria-hidden="true">#</a>Explicit pool size, partial spawns, supervision hooks</h3>
|
|
190
|
+
<p><code>createWasiThreadSpawn</code> options added in 0.8.22:</p>
|
|
191
|
+
<div class="table-wrap"><table><thead><tr><th>Option</th><th>Effect</th></tr></thead><tbody><tr><td><code>poolSize</code></td><td>Browser: pre-start exactly this many workers, independent of <code>hardwareConcurrency - 1</code> (pools are sized for isolation: writers + lanes). Node: cap on live guest threads. Arming is partial: workers that fail to start are dropped, the rest serve.</td></tr><tr><td><code>extraImports</code></td><td>Per-worker import objects, as structured-cloneable descriptors (below). Factories run once per worker.</td></tr><tr><td><code>instanceId</code>, <code>onGuestError(instanceId, tid, error)</code></td><td>Called when a guest thread traps or its worker dies (A36). A dead pooled worker leaves the pool.</td></tr><tr><td><code>onSpawnDeclined({ reason, poolSize, declined })</code></td><td>Called for every spawn that returns -1. Reasons: <code>pool-exhausted</code>, <code>pool-not-armed</code>, <code>threads-unavailable</code>, <code>pool-empty</code>, <code>worker-create-failed</code>, <code>dispatch-failed</code>, <code>hostcall-channel-missing</code>, <code>terminated</code>.</td></tr><tr><td><code>probeTimeoutMs</code></td><td>Browser warm-pool probe deadline.</td></tr><tr><td><code>browserWorkerType: "classic"</code></td><td>Spawn classic workers, for the blob bundles below.</td></tr></tbody></table></div>
|
|
192
|
+
<p>The returned host adds <code>spawnReport()</code>: <code>{ poolSize, armed, failedToArm, spawned, declined, declinedByReason, lastDeclineReason, active, idle }</code>. The implicit (no <code>poolSize</code>) path keeps its all-or-nothing arming.</p>
|
|
193
|
+
<p><code>extraImports</code> entries (the same descriptors work in the engine worker through <code>resolveExtraImports</code>):</p>
|
|
194
|
+
<div class="table-wrap"><table><thead><tr><th>Descriptor</th><th>Imports</th></tr></thead><tbody><tr><td><code>{ provider: "flatsql-io", instanceId, channels, mirror?, trace? }</code></td><td><code>env.flatsql_io_*</code> over one SAB I/O channel per I/O worker. Each worker claims its own request slot.</td></tr><tr><td><code>{ provider: "flatsql-io-node", root, table, instanceId }</code></td><td><code>env.flatsql_io_*</code> over synchronous <code>fs</code> (Node workers).</td></tr><tr><td><code>{ moduleUrl, exportName?, config? }</code></td><td>A factory module (module workers and Node only).</td></tr></tbody></table></div>
|
|
195
|
+
<p>A module that bundles this host into an IIFE or a blob: module worker has no usable <code>import.meta.url</code>. <code>DEFAULT_BROWSER_WORKER_URL</code> is then <code>null</code> instead of a module-evaluation <code>TypeError</code>, and such hosts pass <code>browserWorkerUrl</code>.</p>
|
|
196
|
+
<h3 id="flags-and-statuses"><a class="anchor" href="#flags-and-statuses" aria-hidden="true">#</a>Flags and statuses</h3>
|
|
197
|
+
<p>The partition store adds three open flags and one status. They are flags, so the import set stays at seven. flatsql's <code>flatsql_io.h</code> (T1) must carry the same values.</p>
|
|
198
|
+
<div class="table-wrap"><table><thead><tr><th>Name</th><th>Value</th><th>Meaning</th></tr></thead><tbody><tr><td><code>FLATSQL_IO_CREATE_PARENTS</code></td><td><code>0x0100</code></td><td>mkdir -p; sync each new directory's parent, and the parent of a new file. Best effort on OPFS.</td></tr><tr><td><code>FLATSQL_IO_UNLINK_IF_UNUSED</code></td><td><code>0x0200</code></td><td>Unlink, or <code>BUSY</code> while any handle names the path.</td></tr><tr><td><code>FLATSQL_IO_OPEN_DEFERRED</code></td><td><code>0x0400</code></td><td>Return a handle before the open finishes; the first use waits, or fails with the open's status. Synchronous hosts treat it as a plain open.</td></tr><tr><td><code>FLATSQL_IO_ERR_BUSY</code></td><td><code>-7</code></td><td>Path in use, or its lock held elsewhere.</td></tr></tbody></table></div>
|
|
199
|
+
<p>Every SDK host maps <code>EEXIST</code> to <code>GENERIC</code> (the Go host's mapping), a write on a read-only handle and a read on a write-only handle to <code>IO</code>, and <code>..</code> to <code>ACCESS</code>. OPFS errors map as NotFound -> <code>NOENT</code>, NoModificationAllowed -> <code>BUSY</code>, QuotaExceeded -> <code>NOSPACE</code>; WebKit refuses a second sync handle with InvalidStateError (measured), which maps to <code>BUSY</code> during an open.</p>
|
|
200
|
+
<h3 id="the-sab-io-channel-and-the-io-worker"><a class="anchor" href="#the-sab-io-channel-and-the-io-worker" aria-hidden="true">#</a>The SAB I/O channel and the I/O worker</h3>
|
|
201
|
+
<p><code>sabIoChannel.js</code> extends <code>sabHostcallChannel.js</code> to many guest threads and one server:</p>
|
|
202
|
+
<ul>
|
|
203
|
+
<li>A request ring of slots in one SharedArrayBuffer. Each request owns a slot until its result is read, so a guest never queues behind another guest's request and a slow open blocks only its caller. Idle threads hold no slot, so terminating a pool leaks none; <code>reclaimSabIoSlots(buffer, instanceId)</code> frees the slots of requests that were in flight when an instance's threads died.</li>
|
|
204
|
+
<li>Doorbell: <code>Atomics.waitAsync</code> on a header word. The server publishes the mode; in message mode (<code>doorbell: "message"</code>, or no <code>waitAsync</code>) clients also post on a BroadcastChannel. Completions for non-blocking callers without <code>waitAsync</code> arrive the same way (22.3a-5).</li>
|
|
205
|
+
<li>No timeout, no throw (A36). Blocking waits run in 250 ms slices forever. A supervisor that sees the I/O worker die calls <code>failPendingSabIoRequests</code>, which completes every pending slot with <code>IO</code>.</li>
|
|
206
|
+
<li>Data moves directly between the instance's shared memory and the file when the instance's memory is attached to the I/O worker, else through the slot's data area. Paths are always copied out of shared memory before decoding.</li>
|
|
207
|
+
<li>Revocation (A36): <code>revokeSabIoInstance(buffer, id)</code> resolves after the server has closed the instance's handles; later requests get <code>ACCESS</code> and no byte of the instance is written after it resolves.</li>
|
|
208
|
+
</ul>
|
|
209
|
+
<p><code>opfsIoWorker.mjs</code> runs <code>flatsqlIoServer.js</code> over a backend:</p>
|
|
210
|
+
<ul>
|
|
211
|
+
<li><code>opfs</code>: every open is async (<code>getDirectoryHandle</code>, <code>getFileHandle</code>, <code>createSyncAccessHandle</code>) and awaited on the worker's event loop while the requester waits on its slot. One sync handle per path, shared by all virtual handles. Handles open in <code>"readwrite-unsafe"</code> mode where it exists, so a reader I/O worker can read files a writer I/O worker holds (A7). Views over a SharedArrayBuffer go straight to <code>read</code>/<code>write</code>; a user agent that rejects them gets a private scratch copy.</li>
|
|
212
|
+
<li><code>memory</code>: the dashboard window store (§5.5). Nothing reaches OPFS; <code>reset()</code> drops it; <code>memoryMaxBytes</code> is the per-tab budget (<code>NOSPACE</code>).</li>
|
|
213
|
+
</ul>
|
|
214
|
+
<p>Roles (A7): one writer I/O worker per writer holds the writer's active files; reader I/O workers (one, or two when <code>hardwareConcurrency >= 8</code>) hold sealed files. Reads and writes run in steps of at most 256 KiB with the slots rescanned between steps, so a small read never waits for a whole large write. With several reader I/O workers, <code>createFlatsqlIoImports</code> routes each open by a path hash and every later call to the same worker (the worker index rides in handle bits 24-30).</p>
|
|
215
|
+
<p><code>createFlatsqlIoWorker(options)</code> spawns and supervises one I/O worker: <code>attachMemory</code>, <code>revoke</code>, <code>preopen(paths)</code> (A38: open the registry's active files in parallel at start; later guest opens of those paths return at once), <code>releasePreopen</code>, <code>stats</code>, <code>reset</code>, <code>clear</code>, <code>stop</code>, and <code>onError</code> plus <code>restart: true</code> (A36: pending requests fail with <code>IO</code>, a replacement starts on the same channel with the live memories re-attached; handles do not survive).</p>
|
|
216
|
+
<p>Store lock (A37): with <code>lock: { name }</code> the I/O worker takes that Web Lock before it opens any handle and releases it only after <code>stop()</code> has closed them, so the lock lives exactly as long as the handles. <code>ifAvailable: true</code> fails the start with <code>lockUnavailable</code> instead of waiting; <code>busyRetry</code> retries a handle a previous leader still holds, with backoff. Measured takeover (holder <code>stop()</code> to successor ready): 5 ms Chromium, 52 ms Firefox, 5 ms WebKit. Leadership, follower proxying and heartbeats are the engine's (sdn-js, T10).</p>
|
|
217
|
+
<p>Head mirror (A7): with <code>mirror: { buffer, suffixes: ["/h.fsh"] }</code>, the writer I/O worker copies every write to a matching path into a seqlock mirror in a SharedArrayBuffer (<code>sabIoMirror.js</code>), and reader imports configured with the same mirror serve reads of those paths from it without a round trip.</p>
|
|
218
|
+
<h3 id="node-synchronous-fs-56"><a class="anchor" href="#node-synchronous-fs-56" aria-hidden="true">#</a>Node synchronous fs (§5.6)</h3>
|
|
219
|
+
<p><code>nodeSyncFsIo.js</code>: synchronous <code>fs</code> in each worker over a shared virtual-handle table in a SharedArrayBuffer (<code>createNodeSyncFsIoTable</code>). A handle is <code>(slot << 8) | gen</code>; each worker opens its own fd for a slot on first use and drops stale fds when the generation moves. Paths are confined below <code>root</code>, including through symlinked parents. <code>sync</code> is <code>fdatasync</code> (libuv issues <code>F_FULLFSYNC</code> on darwin). <code>revokeNodeSyncFsIoInstance</code> follows A23: it sets the revoked flag and waits for the instance's in-flight calls to drain. Fault injection (§19, 22.3a-7) belongs to FlatSQL's Node host (T4); <code>interpose</code> wraps every syscall for it.</p>
|
|
220
|
+
<h3 id="link-shim-v2"><a class="anchor" href="#link-shim-v2" aria-hidden="true">#</a>Link shim v2</h3>
|
|
221
|
+
<p><code>FLATSQL_LINK_SHIM_V2_WASM</code> (<code>src/flow/flatsqlLinkShim.js</code>) is a deterministic module that imports the reader instance's SHARED lane memory as <code>flatsql.memory</code> and gives a linked flow the lane mailbox as direct calls. v1 is unchanged (sha256 <code>8d83e69b…</code>), because today's linked flows use it. v2 sha256: <code>67d5b2d9bc2d1b346a14a253a586fd4d08c8056d54eb62701b48000585ed9613</code>.</p>
|
|
222
|
+
<div class="table-wrap"><table><thead><tr><th>Export</th><th>Result</th></tr></thead><tbody><tr><td><code>mb_submit(mailbox, op, req_ptr, req_len)</code></td><td><code>seq</code>, or -1 when the mailbox holds a request</td></tr><tr><td><code>mb_poll(mailbox, seq)</code></td><td>1 when complete</td></tr><tr><td><code>mb_wait(mailbox, seq, poll_ns: i64, max_polls)</code></td><td>the lane's status, or -110 after <code>max_polls</code> bounded waits (<code>max_polls <= 0</code>: no limit)</td></tr><tr><td><code>mb_release(mailbox, seq)</code></td><td>0, or -1 when not complete</td></tr><tr><td><code>mb_cancel(mailbox, seq)</code></td><td>0; the lane polls the cancel word</td></tr><tr><td><code>load32_acquire</code>, <code>store32_release</code>, <code>peek8/32/64</code>, <code>poke8/32</code>, <code>fnv1a64</code>, <code>count_frames</code></td><td>as in v1, over lane memory</td></tr></tbody></table></div>
|
|
223
|
+
<p>Mailbox, 64 bytes, 8-aligned, little-endian: <code>+0 state</code> (IDLE 0, SUBMITTED 1, CLAIMED 2, DONE 3, SUBMITTING 4), <code>+4 seq</code>, <code>+8 done_seq</code>, <code>+12 op</code>, <code>+16 req_ptr</code>, <code>+20 req_len</code>, <code>+24 status</code>, <code>+28 resp_ptr</code>, <code>+32 resp_len</code>, <code>+36 doorbell</code>, <code>+40 cancel</code>, <code>+44 flags</code>, <code>+48 generation (u64)</code>. A lane waits (bounded) on the doorbell, CASes SUBMITTED -> CLAIMED, writes the result, stores <code>done_seq = seq</code>, stores DONE and notifies the state word.</p>
|
|
224
|
+
<p><code>mb_wait</code> never waits unboundedly: each <code>memory.atomic.wait32</code> lasts <code>poll_ns</code> and every wake re-reads the mailbox. WasmEdge keeps waiters per executor, so a lane's notify may never reach a flow on another executor; a lost notify then costs one poll interval and never a completion. <code>poll_ns = 0</code> polls without executing a wait (contexts that may not block).</p>
|
|
225
|
+
<h3 id="worker-bundles-a39"><a class="anchor" href="#worker-bundles-a39" aria-hidden="true">#</a>Worker bundles (A39)</h3>
|
|
226
|
+
<p>The dashboard is one HTML file under <code>worker-src 'self' blob:</code>. <code>space-data-module-sdk/host/worker-bundles</code> ships the pool worker and the I/O worker as self-contained classic scripts (esbuild IIFE, built by <code>scripts/build-host-worker-bundles.mjs</code> into <code>src/host/hostWorkerBundleSources.js</code>):</p>
|
|
227
|
+
<div class="codeblock"><div class="codeblock-head">js</div><pre><code>import { hostWorkerBundleUrl } from "space-data-module-sdk/host/worker-bundles";
|
|
228
|
+
import { createWasiThreadSpawn } from "space-data-module-sdk/host/wasi-threads";
|
|
229
|
+
import { createFlatsqlIoWorker } from "space-data-module-sdk/host/flatsql-io";
|
|
230
|
+
|
|
231
|
+
const writerIo = await createFlatsqlIoWorker({
|
|
232
|
+
workerUrl: hostWorkerBundleUrl("flatsql-io"), workerType: "classic",
|
|
233
|
+
backend: "opfs", role: "writer", rootDirectory: "sdn-store",
|
|
234
|
+
});
|
|
235
|
+
await writerIo.attachMemory(1, writerMemory);
|
|
236
|
+
const pool = await createWasiThreadSpawn({
|
|
237
|
+
wasmModule, memory: writerMemory, poolSize: 1 + 2, instanceId: 1,
|
|
238
|
+
enableBrowserThreads: true,
|
|
239
|
+
browserWorkerUrl: hostWorkerBundleUrl("wasi-thread-pool"), browserWorkerType: "classic",
|
|
240
|
+
extraImports: [{ provider: "flatsql-io", instanceId: 1, channels: [writerIo.buffer] }],
|
|
241
|
+
onGuestError: (instanceId, tid, error) => supervisor.poison(instanceId, tid, error),
|
|
242
|
+
});</code></pre></div>
|
|
243
|
+
<h3 id="browser-capability-matrix"><a class="anchor" href="#browser-capability-matrix" aria-hidden="true">#</a>Browser capability matrix</h3>
|
|
244
|
+
<p><code>probeBrowserCapabilities()</code> probes the page and a blob worker; <code>flatsqlLocalStoreGate(matrix)</code> is the store gate: cross-origin isolation, shared memory, OPFS sync handles in a worker, <code>Atomics.wait</code> in workers, and shared sync-handle modes (§22.4-6: without them there is no local store). Measured 2026-09-27 on the owner's Mac Studio (Apple M3 Ultra, macOS 26.3.1), headless, Playwright 1.63.0, under the dashboard CSP with COOP/COEP:</p>
|
|
245
|
+
<div class="table-wrap"><table><thead><tr><th></th><th>Chromium 153.0.8010.12</th><th>Firefox 155.0</th><th>WebKit 26.6</th></tr></thead><tbody><tr><td><code>crossOriginIsolated</code>, shared wasm memory</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td><code>Atomics.waitAsync</code> (page and worker)</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>OPFS sync access handle in a worker</td><td>yes</td><td>yes</td><td>yes (persistent profile only)</td></tr><tr><td>Second default handle on one file</td><td>NoModificationAllowedError</td><td>NoModificationAllowedError</td><td>InvalidStateError</td></tr><tr><td>Shared modes (<code>readwrite-unsafe</code> x 2)</td><td>yes</td><td>no (mode ignored)</td><td>no (mode ignored)</td></tr><tr><td>SharedArrayBuffer views in <code>read</code>/<code>write</code></td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>Wasm shared-memory views in <code>read</code></td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td><code>removeEntry</code> of an open file</td><td>NoModificationAllowedError</td><td>NoModificationAllowedError</td><td>NoModificationAllowedError</td></tr><tr><td>Nested blob workers</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td><code>performance.now()</code> resolution</td><td>5 µs</td><td>20 µs</td><td>20 µs</td></tr><tr><td>Local-store gate</td><td>supported</td><td>no (shared modes)</td><td>no (shared modes)</td></tr></tbody></table></div>
|
|
246
|
+
<p>WebKit refuses OPFS in Playwright's ephemeral context (<code>UnknownError</code>); the suite uses a persistent profile.</p>
|
|
247
|
+
<h3 id="conformance"><a class="anchor" href="#conformance" aria-hidden="true">#</a>Conformance</h3>
|
|
248
|
+
<p><code>runFlatsqlIoConformance(io)</code> (<code>flatsqlIoConformance.js</code>) is one script for every host (22.3a-6): 15 cases covering statuses, short reads, sparse writes, truncation, EXCL/TRUNC/PROBE/UNLINK/UNLINK_IF_UNUSED/CREATE_PARENTS/ DELETE_ON_CLOSE/OPEN_DEFERRED, access modes, confinement and multi-handle visibility. It passes on the Node sync-fs provider, on the channel over the memory backend (blocking and async clients, both doorbells), on the blob bundle run as a standalone script, and on OPFS and memory in Chromium, Firefox and WebKit. Plain <code>UNLINK</code> of an open path is <code>BUSY</code> in the I/O worker (OPFS cannot remove a file with an open sync handle) and succeeds on POSIX hosts; the script does not test it.</p>
|
|
249
|
+
<h3 id="measured-acceptance-18-t9-a7-a38"><a class="anchor" href="#measured-acceptance-18-t9-a7-a38" aria-hidden="true">#</a>Measured acceptance (§18 T9, A7, A38)</h3>
|
|
250
|
+
<p>Owner's Mac Studio (Apple M3 Ultra, 28 cores, macOS 26.3.1), 2026-09-27, headless, final full run of <code>test/opfs-io-worker.browser.test.js</code>. The machine was shared with other lanes (load average 23-33 on 28 cores). Latencies are guest-observed import calls (<code>trace</code>), in microseconds.</p>
|
|
251
|
+
<div class="table-wrap"><table><thead><tr><th>Item</th><th>Chromium 153</th><th>Firefox 155</th><th>WebKit 26.6</th></tr></thead><tbody><tr><td>#1 8 threads, mixed I/O + 200 async opens + 50 unlinks through 1 I/O worker: errors / lost writes</td><td>0 / 0</td><td>0 / 0</td><td>0 / 0</td></tr><tr><td>same run, 4 KiB read p50 / p99</td><td>50 / 2145</td><td>260 / 8180</td><td>60 / 2660</td></tr><tr><td>same run, message doorbell: errors / lost writes</td><td>0 / 0</td><td>0 / 0</td><td>0 / 0</td></tr><tr><td>A7 lane 4 KiB read p99 during 100 x 4 MiB write+flush, same partition</td><td>790 (active file, shared mode)</td><td>120 (sealed segment)</td><td>40 (sealed segment)</td></tr><tr><td>A7 same, other partition</td><td>45</td><td>120</td><td>40</td></tr><tr><td>A7 writer flush p50 / p99 (4 MiB)</td><td>6365 / 28945</td><td>3540 / 27740</td><td>5260 / 70700</td></tr><tr><td>#2 500 ms open, median of 3 runs: other threads' read p99, baseline / during</td><td>85 / 85</td><td>440 / 400</td><td>60 / 60</td></tr><tr><td>#2 longest other-thread read while the open was pending</td><td>1020</td><td>18500</td><td>320</td></tr><tr><td>A38 OPEN_DEFERRED: open returns in / first write waits (ms)</td><td>3.8 / 504</td><td>28.8 / 511</td><td>5.2 / 755</td></tr><tr><td>#3 <code>poolSize=6</code> on <code>hardwareConcurrency=2</code></td><td>6 workers; 7th spawn -1, reported <code>pool-exhausted</code></td><td>same</td><td>same</td></tr><tr><td>#5 link shim v2 (Node 25): 10,000 calls, lost completions</td><td>0 with a notifying lane, 0 with a lane that never notifies, 0 spinning (<code>poll_ns = 0</code>)</td><td></td><td></td></tr></tbody></table></div>
|
|
252
|
+
<p>Chromium's same-partition A7 p99 varies with host load: 90, 110, 295 and 790 over four runs with the adaptive spin (below), and 125-1165 over four runs before it. The original #1 target (4 KiB read p99 <= 200 µs under the mixed load) is not met through one I/O worker; A7 replaced #1 with the <= 1 ms lane target, which is met. Firefox and WebKit have no shared handle modes, so their lanes read a partition's sealed segment; per §22.4-6 those browsers get no local store.</p>
|
|
253
|
+
<p>Adaptive waits: an idle I/O worker polls its doorbell for 50 µs, and a blocking client polls for its answer for 20 µs, before sleeping (<code>spinMicros</code>), so back-to-back requests skip a thread wake-up. In the Node channel test this took the fast-thread read p50 from 30-39 µs to 11-12 µs.</p>
|
|
177
254
|
<h2 id="tests"><a class="anchor" href="#tests" aria-hidden="true">#</a>Tests</h2>
|
|
178
255
|
<ul>
|
|
179
256
|
<li><code>test/wasi-thread-bundled-consumer-anchor.test.js</code> — the <strong>bundled-consumer guardrail</strong>: the host source is run through esbuild into a directory that does not hold the worker chain (the published-bucket geometry), driven by a Worker mock that resolves URLs on the filesystem the way a browser resolves them against an origin. With no anchor the pooled path must throw <code>WasiThreadWorkerUnreachableError</code> (the silent sequential fallback is a HARD failure); with an explicit base — or the process-wide setter — the same bundle arms its pool and spawns threads. Also pins the precedence table and asserts the resolver neither fetches nor sniffs <code>location</code>.</li>
|
|
@@ -181,6 +258,16 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
181
258
|
<li><code>test/pthreads-artifact-guardrail.test.js</code> — flag-assembler invariants; a positive compile that emits a validated wasi-threads shared-memory/atomics wasm; a single-thread artifact rejected; a <strong>browser-only Emscripten <code>-pthread</code> artifact rejected</strong> (has shared memory + atomics but no wasi-threads contract); a shared-flag-stripped artifact rejected; and a false-positive guard proving the atomics decoder ignores <code>0xFE</code> immediates.</li>
|
|
182
259
|
<li><code>test/guest-link-symbol-prefix.test.js</code> — the guest-link prefix is the full injective hex of the pluginId; two previously-colliding ids now get distinct prefixes; fresh prefixes match the committed modules-branch artifacts byte-for-byte; and the compose path treats the guest-link metadata's <code>symbolPrefix</code> / <code>methodSymbols</code> as authoritative (never re-derived), keeping a legacy truncated-prefix artifact compatible.</li>
|
|
183
260
|
</ul>
|
|
261
|
+
<ul>
|
|
262
|
+
<li><code>test/wasi-thread-pool-size.test.js</code> — explicit <code>poolSize</code> (6 workers on <code>hardwareConcurrency=2</code>, the 7th spawn -1 and reported), partial arming, extraImports delivery, <code>onGuestError</code>, classic workers.</li>
|
|
263
|
+
<li><code>test/sab-io-channel.test.js</code> — the I/O channel under a real wasi-threads guest (<code>test/support/flatsql-io/ioGuestWasm.mjs</code>): 8 threads of mixed I/O in both doorbell modes, a 500 ms open blocking only its caller, OPEN_DEFERRED, revocation, a dead I/O worker, supervisor restart, 256 KiB steps, the head mirror, scratch vs direct transfers, pre-open, and path-hash routing.</li>
|
|
264
|
+
<li><code>test/node-sync-fs-io.test.js</code> — shared handles across workers, stale handles, CREATE_PARENTS, UNLINK_IF_UNUSED, confinement, A23 revocation.</li>
|
|
265
|
+
<li><code>test/flatsql-io-conformance.test.js</code> — the conformance script on every Node host.</li>
|
|
266
|
+
<li><code>test/host-worker-bundles.test.js</code> — the bundles equal a fresh build, and the I/O worker bundle passes the conformance script standalone.</li>
|
|
267
|
+
<li><code>test/link-shim-v2.test.js</code> — shim v2 bytes, contract, and 10,000 calls against a stub lane with and without notify.</li>
|
|
268
|
+
<li><code>test/opfs-io-worker.browser.test.js</code> — the real-browser suite above (env-gated):</li>
|
|
269
|
+
</ul>
|
|
270
|
+
<div class="codeblock"><div class="codeblock-head">sh</div><pre><code>SPACE_DATA_MODULE_SDK_ENABLE_BROWSER_IO=1 node --test test/opfs-io-worker.browser.test.js</code></pre></div>
|
|
184
271
|
<h2 id="see-also"><a class="anchor" href="#see-also" aria-hidden="true">#</a>See also</h2>
|
|
185
272
|
<ul>
|
|
186
273
|
<li><a href="./browser-wasmedge-isomorphic.html"><code>docs/browser-wasmedge-isomorphic.md</code></a> — the one-artifact browser + WasmEdge loading profile.</li>
|
|
@@ -188,6 +275,7 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
188
275
|
<li><code>src/host/wasiThreadHost.js</code> — the isomorphic <code>wasi.thread-spawn</code> host: <code>createWasiThreadSpawn</code>, the browser worker anchor (<code>setBrowserWasiThreadWorkerBase</code>, <code>resolveBrowserWorkerUrl</code>), and <code>WasiThreadWorkerUnreachableError</code>.</li>
|
|
189
276
|
<li><code>src/compiler/pthreadArtifactGuard.js</code> — the flag list + wasm validator.</li>
|
|
190
277
|
<li><code>src/compiler/wasiThreadsToolchain.js</code> — the wasi-threads toolchain resolver.</li>
|
|
278
|
+
<li><code>src/host/sabIoChannel.js</code>, <code>src/host/flatsqlIoServer.js</code>, <code>src/host/opfsIoWorker.mjs</code>, <code>src/host/flatsqlIoWorkers.js</code>, <code>src/host/nodeSyncFsIo.js</code>, <code>src/host/sabIoMirror.js</code>, <code>src/host/flatsqlIoConformance.js</code>, <code>src/host/browserCapabilityProbe.js</code>, <code>src/host/hostWorkerBundles.js</code> — the FlatSQL partition store host I/O (§5).</li>
|
|
191
279
|
<li><code>src/compiler/compileModule.js</code> — <code>ModuleThreadModel</code>, <code>buildCompilerArgs</code>, <code>compileWithWasiThreads</code>, <code>resolveThreadModel</code>, <code>compileModuleFromSource</code>.</li>
|
|
192
280
|
</ul>
|
|
193
281
|
|
|
@@ -204,7 +292,19 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
204
292
|
<li><a class="depth-2" href="#guest-link-symbol-namespacing-collision-proof-metadata-authoritative">Guest-link symbol namespacing (collision-proof, metadata-authoritative)</a></li>
|
|
205
293
|
<li><a class="depth-2" href="#3-compile-time-vs-runtime-an-honest-boundary">3. Compile-Time vs. Runtime: an honest boundary</a></li>
|
|
206
294
|
<li><a class="depth-3" href="#sdk-0820-command-hosts">SDK 0.8.20 command hosts</a></li>
|
|
295
|
+
<li><a class="depth-3" href="#sdk-0821-constructors-on-the-direct-surface">SDK 0.8.21: constructors on the direct surface</a></li>
|
|
296
|
+
<li><a class="depth-3" href="#guest-thread-faults">Guest thread faults</a></li>
|
|
207
297
|
<li><a class="depth-2" href="#4-integrators-the-browser-worker-anchor-required-when-you-bundle">4. Integrators: the browser worker anchor (REQUIRED when you bundle)</a></li>
|
|
298
|
+
<li><a class="depth-2" href="#5-flatsql-partition-store-host-io-sdk-0822">5. FlatSQL partition store host I/O (SDK 0.8.22)</a></li>
|
|
299
|
+
<li><a class="depth-3" href="#explicit-pool-size-partial-spawns-supervision-hooks">Explicit pool size, partial spawns, supervision hooks</a></li>
|
|
300
|
+
<li><a class="depth-3" href="#flags-and-statuses">Flags and statuses</a></li>
|
|
301
|
+
<li><a class="depth-3" href="#the-sab-io-channel-and-the-io-worker">The SAB I/O channel and the I/O worker</a></li>
|
|
302
|
+
<li><a class="depth-3" href="#node-synchronous-fs-56">Node synchronous fs (§5.6)</a></li>
|
|
303
|
+
<li><a class="depth-3" href="#link-shim-v2">Link shim v2</a></li>
|
|
304
|
+
<li><a class="depth-3" href="#worker-bundles-a39">Worker bundles (A39)</a></li>
|
|
305
|
+
<li><a class="depth-3" href="#browser-capability-matrix">Browser capability matrix</a></li>
|
|
306
|
+
<li><a class="depth-3" href="#conformance">Conformance</a></li>
|
|
307
|
+
<li><a class="depth-3" href="#measured-acceptance-18-t9-a7-a38">Measured acceptance (§18 T9, A7, A38)</a></li>
|
|
208
308
|
<li><a class="depth-2" href="#tests">Tests</a></li>
|
|
209
309
|
<li><a class="depth-2" href="#see-also">See also</a></li></ul></nav>
|
|
210
310
|
</div>
|
|
@@ -216,6 +216,51 @@ SPACE_DATA_MODULE_SDK_ENABLE_TRI_RUNTIME_PARITY=1 \
|
|
|
216
216
|
node --test test/wasi-threads-command.test.js
|
|
217
217
|
```
|
|
218
218
|
|
|
219
|
+
### SDK 0.8.21: constructors on the direct surface
|
|
220
|
+
|
|
221
|
+
An artifact built with both the `direct` and the `command` surface links the
|
|
222
|
+
WASI command runtime. Its `_start` sets up the main thread's pthread descriptor,
|
|
223
|
+
runs the global constructors, then runs `main`, which reads stdin. A host that
|
|
224
|
+
serves the direct surface never enters `_start`. From 0.8.21 such an artifact
|
|
225
|
+
also exports `__wasm_call_ctors`: the same descriptor setup and constructors,
|
|
226
|
+
without `main`, at most once per instance. Reactors keep `_initialize`.
|
|
227
|
+
|
|
228
|
+
A direct host runs `_initialize` if the module exports it, otherwise
|
|
229
|
+
`__wasm_call_ctors`, once per instance, before the first direct call. The
|
|
230
|
+
browser harness does this for `surface: "direct"`; a command instance only
|
|
231
|
+
enters `_start`. The wasi-threads runner serves the direct surface of a command
|
|
232
|
+
artifact with `--sdm-direct`, selected by
|
|
233
|
+
`createStandaloneHarness("wasmedge", path, { surface: "direct" })` and by
|
|
234
|
+
`runParityHarness({ surface: "direct" })`. The SDN node uses the same order.
|
|
235
|
+
|
|
236
|
+
Artifacts built with 0.8.20 or earlier have no such export. On their direct
|
|
237
|
+
surface the constructors never run, in every runtime, and a threaded artifact's
|
|
238
|
+
main thread has no pthread descriptor, so a recursive mutex held by the main
|
|
239
|
+
thread does not exclude other threads. Rebuild them with 0.8.21.
|
|
240
|
+
|
|
241
|
+
```sh
|
|
242
|
+
SPACE_DATA_MODULE_SDK_ENABLE_WASMEDGE_PARITY=1 \
|
|
243
|
+
SPACE_DATA_MODULE_SDK_ENABLE_TRI_RUNTIME_PARITY=1 \
|
|
244
|
+
node --test test/direct-call-constructors.test.js
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
### Guest thread faults
|
|
248
|
+
|
|
249
|
+
A guest thread that traps never finishes the pthread exit protocol. WasmEdge
|
|
250
|
+
cancels the whole command. In the browser and Node harnesses the joining thread
|
|
251
|
+
stays blocked inside the guest and cannot run the worker's error event, so the
|
|
252
|
+
worker itself writes `[wasi-thread] guest thread N trapped: ...` to stderr
|
|
253
|
+
(Node) or the console (browser). The call still does not return.
|
|
254
|
+
|
|
255
|
+
V8 (Node 20 to 25) checks bulk memory operations, and every access when the
|
|
256
|
+
WebAssembly trap handler is off (Node on Linux arm64), against a per-instance
|
|
257
|
+
copy of a shared memory's size. That copy is refreshed asynchronously after
|
|
258
|
+
another thread grows the memory, so a thread that writes into memory another
|
|
259
|
+
thread has just grown can trap with "memory access out of bounds", even after
|
|
260
|
+
synchronizing with the growing thread. The guest's allocator uses the heap the
|
|
261
|
+
artifact was linked with and then grows the memory, so a larger imported initial
|
|
262
|
+
memory does not prevent it.
|
|
263
|
+
|
|
219
264
|
The old source path `src/testing/browserModuleHarness.js` remains a pure
|
|
220
265
|
compatibility re-export. New browser consumers should use the public
|
|
221
266
|
`space-data-module-sdk/host/browser-module` entry point.
|
|
@@ -288,6 +333,269 @@ and reports it cannot instantiate the module, a probe timeout, a non-isolated
|
|
|
288
333
|
context, non-shared memory, or a 1-core host all still disable threading and let
|
|
289
334
|
the guest run its proven sequential path (`wasi.thread-spawn` -> `-1`).
|
|
290
335
|
|
|
336
|
+
## 5. FlatSQL partition store host I/O (SDK 0.8.22)
|
|
337
|
+
|
|
338
|
+
The FlatSQL partition store (stack design `docs/architecture/flatsql-partition-store.md`,
|
|
339
|
+
§5.5, §5.6, §18 T9, A7, A36, A38, A39) runs one wasi-threads artifact as a
|
|
340
|
+
writer instance and reader instances, each with guest threads that call
|
|
341
|
+
FlatSQL's seven `env.flatsql_io_*` imports. This section is the SDK half: the
|
|
342
|
+
thread pool, the I/O channel and workers, the Node provider, link shim v2, the
|
|
343
|
+
worker bundles and the browser capability probe.
|
|
344
|
+
|
|
345
|
+
### Explicit pool size, partial spawns, supervision hooks
|
|
346
|
+
|
|
347
|
+
`createWasiThreadSpawn` options added in 0.8.22:
|
|
348
|
+
|
|
349
|
+
| Option | Effect |
|
|
350
|
+
| --- | --- |
|
|
351
|
+
| `poolSize` | Browser: pre-start exactly this many workers, independent of `hardwareConcurrency - 1` (pools are sized for isolation: writers + lanes). Node: cap on live guest threads. Arming is partial: workers that fail to start are dropped, the rest serve. |
|
|
352
|
+
| `extraImports` | Per-worker import objects, as structured-cloneable descriptors (below). Factories run once per worker. |
|
|
353
|
+
| `instanceId`, `onGuestError(instanceId, tid, error)` | Called when a guest thread traps or its worker dies (A36). A dead pooled worker leaves the pool. |
|
|
354
|
+
| `onSpawnDeclined({ reason, poolSize, declined })` | Called for every spawn that returns -1. Reasons: `pool-exhausted`, `pool-not-armed`, `threads-unavailable`, `pool-empty`, `worker-create-failed`, `dispatch-failed`, `hostcall-channel-missing`, `terminated`. |
|
|
355
|
+
| `probeTimeoutMs` | Browser warm-pool probe deadline. |
|
|
356
|
+
| `browserWorkerType: "classic"` | Spawn classic workers, for the blob bundles below. |
|
|
357
|
+
|
|
358
|
+
The returned host adds `spawnReport()`: `{ poolSize, armed, failedToArm, spawned,
|
|
359
|
+
declined, declinedByReason, lastDeclineReason, active, idle }`. The implicit
|
|
360
|
+
(no `poolSize`) path keeps its all-or-nothing arming.
|
|
361
|
+
|
|
362
|
+
`extraImports` entries (the same descriptors work in the engine worker through
|
|
363
|
+
`resolveExtraImports`):
|
|
364
|
+
|
|
365
|
+
| Descriptor | Imports |
|
|
366
|
+
| --- | --- |
|
|
367
|
+
| `{ provider: "flatsql-io", instanceId, channels, mirror?, trace? }` | `env.flatsql_io_*` over one SAB I/O channel per I/O worker. Each worker claims its own request slot. |
|
|
368
|
+
| `{ provider: "flatsql-io-node", root, table, instanceId }` | `env.flatsql_io_*` over synchronous `fs` (Node workers). |
|
|
369
|
+
| `{ moduleUrl, exportName?, config? }` | A factory module (module workers and Node only). |
|
|
370
|
+
|
|
371
|
+
A module that bundles this host into an IIFE or a blob: module worker has no
|
|
372
|
+
usable `import.meta.url`. `DEFAULT_BROWSER_WORKER_URL` is then `null` instead of
|
|
373
|
+
a module-evaluation `TypeError`, and such hosts pass `browserWorkerUrl`.
|
|
374
|
+
|
|
375
|
+
### Flags and statuses
|
|
376
|
+
|
|
377
|
+
The partition store adds three open flags and one status. They are flags, so the
|
|
378
|
+
import set stays at seven. flatsql's `flatsql_io.h` (T1) must carry the same
|
|
379
|
+
values.
|
|
380
|
+
|
|
381
|
+
| Name | Value | Meaning |
|
|
382
|
+
| --- | --- | --- |
|
|
383
|
+
| `FLATSQL_IO_CREATE_PARENTS` | `0x0100` | mkdir -p; sync each new directory's parent, and the parent of a new file. Best effort on OPFS. |
|
|
384
|
+
| `FLATSQL_IO_UNLINK_IF_UNUSED` | `0x0200` | Unlink, or `BUSY` while any handle names the path. |
|
|
385
|
+
| `FLATSQL_IO_OPEN_DEFERRED` | `0x0400` | Return a handle before the open finishes; the first use waits, or fails with the open's status. Synchronous hosts treat it as a plain open. |
|
|
386
|
+
| `FLATSQL_IO_ERR_BUSY` | `-7` | Path in use, or its lock held elsewhere. |
|
|
387
|
+
|
|
388
|
+
Every SDK host maps `EEXIST` to `GENERIC` (the Go host's mapping), a write on a
|
|
389
|
+
read-only handle and a read on a write-only handle to `IO`, and `..` to
|
|
390
|
+
`ACCESS`. OPFS errors map as NotFound -> `NOENT`, NoModificationAllowed ->
|
|
391
|
+
`BUSY`, QuotaExceeded -> `NOSPACE`; WebKit refuses a second sync handle with
|
|
392
|
+
InvalidStateError (measured), which maps to `BUSY` during an open.
|
|
393
|
+
|
|
394
|
+
### The SAB I/O channel and the I/O worker
|
|
395
|
+
|
|
396
|
+
`sabIoChannel.js` extends `sabHostcallChannel.js` to many guest threads and one
|
|
397
|
+
server:
|
|
398
|
+
|
|
399
|
+
- A request ring of slots in one SharedArrayBuffer. Each request owns a slot
|
|
400
|
+
until its result is read, so a guest never queues behind another guest's
|
|
401
|
+
request and a slow open blocks only its caller. Idle threads hold no slot, so
|
|
402
|
+
terminating a pool leaks none; `reclaimSabIoSlots(buffer, instanceId)` frees
|
|
403
|
+
the slots of requests that were in flight when an instance's threads died.
|
|
404
|
+
- Doorbell: `Atomics.waitAsync` on a header word. The server publishes the mode;
|
|
405
|
+
in message mode (`doorbell: "message"`, or no `waitAsync`) clients also post
|
|
406
|
+
on a BroadcastChannel. Completions for non-blocking callers without
|
|
407
|
+
`waitAsync` arrive the same way (22.3a-5).
|
|
408
|
+
- No timeout, no throw (A36). Blocking waits run in 250 ms slices forever. A
|
|
409
|
+
supervisor that sees the I/O worker die calls `failPendingSabIoRequests`,
|
|
410
|
+
which completes every pending slot with `IO`.
|
|
411
|
+
- Data moves directly between the instance's shared memory and the file when the
|
|
412
|
+
instance's memory is attached to the I/O worker, else through the slot's data
|
|
413
|
+
area. Paths are always copied out of shared memory before decoding.
|
|
414
|
+
- Revocation (A36): `revokeSabIoInstance(buffer, id)` resolves after the server
|
|
415
|
+
has closed the instance's handles; later requests get `ACCESS` and no byte of
|
|
416
|
+
the instance is written after it resolves.
|
|
417
|
+
|
|
418
|
+
`opfsIoWorker.mjs` runs `flatsqlIoServer.js` over a backend:
|
|
419
|
+
|
|
420
|
+
- `opfs`: every open is async (`getDirectoryHandle`, `getFileHandle`,
|
|
421
|
+
`createSyncAccessHandle`) and awaited on the worker's event loop while the
|
|
422
|
+
requester waits on its slot. One sync handle per path, shared by all virtual
|
|
423
|
+
handles. Handles open in `"readwrite-unsafe"` mode where it exists, so a
|
|
424
|
+
reader I/O worker can read files a writer I/O worker holds (A7). Views over a
|
|
425
|
+
SharedArrayBuffer go straight to `read`/`write`; a user agent that rejects
|
|
426
|
+
them gets a private scratch copy.
|
|
427
|
+
- `memory`: the dashboard window store (§5.5). Nothing reaches OPFS;
|
|
428
|
+
`reset()` drops it; `memoryMaxBytes` is the per-tab budget (`NOSPACE`).
|
|
429
|
+
|
|
430
|
+
Roles (A7): one writer I/O worker per writer holds the writer's active files;
|
|
431
|
+
reader I/O workers (one, or two when `hardwareConcurrency >= 8`) hold sealed
|
|
432
|
+
files. Reads and writes run in steps of at most 256 KiB with the slots rescanned
|
|
433
|
+
between steps, so a small read never waits for a whole large write. With
|
|
434
|
+
several reader I/O workers, `createFlatsqlIoImports` routes each open by a path
|
|
435
|
+
hash and every later call to the same worker (the worker index rides in handle
|
|
436
|
+
bits 24-30).
|
|
437
|
+
|
|
438
|
+
`createFlatsqlIoWorker(options)` spawns and supervises one I/O worker:
|
|
439
|
+
`attachMemory`, `revoke`, `preopen(paths)` (A38: open the registry's active
|
|
440
|
+
files in parallel at start; later guest opens of those paths return at once),
|
|
441
|
+
`releasePreopen`, `stats`, `reset`, `clear`, `stop`, and `onError` plus
|
|
442
|
+
`restart: true` (A36: pending requests fail with `IO`, a replacement starts on
|
|
443
|
+
the same channel with the live memories re-attached; handles do not survive).
|
|
444
|
+
|
|
445
|
+
Store lock (A37): with `lock: { name }` the I/O worker takes that Web Lock
|
|
446
|
+
before it opens any handle and releases it only after `stop()` has closed them,
|
|
447
|
+
so the lock lives exactly as long as the handles. `ifAvailable: true` fails the
|
|
448
|
+
start with `lockUnavailable` instead of waiting; `busyRetry` retries a handle a
|
|
449
|
+
previous leader still holds, with backoff. Measured takeover (holder `stop()` to
|
|
450
|
+
successor ready): 5 ms Chromium, 52 ms Firefox, 5 ms WebKit. Leadership,
|
|
451
|
+
follower proxying and heartbeats are the engine's (sdn-js, T10).
|
|
452
|
+
|
|
453
|
+
Head mirror (A7): with `mirror: { buffer, suffixes: ["/h.fsh"] }`, the writer
|
|
454
|
+
I/O worker copies every write to a matching path into a seqlock mirror in a
|
|
455
|
+
SharedArrayBuffer (`sabIoMirror.js`), and reader imports configured with the
|
|
456
|
+
same mirror serve reads of those paths from it without a round trip.
|
|
457
|
+
|
|
458
|
+
### Node synchronous fs (§5.6)
|
|
459
|
+
|
|
460
|
+
`nodeSyncFsIo.js`: synchronous `fs` in each worker over a shared virtual-handle
|
|
461
|
+
table in a SharedArrayBuffer (`createNodeSyncFsIoTable`). A handle is
|
|
462
|
+
`(slot << 8) | gen`; each worker opens its own fd for a slot on first use and
|
|
463
|
+
drops stale fds when the generation moves. Paths are confined below `root`,
|
|
464
|
+
including through symlinked parents. `sync` is `fdatasync` (libuv issues
|
|
465
|
+
`F_FULLFSYNC` on darwin). `revokeNodeSyncFsIoInstance` follows A23: it sets the
|
|
466
|
+
revoked flag and waits for the instance's in-flight calls to drain. Fault
|
|
467
|
+
injection (§19, 22.3a-7) belongs to FlatSQL's Node host (T4); `interpose` wraps
|
|
468
|
+
every syscall for it.
|
|
469
|
+
|
|
470
|
+
### Link shim v2
|
|
471
|
+
|
|
472
|
+
`FLATSQL_LINK_SHIM_V2_WASM` (`src/flow/flatsqlLinkShim.js`) is a deterministic
|
|
473
|
+
module that imports the reader instance's SHARED lane memory as
|
|
474
|
+
`flatsql.memory` and gives a linked flow the lane mailbox as direct calls. v1 is
|
|
475
|
+
unchanged (sha256 `8d83e69b…`), because today's linked flows use it. v2 sha256:
|
|
476
|
+
`67d5b2d9bc2d1b346a14a253a586fd4d08c8056d54eb62701b48000585ed9613`.
|
|
477
|
+
|
|
478
|
+
| Export | Result |
|
|
479
|
+
| --- | --- |
|
|
480
|
+
| `mb_submit(mailbox, op, req_ptr, req_len)` | `seq`, or -1 when the mailbox holds a request |
|
|
481
|
+
| `mb_poll(mailbox, seq)` | 1 when complete |
|
|
482
|
+
| `mb_wait(mailbox, seq, poll_ns: i64, max_polls)` | the lane's status, or -110 after `max_polls` bounded waits (`max_polls <= 0`: no limit) |
|
|
483
|
+
| `mb_release(mailbox, seq)` | 0, or -1 when not complete |
|
|
484
|
+
| `mb_cancel(mailbox, seq)` | 0; the lane polls the cancel word |
|
|
485
|
+
| `load32_acquire`, `store32_release`, `peek8/32/64`, `poke8/32`, `fnv1a64`, `count_frames` | as in v1, over lane memory |
|
|
486
|
+
|
|
487
|
+
Mailbox, 64 bytes, 8-aligned, little-endian: `+0 state` (IDLE 0, SUBMITTED 1,
|
|
488
|
+
CLAIMED 2, DONE 3, SUBMITTING 4), `+4 seq`, `+8 done_seq`, `+12 op`,
|
|
489
|
+
`+16 req_ptr`, `+20 req_len`, `+24 status`, `+28 resp_ptr`, `+32 resp_len`,
|
|
490
|
+
`+36 doorbell`, `+40 cancel`, `+44 flags`, `+48 generation (u64)`. A lane waits
|
|
491
|
+
(bounded) on the doorbell, CASes SUBMITTED -> CLAIMED, writes the result,
|
|
492
|
+
stores `done_seq = seq`, stores DONE and notifies the state word.
|
|
493
|
+
|
|
494
|
+
`mb_wait` never waits unboundedly: each `memory.atomic.wait32` lasts `poll_ns`
|
|
495
|
+
and every wake re-reads the mailbox. WasmEdge keeps waiters per executor, so a
|
|
496
|
+
lane's notify may never reach a flow on another executor; a lost notify then
|
|
497
|
+
costs one poll interval and never a completion. `poll_ns = 0` polls without
|
|
498
|
+
executing a wait (contexts that may not block).
|
|
499
|
+
|
|
500
|
+
### Worker bundles (A39)
|
|
501
|
+
|
|
502
|
+
The dashboard is one HTML file under `worker-src 'self' blob:`.
|
|
503
|
+
`space-data-module-sdk/host/worker-bundles` ships the pool worker and the I/O
|
|
504
|
+
worker as self-contained classic scripts (esbuild IIFE, built by
|
|
505
|
+
`scripts/build-host-worker-bundles.mjs` into `src/host/hostWorkerBundleSources.js`):
|
|
506
|
+
|
|
507
|
+
```js
|
|
508
|
+
import { hostWorkerBundleUrl } from "space-data-module-sdk/host/worker-bundles";
|
|
509
|
+
import { createWasiThreadSpawn } from "space-data-module-sdk/host/wasi-threads";
|
|
510
|
+
import { createFlatsqlIoWorker } from "space-data-module-sdk/host/flatsql-io";
|
|
511
|
+
|
|
512
|
+
const writerIo = await createFlatsqlIoWorker({
|
|
513
|
+
workerUrl: hostWorkerBundleUrl("flatsql-io"), workerType: "classic",
|
|
514
|
+
backend: "opfs", role: "writer", rootDirectory: "sdn-store",
|
|
515
|
+
});
|
|
516
|
+
await writerIo.attachMemory(1, writerMemory);
|
|
517
|
+
const pool = await createWasiThreadSpawn({
|
|
518
|
+
wasmModule, memory: writerMemory, poolSize: 1 + 2, instanceId: 1,
|
|
519
|
+
enableBrowserThreads: true,
|
|
520
|
+
browserWorkerUrl: hostWorkerBundleUrl("wasi-thread-pool"), browserWorkerType: "classic",
|
|
521
|
+
extraImports: [{ provider: "flatsql-io", instanceId: 1, channels: [writerIo.buffer] }],
|
|
522
|
+
onGuestError: (instanceId, tid, error) => supervisor.poison(instanceId, tid, error),
|
|
523
|
+
});
|
|
524
|
+
```
|
|
525
|
+
|
|
526
|
+
### Browser capability matrix
|
|
527
|
+
|
|
528
|
+
`probeBrowserCapabilities()` probes the page and a blob worker;
|
|
529
|
+
`flatsqlLocalStoreGate(matrix)` is the store gate: cross-origin isolation,
|
|
530
|
+
shared memory, OPFS sync handles in a worker, `Atomics.wait` in workers, and
|
|
531
|
+
shared sync-handle modes (§22.4-6: without them there is no local store).
|
|
532
|
+
Measured 2026-09-27 on the owner's Mac Studio (Apple M3 Ultra, macOS 26.3.1),
|
|
533
|
+
headless, Playwright 1.63.0, under the dashboard CSP with COOP/COEP:
|
|
534
|
+
|
|
535
|
+
| | Chromium 153.0.8010.12 | Firefox 155.0 | WebKit 26.6 |
|
|
536
|
+
| --- | --- | --- | --- |
|
|
537
|
+
| `crossOriginIsolated`, shared wasm memory | yes | yes | yes |
|
|
538
|
+
| `Atomics.waitAsync` (page and worker) | yes | yes | yes |
|
|
539
|
+
| OPFS sync access handle in a worker | yes | yes | yes (persistent profile only) |
|
|
540
|
+
| Second default handle on one file | NoModificationAllowedError | NoModificationAllowedError | InvalidStateError |
|
|
541
|
+
| Shared modes (`readwrite-unsafe` x 2) | yes | no (mode ignored) | no (mode ignored) |
|
|
542
|
+
| SharedArrayBuffer views in `read`/`write` | yes | yes | yes |
|
|
543
|
+
| Wasm shared-memory views in `read` | yes | yes | yes |
|
|
544
|
+
| `removeEntry` of an open file | NoModificationAllowedError | NoModificationAllowedError | NoModificationAllowedError |
|
|
545
|
+
| Nested blob workers | yes | yes | yes |
|
|
546
|
+
| `performance.now()` resolution | 5 µs | 20 µs | 20 µs |
|
|
547
|
+
| Local-store gate | supported | no (shared modes) | no (shared modes) |
|
|
548
|
+
|
|
549
|
+
WebKit refuses OPFS in Playwright's ephemeral context (`UnknownError`); the suite
|
|
550
|
+
uses a persistent profile.
|
|
551
|
+
|
|
552
|
+
### Conformance
|
|
553
|
+
|
|
554
|
+
`runFlatsqlIoConformance(io)` (`flatsqlIoConformance.js`) is one script for every
|
|
555
|
+
host (22.3a-6): 15 cases covering statuses, short reads, sparse writes,
|
|
556
|
+
truncation, EXCL/TRUNC/PROBE/UNLINK/UNLINK_IF_UNUSED/CREATE_PARENTS/
|
|
557
|
+
DELETE_ON_CLOSE/OPEN_DEFERRED, access modes, confinement and multi-handle
|
|
558
|
+
visibility. It passes on the Node sync-fs provider, on the channel over the
|
|
559
|
+
memory backend (blocking and async clients, both doorbells), on the blob bundle
|
|
560
|
+
run as a standalone script, and on OPFS and memory in Chromium, Firefox and
|
|
561
|
+
WebKit. Plain `UNLINK` of an open path is `BUSY` in the I/O worker (OPFS cannot
|
|
562
|
+
remove a file with an open sync handle) and succeeds on POSIX hosts; the script
|
|
563
|
+
does not test it.
|
|
564
|
+
|
|
565
|
+
### Measured acceptance (§18 T9, A7, A38)
|
|
566
|
+
|
|
567
|
+
Owner's Mac Studio (Apple M3 Ultra, 28 cores, macOS 26.3.1), 2026-09-27,
|
|
568
|
+
headless, final full run of `test/opfs-io-worker.browser.test.js`. The machine
|
|
569
|
+
was shared with other lanes (load average 23-33 on 28 cores). Latencies are
|
|
570
|
+
guest-observed import calls (`trace`), in microseconds.
|
|
571
|
+
|
|
572
|
+
| Item | Chromium 153 | Firefox 155 | WebKit 26.6 |
|
|
573
|
+
| --- | --- | --- | --- |
|
|
574
|
+
| #1 8 threads, mixed I/O + 200 async opens + 50 unlinks through 1 I/O worker: errors / lost writes | 0 / 0 | 0 / 0 | 0 / 0 |
|
|
575
|
+
| same run, 4 KiB read p50 / p99 | 50 / 2145 | 260 / 8180 | 60 / 2660 |
|
|
576
|
+
| same run, message doorbell: errors / lost writes | 0 / 0 | 0 / 0 | 0 / 0 |
|
|
577
|
+
| A7 lane 4 KiB read p99 during 100 x 4 MiB write+flush, same partition | 790 (active file, shared mode) | 120 (sealed segment) | 40 (sealed segment) |
|
|
578
|
+
| A7 same, other partition | 45 | 120 | 40 |
|
|
579
|
+
| A7 writer flush p50 / p99 (4 MiB) | 6365 / 28945 | 3540 / 27740 | 5260 / 70700 |
|
|
580
|
+
| #2 500 ms open, median of 3 runs: other threads' read p99, baseline / during | 85 / 85 | 440 / 400 | 60 / 60 |
|
|
581
|
+
| #2 longest other-thread read while the open was pending | 1020 | 18500 | 320 |
|
|
582
|
+
| A38 OPEN_DEFERRED: open returns in / first write waits (ms) | 3.8 / 504 | 28.8 / 511 | 5.2 / 755 |
|
|
583
|
+
| #3 `poolSize=6` on `hardwareConcurrency=2` | 6 workers; 7th spawn -1, reported `pool-exhausted` | same | same |
|
|
584
|
+
| #5 link shim v2 (Node 25): 10,000 calls, lost completions | 0 with a notifying lane, 0 with a lane that never notifies, 0 spinning (`poll_ns = 0`) | | |
|
|
585
|
+
|
|
586
|
+
Chromium's same-partition A7 p99 varies with host load: 90, 110, 295 and 790
|
|
587
|
+
over four runs with the adaptive spin (below), and 125-1165 over four runs
|
|
588
|
+
before it. The original #1 target (4 KiB read p99 <= 200 µs under the mixed
|
|
589
|
+
load) is not met through one I/O worker; A7 replaced #1 with the <= 1 ms lane
|
|
590
|
+
target, which is met. Firefox and WebKit have no shared handle modes, so their
|
|
591
|
+
lanes read a partition's sealed segment; per §22.4-6 those browsers get no
|
|
592
|
+
local store.
|
|
593
|
+
|
|
594
|
+
Adaptive waits: an idle I/O worker polls its doorbell for 50 µs, and a blocking
|
|
595
|
+
client polls for its answer for 20 µs, before sleeping (`spinMicros`), so
|
|
596
|
+
back-to-back requests skip a thread wake-up. In the Node channel test this took
|
|
597
|
+
the fast-thread read p50 from 30-39 µs to 11-12 µs.
|
|
598
|
+
|
|
291
599
|
## Tests
|
|
292
600
|
|
|
293
601
|
- `test/wasi-thread-bundled-consumer-anchor.test.js` — the **bundled-consumer
|
|
@@ -315,6 +623,28 @@ the guest run its proven sequential path (`wasi.thread-spawn` -> `-1`).
|
|
|
315
623
|
`symbolPrefix` / `methodSymbols` as authoritative (never re-derived), keeping a
|
|
316
624
|
legacy truncated-prefix artifact compatible.
|
|
317
625
|
|
|
626
|
+
- `test/wasi-thread-pool-size.test.js` — explicit `poolSize` (6 workers on
|
|
627
|
+
`hardwareConcurrency=2`, the 7th spawn -1 and reported), partial arming,
|
|
628
|
+
extraImports delivery, `onGuestError`, classic workers.
|
|
629
|
+
- `test/sab-io-channel.test.js` — the I/O channel under a real wasi-threads
|
|
630
|
+
guest (`test/support/flatsql-io/ioGuestWasm.mjs`): 8 threads of mixed I/O in
|
|
631
|
+
both doorbell modes, a 500 ms open blocking only its caller, OPEN_DEFERRED,
|
|
632
|
+
revocation, a dead I/O worker, supervisor restart, 256 KiB steps, the head
|
|
633
|
+
mirror, scratch vs direct transfers, pre-open, and path-hash routing.
|
|
634
|
+
- `test/node-sync-fs-io.test.js` — shared handles across workers, stale handles,
|
|
635
|
+
CREATE_PARENTS, UNLINK_IF_UNUSED, confinement, A23 revocation.
|
|
636
|
+
- `test/flatsql-io-conformance.test.js` — the conformance script on every Node host.
|
|
637
|
+
- `test/host-worker-bundles.test.js` — the bundles equal a fresh build, and the
|
|
638
|
+
I/O worker bundle passes the conformance script standalone.
|
|
639
|
+
- `test/link-shim-v2.test.js` — shim v2 bytes, contract, and 10,000 calls
|
|
640
|
+
against a stub lane with and without notify.
|
|
641
|
+
- `test/opfs-io-worker.browser.test.js` — the real-browser suite above
|
|
642
|
+
(env-gated):
|
|
643
|
+
|
|
644
|
+
```sh
|
|
645
|
+
SPACE_DATA_MODULE_SDK_ENABLE_BROWSER_IO=1 node --test test/opfs-io-worker.browser.test.js
|
|
646
|
+
```
|
|
647
|
+
|
|
318
648
|
## See also
|
|
319
649
|
|
|
320
650
|
- [`docs/browser-wasmedge-isomorphic.md`](./browser-wasmedge-isomorphic.md) —
|
|
@@ -326,5 +656,10 @@ the guest run its proven sequential path (`wasi.thread-spawn` -> `-1`).
|
|
|
326
656
|
`WasiThreadWorkerUnreachableError`.
|
|
327
657
|
- `src/compiler/pthreadArtifactGuard.js` — the flag list + wasm validator.
|
|
328
658
|
- `src/compiler/wasiThreadsToolchain.js` — the wasi-threads toolchain resolver.
|
|
659
|
+
- `src/host/sabIoChannel.js`, `src/host/flatsqlIoServer.js`,
|
|
660
|
+
`src/host/opfsIoWorker.mjs`, `src/host/flatsqlIoWorkers.js`,
|
|
661
|
+
`src/host/nodeSyncFsIo.js`, `src/host/sabIoMirror.js`,
|
|
662
|
+
`src/host/flatsqlIoConformance.js`, `src/host/browserCapabilityProbe.js`,
|
|
663
|
+
`src/host/hostWorkerBundles.js` — the FlatSQL partition store host I/O (§5).
|
|
329
664
|
- `src/compiler/compileModule.js` — `ModuleThreadModel`, `buildCompilerArgs`,
|
|
330
665
|
`compileWithWasiThreads`, `resolveThreadModel`, `compileModuleFromSource`.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "space-data-module-sdk",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.22",
|
|
4
4
|
"license": "Apache-2.0",
|
|
5
5
|
"description": "Module SDK for building, validating, signing, and deploying WebAssembly modules on the Space Data Network.",
|
|
6
6
|
"type": "module",
|
|
@@ -62,6 +62,10 @@
|
|
|
62
62
|
"./host/runtime-target-gate": "./src/host/runtimeTargetGate.js",
|
|
63
63
|
"./host/node-builtin": "./src/host/nodeBuiltinSpecifier.js",
|
|
64
64
|
"./host/worker-module": "./src/host/workerModuleHarness.js",
|
|
65
|
+
"./host/wasi-threads": "./src/host/wasiThreadHost.js",
|
|
66
|
+
"./host/flatsql-io": "./src/host/flatsqlIo.js",
|
|
67
|
+
"./host/flatsql-io/node": "./src/host/nodeSyncFsIo.js",
|
|
68
|
+
"./host/worker-bundles": "./src/host/hostWorkerBundles.js",
|
|
65
69
|
"./testing/browser": "./src/testing/browser.js",
|
|
66
70
|
"./testing/module-flatbuffer-stream-pump": {
|
|
67
71
|
"types": "./src/index.d.ts",
|
|
@@ -129,7 +133,8 @@
|
|
|
129
133
|
},
|
|
130
134
|
"devDependencies": {
|
|
131
135
|
"esbuild": "^0.28.0",
|
|
132
|
-
"express": "^4.21.2"
|
|
136
|
+
"express": "^4.21.2",
|
|
137
|
+
"playwright-core": "1.63.0"
|
|
133
138
|
},
|
|
134
139
|
"engines": {
|
|
135
140
|
"node": ">=20.0.0"
|
package/src/browser.js
CHANGED
|
@@ -41,7 +41,13 @@ export {
|
|
|
41
41
|
resolveBrowserWorkerUrl,
|
|
42
42
|
DEFAULT_BROWSER_WORKER_URL,
|
|
43
43
|
WasiThreadWorkerUnreachableError,
|
|
44
|
+
createWasiThreadSpawn,
|
|
45
|
+
isWasiThreadsModule,
|
|
44
46
|
} from "./host/wasiThreadHost.js";
|
|
47
|
+
// FlatSQL partition store host I/O (T9): the SAB I/O channel, the I/O worker
|
|
48
|
+
// controller, OPFS and memory backends, and the capability probe. The worker
|
|
49
|
+
// bundles for blob: spawning are on the "./host/worker-bundles" subpath.
|
|
50
|
+
export * from "./host/flatsqlIo.js";
|
|
45
51
|
// The runtime-target gate is reachable so a consumer can catch the refusal by
|
|
46
52
|
// CLASS (`error instanceof RuntimeTargetError`) rather than by matching a
|
|
47
53
|
// message string, and can ask the same question the loaders ask before it
|