space-data-module-sdk 0.8.21 → 0.8.22
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/isomorphic-pthreads.html +88 -0
- package/docs/isomorphic-pthreads.md +290 -0
- package/package.json +7 -2
- package/src/browser.js +6 -0
- package/src/flow/flatsqlLinkShim.js +295 -0
- package/src/flow/index.d.ts +31 -0
- package/src/flow/index.js +8 -0
- package/src/host/browserCapabilityProbe.js +251 -0
- package/src/host/flatsqlIo.js +14 -0
- package/src/host/flatsqlIoConformance.js +349 -0
- package/src/host/flatsqlIoContract.js +240 -0
- package/src/host/flatsqlIoImports.js +335 -0
- package/src/host/flatsqlIoMemoryBackend.js +178 -0
- package/src/host/flatsqlIoServer.js +960 -0
- package/src/host/flatsqlIoWorkers.js +323 -0
- package/src/host/hostWorkerBundleSources.js +11 -0
- package/src/host/hostWorkerBundles.js +79 -0
- package/src/host/index.js +9 -0
- package/src/host/nodeSyncFsIo.js +633 -0
- package/src/host/opfsIoBackend.js +280 -0
- package/src/host/opfsIoWorker.mjs +223 -0
- package/src/host/sabIoChannel.js +848 -0
- package/src/host/sabIoMirror.js +306 -0
- package/src/host/wasiThreadBrowserWorker.mjs +18 -11
- package/src/host/wasiThreadHost.js +258 -41
- package/src/host/wasiThreadWorker.mjs +24 -8
- package/src/host/wasiThreadWorkerRuntime.js +54 -0
- package/src/index.d.ts +347 -0
|
@@ -184,6 +184,73 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
184
184
|
<li><strong>An unreachable worker fails LOUD.</strong> When the pooled path was requested (threads enabled + cross-origin isolated + shared memory) and the worker asset at the resolved anchor never loads, <code>createWasiThreadSpawn</code> <strong>throws</strong> <code>WasiThreadWorkerUnreachableError</code> naming the URL it tried. It does not drop quietly to one thread — a silent sequential fallback is how a deployment defect hid behind a passing gate as a pure performance loss.</li>
|
|
185
185
|
</ul>
|
|
186
186
|
<p>Genuine capability negotiation is unaffected and stays soft: a worker that loads and reports it cannot instantiate the module, a probe timeout, a non-isolated context, non-shared memory, or a 1-core host all still disable threading and let the guest run its proven sequential path (<code>wasi.thread-spawn</code> -> <code>-1</code>).</p>
|
|
187
|
+
<h2 id="5-flatsql-partition-store-host-io-sdk-0822"><a class="anchor" href="#5-flatsql-partition-store-host-io-sdk-0822" aria-hidden="true">#</a>5. FlatSQL partition store host I/O (SDK 0.8.22)</h2>
|
|
188
|
+
<p>The FlatSQL partition store (stack design <code>docs/architecture/flatsql-partition-store.md</code>, §5.5, §5.6, §18 T9, A7, A36, A38, A39) runs one wasi-threads artifact as a writer instance and reader instances, each with guest threads that call FlatSQL's seven <code>env.flatsql_io_*</code> imports. This section is the SDK half: the thread pool, the I/O channel and workers, the Node provider, link shim v2, the worker bundles and the browser capability probe.</p>
|
|
189
|
+
<h3 id="explicit-pool-size-partial-spawns-supervision-hooks"><a class="anchor" href="#explicit-pool-size-partial-spawns-supervision-hooks" aria-hidden="true">#</a>Explicit pool size, partial spawns, supervision hooks</h3>
|
|
190
|
+
<p><code>createWasiThreadSpawn</code> options added in 0.8.22:</p>
|
|
191
|
+
<div class="table-wrap"><table><thead><tr><th>Option</th><th>Effect</th></tr></thead><tbody><tr><td><code>poolSize</code></td><td>Browser: pre-start exactly this many workers, independent of <code>hardwareConcurrency - 1</code> (pools are sized for isolation: writers + lanes). Node: cap on live guest threads. Arming is partial: workers that fail to start are dropped, the rest serve.</td></tr><tr><td><code>extraImports</code></td><td>Per-worker import objects, as structured-cloneable descriptors (below). Factories run once per worker.</td></tr><tr><td><code>instanceId</code>, <code>onGuestError(instanceId, tid, error)</code></td><td>Called when a guest thread traps or its worker dies (A36). A dead pooled worker leaves the pool.</td></tr><tr><td><code>onSpawnDeclined({ reason, poolSize, declined })</code></td><td>Called for every spawn that returns -1. Reasons: <code>pool-exhausted</code>, <code>pool-not-armed</code>, <code>threads-unavailable</code>, <code>pool-empty</code>, <code>worker-create-failed</code>, <code>dispatch-failed</code>, <code>hostcall-channel-missing</code>, <code>terminated</code>.</td></tr><tr><td><code>probeTimeoutMs</code></td><td>Browser warm-pool probe deadline.</td></tr><tr><td><code>browserWorkerType: "classic"</code></td><td>Spawn classic workers, for the blob bundles below.</td></tr></tbody></table></div>
|
|
192
|
+
<p>The returned host adds <code>spawnReport()</code>: <code>{ poolSize, armed, failedToArm, spawned, declined, declinedByReason, lastDeclineReason, active, idle }</code>. The implicit (no <code>poolSize</code>) path keeps its all-or-nothing arming.</p>
|
|
193
|
+
<p><code>extraImports</code> entries (the same descriptors work in the engine worker through <code>resolveExtraImports</code>):</p>
|
|
194
|
+
<div class="table-wrap"><table><thead><tr><th>Descriptor</th><th>Imports</th></tr></thead><tbody><tr><td><code>{ provider: "flatsql-io", instanceId, channels, mirror?, trace? }</code></td><td><code>env.flatsql_io_*</code> over one SAB I/O channel per I/O worker. Each worker claims its own request slot.</td></tr><tr><td><code>{ provider: "flatsql-io-node", root, table, instanceId }</code></td><td><code>env.flatsql_io_*</code> over synchronous <code>fs</code> (Node workers).</td></tr><tr><td><code>{ moduleUrl, exportName?, config? }</code></td><td>A factory module (module workers and Node only).</td></tr></tbody></table></div>
|
|
195
|
+
<p>A module that bundles this host into an IIFE or a blob: module worker has no usable <code>import.meta.url</code>. <code>DEFAULT_BROWSER_WORKER_URL</code> is then <code>null</code> instead of a module-evaluation <code>TypeError</code>, and such hosts pass <code>browserWorkerUrl</code>.</p>
|
|
196
|
+
<h3 id="flags-and-statuses"><a class="anchor" href="#flags-and-statuses" aria-hidden="true">#</a>Flags and statuses</h3>
|
|
197
|
+
<p>The partition store adds three open flags and one status. They are flags, so the import set stays at seven. flatsql's <code>flatsql_io.h</code> (T1) must carry the same values.</p>
|
|
198
|
+
<div class="table-wrap"><table><thead><tr><th>Name</th><th>Value</th><th>Meaning</th></tr></thead><tbody><tr><td><code>FLATSQL_IO_CREATE_PARENTS</code></td><td><code>0x0100</code></td><td>mkdir -p; sync each new directory's parent, and the parent of a new file. Best effort on OPFS.</td></tr><tr><td><code>FLATSQL_IO_UNLINK_IF_UNUSED</code></td><td><code>0x0200</code></td><td>Unlink, or <code>BUSY</code> while any handle names the path.</td></tr><tr><td><code>FLATSQL_IO_OPEN_DEFERRED</code></td><td><code>0x0400</code></td><td>Return a handle before the open finishes; the first use waits, or fails with the open's status. Synchronous hosts treat it as a plain open.</td></tr><tr><td><code>FLATSQL_IO_ERR_BUSY</code></td><td><code>-7</code></td><td>Path in use, or its lock held elsewhere.</td></tr></tbody></table></div>
|
|
199
|
+
<p>Every SDK host maps <code>EEXIST</code> to <code>GENERIC</code> (the Go host's mapping), a write on a read-only handle and a read on a write-only handle to <code>IO</code>, and <code>..</code> to <code>ACCESS</code>. OPFS errors map as NotFound -> <code>NOENT</code>, NoModificationAllowed -> <code>BUSY</code>, QuotaExceeded -> <code>NOSPACE</code>; WebKit refuses a second sync handle with InvalidStateError (measured), which maps to <code>BUSY</code> during an open.</p>
|
|
200
|
+
<h3 id="the-sab-io-channel-and-the-io-worker"><a class="anchor" href="#the-sab-io-channel-and-the-io-worker" aria-hidden="true">#</a>The SAB I/O channel and the I/O worker</h3>
|
|
201
|
+
<p><code>sabIoChannel.js</code> extends <code>sabHostcallChannel.js</code> to many guest threads and one server:</p>
|
|
202
|
+
<ul>
|
|
203
|
+
<li>A request ring of slots in one SharedArrayBuffer. Each request owns a slot until its result is read, so a guest never queues behind another guest's request and a slow open blocks only its caller. Idle threads hold no slot, so terminating a pool leaks none; <code>reclaimSabIoSlots(buffer, instanceId)</code> frees the slots of requests that were in flight when an instance's threads died.</li>
|
|
204
|
+
<li>Doorbell: <code>Atomics.waitAsync</code> on a header word. The server publishes the mode; in message mode (<code>doorbell: "message"</code>, or no <code>waitAsync</code>) clients also post on a BroadcastChannel. Completions for non-blocking callers without <code>waitAsync</code> arrive the same way (22.3a-5).</li>
|
|
205
|
+
<li>No timeout, no throw (A36). Blocking waits run in 250 ms slices forever. A supervisor that sees the I/O worker die calls <code>failPendingSabIoRequests</code>, which completes every pending slot with <code>IO</code>.</li>
|
|
206
|
+
<li>Data moves directly between the instance's shared memory and the file when the instance's memory is attached to the I/O worker, else through the slot's data area. Paths are always copied out of shared memory before decoding.</li>
|
|
207
|
+
<li>Revocation (A36): <code>revokeSabIoInstance(buffer, id)</code> resolves after the server has closed the instance's handles; later requests get <code>ACCESS</code> and no byte of the instance is written after it resolves.</li>
|
|
208
|
+
</ul>
|
|
209
|
+
<p><code>opfsIoWorker.mjs</code> runs <code>flatsqlIoServer.js</code> over a backend:</p>
|
|
210
|
+
<ul>
|
|
211
|
+
<li><code>opfs</code>: every open is async (<code>getDirectoryHandle</code>, <code>getFileHandle</code>, <code>createSyncAccessHandle</code>) and awaited on the worker's event loop while the requester waits on its slot. One sync handle per path, shared by all virtual handles. Handles open in <code>"readwrite-unsafe"</code> mode where it exists, so a reader I/O worker can read files a writer I/O worker holds (A7). Views over a SharedArrayBuffer go straight to <code>read</code>/<code>write</code>; a user agent that rejects them gets a private scratch copy.</li>
|
|
212
|
+
<li><code>memory</code>: the dashboard window store (§5.5). Nothing reaches OPFS; <code>reset()</code> drops it; <code>memoryMaxBytes</code> is the per-tab budget (<code>NOSPACE</code>).</li>
|
|
213
|
+
</ul>
|
|
214
|
+
<p>Roles (A7): one writer I/O worker per writer holds the writer's active files; reader I/O workers (one, or two when <code>hardwareConcurrency >= 8</code>) hold sealed files. Reads and writes run in steps of at most 256 KiB with the slots rescanned between steps, so a small read never waits for a whole large write. With several reader I/O workers, <code>createFlatsqlIoImports</code> routes each open by a path hash and every later call to the same worker (the worker index rides in handle bits 24-30).</p>
|
|
215
|
+
<p><code>createFlatsqlIoWorker(options)</code> spawns and supervises one I/O worker: <code>attachMemory</code>, <code>revoke</code>, <code>preopen(paths)</code> (A38: open the registry's active files in parallel at start; later guest opens of those paths return at once), <code>releasePreopen</code>, <code>stats</code>, <code>reset</code>, <code>clear</code>, <code>stop</code>, and <code>onError</code> plus <code>restart: true</code> (A36: pending requests fail with <code>IO</code>, a replacement starts on the same channel with the live memories re-attached; handles do not survive).</p>
|
|
216
|
+
<p>Store lock (A37): with <code>lock: { name }</code> the I/O worker takes that Web Lock before it opens any handle and releases it only after <code>stop()</code> has closed them, so the lock lives exactly as long as the handles. <code>ifAvailable: true</code> fails the start with <code>lockUnavailable</code> instead of waiting; <code>busyRetry</code> retries a handle a previous leader still holds, with backoff. Measured takeover (holder <code>stop()</code> to successor ready): 5 ms Chromium, 52 ms Firefox, 5 ms WebKit. Leadership, follower proxying and heartbeats are the engine's (sdn-js, T10).</p>
|
|
217
|
+
<p>Head mirror (A7): with <code>mirror: { buffer, suffixes: ["/h.fsh"] }</code>, the writer I/O worker copies every write to a matching path into a seqlock mirror in a SharedArrayBuffer (<code>sabIoMirror.js</code>), and reader imports configured with the same mirror serve reads of those paths from it without a round trip.</p>
|
|
218
|
+
<h3 id="node-synchronous-fs-56"><a class="anchor" href="#node-synchronous-fs-56" aria-hidden="true">#</a>Node synchronous fs (§5.6)</h3>
|
|
219
|
+
<p><code>nodeSyncFsIo.js</code>: synchronous <code>fs</code> in each worker over a shared virtual-handle table in a SharedArrayBuffer (<code>createNodeSyncFsIoTable</code>). A handle is <code>(slot << 8) | gen</code>; each worker opens its own fd for a slot on first use and drops stale fds when the generation moves. Paths are confined below <code>root</code>, including through symlinked parents. <code>sync</code> is <code>fdatasync</code> (libuv issues <code>F_FULLFSYNC</code> on darwin). <code>revokeNodeSyncFsIoInstance</code> follows A23: it sets the revoked flag and waits for the instance's in-flight calls to drain. Fault injection (§19, 22.3a-7) belongs to FlatSQL's Node host (T4); <code>interpose</code> wraps every syscall for it.</p>
|
|
220
|
+
<h3 id="link-shim-v2"><a class="anchor" href="#link-shim-v2" aria-hidden="true">#</a>Link shim v2</h3>
|
|
221
|
+
<p><code>FLATSQL_LINK_SHIM_V2_WASM</code> (<code>src/flow/flatsqlLinkShim.js</code>) is a deterministic module that imports the reader instance's SHARED lane memory as <code>flatsql.memory</code> and gives a linked flow the lane mailbox as direct calls. v1 is unchanged (sha256 <code>8d83e69b…</code>), because today's linked flows use it. v2 sha256: <code>67d5b2d9bc2d1b346a14a253a586fd4d08c8056d54eb62701b48000585ed9613</code>.</p>
|
|
222
|
+
<div class="table-wrap"><table><thead><tr><th>Export</th><th>Result</th></tr></thead><tbody><tr><td><code>mb_submit(mailbox, op, req_ptr, req_len)</code></td><td><code>seq</code>, or -1 when the mailbox holds a request</td></tr><tr><td><code>mb_poll(mailbox, seq)</code></td><td>1 when complete</td></tr><tr><td><code>mb_wait(mailbox, seq, poll_ns: i64, max_polls)</code></td><td>the lane's status, or -110 after <code>max_polls</code> bounded waits (<code>max_polls <= 0</code>: no limit)</td></tr><tr><td><code>mb_release(mailbox, seq)</code></td><td>0, or -1 when not complete</td></tr><tr><td><code>mb_cancel(mailbox, seq)</code></td><td>0; the lane polls the cancel word</td></tr><tr><td><code>load32_acquire</code>, <code>store32_release</code>, <code>peek8/32/64</code>, <code>poke8/32</code>, <code>fnv1a64</code>, <code>count_frames</code></td><td>as in v1, over lane memory</td></tr></tbody></table></div>
|
|
223
|
+
<p>Mailbox, 64 bytes, 8-aligned, little-endian: <code>+0 state</code> (IDLE 0, SUBMITTED 1, CLAIMED 2, DONE 3, SUBMITTING 4), <code>+4 seq</code>, <code>+8 done_seq</code>, <code>+12 op</code>, <code>+16 req_ptr</code>, <code>+20 req_len</code>, <code>+24 status</code>, <code>+28 resp_ptr</code>, <code>+32 resp_len</code>, <code>+36 doorbell</code>, <code>+40 cancel</code>, <code>+44 flags</code>, <code>+48 generation (u64)</code>. A lane waits (bounded) on the doorbell, CASes SUBMITTED -> CLAIMED, writes the result, stores <code>done_seq = seq</code>, stores DONE and notifies the state word.</p>
|
|
224
|
+
<p><code>mb_wait</code> never waits unboundedly: each <code>memory.atomic.wait32</code> lasts <code>poll_ns</code> and every wake re-reads the mailbox. WasmEdge keeps waiters per executor, so a lane's notify may never reach a flow on another executor; a lost notify then costs one poll interval and never a completion. <code>poll_ns = 0</code> polls without executing a wait (contexts that may not block).</p>
|
|
225
|
+
<h3 id="worker-bundles-a39"><a class="anchor" href="#worker-bundles-a39" aria-hidden="true">#</a>Worker bundles (A39)</h3>
|
|
226
|
+
<p>The dashboard is one HTML file under <code>worker-src 'self' blob:</code>. <code>space-data-module-sdk/host/worker-bundles</code> ships the pool worker and the I/O worker as self-contained classic scripts (esbuild IIFE, built by <code>scripts/build-host-worker-bundles.mjs</code> into <code>src/host/hostWorkerBundleSources.js</code>):</p>
|
|
227
|
+
<div class="codeblock"><div class="codeblock-head">js</div><pre><code>import { hostWorkerBundleUrl } from "space-data-module-sdk/host/worker-bundles";
|
|
228
|
+
import { createWasiThreadSpawn } from "space-data-module-sdk/host/wasi-threads";
|
|
229
|
+
import { createFlatsqlIoWorker } from "space-data-module-sdk/host/flatsql-io";
|
|
230
|
+
|
|
231
|
+
const writerIo = await createFlatsqlIoWorker({
|
|
232
|
+
workerUrl: hostWorkerBundleUrl("flatsql-io"), workerType: "classic",
|
|
233
|
+
backend: "opfs", role: "writer", rootDirectory: "sdn-store",
|
|
234
|
+
});
|
|
235
|
+
await writerIo.attachMemory(1, writerMemory);
|
|
236
|
+
const pool = await createWasiThreadSpawn({
|
|
237
|
+
wasmModule, memory: writerMemory, poolSize: 1 + 2, instanceId: 1,
|
|
238
|
+
enableBrowserThreads: true,
|
|
239
|
+
browserWorkerUrl: hostWorkerBundleUrl("wasi-thread-pool"), browserWorkerType: "classic",
|
|
240
|
+
extraImports: [{ provider: "flatsql-io", instanceId: 1, channels: [writerIo.buffer] }],
|
|
241
|
+
onGuestError: (instanceId, tid, error) => supervisor.poison(instanceId, tid, error),
|
|
242
|
+
});</code></pre></div>
|
|
243
|
+
<h3 id="browser-capability-matrix"><a class="anchor" href="#browser-capability-matrix" aria-hidden="true">#</a>Browser capability matrix</h3>
|
|
244
|
+
<p><code>probeBrowserCapabilities()</code> probes the page and a blob worker; <code>flatsqlLocalStoreGate(matrix)</code> is the store gate: cross-origin isolation, shared memory, OPFS sync handles in a worker, <code>Atomics.wait</code> in workers, and shared sync-handle modes (§22.4-6: without them there is no local store). Measured 2026-09-27 on the owner's Mac Studio (Apple M3 Ultra, macOS 26.3.1), headless, Playwright 1.63.0, under the dashboard CSP with COOP/COEP:</p>
|
|
245
|
+
<div class="table-wrap"><table><thead><tr><th></th><th>Chromium 153.0.8010.12</th><th>Firefox 155.0</th><th>WebKit 26.6</th></tr></thead><tbody><tr><td><code>crossOriginIsolated</code>, shared wasm memory</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td><code>Atomics.waitAsync</code> (page and worker)</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>OPFS sync access handle in a worker</td><td>yes</td><td>yes</td><td>yes (persistent profile only)</td></tr><tr><td>Second default handle on one file</td><td>NoModificationAllowedError</td><td>NoModificationAllowedError</td><td>InvalidStateError</td></tr><tr><td>Shared modes (<code>readwrite-unsafe</code> x 2)</td><td>yes</td><td>no (mode ignored)</td><td>no (mode ignored)</td></tr><tr><td>SharedArrayBuffer views in <code>read</code>/<code>write</code></td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>Wasm shared-memory views in <code>read</code></td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td><code>removeEntry</code> of an open file</td><td>NoModificationAllowedError</td><td>NoModificationAllowedError</td><td>NoModificationAllowedError</td></tr><tr><td>Nested blob workers</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td><code>performance.now()</code> resolution</td><td>5 µs</td><td>20 µs</td><td>20 µs</td></tr><tr><td>Local-store gate</td><td>supported</td><td>no (shared modes)</td><td>no (shared modes)</td></tr></tbody></table></div>
|
|
246
|
+
<p>WebKit refuses OPFS in Playwright's ephemeral context (<code>UnknownError</code>); the suite uses a persistent profile.</p>
|
|
247
|
+
<h3 id="conformance"><a class="anchor" href="#conformance" aria-hidden="true">#</a>Conformance</h3>
|
|
248
|
+
<p><code>runFlatsqlIoConformance(io)</code> (<code>flatsqlIoConformance.js</code>) is one script for every host (22.3a-6): 15 cases covering statuses, short reads, sparse writes, truncation, EXCL/TRUNC/PROBE/UNLINK/UNLINK_IF_UNUSED/CREATE_PARENTS/ DELETE_ON_CLOSE/OPEN_DEFERRED, access modes, confinement and multi-handle visibility. It passes on the Node sync-fs provider, on the channel over the memory backend (blocking and async clients, both doorbells), on the blob bundle run as a standalone script, and on OPFS and memory in Chromium, Firefox and WebKit. Plain <code>UNLINK</code> of an open path is <code>BUSY</code> in the I/O worker (OPFS cannot remove a file with an open sync handle) and succeeds on POSIX hosts; the script does not test it.</p>
|
|
249
|
+
<h3 id="measured-acceptance-18-t9-a7-a38"><a class="anchor" href="#measured-acceptance-18-t9-a7-a38" aria-hidden="true">#</a>Measured acceptance (§18 T9, A7, A38)</h3>
|
|
250
|
+
<p>Owner's Mac Studio (Apple M3 Ultra, 28 cores, macOS 26.3.1), 2026-09-27, headless, final full run of <code>test/opfs-io-worker.browser.test.js</code>. The machine was shared with other lanes (load average 23-33 on 28 cores). Latencies are guest-observed import calls (<code>trace</code>), in microseconds.</p>
|
|
251
|
+
<div class="table-wrap"><table><thead><tr><th>Item</th><th>Chromium 153</th><th>Firefox 155</th><th>WebKit 26.6</th></tr></thead><tbody><tr><td>#1 8 threads, mixed I/O + 200 async opens + 50 unlinks through 1 I/O worker: errors / lost writes</td><td>0 / 0</td><td>0 / 0</td><td>0 / 0</td></tr><tr><td>same run, 4 KiB read p50 / p99</td><td>50 / 2145</td><td>260 / 8180</td><td>60 / 2660</td></tr><tr><td>same run, message doorbell: errors / lost writes</td><td>0 / 0</td><td>0 / 0</td><td>0 / 0</td></tr><tr><td>A7 lane 4 KiB read p99 during 100 x 4 MiB write+flush, same partition</td><td>790 (active file, shared mode)</td><td>120 (sealed segment)</td><td>40 (sealed segment)</td></tr><tr><td>A7 same, other partition</td><td>45</td><td>120</td><td>40</td></tr><tr><td>A7 writer flush p50 / p99 (4 MiB)</td><td>6365 / 28945</td><td>3540 / 27740</td><td>5260 / 70700</td></tr><tr><td>#2 500 ms open, median of 3 runs: other threads' read p99, baseline / during</td><td>85 / 85</td><td>440 / 400</td><td>60 / 60</td></tr><tr><td>#2 longest other-thread read while the open was pending</td><td>1020</td><td>18500</td><td>320</td></tr><tr><td>A38 OPEN_DEFERRED: open returns in / first write waits (ms)</td><td>3.8 / 504</td><td>28.8 / 511</td><td>5.2 / 755</td></tr><tr><td>#3 <code>poolSize=6</code> on <code>hardwareConcurrency=2</code></td><td>6 workers; 7th spawn -1, reported <code>pool-exhausted</code></td><td>same</td><td>same</td></tr><tr><td>#5 link shim v2 (Node 25): 10,000 calls, lost completions</td><td>0 with a notifying lane, 0 with a lane that never notifies, 0 spinning (<code>poll_ns = 0</code>)</td><td></td><td></td></tr></tbody></table></div>
|
|
252
|
+
<p>Chromium's same-partition A7 p99 varies with host load: 90, 110, 295 and 790 over four runs with the adaptive spin (below), and 125-1165 over four runs before it. The original #1 target (4 KiB read p99 <= 200 µs under the mixed load) is not met through one I/O worker; A7 replaced #1 with the <= 1 ms lane target, which is met. Firefox and WebKit have no shared handle modes, so their lanes read a partition's sealed segment; per §22.4-6 those browsers get no local store.</p>
|
|
253
|
+
<p>Adaptive waits: an idle I/O worker polls its doorbell for 50 µs, and a blocking client polls for its answer for 20 µs, before sleeping (<code>spinMicros</code>), so back-to-back requests skip a thread wake-up. In the Node channel test this took the fast-thread read p50 from 30-39 µs to 11-12 µs.</p>
|
|
187
254
|
<h2 id="tests"><a class="anchor" href="#tests" aria-hidden="true">#</a>Tests</h2>
|
|
188
255
|
<ul>
|
|
189
256
|
<li><code>test/wasi-thread-bundled-consumer-anchor.test.js</code> — the <strong>bundled-consumer guardrail</strong>: the host source is run through esbuild into a directory that does not hold the worker chain (the published-bucket geometry), driven by a Worker mock that resolves URLs on the filesystem the way a browser resolves them against an origin. With no anchor the pooled path must throw <code>WasiThreadWorkerUnreachableError</code> (the silent sequential fallback is a HARD failure); with an explicit base — or the process-wide setter — the same bundle arms its pool and spawns threads. Also pins the precedence table and asserts the resolver neither fetches nor sniffs <code>location</code>.</li>
|
|
@@ -191,6 +258,16 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
191
258
|
<li><code>test/pthreads-artifact-guardrail.test.js</code> — flag-assembler invariants; a positive compile that emits a validated wasi-threads shared-memory/atomics wasm; a single-thread artifact rejected; a <strong>browser-only Emscripten <code>-pthread</code> artifact rejected</strong> (has shared memory + atomics but no wasi-threads contract); a shared-flag-stripped artifact rejected; and a false-positive guard proving the atomics decoder ignores <code>0xFE</code> immediates.</li>
|
|
192
259
|
<li><code>test/guest-link-symbol-prefix.test.js</code> — the guest-link prefix is the full injective hex of the pluginId; two previously-colliding ids now get distinct prefixes; fresh prefixes match the committed modules-branch artifacts byte-for-byte; and the compose path treats the guest-link metadata's <code>symbolPrefix</code> / <code>methodSymbols</code> as authoritative (never re-derived), keeping a legacy truncated-prefix artifact compatible.</li>
|
|
193
260
|
</ul>
|
|
261
|
+
<ul>
|
|
262
|
+
<li><code>test/wasi-thread-pool-size.test.js</code> — explicit <code>poolSize</code> (6 workers on <code>hardwareConcurrency=2</code>, the 7th spawn -1 and reported), partial arming, extraImports delivery, <code>onGuestError</code>, classic workers.</li>
|
|
263
|
+
<li><code>test/sab-io-channel.test.js</code> — the I/O channel under a real wasi-threads guest (<code>test/support/flatsql-io/ioGuestWasm.mjs</code>): 8 threads of mixed I/O in both doorbell modes, a 500 ms open blocking only its caller, OPEN_DEFERRED, revocation, a dead I/O worker, supervisor restart, 256 KiB steps, the head mirror, scratch vs direct transfers, pre-open, and path-hash routing.</li>
|
|
264
|
+
<li><code>test/node-sync-fs-io.test.js</code> — shared handles across workers, stale handles, CREATE_PARENTS, UNLINK_IF_UNUSED, confinement, A23 revocation.</li>
|
|
265
|
+
<li><code>test/flatsql-io-conformance.test.js</code> — the conformance script on every Node host.</li>
|
|
266
|
+
<li><code>test/host-worker-bundles.test.js</code> — the bundles equal a fresh build, and the I/O worker bundle passes the conformance script standalone.</li>
|
|
267
|
+
<li><code>test/link-shim-v2.test.js</code> — shim v2 bytes, contract, and 10,000 calls against a stub lane with and without notify.</li>
|
|
268
|
+
<li><code>test/opfs-io-worker.browser.test.js</code> — the real-browser suite above (env-gated):</li>
|
|
269
|
+
</ul>
|
|
270
|
+
<div class="codeblock"><div class="codeblock-head">sh</div><pre><code>SPACE_DATA_MODULE_SDK_ENABLE_BROWSER_IO=1 node --test test/opfs-io-worker.browser.test.js</code></pre></div>
|
|
194
271
|
<h2 id="see-also"><a class="anchor" href="#see-also" aria-hidden="true">#</a>See also</h2>
|
|
195
272
|
<ul>
|
|
196
273
|
<li><a href="./browser-wasmedge-isomorphic.html"><code>docs/browser-wasmedge-isomorphic.md</code></a> — the one-artifact browser + WasmEdge loading profile.</li>
|
|
@@ -198,6 +275,7 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
198
275
|
<li><code>src/host/wasiThreadHost.js</code> — the isomorphic <code>wasi.thread-spawn</code> host: <code>createWasiThreadSpawn</code>, the browser worker anchor (<code>setBrowserWasiThreadWorkerBase</code>, <code>resolveBrowserWorkerUrl</code>), and <code>WasiThreadWorkerUnreachableError</code>.</li>
|
|
199
276
|
<li><code>src/compiler/pthreadArtifactGuard.js</code> — the flag list + wasm validator.</li>
|
|
200
277
|
<li><code>src/compiler/wasiThreadsToolchain.js</code> — the wasi-threads toolchain resolver.</li>
|
|
278
|
+
<li><code>src/host/sabIoChannel.js</code>, <code>src/host/flatsqlIoServer.js</code>, <code>src/host/opfsIoWorker.mjs</code>, <code>src/host/flatsqlIoWorkers.js</code>, <code>src/host/nodeSyncFsIo.js</code>, <code>src/host/sabIoMirror.js</code>, <code>src/host/flatsqlIoConformance.js</code>, <code>src/host/browserCapabilityProbe.js</code>, <code>src/host/hostWorkerBundles.js</code> — the FlatSQL partition store host I/O (§5).</li>
|
|
201
279
|
<li><code>src/compiler/compileModule.js</code> — <code>ModuleThreadModel</code>, <code>buildCompilerArgs</code>, <code>compileWithWasiThreads</code>, <code>resolveThreadModel</code>, <code>compileModuleFromSource</code>.</li>
|
|
202
280
|
</ul>
|
|
203
281
|
|
|
@@ -217,6 +295,16 @@ setBrowserWasiThreadWorkerBase("js/vendor/space-data-module-sdk/src/host/&q
|
|
|
217
295
|
<li><a class="depth-3" href="#sdk-0821-constructors-on-the-direct-surface">SDK 0.8.21: constructors on the direct surface</a></li>
|
|
218
296
|
<li><a class="depth-3" href="#guest-thread-faults">Guest thread faults</a></li>
|
|
219
297
|
<li><a class="depth-2" href="#4-integrators-the-browser-worker-anchor-required-when-you-bundle">4. Integrators: the browser worker anchor (REQUIRED when you bundle)</a></li>
|
|
298
|
+
<li><a class="depth-2" href="#5-flatsql-partition-store-host-io-sdk-0822">5. FlatSQL partition store host I/O (SDK 0.8.22)</a></li>
|
|
299
|
+
<li><a class="depth-3" href="#explicit-pool-size-partial-spawns-supervision-hooks">Explicit pool size, partial spawns, supervision hooks</a></li>
|
|
300
|
+
<li><a class="depth-3" href="#flags-and-statuses">Flags and statuses</a></li>
|
|
301
|
+
<li><a class="depth-3" href="#the-sab-io-channel-and-the-io-worker">The SAB I/O channel and the I/O worker</a></li>
|
|
302
|
+
<li><a class="depth-3" href="#node-synchronous-fs-56">Node synchronous fs (§5.6)</a></li>
|
|
303
|
+
<li><a class="depth-3" href="#link-shim-v2">Link shim v2</a></li>
|
|
304
|
+
<li><a class="depth-3" href="#worker-bundles-a39">Worker bundles (A39)</a></li>
|
|
305
|
+
<li><a class="depth-3" href="#browser-capability-matrix">Browser capability matrix</a></li>
|
|
306
|
+
<li><a class="depth-3" href="#conformance">Conformance</a></li>
|
|
307
|
+
<li><a class="depth-3" href="#measured-acceptance-18-t9-a7-a38">Measured acceptance (§18 T9, A7, A38)</a></li>
|
|
220
308
|
<li><a class="depth-2" href="#tests">Tests</a></li>
|
|
221
309
|
<li><a class="depth-2" href="#see-also">See also</a></li></ul></nav>
|
|
222
310
|
</div>
|
|
@@ -333,6 +333,269 @@ and reports it cannot instantiate the module, a probe timeout, a non-isolated
|
|
|
333
333
|
context, non-shared memory, or a 1-core host all still disable threading and let
|
|
334
334
|
the guest run its proven sequential path (`wasi.thread-spawn` -> `-1`).
|
|
335
335
|
|
|
336
|
+
## 5. FlatSQL partition store host I/O (SDK 0.8.22)
|
|
337
|
+
|
|
338
|
+
The FlatSQL partition store (stack design `docs/architecture/flatsql-partition-store.md`,
|
|
339
|
+
§5.5, §5.6, §18 T9, A7, A36, A38, A39) runs one wasi-threads artifact as a
|
|
340
|
+
writer instance and reader instances, each with guest threads that call
|
|
341
|
+
FlatSQL's seven `env.flatsql_io_*` imports. This section is the SDK half: the
|
|
342
|
+
thread pool, the I/O channel and workers, the Node provider, link shim v2, the
|
|
343
|
+
worker bundles and the browser capability probe.
|
|
344
|
+
|
|
345
|
+
### Explicit pool size, partial spawns, supervision hooks
|
|
346
|
+
|
|
347
|
+
`createWasiThreadSpawn` options added in 0.8.22:
|
|
348
|
+
|
|
349
|
+
| Option | Effect |
|
|
350
|
+
| --- | --- |
|
|
351
|
+
| `poolSize` | Browser: pre-start exactly this many workers, independent of `hardwareConcurrency - 1` (pools are sized for isolation: writers + lanes). Node: cap on live guest threads. Arming is partial: workers that fail to start are dropped, the rest serve. |
|
|
352
|
+
| `extraImports` | Per-worker import objects, as structured-cloneable descriptors (below). Factories run once per worker. |
|
|
353
|
+
| `instanceId`, `onGuestError(instanceId, tid, error)` | Called when a guest thread traps or its worker dies (A36). A dead pooled worker leaves the pool. |
|
|
354
|
+
| `onSpawnDeclined({ reason, poolSize, declined })` | Called for every spawn that returns -1. Reasons: `pool-exhausted`, `pool-not-armed`, `threads-unavailable`, `pool-empty`, `worker-create-failed`, `dispatch-failed`, `hostcall-channel-missing`, `terminated`. |
|
|
355
|
+
| `probeTimeoutMs` | Browser warm-pool probe deadline. |
|
|
356
|
+
| `browserWorkerType: "classic"` | Spawn classic workers, for the blob bundles below. |
|
|
357
|
+
|
|
358
|
+
The returned host adds `spawnReport()`: `{ poolSize, armed, failedToArm, spawned,
|
|
359
|
+
declined, declinedByReason, lastDeclineReason, active, idle }`. The implicit
|
|
360
|
+
(no `poolSize`) path keeps its all-or-nothing arming.
|
|
361
|
+
|
|
362
|
+
`extraImports` entries (the same descriptors work in the engine worker through
|
|
363
|
+
`resolveExtraImports`):
|
|
364
|
+
|
|
365
|
+
| Descriptor | Imports |
|
|
366
|
+
| --- | --- |
|
|
367
|
+
| `{ provider: "flatsql-io", instanceId, channels, mirror?, trace? }` | `env.flatsql_io_*` over one SAB I/O channel per I/O worker. Each worker claims its own request slot. |
|
|
368
|
+
| `{ provider: "flatsql-io-node", root, table, instanceId }` | `env.flatsql_io_*` over synchronous `fs` (Node workers). |
|
|
369
|
+
| `{ moduleUrl, exportName?, config? }` | A factory module (module workers and Node only). |
|
|
370
|
+
|
|
371
|
+
A module that bundles this host into an IIFE or a blob: module worker has no
|
|
372
|
+
usable `import.meta.url`. `DEFAULT_BROWSER_WORKER_URL` is then `null` instead of
|
|
373
|
+
a module-evaluation `TypeError`, and such hosts pass `browserWorkerUrl`.
|
|
374
|
+
|
|
375
|
+
### Flags and statuses
|
|
376
|
+
|
|
377
|
+
The partition store adds three open flags and one status. They are flags, so the
|
|
378
|
+
import set stays at seven. flatsql's `flatsql_io.h` (T1) must carry the same
|
|
379
|
+
values.
|
|
380
|
+
|
|
381
|
+
| Name | Value | Meaning |
|
|
382
|
+
| --- | --- | --- |
|
|
383
|
+
| `FLATSQL_IO_CREATE_PARENTS` | `0x0100` | mkdir -p; sync each new directory's parent, and the parent of a new file. Best effort on OPFS. |
|
|
384
|
+
| `FLATSQL_IO_UNLINK_IF_UNUSED` | `0x0200` | Unlink, or `BUSY` while any handle names the path. |
|
|
385
|
+
| `FLATSQL_IO_OPEN_DEFERRED` | `0x0400` | Return a handle before the open finishes; the first use waits, or fails with the open's status. Synchronous hosts treat it as a plain open. |
|
|
386
|
+
| `FLATSQL_IO_ERR_BUSY` | `-7` | Path in use, or its lock held elsewhere. |
|
|
387
|
+
|
|
388
|
+
Every SDK host maps `EEXIST` to `GENERIC` (the Go host's mapping), a write on a
|
|
389
|
+
read-only handle and a read on a write-only handle to `IO`, and `..` to
|
|
390
|
+
`ACCESS`. OPFS errors map as NotFound -> `NOENT`, NoModificationAllowed ->
|
|
391
|
+
`BUSY`, QuotaExceeded -> `NOSPACE`; WebKit refuses a second sync handle with
|
|
392
|
+
InvalidStateError (measured), which maps to `BUSY` during an open.
|
|
393
|
+
|
|
394
|
+
### The SAB I/O channel and the I/O worker
|
|
395
|
+
|
|
396
|
+
`sabIoChannel.js` extends `sabHostcallChannel.js` to many guest threads and one
|
|
397
|
+
server:
|
|
398
|
+
|
|
399
|
+
- A request ring of slots in one SharedArrayBuffer. Each request owns a slot
|
|
400
|
+
until its result is read, so a guest never queues behind another guest's
|
|
401
|
+
request and a slow open blocks only its caller. Idle threads hold no slot, so
|
|
402
|
+
terminating a pool leaks none; `reclaimSabIoSlots(buffer, instanceId)` frees
|
|
403
|
+
the slots of requests that were in flight when an instance's threads died.
|
|
404
|
+
- Doorbell: `Atomics.waitAsync` on a header word. The server publishes the mode;
|
|
405
|
+
in message mode (`doorbell: "message"`, or no `waitAsync`) clients also post
|
|
406
|
+
on a BroadcastChannel. Completions for non-blocking callers without
|
|
407
|
+
`waitAsync` arrive the same way (22.3a-5).
|
|
408
|
+
- No timeout, no throw (A36). Blocking waits run in 250 ms slices forever. A
|
|
409
|
+
supervisor that sees the I/O worker die calls `failPendingSabIoRequests`,
|
|
410
|
+
which completes every pending slot with `IO`.
|
|
411
|
+
- Data moves directly between the instance's shared memory and the file when the
|
|
412
|
+
instance's memory is attached to the I/O worker, else through the slot's data
|
|
413
|
+
area. Paths are always copied out of shared memory before decoding.
|
|
414
|
+
- Revocation (A36): `revokeSabIoInstance(buffer, id)` resolves after the server
|
|
415
|
+
has closed the instance's handles; later requests get `ACCESS` and no byte of
|
|
416
|
+
the instance is written after it resolves.
|
|
417
|
+
|
|
418
|
+
`opfsIoWorker.mjs` runs `flatsqlIoServer.js` over a backend:
|
|
419
|
+
|
|
420
|
+
- `opfs`: every open is async (`getDirectoryHandle`, `getFileHandle`,
|
|
421
|
+
`createSyncAccessHandle`) and awaited on the worker's event loop while the
|
|
422
|
+
requester waits on its slot. One sync handle per path, shared by all virtual
|
|
423
|
+
handles. Handles open in `"readwrite-unsafe"` mode where it exists, so a
|
|
424
|
+
reader I/O worker can read files a writer I/O worker holds (A7). Views over a
|
|
425
|
+
SharedArrayBuffer go straight to `read`/`write`; a user agent that rejects
|
|
426
|
+
them gets a private scratch copy.
|
|
427
|
+
- `memory`: the dashboard window store (§5.5). Nothing reaches OPFS;
|
|
428
|
+
`reset()` drops it; `memoryMaxBytes` is the per-tab budget (`NOSPACE`).
|
|
429
|
+
|
|
430
|
+
Roles (A7): one writer I/O worker per writer holds the writer's active files;
|
|
431
|
+
reader I/O workers (one, or two when `hardwareConcurrency >= 8`) hold sealed
|
|
432
|
+
files. Reads and writes run in steps of at most 256 KiB with the slots rescanned
|
|
433
|
+
between steps, so a small read never waits for a whole large write. With
|
|
434
|
+
several reader I/O workers, `createFlatsqlIoImports` routes each open by a path
|
|
435
|
+
hash and every later call to the same worker (the worker index rides in handle
|
|
436
|
+
bits 24-30).
|
|
437
|
+
|
|
438
|
+
`createFlatsqlIoWorker(options)` spawns and supervises one I/O worker:
|
|
439
|
+
`attachMemory`, `revoke`, `preopen(paths)` (A38: open the registry's active
|
|
440
|
+
files in parallel at start; later guest opens of those paths return at once),
|
|
441
|
+
`releasePreopen`, `stats`, `reset`, `clear`, `stop`, and `onError` plus
|
|
442
|
+
`restart: true` (A36: pending requests fail with `IO`, a replacement starts on
|
|
443
|
+
the same channel with the live memories re-attached; handles do not survive).
|
|
444
|
+
|
|
445
|
+
Store lock (A37): with `lock: { name }` the I/O worker takes that Web Lock
|
|
446
|
+
before it opens any handle and releases it only after `stop()` has closed them,
|
|
447
|
+
so the lock lives exactly as long as the handles. `ifAvailable: true` fails the
|
|
448
|
+
start with `lockUnavailable` instead of waiting; `busyRetry` retries a handle a
|
|
449
|
+
previous leader still holds, with backoff. Measured takeover (holder `stop()` to
|
|
450
|
+
successor ready): 5 ms Chromium, 52 ms Firefox, 5 ms WebKit. Leadership,
|
|
451
|
+
follower proxying and heartbeats are the engine's (sdn-js, T10).
|
|
452
|
+
|
|
453
|
+
Head mirror (A7): with `mirror: { buffer, suffixes: ["/h.fsh"] }`, the writer
|
|
454
|
+
I/O worker copies every write to a matching path into a seqlock mirror in a
|
|
455
|
+
SharedArrayBuffer (`sabIoMirror.js`), and reader imports configured with the
|
|
456
|
+
same mirror serve reads of those paths from it without a round trip.
|
|
457
|
+
|
|
458
|
+
### Node synchronous fs (§5.6)
|
|
459
|
+
|
|
460
|
+
`nodeSyncFsIo.js`: synchronous `fs` in each worker over a shared virtual-handle
|
|
461
|
+
table in a SharedArrayBuffer (`createNodeSyncFsIoTable`). A handle is
|
|
462
|
+
`(slot << 8) | gen`; each worker opens its own fd for a slot on first use and
|
|
463
|
+
drops stale fds when the generation moves. Paths are confined below `root`,
|
|
464
|
+
including through symlinked parents. `sync` is `fdatasync` (libuv issues
|
|
465
|
+
`F_FULLFSYNC` on darwin). `revokeNodeSyncFsIoInstance` follows A23: it sets the
|
|
466
|
+
revoked flag and waits for the instance's in-flight calls to drain. Fault
|
|
467
|
+
injection (§19, 22.3a-7) belongs to FlatSQL's Node host (T4); `interpose` wraps
|
|
468
|
+
every syscall for it.
|
|
469
|
+
|
|
470
|
+
### Link shim v2
|
|
471
|
+
|
|
472
|
+
`FLATSQL_LINK_SHIM_V2_WASM` (`src/flow/flatsqlLinkShim.js`) is a deterministic
|
|
473
|
+
module that imports the reader instance's SHARED lane memory as
|
|
474
|
+
`flatsql.memory` and gives a linked flow the lane mailbox as direct calls. v1 is
|
|
475
|
+
unchanged (sha256 `8d83e69b…`), because today's linked flows use it. v2 sha256:
|
|
476
|
+
`67d5b2d9bc2d1b346a14a253a586fd4d08c8056d54eb62701b48000585ed9613`.
|
|
477
|
+
|
|
478
|
+
| Export | Result |
|
|
479
|
+
| --- | --- |
|
|
480
|
+
| `mb_submit(mailbox, op, req_ptr, req_len)` | `seq`, or -1 when the mailbox holds a request |
|
|
481
|
+
| `mb_poll(mailbox, seq)` | 1 when complete |
|
|
482
|
+
| `mb_wait(mailbox, seq, poll_ns: i64, max_polls)` | the lane's status, or -110 after `max_polls` bounded waits (`max_polls <= 0`: no limit) |
|
|
483
|
+
| `mb_release(mailbox, seq)` | 0, or -1 when not complete |
|
|
484
|
+
| `mb_cancel(mailbox, seq)` | 0; the lane polls the cancel word |
|
|
485
|
+
| `load32_acquire`, `store32_release`, `peek8/32/64`, `poke8/32`, `fnv1a64`, `count_frames` | as in v1, over lane memory |
|
|
486
|
+
|
|
487
|
+
Mailbox, 64 bytes, 8-aligned, little-endian: `+0 state` (IDLE 0, SUBMITTED 1,
|
|
488
|
+
CLAIMED 2, DONE 3, SUBMITTING 4), `+4 seq`, `+8 done_seq`, `+12 op`,
|
|
489
|
+
`+16 req_ptr`, `+20 req_len`, `+24 status`, `+28 resp_ptr`, `+32 resp_len`,
|
|
490
|
+
`+36 doorbell`, `+40 cancel`, `+44 flags`, `+48 generation (u64)`. A lane waits
|
|
491
|
+
(bounded) on the doorbell, CASes SUBMITTED -> CLAIMED, writes the result,
|
|
492
|
+
stores `done_seq = seq`, stores DONE and notifies the state word.
|
|
493
|
+
|
|
494
|
+
`mb_wait` never waits unboundedly: each `memory.atomic.wait32` lasts `poll_ns`
|
|
495
|
+
and every wake re-reads the mailbox. WasmEdge keeps waiters per executor, so a
|
|
496
|
+
lane's notify may never reach a flow on another executor; a lost notify then
|
|
497
|
+
costs one poll interval and never a completion. `poll_ns = 0` polls without
|
|
498
|
+
executing a wait (contexts that may not block).
|
|
499
|
+
|
|
500
|
+
### Worker bundles (A39)
|
|
501
|
+
|
|
502
|
+
The dashboard is one HTML file under `worker-src 'self' blob:`.
|
|
503
|
+
`space-data-module-sdk/host/worker-bundles` ships the pool worker and the I/O
|
|
504
|
+
worker as self-contained classic scripts (esbuild IIFE, built by
|
|
505
|
+
`scripts/build-host-worker-bundles.mjs` into `src/host/hostWorkerBundleSources.js`):
|
|
506
|
+
|
|
507
|
+
```js
|
|
508
|
+
import { hostWorkerBundleUrl } from "space-data-module-sdk/host/worker-bundles";
|
|
509
|
+
import { createWasiThreadSpawn } from "space-data-module-sdk/host/wasi-threads";
|
|
510
|
+
import { createFlatsqlIoWorker } from "space-data-module-sdk/host/flatsql-io";
|
|
511
|
+
|
|
512
|
+
const writerIo = await createFlatsqlIoWorker({
|
|
513
|
+
workerUrl: hostWorkerBundleUrl("flatsql-io"), workerType: "classic",
|
|
514
|
+
backend: "opfs", role: "writer", rootDirectory: "sdn-store",
|
|
515
|
+
});
|
|
516
|
+
await writerIo.attachMemory(1, writerMemory);
|
|
517
|
+
const pool = await createWasiThreadSpawn({
|
|
518
|
+
wasmModule, memory: writerMemory, poolSize: 1 + 2, instanceId: 1,
|
|
519
|
+
enableBrowserThreads: true,
|
|
520
|
+
browserWorkerUrl: hostWorkerBundleUrl("wasi-thread-pool"), browserWorkerType: "classic",
|
|
521
|
+
extraImports: [{ provider: "flatsql-io", instanceId: 1, channels: [writerIo.buffer] }],
|
|
522
|
+
onGuestError: (instanceId, tid, error) => supervisor.poison(instanceId, tid, error),
|
|
523
|
+
});
|
|
524
|
+
```
|
|
525
|
+
|
|
526
|
+
### Browser capability matrix
|
|
527
|
+
|
|
528
|
+
`probeBrowserCapabilities()` probes the page and a blob worker;
|
|
529
|
+
`flatsqlLocalStoreGate(matrix)` is the store gate: cross-origin isolation,
|
|
530
|
+
shared memory, OPFS sync handles in a worker, `Atomics.wait` in workers, and
|
|
531
|
+
shared sync-handle modes (§22.4-6: without them there is no local store).
|
|
532
|
+
Measured 2026-09-27 on the owner's Mac Studio (Apple M3 Ultra, macOS 26.3.1),
|
|
533
|
+
headless, Playwright 1.63.0, under the dashboard CSP with COOP/COEP:
|
|
534
|
+
|
|
535
|
+
| | Chromium 153.0.8010.12 | Firefox 155.0 | WebKit 26.6 |
|
|
536
|
+
| --- | --- | --- | --- |
|
|
537
|
+
| `crossOriginIsolated`, shared wasm memory | yes | yes | yes |
|
|
538
|
+
| `Atomics.waitAsync` (page and worker) | yes | yes | yes |
|
|
539
|
+
| OPFS sync access handle in a worker | yes | yes | yes (persistent profile only) |
|
|
540
|
+
| Second default handle on one file | NoModificationAllowedError | NoModificationAllowedError | InvalidStateError |
|
|
541
|
+
| Shared modes (`readwrite-unsafe` x 2) | yes | no (mode ignored) | no (mode ignored) |
|
|
542
|
+
| SharedArrayBuffer views in `read`/`write` | yes | yes | yes |
|
|
543
|
+
| Wasm shared-memory views in `read` | yes | yes | yes |
|
|
544
|
+
| `removeEntry` of an open file | NoModificationAllowedError | NoModificationAllowedError | NoModificationAllowedError |
|
|
545
|
+
| Nested blob workers | yes | yes | yes |
|
|
546
|
+
| `performance.now()` resolution | 5 µs | 20 µs | 20 µs |
|
|
547
|
+
| Local-store gate | supported | no (shared modes) | no (shared modes) |
|
|
548
|
+
|
|
549
|
+
WebKit refuses OPFS in Playwright's ephemeral context (`UnknownError`); the suite
|
|
550
|
+
uses a persistent profile.
|
|
551
|
+
|
|
552
|
+
### Conformance
|
|
553
|
+
|
|
554
|
+
`runFlatsqlIoConformance(io)` (`flatsqlIoConformance.js`) is one script for every
|
|
555
|
+
host (22.3a-6): 15 cases covering statuses, short reads, sparse writes,
|
|
556
|
+
truncation, EXCL/TRUNC/PROBE/UNLINK/UNLINK_IF_UNUSED/CREATE_PARENTS/
|
|
557
|
+
DELETE_ON_CLOSE/OPEN_DEFERRED, access modes, confinement and multi-handle
|
|
558
|
+
visibility. It passes on the Node sync-fs provider, on the channel over the
|
|
559
|
+
memory backend (blocking and async clients, both doorbells), on the blob bundle
|
|
560
|
+
run as a standalone script, and on OPFS and memory in Chromium, Firefox and
|
|
561
|
+
WebKit. Plain `UNLINK` of an open path is `BUSY` in the I/O worker (OPFS cannot
|
|
562
|
+
remove a file with an open sync handle) and succeeds on POSIX hosts; the script
|
|
563
|
+
does not test it.
|
|
564
|
+
|
|
565
|
+
### Measured acceptance (§18 T9, A7, A38)
|
|
566
|
+
|
|
567
|
+
Owner's Mac Studio (Apple M3 Ultra, 28 cores, macOS 26.3.1), 2026-09-27,
|
|
568
|
+
headless, final full run of `test/opfs-io-worker.browser.test.js`. The machine
|
|
569
|
+
was shared with other lanes (load average 23-33 on 28 cores). Latencies are
|
|
570
|
+
guest-observed import calls (`trace`), in microseconds.
|
|
571
|
+
|
|
572
|
+
| Item | Chromium 153 | Firefox 155 | WebKit 26.6 |
|
|
573
|
+
| --- | --- | --- | --- |
|
|
574
|
+
| #1 8 threads, mixed I/O + 200 async opens + 50 unlinks through 1 I/O worker: errors / lost writes | 0 / 0 | 0 / 0 | 0 / 0 |
|
|
575
|
+
| same run, 4 KiB read p50 / p99 | 50 / 2145 | 260 / 8180 | 60 / 2660 |
|
|
576
|
+
| same run, message doorbell: errors / lost writes | 0 / 0 | 0 / 0 | 0 / 0 |
|
|
577
|
+
| A7 lane 4 KiB read p99 during 100 x 4 MiB write+flush, same partition | 790 (active file, shared mode) | 120 (sealed segment) | 40 (sealed segment) |
|
|
578
|
+
| A7 same, other partition | 45 | 120 | 40 |
|
|
579
|
+
| A7 writer flush p50 / p99 (4 MiB) | 6365 / 28945 | 3540 / 27740 | 5260 / 70700 |
|
|
580
|
+
| #2 500 ms open, median of 3 runs: other threads' read p99, baseline / during | 85 / 85 | 440 / 400 | 60 / 60 |
|
|
581
|
+
| #2 longest other-thread read while the open was pending | 1020 | 18500 | 320 |
|
|
582
|
+
| A38 OPEN_DEFERRED: open returns in / first write waits (ms) | 3.8 / 504 | 28.8 / 511 | 5.2 / 755 |
|
|
583
|
+
| #3 `poolSize=6` on `hardwareConcurrency=2` | 6 workers; 7th spawn -1, reported `pool-exhausted` | same | same |
|
|
584
|
+
| #5 link shim v2 (Node 25): 10,000 calls, lost completions | 0 with a notifying lane, 0 with a lane that never notifies, 0 spinning (`poll_ns = 0`) | | |
|
|
585
|
+
|
|
586
|
+
Chromium's same-partition A7 p99 varies with host load: 90, 110, 295 and 790
|
|
587
|
+
over four runs with the adaptive spin (below), and 125-1165 over four runs
|
|
588
|
+
before it. The original #1 target (4 KiB read p99 <= 200 µs under the mixed
|
|
589
|
+
load) is not met through one I/O worker; A7 replaced #1 with the <= 1 ms lane
|
|
590
|
+
target, which is met. Firefox and WebKit have no shared handle modes, so their
|
|
591
|
+
lanes read a partition's sealed segment; per §22.4-6 those browsers get no
|
|
592
|
+
local store.
|
|
593
|
+
|
|
594
|
+
Adaptive waits: an idle I/O worker polls its doorbell for 50 µs, and a blocking
|
|
595
|
+
client polls for its answer for 20 µs, before sleeping (`spinMicros`), so
|
|
596
|
+
back-to-back requests skip a thread wake-up. In the Node channel test this took
|
|
597
|
+
the fast-thread read p50 from 30-39 µs to 11-12 µs.
|
|
598
|
+
|
|
336
599
|
## Tests
|
|
337
600
|
|
|
338
601
|
- `test/wasi-thread-bundled-consumer-anchor.test.js` — the **bundled-consumer
|
|
@@ -360,6 +623,28 @@ the guest run its proven sequential path (`wasi.thread-spawn` -> `-1`).
|
|
|
360
623
|
`symbolPrefix` / `methodSymbols` as authoritative (never re-derived), keeping a
|
|
361
624
|
legacy truncated-prefix artifact compatible.
|
|
362
625
|
|
|
626
|
+
- `test/wasi-thread-pool-size.test.js` — explicit `poolSize` (6 workers on
|
|
627
|
+
`hardwareConcurrency=2`, the 7th spawn -1 and reported), partial arming,
|
|
628
|
+
extraImports delivery, `onGuestError`, classic workers.
|
|
629
|
+
- `test/sab-io-channel.test.js` — the I/O channel under a real wasi-threads
|
|
630
|
+
guest (`test/support/flatsql-io/ioGuestWasm.mjs`): 8 threads of mixed I/O in
|
|
631
|
+
both doorbell modes, a 500 ms open blocking only its caller, OPEN_DEFERRED,
|
|
632
|
+
revocation, a dead I/O worker, supervisor restart, 256 KiB steps, the head
|
|
633
|
+
mirror, scratch vs direct transfers, pre-open, and path-hash routing.
|
|
634
|
+
- `test/node-sync-fs-io.test.js` — shared handles across workers, stale handles,
|
|
635
|
+
CREATE_PARENTS, UNLINK_IF_UNUSED, confinement, A23 revocation.
|
|
636
|
+
- `test/flatsql-io-conformance.test.js` — the conformance script on every Node host.
|
|
637
|
+
- `test/host-worker-bundles.test.js` — the bundles equal a fresh build, and the
|
|
638
|
+
I/O worker bundle passes the conformance script standalone.
|
|
639
|
+
- `test/link-shim-v2.test.js` — shim v2 bytes, contract, and 10,000 calls
|
|
640
|
+
against a stub lane with and without notify.
|
|
641
|
+
- `test/opfs-io-worker.browser.test.js` — the real-browser suite above
|
|
642
|
+
(env-gated):
|
|
643
|
+
|
|
644
|
+
```sh
|
|
645
|
+
SPACE_DATA_MODULE_SDK_ENABLE_BROWSER_IO=1 node --test test/opfs-io-worker.browser.test.js
|
|
646
|
+
```
|
|
647
|
+
|
|
363
648
|
## See also
|
|
364
649
|
|
|
365
650
|
- [`docs/browser-wasmedge-isomorphic.md`](./browser-wasmedge-isomorphic.md) —
|
|
@@ -371,5 +656,10 @@ the guest run its proven sequential path (`wasi.thread-spawn` -> `-1`).
|
|
|
371
656
|
`WasiThreadWorkerUnreachableError`.
|
|
372
657
|
- `src/compiler/pthreadArtifactGuard.js` — the flag list + wasm validator.
|
|
373
658
|
- `src/compiler/wasiThreadsToolchain.js` — the wasi-threads toolchain resolver.
|
|
659
|
+
- `src/host/sabIoChannel.js`, `src/host/flatsqlIoServer.js`,
|
|
660
|
+
`src/host/opfsIoWorker.mjs`, `src/host/flatsqlIoWorkers.js`,
|
|
661
|
+
`src/host/nodeSyncFsIo.js`, `src/host/sabIoMirror.js`,
|
|
662
|
+
`src/host/flatsqlIoConformance.js`, `src/host/browserCapabilityProbe.js`,
|
|
663
|
+
`src/host/hostWorkerBundles.js` — the FlatSQL partition store host I/O (§5).
|
|
374
664
|
- `src/compiler/compileModule.js` — `ModuleThreadModel`, `buildCompilerArgs`,
|
|
375
665
|
`compileWithWasiThreads`, `resolveThreadModel`, `compileModuleFromSource`.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "space-data-module-sdk",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.22",
|
|
4
4
|
"license": "Apache-2.0",
|
|
5
5
|
"description": "Module SDK for building, validating, signing, and deploying WebAssembly modules on the Space Data Network.",
|
|
6
6
|
"type": "module",
|
|
@@ -62,6 +62,10 @@
|
|
|
62
62
|
"./host/runtime-target-gate": "./src/host/runtimeTargetGate.js",
|
|
63
63
|
"./host/node-builtin": "./src/host/nodeBuiltinSpecifier.js",
|
|
64
64
|
"./host/worker-module": "./src/host/workerModuleHarness.js",
|
|
65
|
+
"./host/wasi-threads": "./src/host/wasiThreadHost.js",
|
|
66
|
+
"./host/flatsql-io": "./src/host/flatsqlIo.js",
|
|
67
|
+
"./host/flatsql-io/node": "./src/host/nodeSyncFsIo.js",
|
|
68
|
+
"./host/worker-bundles": "./src/host/hostWorkerBundles.js",
|
|
65
69
|
"./testing/browser": "./src/testing/browser.js",
|
|
66
70
|
"./testing/module-flatbuffer-stream-pump": {
|
|
67
71
|
"types": "./src/index.d.ts",
|
|
@@ -129,7 +133,8 @@
|
|
|
129
133
|
},
|
|
130
134
|
"devDependencies": {
|
|
131
135
|
"esbuild": "^0.28.0",
|
|
132
|
-
"express": "^4.21.2"
|
|
136
|
+
"express": "^4.21.2",
|
|
137
|
+
"playwright-core": "1.63.0"
|
|
133
138
|
},
|
|
134
139
|
"engines": {
|
|
135
140
|
"node": ">=20.0.0"
|
package/src/browser.js
CHANGED
|
@@ -41,7 +41,13 @@ export {
|
|
|
41
41
|
resolveBrowserWorkerUrl,
|
|
42
42
|
DEFAULT_BROWSER_WORKER_URL,
|
|
43
43
|
WasiThreadWorkerUnreachableError,
|
|
44
|
+
createWasiThreadSpawn,
|
|
45
|
+
isWasiThreadsModule,
|
|
44
46
|
} from "./host/wasiThreadHost.js";
|
|
47
|
+
// FlatSQL partition store host I/O (T9): the SAB I/O channel, the I/O worker
|
|
48
|
+
// controller, OPFS and memory backends, and the capability probe. The worker
|
|
49
|
+
// bundles for blob: spawning are on the "./host/worker-bundles" subpath.
|
|
50
|
+
export * from "./host/flatsqlIo.js";
|
|
45
51
|
// The runtime-target gate is reachable so a consumer can catch the refusal by
|
|
46
52
|
// CLASS (`error instanceof RuntimeTargetError`) rather than by matching a
|
|
47
53
|
// message string, and can ask the same question the loaders ask before it
|