space-data-module-sdk 0.5.19 → 0.5.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +52 -7
- package/package.json +5 -1
- package/src/AGENTS.md +33 -0
- package/src/auth/AGENTS.md +25 -0
- package/src/browser.js +2 -0
- package/src/bundle/AGENTS.md +21 -0
- package/src/compiler/AGENTS.md +35 -0
- package/src/compliance/AGENTS.md +18 -0
- package/src/host/AGENTS.md +39 -0
- package/src/host/isomorphicLoader.js +133 -0
- package/src/host/wasiShim.js +12 -1
- package/src/index.d.ts +100 -0
- package/src/manifest/AGENTS.md +18 -0
- package/src/runtime-host/AGENTS.md +32 -0
- package/src/runtime-host/flatbufferStreamIngestor.js +181 -0
- package/src/runtime-host/flatsqlRuntimeStore.js +120 -49
- package/src/runtime-host/index.js +2 -0
- package/src/testing/AGENTS.md +33 -0
- package/src/testing/index.js +1 -0
- package/src/testing/moduleFlatbufferStreamPump.js +255 -0
package/README.md
CHANGED
|
@@ -52,6 +52,10 @@ The minimal host surface lives under `src/runtime-host/` and covers three
|
|
|
52
52
|
responsibilities:
|
|
53
53
|
|
|
54
54
|
- `createFlatSqlRuntimeStore()` for append-only row handles and row resolution
|
|
55
|
+
- `createFlatBufferStreamIngestor()` for little-endian size-prefixed FlatBuffer
|
|
56
|
+
transport ingest into host-owned row storage without JSON transcoding
|
|
57
|
+
- `createModuleFlatBufferStreamPump()` for feeding those same FlatBuffer stream
|
|
58
|
+
chunks into a resident stateful module instance without JSON envelopes
|
|
55
59
|
- `createRuntimeRegionStore()` for host-allocated aligned-binary regions and
|
|
56
60
|
externally backed record-view descriptors
|
|
57
61
|
- `createModuleRegistry()` for dynamic install/load/unload/invoke
|
|
@@ -60,6 +64,9 @@ responsibilities:
|
|
|
60
64
|
OrbPro layers entity/view helpers on top of that host instead of inventing a
|
|
61
65
|
separate durable identity model.
|
|
62
66
|
|
|
67
|
+
The canonical FlatBuffer-to-FlatSQL streaming contract is documented in
|
|
68
|
+
[`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md).
|
|
69
|
+
|
|
63
70
|
## Module Artifact Model
|
|
64
71
|
|
|
65
72
|
A compliant module built with this SDK is always a valid `.wasm` artifact with:
|
|
@@ -236,6 +243,12 @@ The checked-in same-artifact demo lives in
|
|
|
236
243
|
- [`wasmedge-demo.mjs`](./examples/isomorphic-loader/wasmedge-demo.mjs) loads
|
|
237
244
|
that same artifact in WasmEdge
|
|
238
245
|
|
|
246
|
+
Large streamed FlatBuffer ingest should use either the runtime-host transport
|
|
247
|
+
path or the resident-module stream pump, not one giant invoke envelope. See
|
|
248
|
+
[`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md)
|
|
249
|
+
for the recommended browser/WasmEdge/FlatSQL split and the canonical
|
|
250
|
+
direct-binary ingest contract.
|
|
251
|
+
|
|
239
252
|
## Testing
|
|
240
253
|
|
|
241
254
|
This repo now exposes a manifest-driven harness generator from
|
|
@@ -260,10 +273,23 @@ This repo now exposes a manifest-driven harness generator from
|
|
|
260
273
|
- `npm run test:host-surfaces`
|
|
261
274
|
- authoritative Node-host coverage for HTTP, TCP, UDP, TLS, WebSocket, MQTT,
|
|
262
275
|
process execution, timers, filesystem, and the sync `sdn_host` ABI
|
|
276
|
+
- `npm run test:stream-ingest`
|
|
277
|
+
- correctness coverage for chunked size-prefixed FlatBuffer ingest into the
|
|
278
|
+
runtime-host row store
|
|
279
|
+
- `npm run test:module-stream`
|
|
280
|
+
- correctness coverage for chunked FlatBuffer streaming into a resident
|
|
281
|
+
direct-surface module instance
|
|
282
|
+
- `npm run benchmark:stream-1gib`
|
|
283
|
+
- env-gated local stress benchmark for 1 GiB total FlatBuffer transport ingest
|
|
284
|
+
- `npm run benchmark:module-stream-1gib`
|
|
285
|
+
- env-gated local stress benchmark for 1 GiB total chunked module stream-pump
|
|
286
|
+
ingest
|
|
263
287
|
|
|
264
288
|
The detailed edge cases and the current WASI-vs-host portability boundary are
|
|
265
289
|
documented in
|
|
266
|
-
[`docs/testing-harness.md`](./docs/testing-harness.md).
|
|
290
|
+
[`docs/testing-harness.md`](./docs/testing-harness.md). The canonical
|
|
291
|
+
FlatBuffer-to-FlatSQL ingest contract is documented in
|
|
292
|
+
[`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md).
|
|
267
293
|
|
|
268
294
|
## Install
|
|
269
295
|
|
|
@@ -437,6 +463,22 @@ For npm packages, the simplest form is:
|
|
|
437
463
|
}
|
|
438
464
|
```
|
|
439
465
|
|
|
466
|
+
For module repos themselves, the canonical compiled artifact should live at:
|
|
467
|
+
|
|
468
|
+
```text
|
|
469
|
+
dist/isomorphic/module.wasm
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
If the repo also ships a direct browser adapter, publish it under:
|
|
473
|
+
|
|
474
|
+
```text
|
|
475
|
+
dist/browser/module.js
|
|
476
|
+
dist/browser/module.wasm
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
That keeps the shared browser/WasmEdge artifact path stable across repos while
|
|
480
|
+
leaving the browser-specific adapter optional.
|
|
481
|
+
|
|
440
482
|
When publication metadata is published in the same file, it belongs in an
|
|
441
483
|
appended SDS `REC` trailer. Loaders scan from the end of the protected blob,
|
|
442
484
|
resolve `PNM` / `ENC`, strip or decrypt as needed, and only then instantiate
|
|
@@ -492,16 +534,16 @@ Manifests can also declare coarse runtime targets for planning and compliance:
|
|
|
492
534
|
|
|
493
535
|
```bash
|
|
494
536
|
# Validate a manifest + wasm pair
|
|
495
|
-
npx space-data-module check --manifest ./manifest.json --wasm ./dist/module.wasm
|
|
537
|
+
npx space-data-module check --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm
|
|
496
538
|
|
|
497
539
|
# Compile C/C++ source and embed the manifest
|
|
498
|
-
npx space-data-module compile --manifest ./manifest.json --source ./src/module.c --out ./dist/module.wasm
|
|
540
|
+
npx space-data-module compile --manifest ./manifest.json --source ./src/module.c --out ./dist/isomorphic/module.wasm
|
|
499
541
|
|
|
500
542
|
# Sign and encrypt a deployment payload
|
|
501
|
-
npx space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --json
|
|
543
|
+
npx space-data-module protect --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm --json
|
|
502
544
|
|
|
503
545
|
# Emit a single-file bundled wasm
|
|
504
|
-
npx space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --single-file-bundle --out ./dist/module.bundle.wasm
|
|
546
|
+
npx space-data-module protect --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm --single-file-bundle --out ./dist/module.bundle.wasm
|
|
505
547
|
```
|
|
506
548
|
|
|
507
549
|
## Module Lab
|
|
@@ -534,8 +576,11 @@ npm run check:compliance
|
|
|
534
576
|
```
|
|
535
577
|
|
|
536
578
|
Node.js `>=20` is required. The compiler uses `sdn-emception` and `flatc-wasm`
|
|
537
|
-
by default for the embedded toolchain path.
|
|
538
|
-
|
|
579
|
+
by default for the embedded toolchain path. For multi-repo module builds, use a
|
|
580
|
+
repo-local `deps/emsdk` checkout by default instead of Homebrew or any other
|
|
581
|
+
machine-global Emscripten install. Treat `PATH` Emscripten as an explicit escape
|
|
582
|
+
hatch, not the default. WasmEdge pthread builds still require a system
|
|
583
|
+
Emscripten toolchain on `PATH`.
|
|
539
584
|
|
|
540
585
|
If another repo needs the same compiler runtime, the package also exposes a
|
|
541
586
|
shared emception session at `space-data-module-sdk/compiler/emception` with
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "space-data-module-sdk",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.21",
|
|
4
4
|
"description": "Module SDK for building, validating, signing, and deploying WebAssembly modules on the Space Data Network.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"types": "./src/index.d.ts",
|
|
@@ -53,6 +53,10 @@
|
|
|
53
53
|
"test": "node --test",
|
|
54
54
|
"test:host-surfaces": "node --test test/node-host.test.js test/host-abi.test.js",
|
|
55
55
|
"test:runtime-matrix": "SPACE_DATA_MODULE_SDK_ENABLE_RUNTIME_MATRIX=1 node --test test/runtime-matrix.test.js",
|
|
56
|
+
"test:stream-ingest": "node --test test/runtime-host-stream-ingest.test.js",
|
|
57
|
+
"test:module-stream": "node --test test/module-flatbuffer-stream-pump.test.js",
|
|
58
|
+
"benchmark:stream-1gib": "SPACE_DATA_MODULE_SDK_ENABLE_1GB_STREAM_TEST=1 node --test test/runtime-host-stream-ingest.test.js",
|
|
59
|
+
"benchmark:module-stream-1gib": "SPACE_DATA_MODULE_SDK_ENABLE_1GB_MODULE_STREAM_TEST=1 node --test test/module-flatbuffer-stream-pump.test.js",
|
|
56
60
|
"start:lab": "node ./lab/server.js",
|
|
57
61
|
"check:compliance": "node ./bin/space-data-module.js check --repo-root .",
|
|
58
62
|
"generate:vectors": "node ./examples/single-file-bundle/generate-vectors.mjs",
|
package/src/AGENTS.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# AGENTS
|
|
2
|
+
|
|
3
|
+
Apply the root `AGENTS.md` first. This file routes source edits to the correct
|
|
4
|
+
subsystem.
|
|
5
|
+
|
|
6
|
+
## Source Routing
|
|
7
|
+
|
|
8
|
+
- `src/manifest`, `src/compliance`, `src/standards`: manifest shape, type refs,
|
|
9
|
+
standards validation, and compliance warnings/errors.
|
|
10
|
+
- `src/manifest/AGENTS.md`, `src/compliance/AGENTS.md`, `src/bundle/AGENTS.md`,
|
|
11
|
+
and `src/auth/AGENTS.md` provide narrower rules for those owners.
|
|
12
|
+
- `src/compiler`: compile flow, manifest embedding, runtime targets, toolchains,
|
|
13
|
+
and artifact layout.
|
|
14
|
+
- `src/bundle`: `sds.bundle`, wasm custom sections, and vector compatibility.
|
|
15
|
+
- `src/host`: Node host, browser shims, `sdn_host`, WasmEdge runners, and
|
|
16
|
+
isomorphic loaders.
|
|
17
|
+
- `src/runtime-host`: FlatSQL-backed runtime host, row/region identity, and
|
|
18
|
+
binary stream ingest.
|
|
19
|
+
- `src/testing`: harnesses, browser module loading, process invoke clients, and
|
|
20
|
+
module-owned FlatBuffer stream pumps.
|
|
21
|
+
- `src/auth`, `src/transport`, `src/deployment`: signing, encryption, REC/PNM/
|
|
22
|
+
ENC records, deployment plans, and publication protection.
|
|
23
|
+
- `src/generated`: generated code. Do not hand-edit unless regeneration is part
|
|
24
|
+
of the task and you update the generation path or fixtures as needed.
|
|
25
|
+
|
|
26
|
+
## Working Rules
|
|
27
|
+
|
|
28
|
+
- Keep module contracts portable unless the task is explicitly runtime-specific.
|
|
29
|
+
- Prefer changes in the smallest subsystem that owns the behavior.
|
|
30
|
+
- If a change crosses compiler, host, and docs, update each owner explicitly
|
|
31
|
+
rather than hiding policy in one layer.
|
|
32
|
+
- Add or update tests near the owning subsystem and then wire them into the
|
|
33
|
+
repo-level verification set in `test/AGENTS.md`.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# AGENTS
|
|
2
|
+
|
|
3
|
+
You are in `src/auth`, which owns deployment authorization, signing helpers,
|
|
4
|
+
and related record handling used by module distribution.
|
|
5
|
+
|
|
6
|
+
## Rules
|
|
7
|
+
|
|
8
|
+
- Keep signing and authorization flows deterministic and explicit.
|
|
9
|
+
- Do not invent app-local signature records when a repo-level contract belongs
|
|
10
|
+
here.
|
|
11
|
+
- When auth changes affect transport or deployment plans, update the related
|
|
12
|
+
surfaces together rather than splitting incompatible changes across repos.
|
|
13
|
+
|
|
14
|
+
## Related Surfaces
|
|
15
|
+
|
|
16
|
+
- `src/transport`
|
|
17
|
+
- `src/deployment`
|
|
18
|
+
- `test/transport-records.test.js`
|
|
19
|
+
- `test/deployment-plan.test.js`
|
|
20
|
+
|
|
21
|
+
## Check Before You Finish
|
|
22
|
+
|
|
23
|
+
- `npm test`
|
|
24
|
+
- `npm run check:compliance`
|
|
25
|
+
- `node --test test/transport-records.test.js test/deployment-plan.test.js`
|
package/src/browser.js
CHANGED
|
@@ -6,9 +6,11 @@ export * from "./capabilities.js";
|
|
|
6
6
|
export * from "./deployment/index.js";
|
|
7
7
|
export * from "./invoke/index.js";
|
|
8
8
|
export * from "./runtime/index.js";
|
|
9
|
+
export * from "./runtime-host/index.js";
|
|
9
10
|
export * from "./host/browserHost.js";
|
|
10
11
|
export * from "./host/browserEdgeShims.js";
|
|
11
12
|
export * from "./host/wasiShim.js";
|
|
12
13
|
export * from "./host/abi.js";
|
|
13
14
|
export * from "./host/isomorphicLoader.js";
|
|
14
15
|
export { createBrowserModuleHarness, detectArtifactProfile } from "./testing/browserModuleHarness.js";
|
|
16
|
+
export { createModuleFlatBufferStreamPump } from "./testing/moduleFlatbufferStreamPump.js";
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# AGENTS
|
|
2
|
+
|
|
3
|
+
You are in `src/bundle`, which owns `sds.bundle` and the wasm custom-section
|
|
4
|
+
packaging rules.
|
|
5
|
+
|
|
6
|
+
## Rules
|
|
7
|
+
|
|
8
|
+
- `sds.bundle` is the single-file delivery format.
|
|
9
|
+
- Put bundle data in a wasm custom section; do not append raw bytes after the
|
|
10
|
+
wasm binary.
|
|
11
|
+
- Keep parser, encoder, CLI behavior, vectors, and cross-language fixtures in
|
|
12
|
+
sync.
|
|
13
|
+
- If the bundle contract changes, regenerate vectors and update the bundle docs
|
|
14
|
+
and demos in the same change.
|
|
15
|
+
|
|
16
|
+
## Check Before You Finish
|
|
17
|
+
|
|
18
|
+
- `npm test`
|
|
19
|
+
- `npm run check:compliance`
|
|
20
|
+
- `node --test test/module-bundle.test.js test/module-bundle-vectors.test.js test/module-bundle-cli.test.js test/module-bundle-go.test.js test/module-bundle-python.test.js`
|
|
21
|
+
- `npm run generate:vectors` when vectors or the contract changed
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# AGENTS
|
|
2
|
+
|
|
3
|
+
Apply the root and `src/AGENTS.md` files first.
|
|
4
|
+
|
|
5
|
+
## Area Ownership
|
|
6
|
+
|
|
7
|
+
This directory owns source-to-wasm compile behavior, manifest embedding, runtime
|
|
8
|
+
target inference, toolchain integration, and canonical artifact layout.
|
|
9
|
+
|
|
10
|
+
## Compiler Rules
|
|
11
|
+
|
|
12
|
+
- Preserve the canonical manifest accessor exports and validate them together
|
|
13
|
+
with compiled wasm.
|
|
14
|
+
- For shared browser/WasmEdge modules, emit the standalone artifact at
|
|
15
|
+
`dist/isomorphic/module.wasm`.
|
|
16
|
+
- Optional browser adapters belong under `dist/browser/`.
|
|
17
|
+
- Do not make browser/WasmEdge shared artifacts depend on pthread-style `env.*`
|
|
18
|
+
imports.
|
|
19
|
+
- Prefer repo-local Emscripten or `sdn-emception`. Do not assume Homebrew
|
|
20
|
+
toolchains.
|
|
21
|
+
- If you change runtime-target inference, keep the browser/WasmEdge shared path
|
|
22
|
+
on the standalone profile unless the docs and tests are updated deliberately.
|
|
23
|
+
|
|
24
|
+
## Key Files
|
|
25
|
+
|
|
26
|
+
- `compileModule.js`
|
|
27
|
+
- `compileModuleFromSource.js`
|
|
28
|
+
- `emception*.js`
|
|
29
|
+
- `invokeGlue.js`
|
|
30
|
+
|
|
31
|
+
## Verification
|
|
32
|
+
|
|
33
|
+
- `node --test test/module-sdk.test.js test/compliance.test.js`
|
|
34
|
+
- `node --test test/browser-harness.test.js test/isomorphic-loader.test.js`
|
|
35
|
+
- `node --test test/compiler-emception-subpath-export.test.js`
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# AGENTS
|
|
2
|
+
|
|
3
|
+
You are in `src/compliance`, which defines what counts as a compliant module or
|
|
4
|
+
artifact.
|
|
5
|
+
|
|
6
|
+
## Rules
|
|
7
|
+
|
|
8
|
+
- Treat this directory as policy, not convenience code.
|
|
9
|
+
- When tightening validation, update tests and error messages together so the
|
|
10
|
+
failure mode is actionable.
|
|
11
|
+
- When relaxing validation, make sure the compiler, docs, and examples still
|
|
12
|
+
describe the broader contract accurately.
|
|
13
|
+
|
|
14
|
+
## Check Before You Finish
|
|
15
|
+
|
|
16
|
+
- `npm test`
|
|
17
|
+
- `npm run check:compliance`
|
|
18
|
+
- `node --test test/module-sdk.test.js test/compliance.test.js`
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# AGENTS
|
|
2
|
+
|
|
3
|
+
Apply the root and `src/AGENTS.md` files first.
|
|
4
|
+
|
|
5
|
+
## Area Ownership
|
|
6
|
+
|
|
7
|
+
This directory owns the Node reference host, browser host, browser edge shims,
|
|
8
|
+
WASI shims, the sync `sdn_host` ABI bridge, WasmEdge launch planning, and the
|
|
9
|
+
isomorphic loader path.
|
|
10
|
+
|
|
11
|
+
## Host Rules
|
|
12
|
+
|
|
13
|
+
- Shared browser/WasmEdge artifacts must stay within standalone WASI plus the
|
|
14
|
+
optional `sdn_host` bridge. Raw WasmEdge-native extension imports are not
|
|
15
|
+
browser-isomorphic.
|
|
16
|
+
- Browser shims are host-side replacements, not a promise that the guest can
|
|
17
|
+
import WasmEdge-native sockets, TLS, or filesystem extensions directly.
|
|
18
|
+
- Keep the sync guest-host ABI narrow and deliberate. Avoid adding new hostcalls
|
|
19
|
+
unless the Node host, browser path, docs, and tests all agree on semantics.
|
|
20
|
+
- Scope host capabilities tightly: filesystem roots, network allowlists, TLS,
|
|
21
|
+
exec, timers, and crypto should stay explicit.
|
|
22
|
+
|
|
23
|
+
## Key Files
|
|
24
|
+
|
|
25
|
+
- `abi.js`
|
|
26
|
+
- `browserEdgeShims.js`
|
|
27
|
+
- `browserHost.js`
|
|
28
|
+
- `isomorphicLoader.js`
|
|
29
|
+
- `nodeHost.js`
|
|
30
|
+
- `wasiShim.js`
|
|
31
|
+
|
|
32
|
+
## Verification
|
|
33
|
+
|
|
34
|
+
- `node --test test/node-host.test.js test/host-abi.test.js`
|
|
35
|
+
- `node --test test/browser-harness.test.js test/isomorphic-loader.test.js`
|
|
36
|
+
- `node --test test/process-invoke.test.js`
|
|
37
|
+
- `node --test test/wasmedge-runner-build.test.js`
|
|
38
|
+
- `node --test test/wasmedge-runner-runtime.test.js` when WasmEdge behavior is
|
|
39
|
+
part of the change
|
|
@@ -18,6 +18,123 @@ const isBrowser =
|
|
|
18
18
|
typeof globalThis.window !== "undefined" &&
|
|
19
19
|
typeof globalThis.document !== "undefined";
|
|
20
20
|
|
|
21
|
+
async function createWasmEdgeCommandHarness(options = {}) {
|
|
22
|
+
const [
|
|
23
|
+
{ spawn },
|
|
24
|
+
{ readFile },
|
|
25
|
+
pathModule,
|
|
26
|
+
{ encodePluginInvokeRequest, decodePluginInvokeResponse },
|
|
27
|
+
{ buildWasmEdgeSpawnEnv },
|
|
28
|
+
{ DefaultInvokeExports },
|
|
29
|
+
{ toUint8Array },
|
|
30
|
+
] = await Promise.all([
|
|
31
|
+
import("node:child_process"),
|
|
32
|
+
import("node:fs/promises"),
|
|
33
|
+
import("node:path"),
|
|
34
|
+
import("../invoke/codec.js"),
|
|
35
|
+
import("../testing/processInvoke.js"),
|
|
36
|
+
import("../runtime/constants.js"),
|
|
37
|
+
import("../runtime/bufferLike.js"),
|
|
38
|
+
]);
|
|
39
|
+
const path = pathModule.default ?? pathModule;
|
|
40
|
+
const wasmPath = path.resolve(String(options.wasmSource));
|
|
41
|
+
const wasmBytes = await readFile(wasmPath);
|
|
42
|
+
const inspection = await inspectModule(wasmBytes);
|
|
43
|
+
|
|
44
|
+
if (!inspection.exports.includes(DefaultInvokeExports.commandSymbol)) {
|
|
45
|
+
throw new Error(
|
|
46
|
+
"Standalone WasmEdge loading requires a command-surface artifact with the _start export.",
|
|
47
|
+
);
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
const command = options.wasmEdgeBinary ?? "wasmedge";
|
|
51
|
+
const args = [
|
|
52
|
+
...(options.enableThreads === false ? [] : ["--enable-threads"]),
|
|
53
|
+
wasmPath,
|
|
54
|
+
...(Array.isArray(options.args) ? options.args : []),
|
|
55
|
+
];
|
|
56
|
+
const launchPlan = {
|
|
57
|
+
command,
|
|
58
|
+
args,
|
|
59
|
+
env: buildWasmEdgeSpawnEnv(options.env),
|
|
60
|
+
cwd: options.cwd ?? process.cwd(),
|
|
61
|
+
wasmPath,
|
|
62
|
+
};
|
|
63
|
+
|
|
64
|
+
async function invokeRaw(requestBytes) {
|
|
65
|
+
const normalizedRequest = toUint8Array(requestBytes);
|
|
66
|
+
if (!normalizedRequest) {
|
|
67
|
+
throw new TypeError(
|
|
68
|
+
"Expected Uint8Array, ArrayBufferView, or ArrayBuffer request bytes.",
|
|
69
|
+
);
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
return new Promise((resolve, reject) => {
|
|
73
|
+
const child = spawn(launchPlan.command, launchPlan.args, {
|
|
74
|
+
cwd: launchPlan.cwd,
|
|
75
|
+
env: launchPlan.env,
|
|
76
|
+
stdio: ["pipe", "pipe", "pipe"],
|
|
77
|
+
});
|
|
78
|
+
const stdoutChunks = [];
|
|
79
|
+
const stderrChunks = [];
|
|
80
|
+
|
|
81
|
+
function formatFailure(message, cause = null) {
|
|
82
|
+
const stderrText = Buffer.concat(stderrChunks).toString("utf8").trim();
|
|
83
|
+
const details = stderrText ? `${message}\n${stderrText}` : message;
|
|
84
|
+
return cause ? new Error(details, { cause }) : new Error(details);
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
child.stdout.on("data", (chunk) => {
|
|
88
|
+
stdoutChunks.push(Buffer.from(chunk));
|
|
89
|
+
});
|
|
90
|
+
child.stderr.on("data", (chunk) => {
|
|
91
|
+
stderrChunks.push(Buffer.from(chunk));
|
|
92
|
+
});
|
|
93
|
+
child.on("error", (error) => {
|
|
94
|
+
reject(
|
|
95
|
+
formatFailure(
|
|
96
|
+
"Failed to launch WasmEdge command harness.",
|
|
97
|
+
error,
|
|
98
|
+
),
|
|
99
|
+
);
|
|
100
|
+
});
|
|
101
|
+
child.on("close", (code, signal) => {
|
|
102
|
+
if (code !== 0 || signal !== null) {
|
|
103
|
+
reject(
|
|
104
|
+
formatFailure(
|
|
105
|
+
`WasmEdge command harness exited with ${
|
|
106
|
+
signal ? `signal ${signal}` : `code ${code}`
|
|
107
|
+
}.`,
|
|
108
|
+
),
|
|
109
|
+
);
|
|
110
|
+
return;
|
|
111
|
+
}
|
|
112
|
+
resolve(new Uint8Array(Buffer.concat(stdoutChunks)));
|
|
113
|
+
});
|
|
114
|
+
child.stdin.end(Buffer.from(normalizedRequest));
|
|
115
|
+
});
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
return {
|
|
119
|
+
runtime: {
|
|
120
|
+
kind: "wasmedge",
|
|
121
|
+
profile: inspection.profile,
|
|
122
|
+
surface: "command",
|
|
123
|
+
},
|
|
124
|
+
launchPlan,
|
|
125
|
+
invokeRaw,
|
|
126
|
+
async invoke(request = {}) {
|
|
127
|
+
const requestBytes = encodePluginInvokeRequest(request);
|
|
128
|
+
const responseBytes = await invokeRaw(requestBytes);
|
|
129
|
+
return decodePluginInvokeResponse(responseBytes);
|
|
130
|
+
},
|
|
131
|
+
readManifest() {
|
|
132
|
+
return null;
|
|
133
|
+
},
|
|
134
|
+
async destroy() {},
|
|
135
|
+
};
|
|
136
|
+
}
|
|
137
|
+
|
|
21
138
|
/**
|
|
22
139
|
* Load a WASM module isomorphically.
|
|
23
140
|
*
|
|
@@ -46,6 +163,22 @@ export async function loadModule(options = {}) {
|
|
|
46
163
|
|
|
47
164
|
const runtimeKind = options.runtimeKind ?? "wasmedge";
|
|
48
165
|
if (runtimeKind === "wasmedge") {
|
|
166
|
+
const runtimeHostRequested =
|
|
167
|
+
String(options.hostProfile ?? "").trim().toLowerCase() === "runtime-host" ||
|
|
168
|
+
Array.isArray(options.modules) ||
|
|
169
|
+
(typeof options.defaultModuleId === "string" &&
|
|
170
|
+
options.defaultModuleId.trim().length > 0);
|
|
171
|
+
if (!runtimeHostRequested && !options.wasmEdgeRunnerBinary) {
|
|
172
|
+
const { readFile } = await import("node:fs/promises");
|
|
173
|
+
const inspection = await inspectModule(await readFile(source));
|
|
174
|
+
if (
|
|
175
|
+
(inspection.profile === "standalone" || inspection.profile === "sdn-abi") &&
|
|
176
|
+
inspection.exports.includes("_start")
|
|
177
|
+
) {
|
|
178
|
+
return createWasmEdgeCommandHarness(options);
|
|
179
|
+
}
|
|
180
|
+
}
|
|
181
|
+
|
|
49
182
|
return createModuleHarness({
|
|
50
183
|
runtime: {
|
|
51
184
|
kind: "wasmedge",
|
package/src/host/wasiShim.js
CHANGED
|
@@ -7,7 +7,8 @@
|
|
|
7
7
|
*
|
|
8
8
|
* Covers the 11 imports observed across all SDN plugin standalone builds:
|
|
9
9
|
* clock_time_get, fd_write, fd_read, fd_close, fd_seek, fd_fdstat_get,
|
|
10
|
-
* environ_sizes_get, environ_get, proc_exit, args_get, args_sizes_get
|
|
10
|
+
* environ_sizes_get, environ_get, proc_exit, args_get, args_sizes_get,
|
|
11
|
+
* random_get
|
|
11
12
|
*/
|
|
12
13
|
|
|
13
14
|
const ERRNO_SUCCESS = 0;
|
|
@@ -38,6 +39,7 @@ export function createBrowserWasiShim(options = {}) {
|
|
|
38
39
|
now: () => Date.now(),
|
|
39
40
|
timeOrigin: 0,
|
|
40
41
|
};
|
|
42
|
+
const cryptoApi = options.crypto ?? globalThis.crypto ?? null;
|
|
41
43
|
const stdoutChunks = [];
|
|
42
44
|
const stderrChunks = [];
|
|
43
45
|
let stdinOffset = 0;
|
|
@@ -206,6 +208,14 @@ export function createBrowserWasiShim(options = {}) {
|
|
|
206
208
|
return ERRNO_SUCCESS;
|
|
207
209
|
}
|
|
208
210
|
|
|
211
|
+
function random_get(bufPtr, bufLen) {
|
|
212
|
+
if (!cryptoApi?.getRandomValues) {
|
|
213
|
+
return ERRNO_NOSYS;
|
|
214
|
+
}
|
|
215
|
+
cryptoApi.getRandomValues(mem8().subarray(bufPtr, bufPtr + bufLen));
|
|
216
|
+
return ERRNO_SUCCESS;
|
|
217
|
+
}
|
|
218
|
+
|
|
209
219
|
function proc_exit(code) {
|
|
210
220
|
if (logOutput) {
|
|
211
221
|
flushOutput();
|
|
@@ -259,6 +269,7 @@ export function createBrowserWasiShim(options = {}) {
|
|
|
259
269
|
environ_get,
|
|
260
270
|
args_sizes_get,
|
|
261
271
|
args_get,
|
|
272
|
+
random_get,
|
|
262
273
|
proc_exit,
|
|
263
274
|
},
|
|
264
275
|
},
|
package/src/index.d.ts
CHANGED
|
@@ -277,6 +277,22 @@ export interface RuntimeRowQueryResult {
|
|
|
277
277
|
rowCount: number;
|
|
278
278
|
}
|
|
279
279
|
|
|
280
|
+
export interface FlatBufferStreamIngestStats {
|
|
281
|
+
bytesReceived: number;
|
|
282
|
+
chunksReceived: number;
|
|
283
|
+
framesDecoded: number;
|
|
284
|
+
framesAppended: number;
|
|
285
|
+
framesRouted: number;
|
|
286
|
+
parseErrors: number;
|
|
287
|
+
}
|
|
288
|
+
|
|
289
|
+
export interface FlatBufferStreamIngestContext {
|
|
290
|
+
rawFileIdentifier: string;
|
|
291
|
+
schemaFileId: string;
|
|
292
|
+
rows: FlatSqlRuntimeStore;
|
|
293
|
+
stats: FlatBufferStreamIngestStats;
|
|
294
|
+
}
|
|
295
|
+
|
|
280
296
|
export interface FlatSqlRuntimeStore {
|
|
281
297
|
appendRow(options: { schemaFileId: string; payload?: unknown }): RowHandle;
|
|
282
298
|
listRows(schemaFileId?: string | null): RuntimeRowView[];
|
|
@@ -284,6 +300,13 @@ export interface FlatSqlRuntimeStore {
|
|
|
284
300
|
resolveRow(handle: RowHandle): RuntimeRowView | null;
|
|
285
301
|
}
|
|
286
302
|
|
|
303
|
+
export interface FlatBufferStreamIngestor {
|
|
304
|
+
rows: FlatSqlRuntimeStore;
|
|
305
|
+
stats: FlatBufferStreamIngestStats;
|
|
306
|
+
pushBytes(data: Uint8Array | ArrayBuffer | ArrayBufferView): number;
|
|
307
|
+
finish(): 0;
|
|
308
|
+
}
|
|
309
|
+
|
|
287
310
|
export interface RuntimeRegionDescriptor {
|
|
288
311
|
regionId: number;
|
|
289
312
|
layoutId: string;
|
|
@@ -388,6 +411,25 @@ export interface RuntimeHost {
|
|
|
388
411
|
moduleRegistry: RuntimeModuleRegistry;
|
|
389
412
|
}
|
|
390
413
|
|
|
414
|
+
export function createFlatBufferStreamIngestor(options?: {
|
|
415
|
+
rows?: FlatSqlRuntimeStore;
|
|
416
|
+
frameRouter?:
|
|
417
|
+
| ((
|
|
418
|
+
payload: Uint8Array,
|
|
419
|
+
context: FlatBufferStreamIngestContext,
|
|
420
|
+
) => boolean | void)
|
|
421
|
+
| Record<
|
|
422
|
+
string,
|
|
423
|
+
(
|
|
424
|
+
payload: Uint8Array,
|
|
425
|
+
context: FlatBufferStreamIngestContext,
|
|
426
|
+
) => boolean | void
|
|
427
|
+
>;
|
|
428
|
+
appendFrame?: (
|
|
429
|
+
payload: Uint8Array,
|
|
430
|
+
context: FlatBufferStreamIngestContext,
|
|
431
|
+
) => void;
|
|
432
|
+
}): FlatBufferStreamIngestor;
|
|
391
433
|
export function createFlatSqlRuntimeStore(): FlatSqlRuntimeStore;
|
|
392
434
|
export function createRuntimeRegionStore(): RuntimeRegionStore;
|
|
393
435
|
export function createModuleRegistry(): RuntimeModuleRegistry;
|
|
@@ -1563,6 +1605,64 @@ export function createBrowserModuleHarness(options?: {
|
|
|
1563
1605
|
};
|
|
1564
1606
|
logOutput?: boolean;
|
|
1565
1607
|
}): Promise<BrowserModuleHarness>;
|
|
1608
|
+
export interface ModuleFlatBufferStreamPumpStats {
|
|
1609
|
+
bytesReceived: number;
|
|
1610
|
+
chunksReceived: number;
|
|
1611
|
+
framesDecoded: number;
|
|
1612
|
+
framesInvoked: number;
|
|
1613
|
+
invokes: number;
|
|
1614
|
+
parseErrors: number;
|
|
1615
|
+
}
|
|
1616
|
+
export interface ModuleFlatBufferStreamPumpContext {
|
|
1617
|
+
rawFileIdentifier: string;
|
|
1618
|
+
schemaFileId: string;
|
|
1619
|
+
methodId: string;
|
|
1620
|
+
portId: string;
|
|
1621
|
+
streamId: number;
|
|
1622
|
+
sequence: number;
|
|
1623
|
+
stats: ModuleFlatBufferStreamPumpStats;
|
|
1624
|
+
}
|
|
1625
|
+
export interface ModuleFlatBufferStreamPump {
|
|
1626
|
+
stats: ModuleFlatBufferStreamPumpStats;
|
|
1627
|
+
lastResponse: PluginInvokeResponseEnvelope | null;
|
|
1628
|
+
pushBytes(data: Uint8Array | ArrayBuffer | ArrayBufferView): Promise<number>;
|
|
1629
|
+
finish(): Promise<PluginInvokeResponseEnvelope | null>;
|
|
1630
|
+
}
|
|
1631
|
+
export function createModuleFlatBufferStreamPump(options: {
|
|
1632
|
+
harness?: {
|
|
1633
|
+
invoke(
|
|
1634
|
+
request: PluginInvokeRequestEnvelope,
|
|
1635
|
+
): Promise<PluginInvokeResponseEnvelope>;
|
|
1636
|
+
};
|
|
1637
|
+
invoke?: (
|
|
1638
|
+
request: PluginInvokeRequestEnvelope,
|
|
1639
|
+
) => Promise<PluginInvokeResponseEnvelope>;
|
|
1640
|
+
methodId: string;
|
|
1641
|
+
portId: string;
|
|
1642
|
+
maxFramesPerInvoke?: number;
|
|
1643
|
+
streamId?: number;
|
|
1644
|
+
sequenceStart?: number;
|
|
1645
|
+
typeResolver?: (
|
|
1646
|
+
payload: Uint8Array,
|
|
1647
|
+
context: ModuleFlatBufferStreamPumpContext,
|
|
1648
|
+
) => PayloadTypeRef | null | undefined;
|
|
1649
|
+
frameTemplate?:
|
|
1650
|
+
| Partial<InvokeFrame>
|
|
1651
|
+
| ((
|
|
1652
|
+
payload: Uint8Array,
|
|
1653
|
+
context: ModuleFlatBufferStreamPumpContext,
|
|
1654
|
+
) => Partial<InvokeFrame> | null | undefined);
|
|
1655
|
+
onResponse?: (
|
|
1656
|
+
response: PluginInvokeResponseEnvelope,
|
|
1657
|
+
context: {
|
|
1658
|
+
methodId: string;
|
|
1659
|
+
portId: string;
|
|
1660
|
+
frames: InvokeFrame[];
|
|
1661
|
+
isFinalBatch: boolean;
|
|
1662
|
+
stats: ModuleFlatBufferStreamPumpStats;
|
|
1663
|
+
},
|
|
1664
|
+
) => void | Promise<void>;
|
|
1665
|
+
}): ModuleFlatBufferStreamPump;
|
|
1566
1666
|
export function loadModule(options?: {
|
|
1567
1667
|
wasmSource: Uint8Array | ArrayBuffer | string | WebAssembly.Module | unknown;
|
|
1568
1668
|
host?: BrowserHost;
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# AGENTS
|
|
2
|
+
|
|
3
|
+
You are in `src/manifest`, which owns the manifest schema codecs and
|
|
4
|
+
normalization rules.
|
|
5
|
+
|
|
6
|
+
## Rules
|
|
7
|
+
|
|
8
|
+
- Keep manifest encode/decode round-trips stable.
|
|
9
|
+
- Prefer canonical SDS schema names and file identifiers; do not add repo-local
|
|
10
|
+
aliases when a standards name already exists.
|
|
11
|
+
- If you change a manifest field or normalization rule, update the compiler,
|
|
12
|
+
compliance checks, and any affected examples/tests together.
|
|
13
|
+
|
|
14
|
+
## Check Before You Finish
|
|
15
|
+
|
|
16
|
+
- `npm test`
|
|
17
|
+
- `npm run check:compliance`
|
|
18
|
+
- `node --test test/module-sdk.test.js test/compliance.test.js`
|