space-data-module-sdk 0.5.19 → 0.5.21

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -52,6 +52,10 @@ The minimal host surface lives under `src/runtime-host/` and covers three
52
52
  responsibilities:
53
53
 
54
54
  - `createFlatSqlRuntimeStore()` for append-only row handles and row resolution
55
+ - `createFlatBufferStreamIngestor()` for little-endian size-prefixed FlatBuffer
56
+ transport ingest into host-owned row storage without JSON transcoding
57
+ - `createModuleFlatBufferStreamPump()` for feeding those same FlatBuffer stream
58
+ chunks into a resident stateful module instance without JSON envelopes
55
59
  - `createRuntimeRegionStore()` for host-allocated aligned-binary regions and
56
60
  externally backed record-view descriptors
57
61
  - `createModuleRegistry()` for dynamic install/load/unload/invoke
@@ -60,6 +64,9 @@ responsibilities:
60
64
  OrbPro layers entity/view helpers on top of that host instead of inventing a
61
65
  separate durable identity model.
62
66
 
67
+ The canonical FlatBuffer-to-FlatSQL streaming contract is documented in
68
+ [`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md).
69
+
63
70
  ## Module Artifact Model
64
71
 
65
72
  A compliant module built with this SDK is always a valid `.wasm` artifact with:
@@ -236,6 +243,12 @@ The checked-in same-artifact demo lives in
236
243
  - [`wasmedge-demo.mjs`](./examples/isomorphic-loader/wasmedge-demo.mjs) loads
237
244
  that same artifact in WasmEdge
238
245
 
246
+ Large streamed FlatBuffer ingest should use either the runtime-host transport
247
+ path or the resident-module stream pump, not one giant invoke envelope. See
248
+ [`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md)
249
+ for the recommended browser/WasmEdge/FlatSQL split and the canonical
250
+ direct-binary ingest contract.
251
+
239
252
  ## Testing
240
253
 
241
254
  This repo now exposes a manifest-driven harness generator from
@@ -260,10 +273,23 @@ This repo now exposes a manifest-driven harness generator from
260
273
  - `npm run test:host-surfaces`
261
274
  - authoritative Node-host coverage for HTTP, TCP, UDP, TLS, WebSocket, MQTT,
262
275
  process execution, timers, filesystem, and the sync `sdn_host` ABI
276
+ - `npm run test:stream-ingest`
277
+ - correctness coverage for chunked size-prefixed FlatBuffer ingest into the
278
+ runtime-host row store
279
+ - `npm run test:module-stream`
280
+ - correctness coverage for chunked FlatBuffer streaming into a resident
281
+ direct-surface module instance
282
+ - `npm run benchmark:stream-1gib`
283
+ - env-gated local stress benchmark for 1 GiB total FlatBuffer transport ingest
284
+ - `npm run benchmark:module-stream-1gib`
285
+ - env-gated local stress benchmark for 1 GiB total chunked module stream-pump
286
+ ingest
263
287
 
264
288
  The detailed edge cases and the current WASI-vs-host portability boundary are
265
289
  documented in
266
- [`docs/testing-harness.md`](./docs/testing-harness.md).
290
+ [`docs/testing-harness.md`](./docs/testing-harness.md). The canonical
291
+ FlatBuffer-to-FlatSQL ingest contract is documented in
292
+ [`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md).
267
293
 
268
294
  ## Install
269
295
 
@@ -437,6 +463,22 @@ For npm packages, the simplest form is:
437
463
  }
438
464
  ```
439
465
 
466
+ For module repos themselves, the canonical compiled artifact should live at:
467
+
468
+ ```text
469
+ dist/isomorphic/module.wasm
470
+ ```
471
+
472
+ If the repo also ships a direct browser adapter, publish it under:
473
+
474
+ ```text
475
+ dist/browser/module.js
476
+ dist/browser/module.wasm
477
+ ```
478
+
479
+ That keeps the shared browser/WasmEdge artifact path stable across repos while
480
+ leaving the browser-specific adapter optional.
481
+
440
482
  When publication metadata is published in the same file, it belongs in an
441
483
  appended SDS `REC` trailer. Loaders scan from the end of the protected blob,
442
484
  resolve `PNM` / `ENC`, strip or decrypt as needed, and only then instantiate
@@ -492,16 +534,16 @@ Manifests can also declare coarse runtime targets for planning and compliance:
492
534
 
493
535
  ```bash
494
536
  # Validate a manifest + wasm pair
495
- npx space-data-module check --manifest ./manifest.json --wasm ./dist/module.wasm
537
+ npx space-data-module check --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm
496
538
 
497
539
  # Compile C/C++ source and embed the manifest
498
- npx space-data-module compile --manifest ./manifest.json --source ./src/module.c --out ./dist/module.wasm
540
+ npx space-data-module compile --manifest ./manifest.json --source ./src/module.c --out ./dist/isomorphic/module.wasm
499
541
 
500
542
  # Sign and encrypt a deployment payload
501
- npx space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --json
543
+ npx space-data-module protect --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm --json
502
544
 
503
545
  # Emit a single-file bundled wasm
504
- npx space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --single-file-bundle --out ./dist/module.bundle.wasm
546
+ npx space-data-module protect --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm --single-file-bundle --out ./dist/module.bundle.wasm
505
547
  ```
506
548
 
507
549
  ## Module Lab
@@ -534,8 +576,11 @@ npm run check:compliance
534
576
  ```
535
577
 
536
578
  Node.js `>=20` is required. The compiler uses `sdn-emception` and `flatc-wasm`
537
- by default for the embedded toolchain path. WasmEdge pthread builds require a
538
- system Emscripten toolchain on `PATH`.
579
+ by default for the embedded toolchain path. For multi-repo module builds, use a
580
+ repo-local `deps/emsdk` checkout by default instead of Homebrew or any other
581
+ machine-global Emscripten install. Treat `PATH` Emscripten as an explicit escape
582
+ hatch, not the default. WasmEdge pthread builds still require a system
583
+ Emscripten toolchain on `PATH`.
539
584
 
540
585
  If another repo needs the same compiler runtime, the package also exposes a
541
586
  shared emception session at `space-data-module-sdk/compiler/emception` with
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "space-data-module-sdk",
3
- "version": "0.5.19",
3
+ "version": "0.5.21",
4
4
  "description": "Module SDK for building, validating, signing, and deploying WebAssembly modules on the Space Data Network.",
5
5
  "type": "module",
6
6
  "types": "./src/index.d.ts",
@@ -53,6 +53,10 @@
53
53
  "test": "node --test",
54
54
  "test:host-surfaces": "node --test test/node-host.test.js test/host-abi.test.js",
55
55
  "test:runtime-matrix": "SPACE_DATA_MODULE_SDK_ENABLE_RUNTIME_MATRIX=1 node --test test/runtime-matrix.test.js",
56
+ "test:stream-ingest": "node --test test/runtime-host-stream-ingest.test.js",
57
+ "test:module-stream": "node --test test/module-flatbuffer-stream-pump.test.js",
58
+ "benchmark:stream-1gib": "SPACE_DATA_MODULE_SDK_ENABLE_1GB_STREAM_TEST=1 node --test test/runtime-host-stream-ingest.test.js",
59
+ "benchmark:module-stream-1gib": "SPACE_DATA_MODULE_SDK_ENABLE_1GB_MODULE_STREAM_TEST=1 node --test test/module-flatbuffer-stream-pump.test.js",
56
60
  "start:lab": "node ./lab/server.js",
57
61
  "check:compliance": "node ./bin/space-data-module.js check --repo-root .",
58
62
  "generate:vectors": "node ./examples/single-file-bundle/generate-vectors.mjs",
package/src/AGENTS.md ADDED
@@ -0,0 +1,33 @@
1
+ # AGENTS
2
+
3
+ Apply the root `AGENTS.md` first. This file routes source edits to the correct
4
+ subsystem.
5
+
6
+ ## Source Routing
7
+
8
+ - `src/manifest`, `src/compliance`, `src/standards`: manifest shape, type refs,
9
+ standards validation, and compliance warnings/errors.
10
+ - `src/manifest/AGENTS.md`, `src/compliance/AGENTS.md`, `src/bundle/AGENTS.md`,
11
+ and `src/auth/AGENTS.md` provide narrower rules for those owners.
12
+ - `src/compiler`: compile flow, manifest embedding, runtime targets, toolchains,
13
+ and artifact layout.
14
+ - `src/bundle`: `sds.bundle`, wasm custom sections, and vector compatibility.
15
+ - `src/host`: Node host, browser shims, `sdn_host`, WasmEdge runners, and
16
+ isomorphic loaders.
17
+ - `src/runtime-host`: FlatSQL-backed runtime host, row/region identity, and
18
+ binary stream ingest.
19
+ - `src/testing`: harnesses, browser module loading, process invoke clients, and
20
+ module-owned FlatBuffer stream pumps.
21
+ - `src/auth`, `src/transport`, `src/deployment`: signing, encryption, REC/PNM/
22
+ ENC records, deployment plans, and publication protection.
23
+ - `src/generated`: generated code. Do not hand-edit unless regeneration is part
24
+ of the task and you update the generation path or fixtures as needed.
25
+
26
+ ## Working Rules
27
+
28
+ - Keep module contracts portable unless the task is explicitly runtime-specific.
29
+ - Prefer changes in the smallest subsystem that owns the behavior.
30
+ - If a change crosses compiler, host, and docs, update each owner explicitly
31
+ rather than hiding policy in one layer.
32
+ - Add or update tests near the owning subsystem and then wire them into the
33
+ repo-level verification set in `test/AGENTS.md`.
@@ -0,0 +1,25 @@
1
+ # AGENTS
2
+
3
+ You are in `src/auth`, which owns deployment authorization, signing helpers,
4
+ and related record handling used by module distribution.
5
+
6
+ ## Rules
7
+
8
+ - Keep signing and authorization flows deterministic and explicit.
9
+ - Do not invent app-local signature records when a repo-level contract belongs
10
+ here.
11
+ - When auth changes affect transport or deployment plans, update the related
12
+ surfaces together rather than splitting incompatible changes across repos.
13
+
14
+ ## Related Surfaces
15
+
16
+ - `src/transport`
17
+ - `src/deployment`
18
+ - `test/transport-records.test.js`
19
+ - `test/deployment-plan.test.js`
20
+
21
+ ## Check Before You Finish
22
+
23
+ - `npm test`
24
+ - `npm run check:compliance`
25
+ - `node --test test/transport-records.test.js test/deployment-plan.test.js`
package/src/browser.js CHANGED
@@ -6,9 +6,11 @@ export * from "./capabilities.js";
6
6
  export * from "./deployment/index.js";
7
7
  export * from "./invoke/index.js";
8
8
  export * from "./runtime/index.js";
9
+ export * from "./runtime-host/index.js";
9
10
  export * from "./host/browserHost.js";
10
11
  export * from "./host/browserEdgeShims.js";
11
12
  export * from "./host/wasiShim.js";
12
13
  export * from "./host/abi.js";
13
14
  export * from "./host/isomorphicLoader.js";
14
15
  export { createBrowserModuleHarness, detectArtifactProfile } from "./testing/browserModuleHarness.js";
16
+ export { createModuleFlatBufferStreamPump } from "./testing/moduleFlatbufferStreamPump.js";
@@ -0,0 +1,21 @@
1
+ # AGENTS
2
+
3
+ You are in `src/bundle`, which owns `sds.bundle` and the wasm custom-section
4
+ packaging rules.
5
+
6
+ ## Rules
7
+
8
+ - `sds.bundle` is the single-file delivery format.
9
+ - Put bundle data in a wasm custom section; do not append raw bytes after the
10
+ wasm binary.
11
+ - Keep parser, encoder, CLI behavior, vectors, and cross-language fixtures in
12
+ sync.
13
+ - If the bundle contract changes, regenerate vectors and update the bundle docs
14
+ and demos in the same change.
15
+
16
+ ## Check Before You Finish
17
+
18
+ - `npm test`
19
+ - `npm run check:compliance`
20
+ - `node --test test/module-bundle.test.js test/module-bundle-vectors.test.js test/module-bundle-cli.test.js test/module-bundle-go.test.js test/module-bundle-python.test.js`
21
+ - `npm run generate:vectors` when vectors or the contract changed
@@ -0,0 +1,35 @@
1
+ # AGENTS
2
+
3
+ Apply the root and `src/AGENTS.md` files first.
4
+
5
+ ## Area Ownership
6
+
7
+ This directory owns source-to-wasm compile behavior, manifest embedding, runtime
8
+ target inference, toolchain integration, and canonical artifact layout.
9
+
10
+ ## Compiler Rules
11
+
12
+ - Preserve the canonical manifest accessor exports and validate them together
13
+ with compiled wasm.
14
+ - For shared browser/WasmEdge modules, emit the standalone artifact at
15
+ `dist/isomorphic/module.wasm`.
16
+ - Optional browser adapters belong under `dist/browser/`.
17
+ - Do not make browser/WasmEdge shared artifacts depend on pthread-style `env.*`
18
+ imports.
19
+ - Prefer repo-local Emscripten or `sdn-emception`. Do not assume Homebrew
20
+ toolchains.
21
+ - If you change runtime-target inference, keep the browser/WasmEdge shared path
22
+ on the standalone profile unless the docs and tests are updated deliberately.
23
+
24
+ ## Key Files
25
+
26
+ - `compileModule.js`
27
+ - `compileModuleFromSource.js`
28
+ - `emception*.js`
29
+ - `invokeGlue.js`
30
+
31
+ ## Verification
32
+
33
+ - `node --test test/module-sdk.test.js test/compliance.test.js`
34
+ - `node --test test/browser-harness.test.js test/isomorphic-loader.test.js`
35
+ - `node --test test/compiler-emception-subpath-export.test.js`
@@ -0,0 +1,18 @@
1
+ # AGENTS
2
+
3
+ You are in `src/compliance`, which defines what counts as a compliant module or
4
+ artifact.
5
+
6
+ ## Rules
7
+
8
+ - Treat this directory as policy, not convenience code.
9
+ - When tightening validation, update tests and error messages together so the
10
+ failure mode is actionable.
11
+ - When relaxing validation, make sure the compiler, docs, and examples still
12
+ describe the broader contract accurately.
13
+
14
+ ## Check Before You Finish
15
+
16
+ - `npm test`
17
+ - `npm run check:compliance`
18
+ - `node --test test/module-sdk.test.js test/compliance.test.js`
@@ -0,0 +1,39 @@
1
+ # AGENTS
2
+
3
+ Apply the root and `src/AGENTS.md` files first.
4
+
5
+ ## Area Ownership
6
+
7
+ This directory owns the Node reference host, browser host, browser edge shims,
8
+ WASI shims, the sync `sdn_host` ABI bridge, WasmEdge launch planning, and the
9
+ isomorphic loader path.
10
+
11
+ ## Host Rules
12
+
13
+ - Shared browser/WasmEdge artifacts must stay within standalone WASI plus the
14
+ optional `sdn_host` bridge. Raw WasmEdge-native extension imports are not
15
+ browser-isomorphic.
16
+ - Browser shims are host-side replacements, not a promise that the guest can
17
+ import WasmEdge-native sockets, TLS, or filesystem extensions directly.
18
+ - Keep the sync guest-host ABI narrow and deliberate. Avoid adding new hostcalls
19
+ unless the Node host, browser path, docs, and tests all agree on semantics.
20
+ - Scope host capabilities tightly: filesystem roots, network allowlists, TLS,
21
+ exec, timers, and crypto should stay explicit.
22
+
23
+ ## Key Files
24
+
25
+ - `abi.js`
26
+ - `browserEdgeShims.js`
27
+ - `browserHost.js`
28
+ - `isomorphicLoader.js`
29
+ - `nodeHost.js`
30
+ - `wasiShim.js`
31
+
32
+ ## Verification
33
+
34
+ - `node --test test/node-host.test.js test/host-abi.test.js`
35
+ - `node --test test/browser-harness.test.js test/isomorphic-loader.test.js`
36
+ - `node --test test/process-invoke.test.js`
37
+ - `node --test test/wasmedge-runner-build.test.js`
38
+ - `node --test test/wasmedge-runner-runtime.test.js` when WasmEdge behavior is
39
+ part of the change
@@ -18,6 +18,123 @@ const isBrowser =
18
18
  typeof globalThis.window !== "undefined" &&
19
19
  typeof globalThis.document !== "undefined";
20
20
 
21
+ async function createWasmEdgeCommandHarness(options = {}) {
22
+ const [
23
+ { spawn },
24
+ { readFile },
25
+ pathModule,
26
+ { encodePluginInvokeRequest, decodePluginInvokeResponse },
27
+ { buildWasmEdgeSpawnEnv },
28
+ { DefaultInvokeExports },
29
+ { toUint8Array },
30
+ ] = await Promise.all([
31
+ import("node:child_process"),
32
+ import("node:fs/promises"),
33
+ import("node:path"),
34
+ import("../invoke/codec.js"),
35
+ import("../testing/processInvoke.js"),
36
+ import("../runtime/constants.js"),
37
+ import("../runtime/bufferLike.js"),
38
+ ]);
39
+ const path = pathModule.default ?? pathModule;
40
+ const wasmPath = path.resolve(String(options.wasmSource));
41
+ const wasmBytes = await readFile(wasmPath);
42
+ const inspection = await inspectModule(wasmBytes);
43
+
44
+ if (!inspection.exports.includes(DefaultInvokeExports.commandSymbol)) {
45
+ throw new Error(
46
+ "Standalone WasmEdge loading requires a command-surface artifact with the _start export.",
47
+ );
48
+ }
49
+
50
+ const command = options.wasmEdgeBinary ?? "wasmedge";
51
+ const args = [
52
+ ...(options.enableThreads === false ? [] : ["--enable-threads"]),
53
+ wasmPath,
54
+ ...(Array.isArray(options.args) ? options.args : []),
55
+ ];
56
+ const launchPlan = {
57
+ command,
58
+ args,
59
+ env: buildWasmEdgeSpawnEnv(options.env),
60
+ cwd: options.cwd ?? process.cwd(),
61
+ wasmPath,
62
+ };
63
+
64
+ async function invokeRaw(requestBytes) {
65
+ const normalizedRequest = toUint8Array(requestBytes);
66
+ if (!normalizedRequest) {
67
+ throw new TypeError(
68
+ "Expected Uint8Array, ArrayBufferView, or ArrayBuffer request bytes.",
69
+ );
70
+ }
71
+
72
+ return new Promise((resolve, reject) => {
73
+ const child = spawn(launchPlan.command, launchPlan.args, {
74
+ cwd: launchPlan.cwd,
75
+ env: launchPlan.env,
76
+ stdio: ["pipe", "pipe", "pipe"],
77
+ });
78
+ const stdoutChunks = [];
79
+ const stderrChunks = [];
80
+
81
+ function formatFailure(message, cause = null) {
82
+ const stderrText = Buffer.concat(stderrChunks).toString("utf8").trim();
83
+ const details = stderrText ? `${message}\n${stderrText}` : message;
84
+ return cause ? new Error(details, { cause }) : new Error(details);
85
+ }
86
+
87
+ child.stdout.on("data", (chunk) => {
88
+ stdoutChunks.push(Buffer.from(chunk));
89
+ });
90
+ child.stderr.on("data", (chunk) => {
91
+ stderrChunks.push(Buffer.from(chunk));
92
+ });
93
+ child.on("error", (error) => {
94
+ reject(
95
+ formatFailure(
96
+ "Failed to launch WasmEdge command harness.",
97
+ error,
98
+ ),
99
+ );
100
+ });
101
+ child.on("close", (code, signal) => {
102
+ if (code !== 0 || signal !== null) {
103
+ reject(
104
+ formatFailure(
105
+ `WasmEdge command harness exited with ${
106
+ signal ? `signal ${signal}` : `code ${code}`
107
+ }.`,
108
+ ),
109
+ );
110
+ return;
111
+ }
112
+ resolve(new Uint8Array(Buffer.concat(stdoutChunks)));
113
+ });
114
+ child.stdin.end(Buffer.from(normalizedRequest));
115
+ });
116
+ }
117
+
118
+ return {
119
+ runtime: {
120
+ kind: "wasmedge",
121
+ profile: inspection.profile,
122
+ surface: "command",
123
+ },
124
+ launchPlan,
125
+ invokeRaw,
126
+ async invoke(request = {}) {
127
+ const requestBytes = encodePluginInvokeRequest(request);
128
+ const responseBytes = await invokeRaw(requestBytes);
129
+ return decodePluginInvokeResponse(responseBytes);
130
+ },
131
+ readManifest() {
132
+ return null;
133
+ },
134
+ async destroy() {},
135
+ };
136
+ }
137
+
21
138
  /**
22
139
  * Load a WASM module isomorphically.
23
140
  *
@@ -46,6 +163,22 @@ export async function loadModule(options = {}) {
46
163
 
47
164
  const runtimeKind = options.runtimeKind ?? "wasmedge";
48
165
  if (runtimeKind === "wasmedge") {
166
+ const runtimeHostRequested =
167
+ String(options.hostProfile ?? "").trim().toLowerCase() === "runtime-host" ||
168
+ Array.isArray(options.modules) ||
169
+ (typeof options.defaultModuleId === "string" &&
170
+ options.defaultModuleId.trim().length > 0);
171
+ if (!runtimeHostRequested && !options.wasmEdgeRunnerBinary) {
172
+ const { readFile } = await import("node:fs/promises");
173
+ const inspection = await inspectModule(await readFile(source));
174
+ if (
175
+ (inspection.profile === "standalone" || inspection.profile === "sdn-abi") &&
176
+ inspection.exports.includes("_start")
177
+ ) {
178
+ return createWasmEdgeCommandHarness(options);
179
+ }
180
+ }
181
+
49
182
  return createModuleHarness({
50
183
  runtime: {
51
184
  kind: "wasmedge",
@@ -7,7 +7,8 @@
7
7
  *
8
8
  * Covers the 11 imports observed across all SDN plugin standalone builds:
9
9
  * clock_time_get, fd_write, fd_read, fd_close, fd_seek, fd_fdstat_get,
10
- * environ_sizes_get, environ_get, proc_exit, args_get, args_sizes_get
10
+ * environ_sizes_get, environ_get, proc_exit, args_get, args_sizes_get,
11
+ * random_get
11
12
  */
12
13
 
13
14
  const ERRNO_SUCCESS = 0;
@@ -38,6 +39,7 @@ export function createBrowserWasiShim(options = {}) {
38
39
  now: () => Date.now(),
39
40
  timeOrigin: 0,
40
41
  };
42
+ const cryptoApi = options.crypto ?? globalThis.crypto ?? null;
41
43
  const stdoutChunks = [];
42
44
  const stderrChunks = [];
43
45
  let stdinOffset = 0;
@@ -206,6 +208,14 @@ export function createBrowserWasiShim(options = {}) {
206
208
  return ERRNO_SUCCESS;
207
209
  }
208
210
 
211
+ function random_get(bufPtr, bufLen) {
212
+ if (!cryptoApi?.getRandomValues) {
213
+ return ERRNO_NOSYS;
214
+ }
215
+ cryptoApi.getRandomValues(mem8().subarray(bufPtr, bufPtr + bufLen));
216
+ return ERRNO_SUCCESS;
217
+ }
218
+
209
219
  function proc_exit(code) {
210
220
  if (logOutput) {
211
221
  flushOutput();
@@ -259,6 +269,7 @@ export function createBrowserWasiShim(options = {}) {
259
269
  environ_get,
260
270
  args_sizes_get,
261
271
  args_get,
272
+ random_get,
262
273
  proc_exit,
263
274
  },
264
275
  },
package/src/index.d.ts CHANGED
@@ -277,6 +277,22 @@ export interface RuntimeRowQueryResult {
277
277
  rowCount: number;
278
278
  }
279
279
 
280
+ export interface FlatBufferStreamIngestStats {
281
+ bytesReceived: number;
282
+ chunksReceived: number;
283
+ framesDecoded: number;
284
+ framesAppended: number;
285
+ framesRouted: number;
286
+ parseErrors: number;
287
+ }
288
+
289
+ export interface FlatBufferStreamIngestContext {
290
+ rawFileIdentifier: string;
291
+ schemaFileId: string;
292
+ rows: FlatSqlRuntimeStore;
293
+ stats: FlatBufferStreamIngestStats;
294
+ }
295
+
280
296
  export interface FlatSqlRuntimeStore {
281
297
  appendRow(options: { schemaFileId: string; payload?: unknown }): RowHandle;
282
298
  listRows(schemaFileId?: string | null): RuntimeRowView[];
@@ -284,6 +300,13 @@ export interface FlatSqlRuntimeStore {
284
300
  resolveRow(handle: RowHandle): RuntimeRowView | null;
285
301
  }
286
302
 
303
+ export interface FlatBufferStreamIngestor {
304
+ rows: FlatSqlRuntimeStore;
305
+ stats: FlatBufferStreamIngestStats;
306
+ pushBytes(data: Uint8Array | ArrayBuffer | ArrayBufferView): number;
307
+ finish(): 0;
308
+ }
309
+
287
310
  export interface RuntimeRegionDescriptor {
288
311
  regionId: number;
289
312
  layoutId: string;
@@ -388,6 +411,25 @@ export interface RuntimeHost {
388
411
  moduleRegistry: RuntimeModuleRegistry;
389
412
  }
390
413
 
414
+ export function createFlatBufferStreamIngestor(options?: {
415
+ rows?: FlatSqlRuntimeStore;
416
+ frameRouter?:
417
+ | ((
418
+ payload: Uint8Array,
419
+ context: FlatBufferStreamIngestContext,
420
+ ) => boolean | void)
421
+ | Record<
422
+ string,
423
+ (
424
+ payload: Uint8Array,
425
+ context: FlatBufferStreamIngestContext,
426
+ ) => boolean | void
427
+ >;
428
+ appendFrame?: (
429
+ payload: Uint8Array,
430
+ context: FlatBufferStreamIngestContext,
431
+ ) => void;
432
+ }): FlatBufferStreamIngestor;
391
433
  export function createFlatSqlRuntimeStore(): FlatSqlRuntimeStore;
392
434
  export function createRuntimeRegionStore(): RuntimeRegionStore;
393
435
  export function createModuleRegistry(): RuntimeModuleRegistry;
@@ -1563,6 +1605,64 @@ export function createBrowserModuleHarness(options?: {
1563
1605
  };
1564
1606
  logOutput?: boolean;
1565
1607
  }): Promise<BrowserModuleHarness>;
1608
+ export interface ModuleFlatBufferStreamPumpStats {
1609
+ bytesReceived: number;
1610
+ chunksReceived: number;
1611
+ framesDecoded: number;
1612
+ framesInvoked: number;
1613
+ invokes: number;
1614
+ parseErrors: number;
1615
+ }
1616
+ export interface ModuleFlatBufferStreamPumpContext {
1617
+ rawFileIdentifier: string;
1618
+ schemaFileId: string;
1619
+ methodId: string;
1620
+ portId: string;
1621
+ streamId: number;
1622
+ sequence: number;
1623
+ stats: ModuleFlatBufferStreamPumpStats;
1624
+ }
1625
+ export interface ModuleFlatBufferStreamPump {
1626
+ stats: ModuleFlatBufferStreamPumpStats;
1627
+ lastResponse: PluginInvokeResponseEnvelope | null;
1628
+ pushBytes(data: Uint8Array | ArrayBuffer | ArrayBufferView): Promise<number>;
1629
+ finish(): Promise<PluginInvokeResponseEnvelope | null>;
1630
+ }
1631
+ export function createModuleFlatBufferStreamPump(options: {
1632
+ harness?: {
1633
+ invoke(
1634
+ request: PluginInvokeRequestEnvelope,
1635
+ ): Promise<PluginInvokeResponseEnvelope>;
1636
+ };
1637
+ invoke?: (
1638
+ request: PluginInvokeRequestEnvelope,
1639
+ ) => Promise<PluginInvokeResponseEnvelope>;
1640
+ methodId: string;
1641
+ portId: string;
1642
+ maxFramesPerInvoke?: number;
1643
+ streamId?: number;
1644
+ sequenceStart?: number;
1645
+ typeResolver?: (
1646
+ payload: Uint8Array,
1647
+ context: ModuleFlatBufferStreamPumpContext,
1648
+ ) => PayloadTypeRef | null | undefined;
1649
+ frameTemplate?:
1650
+ | Partial<InvokeFrame>
1651
+ | ((
1652
+ payload: Uint8Array,
1653
+ context: ModuleFlatBufferStreamPumpContext,
1654
+ ) => Partial<InvokeFrame> | null | undefined);
1655
+ onResponse?: (
1656
+ response: PluginInvokeResponseEnvelope,
1657
+ context: {
1658
+ methodId: string;
1659
+ portId: string;
1660
+ frames: InvokeFrame[];
1661
+ isFinalBatch: boolean;
1662
+ stats: ModuleFlatBufferStreamPumpStats;
1663
+ },
1664
+ ) => void | Promise<void>;
1665
+ }): ModuleFlatBufferStreamPump;
1566
1666
  export function loadModule(options?: {
1567
1667
  wasmSource: Uint8Array | ArrayBuffer | string | WebAssembly.Module | unknown;
1568
1668
  host?: BrowserHost;
@@ -0,0 +1,18 @@
1
+ # AGENTS
2
+
3
+ You are in `src/manifest`, which owns the manifest schema codecs and
4
+ normalization rules.
5
+
6
+ ## Rules
7
+
8
+ - Keep manifest encode/decode round-trips stable.
9
+ - Prefer canonical SDS schema names and file identifiers; do not add repo-local
10
+ aliases when a standards name already exists.
11
+ - If you change a manifest field or normalization rule, update the compiler,
12
+ compliance checks, and any affected examples/tests together.
13
+
14
+ ## Check Before You Finish
15
+
16
+ - `npm test`
17
+ - `npm run check:compliance`
18
+ - `node --test test/module-sdk.test.js test/compliance.test.js`