space-data-module-sdk 0.5.18 → 0.5.21

package/README.md

@@ -52,6 +52,10 @@ The minimal host surface lives under `src/runtime-host/` and covers three
 responsibilities:
 
 - `createFlatSqlRuntimeStore()` for append-only row handles and row resolution
+- `createFlatBufferStreamIngestor()` for little-endian size-prefixed FlatBuffer
+  transport ingest into host-owned row storage without JSON transcoding
+- `createModuleFlatBufferStreamPump()` for feeding those same FlatBuffer stream
+  chunks into a resident stateful module instance without JSON envelopes
 - `createRuntimeRegionStore()` for host-allocated aligned-binary regions and
   externally backed record-view descriptors
 - `createModuleRegistry()` for dynamic install/load/unload/invoke
@@ -60,6 +64,9 @@ responsibilities:
 OrbPro layers entity/view helpers on top of that host instead of inventing a
 separate durable identity model.
 
+The canonical FlatBuffer-to-FlatSQL streaming contract is documented in
+[`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md).
+
 ## Module Artifact Model
 
 A compliant module built with this SDK is always a valid `.wasm` artifact with:
@@ -189,6 +196,11 @@ the preferred server-side target when the guest needs network-oriented runtime
 features such as sockets or TLS. Plain `wasi` remains the strict portability
 baseline; `wasmedge` is the practical higher-capability target.
 
+If a manifest declares `runtimeTargets: ["browser", "wasmedge"]`, this SDK
+treats that as the explicit "one binary for both" profile. That pair now
+defaults to a shared `single-thread` artifact so the compiled wasm can be loaded
+unchanged by the browser harness and the WasmEdge harness.
+
 ## WasmEdge Pthreads
 
 `space-data-module-sdk` is also the source of truth for module thread-model
@@ -197,6 +209,7 @@ selection.
 - `compileModuleFromSource({ threadModel })` accepts an explicit thread model.
 - If `threadModel` is omitted, the SDK resolves it from `manifest.runtimeTargets`.
 - `runtimeTargets: ["wasmedge"]` defaults to `emscripten-pthreads`.
+- `runtimeTargets: ["browser", "wasmedge"]` defaults to `single-thread`.
 - Other targets currently default to `single-thread`.
 
 WasmEdge-targeted pthread builds do not use the embedded `sdn-emception`
@@ -214,11 +227,38 @@ If a runtime cannot interoperate with the guest pthread contract directly,
 document that as a wrapper requirement instead of changing the guest artifact
 semantics.
 
+## Browser + WasmEdge Isomorphism
+
+The supported isomorphic profile and browser edge shims are documented in
+[`docs/browser-wasmedge-isomorphic.md`](./docs/browser-wasmedge-isomorphic.md).
+
+The checked-in same-artifact demo lives in
+[`examples/isomorphic-loader`](./examples/isomorphic-loader):
+
+- [`build-demo.mjs`](./examples/isomorphic-loader/build-demo.mjs) compiles the
+  shared artifact
+- [`browser-demo.html`](./examples/isomorphic-loader/browser-demo.html) and
+  [`browser-demo.mjs`](./examples/isomorphic-loader/browser-demo.mjs) load that
+  artifact in the browser harness with browser edge shims
+- [`wasmedge-demo.mjs`](./examples/isomorphic-loader/wasmedge-demo.mjs) loads
+  that same artifact in WasmEdge
+
+Large streamed FlatBuffer ingest should use either the runtime-host transport
+path or the resident-module stream pump, not one giant invoke envelope. See
+[`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md)
+for the recommended browser/WasmEdge/FlatSQL split and the canonical
+direct-binary ingest contract.
+
 ## Testing
 
 This repo now exposes a manifest-driven harness generator from
 `space-data-module-sdk/testing` and two complementary integration suites:
 
+- browser/isomorphic helpers:
+  - `createBrowserModuleHarness(...)`
+  - `detectArtifactProfile(...)`
+  - `loadModule(...)`
+
 - shared process-level helpers for command-surface runtimes:
   - `createPluginInvokeProcessClient(...)`
   - `resolveWasmEdgePluginLaunchPlan(...)`
@@ -233,10 +273,23 @@ This repo now exposes a manifest-driven harness generator from
 - `npm run test:host-surfaces`
   - authoritative Node-host coverage for HTTP, TCP, UDP, TLS, WebSocket, MQTT,
     process execution, timers, filesystem, and the sync `sdn_host` ABI
+- `npm run test:stream-ingest`
+  - correctness coverage for chunked size-prefixed FlatBuffer ingest into the
+    runtime-host row store
+- `npm run test:module-stream`
+  - correctness coverage for chunked FlatBuffer streaming into a resident
+    direct-surface module instance
+- `npm run benchmark:stream-1gib`
+  - env-gated local stress benchmark for 1 GiB total FlatBuffer transport ingest
+- `npm run benchmark:module-stream-1gib`
+  - env-gated local stress benchmark for 1 GiB total chunked module stream-pump
+    ingest
 
 The detailed edge cases and the current WASI-vs-host portability boundary are
 documented in
-[`docs/testing-harness.md`](./docs/testing-harness.md).
+[`docs/testing-harness.md`](./docs/testing-harness.md). The canonical
+FlatBuffer-to-FlatSQL ingest contract is documented in
+[`docs/flatsql-streaming-standard.md`](./docs/flatsql-streaming-standard.md).
 
 ## Install
 
@@ -410,6 +463,22 @@ For npm packages, the simplest form is:
 }
 ```
 
+For module repos themselves, the canonical compiled artifact should live at:
+
+```text
+dist/isomorphic/module.wasm
+```
+
+If the repo also ships a direct browser adapter, publish it under:
+
+```text
+dist/browser/module.js
+dist/browser/module.wasm
+```
+
+That keeps the shared browser/WasmEdge artifact path stable across repos while
+leaving the browser-specific adapter optional.
+
 When publication metadata is published in the same file, it belongs in an
 appended SDS `REC` trailer. Loaders scan from the end of the protected blob,
 resolve `PNM` / `ENC`, strip or decrypt as needed, and only then instantiate
@@ -465,16 +534,16 @@ Manifests can also declare coarse runtime targets for planning and compliance:
 
 ```bash
 # Validate a manifest + wasm pair
-npx space-data-module check --manifest ./manifest.json --wasm ./dist/module.wasm
+npx space-data-module check --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm
 
 # Compile C/C++ source and embed the manifest
-npx space-data-module compile --manifest ./manifest.json --source ./src/module.c --out ./dist/module.wasm
+npx space-data-module compile --manifest ./manifest.json --source ./src/module.c --out ./dist/isomorphic/module.wasm
 
 # Sign and encrypt a deployment payload
-npx space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --json
+npx space-data-module protect --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm --json
 
 # Emit a single-file bundled wasm
-npx space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --single-file-bundle --out ./dist/module.bundle.wasm
+npx space-data-module protect --manifest ./manifest.json --wasm ./dist/isomorphic/module.wasm --single-file-bundle --out ./dist/module.bundle.wasm
 ```
 
 ## Module Lab
@@ -507,8 +576,11 @@ npm run check:compliance
 ```
 
 Node.js `>=20` is required. The compiler uses `sdn-emception` and `flatc-wasm`
-by default for the embedded toolchain path. WasmEdge pthread builds require a
-system Emscripten toolchain on `PATH`.
+by default for the embedded toolchain path. For multi-repo module builds, use a
+repo-local `deps/emsdk` checkout by default instead of Homebrew or any other
+machine-global Emscripten install. Treat `PATH` Emscripten as an explicit escape
+hatch, not the default. WasmEdge pthread builds still require a system
+Emscripten toolchain on `PATH`.
 
 If another repo needs the same compiler runtime, the package also exposes a
 shared emception session at `space-data-module-sdk/compiler/emception` with

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "space-data-module-sdk",
-  "version": "0.5.18",
+  "version": "0.5.21",
   "description": "Module SDK for building, validating, signing, and deploying WebAssembly modules on the Space Data Network.",
   "type": "module",
   "types": "./src/index.d.ts",
@@ -35,6 +35,11 @@
       "types": "./src/index.d.ts",
       "default": "./src/runtime-host/index.js"
     },
+    "./host/browser": "./src/host/browserHost.js",
+    "./host/browser-edge-shims": "./src/host/browserEdgeShims.js",
+    "./host/wasi-shim": "./src/host/wasiShim.js",
+    "./host/isomorphic": "./src/host/isomorphicLoader.js",
+    "./testing/browser": "./src/testing/browserModuleHarness.js",
     "./testing": "./src/testing/index.js",
     "./standards": "./src/standards/index.js",
     "./schemas/*": "./schemas/*"
@@ -48,6 +53,10 @@
     "test": "node --test",
     "test:host-surfaces": "node --test test/node-host.test.js test/host-abi.test.js",
     "test:runtime-matrix": "SPACE_DATA_MODULE_SDK_ENABLE_RUNTIME_MATRIX=1 node --test test/runtime-matrix.test.js",
+    "test:stream-ingest": "node --test test/runtime-host-stream-ingest.test.js",
+    "test:module-stream": "node --test test/module-flatbuffer-stream-pump.test.js",
+    "benchmark:stream-1gib": "SPACE_DATA_MODULE_SDK_ENABLE_1GB_STREAM_TEST=1 node --test test/runtime-host-stream-ingest.test.js",
+    "benchmark:module-stream-1gib": "SPACE_DATA_MODULE_SDK_ENABLE_1GB_MODULE_STREAM_TEST=1 node --test test/module-flatbuffer-stream-pump.test.js",
     "start:lab": "node ./lab/server.js",
     "check:compliance": "node ./bin/space-data-module.js check --repo-root .",
     "generate:vectors": "node ./examples/single-file-bundle/generate-vectors.mjs",

package/src/AGENTS.md

@@ -0,0 +1,33 @@
+# AGENTS
+
+Apply the root `AGENTS.md` first. This file routes source edits to the correct
+subsystem.
+
+## Source Routing
+
+- `src/manifest`, `src/compliance`, `src/standards`: manifest shape, type refs,
+  standards validation, and compliance warnings/errors.
+- `src/manifest/AGENTS.md`, `src/compliance/AGENTS.md`, `src/bundle/AGENTS.md`,
+  and `src/auth/AGENTS.md` provide narrower rules for those owners.
+- `src/compiler`: compile flow, manifest embedding, runtime targets, toolchains,
+  and artifact layout.
+- `src/bundle`: `sds.bundle`, wasm custom sections, and vector compatibility.
+- `src/host`: Node host, browser shims, `sdn_host`, WasmEdge runners, and
+  isomorphic loaders.
+- `src/runtime-host`: FlatSQL-backed runtime host, row/region identity, and
+  binary stream ingest.
+- `src/testing`: harnesses, browser module loading, process invoke clients, and
+  module-owned FlatBuffer stream pumps.
+- `src/auth`, `src/transport`, `src/deployment`: signing, encryption, REC/PNM/
+  ENC records, deployment plans, and publication protection.
+- `src/generated`: generated code. Do not hand-edit unless regeneration is part
+  of the task and you update the generation path or fixtures as needed.
+
+## Working Rules
+
+- Keep module contracts portable unless the task is explicitly runtime-specific.
+- Prefer changes in the smallest subsystem that owns the behavior.
+- If a change crosses compiler, host, and docs, update each owner explicitly
+  rather than hiding policy in one layer.
+- Add or update tests near the owning subsystem and then wire them into the
+  repo-level verification set in `test/AGENTS.md`.

package/src/auth/AGENTS.md

@@ -0,0 +1,25 @@
+# AGENTS
+
+You are in `src/auth`, which owns deployment authorization, signing helpers,
+and related record handling used by module distribution.
+
+## Rules
+
+- Keep signing and authorization flows deterministic and explicit.
+- Do not invent app-local signature records when a repo-level contract belongs
+  here.
+- When auth changes affect transport or deployment plans, update the related
+  surfaces together rather than splitting incompatible changes across repos.
+
+## Related Surfaces
+
+- `src/transport`
+- `src/deployment`
+- `test/transport-records.test.js`
+- `test/deployment-plan.test.js`
+
+## Check Before You Finish
+
+- `npm test`
+- `npm run check:compliance`
+- `node --test test/transport-records.test.js test/deployment-plan.test.js`

package/src/browser.js

@@ -6,3 +6,11 @@ export * from "./capabilities.js";
 export * from "./deployment/index.js";
 export * from "./invoke/index.js";
 export * from "./runtime/index.js";
+export * from "./runtime-host/index.js";
+export * from "./host/browserHost.js";
+export * from "./host/browserEdgeShims.js";
+export * from "./host/wasiShim.js";
+export * from "./host/abi.js";
+export * from "./host/isomorphicLoader.js";
+export { createBrowserModuleHarness, detectArtifactProfile } from "./testing/browserModuleHarness.js";
+export { createModuleFlatBufferStreamPump } from "./testing/moduleFlatbufferStreamPump.js";

package/src/bundle/AGENTS.md

@@ -0,0 +1,21 @@
+# AGENTS
+
+You are in `src/bundle`, which owns `sds.bundle` and the wasm custom-section
+packaging rules.
+
+## Rules
+
+- `sds.bundle` is the single-file delivery format.
+- Put bundle data in a wasm custom section; do not append raw bytes after the
+  wasm binary.
+- Keep parser, encoder, CLI behavior, vectors, and cross-language fixtures in
+  sync.
+- If the bundle contract changes, regenerate vectors and update the bundle docs
+  and demos in the same change.
+
+## Check Before You Finish
+
+- `npm test`
+- `npm run check:compliance`
+- `node --test test/module-bundle.test.js test/module-bundle-vectors.test.js test/module-bundle-cli.test.js test/module-bundle-go.test.js test/module-bundle-python.test.js`
+- `npm run generate:vectors` when vectors or the contract changed

package/src/compiler/AGENTS.md

@@ -0,0 +1,35 @@
+# AGENTS
+
+Apply the root and `src/AGENTS.md` files first.
+
+## Area Ownership
+
+This directory owns source-to-wasm compile behavior, manifest embedding, runtime
+target inference, toolchain integration, and canonical artifact layout.
+
+## Compiler Rules
+
+- Preserve the canonical manifest accessor exports and validate them together
+  with compiled wasm.
+- For shared browser/WasmEdge modules, emit the standalone artifact at
+  `dist/isomorphic/module.wasm`.
+- Optional browser adapters belong under `dist/browser/`.
+- Do not make browser/WasmEdge shared artifacts depend on pthread-style `env.*`
+  imports.
+- Prefer repo-local Emscripten or `sdn-emception`. Do not assume Homebrew
+  toolchains.
+- If you change runtime-target inference, keep the browser/WasmEdge shared path
+  on the standalone profile unless the docs and tests are updated deliberately.
+
+## Key Files
+
+- `compileModule.js`
+- `compileModuleFromSource.js`
+- `emception*.js`
+- `invokeGlue.js`
+
+## Verification
+
+- `node --test test/module-sdk.test.js test/compliance.test.js`
+- `node --test test/browser-harness.test.js test/isomorphic-loader.test.js`
+- `node --test test/compiler-emception-subpath-export.test.js`

package/src/compiler/compileModule.js

@@ -142,6 +142,9 @@ function resolveThreadModel({ manifest, threadModel } = {}) {
         .map((target) => String(target ?? "").trim().toLowerCase())
         .filter(Boolean)
     : [];
+  if (runtimeTargets.includes(RuntimeTarget.BROWSER)) {
+    return ModuleThreadModel.SINGLE_THREAD;
+  }
   if (runtimeTargets.includes(RuntimeTarget.WASMEDGE)) {
     return ModuleThreadModel.EMSCRIPTEN_PTHREADS;
   }

package/src/compliance/AGENTS.md

@@ -0,0 +1,18 @@
+# AGENTS
+
+You are in `src/compliance`, which defines what counts as a compliant module or
+artifact.
+
+## Rules
+
+- Treat this directory as policy, not convenience code.
+- When tightening validation, update tests and error messages together so the
+  failure mode is actionable.
+- When relaxing validation, make sure the compiler, docs, and examples still
+  describe the broader contract accurately.
+
+## Check Before You Finish
+
+- `npm test`
+- `npm run check:compliance`
+- `node --test test/module-sdk.test.js test/compliance.test.js`

package/src/compliance/pluginCompliance.js

@@ -30,7 +30,6 @@ const ExternalInterfaceKindSet = new Set(Object.values(ExternalInterfaceKind));
 const ProtocolRoleSet = new Set(Object.values(ProtocolRole));
 const ProtocolTransportKindSet = new Set(Object.values(ProtocolTransportKind));
 const BrowserIncompatibleCapabilitySet = new Set([
-  "filesystem",
   "pipe",
   "network",
   "tcp",

package/src/host/AGENTS.md

@@ -0,0 +1,39 @@
+# AGENTS
+
+Apply the root and `src/AGENTS.md` files first.
+
+## Area Ownership
+
+This directory owns the Node reference host, browser host, browser edge shims,
+WASI shims, the sync `sdn_host` ABI bridge, WasmEdge launch planning, and the
+isomorphic loader path.
+
+## Host Rules
+
+- Shared browser/WasmEdge artifacts must stay within standalone WASI plus the
+  optional `sdn_host` bridge. Raw WasmEdge-native extension imports are not
+  browser-isomorphic.
+- Browser shims are host-side replacements, not a promise that the guest can
+  import WasmEdge-native sockets, TLS, or filesystem extensions directly.
+- Keep the sync guest-host ABI narrow and deliberate. Avoid adding new hostcalls
+  unless the Node host, browser path, docs, and tests all agree on semantics.
+- Scope host capabilities tightly: filesystem roots, network allowlists, TLS,
+  exec, timers, and crypto should stay explicit.
+
+## Key Files
+
+- `abi.js`
+- `browserEdgeShims.js`
+- `browserHost.js`
+- `isomorphicLoader.js`
+- `nodeHost.js`
+- `wasiShim.js`
+
+## Verification
+
+- `node --test test/node-host.test.js test/host-abi.test.js`
+- `node --test test/browser-harness.test.js test/isomorphic-loader.test.js`
+- `node --test test/process-invoke.test.js`
+- `node --test test/wasmedge-runner-build.test.js`
+- `node --test test/wasmedge-runner-runtime.test.js` when WasmEdge behavior is
+  part of the change