space-data-module-sdk 0.8.4 → 0.8.5

package/README.md

@@ -32,8 +32,8 @@ This repo owns the generic module-side runtime harnesses used across the stack:
 - `buildWasmEdgeEmscriptenPthreadRunner(...)` plus the shared native runner
   source and runner-level pthread smoke test
 
-Flow-specific standalone harnessing lives in `sdn-flow`, which layers flow
-enqueue/drain behavior on top of its compiled standalone runtime host. Package-
+Flow-specific standalone harnessing now lives in host runtime packages that
+layer flow enqueue/drain behavior on top of the SDK runtime host. Package-
 specific validation suites, including conjunction replay/V&V, are expected to
 sit on top of these shared harnesses instead of defining their own runtime
 process model.
@@ -107,16 +107,43 @@ Modules can now declare one or both canonical invoke surfaces in
 - `direct`: in-memory invocation through `plugin_invoke_stream`
 - `command`: WASI command-mode invocation through `_start`
 
-Both surfaces consume and produce the same FlatBuffer envelopes:
-
-- `PluginInvokeRequest`
-- `PluginInvokeResponse`
-
-Those envelopes route SDS payload frames by `portId` using
-`TypedArenaBuffer.fbs` and a shared payload arena. Command mode reads one
-request from `stdin` and writes one response to `stdout`. Direct mode takes the
-same request bytes from guest memory and returns response bytes in guest
-memory.
+Both surfaces consume and produce the SDS `PIV` FlatBuffer envelope
+(`file_identifier "$PIV"`). `PIV.REQUEST` and `PIV.RESPONSE` route SDS payload
+frames by `TAB.PORT_ID` using a shared `PAYLOAD_ARENA`. `TAB.WIRE_FORMAT`
+declares whether each frame body is a regular FlatBuffer or an aligned-binary
+payload. Command mode reads one `PIV` request from `stdin` and writes one `PIV`
+response to `stdout`. Direct mode takes the same request bytes from guest
+memory and returns response bytes in guest memory. Direct-mode hosts must pass
+request and response-length pointers allocated by the module's `plugin_alloc`;
+the generated bridge validates those guest-memory ranges and returns a
+canonical `invalid-request-pointer` response, or `0` for an invalid
+response-length pointer, instead of dereferencing arbitrary addresses. When a
+PIV request has an empty `PAYLOAD_ARENA`, the generated direct bridge treats
+`TAB.OFFSET` as an absolute guest-memory pointer only if that range was
+allocated by `plugin_alloc`; otherwise it fails closed with
+`invalid-request-pointer`. Public JS decoders can materialize the same
+empty-arena TAB offsets by passing `{ externalArena }` to
+`decodePluginInvokeRequest` or `decodePluginInvokeResponse`.
+Public JS encoders can also emit empty-arena PIV descriptors by passing
+`externalArena` to `encodePluginInvokeRequest` or
+`encodePluginInvokeResponse` and supplying frame `offset`/`size` values into
+that arena. Browser direct-mode harness invocation requires `externalArena` to
+be a full view of the module's active SharedArrayBuffer-backed memory and
+passes frame `offset`/`size` descriptors through without copying the selected
+frame bytes into a separate allocation. The browser direct harness also authors
+the PIV request envelope itself into `plugin_alloc`-owned module memory before calling
+`plugin_invoke_stream`; it does not route external-arena direct calls through a
+JS-owned `invokeRaw` request-byte frame. Command-mode invocation rejects
+`externalArena` because stdin has no guest-memory address space.
+
+SDS `TAB` is the canonical transport descriptor. It preserves port id, payload
+offset/size, alignment, wire format, type identity, ownership, mutability, and
+frame id. Older SDK-local invoke descriptors and fields are not part of the
+wire ABI. For stream-pump compatibility, the SDK uses `TAB.FRAME_ID` as packed
+stream bookkeeping: `(sequence << 1) | endOfStream`.
+Source-built modules can set this on an emitted output frame with
+`plugin_set_output_stream_frame(outputIndex, sequence, endOfStream)` or set the
+raw frame id with `plugin_set_output_frame_id(outputIndex, frameId)`.
 
 For simple single-input / single-output methods, command mode also supports a
 raw shortcut:
@@ -174,12 +201,12 @@ This repo currently includes:
   transport, bundle handling, and compilation
 - deterministic REC+MBL conformance vectors under
   [`examples/single-file-bundle/vectors`](./examples/single-file-bundle/vectors)
-- reference Node and browser hosts plus the legacy sync `sdn_host` bridge for
+- reference Node and browser hosts plus the legacy sync `space_data_module_host` bridge for
   sync-safe guest hostcalls
 
 The current host/runtime contract is two-tiered:
 
-- raw guest `sdn_host` imports stay sync-only
+- raw guest `space_data_module_host` imports stay sync-only
 - generic async capabilities such as filesystem, network, IPFS, and protocol
   adapters are exposed through `NodeHost`, `BrowserHost`,
   `createRuntimeHost()` capability registries, and the browser/module harness
@@ -245,6 +272,27 @@ semantics.
 The supported isomorphic profile and browser edge shims are documented in
 [`docs/browser-wasmedge-isomorphic.md`](./docs/browser-wasmedge-isomorphic.md).
 
+GPU-accelerated modules should follow the host-owned GPU capability standard in
+[`docs/gpu-module-abi.md`](./docs/gpu-module-abi.md). The reusable C/C++ layout
+header and starter manifest live in
+[`templates/gpu-module`](./templates/gpu-module).
+
+The GPU surface is intentionally host-owned. A portable module advertises the
+optional `gpu_compute` capability with scope `webgpu.v1`, keeps a correct CPU
+fallback in `dist/isomorphic/module.wasm`, and lets the embedding host choose a
+backend:
+
+- browser hosts use WebGPU through a browser adapter
+- WasmEdge deployments use a native host extension backed by Dawn
+- modules share the SDK C/C++ layout header for dispatch descriptors, result
+  records, buffer roles, and split high/low `f32` representations of `f64`
+  values
+
+The public invoke API should remain domain-specific. GPU command buffers,
+devices, queue ownership, and asynchronous buffer mapping stay behind the host
+capability boundary rather than becoming new sync `space_data_module_host`
+imports.
+
 The checked-in same-artifact demo lives in
 [`examples/isomorphic-loader`](./examples/isomorphic-loader):
 
@@ -285,7 +333,7 @@ This repo now exposes a manifest-driven harness generator from
     clock/time smoke
 - `npm run test:host-surfaces`
   - authoritative Node-host coverage for HTTP, TCP, UDP, TLS, WebSocket, MQTT,
-    process execution, timers, filesystem, and the sync `sdn_host` ABI
+    process execution, timers, filesystem, and the sync `space_data_module_host` ABI
 - `npm run test:stream-ingest`
   - correctness coverage for chunked size-prefixed FlatBuffer ingest into the
     runtime-host row store
@@ -506,11 +554,11 @@ the remaining raw wasm bytes.
 ## Host ABI
 
 This repo also defines the module-facing capability vocabulary and the first
-synchronous hostcall bridge under the import module `sdn_host`.
+synchronous hostcall bridge under the import module `space_data_module_host`.
 
 The current sync import surface is:
 
-- `call_json(operation_ptr, operation_len, payload_ptr, payload_len) -> i32`
+- `call(request_ptr, request_len) -> i32`
 - `response_len() -> i32`
 - `read_response(dst_ptr, dst_len) -> i32`
 - `last_status_code() -> i32`
@@ -530,13 +578,52 @@ includes:
 `protocol_dial` `database` `storage_adapter` `storage_query` `storage_write`
 `context_read` `context_write` `process_exec` `crypto_hash` `crypto_sign`
 `crypto_verify` `crypto_encrypt` `crypto_decrypt` `crypto_key_agreement`
-`crypto_kdf` `wallet_sign` `ipfs` `scene_access` `entity_access`
-`render_hooks`
+`crypto_kdf` `wallet_sign` `ipfs` `gpu_compute` `scene_access`
+`entity_access` `render_hooks`
 
 Manifests can also declare coarse runtime targets for planning and compliance:
 
 `node` `browser` `wasi` `wasmedge` `server` `desktop` `edge`
 
+## Credential Lanes (`secrets:<lane>`)
+
+A module that needs an operator's third-party credential — a Space-Track login,
+a data-provider API password, an account at any service the operator uses —
+reads it at runtime from the node's encrypted credential keystore instead of
+carrying it in the manifest, the flow, or a record.
+
+Lane ids are **operator-defined** (lowercase `[a-z0-9_-]{2,64}`). A few are
+well-known (`spacetrack`, `edc_cpf`, `myintelsat`) because the node ships a
+verifier for them, but an operator may create a lane for any service; take the
+lane id as module configuration, not as a compile-time constant.
+
+Reading a lane requires the capability `secrets:<lane>` **and** an operator
+approval recorded against your module's exact content hash. Approval is per
+lane: `secrets:spacetrack` conveys nothing about any other lane, and the host
+re-checks the exact lane on every call. Without an approval the module is denied
+at load. There is no list or export operation — a guest can never enumerate the
+keystore.
+
+The guest helper is [`src/host/cpp/secretsClient.hpp`](./src/host/cpp/secretsClient.hpp):
+
+```cpp
+#include "sdm_hostcall_wire.hpp"   // must be included first
+#include "secretsClient.hpp"
+
+sdm_secrets::Credential cred;
+if (!sdm_secrets::secrets_get(lane, &cred)) {
+  log_error("provider credential unavailable");   // the CONDITION, never the contents
+  return false;
+}
+const bool ok = provider_login(cred.username, cred.secret);
+sdm_secrets::wipe(&cred);   // on every path, including failures
+```
+
+The credential arrives as plaintext: never log it, never persist it, never
+forward it anywhere but the provider it belongs to, and wipe it as soon as the
+call is done. Full contract, wire shapes, and failure modes:
+[`docs/secrets-capability.md`](./docs/secrets-capability.md).
+
 ## Environment Notes
 
 | Surface | Node.js | Browser |
@@ -586,7 +673,6 @@ inspect the parsed `REC`, `PNM`, and `ENC` records produced by the real
 ## Related Projects
 
 - [`spacedatastandards.org`](https://spacedatastandards.org)
-- [`sdn-flow`](https://github.com/DigitalArsenal/sdn-flow)
 - [`hd-wallet-wasm`](https://github.com/nicktj-dev/hd-wallet-wasm)
 
 ## Development
@@ -609,6 +695,17 @@ generated bindings must come from upstream SDS sources or published SDS
 artifacts generated with `flatc-wasm`. Do not add repo-local shadow `.fbs`
 families or hand-authored generated classes in this repo.
 
+PLG bindings are mirrored from SDS generated artifacts. During cross-repo schema
+work, point the generator at the active SDS checkout:
+
+```sh
+SPACE_DATA_STANDARDS_ROOT=/path/to/spacedatastandards.org \
+  node scripts/generate-plg-bindings.mjs
+```
+
+Without the override, the generator uses the installed `spacedatastandards.org`
+package and still refuses to read SDK-local schema copies.
+
 If another repo needs the same compiler runtime, the package also exposes a
 shared emception session at `space-data-module-sdk/compiler/emception` with
 helpers for serialized command execution and virtual filesystem access.

package/bin/space-data-module.js

@@ -16,6 +16,16 @@ import {
   validateManifestWithStandards,
 } from "../src/compliance/index.js";
 import { bytesToBase64 } from "../src/utils/encoding.js";
+import {
+  signModuleArtifact,
+  verifyModuleArtifact,
+} from "../src/bundle/signing.js";
+import {
+  checkFlowProgram,
+  compileFlowProgram,
+  loadFlowDocument,
+  resolveFlowDependencies,
+} from "../src/flow/flowCompiler.js";
 
 async function main(argv) {
   const [command, ...rest] = argv;
@@ -24,8 +34,16 @@ async function main(argv) {
       return runCheck(rest);
     case "compile":
       return runCompile(rest);
+    case "flow":
+      return runFlow(rest);
+    case "parity":
+      return runParity(rest);
     case "protect":
       return runProtect(rest);
+    case "sign":
+      return runSign(rest);
+    case "verify":
+      return runVerify(rest);
     case "help":
     case "--help":
     case "-h":
@@ -84,6 +102,46 @@ function parseArgs(argv) {
       case "--mnemonic":
         options.mnemonic = requireValue(argv, ++index, value);
         break;
+      case "--deps":
+        options.depsPath = requireValue(argv, ++index, value);
+        break;
+      case "--key":
+        options.keyPath = path.resolve(requireValue(argv, ++index, value));
+        break;
+      case "--trusted":
+        options.trustedKeys = requireValue(argv, ++index, value);
+        break;
+      case "--require-signature":
+        options.requireSignature = true;
+        break;
+      case "--fixture":
+        options.fixturePath = path.resolve(requireValue(argv, ++index, value));
+        break;
+      case "--lanes":
+        options.lanes = requireValue(argv, ++index, value)
+          .split(",")
+          .map((lane) => lane.trim())
+          .filter(Boolean);
+        break;
+      case "--timeout-sec":
+        options.timeoutMs =
+          Number(requireValue(argv, ++index, value)) * 1000;
+        break;
+      case "--self-test-divergence":
+        options.injectDivergence = requireValue(argv, ++index, value);
+        break;
+      case "--chrome-binary":
+        options.chromeBinary = requireValue(argv, ++index, value);
+        break;
+      case "--wasmedge-binary":
+        options.wasmedgeBinary = requireValue(argv, ++index, value);
+        break;
+      case "--docker-platform":
+        options.dockerPlatform = requireValue(argv, ++index, value);
+        break;
+      case "--allow-single-lane":
+        options.allowSingleLane = true;
+        break;
       default:
         throw new Error(`Unknown argument: ${value}`);
     }
@@ -105,9 +163,17 @@ function printUsage() {
   space-data-module check --repo-root .
   space-data-module check --manifest ./manifest.json --wasm ./dist/module.wasm
   space-data-module compile --manifest ./manifest.json --source ./src/module.c --out ./dist/module.wasm
+  space-data-module parity --wasm ./dist/isomorphic/module.wasm --fixture ./fixtures/parity/basic.json
+  space-data-module parity --wasm ./dist/isomorphic/module.wasm --fixture ./f.json --lanes browser,wasmedge,docker-wasmedge [--json]
+  space-data-module parity ... --self-test-divergence docker-wasmedge   (fire drill: prove the diff fails loudly)
+  space-data-module flow check ./flows/my.flow.json --deps ./modules-root
+  space-data-module flow compile ./flows/my.flow.json --deps ./modules-root [--out ./flows/my/dist]
   space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --json
   space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --recipient-public-key <hex> --out ./dist/module.wasm.enc
   space-data-module protect --manifest ./manifest.json --wasm ./dist/module.wasm --single-file-bundle --out ./dist/module.bundle.wasm
+  space-data-module sign --wasm ./dist/module.wasm --key ./test/support/dev-module-signing-keypair.json [--out ./dist/module.signed.wasm]
+  space-data-module verify --wasm ./dist/module.wasm --trusted <pubKeyHex[,pubKeyHex...]> [--require-signature]
+  space-data-module verify --wasm ./dist/module.wasm --key ./test/support/dev-module-signing-keypair.json
 `);
 }
 
@@ -123,6 +189,139 @@ function printReport(report) {
   }
 }
 
+// space-data-module flow check|compile <flow.json> [--deps <path>] [--out <dir>]
+async function runFlow(argv) {
+  const [subcommand, ...rest] = argv;
+  if (subcommand !== "check" && subcommand !== "compile") {
+    printUsage();
+    return 1;
+  }
+  const positionals = rest.filter((value) => !value.startsWith("--"));
+  const flags = [];
+  for (let index = 0; index < rest.length; index += 1) {
+    const value = rest[index];
+    if (!value.startsWith("--")) continue;
+    flags.push(value);
+    if (value !== "--json" && rest[index + 1] !== undefined) {
+      flags.push(rest[++index]);
+    }
+  }
+  const flowPath = positionals[0] ? path.resolve(positionals[0]) : null;
+  if (!flowPath) {
+    throw new Error(`flow ${subcommand} requires a <flow.json> path.`);
+  }
+  const options = parseArgs(flags);
+
+  const flow = await loadFlowDocument(flowPath);
+  const dependencies = await resolveFlowDependencies({
+    depsPath: options.depsPath ?? path.dirname(flowPath),
+  });
+
+  if (subcommand === "check") {
+    const check = checkFlowProgram({ flow, dependencies });
+    if (options.json) {
+      console.log(JSON.stringify(check, null, 2));
+    } else {
+      console.log(`${check.ok ? "PASS" : "FAIL"} ${flowPath}`);
+      for (const issue of check.issues) {
+        console.log(`  ${issue.severity.toUpperCase()} ${issue.code}: ${issue.message}`);
+      }
+      console.log(`  capabilities: [${check.capabilities.join(", ")}]`);
+      for (const node of check.nodes) {
+        console.log(`  node ${node.nodeId}: ${node.pluginId}:${node.methodId} (${node.dispatchModel})`);
+      }
+      for (const component of check.componentDependencies) {
+        console.log(
+          `  component ${component.pluginId}${component.minVersion ? `@${component.minVersion}` : ""}` +
+            `${component.resolved ? "" : " (unresolved)"}`,
+        );
+      }
+    }
+    return check.ok ? 0 : 1;
+  }
+
+  const outDir =
+    options.outputPath ??
+    path.join(
+      path.dirname(flowPath),
+      path.basename(flowPath).replace(/\.flow\.json$/, "").replace(/\.json$/, ""),
+      "dist",
+    );
+  const result = await compileFlowProgram({
+    flow,
+    dependencies,
+    outDir,
+    flowSourcePath: flowPath,
+  });
+  if (options.json) {
+    console.log(
+      JSON.stringify(
+        {
+          outDir,
+          outputs: result.outputs,
+          capabilities: result.check.capabilities,
+          artifact: result.artifact,
+          report: result.report,
+        },
+        null,
+        2,
+      ),
+    );
+  } else {
+    console.log(`Wrote ${result.outputs.moduleWasmPath}`);
+    console.log(`  capabilities: [${result.check.capabilities.join(", ")}]`);
+    for (const node of result.check.nodes) {
+      console.log(`  node ${node.nodeId}: ${node.pluginId}:${node.methodId} (${node.dispatchModel})`);
+    }
+    for (const issue of result.check.warnings) {
+      console.log(`  WARNING ${issue.code}: ${issue.message}`);
+    }
+    printReport(result.report);
+  }
+  return result.report.ok ? 0 : 1;
+}
+
+// space-data-module parity --wasm <module.wasm> --fixture <fixture.json>
+//   [--lanes browser,wasmedge,docker-wasmedge] [--json] [--timeout-sec N]
+//   [--self-test-divergence <lane>] [--chrome-binary|--wasmedge-binary <bin>]
+//   [--docker-platform <p>] [--allow-single-lane]
+//
+// ONE module.wasm, three runtimes, byte-identical behavior. Exit 0 only when
+// every lane × thread-count run matches; any divergence, missing lane, or
+// WasmEdge pin drift exits non-zero with a per-case mismatch report.
+async function runParity(argv) {
+  const options = parseArgs(argv);
+  if (!options.wasmPath || !options.fixturePath) {
+    throw new Error("parity requires --wasm and --fixture.");
+  }
+  const { runParityHarness, formatParityReport } = await import(
+    "../src/testing/parityHarness.js"
+  );
+  const report = await runParityHarness({
+    wasmPath: options.wasmPath,
+    fixturePath: options.fixturePath,
+    lanes: options.lanes,
+    timeoutMs: options.timeoutMs,
+    injectDivergence: options.injectDivergence,
+    chromeBinary: options.chromeBinary,
+    wasmedgeBinary: options.wasmedgeBinary,
+    dockerPlatform: options.dockerPlatform,
+    allowSingleLane: options.allowSingleLane === true,
+    log: (line) => console.error(line),
+  });
+  if (options.json) {
+    console.log(JSON.stringify(report, null, 2));
+  } else {
+    const text = formatParityReport(report);
+    if (report.ok) {
+      console.log(text);
+    } else {
+      console.error(text);
+    }
+  }
+  return report.ok ? 0 : 2;
+}
+
 async function runCheck(argv) {
   const options = parseArgs(argv);
   if (options.manifestPath) {
@@ -248,6 +447,70 @@ async function runProtect(argv) {
   return 0;
 }
 
+async function runSign(argv) {
+  const options = parseArgs(argv);
+  if (!options.wasmPath || !options.keyPath) {
+    throw new Error("sign requires --wasm and --key <keypair.json>.");
+  }
+  const keypair = JSON.parse(await readFile(options.keyPath, "utf8"));
+  if (!keypair.privateKeySeedHex) {
+    throw new Error("key file must contain privateKeySeedHex.");
+  }
+  const wasmBytes = new Uint8Array(await readFile(options.wasmPath));
+  const result = await signModuleArtifact(wasmBytes, {
+    privateKeySeedHex: keypair.privateKeySeedHex,
+    keyId: keypair.keyId ?? null,
+  });
+  const outputPath = options.outputPath ?? options.wasmPath;
+  await writeFile(outputPath, result.wasmBytes);
+  if (options.json) {
+    console.log(
+      JSON.stringify(
+        {
+          outputPath,
+          keyId: result.signature.keyId,
+          publicKeyHex: result.signature.publicKeyHex,
+          canonicalModuleHashHex: result.canonicalModuleHashHex,
+        },
+        null,
+        2,
+      ),
+    );
+  } else {
+    console.log(`signed=${outputPath}`);
+    console.log(`keyId=${result.signature.keyId}`);
+    console.log(`publicKeyHex=${result.signature.publicKeyHex}`);
+    console.log(`canonicalModuleHashHex=${result.canonicalModuleHashHex}`);
+  }
+  return 0;
+}
+
+async function runVerify(argv) {
+  const options = parseArgs(argv);
+  if (!options.wasmPath) {
+    throw new Error("verify requires --wasm.");
+  }
+  let trustedPublicKeys = options.trustedKeys;
+  if (!trustedPublicKeys && options.keyPath) {
+    const keypair = JSON.parse(await readFile(options.keyPath, "utf8"));
+    trustedPublicKeys = keypair.publicKeyHex;
+  }
+  const wasmBytes = new Uint8Array(await readFile(options.wasmPath));
+  const result = await verifyModuleArtifact(wasmBytes, {
+    trustedPublicKeys,
+    requireSignature: options.requireSignature ?? true,
+  });
+  if (options.json) {
+    console.log(JSON.stringify(result, null, 2));
+  } else {
+    console.log(`verified=${result.verified}`);
+    console.log(`signed=${result.signed}`);
+    if (result.keyId) console.log(`keyId=${result.keyId}`);
+    if (result.publicKeyHex) console.log(`publicKeyHex=${result.publicKeyHex}`);
+  }
+  return result.verified ? 0 : 1;
+}
+
 main(process.argv.slice(2))
   .then((exitCode) => {
     process.exitCode = exitCode;

package/docs/.nojekyll

@@ -0,0 +1 @@
+

package/docs/AGENTS.md

@@ -0,0 +1,34 @@
+# AGENTS
+
+Apply the root `AGENTS.md` first. The docs in this repo are normative whenever
+they use words like "canonical", "required", or "must".
+
+## What Authors Should Expect From These Docs
+
+- When behavior changes, update the relevant doc in the same change. Do not let
+  README, examples, and normative docs drift.
+- Keep browser/WasmEdge portability aligned with
+  `docs/browser-wasmedge-isomorphic.md`.
+- Keep the async host adapter boundary aligned with
+  `docs/browser-wasmedge-isomorphic.md` and `docs/testing-harness.md`.
+- When the generic async capability boundary changes, update the exported types
+  and public entry points in the same change so `NodeHost`, `BrowserHost`,
+  `createRuntimeHost()`, and harness `callHost(...)` stay in sync.
+- Keep publication layout and release expectations aligned with
+  `docs/module-publication-standard.md`.
+- Keep binary FlatBuffer ingest guidance aligned with
+  `docs/flatsql-streaming-standard.md`.
+- Keep testing claims aligned with `docs/testing-harness.md`.
+- If you change user-facing behavior, update the nearest example README as well.
+
+## Read These First
+
+- `docs/browser-wasmedge-isomorphic.md`
+- `docs/flatsql-streaming-standard.md`
+- `docs/module-publication-standard.md`
+- `docs/testing-harness.md`
+
+## Note
+
+If you are only authoring a module, prefer following these docs over editing
+them. Edit docs here only when the canonical guidance itself needs to change.