space-data-module-sdk 0.5.14 → 0.5.18

package/README.md

@@ -14,11 +14,52 @@ This repository is the source of truth for module-level concerns:
 - the `sds.bundle` single-file container
 - deployment authorization plus SDS publication records (`REC`, `PNM`, `ENC`)
 - the first canonical module hostcall/import ABI surface
+- the canonical runtime-host model for append-only standards rows, host-managed
+  runtime regions, and dynamic module installation/loading
+- shared module-level testing/runtime harnesses, including the WasmEdge
+  process runner used by package-level validation suites
 
 <p align="center">
   <img src="docs/architecture.svg" alt="Module architecture overview" width="820" />
 </p>
 
+## Shared Harness Ownership
+
+This repo owns the generic module-side runtime harnesses used across the stack:
+
+- `createModuleHarness(...)` for process and WasmEdge-backed module invocation
+- `resolveModuleHarnessLaunchPlan(...)` for portable launch planning
+- `buildWasmEdgeEmscriptenPthreadRunner(...)` plus the shared native runner
+  source and runner-level pthread smoke test
+
+Flow-specific standalone harnessing lives in `sdn-flow`, which layers flow
+enqueue/drain behavior on top of its compiled standalone runtime host. Package-
+specific validation suites, including conjunction replay/V&V, are expected to
+sit on top of these shared harnesses instead of defining their own runtime
+process model.
+
+## Canonical Runtime Host
+
+The SDK now owns the durable runtime identity model used by OrbPro, browser and
+server SDN hosts, and the evolving WasmEdge harness:
+
+- standards rows are addressed only by `($SCHEMA_FILE_ID, rowId)`
+- `rowId` is append-only and never reused
+- aligned-binary runtime state is addressed only by `(regionId, recordIndex)`
+- raw pointers remain internal execution details and are not durable APIs
+
+The minimal host surface lives under `src/runtime-host/` and covers three
+responsibilities:
+
+- `createFlatSqlRuntimeStore()` for append-only row handles and row resolution
+- `createRuntimeRegionStore()` for host-allocated aligned-binary regions and
+  externally backed record-view descriptors
+- `createModuleRegistry()` for dynamic install/load/unload/invoke
+
+`createRuntimeHost()` composes those stores into the canonical SDK host model.
+OrbPro layers entity/view helpers on top of that host instead of inventing a
+separate durable identity model.
+
 ## Module Artifact Model
 
 A compliant module built with this SDK is always a valid `.wasm` artifact with:
@@ -178,6 +219,11 @@ semantics.
 This repo now exposes a manifest-driven harness generator from
 `space-data-module-sdk/testing` and two complementary integration suites:
 
+- shared process-level helpers for command-surface runtimes:
+  - `createPluginInvokeProcessClient(...)`
+  - `resolveWasmEdgePluginLaunchPlan(...)`
+  - `buildWasmEdgeEmscriptenPthreadRunner(...)`
+
 - `npm run test:runtime-matrix`
   - cross-language runtime smoke across the same WASM in Node.js, Go, Python,
     Rust, Java, C#, and Swift
@@ -299,6 +345,46 @@ The reference path lives in
 Standard bundle payloads now include the optional `deployment-plan` JSON entry
 for resolved protocol installations and producer input bindings.
 
+## Publication Protection Extensions
+
+Digital signature and encrypted-delivery metadata are publication-layer
+extensions, not a second module format. The canonical module is still:
+
+- valid `.wasm`
+- optionally carrying `sds.bundle` as a wasm custom section
+- optionally carrying one appended SDS `REC` trailer after the wasm bytes
+
+That trailing `REC` FlatBuffer is the standards-backed container for
+publication records:
+
+- `PNM` is the publication notice / digital-signature record
+  - identifies the artifact
+  - carries the CID and publish timestamp
+  - carries signature metadata so a loader can verify who published the module
+- `ENC` is the encrypted-delivery record
+  - carries the X25519 / AES-CTR / HKDF parameters needed to decrypt the
+    protected transport payload
+  - does not change the canonical module ABI or manifest shape
+
+The loader contract is always:
+
+1. Start with the protected blob.
+2. Scan backward for the trailing `REC` footer.
+3. Parse `REC` using the standards-generated `REC`, `PNM`, and `ENC`
+   FlatBuffers from `spacedatastandards.org`.
+4. Resolve `PNM` for signature/publication metadata.
+5. Resolve `ENC` if the delivery was encrypted.
+6. Strip the trailer or decrypt the protected payload.
+7. Hand the remaining raw wasm bytes to the runtime.
+
+Aligned-binary payloads are a separate invoke-ABI optimization. They do not
+replace the canonical FlatBuffer schema. If a port advertises
+`wireFormat: "aligned-binary"`, it must also advertise the regular
+`wireFormat: "flatbuffer"` fallback for the same schema and file identifier in
+the same accepted type set. The publication demo in the local lab uses a
+regular `OMM.fbs` request and a `StateVector.fbs` response that advertises both
+the canonical FlatBuffer fallback and the aligned-binary layout metadata.
+
 ## Module Publication
 
 Packages that publish Space Data modules use the canonical `sdn-module`
@@ -402,10 +488,14 @@ npm run start:lab
 
 Then open `http://localhost:4318`.
 
+Use the `Run Publication Demo` button to generate a protected demo artifact and
+inspect the parsed `REC`, `PNM`, and `ENC` records produced by the real
+`spacedatastandards.org` generated message classes.
+
 ## Related Projects
 
 - [`spacedatastandards.org`](https://spacedatastandards.org)
-- [`@digitalarsenal/sdn-flow`](https://github.com/DigitalArsenal/sdn-flow)
+- [`sdn-flow`](https://github.com/DigitalArsenal/sdn-flow)
 - [`hd-wallet-wasm`](https://github.com/nicktj-dev/hd-wallet-wasm)
 
 ## Development

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "space-data-module-sdk",
-  "version": "0.5.14",
+  "version": "0.5.18",
   "description": "Module SDK for building, validating, signing, and deploying WebAssembly modules on the Space Data Network.",
   "type": "module",
   "types": "./src/index.d.ts",
@@ -31,6 +31,10 @@
     "./deployment": "./src/deployment/index.js",
     "./invoke": "./src/invoke/index.js",
     "./runtime": "./src/runtime/index.js",
+    "./runtime-host": {
+      "types": "./src/index.d.ts",
+      "default": "./src/runtime-host/index.js"
+    },
     "./testing": "./src/testing/index.js",
     "./standards": "./src/standards/index.js",
     "./schemas/*": "./schemas/*"
@@ -50,15 +54,15 @@
     "prepublishOnly": "npm test && npm run check:compliance"
   },
   "dependencies": {
-    "flatc-wasm": "25.12.19-wasm.43",
     "flatbuffers": "^25.9.23",
+    "flatc-wasm": "25.12.19-wasm.43",
+    "flatsql": "^0.4.0",
     "hd-wallet-wasm": "1.6.0",
     "sdn-emception": "1.0.0",
     "spacedatastandards.org": "git+https://github.com/DigitalArsenal/spacedatastandards.org.git#09af351a1"
   },
   "devDependencies": {
-    "express": "^4.21.2",
-    "flatsql": "^0.4.0"
+    "express": "^4.21.2"
   },
   "engines": {
     "node": ">=20.0.0"

package/schemas/HostStorageAbi.fbs

@@ -0,0 +1,53 @@
+namespace orbpro.hoststorage;
+
+table RowHandle {
+  schema_file_id: string (required);
+  row_id: uint64;
+}
+
+table RuntimeRegionRecord {
+  region_id: uint64;
+  record_index: uint32;
+  layout_id: string (required);
+  record_byte_length: uint32;
+  byte_length: uint32;
+  alignment: uint16 = 1;
+  bytes: [ubyte];
+}
+
+table ResolveRowViewRequest {
+  handle: RowHandle;
+}
+
+table ResolveRowViewResult {
+  handle: RowHandle;
+  found: bool = false;
+}
+
+table ResolveRegionRecordRequest {
+  region_id: uint64;
+  record_index: uint32;
+}
+
+table ResolveRegionRecordResult {
+  record: RuntimeRegionRecord;
+  found: bool = false;
+}
+
+table AllocateRuntimeRegionRequest {
+  layout_id: string (required);
+  record_byte_length: uint32;
+  alignment: uint16 = 1;
+  initial_record_count: uint32;
+}
+
+table AllocateRuntimeRegionResult {
+  region_id: uint64;
+  layout_id: string (required);
+  record_byte_length: uint32;
+  alignment: uint16 = 1;
+  record_count: uint32;
+}
+
+root_type ResolveRowViewRequest;
+file_identifier "HSTR";

package/src/bundle/codec.js

@@ -1,4 +1,4 @@
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 
 import {
   CanonicalizationRuleT,

package/src/compliance/pluginCompliance.js

@@ -167,134 +167,6 @@ function normalizePayloadWireFormatName(value) {
   return null;
 }
 
-function normalizedPluginFamilyName(value) {
-  if (!isNonEmptyString(value)) {
-    return null;
-  }
-  return value
-    .trim()
-    .toLowerCase()
-    .replace(/-/g, "_");
-}
-
-function typeSetHasDualWireFormats(typeSet) {
-  if (!Array.isArray(typeSet?.allowedTypes) || typeSet.allowedTypes.length === 0) {
-    return false;
-  }
-  let hasFlatbuffer = false;
-  let hasAlignedBinary = false;
-  for (const allowedType of typeSet.allowedTypes) {
-    const wireFormat = normalizePayloadWireFormatName(allowedType?.wireFormat);
-    if (wireFormat === "flatbuffer") {
-      hasFlatbuffer = true;
-    } else if (wireFormat === "aligned-binary") {
-      hasAlignedBinary = true;
-    }
-  }
-  return hasFlatbuffer && hasAlignedBinary;
-}
-
-function validateCanonicalDualFormatPort(port, issues, location, methodId, portId) {
-  if (!port) {
-    pushIssue(
-      issues,
-      "error",
-      "propagator-missing-canonical-port",
-      `Canonical propagator method "${methodId}" must declare port "${portId}".`,
-      location,
-    );
-    return;
-  }
-  if (!Array.isArray(port.acceptedTypeSets) || port.acceptedTypeSets.length === 0) {
-    pushIssue(
-      issues,
-      "error",
-      "propagator-missing-canonical-port",
-      `Canonical propagator method "${methodId}" port "${portId}" must declare acceptedTypeSets.`,
-      `${location}.acceptedTypeSets`,
-    );
-    return;
-  }
-  port.acceptedTypeSets.forEach((typeSet, index) => {
-    if (!typeSetHasDualWireFormats(typeSet)) {
-      pushIssue(
-        issues,
-        "error",
-        "canonical-port-missing-dual-wire-format",
-        `Canonical propagator method "${methodId}" port "${portId}" acceptedTypeSet "${typeSet?.setId ?? index}" must accept both regular flatbuffer and aligned-binary payloads.`,
-        `${location}.acceptedTypeSets[${index}]`,
-      );
-    }
-  });
-}
-
-function validateCanonicalPropagatorContract(manifest, issues, sourceName) {
-  if (normalizedPluginFamilyName(manifest?.pluginFamily) !== "propagator") {
-    return;
-  }
-
-  const methods = Array.isArray(manifest?.methods) ? manifest.methods : [];
-  const methodRequirements = [
-    {
-      methodId: "ingest_omm",
-      inputPorts: ["omm"],
-      outputPorts: [],
-    },
-    {
-      methodId: "describe_sources_batch",
-      inputPorts: ["request"],
-      outputPorts: ["result"],
-    },
-    {
-      methodId: "propagate_state",
-      inputPorts: ["request"],
-      outputPorts: ["state"],
-    },
-  ];
-
-  for (const requirement of methodRequirements) {
-    const methodIndex = methods.findIndex(
-      (method) => method?.methodId === requirement.methodId,
-    );
-    if (methodIndex === -1) {
-      pushIssue(
-        issues,
-        "error",
-        "propagator-missing-canonical-method",
-        `Propagator plugins must declare canonical method "${requirement.methodId}".`,
-        `${sourceName}.methods`,
-      );
-      continue;
-    }
-    const method = methods[methodIndex];
-    const location = `${sourceName}.methods[${methodIndex}]`;
-    for (const portId of requirement.inputPorts) {
-      const port = Array.isArray(method.inputPorts)
-        ? method.inputPorts.find((entry) => entry?.portId === portId)
-        : null;
-      validateCanonicalDualFormatPort(
-        port,
-        issues,
-        `${location}.inputPorts`,
-        requirement.methodId,
-        portId,
-      );
-    }
-    for (const portId of requirement.outputPorts) {
-      const port = Array.isArray(method.outputPorts)
-        ? method.outputPorts.find((entry) => entry?.portId === portId)
-        : null;
-      validateCanonicalDualFormatPort(
-        port,
-        issues,
-        `${location}.outputPorts`,
-        requirement.methodId,
-        portId,
-      );
-    }
-  }
-}
-
 function validateStringField(issues, value, location, label) {
   if (!isNonEmptyString(value)) {
     pushIssue(issues, "error", "missing-string", `${label} must be a non-empty string.`, location);
@@ -1387,8 +1259,6 @@ export function validatePluginManifest(manifest, options = {}) {
       }
     });
 
-    validateCanonicalPropagatorContract(manifest, issues, sourceName);
-
     if (manifest.timers !== undefined && !Array.isArray(manifest.timers)) {
       pushIssue(
         issues,

package/src/generated/orbpro/invoke/plugin-invoke-request.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from 'flatbuffers';
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { TypedArenaBuffer } from '../../orbpro/stream/typed-arena-buffer.js';
 /**
  * Canonical invoke envelope consumed by direct ABI and command-mode execution.

package/src/generated/orbpro/invoke/plugin-invoke-response.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from 'flatbuffers';
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { TypedArenaBuffer } from '../../orbpro/stream/typed-arena-buffer.js';
 /**
  * Canonical invoke result emitted by direct ABI and command-mode execution.

package/src/generated/orbpro/manifest/accepted-type-set.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { FlatBufferTypeRef, } from "../../orbpro/stream/flat-buffer-type-ref.js";
 /**
  * Accepted schema family for a port.

package/src/generated/orbpro/manifest/build-artifact.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 /**
  * Build artifact emitted by the plugin toolchain.
  */

package/src/generated/orbpro/manifest/host-capability.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { CapabilityKind } from "../../orbpro/manifest/capability-kind.js";
 /**
  * One host capability dependency.

package/src/generated/orbpro/manifest/method-manifest.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { DrainPolicy } from "../../orbpro/manifest/drain-policy.js";
 import { PortManifest, } from "../../orbpro/manifest/port-manifest.js";
 /**

package/src/generated/orbpro/manifest/plugin-manifest.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from 'flatbuffers';
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { BuildArtifact } from '../../orbpro/manifest/build-artifact.js';
 import { HostCapability } from '../../orbpro/manifest/host-capability.js';
 import { MethodManifest } from '../../orbpro/manifest/method-manifest.js';

package/src/generated/orbpro/manifest/port-manifest.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { AcceptedTypeSet, } from "../../orbpro/manifest/accepted-type-set.js";
 /**
  * One input or output port on a method.

package/src/generated/orbpro/manifest/protocol-spec.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 /**
  * Protocol handler declared by a plugin.
  */

package/src/generated/orbpro/manifest/timer-spec.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 /**
  * Timer entry declared by a plugin.
  */

package/src/generated/orbpro/module/canonicalization-rule.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from 'flatbuffers';
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 /**
  * Canonicalization rule applied before hashing or signature verification.
  */

package/src/generated/orbpro/module/module-bundle-entry.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from 'flatbuffers';
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { ModuleBundleEntryRole } from '../../orbpro/module/module-bundle-entry-role.js';
 import { ModulePayloadEncoding } from '../../orbpro/module/module-payload-encoding.js';
 import { FlatBufferTypeRef } from '../../orbpro/stream/flat-buffer-type-ref.js';

package/src/generated/orbpro/module/module-bundle.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from 'flatbuffers';
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { CanonicalizationRule } from '../../orbpro/module/canonicalization-rule.js';
 import { ModuleBundleEntry } from '../../orbpro/module/module-bundle-entry.js';
 /**

package/src/generated/orbpro/stream/flat-buffer-type-ref.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { PayloadWireFormat } from "../../orbpro/stream/payload-wire-format.js";
 function normalizePayloadWireFormat(value) {
     if (value === PayloadWireFormat.ALIGNED_BINARY) {

package/src/generated/orbpro/stream/typed-arena-buffer.js

@@ -1,6 +1,6 @@
 // automatically generated by the FlatBuffers compiler, do not modify
 /* eslint-disable @typescript-eslint/no-unused-vars, @typescript-eslint/no-explicit-any, @typescript-eslint/no-non-null-assertion */
-import * as flatbuffers from "flatbuffers";
+import * as flatbuffers from "flatbuffers/mjs/flatbuffers.js";
 import { BufferMutability } from "../../orbpro/stream/buffer-mutability.js";
 import { BufferOwnership } from "../../orbpro/stream/buffer-ownership.js";
 import { FlatBufferTypeRef, } from "../../orbpro/stream/flat-buffer-type-ref.js";