space-data-module-sdk 0.5.5 → 0.5.9

package/README.md

@@ -1,9 +1,9 @@
 # Space Data Module SDK
 
-`space-data-module-sdk` defines the canonical module artifact for the Space Data
-stack: WebAssembly code, an embedded FlatBuffer manifest, a stable capability
-vocabulary, a single-file bundle format, and the signing and transport records
-used to move modules between hosts.
+`space-data-module-sdk` defines the canonical Space Data module architecture for
+WebAssembly artifacts: an embedded FlatBuffer manifest, canonical ABI exports,
+a stable capability vocabulary, single-file packaging, and the signing and
+transport records used to move modules between hosts.
 
 This repository is the source of truth for module-level concerns:
 
@@ -11,17 +11,34 @@ This repository is the source of truth for module-level concerns:
 - embedded manifest exports inside `.wasm` modules
 - standards-aware compliance and capability validation
 - module compilation and protection
-- the `sds.bundle` single-file custom section
+- the `sds.bundle` single-file container
 - deployment authorization plus SDS publication records (`REC`, `PNM`, `ENC`)
 - the first canonical module hostcall/import ABI surface
+- shared module-level testing/runtime harnesses, including the WasmEdge
+  process runner used by package-level validation suites
 
 <p align="center">
   <img src="docs/architecture.svg" alt="Module architecture overview" width="820" />
 </p>
 
-## Artifact Model
+## Shared Harness Ownership
 
-A module built with this SDK is a `.wasm` artifact with:
+This repo owns the generic module-side runtime harnesses used across the stack:
+
+- `createModuleHarness(...)` for process and WasmEdge-backed module invocation
+- `resolveModuleHarnessLaunchPlan(...)` for portable launch planning
+- `buildWasmEdgeEmscriptenPthreadRunner(...)` plus the shared native runner
+  source and runner-level pthread smoke test
+
+Flow-specific standalone harnessing lives in `sdn-flow`, which layers flow
+enqueue/drain behavior on top of its compiled standalone runtime host. Package-
+specific validation suites, including conjunction replay/V&V, are expected to
+sit on top of these shared harnesses instead of defining their own runtime
+process model.
+
+## Module Artifact Model
+
+A compliant module built with this SDK is always a valid `.wasm` artifact with:
 
 - an embedded `PluginManifest.fbs`
 - exported manifest accessors:
@@ -37,10 +54,15 @@ A module built with this SDK is a `.wasm` artifact with:
   - resolved deployment plans and input bindings
   - deployment authorization
   - auxiliary FlatBuffer or raw payloads
-- optional appended SDS `REC` publication trailers carrying:
+- optional appended SDS FlatBuffer publication records in a trailing `REC`
+  container carrying:
   - `PNM` digital-signature/publication metadata
   - `ENC` encrypted-delivery metadata
-- auxiliary FlatBuffer or raw payloads
+
+Single-file bundling does not append arbitrary bytes to the end of the wasm
+module. The one-file module container is `sds.bundle`, stored as a standard
+wasm custom section. Appended bytes are reserved for SDS publication records
+such as `PNM` and `ENC`, wrapped in a trailing `REC` FlatBuffer collection.
 
 The module contract stays the same whether the artifact is loaded directly,
 wrapped in a deployment envelope, or shipped as one bundled `.wasm` file.
@@ -65,7 +87,7 @@ same request bytes from guest memory and returns response bytes in guest
 memory.
 
 For simple single-input / single-output methods, command mode also supports a
-degenerate raw shortcut:
+raw shortcut:
 
 ```bash
 wasmedge module.wasm --method echo < input.fb > output.fb
@@ -101,8 +123,8 @@ anywhere it can:
 - read FlatBuffers
 - honor the module capability and host ABI contract
 
-That target set matches the WebAssembly and FlatBuffers runtime families already
-used in the companion `flatbuffers/wasm` work:
+That architecture is intended to stay portable across the common WebAssembly
+and FlatBuffer host environments:
 
 - browser
 - Node.js
@@ -126,11 +148,58 @@ This repo currently includes:
 - a reference Node host and sync `sdn_host` bridge for the first hostcall
   surface
 
+## Runtime Targets
+
+Manifests can declare coarse runtime targets in `manifest.runtimeTargets`.
+
+If a manifest declares `runtimeTargets: ["wasi"]`, this SDK treats that as
+"standalone WASI, no host wrapper required." In practice that currently means:
+
+- the artifact must declare the `command` invoke surface
+- declared capabilities must stay within the pure WASI subset:
+  `logging`, `clock`, `random`, `filesystem`, `pipe`
+- hosted protocols may only use `wasi-pipe` transport
+
+If a manifest declares `runtimeTargets: ["wasmedge"]`, this SDK treats that as
+the preferred server-side target when the guest needs network-oriented runtime
+features such as sockets or TLS. Plain `wasi` remains the strict portability
+baseline; `wasmedge` is the practical higher-capability target.
+
+## WasmEdge Pthreads
+
+`space-data-module-sdk` is also the source of truth for module thread-model
+selection.
+
+- `compileModuleFromSource({ threadModel })` accepts an explicit thread model.
+- If `threadModel` is omitted, the SDK resolves it from `manifest.runtimeTargets`.
+- `runtimeTargets: ["wasmedge"]` defaults to `emscripten-pthreads`.
+- Other targets currently default to `single-thread`.
+
+WasmEdge-targeted pthread builds do not use the embedded `sdn-emception`
+toolchain. They require a real system Emscripten installation on `PATH`, and
+the compiler result plus guest-link bundle metadata preserve the selected
+`threadModel`.
+
+As emitted by current Emscripten, these pthread artifacts still import
+Emscripten `env.*` host functions plus imported shared memory. That means a
+bare `wasmedge` CLI invocation is not yet the direct execution path for them;
+they currently require a WasmEdge-side host shim that satisfies the Emscripten
+pthread contract.
+
+If a runtime cannot interoperate with the guest pthread contract directly,
+document that as a wrapper requirement instead of changing the guest artifact
+semantics.
+
 ## Testing
 
 This repo now exposes a manifest-driven harness generator from
 `space-data-module-sdk/testing` and two complementary integration suites:
 
+- shared process-level helpers for command-surface runtimes:
+  - `createPluginInvokeProcessClient(...)`
+  - `resolveWasmEdgePluginLaunchPlan(...)`
+  - `buildWasmEdgeEmscriptenPthreadRunner(...)`
+
 - `npm run test:runtime-matrix`
   - cross-language runtime smoke across the same WASM in Node.js, Go, Python,
     Rust, Java, C#, and Swift
@@ -145,20 +214,6 @@ The detailed edge cases and the current WASI-vs-host portability boundary are
 documented in
 [`docs/testing-harness.md`](./docs/testing-harness.md).
 
-If a manifest declares `runtimeTargets: ["wasi"]`, this SDK now treats that as
-"standalone WASI, no host wrapper required." In practice that currently means:
-
-- the artifact must declare the `command` invoke surface
-- declared capabilities must stay within the pure WASI subset:
-  `logging`, `clock`, `random`, `filesystem`, `pipe`
-- hosted protocols may only use `wasi-pipe` transport
-
-For maximum server-side portability with guest-owned network services, use
-`runtimeTargets: ["wasmedge"]`. That target is intended for WasmEdge
-environments with socket/TLS extensions, while plain `wasi` remains the strict
-no-wrapper baseline. The Node-RED-oriented parity map lives in
-[`docs/node-red-default-node-parity.md`](./docs/node-red-default-node-parity.md).
-
 ## Install
 
 ```bash
@@ -176,6 +231,8 @@ import {
   validateManifestWithStandards,
 } from "space-data-module-sdk";
 
+manifest.runtimeTargets = ["wasmedge"];
+
 const manifestBytes = encodePluginManifest(manifest);
 const validation = await validateManifestWithStandards(manifest);
 if (!validation.ok) {
@@ -186,6 +243,8 @@ const compilation = await compileModuleFromSource({
   manifest,
   sourceCode,
   language: "c",
+  // Optional. Defaults from manifest.runtimeTargets.
+  // threadModel: "emscripten-pthreads",
 });
 
 const bundle = await createSingleFileBundle({
@@ -243,10 +302,11 @@ The full contract split is documented in
 
 `sds.bundle` keeps module delivery to one file without changing WebAssembly
 loadability for the runtime payload itself. The SDK writes the bundle as a
-standard custom section inside the wasm module and, when the artifact is
-signed or encrypted for publication, appends an SDS `REC` trailer after the
-wasm bytes. Loaders must scan and strip that trailer before handing bytes to a
-runtime such as WasmEdge.
+standard custom section inside the wasm module. When the artifact is signed or
+encrypted for publication, SDS appends FlatBuffer publication records after the
+wasm bytes in a trailing `REC` container. Loaders must scan from the end,
+resolve `PNM` / `ENC`, strip or decrypt as needed, and only then hand the
+remaining raw wasm bytes to a runtime such as WasmEdge.
 
 The reference path lives in
 [`examples/single-file-bundle`](./examples/single-file-bundle):
@@ -263,7 +323,7 @@ for resolved protocol installations and producer input bindings.
 
 ## Module Publication
 
-Packages that publish SDN modules now use the canonical `sdn-module`
+Packages that publish Space Data modules use the canonical `sdn-module`
 publication descriptor. That descriptor covers:
 
 - standalone module packages
@@ -323,7 +383,7 @@ includes:
 
 Manifests can also declare coarse runtime targets for planning and compliance:
 
-`node` `browser` `wasi` `server` `desktop` `edge`
+`node` `browser` `wasi` `wasmedge` `server` `desktop` `edge`
 
 ## Environment Notes
 
@@ -379,7 +439,8 @@ npm run check:compliance
 ```
 
 Node.js `>=20` is required. The compiler uses `sdn-emception` and `flatc-wasm`
-by default.
+by default for the embedded toolchain path. WasmEdge pthread builds require a
+system Emscripten toolchain on `PATH`.
 
 If another repo needs the same compiler runtime, the package also exposes a
 shared emception session at `space-data-module-sdk/compiler/emception` with

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "space-data-module-sdk",
-  "version": "0.5.5",
+  "version": "0.5.9",
   "description": "Module SDK for building, validating, signing, and deploying WebAssembly modules on the Space Data Network.",
   "type": "module",
   "types": "./src/index.d.ts",

package/src/compiler/compileModule.js

@@ -1,7 +1,11 @@
 import os from "node:os";
 import path from "node:path";
+import { execFile } from "node:child_process";
+import { promisify } from "node:util";
 import {
   mkdtemp,
+  mkdir,
+  readFile,
   rm,
   writeFile,
 } from "node:fs/promises";
@@ -24,7 +28,11 @@ import {
 } from "./flatcSupport.js";
 import { runWithEmceptionLock } from "./emceptionNode.js";
 import { encodePluginManifest, toEmbeddedPluginManifest } from "../manifest/index.js";
-import { DefaultInvokeExports, InvokeSurface } from "../runtime/constants.js";
+import {
+  DefaultInvokeExports,
+  InvokeSurface,
+  RuntimeTarget,
+} from "../runtime/constants.js";
 import {
   appendPublicationRecordCollection,
   createEncryptedEnvelopePayload,
@@ -55,6 +63,12 @@ import { sha256Bytes } from "../utils/crypto.js";
 import { getWasmWallet } from "../utils/wasmCrypto.js";
 
 const C_IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_]*$/;
+const execFileAsync = promisify(execFile);
+
+export const ModuleThreadModel = Object.freeze({
+  SINGLE_THREAD: "single-thread",
+  EMSCRIPTEN_PTHREADS: "emscripten-pthreads",
+});
 
 function selectCompiler(language) {
   const normalized = String(language ?? "c").trim().toLowerCase();
@@ -81,16 +95,63 @@ function buildCompilerArgs(exportedSymbols, options = {}) {
     (symbol) => "-Wl,--export=" + symbol,
   );
   const extraArgs = [];
+  const threadArgs = [];
+  if (options.threadModel === ModuleThreadModel.EMSCRIPTEN_PTHREADS) {
+    threadArgs.push("-pthread");
+  }
   if (options.allowUndefinedImports === true) {
     extraArgs.push("-s", "ERROR_ON_UNDEFINED_SYMBOLS=0", "-Wl,--allow-undefined");
   }
-  const args = ["-O2", "-s", "STANDALONE_WASM=1", ...extraArgs, ...linkerExports];
+  const args = [
+    "-O2",
+    ...threadArgs,
+    "-s",
+    "STANDALONE_WASM=1",
+    ...extraArgs,
+    ...linkerExports,
+  ];
   if (options.noEntry === true) {
     args.splice(1, 0, "--no-entry");
   }
   return args;
 }
 
+function normalizeThreadModel(value) {
+  if (value === undefined || value === null || value === "") {
+    return null;
+  }
+  const normalized = String(value).trim().toLowerCase();
+  if (normalized === ModuleThreadModel.SINGLE_THREAD) {
+    return ModuleThreadModel.SINGLE_THREAD;
+  }
+  if (normalized === ModuleThreadModel.EMSCRIPTEN_PTHREADS) {
+    return ModuleThreadModel.EMSCRIPTEN_PTHREADS;
+  }
+  throw new Error(
+    `Unsupported threadModel "${value}". Expected "${ModuleThreadModel.SINGLE_THREAD}" or "${ModuleThreadModel.EMSCRIPTEN_PTHREADS}".`,
+  );
+}
+
+function resolveThreadModel({ manifest, threadModel } = {}) {
+  const explicit = normalizeThreadModel(threadModel);
+  if (explicit) {
+    return explicit;
+  }
+  const runtimeTargets = Array.isArray(manifest?.runtimeTargets)
+    ? manifest.runtimeTargets
+        .map((target) => String(target ?? "").trim().toLowerCase())
+        .filter(Boolean)
+    : [];
+  if (runtimeTargets.includes(RuntimeTarget.WASMEDGE)) {
+    return ModuleThreadModel.EMSCRIPTEN_PTHREADS;
+  }
+  return ModuleThreadModel.SINGLE_THREAD;
+}
+
+function requiresSystemEmscripten(threadModel) {
+  return threadModel === ModuleThreadModel.EMSCRIPTEN_PTHREADS;
+}
+
 function guestLinkSymbolPrefix(pluginId) {
   const normalized = String(pluginId ?? "module");
   const ascii = Array.from(new TextEncoder().encode(normalized))
@@ -280,6 +341,7 @@ function createGuestLinkBundleEntries(guestLink) {
         methodSymbols: guestLink.methodSymbols,
         methodIds: Object.keys(guestLink.methodSymbols ?? {}),
         language: guestLink.language,
+        threadModel: guestLink.threadModel ?? ModuleThreadModel.SINGLE_THREAD,
       },
       description: "Guest-link metadata for monolithic flow linking.",
     },
@@ -302,6 +364,14 @@ async function writeFilesToEmception(emception, rootDir, files) {
   }
 }
 
+async function writeFilesToDirectory(rootDir, files) {
+  for (const [relativePath, content] of Object.entries(files)) {
+    const filePath = path.join(rootDir, relativePath);
+    await mkdir(path.dirname(filePath), { recursive: true });
+    await writeFile(filePath, content);
+  }
+}
+
 function removeEmceptionDirectory(emception, directoryPath) {
   if (!emception.FS.analyzePath(directoryPath).exists) {
     return;
@@ -466,6 +536,8 @@ async function compileWithEmception(options = {}) {
             language,
             symbolPrefix: guestLink.prefix,
             methodSymbols: guestLink.methodSymbols,
+            threadModel:
+              compileOptions.threadModel ?? ModuleThreadModel.SINGLE_THREAD,
             objectBytes: linkObjectBytes,
           },
         };
@@ -483,6 +555,166 @@ async function compileWithEmception(options = {}) {
   }
 }
 
+async function ensureSystemCompilerAvailable(command) {
+  try {
+    await execFileAsync(command, ["--version"]);
+  } catch (error) {
+    if (error?.code === "ENOENT") {
+      throw new Error(
+        `System Emscripten toolchain is required for "${ModuleThreadModel.EMSCRIPTEN_PTHREADS}" builds, but "${command}" was not found on PATH.`,
+      );
+    }
+    throw error;
+  }
+}
+
+async function runSystemCompiler(command, args, options = {}) {
+  try {
+    await execFileAsync(command, args, {
+      cwd: options.cwd,
+      maxBuffer: 32 * 1024 * 1024,
+    });
+  } catch (error) {
+    const stderr = typeof error?.stderr === "string" ? error.stderr.trim() : "";
+    const stdout = typeof error?.stdout === "string" ? error.stdout.trim() : "";
+    const detail = stderr || stdout || error?.message || "unknown error";
+    throw new Error(
+      `Compilation failed with ${command} (system emscripten): ${detail}`,
+    );
+  }
+}
+
+async function compileWithSystemEmscripten(options = {}) {
+  const {
+    manifest,
+    language,
+    sourceCompilerCommand,
+    sourceExtension,
+    sourceCode,
+    manifestSource,
+    invokeHeaderSource,
+    invokeSource,
+    exportedSymbols,
+    outputPath,
+    compileOptions,
+  } = options;
+  const tempDir = await mkdtemp(
+    path.join(os.tmpdir(), "space-data-module-sdk-compile-"),
+  );
+  const resolvedOutputPath = path.resolve(
+    outputPath ?? path.join(tempDir, "module.wasm"),
+  );
+  const runtimeIncludeDir = path.join(tempDir, "flatbuffers-runtime");
+  const sourcePath = path.join(tempDir, `module.${sourceExtension}`);
+  const manifestSourcePath = path.join(tempDir, "plugin-manifest-exports.cpp");
+  const invokeHeaderPath = path.join(tempDir, "space_data_module_invoke.h");
+  const invokeSourcePath = path.join(tempDir, "plugin-invoke-bridge.cpp");
+  const sourceObjectPath = path.join(tempDir, "module.o");
+  const linkObjectPath = path.join(tempDir, "module-link.o");
+  const manifestObjectPath = path.join(tempDir, "plugin-manifest-exports.o");
+  const invokeObjectPath = path.join(tempDir, "plugin-invoke-bridge.o");
+  const wasmOutputPath = path.join(tempDir, "module.wasm");
+
+  try {
+    await ensureSystemCompilerAvailable(sourceCompilerCommand);
+    await ensureSystemCompilerAvailable("em++");
+    const { runtimeHeaders, schemaHeaders } = await getInvokeCppSupportFiles();
+    const args = buildCompilerArgs(exportedSymbols, compileOptions);
+    await writeFilesToDirectory(runtimeIncludeDir, runtimeHeaders);
+    await writeFilesToDirectory(tempDir, schemaHeaders);
+    await writeFile(sourcePath, sourceCode);
+    await writeFile(manifestSourcePath, manifestSource);
+    await writeFile(invokeHeaderPath, invokeHeaderSource);
+    await writeFile(invokeSourcePath, invokeSource);
+
+    const sourceCompileArgs = [
+      "-c",
+      sourcePath,
+      `-I${tempDir}`,
+      ...(compileOptions.threadModel === ModuleThreadModel.EMSCRIPTEN_PTHREADS
+        ? ["-pthread"]
+        : []),
+      "-o",
+      sourceObjectPath,
+    ];
+    await runSystemCompiler(sourceCompilerCommand, sourceCompileArgs);
+
+    const sourceObjectBytes = new Uint8Array(await readFile(sourceObjectPath));
+    const guestLink = deriveGuestLinkRenameArgs({
+      objectBytes: sourceObjectBytes,
+      pluginId: manifest?.pluginId,
+      methodIds: Array.isArray(manifest?.methods)
+        ? manifest.methods.map((method) => String(method?.methodId ?? ""))
+        : [],
+    });
+
+    const linkCompileArgs = [
+      "-c",
+      sourcePath,
+      `-I${tempDir}`,
+      ...(compileOptions.threadModel === ModuleThreadModel.EMSCRIPTEN_PTHREADS
+        ? ["-pthread"]
+        : []),
+      ...guestLink.renameArgs,
+      "-o",
+      linkObjectPath,
+    ];
+    await runSystemCompiler(sourceCompilerCommand, linkCompileArgs);
+    await runSystemCompiler("em++", [
+      "-c",
+      manifestSourcePath,
+      "-std=c++17",
+      `-I${tempDir}`,
+      `-I${runtimeIncludeDir}`,
+      ...(compileOptions.threadModel === ModuleThreadModel.EMSCRIPTEN_PTHREADS
+        ? ["-pthread"]
+        : []),
+      "-o",
+      manifestObjectPath,
+    ]);
+    await runSystemCompiler("em++", [
+      "-c",
+      invokeSourcePath,
+      "-std=c++17",
+      `-I${tempDir}`,
+      `-I${runtimeIncludeDir}`,
+      ...(compileOptions.threadModel === ModuleThreadModel.EMSCRIPTEN_PTHREADS
+        ? ["-pthread"]
+        : []),
+      "-o",
+      invokeObjectPath,
+    ]);
+    await runSystemCompiler("em++", [
+      sourceObjectPath,
+      manifestObjectPath,
+      invokeObjectPath,
+      ...args,
+      "-o",
+      wasmOutputPath,
+    ]);
+
+    const wasmBytes = new Uint8Array(await readFile(wasmOutputPath));
+    const linkObjectBytes = new Uint8Array(await readFile(linkObjectPath));
+    await writeFile(resolvedOutputPath, wasmBytes);
+    return {
+      wasmBytes,
+      outputPath: resolvedOutputPath,
+      tempDir,
+      guestLink: {
+        format: "wasm-object",
+        language,
+        symbolPrefix: guestLink.prefix,
+        methodSymbols: guestLink.methodSymbols,
+        threadModel: compileOptions.threadModel,
+        objectBytes: linkObjectBytes,
+      },
+    };
+  } catch (error) {
+    await rm(tempDir, { recursive: true, force: true });
+    throw error;
+  }
+}
+
 export async function compileModuleFromSource(options = {}) {
   const manifest = options.manifest ?? {};
   const sourceCode = String(options.sourceCode ?? "");
@@ -531,11 +763,19 @@ export async function compileModuleFromSource(options = {}) {
   let wasmBytes;
   let resolvedOutputPath = null;
   let tempDir = null;
+  const threadModel = resolveThreadModel({
+    manifest,
+    threadModel: options.threadModel,
+  });
   const compileOptions = {
     ...options,
     noEntry: includeCommandMain !== true,
+    threadModel,
   };
-  const result = await compileWithEmception({
+  const compileFunction = requiresSystemEmscripten(threadModel)
+    ? compileWithSystemEmscripten
+    : compileWithEmception;
+  const result = await compileFunction({
     manifest,
     language: compiler.language,
     sourceCompilerCommand: compiler.command,
@@ -559,8 +799,11 @@ export async function compileModuleFromSource(options = {}) {
   });
 
   return {
-    compiler: "em++ (emception)",
+    compiler: requiresSystemEmscripten(threadModel)
+      ? "em++ (system emscripten pthreads)"
+      : "em++ (emception)",
     language: compiler.language,
+    threadModel,
     outputPath: resolvedOutputPath,
     tempDir,
     wasmBytes,

package/src/compiler/index.d.ts

@@ -1,5 +1,7 @@
 export type {
   CompilationResult,
+  GuestLinkArtifact,
+  ModuleThreadModelName,
   ProtectedArtifact,
 } from "../index.js";
 
@@ -7,6 +9,7 @@ export {
   cleanupCompilation,
   compileModuleFromSource,
   createRecipientKeypairHex,
+  ModuleThreadModel,
   protectModuleArtifact,
 } from "../index.js";
 

package/src/compiler/index.js

@@ -2,6 +2,7 @@ export {
   cleanupCompilation,
   compileModuleFromSource,
   createRecipientKeypairHex,
+  ModuleThreadModel,
   protectModuleArtifact,
 } from "./compileModule.js";
 

package/src/index.d.ts

@@ -556,6 +556,7 @@ export function decodeProtectedBlobBase64(
 export interface CompilationResult {
   compiler: string;
   language: string;
+  threadModel: ModuleThreadModelName;
   outputPath: string | null;
   tempDir: string | null;
   wasmBytes: Uint8Array;
@@ -569,9 +570,19 @@ export interface GuestLinkArtifact {
   language: string;
   symbolPrefix: string;
   methodSymbols: Record<string, string>;
+  threadModel: ModuleThreadModelName;
   objectBytes: Uint8Array;
 }
 
+export type ModuleThreadModelName =
+  | "single-thread"
+  | "emscripten-pthreads";
+
+export const ModuleThreadModel: {
+  readonly SINGLE_THREAD: "single-thread";
+  readonly EMSCRIPTEN_PTHREADS: "emscripten-pthreads";
+};
+
 export interface ProtectedArtifact {
   mnemonic: string;
   signingPublicKeyHex: string;
@@ -602,6 +613,7 @@ export function compileModuleFromSource(options: {
   manifest: PluginManifest;
   sourceCode: string;
   language?: string;
+  threadModel?: ModuleThreadModelName;
   outputPath?: string;
   allowUndefinedImports?: boolean;
 }): Promise<CompilationResult>;
@@ -650,12 +662,25 @@ export type {
   HarnessInvokeScenario,
   HarnessRawScenario,
   ManifestHarnessPlan,
+  ModuleHarness,
+  ModuleHarnessRuntimeDescriptor,
+  PluginInvokeProcessClient,
+  PluginInvokeProcessLaunchPlan,
+  WasmEdgeRunnerBuildPlan,
 } from "./testing/index.js";
 
 export {
+  buildWasmEdgeEmscriptenPthreadRunner,
+  buildWasmEdgeSpawnEnv,
+  createModuleHarness,
+  createPluginInvokeProcessClient,
   describeCapabilityRuntimeSurface,
   generateManifestHarnessPlan,
   materializeHarnessScenario,
+  resolveModuleHarnessLaunchPlan,
+  resolveWasmEdgeRunnerBuildPlan,
+  resolveWasmEdgeRunnerSourcePath,
+  resolveWasmEdgePluginLaunchPlan,
   serializeHarnessPlan,
 } from "./testing/index.js";
 

package/src/invoke/codec.js

@@ -12,7 +12,7 @@ import { InvokeSurface } from "../generated/orbpro/manifest/invoke-surface.js";
 import { BufferMutability } from "../generated/orbpro/stream/buffer-mutability.js";
 import { BufferOwnership } from "../generated/orbpro/stream/buffer-ownership.js";
 import { FlatBufferTypeRefT } from "../generated/orbpro/stream/flat-buffer-type-ref.js";
-import { TypedArenaBuffer, TypedArenaBufferT } from "../generated/orbpro/stream/typed-arena-buffer.js";
+import { TypedArenaBufferT } from "../generated/orbpro/stream/typed-arena-buffer.js";
 import { toUint8Array } from "../runtime/bufferLike.js";
 
 function toByteBuffer(data) {
@@ -149,20 +149,22 @@ function normalizeArenaFrame(frame = {}, offset) {
 
 function packArenaFrames(frames = []) {
   const packedFrames = [];
-  const arena = [];
+  const normalizedFrames = [];
   let offset = 0;
   for (const frame of frames) {
     const normalized = normalizeArenaFrame(frame, offset);
-    for (let index = 0; index < normalized.padding; index += 1) {
-      arena.push(0);
-    }
-    arena.push(...normalized.payload);
     offset = normalized.buffer.offset + normalized.buffer.size;
     packedFrames.push(normalized.buffer);
+    normalizedFrames.push(normalized);
+  }
+
+  const arena = new Uint8Array(offset);
+  for (const normalized of normalizedFrames) {
+    arena.set(normalized.payload, normalized.buffer.offset);
   }
   return {
     frames: packedFrames,
-    arena: Uint8Array.from(arena),
+    arena,
   };
 }