knitting 0.1.55 → 0.1.60

package/map.md

@@ -20,6 +20,8 @@ The core flow is:
   `importTask`, `isMain`, `Envelope`, and `workerMainLoop`.
 - `shared-memory.ts`: Secondary public export for process-safe shared-memory
   helpers.
+- `process-shared-buffer.ts`: Backward-compatible alias for the shared-memory
+  public export.
 - `unsafe.ts`: Public export for experimental lower-level capabilities such as
   `BufferReference`.
 - `utils.ts`: Public export for request/response buffer conversion helpers.
@@ -34,11 +36,13 @@ The core flow is:
 
 ## Build And Scripts
 
-- `build.ts`: Bundles each public entry to one root `.js` file with Bun for a
-  Node ESM target.
-- `tsconfig.npm.json`: TypeScript config used by npm declaration bundling.
-- `scripts/build-declarations.mjs`: Bundles each public entry to one root
-  `.d.ts` file for npm consumers.
+- `build.ts`: Runs the generated-output cleaner, then compiles public entries
+  and `src/**/*.ts` to `.js` and `.d.ts` files with TypeScript.
+- `scripts/clean-generated.ts`: Removes npm-generated root entry files and
+  generated `.js`/`.d.ts` files under `src/`; exposed as `npm run clean`.
+- `tsconfig.npm.json`: TypeScript config used by the npm release build.
+- `scripts/rewrite-declaration-imports.mjs`: Rewrites generated declaration
+  imports from `.ts` to `.js` so npm consumers resolve the compiled files.
 - `scripts/build-native-addons.ts`: Compiles the native Node addons into
   `build/Release/` on Linux, macOS, and Windows. It finds Node headers/libs,
   splits user flags, builds the shared-memory and futex addons, and emits the
@@ -63,8 +67,8 @@ The core flow is:
 - `docs/AGENTS.md`: Agent/contributor orientation, invariants, and workflow
   notes.
 - `docs/CLAUDE.md`: Pointer to the shared agent guidance.
-- `docs/buffer-reference-ownership-move.md`: Design record for
-  `BufferReference` ownership transfer.
+- `docs/buffer-reference-ownership-move.md`: Design record for `BufferReference`
+  ownership transfer.
 - `docs/knitting.pdf`: Longer-form project design/theory document.
 
 ## Public API Layer
@@ -118,6 +122,12 @@ The core flow is:
 - `src/runtime/process-worker.ts`: Process-worker spawning, command/runtime
   selection, process shared-memory layout, inherited/named mapping metadata, and
   child boot payload construction.
+- `src/runtime/compiled-artifact.ts`: Resolves `.knt` executables and validates
+  their versioned manifests, source/task identity, target platform, and execute
+  permission without launching them.
+- `src/runtime/compiled-worker.ts`: Experimental host adapter for compiled
+  worker executables. Spawns the artifact and maps pool calls onto the current
+  numeric framed-pipe protocol.
 
 ## Worker Side
 
@@ -194,14 +204,16 @@ The core flow is:
   parsing, mapping support, and descriptor lifecycle helpers.
 - `src/connections/node-addons.ts`: Native addon specifier resolution for
   committed Node ABI prebuilds with `build/Release` fallback loading.
-- `src/connections/node.ts`: Loads POSIX Node native addons and exposes Node
-  shared memory, mapping, unlink, and futex primitives.
+- `src/connections/node.ts`: Selects the Node 22/24 addon backend or the Node 26
+  FFI backend and exposes shared-memory and futex primitives.
+- `src/connections/node-ffi.ts`: Node 26 `node:ffi` implementation for POSIX
+  shared-memory creation, mapping, cleanup, and buffer-pointer aliases.
 - `src/connections/bun.ts`: Bun FFI implementation for POSIX shared memory.
 - `src/connections/deno.ts`: Deno FFI implementation for POSIX shared memory.
 - `src/connections/posix.ts`: POSIX constants, shared-memory naming, libc path
   detection, and close-on-exec helpers.
 - `src/connections/windows.ts`: Windows FFI loader and named file-mapping
-  primitives used by Deno and Bun.
+  primitives used by Node 26, Deno, and Bun.
 - `src/knitting_shared_memory.cc`: Native Node addon for shared-memory create,
   map, unlink, and descriptor operations, including Windows named mappings.
 - `src/knitting_shm.cc`: Native Node addon for futex/wait helpers used by parked
@@ -211,8 +223,8 @@ The core flow is:
   FFI runtimes.
 - `prebuilds/*/*.node`: Tracked Node native-addon prebuilds for supported
   platform/Node ABI combinations.
-- `prebuilds/win32-x64/*.dll`: Tracked Windows FFI DLL prebuild used by Bun and
-  Deno shared-memory primitives on Windows.
+- `prebuilds/win32-x64/*.dll`: Tracked Windows FFI DLL prebuild used by Node 26,
+  Bun, and Deno shared-memory primitives on Windows.
 
 ## Permissions
 
@@ -269,6 +281,8 @@ The core flow is:
 - `test/_runner.ts`: Runtime-neutral test runner shim used by the test suite.
 - `test/abortSignal.test.ts`: Shared abort bitset behavior.
 - `test/api-cap.test.ts`: API limits such as maximum task id count.
+- `test/compiled-worker.test.ts`: Compiled-artifact validation, public pool
+  integration, source-derived `.knt` paths, and unsupported-feature guards.
 - `test/shared-buffer-io.test.ts`: Shared-buffer IO read/write behavior.
 - `test/file-descriptor.test.ts`: File descriptor metadata and mapping behavior.
 - `test/inliner.test.ts`: Inline executor behavior and thresholds.
@@ -309,9 +323,11 @@ The core flow is:
 
 - `build/Release/*.node`: Native addon output produced by
   `scripts/build-native-addons.ts`.
-- `knitting.js`, `knitting.d.ts`, `shared-memory.js`, `shared-memory.d.ts`,
-  `unsafe.js`, `unsafe.d.ts`, `utils.js`, and `utils.d.ts`: npm release build
-  artifacts produced by `build.ts` and `scripts/build-declarations.mjs`.
+- `knitting.js`, `knitting.d.ts`, `process-shared-buffer.js`,
+  `process-shared-buffer.d.ts`, `shared-memory.js`, `shared-memory.d.ts`,
+  `unsafe.js`, `unsafe.d.ts`, `utils.js`, `utils.d.ts`, and generated
+  `src/**/*.js`/`src/**/*.d.ts`: npm release build artifacts produced by
+  `build.ts` and `scripts/rewrite-declaration-imports.mjs`.
 - `results/`: Benchmark output produced by `run.sh`.
 - `log/`, `logs`, and `*.log`: Local runtime/log output.
 - `node_modules/`: Installed dependencies. Not part of the source map.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "knitting",
-  "version": "0.1.55",
+  "version": "0.1.60",
   "description": "Shared-memory IPC runtime for Node.js, Deno, and Bun.",
   "license": "Apache-2.0",
   "repository": {
@@ -20,6 +20,10 @@
       "types": "./knitting.d.ts",
       "default": "./knitting.js"
     },
+    "./process-shared-buffer": {
+      "types": "./process-shared-buffer.d.ts",
+      "default": "./process-shared-buffer.js"
+    },
     "./shared-memory": {
       "types": "./shared-memory.d.ts",
       "default": "./shared-memory.js"
@@ -40,9 +44,16 @@
     "prebuilds/**/*.node",
     "prebuilds/**/*.dll",
     "scripts/build-native-addons.ts",
+    "scripts/build-compiled-worker.ts",
+    "scripts/compiled-worker/**/*.ts",
+    "scripts/compiled-worker/**/*.d.ts",
     "src/**/*.cc",
+    "src/**/*.d.ts",
+    "src/**/*.js",
     "knitting.d.ts",
     "knitting.js",
+    "process-shared-buffer.d.ts",
+    "process-shared-buffer.js",
     "shared-memory.d.ts",
     "shared-memory.js",
     "unsafe.d.ts",
@@ -51,19 +62,22 @@
     "utils.js"
   ],
   "scripts": {
-    "build": "bun run build.ts && node scripts/build-declarations.mjs",
-    "build:js": "bun run build.ts",
-    "build:types": "node scripts/build-declarations.mjs",
-    "build:npm": "npm run build",
+    "clean": "bun run scripts/clean-generated.ts",
+    "build": "bun run build.ts && node scripts/rewrite-declaration-imports.mjs",
+    "build:js": "bun run build.ts && node scripts/rewrite-declaration-imports.mjs",
+    "build:types": "bun run build.ts && node scripts/rewrite-declaration-imports.mjs",
+    "build:npm": "bun run build.ts && node scripts/rewrite-declaration-imports.mjs",
     "build:native": "bun run scripts/build-native-addons.ts",
+    "build:compiled": "bun run scripts/build-compiled-worker.ts",
     "build:release": "bun run build:native && npm run build",
-    "build:node": "npm run build:js",
+    "build:node": "bun run build.ts && node scripts/rewrite-declaration-imports.mjs",
     "prepack": "npm run build",
     "release:npm": "npm run test:all && npm publish",
     "release:npm:dry-run": "npm publish --dry-run",
     "test": "npm run test:deno",
     "test:deno": "deno test -A --ignore=test/runtime.node.test.ts --ignore=test/runtime.process.test.ts",
     "test:node": "node --no-warnings --experimental-transform-types --test \"./test/*.test.ts\"",
+    "test:node:26": "node --no-warnings --experimental-ffi test/node26-ffi.integration.mjs",
     "test:bun": "bun test ./test/*.test.ts --path-ignore-patterns=**/tx-queue.test.ts",
     "test:runtimes": "npm run test:node && npm run test:bun",
     "test:all": "npm run test:deno && npm run test:runtimes",

package/process-shared-buffer.d.ts

@@ -0,0 +1 @@
+export { getDefaultProcessSharedBufferPrimitives, parseProcessSharedBufferMetadata, PROCESS_SHARED_BUFFER_BRAND, ProcessSharedBuffer, setDefaultProcessSharedBufferPrimitives, type ProcessSharedBufferCreator, type ProcessSharedBufferMapper, type ProcessSharedBufferMetadata, type ProcessSharedBufferPrimitives, type ProcessSharedBufferRange, type ProcessSharedBufferView, type ProcessSharedBufferViewConstructor, } from "./src/connections/process-shared-buffer.js";

package/process-shared-buffer.js

@@ -0,0 +1 @@
+export { getDefaultProcessSharedBufferPrimitives, parseProcessSharedBufferMetadata, PROCESS_SHARED_BUFFER_BRAND, ProcessSharedBuffer, setDefaultProcessSharedBufferPrimitives, } from "./src/connections/process-shared-buffer.js";

package/scripts/build-compiled-worker.ts

@@ -0,0 +1,276 @@
+#!/usr/bin/env bun
+/// <reference types="bun" />
+
+import {
+  chmodSync,
+  existsSync,
+  mkdirSync,
+  renameSync,
+  rmSync,
+  statSync,
+} from "node:fs";
+import { dirname, isAbsolute, join, relative, resolve } from "node:path";
+import { homedir } from "node:os";
+import { fileURLToPath } from "node:url";
+
+type Options = {
+  modulePath: string;
+  outputPath: string;
+  manifestPath: string;
+  tasks: string[];
+  compiler?: string;
+};
+
+const here = dirname(fileURLToPath(import.meta.url));
+const runtime = resolve(here, "compiled-worker/runtime.ts");
+const taskShim = resolve(here, "compiled-worker/task-shim.ts");
+const repositoryEntry = resolve(here, "../knitting.ts");
+const PORFFOR_REPOSITORY = "https://github.com/CanadaHonk/porffor.git";
+const PORFFOR_REVISION = "747d551844750fc5ed32cf88cdf0b3854aee267e";
+
+const usage = (): never => {
+  console.error(
+    "Usage: bun scripts/build-compiled-worker.ts --module tasks.ts " +
+      "--out tasks.knt --tasks taskA,taskB [--porf /path/to/porf]",
+  );
+  process.exit(2);
+};
+
+const readOptions = (): Options => {
+  const args = process.argv.slice(2);
+  let modulePath: string | undefined;
+  let outputPath: string | undefined;
+  let manifestPath: string | undefined;
+  let tasks: string[] | undefined;
+  let compiler: string | undefined;
+  for (let index = 0; index < args.length; index++) {
+    const argument = args[index]!;
+    const value = args[index + 1];
+    if (value === undefined) usage();
+    if (argument === "--module") modulePath = value;
+    else if (argument === "--out") outputPath = value;
+    else if (argument === "--manifest") manifestPath = value;
+    else if (argument === "--tasks") tasks = value.split(",").filter(Boolean);
+    else if (argument === "--porf") compiler = value;
+    else usage();
+    index++;
+  }
+  if (modulePath === undefined || outputPath === undefined || !tasks?.length) {
+    usage();
+  }
+  for (const name of tasks) {
+    if (!/^[$A-Z_a-z][$\w]*$/.test(name) && name !== "default") {
+      throw new Error("Compiled task name is not an export identifier: " + name);
+    }
+  }
+  return {
+    modulePath: resolve(modulePath),
+    outputPath: resolve(outputPath),
+    manifestPath: manifestPath === undefined
+      ? resolve(outputPath) + ".json"
+      : resolve(manifestPath),
+    tasks,
+    compiler,
+  };
+};
+
+const commandWorks = (command: string): boolean => {
+  try {
+    const result = Bun.spawnSync([command, "--version"], {
+      stdout: "pipe",
+      stderr: "pipe",
+    });
+    return result.exitCode === 0 &&
+      new TextDecoder().decode(result.stdout).startsWith("alpha ");
+  } catch {
+    return false;
+  }
+};
+
+const compilerPath = (input: string): string => {
+  const decoded = input.startsWith("file:") ? fileURLToPath(input) : input;
+  const path = decoded.includes("/") || decoded.includes("\\")
+    ? resolve(decoded)
+    : decoded;
+  return existsSync(path) && statSync(path).isDirectory()
+    ? join(path, "porf")
+    : path;
+};
+
+const installCompiler = (): string => {
+  const cacheRoot = process.env.XDG_CACHE_HOME ?? join(homedir(), ".cache");
+  const cache = join(
+    cacheRoot,
+    "knitting",
+    "porffor-" + PORFFOR_REVISION.slice(0, 12),
+  );
+  const executable = join(cache, "porf");
+  if (commandWorks(executable)) return executable;
+
+  const temporary = cache + ".tmp-" + process.pid;
+  rmSync(temporary, { recursive: true, force: true });
+  mkdirSync(temporary, { recursive: true });
+  const commands = [
+    ["git", "-C", temporary, "init", "--quiet"],
+    ["git", "-C", temporary, "remote", "add", "origin", PORFFOR_REPOSITORY],
+    [
+      "git",
+      "-C",
+      temporary,
+      "fetch",
+      "--quiet",
+      "--depth",
+      "1",
+      "origin",
+      PORFFOR_REVISION,
+    ],
+    ["git", "-C", temporary, "checkout", "--quiet", "--detach", "FETCH_HEAD"],
+  ];
+  console.error("Knitting: downloading the pinned Porffor compiler (first run only)");
+  try {
+    for (const command of commands) {
+      const result = Bun.spawnSync(command, { stdout: "inherit", stderr: "inherit" });
+      if (result.exitCode !== 0) {
+        throw new Error(command[0] + " exited with code " + result.exitCode);
+      }
+    }
+    rmSync(cache, { recursive: true, force: true });
+    renameSync(temporary, cache);
+  } catch (error) {
+    rmSync(temporary, { recursive: true, force: true });
+    throw error;
+  }
+  if (!commandWorks(executable)) {
+    throw new Error("Downloaded Porffor compiler is unavailable");
+  }
+  return executable;
+};
+
+const resolveCompiler = (options: Options): string => {
+  const configured = options.compiler ?? process.env.PORFFOR_MAIN ??
+    process.env.PORF;
+  if (configured !== undefined) {
+    const candidate = compilerPath(configured);
+    if (!commandWorks(candidate)) {
+      throw new Error("Porffor main compiler is unavailable: " + candidate);
+    }
+    return candidate;
+  }
+  if (commandWorks("porf")) return "porf";
+  return installCompiler();
+};
+
+const quote = (value: string): string => JSON.stringify(value);
+
+const makeEntry = (modulePath: string, tasks: string[]): string => {
+  const imports = tasks.map((name, index) =>
+    name === "default"
+      ? `import __knitExport${index} from ${quote(modulePath)};`
+      : `import { ${name} as __knitExport${index} } from ${quote(modulePath)};`
+  );
+  const functions = tasks.map((_, index) =>
+    `typeof __knitExport${index} === "function" ` +
+    `? __knitExport${index} : __knitExport${index}.f`
+  );
+  return [
+    ...imports,
+    `import { runJsonWorker } from ${quote(runtime)};`,
+    `runJsonWorker([${functions.join(", ")}]);`,
+    "",
+  ].join("\n");
+};
+
+const resolvesToKnitting = (specifier: string, resolveDir: string): boolean => {
+  if (specifier === "knitting" || specifier === "@vixeny/knitting") return true;
+  if (!specifier.startsWith(".") && !isAbsolute(specifier)) return false;
+  const candidate = resolve(resolveDir, specifier).replace(/\.(?:js|ts)$/, "");
+  return candidate === repositoryEntry.replace(/\.ts$/, "");
+};
+
+const run = async (): Promise<void> => {
+  const options = readOptions();
+  const outputDirectory = dirname(options.outputPath);
+  const workDirectory = join(outputDirectory, ".knitting-compiled");
+  const entryPath = join(workDirectory, "entry.ts");
+  const bundlePath = join(workDirectory, "worker.bundle.js");
+  mkdirSync(workDirectory, { recursive: true });
+  mkdirSync(dirname(options.manifestPath), { recursive: true });
+  await Bun.write(entryPath, makeEntry(options.modulePath, options.tasks));
+
+  const build = await Bun.build({
+    entrypoints: [entryPath],
+    format: "esm",
+    target: "browser",
+    minify: { syntax: true, whitespace: false, identifiers: false },
+    plugins: [{
+      name: "knitting-compiled-worker-api",
+      setup(builder) {
+        builder.onResolve({ filter: /.*/ }, (args) =>
+          resolvesToKnitting(args.path, args.resolveDir)
+            ? { path: taskShim }
+            : undefined
+        );
+      },
+    }],
+  });
+  if (!build.success || build.outputs[0] === undefined) {
+    for (const log of build.logs) console.error(log);
+    throw new Error("Bun could not bundle the compiled worker");
+  }
+  await Bun.write(bundlePath, build.outputs[0]);
+
+  const porf = resolveCompiler(options);
+  const versionResult = Bun.spawnSync([porf, "--version"], {
+    stdout: "pipe",
+    stderr: "pipe",
+  });
+  const version = new TextDecoder().decode(versionResult.stdout).trim();
+  console.error("Knitting: compiling " + relative(process.cwd(), options.modulePath));
+  const compilerEnvironment = { ...process.env };
+  if (process.platform === "linux" && compilerEnvironment.CC === undefined) {
+    compilerEnvironment.CC = "cc -Wno-stringop-overflow";
+  }
+  const compile = Bun.spawnSync([
+    porf,
+    "native",
+    "--module",
+    "--quiet",
+    "-O3",
+    "--flto=auto",
+    bundlePath,
+    "-o",
+    options.outputPath,
+  ], {
+    stdout: "inherit",
+    stderr: "inherit",
+    env: compilerEnvironment,
+  });
+  if (compile.exitCode !== 0) {
+    throw new Error("Porffor exited with code " + compile.exitCode);
+  }
+  chmodSync(options.outputPath, 0o755);
+
+  await Bun.write(
+    options.manifestPath,
+    JSON.stringify({
+      format: "knitting-compiled-worker",
+      version: 1,
+      protocol: "knitting-json-v1",
+      compiler: { name: "porffor", version },
+      target: { platform: process.platform, arch: process.arch },
+      source: relative(outputDirectory, options.modulePath).replaceAll("\\", "/"),
+      sourceMtimeMs: statSync(options.modulePath).mtimeMs,
+      capabilities: {
+        input: "json",
+        output: "json",
+        async: false,
+        abortSignal: true,
+        binaryValues: true,
+      },
+      tasks: options.tasks.map((exportName, index) => ({ index, exportName })),
+    }, null, 2) + "\n",
+  );
+  rmSync(workDirectory, { recursive: true, force: true });
+};
+
+await run();

package/scripts/compiled-worker/porffor.d.ts

@@ -0,0 +1,5 @@
+declare namespace Porffor {
+  function c(strings: TemplateStringsArray, ...values: unknown[]): void;
+}
+
+type i32 = number;