knitting 0.1.46 → 0.1.50

package/README.md

@@ -2,6 +2,7 @@
 
 [![JSR Version](https://jsr.io/badges/@vixeny/knitting)](https://jsr.io/@vixeny/knitting)
 [![JSR Score](https://jsr.io/badges/@vixeny/knitting/score)](https://jsr.io/@vixeny/knitting)
+[![npm Version](https://img.shields.io/npm/v/knitting?logo=npm&logoColor=white)](https://www.npmjs.com/package/knitting)
 [![Tests](https://github.com/mimiMonads/knitting/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/mimiMonads/knitting/actions/workflows/test.yml)
 [![Coverage Workflow](https://github.com/mimiMonads/knitting/actions/workflows/coverage.yml/badge.svg?branch=main)](https://github.com/mimiMonads/knitting/actions/workflows/coverage.yml)
 [![Coverage](https://img.shields.io/badge/coverage-92.10%25-brightgreen)](https://github.com/mimiMonads/knitting/actions/workflows/coverage.yml)
@@ -10,45 +11,37 @@
 [![Deno](https://img.shields.io/badge/deno-2%2B-000000?logo=deno&logoColor=white)](https://deno.com/)
 [![Bun](https://img.shields.io/badge/bun-1%2B-f472b6?logo=bun&logoColor=white)](https://bun.sh/)
 
-Knitting is a worker-pool over shared-memory IPC runtime for Node.js,
-Deno, and Bun. Which mission is make javascript a real multicore language,
-Thanks to its memory design, it can be from 5x to 25x faster than using `postMessages`
-bypassing OS sockect comunnication entielly with a novel protocol written from scratch.
+Knitting is a worker pool over a shared-memory IPC runtime for Node.js, Deno, and Bun. Our mission is to make JavaScript a multicore language with real inter-runtime communication. 
 
-Use it for the parts of your program that should run somewhere else: CPU-heavy to small
-work, runtime-isolated jobs, long-running tools, or anything that needs to move
-speed and type felixbility.
+Thanks to its memory design, it can be 5x to 25x faster than using `postMessages` , bypassing OS socket communication entirely with a novel protocol written from scratch.
 
+Use it for parts of your program that should run in different environments, such as CPU-intensive tasks, small jobs, runtime-isolated tasks, custom isolation for workers in Docker or bwrap environments, long-running tools, or any processes that require speed and type flexibility.
 
-You define a task once, spin up a pool, and call it like a normal async
-function:
+You define a task once, spin up a pool, and call it like a normal async function:
 
 ```ts
+
 const result = await pool.call.resizeImage(file);
+
 ```
 
-Under the hood, Knitting schedules work across worker threads or separate
-processes (depending on runtime and your settings), keeps arguments typed, and
-uses shared memory for transport.
+So you only have to take care of 4 things:
+ - Create a task
+ - Create a pool
+ - Call your task
+ - Terminate the pool
+
 
-## So
+Under the hood, we take care of scheduling and orchestration across worker threads or separate processes, also handling signals, timeouts, life cycles, memory allocation, garbage collection, and cross-runtime memory over the processes.
 
-- Call worker code like `pool.call.myTask(arg)`.
-- Keep the resolved types end-to-end
-- Choose `threads` for speed or `processes` for stronger isolation.
-- Move supported payloads via shared memory (great for binary data).
-- Run on Node.js, Deno, or Bun.
 
 ## Why use it?
 
-- Easy to use: Have a multi-thread enviroment or process with few lines of code.
-- Great type support: pass primitives, JSON, Promise of these and special types (typed arrays, Node
-  `Buffer`, `Envelope`, and `ProcessSharedBuffer`).
+- Easy to use: Have a multithreaded environment or process with few lines of code.
+- Great type support: pass primitives, JSON, Promise of these, and special types (typed arrays, `Node Buffer`, `Envelope`, and `ProcessSharedBuffer`).
 - Runtime flexibility: the same API across Node.js, Deno, and Bun.
-- Worker choices: use threads for fast pools, or processes for stronger
-  isolation.
-- All out-of-the-box expierince: strict-by-default permissions, payload-size limits, task
-  timeouts, abort-aware tasks, and worker hard timeouts.
+- Worker choices: use threads for fast pools or processes for stronger isolation.
+- All out-of-the-box experiences: strict-by-default permissions, payload-size limits, task timeouts, abort-aware tasks, and worker hard timeouts.
 
 ## Requirements
 
@@ -56,27 +49,18 @@ uses shared memory for transport.
 - Deno 2+ 
 - Bun 1+ 
 
-Process workers and `ProcessSharedBuffer` use the native shared-memory layer.
-On Linux and macOS, build it before running process/shared-memory tests.
-If you're only using thread workers, you can typically ignore this and use
-Knitting without building native addons.
-
 ## Install
 
-```bash
-npm install knitting
-```
-
-Via JSR's npm compatibility:
+From npm:
 
 ```bash
-jsr add --npm @vixeny/knitting
+npm install knitting
 ```
 
 For Deno projects:
 
 ```bash
-deno add jsr:@vixeny/knitting
+deno add --npm jsr:@vixeny/knitting
 ```
 
 ## Quick Start
@@ -301,12 +285,37 @@ Common options you might tweak:
 | `payload` | Shared payload-buffer settings: `mode`, `payloadInitialBytes`, `payloadMaxByteLength`, and `maxPayloadBytes`. |
 | `abortSignalCapacity` | Number of shared abort slots available to abort-aware calls. |
 | `worker.resolveAfterFinishingAll` | Let submitted calls finish before shutdown resolves. |
+| `worker.bootstrap` | Privileged async hook imported and awaited before task modules load. |
 | `worker.hardTimeoutMs` | Force pool shutdown when a task exceeds this many milliseconds. |
 | `worker.runtime` | Choose `"thread"` or `"process"` workers. |
+| `worker.processSharedMemory` | Process-worker memory discovery: `"inherit"` by default on POSIX, or `"named"` for wrappers/containers that cannot preserve fd 0. |
 | `permission` | Runtime permission policy for workers. |
 | `debug` | Enable extra diagnostics. |
 | `source` | Worker source override for advanced runtimes. |
 
+### Worker bootstrap
+
+Use `worker.bootstrap` when a worker needs privileged setup before task modules
+are imported. The bootstrap module is imported once per worker, and its selected
+export is awaited before Knitting imports task definitions.
+
+```ts
+const pool = createPool({
+  worker: {
+    bootstrap: {
+      href: "./worker-bootstrap.ts",
+      name: "setup",
+      data: { env: "worker-only" },
+    },
+  },
+})({ add });
+```
+
+Bootstrap code runs with worker startup privileges, so keep it trusted. It is a
+good place to remove environment variables, install runtime guards, open shared
+memory metadata, or prepare globals that task modules should see at import time.
+Bootstrap is worker-only and cannot be combined with the inline host lane.
+
 ## Worker Runtimes
 
 By default, workers use runtime-local threads where possible (the lowest
@@ -326,14 +335,15 @@ const pool = createPool({
   threads: 2,
   worker: {
     runtime: "process",
-    processRuntime: "node",
+    processRuntime: "deno",
   },
 })({ add });
 ```
 
-`processRuntime` can be `"node"`, `"deno"`, or `"bun"`. You can also provide a
-`processCommandPrefix` when workers need to be launched through a wrapper such
-as a package manager, container command, or runtime shim.
+`processRuntime` can be `"node"`, `"deno"`, or `"bun"` and defaults to
+`"deno"`. You can also provide a `processCommandPrefix` when workers need to be
+launched through a wrapper such as a package manager, container command, or
+runtime shim.
 
 That prefix is also useful for sandbox and resource-control tools. The one
 important detail is that process workers receive their shared-memory handle on
@@ -341,6 +351,68 @@ stdin, which is file descriptor 0. Wrappers that leave stdin alone usually work;
 wrappers that replace, close, or proxy stdin without passing the fd through will
 stop the worker from booting.
 
+For wrappers that cannot preserve fd 0, use named process-worker memory instead.
+The worker process must share the same OS IPC namespace as the host so it can
+reopen the named mapping.
+
+```ts
+const pool = createPool({
+  threads: 2,
+  worker: {
+    runtime: "process",
+    processRuntime: "node",
+    processSharedMemory: {
+      mode: "named",
+      namePrefix: "knit_worker",
+    },
+    processCommandPrefix: [
+      // The prefix runs before Knitting appends:
+      // node --no-warnings --experimental-transform-types <worker-file>
+      "docker",
+      "run",
+      // Remove the container when the worker exits.
+      "--rm",
+      // Required for named POSIX shared memory across host/container.
+      "--ipc=host",
+      // The worker imports the same files as the host, at the same path.
+      "-v",
+      `${process.cwd()}:${process.cwd()}`,
+      "-w",
+      process.cwd(),
+      // Forward Knitting's process-worker boot metadata into the container.
+      "-e",
+      "KNITTING_PROCESS_WORKER",
+      "-e",
+      "KNITTING_PROCESS_WORKER_BOOT",
+      "knitting-node-worker",
+    ],
+  },
+})({ add });
+```
+
+### Windows process workers
+
+On Windows, Knitting automatically uses named shared memory for the
+process-worker control channel. You do not need to set
+`processSharedMemory: "named"` yourself — the runtime detects Windows and
+forces it.
+
+```ts
+// Works on Windows without extra options.
+const pool = createPool({
+  threads: 2,
+  worker: {
+    runtime: "process",
+    processRuntime: "node",
+  },
+})({ add });
+```
+
+If you also pass `ProcessSharedBuffer` payloads to Docker workers running on
+Windows, create the payload buffer with `mode: "create"` and a name, and add
+`--ipc=host` to the Docker prefix — the pool-level control channel is already
+named, but the payload buffer needs its own name so the container can reopen it.
+
 When the goal is isolation, define the worker code with `importTask()` instead
 of importing the task function directly into the host. That keeps the code you
 want to isolate out of the host process; only the worker imports and runs it.
@@ -436,6 +508,41 @@ shouldn't) cross the boundary:
 - Objects with behavior that depends on prototypes, getters, setters, or hidden
   process-local state.
 
+### Envelope
+
+`Envelope` pairs a JSON-serializable header with a binary `ArrayBuffer`
+payload. Use it when a call needs both structured metadata and raw bytes in a
+single argument.
+
+```ts
+import { Envelope, createPool, isMain, task } from "knitting";
+
+export const processImage = task<
+  Envelope<{ format: string }>,
+  Envelope<{ width: number; height: number }>
+>({
+  f: (envelope) => {
+    const pixels = new Uint8Array(envelope.payload);
+    // ... process pixels
+    return new Envelope({ width: 800, height: 600 }, pixels.buffer);
+  },
+});
+
+if (isMain) {
+  const pool = createPool({ threads: 2 })({ processImage });
+
+  try {
+    const buffer = new ArrayBuffer(1024);
+    const result = await pool.call.processImage(
+      new Envelope({ format: "png" }, buffer),
+    );
+    console.log(result.header); // { width: 800, height: 600 }
+  } finally {
+    await pool.shutdown();
+  }
+}
+```
+
 If a payload is large, set `payload.maxPayloadBytes` deliberately and prefer
 binary/shared-memory shapes over deeply nested objects.
 
@@ -446,7 +553,10 @@ memory. Use it when two workers or processes need to see the same bytes without
 copying the whole payload for every call.
 
 ```ts
-import { ProcessSharedBuffer } from "knitting/process-shared-buffer";
+import {
+  getDefaultProcessSharedBufferPrimitives,
+  ProcessSharedBuffer,
+} from "knitting/process-shared-buffer";
 import { createPool, isMain, task } from "knitting";
 
 export const readFirstCell = task<ProcessSharedBuffer, number>({
@@ -455,13 +565,14 @@ export const readFirstCell = task<ProcessSharedBuffer, number>({
 
 if (isMain) {
   const pool = createPool({ threads: 1 })({ readFirstCell });
-  const shared = ProcessSharedBuffer.create(64);
+  const primitives = getDefaultProcessSharedBufferPrimitives();
+  const shared = ProcessSharedBuffer.create(64, primitives);
 
   try {
     Atomics.store(shared.view(Int32Array), 0, 42);
     console.log(await pool.call.readFirstCell(shared));
   } finally {
-    shared.close();
+    shared.descriptor.mapping?.close?.();
     await pool.shutdown();
   }
 }
@@ -482,19 +593,24 @@ programs do not accidentally inherit them.
 
 ### Named channels for independent processes
 
-Sometimes you do want two unrelated processes to rendezvous on purpose. Use a
-named channel for that.
+Use a named channel when two processes need to find the same shared memory
+without inheriting an fd from each other. One process creates the channel by
+name; the other opens that same name.
 
 ```ts
-import { ProcessSharedBuffer } from "knitting/process-shared-buffer";
+import {
+  getDefaultProcessSharedBufferPrimitives,
+  ProcessSharedBuffer,
+} from "knitting/process-shared-buffer";
 
 const name = "knitting-demo-channel";
+const primitives = getDefaultProcessSharedBufferPrimitives();
 
 const owner = ProcessSharedBuffer.create({
   name,
   size: 64,
   mode: "create",
-});
+}, primitives);
 
 try {
   Atomics.store(owner.view(Int32Array), 0, 7);
@@ -503,46 +619,121 @@ try {
     name,
     size: 64,
     mode: "open",
-  });
+  }, primitives);
 
   try {
     console.log(Atomics.load(peer.view(Int32Array), 0));
   } finally {
-    peer.close();
+    peer.descriptor.mapping?.close?.();
   }
 } finally {
-  owner.close();
-  ProcessSharedBuffer.unlink(name);
+  owner.descriptor.mapping?.close?.();
+  primitives.unlinkSharedMemory?.(name);
 }
 ```
 
-Use `"create"` for the process that owns the channel and `"open"` for peers.
-Treat the channel name like a capability: make it unique, do not accept it from
-untrusted input without validation, and clean it up when the channel is no
-longer needed. On POSIX runtimes `unlink` removes the name. On platforms where
-named mappings are lifetime-managed by the OS, closing the last handle is the
-important cleanup step.
+Use `"create"` on the owner side and `"open"` on the peer side. The name is the
+thing that grants access, so generate a hard-to-guess name and keep it private.
+When you are done, close the mappings and unlink the name where the runtime
+supports it.
+
+### Sending `ProcessSharedBuffer` to Docker workers
+
+Docker process workers can receive a `ProcessSharedBuffer`, but it needs to be
+named. The default anonymous form is fd-backed and private to the parent-child
+process path; Docker does not inherit that fd in a way the worker can reopen.
+
+Use a named buffer for the payload and named process-worker memory for the pool:
+
+```ts
+import { createPool, isMain, task } from "knitting";
+import {
+  getDefaultProcessSharedBufferPrimitives,
+  ProcessSharedBuffer,
+} from "knitting/process-shared-buffer";
+
+export const readCounter = task<ProcessSharedBuffer, number>({
+  f: (shared) => Atomics.load(shared.view(Int32Array), 0),
+});
+
+if (isMain) {
+  const cwd = process.cwd();
+  const name = `knitting-docker-counter-${process.pid}`;
+  const primitives = getDefaultProcessSharedBufferPrimitives();
+  const shared = ProcessSharedBuffer.create({
+    mode: "create",
+    name,
+    size: 64,
+  }, primitives);
+
+  const pool = createPool({
+    threads: 1,
+    worker: {
+      runtime: "process",
+      processRuntime: "node",
+      processSharedMemory: "named",
+      processCommandPrefix: [
+        // Knitting appends the actual Node worker command after this prefix.
+        "docker",
+        "run",
+        // Named shared memory needs a shared IPC namespace.
+        "--ipc=host",
+        // Mount the project so the container can import the worker module.
+        "-v",
+        `${cwd}:${cwd}`,
+        "-w",
+        cwd,
+        // Pass Knitting's boot payload through Docker.
+        "-e",
+        "KNITTING_PROCESS_WORKER",
+        "-e",
+        "KNITTING_PROCESS_WORKER_BOOT",
+        "node:24-trixie-slim",
+      ],
+    },
+    permission: "unsafe",
+  })({ readCounter });
+
+  try {
+    Atomics.store(shared.view(Int32Array), 0, 42);
+    console.log(await pool.call.readCounter(shared));
+  } finally {
+    await pool.shutdown();
+    shared.descriptor.mapping?.close?.();
+    primitives.unlinkSharedMemory?.(name);
+  }
+}
+```
+
+There are three moving parts:
+
+- `processSharedMemory: "named"` lets the Docker worker find Knitting's control
+  channel.
+- `ProcessSharedBuffer.create({ mode: "create", name, size })` makes the
+  payload buffer reopenable by name.
+- `--ipc=host` lets the container see the same POSIX shared-memory namespace.
+
+This is same-host communication. It is fast because both sides map the same
+bytes, but it is not a network transport and it deliberately shares IPC with the
+container. Use names like capabilities: generate them, keep them private, and
+unlink them when the shared memory is no longer needed.
 
 ### Current support
 
-Thread workers are the broadest path: they do not need native prebuilds or FFI.
-Process workers and `ProcessSharedBuffer` both use OS-backed shared memory, so
-their support follows the native backend for each runtime.
-
-For now, Windows support means thread workers only. Process workers and
-`ProcessSharedBuffer` are supported on POSIX targets: Linux and macOS.
-
-| Runtime and target | Thread workers | Process workers | `ProcessSharedBuffer` | Native path |
-| --- | --- | --- | --- | --- |
-| Node.js 22 / 24 on Linux x64 | Supported | Supported | Supported | Shipped Node `.node` prebuilds. |
-| Node.js 22 / 24 on macOS x64 | Supported | Supported | Supported | Shipped Node `.node` prebuilds. |
-| Node.js 22 / 24 on macOS arm64 | Supported | Supported | Supported | Shipped Node `.node` prebuilds. |
-| Node.js 22 / 24 on Windows x64 | Supported | Not supported | Not supported | Native shared memory is POSIX-only. |
-| Other POSIX Node.js ABI or arch | Supported | Local native build needed | Local native build needed | Run `bun run build:native` before using native shared memory. |
-| Deno 2+ on Linux/macOS, runtime-supported arch | Supported | Supported | Supported | Uses Deno FFI into libc; allow FFI permission when permissions are enabled. |
-| Deno 2+ on Windows | Supported | Not supported | Not supported | Current Deno backend is POSIX-only. |
-| Bun 1+ on Linux/macOS, runtime-supported arch | Supported | Supported | Supported | Uses Bun FFI into libc. |
-| Bun 1+ on Windows | Supported | Not supported | Not supported | Current Bun backend is POSIX-only. |
+Knitting supports Node.js 22+, Deno 2+, and Bun 1+ on Linux, macOS, and
+Windows.
+
+Thread workers work without native pieces. Process workers and
+`ProcessSharedBuffer` use the platform's shared-memory APIs. Release packages
+include the native prebuilds needed for the supported Node targets and Windows
+FFI path; if you are developing locally on a new Node ABI or architecture, run:
+
+```bash
+bun run build:native
+```
+
+For Deno projects with permissions enabled, allow FFI when using process
+workers or `ProcessSharedBuffer`.
 
 ## Runtime Safety
 
@@ -604,7 +795,7 @@ Build the package:
 bun run build
 ```
 
-Build the native shared-memory addon on Linux or macOS:
+Build the native shared-memory addon/prebuild for the current platform:
 
 ```bash
 bun run build:native
@@ -625,6 +816,12 @@ Emit JSON benchmark results:
 ./run.sh --json
 ```
 
+Compare inherited-fd and named-shared-memory process-worker startup:
+
+```bash
+node --no-warnings --experimental-transform-types bench/startup.ts --named-process-shm
+```
+
 For a file-by-file orientation, see [map.md](./map.md).
 
 ## License

package/map.md

@@ -33,8 +33,9 @@ The core flow is:
 
 - `build.ts`: Bundles `knitting.ts` to `out/` with Bun for a Node ESM target.
 - `scripts/build-native-addons.ts`: Compiles the native Node addons into
-  `build/Release/` on Linux and macOS. It finds Node headers/libs, splits user
-  flags, and builds the shared-memory and futex addons.
+  `build/Release/` on Linux, macOS, and Windows. It finds Node headers/libs,
+  splits user flags, builds the shared-memory and futex addons, and emits the
+  Windows FFI DLL used by Deno and Bun.
 - `run.sh`: Runs every top-level benchmark in `bench/` across Node, Deno, and
   Bun. `--json` writes JSON result files.
 
@@ -159,10 +160,15 @@ The core flow is:
 - `src/connections/deno.ts`: Deno FFI implementation for POSIX shared memory.
 - `src/connections/posix.ts`: POSIX constants, shared-memory naming, libc path
   detection, and close-on-exec helpers.
+- `src/connections/windows.ts`: Windows FFI loader and named file-mapping
+  primitives used by Deno and Bun.
 - `src/knitting_shared_memory.cc`: Native Node addon for shared-memory create,
-  map, unlink, and descriptor operations.
+  map, unlink, and descriptor operations, including Windows named mappings.
 - `src/knitting_shm.cc`: Native Node addon for futex/wait helpers used by parked
   workers.
+- `src/knitting_windows_shared_memory.cc`: Runtime-neutral Windows DLL exports
+  for creating, opening, mapping, and closing named shared-memory objects from
+  FFI runtimes.
 
 ## Permissions
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "knitting",
-  "version": "0.1.46",
+  "version": "0.1.50",
   "description": "Shared-memory IPC runtime for Node.js, Deno, and Bun.",
   "license": "Apache-2.0",
   "repository": {
@@ -30,6 +30,7 @@
     "map.md",
     "LICENSE",
     "prebuilds/**/*.node",
+    "prebuilds/**/*.dll",
     "scripts/build-native-addons.ts",
     "src/**/*.cc",
     "src/**/*.d.ts",

package/scripts/build-native-addons.ts

@@ -70,11 +70,6 @@ const nodeInfo = JSON.parse(runCapture(nodeBinary, [
   "JSON.stringify({arch:process.arch,execPath:process.execPath,modules:process.versions.modules,nodedir:process.config.variables.nodedir||null,platform:process.platform,version:process.versions.node})",
 ])) as NodeInfo;
 const isWindows = nodeInfo.platform === "win32";
-if (isWindows) {
-  throw new Error(
-    "Knitting native shared-memory addons are supported on Linux and macOS only",
-  );
-}
 const cacheRoot = join(
   resolve(
     Bun.env.KNITTING_NODE_CACHE_DIR ??
@@ -238,11 +233,25 @@ const addons = [
     output: "build/Release/knitting_shm.node",
   },
 ];
+const ffiLibraries = isWindows
+  ? [
+    {
+      name: "knitting_windows_shared_memory",
+      source: "src/knitting_windows_shared_memory.cc",
+      output: "build/Release/knitting_windows_shared_memory.dll",
+    },
+  ]
+  : [];
 const prebuildDir = join(
   root,
   "prebuilds",
   `${nodeInfo.platform}-${nodeInfo.arch}-node-${nodeInfo.modules}`,
 );
+const ffiPrebuildDir = join(
+  root,
+  "prebuilds",
+  `${nodeInfo.platform}-${nodeInfo.arch}`,
+);
 
 console.log(`Using Node: ${nodeInfo.execPath}`);
 console.log(`Using Node module ABI: ${nodeInfo.modules}`);
@@ -282,6 +291,23 @@ for (const addon of addons) {
   copiedPrebuilds.push(prebuildPath);
 }
 
+for (const library of ffiLibraries) {
+  const outputPath = join(root, library.output);
+  run(cxx, [
+    ...compileFlags,
+    join(root, library.source),
+    "/link",
+    `/OUT:${outputPath}`,
+    ...extraLdFlags,
+  ]);
+  builtAddons.push(library.output);
+
+  mkdirSync(ffiPrebuildDir, { recursive: true });
+  const prebuildPath = join(ffiPrebuildDir, `${library.name}.dll`);
+  copyFileSync(outputPath, prebuildPath);
+  copiedPrebuilds.push(prebuildPath);
+}
+
 console.log(
   `Built ${builtAddons.length} native addon${
     builtAddons.length === 1 ? "" : "s"