@ricsam/r5d-worker 0.0.111 → 0.0.112

package/README.md

@@ -71,16 +71,21 @@ worker exec macos bun test
 
 Agent shell commands receive no stdin by default: prompts read immediate EOF instead of hanging. A command started with `interactive: true` is spawned with a writable stdin pipe, and the agent's `shell_write` tool delivers input through the `exec_stdin` protocol message. The worker acknowledges each write with an `operation_result` carrying the delivered byte count, closes stdin on an explicit `eof`, on cancellation, on timeout, and at process exit, and advertises this support through the `execStdinV1` capability in its hello message. Servers refuse interactive runs and stdin writes for workers without the capability, so this is a protocol-compatible release that uses the ordinary worker update flow. Delivered stdin is audited server-side in the run's `process-runs/<runId>/stdin.log`.
 
-### Command resource limits
+### Optional command launcher
 
-Managed Linux workers require cgroup v2 delegation with `Delegate=memory pids`, `DelegateSubgroup=control`, and `OOMPolicy=continue`. The supervisor and control runtime stay in the service's `control` subgroup. Every command and PTY enters its own memory-limited cgroup before executing user code, and all descendants remain under `r5d-worker.service` for credential-generation cleanup. Each job has a process limit of 512 and group OOM termination; a job that exhausts memory reports a command/terminal resource failure instead of stopping the worker. Control processes receive OOM protection, while the command wrapper removes that protection before executing user code.
+By default the worker launches commands and terminals directly on every platform. It imposes no memory limit, concurrency limit, or OS setup requirement. The operating system may terminate a command or the worker itself under memory pressure; a killed process alone is not evidence of an OOM. Inline RPC output is limited to 8 MiB per stream to bound worker-side buffering; use streaming execution for larger output.
 
-Admission is shared across all projects in the runtime: two general commands or terminals can run concurrently. A separate two-slot utility lane, capped at 256 MiB per job, serves explicitly effect-free inline RPC commands. File operations, cancellation, and connection recovery do not wait in either queue. Inline RPC output is limited to 8 MiB per stream so a small producer cannot fill the control runtime; larger output requires streaming execution. General jobs reserve the rest of the configured aggregate budget. The managed VM sets direct jobs to 50% of effective RAM and uses a separately bounded default BuildKit builder (up to 25%, capped at 2 GiB), leaving at least 25% for the OS and control services. Effective RAM honors the tightest ancestor cgroup limit.
+Operators may set `R5D_WORKER_COMMAND_LAUNCHER` to a JSON argv array, such as `["node", "/opt/worker/launcher.mjs", "/etc/worker/policy.json"]`. The worker starts this trusted executable once after acquiring credential authority and keeps it alive across websocket reconnects. Configuration is read from the worker's environment, never from command message environments, and the controller inherits only basic non-secret host environment fields. Arguments are passed literally without shell splitting.
 
-Queued commands retain their durable operation identities across reconnects. They wait at most 60 seconds before returning a capacity error without starting a command; cancellation also prevents a command whose asynchronous preparation is still in progress from starting. This bound prevents indefinite capacity deadlocks when all running shells wait on nested `r5dctl` work. Retry later or stop an occupied command/terminal when capacity is exhausted. Transient disconnects preserve running commands and PTYs; expiry of the five-minute execution lease closes them and cancels queued work.
+The launcher owns admission, OS isolation, budgets, and diagnostic evidence. Its JSONL protocol uses `{requestId, method, ...fields}` requests and `{requestId, result}` or `{requestId, error}` replies:
 
-The standalone default aggregate limit is 70% of effective RAM. Operators can set `R5D_WORKER_JOBS_MEMORY_PERCENT` (1–70), `R5D_WORKER_JOBS_MEMORY_BYTES`, `R5D_WORKER_JOB_MEMORY_BYTES`, `R5D_WORKER_UTILITY_MEMORY_BYTES`, `R5D_WORKER_JOB_CONCURRENCY`, and `R5D_WORKER_UTILITY_CONCURRENCY`. The sum of all slot reservations must fit the aggregate budget. `R5D_WORKER_JOB_QUEUE_TIMEOUT_MS` is capped at 60000; `R5D_WORKER_JOB_QUEUE_LIMIT` defaults to 128. Managed services set `R5D_WORKER_RESOURCE_ISOLATION=required` and refuse command service if delegation is unavailable. Direct workers on other platforms use bounded concurrency and explicitly report `concurrency-only`; they do not claim kernel memory isolation.
+- `initialize` returns `{}` once ready; failure prevents command admission.
+- `acquire` receives `{id, kind, workspaceEffect?, sessionId?}` and returns `{argvPrefix: string[]}`. The worker prepends this argv to the command or terminal shell without interpreting it.
+- `cancel` receives `{id}` and irreversibly revokes queued or acquired work. `release` receives `{id}` and acknowledges only after cleanup. Both must be idempotent, including release before a delayed acquisition response.
+- `diagnose` receives `{id}` and returns `{message?: string}` with launcher-specific evidence. The worker does not infer a memory failure from an exit signal.
 
-Cgroups constrain direct descendants, not workloads launched by separate daemons. The managed default `BUILDX_BUILDER` selects the bounded build daemon, but explicit alternate builders and arbitrary `docker run`/Compose containers need their own memory limits. These controls prevent accidental resource exhaustion; they are not a hostile same-UID sandbox.
+Requests must be processed concurrently so cancellation and cleanup cannot wait behind queued admission. Controller RPCs are bounded (10 seconds, or 90 seconds for acquisition); external policy must reject excessive queue waits within that bound. Controller loss or protocol failure stops command admission, preserves uncertain outcomes for recovery, reaps ordinary children, and restarts the runtime through its supervisor. It never falls back to direct spawning. Shutdown closes controller stdin; the controller must reap its jobs before exiting.
+
+Cancellation is checked again immediately before spawn. Transient disconnects preserve commands and terminals; expiry of the five-minute execution lease cancels agent work and closes terminals. File operations, communication recovery, and cancellation do not enter launcher admission. Internal worker maintenance subprocesses are also outside this hook. Deployment-specific limits and service protection are documented in [Worker deployment resources](../../docs/worker-deployment-resources.md).
 
 When the connected `r5d-browser` requests a port forward, the worker opens each relayed connection only to `127.0.0.1` on the requested worker port. Browser-side and worker-side ports may differ. The worker never opens a public listener, and a disconnected worker leaves the browser's long-lived mapping unavailable until the same worker label reconnects.

package/dist/cjs/command-launcher.cjs

@@ -0,0 +1,345 @@
+"use strict";
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+var command_launcher_exports = {};
+__export(command_launcher_exports, {
+  CommandLaunchCancelledError: () => CommandLaunchCancelledError,
+  CommandLauncherUnavailableError: () => CommandLauncherUnavailableError,
+  WorkerCommandLauncher: () => WorkerCommandLauncher,
+  commandLauncherEnvironment: () => commandLauncherEnvironment,
+  createWorkerCommandLauncher: () => createWorkerCommandLauncher,
+  parseCommandLauncher: () => parseCommandLauncher
+});
+module.exports = __toCommonJS(command_launcher_exports);
+var import_node_child_process = require("node:child_process");
+class CommandLauncherUnavailableError extends Error {
+  constructor(message, cause) {
+    super(message, { cause });
+    this.name = "CommandLauncherUnavailableError";
+  }
+}
+class CommandLaunchCancelledError extends Error {
+  constructor() {
+    super("Command cancelled before it started");
+    this.name = "CommandLaunchCancelledError";
+  }
+}
+function parseCommandLauncher(configuration) {
+  if (configuration === void 0) return void 0;
+  let argv;
+  try {
+    argv = JSON.parse(configuration);
+  } catch {
+    throw new Error("R5D_WORKER_COMMAND_LAUNCHER must be a JSON argv array");
+  }
+  if (!Array.isArray(argv) || argv.length === 0 || argv.some((part) => typeof part !== "string" || part.includes("\0")) || !argv[0])
+    throw new Error("R5D_WORKER_COMMAND_LAUNCHER must be a nonempty JSON argv array without NUL characters");
+  return argv;
+}
+function commandLauncherEnvironment(source = process.env) {
+  const result = {};
+  for (const [key, value] of Object.entries(source)) {
+    if (/^(PATH|HOME|USER|LOGNAME|TMP|TMPDIR|TEMP|SHELL|SYSTEMROOT|COMSPEC|PATHEXT|LANG|LC_[A-Z_]+)$/i.test(key)) result[key] = value;
+  }
+  return result;
+}
+class WorkerCommandLauncher {
+  constructor(options = {}) {
+    this.options = options;
+    this.mode = options.argv ? "external" : "direct";
+    if (!options.argv) {
+      this.exited = Promise.resolve();
+      return;
+    }
+    this.child = (0, import_node_child_process.spawn)(options.argv[0], options.argv.slice(1), {
+      env: options.env ?? commandLauncherEnvironment(),
+      stdio: ["pipe", "pipe", "pipe"]
+    });
+    this.exited = new Promise((resolve) => {
+      this.child.once("close", (code, signal) => {
+        if (!this.closing) this.fail(new Error(`Configured command launcher exited (${signal ?? code ?? "unknown"})`));
+        resolve();
+      });
+    });
+    this.child.once("error", (error) => this.fail(error));
+    this.child.stdin.on("error", (error) => {
+      if (!this.closing) this.fail(error);
+    });
+    let buffered = "";
+    this.child.stdout.setEncoding("utf8");
+    this.child.stdout.on("data", (data) => {
+      buffered += data;
+      if (Buffer.byteLength(buffered) > 1024 * 1024) {
+        this.fail(new Error("Command launcher response exceeded the protocol limit"));
+        return;
+      }
+      while (buffered.includes("\n")) {
+        const newline = buffered.indexOf("\n");
+        const line = buffered.slice(0, newline);
+        buffered = buffered.slice(newline + 1);
+        try {
+          this.receive(JSON.parse(line));
+        } catch (error) {
+          this.fail(error instanceof Error ? error : new Error(String(error)));
+          return;
+        }
+      }
+    });
+    this.child.stderr.setEncoding("utf8");
+    this.child.stderr.on("data", (data) => options.onStderr?.(data));
+  }
+  options;
+  mode;
+  child;
+  pending = /* @__PURE__ */ new Map();
+  launches = /* @__PURE__ */ new Map();
+  exited;
+  failure;
+  closing = false;
+  initialized = false;
+  closePromise;
+  sequence = 0;
+  async initialize() {
+    this.assertHealthy();
+    if (this.initialized) return;
+    if (this.child) {
+      try {
+        await this.rpc("initialize");
+      } catch (error) {
+        this.fail(error instanceof Error ? error : new Error(String(error)));
+        await this.close();
+        throw error;
+      }
+    }
+    this.initialized = true;
+  }
+  get activeIds() {
+    return [...this.launches.keys()];
+  }
+  async acquire(request) {
+    this.assertHealthy();
+    if (!this.initialized) throw new Error("Command launcher is not initialized");
+    request.assertAdmission();
+    if (this.launches.has(request.id)) throw new Error(`Duplicate command launch identity: ${request.id}`);
+    let rejectAdmission;
+    const cancellation = new Promise((_resolve, reject) => {
+      rejectAdmission = reject;
+    });
+    void cancellation.catch(() => {
+    });
+    const record = { request, revoked: false, released: false, rejectAdmission };
+    this.launches.set(request.id, record);
+    let prefix = [];
+    let admissionTimer;
+    try {
+      if (this.child) {
+        const { assertAdmission: _assertAdmission, ...fields } = request;
+        const acquisition = this.rpc("acquire", fields, this.options.acquisitionTimeoutMs ?? 9e4).then(async (result) => {
+          if (record.revoked) {
+            await this.releaseRecord(record);
+            throw new CommandLaunchCancelledError();
+          }
+          if (!Array.isArray(result.argvPrefix) || result.argvPrefix.some((part) => typeof part !== "string" || part.includes("\0")))
+            throw this.fail(new Error("Command launcher acquire returned an invalid argvPrefix"));
+          return result.argvPrefix;
+        });
+        admissionTimer = setInterval(() => {
+          try {
+            request.assertAdmission();
+          } catch (error) {
+            record.revoked = true;
+            record.rejectAdmission(error);
+            void this.cancel(request.id).catch(() => {
+            });
+          }
+        }, 100);
+        prefix = await Promise.race([acquisition, cancellation]);
+      }
+      request.assertAdmission();
+      this.assertHealthy();
+      if (record.revoked) throw new CommandLaunchCancelledError();
+    } catch (error) {
+      if (!this.failure && !this.closing) await this.releaseRecord(record);
+      throw error;
+    } finally {
+      if (admissionTimer) clearInterval(admissionTimer);
+    }
+    return {
+      wrap: (argv) => {
+        this.assertHealthy();
+        request.assertAdmission();
+        if (record.revoked) throw new CommandLaunchCancelledError();
+        if (record.released) throw new Error("Command launch lease is already released");
+        if (!argv.length) throw new Error("Command argv must not be empty");
+        return [...prefix, ...argv];
+      },
+      cancel: () => this.cancel(request.id),
+      diagnose: async () => {
+        this.assertHealthy();
+        if (!this.child) return void 0;
+        try {
+          const result = await this.rpc("diagnose", { id: request.id });
+          if (result.message !== void 0 && typeof result.message !== "string")
+            throw new Error("Command launcher returned an invalid diagnostic");
+          return result.message;
+        } catch (error) {
+          throw this.fail(error instanceof Error ? error : new Error(String(error)));
+        }
+      },
+      release: () => this.releaseRecord(record)
+    };
+  }
+  async cancel(id) {
+    const record = this.launches.get(id);
+    if (!record || record.released) return;
+    record.revoked = true;
+    record.rejectAdmission(new CommandLaunchCancelledError());
+    if (this.child) {
+      try {
+        await this.rpc("cancel", { id });
+      } catch (error) {
+        throw this.fail(error instanceof Error ? error : new Error(String(error)));
+      }
+    }
+  }
+  async cancelSession(sessionId) {
+    await this.cancelMatching((_id, ownerSessionId) => ownerSessionId === sessionId);
+  }
+  async cancelMatching(matches) {
+    const results = await Promise.allSettled(
+      [...this.launches].filter(([id, record]) => matches(id, record.request.sessionId)).map(([id]) => this.cancel(id))
+    );
+    const errors = results.flatMap((result) => result.status === "rejected" ? [result.reason] : []);
+    if (errors.length) throw new AggregateError(errors, "Command launcher cancellation was incomplete");
+  }
+  close() {
+    return this.closePromise ??= (async () => {
+      this.closing = true;
+      for (const record of this.launches.values()) {
+        record.revoked = true;
+        record.rejectAdmission(new CommandLaunchCancelledError());
+      }
+      this.rejectPending(new CommandLauncherUnavailableError("Command launcher is shutting down"));
+      this.child?.stdin.end();
+      const exited = await this.waitForExit(6e3);
+      if (!exited) {
+        this.child?.kill("SIGKILL");
+        await this.waitForExit(1e3);
+      }
+    })();
+  }
+  async waitForExit(timeoutMs) {
+    let timer;
+    try {
+      return await Promise.race([
+        this.exited.then(() => true),
+        new Promise((resolve) => {
+          timer = setTimeout(() => resolve(false), timeoutMs);
+        })
+      ]);
+    } finally {
+      if (timer) clearTimeout(timer);
+    }
+  }
+  releaseRecord(record) {
+    return record.release ??= (async () => {
+      record.revoked = true;
+      if (this.child) {
+        try {
+          await this.rpc("release", { id: record.request.id });
+        } catch (error) {
+          throw this.fail(error instanceof Error ? error : new Error(String(error)));
+        }
+      }
+      record.released = true;
+      this.launches.delete(record.request.id);
+    })();
+  }
+  assertHealthy() {
+    if (this.failure) throw this.failure;
+    if (this.closing) throw new CommandLauncherUnavailableError("Command launcher is shutting down");
+  }
+  rpc(method, fields = {}, timeoutMs = this.options.requestTimeoutMs ?? 1e4) {
+    try {
+      this.assertHealthy();
+    } catch (error) {
+      return Promise.reject(error);
+    }
+    const requestId = String(++this.sequence);
+    return new Promise((resolve, reject) => {
+      const timer = setTimeout(() => this.fail(new Error(`Command launcher ${method} request timed out`)), timeoutMs);
+      this.pending.set(requestId, { resolve, reject, timer });
+      this.child.stdin.write(`${JSON.stringify({ requestId, method, ...fields })}
+`, (error) => {
+        if (error) this.fail(error);
+      });
+    });
+  }
+  receive(value) {
+    if (!value || typeof value !== "object") throw new Error("Invalid command launcher response");
+    const response = value;
+    if (typeof response.requestId !== "string") throw new Error("Command launcher response is missing requestId");
+    const request = this.pending.get(response.requestId);
+    if (!request) throw new Error("Command launcher returned an unknown requestId");
+    this.pending.delete(response.requestId);
+    clearTimeout(request.timer);
+    if (typeof response.error === "string") {
+      request.reject(new Error(response.error));
+      return;
+    }
+    if (!response.result || typeof response.result !== "object" || Array.isArray(response.result)) {
+      const error = new Error("Command launcher response is missing result");
+      request.reject(error);
+      throw error;
+    }
+    request.resolve(response.result);
+  }
+  rejectPending(error) {
+    for (const request of this.pending.values()) {
+      clearTimeout(request.timer);
+      request.reject(error);
+    }
+    this.pending.clear();
+  }
+  fail(error) {
+    if (this.failure || this.closing) return this.failure ?? error;
+    this.failure = new CommandLauncherUnavailableError(`Command launcher unavailable: ${error.message}`, error);
+    for (const record of this.launches.values()) {
+      record.revoked = true;
+      record.rejectAdmission(this.failure);
+    }
+    this.rejectPending(this.failure);
+    this.child?.kill("SIGTERM");
+    this.options.onFailure?.(this.failure, this.activeIds);
+    return this.failure;
+  }
+}
+async function createWorkerCommandLauncher(options = {}) {
+  const launcher = new WorkerCommandLauncher({ ...options, argv: parseCommandLauncher(options.configuration) });
+  await launcher.initialize();
+  return launcher;
+}
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  CommandLaunchCancelledError,
+  CommandLauncherUnavailableError,
+  WorkerCommandLauncher,
+  commandLauncherEnvironment,
+  createWorkerCommandLauncher,
+  parseCommandLauncher
+});

package/dist/cjs/inline-output.cjs

@@ -0,0 +1,62 @@
+"use strict";
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+var inline_output_exports = {};
+__export(inline_output_exports, {
+  WORKER_INLINE_OUTPUT_MAX_BYTES: () => WORKER_INLINE_OUTPUT_MAX_BYTES,
+  collectWorkerInlineOutput: () => collectWorkerInlineOutput
+});
+module.exports = __toCommonJS(inline_output_exports);
+const WORKER_INLINE_OUTPUT_MAX_BYTES = 8 * 1024 * 1024;
+async function collectWorkerInlineOutput(stream, maxBytes = WORKER_INLINE_OUTPUT_MAX_BYTES, signal) {
+  if (!stream) return "";
+  signal?.throwIfAborted();
+  const reader = stream.getReader();
+  const abort = () => {
+    void reader.cancel().catch(() => {
+    });
+  };
+  signal?.addEventListener("abort", abort, { once: true });
+  const decoder = new TextDecoder();
+  const parts = [];
+  let bytes = 0;
+  try {
+    while (true) {
+      const next = await reader.read();
+      signal?.throwIfAborted();
+      if (next.done) break;
+      bytes += next.value.byteLength;
+      if (bytes > maxBytes) {
+        void reader.cancel().catch(() => {
+        });
+        throw new Error(`Inline command output exceeded ${maxBytes} bytes. Use streaming execution for large output.`);
+      }
+      parts.push(decoder.decode(next.value, { stream: true }));
+    }
+    parts.push(decoder.decode());
+    return parts.join("");
+  } finally {
+    signal?.removeEventListener("abort", abort);
+    reader.releaseLock();
+  }
+}
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  WORKER_INLINE_OUTPUT_MAX_BYTES,
+  collectWorkerInlineOutput
+});