@ai-ecoverse/slicc-kernel 1.8.12 → 1.10.0

package/README.md

@@ -67,6 +67,29 @@ Starts `argv` as the session leader on a new terminal and resolves once it is ru
 
 Stops the kernel worker and every process. Pending and later calls reject.
 
+### `kernel.connect() → Promise<MessagePort>`
+
+Makes a port for another client of the same kernel. Hand it to a dedicated worker, or through a SharedWorker to another tab: it can be transferred any number of times. All clients share one process table, so a terminal on the page can `ps` and `kill` what a worker started.
+
+### `attachKernel(port, options?) → Promise<KernelClient>`
+
+Attaches to a kernel over a port from `kernel.connect()`, in any realm: a worker, another tab, or Node. `options.timeoutMs` (10 s by default) bounds the handshake: a port with no kernel behind it rejects with `KernelGoneError`.
+
+| Client | |
+|---|---|
+| `spawn(argv, { cwd, env, stdin, onStdout, onStderr })` | runs over pipes and resolves with `{ pid, pgid, exited, signal(name) }` once the process has started; output arrives in chunks, as bytes, and the kernel keeps no copy; every spawn leads its own process group, which `signal` (default `SIGTERM`) signals; a program that cannot start rejects with `code: 'ENOENT'` |
+| `run(argv, options?)` | `spawn` and wait: `{ pid, status, stdout, stderr }` as text, with `onStdout`/`onStderr` streaming text |
+| `openTerminal(argv, options?)` | a pty session, as `kernel.openTerminal` |
+| `ps()` | the kernel's processes: `{ pid, ppid, pgid, sid, argv, tty, started, state }`, `state` being `'S'` or `'Z'` |
+| `kill(pid, signal?)` | any process, or a process group with a negative pid; no such process rejects with `code: 'ESRCH'` |
+| `fs` | `readFile`, `readText`, `writeFile(path, string \| bytes)`, `stat`, `lstat`, `readdir`, `mkdir` (with parents), `rm(path, { force })` (recursive), `rename`, `realpath`, `symlink(target, path)`, `readlink`, `exists`, on the processes' file system; failures reject with `KernelCallError` and a POSIX `code` |
+| `fetch({ url, method, headers, body, signal })` | through the kernel's network transport (the one the page passed to `createKernel`), with a streaming body; `transport` is the same as a `NetworkTransport` |
+| `fs.watch(paths, { recursive }, onChange)` | resolves with `{ close() }`; `onChange` gets `{ paths }`, the changed paths at, below (or, without `recursive`, directly in) the watched ones, batched per task, or `{ overflow: true }` when there were too many to list, to rescan; it covers every change made through this kernel, by its processes and its clients |
+| `close({ kill })` | detaches: the client's processes keep running unless `kill` is set, which ends their process groups with `SIGKILL`; its terminals are hung up |
+| `closed` | resolves with the error that ended the client |
+
+When the kernel goes away (the page closed or reloaded, `terminate()`), pending calls and `exited` reject with `KernelGoneError`, and so do later calls. Each side holds a Web Lock and waits on the other's, since a `MessagePort` reports no close in browsers; in Node, the port's `close` event does the same. The first message is a handshake on the protocol version, `1.1` (`1.0` had no `watch`, which rejects with `code: 'ENOSYS'` on such a kernel): a client or kernel of another major version is refused with an error naming both.
+
 ### Headless in Node, for tests
 
 `@ai-ecoverse/slicc-kernel/node` runs the same kernel in Node without a browser, so packages built for SLICC can test against it. It is a testing entry, not a supported runtime: there is no OPFS and no isolation, and nothing beyond what its tests use is promised.
@@ -80,7 +103,7 @@ const { status, stdout } = await kernel.run(['bash', '-c', 'echo hi'], { cwd: '/
 kernel.terminate();
 ```
 
-`createNodeKernel({ root, modules, env, network, worker })` takes the options of `createKernel` except `metadata`. `root` is an in-memory directory by default (`memoryRoot()` makes another), and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal` and `terminate` as above, plus `root`, `writeFile(path, data)` (creating the parent directories) and `readFile(path)` to put files in place and read results. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`).
+`createNodeKernel({ root, modules, env, network, worker })` takes the options of `createKernel` except `metadata`. `root` is an in-memory directory by default (`memoryRoot()` makes another), and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal` and `terminate` as above, plus `root`, `writeFile(path, data)` (creating the parent directories) and `readFile(path)` to put files in place and read results. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`). `connect()` and `attachKernel` work as in the browser: the port is a `worker_threads` `MessagePort`, which a worker thread can attach with, and `terminate()` ends every attached client.
 
 ## Commands
 
@@ -156,6 +179,10 @@ The proxy answers `200` with `Content-Type: application/vnd.slicc.raw-fetch`: a
 
 Directories are renamed with `FileSystemHandle.move()` where available, else by copy and delete.
 
+### `/proc`
+
+Every process of the kernel has `/proc/<pid>/` with `cmdline`, `comm`, `stat`, `statm` and `status`, in the formats procps reads, and `/proc/self/` has the same files for the process that reads them. They come from the kernel's process table when they are opened, so a terminal sees the processes of every client; an exec'd program shows under the pid its parent knows. `/proc/uptime`, `/proc/loadavg`, `/proc/stat` and `/proc/meminfo` are there too, and `/proc/mounts`. Pids, parents, process groups, sessions, command lines, terminals, start and boot times are real; CPU times, memory sizes, load and `MemTotal` (from `navigator.deviceMemory`, else 4 GiB) are placeholders. Only Emscripten processes see this `/proc`.
+
 ### Metadata
 
 OPFS stores names, bytes, sizes and modification times, nothing else. Everything POSIX needs on top of that lives in an IndexedDB sidecar, so it survives reloads and new kernels:

package/dist/index.js

@@ -164,6 +164,424 @@ function serveTransport(peer, transport) {
   };
 }
 
+// src/kernel/net/remote-transport.ts
+var TransportError = class extends Error {
+  status;
+  code;
+  constructor(message, status, code) {
+    super(message);
+    this.status = status;
+    this.code = code;
+  }
+};
+var RemoteTransport = class {
+  traits;
+  port;
+  waiting = /* @__PURE__ */ new Map();
+  nextId = 0;
+  failure;
+  constructor(port, traits) {
+    this.port = port;
+    this.traits = traits;
+  }
+  receive(reply) {
+    const waiter = this.waiting.get(reply.nid);
+    if (!waiter) return;
+    this.waiting.delete(reply.nid);
+    if (reply.net === "error") {
+      waiter.reject(new TransportError(reply.message, reply.status, reply.code));
+    } else waiter.resolve(reply);
+  }
+  fail(error) {
+    this.failure = error;
+    for (const waiter of this.waiting.values()) waiter.reject(error);
+    this.waiting.clear();
+  }
+  ask(call) {
+    if (this.failure) return Promise.reject(this.failure);
+    return new Promise((resolve, reject) => {
+      this.waiting.set(call.nid, { resolve, reject });
+      this.port.postMessage(call);
+    });
+  }
+  cancel(nid) {
+    const waiter = this.waiting.get(nid);
+    this.waiting.delete(nid);
+    waiter?.reject(new TransportError("the request was cancelled"));
+    this.port.postMessage({ net: "cancel", nid });
+  }
+  async fetch(request) {
+    const nid = ++this.nextId;
+    const { signal } = request;
+    signal.throwIfAborted();
+    let finished = false;
+    const abort = () => {
+      finished = true;
+      this.cancel(nid);
+    };
+    signal.addEventListener("abort", abort, { once: true });
+    const finish = () => {
+      if (finished) return;
+      finished = true;
+      signal.removeEventListener("abort", abort);
+    };
+    let head;
+    try {
+      head = await this.ask({
+        net: "fetch",
+        nid,
+        url: request.url,
+        method: request.method,
+        headers: request.headers,
+        ...request.body ? { body: request.body } : {}
+      });
+    } catch (e) {
+      finish();
+      throw e;
+    }
+    const { status, statusText, headers } = head;
+    const read = async () => {
+      if (finished) return void 0;
+      const reply = await this.ask({ net: "read", nid }).catch((e) => {
+        finish();
+        throw e;
+      });
+      if (reply.net === "chunk") return reply.bytes;
+      finish();
+      return void 0;
+    };
+    const cancel = async () => {
+      if (finished) return;
+      finish();
+      this.cancel(nid);
+    };
+    return {
+      status,
+      statusText,
+      headers,
+      cancel,
+      body: {
+        [Symbol.asyncIterator]: () => ({
+          next: async () => {
+            const bytes = await read();
+            return bytes ? { value: bytes, done: false } : { value: void 0, done: true };
+          },
+          return: async () => {
+            await cancel();
+            return { value: void 0, done: true };
+          }
+        })
+      }
+    };
+  }
+};
+
+// src/client/protocol.ts
+var PROTOCOL = [1, 1];
+function locksOf(scope = globalThis) {
+  const locks = scope.navigator?.locks;
+  return typeof locks?.request === "function" ? locks : void 0;
+}
+function versionError(theirs) {
+  const major = Array.isArray(theirs) ? theirs[0] : void 0;
+  if (major === PROTOCOL[0]) return void 0;
+  const named = typeof major === "number" ? `${major}.x` : "without a version";
+  return `slicc-kernel client protocol ${named} is not supported: this side speaks ${PROTOCOL.join(".")}`;
+}
+
+// src/client/attach.ts
+var KernelGoneError = class extends Error {
+  constructor(message = "the slicc-kernel this client was attached to is gone") {
+    super(message);
+    this.name = "KernelGoneError";
+  }
+};
+var KernelCallError = class extends Error {
+  code;
+  constructor(message, code) {
+    super(message);
+    this.name = "KernelCallError";
+    this.code = code;
+  }
+};
+var encoder = new TextEncoder();
+var bytesOf = (data) => typeof data === "string" ? encoder.encode(data) : data;
+var UNAVAILABLE = {
+  manualRedirects: false,
+  encodedBodies: false,
+  maxRequestBody: 0,
+  unavailable: true
+};
+async function holdLock(locks) {
+  if (!locks) return { release() {
+  } };
+  const name = `slicc-kernel-client:${crypto.randomUUID()}`;
+  const granted = Promise.withResolvers();
+  const held = Promise.withResolvers();
+  void locks.request(name, () => {
+    granted.resolve();
+    return held.promise;
+  });
+  await granted.promise;
+  return { name, release: () => held.resolve() };
+}
+function concat(chunks3) {
+  const out = new Uint8Array(chunks3.reduce((n, c) => n + c.length, 0));
+  let at = 0;
+  for (const chunk of chunks3) {
+    out.set(chunk, at);
+    at += chunk.length;
+  }
+  return out;
+}
+var Channel = class {
+  greeted = Promise.withResolvers();
+  ended = Promise.withResolvers();
+  remote;
+  failure;
+  pending = /* @__PURE__ */ new Map();
+  watching = /* @__PURE__ */ new Map();
+  protocol = PROTOCOL;
+  nextId = 0;
+  port;
+  release;
+  constructor(port, release) {
+    this.port = port;
+    this.release = release;
+    this.greeted.promise.catch(() => void 0);
+    port.addEventListener("message", (event) => this.receive(event.data));
+    port.addEventListener("close", () => this.end(new KernelGoneError()));
+    port.start?.();
+  }
+  post(message) {
+    if (!this.failure) this.port.postMessage(message);
+  }
+  end(error) {
+    if (this.failure) return;
+    this.failure = error;
+    this.greeted.reject(error);
+    this.remote?.fail(error);
+    for (const call of this.pending.values()) call.reject(error);
+    this.pending.clear();
+    this.release();
+    this.port.close?.();
+    this.ended.resolve(error);
+  }
+  request(call, handlers = {}) {
+    const id = ++this.nextId;
+    const { failure: failure2 } = this;
+    if (failure2) return { id, done: Promise.reject(failure2) };
+    const done = new Promise((resolve, reject) => {
+      this.pending.set(id, { resolve, reject, ...handlers });
+      this.port.postMessage({ ...call, id });
+    });
+    return { id, done };
+  }
+  call(call) {
+    return this.request(call).done;
+  }
+  receive(data) {
+    if (data.watch !== void 0) {
+      this.watching.get(data.watch)?.(data.change);
+      return;
+    }
+    if (data.bye !== void 0) {
+      this.end(new KernelGoneError(data.bye));
+      return;
+    }
+    if (data.hello) {
+      this.greeted.resolve(data.hello);
+      return;
+    }
+    if (data.net !== void 0) {
+      this.remote?.receive(data);
+      return;
+    }
+    const reply = data;
+    const call = this.pending.get(reply.id);
+    if (!call) return;
+    if (reply.fd !== void 0) {
+      call.output?.(reply.fd, reply.bytes);
+      return;
+    }
+    if (reply.started !== void 0) {
+      call.started?.(reply.started);
+      return;
+    }
+    this.pending.delete(reply.id);
+    if (reply.error !== void 0) call.reject(new KernelCallError(reply.error, reply.code));
+    else call.resolve(reply.result);
+  }
+};
+async function spawnOn(channel, argv, opts) {
+  const { onStdout, onStderr, stdin, group: _, ...rest } = opts;
+  const started = Promise.withResolvers();
+  const options = { ...rest, ...stdin !== void 0 ? { stdin: bytesOf(stdin) } : {} };
+  const { done } = channel.request(
+    { op: "spawn", argv, options },
+    {
+      output: (fd, bytes) => (fd === 1 ? onStdout : onStderr)?.(bytes),
+      started: started.resolve
+    }
+  );
+  done.catch(started.reject);
+  const pid = await started.promise;
+  return {
+    pid,
+    pgid: pid,
+    exited: done,
+    signal: async (name = "SIGTERM") => void await channel.call({ op: "kill", pid: -pid, signal: name })
+  };
+}
+async function runOn(channel, argv, opts) {
+  const { onStdout, onStderr, ...rest } = opts;
+  const chunks3 = { 1: [], 2: [] };
+  const live = { 1: new TextDecoder(), 2: new TextDecoder() };
+  const tee = (fd, on) => (bytes) => {
+    chunks3[fd].push(bytes);
+    on?.(live[fd].decode(bytes, { stream: true }));
+  };
+  const child = await spawnOn(channel, argv, {
+    ...rest,
+    onStdout: tee(1, onStdout),
+    onStderr: tee(2, onStderr)
+  });
+  const status = await child.exited;
+  const text = (fd) => new TextDecoder().decode(concat(chunks3[fd]));
+  return { pid: child.pid, status, stdout: text(1), stderr: text(2) };
+}
+async function terminalOn(channel, argv, opts) {
+  const { onData, ...options } = opts;
+  let listener = onData ?? null;
+  const backlog = [];
+  const started = Promise.withResolvers();
+  const { id, done } = channel.request(
+    { op: "open-terminal", argv, options },
+    {
+      output: (_fd, bytes) => listener ? listener(bytes) : backlog.push(bytes),
+      started: started.resolve
+    }
+  );
+  done.catch(started.reject);
+  const pid = await started.promise;
+  const send2 = (action) => void channel.call({ op: "terminal", terminal: id, ...action }).catch(() => void 0);
+  return {
+    pid,
+    exited: done,
+    get onData() {
+      return listener;
+    },
+    set onData(next) {
+      listener = next;
+      if (next) for (const bytes of backlog.splice(0)) next(bytes);
+    },
+    write: (data) => send2({ action: "write", bytes: bytesOf(data) }),
+    resize: (cols, rows) => send2({ action: "resize", cols, rows }),
+    signal: (name) => send2({ action: "signal", signal: name }),
+    close: () => send2({ action: "close" })
+  };
+}
+function fsOn(channel) {
+  const fs = (method, ...args) => channel.call({ op: "fs", method, args });
+  const done = async (method, ...args) => void await fs(method, ...args);
+  return {
+    readFile: (path) => fs("readFile", path),
+    readText: async (path) => new TextDecoder().decode(await fs("readFile", path)),
+    writeFile: (path, data) => done("writeFile", path, data),
+    stat: (path) => fs("stat", path),
+    lstat: (path) => fs("lstat", path),
+    readdir: (path) => fs("readdir", path),
+    mkdir: (path) => done("mkdir", path),
+    rm: (path, o = {}) => done("rm", path, o.force === true),
+    rename: (from, to) => done("rename", from, to),
+    realpath: (path) => fs("realpath", path),
+    symlink: (target, path) => done("symlink", target, path),
+    readlink: (path) => fs("readlink", path),
+    exists: (path) => fs("exists", path),
+    async watch(paths, options, onChange) {
+      if (channel.protocol[1] < 1) {
+        throw new KernelCallError(
+          `this kernel speaks protocol ${channel.protocol.join(".")}, which has no watch`,
+          "ENOSYS"
+        );
+      }
+      const { id, done: done2 } = channel.request({
+        op: "watch",
+        paths,
+        recursive: options.recursive === true
+      });
+      channel.watching.set(id, onChange);
+      try {
+        await done2;
+      } catch (error) {
+        channel.watching.delete(id);
+        throw error;
+      }
+      return {
+        close: () => {
+          if (!channel.watching.delete(id)) return;
+          void channel.call({ op: "unwatch", watch: id }).catch(() => void 0);
+        }
+      };
+    }
+  };
+}
+async function greet(channel, lock, timeoutMs) {
+  channel.post({ hello: { protocol: PROTOCOL, ...lock ? { lock } : {} } });
+  const timer = setTimeout(
+    () => channel.end(
+      new KernelGoneError(`no slicc-kernel answered on this port within ${timeoutMs} ms`)
+    ),
+    timeoutMs
+  );
+  let hello;
+  try {
+    hello = await channel.greeted.promise;
+  } finally {
+    clearTimeout(timer);
+  }
+  const refused = hello.error ?? versionError(hello.protocol);
+  if (refused) {
+    const error = new Error(refused);
+    channel.end(error);
+    throw error;
+  }
+  return hello;
+}
+async function attachKernel(port, options = {}) {
+  const locks = options.locks === null ? void 0 : options.locks ?? locksOf();
+  const own = await holdLock(locks);
+  const channel = new Channel(port, own.release);
+  const hello = await greet(channel, own.name, options.timeoutMs ?? 1e4);
+  channel.protocol = hello.protocol;
+  if (hello.lock && locks) void locks.request(hello.lock, () => channel.end(new KernelGoneError()));
+  const transport = new RemoteTransport(
+    { postMessage: (call) => channel.post(call) },
+    hello.traits ?? UNAVAILABLE
+  );
+  channel.remote = transport;
+  return {
+    protocol: hello.protocol,
+    transport,
+    closed: channel.ended.promise,
+    spawn: (argv, opts = {}) => spawnOn(channel, argv, opts),
+    run: (argv, opts = {}) => runOn(channel, argv, opts),
+    openTerminal: (argv, opts = {}) => terminalOn(channel, argv, opts),
+    ps: () => channel.call({ op: "ps" }),
+    kill: async (pid, signal = "SIGTERM") => void await channel.call({ op: "kill", pid, signal }),
+    fetch: (req) => {
+      const { signal, ...rest } = req;
+      return transport.fetch({ ...rest, signal: signal ?? new AbortController().signal });
+    },
+    fs: fsOn(channel),
+    async close(o = {}) {
+      if (channel.failure) return;
+      await channel.call({ op: "detach", ...o.kill ? { kill: true } : {} }).catch(() => void 0);
+      channel.end(new KernelGoneError("this client is closed"));
+    }
+  };
+}
+
 // src/local-proxy-transport.ts
 var PATH = "/api/fetch-proxy";
 var RAW_CONTENT_TYPE = "application/vnd.slicc.raw-fetch";
@@ -212,7 +630,7 @@ function isHead(value) {
     (pair) => Array.isArray(pair) && pair.length === 2 && typeof pair[0] === "string" && typeof pair[1] === "string"
   );
 }
-function concat(a, b) {
+function concat2(a, b) {
   const out = new Uint8Array(a.byteLength + b.byteLength);
   out.set(a);
   out.set(b, a.byteLength);
@@ -236,7 +654,7 @@ async function readHead(reader, idleMs) {
     }
     const next = await readWithin(reader, idleMs);
     if (next.done) throw failure("closed before the response head");
-    buffer = concat(buffer, next.value);
+    buffer = concat2(buffer, next.value);
   }
 }
 async function* chunks2(rest, reader, idleMs) {
@@ -446,6 +864,7 @@ async function createKernel(options = {}) {
       };
     },
     openTerminal,
+    connect: async () => await call({ op: "connect" }),
     terminate() {
       fail(new Error("slicc-kernel terminated"));
       worker.terminate();
@@ -453,6 +872,9 @@ async function createKernel(options = {}) {
   };
 }
 export {
+  KernelCallError,
+  KernelGoneError,
+  attachKernel,
   checkLocalProxy,
   createKernel,
   fetchTransport,