@ai-ecoverse/slicc-kernel 1.18.1 → 1.20.0

package/README.md

@@ -76,13 +76,22 @@ Stops the kernel worker and every process. Pending and later calls reject.
 
 Makes a port for another client of the same kernel. Hand it to a dedicated worker, or through a SharedWorker to another tab: it can be transferred any number of times. All clients share one process table, so a terminal on the page can `ps` and `kill` what a worker started.
 
+### `kernel.dial({ port, host? })` and `kernel.loopbackFetch(input, { port, host?, signal? })`
+
+Reach a server a program runs on the kernel's loopback (a dev server, `python -m http.server`) from the page, a service worker or an attached client, which have both calls too (protocol 1.4).
+
+- **`dial`** connects to `127.0.0.1:<port>` inside the kernel (`host` may also be `localhost` or `::1`) and resolves with `{ readable, writable, close() }`: a `ReadableStream` and a `WritableStream` of bytes. `writable.close()` half-closes the connection, as `shutdown(SHUT_WR)` does, and `close()` closes it. Each connection has a `MessagePort` of its own, and an attached client's connections close when it detaches; the other end then reads `ECONNRESET`.
+- **`loopbackFetch`** sends an HTTP/1.1 request over `dial` and resolves with a `Response` as soon as the headers arrive. The body streams (chunked, counted, or until the connection closes), so server-sent events and long polls work; cancelling the body or aborting `signal` closes the connection. The request is routed by `port`, not by the URL, whose host is sent as `Host`; a request body is sent with `Content-Length`, and every request uses a connection of its own (`Connection: close`).
+- **Errors** reject with an `Error` whose `code` is the errno: `ECONNREFUSED` when nothing listens on the port, `ENETUNREACH` for a host that is not the loopback, `EINVAL` for a port outside 1–65535.
+- **Names**: a page reaches the kernel's loopback as `<port>.kernel.localhost`, which is the loopback and a secure context, so a request no service worker routes lands on port 80 of the real machine instead of a real server on that port; the service worker routes the name to `loopbackFetch` with that port. Programs find the name in `SLICC_PAGE_LOOPBACK` (`kernel.localhost`) and `SLICC_PAGE_LOOPBACK_URL` (`http://{port}.kernel.localhost`), to print or inject the right URL.
+
 ### `attachKernel(port, options?) → Promise<KernelClient>`
 
 Attaches to a kernel over a port from `kernel.connect()`, in any realm: a worker, another tab, or Node. `options.timeoutMs` (10 s by default) bounds the handshake: a port with no kernel behind it rejects with `KernelGoneError`.
 
 | Client | |
 |---|---|
-| `spawn(argv, { cwd, env, stdin, onStdout, onStderr })` | runs over pipes and resolves with `{ pid, pgid, exited, signal(name) }` once the process has started; output arrives in chunks, as bytes, and the kernel keeps no copy; every spawn leads its own process group, which `signal` (default `SIGTERM`) signals; a program that cannot start rejects with `code: 'ENOENT'` |
+| `spawn(argv, { cwd, env, stdin, pgid, onStdout, onStderr })` | runs over pipes and resolves with `{ pid, pgid, exited, signal(name) }` once the process has started; output arrives in chunks, as bytes, and the kernel keeps no copy; every spawn leads its own process group, which `signal` (default `SIGTERM`) signals, unless `pgid` names an existing group to join (protocol 1.3): the process then joins that group and that group's session, as `setpgid` would (a client has no session of its own: each of its spawns and terminals leads one, and `setpgid` cannot cross sessions), and `signal` signals the whole group. The group must have been started by this client (a spawn or a terminal it opened), even if its leader has exited; an unknown group rejects with `code: 'ESRCH'`, another session's with `code: 'EPERM'`, and against a 1.2 kernel the client rejects with `code: 'ENOSYS'` before sending anything, as an older kernel would ignore the option; a program that cannot start rejects with `code: 'ENOENT'` |
 | `run(argv, options?)` | `spawn` and wait: `{ pid, status, stdout, stderr }` as text, with `onStdout`/`onStderr` streaming text |
 | `openTerminal(argv, options?)` | a pty session, as `kernel.openTerminal` |
 | `ps()` | the kernel's processes: `{ pid, ppid, pgid, sid, argv, tty, started, state }`, `state` being `'S'` or `'Z'` |
@@ -93,7 +102,7 @@ Attaches to a kernel over a port from `kernel.connect()`, in any realm: a worker
 | `close({ kill })` | detaches: the client's processes keep running unless `kill` is set, which ends their process groups with `SIGKILL`; its terminals are hung up |
 | `closed` | resolves with the error that ended the client |
 
-When the kernel goes away (the page closed or reloaded, `terminate()`), pending calls and `exited` reject with `KernelGoneError`, and so do later calls. Each side holds a Web Lock and waits on the other's, since a `MessagePort` reports no close in browsers; in Node, the port's `close` event does the same. The first message is a handshake on the protocol version, `1.1` (`1.0` had no `watch`, which rejects with `code: 'ENOSYS'` on such a kernel): a client or kernel of another major version is refused with an error naming both.
+When the kernel goes away (the page closed or reloaded, `terminate()`), pending calls and `exited` reject with `KernelGoneError`, and so do later calls. Each side holds a Web Lock and waits on the other's, since a `MessagePort` reports no close in browsers; in Node, the port's `close` event does the same. The first message is a handshake on the protocol version, `1.4` (`1.0` had no `watch`, `1.1` no mounts, `1.2` no `spawn` into a group and `1.3` no `dial`, which reject with `code: 'ENOSYS'` on such a kernel): a client or kernel of another major version is refused with an error naming both.
 
 ### Headless in Node, for tests
 
@@ -164,6 +173,7 @@ await kernel.run(['curl', '-sS', 'https://registry.npmjs.org/@ai-ecoverse/wasm-b
 - **TLS** needs [`@ai-ecoverse/wasm-tls-engine`](https://www.npmjs.com/package/@ai-ecoverse/wasm-tls-engine) installed under `modules`, like a command. Without it, `CONNECT` is answered `501` with the reason.
 - **The transport** is an object on the page with `traits` (`manualRedirects`, `encodedBodies`, `maxRequestBody`, and `crossOrigin`: `'cors'` or `'any'`) and `fetch(request)`, which answers `{ status, statusText, headers, body, cancel }` with `body` an async iterable of `Uint8Array`. The kernel pulls the body one chunk at a time, so a slow program slows the download. A rejection with a numeric `status` is answered with that status; any other with `502`. `fetchTransport()` is the plain `fetch` of the page, bound by CORS; its `hint` option is appended to the message of a request it could not make, so a program's 502 can name a way around CORS (`registry.npmjs.org` and jsDelivr allow it); an embedder with a way around CORS passes its own. A response body that delivers no data for `bodyIdleMs` (default 5 minutes; `0` or `Infinity` turns it off) fails with `ETIMEDOUT` instead of waiting forever, which matters because Chrome under heavy load can stop handing a page the rest of a body. A slow body that keeps delivering is never cut off.
 - **`localProxyTransport({ url, key })`** sends every request to a local proxy on loopback, such as [slicc-node](https://github.com/ai-ecoverse/slicc-node) or slicc-swift, which fetches it without CORS. Redirects reach the program unfollowed, with every `Set-Cookie`; bodies arrive decoded. `probeLocalProxy({ url, key })` resolves with the proxy's capabilities, or `null` when it is gone or refuses the key, so a page can fall back to `fetchTransport()`. `checkLocalProxy({ url, key })` says why: it resolves with `{ state }`, which is `ready` (with `probe`), `blocked` (the user denied Chrome's Local Network Access permission, `loopback-network`), `unreachable` (nothing answered; its `permission` is `granted`, `prompt` or `unknown`, and with `prompt` the page cannot tell a dismissed prompt from a proxy that is not running, since Chrome asks only once it has a connection), `refused` (with `status` and `error`, such as a stale key) or `incompatible`. The permission is not consulted when the page itself is on loopback. Both take a `fetch` option, and the transport `maxRequestBody` (default 64 MiB) and `bodyIdleMs` like `fetchTransport`.
+- **The host machine** is `host.slicc.internal` (`SLICC_HOST_LOOPBACK`), as `host.docker.internal` is in Docker, while `localhost` and `127.0.0.1` stay the kernel's own loopback. The proxy forwards `http(s)://host.slicc.internal:<port>/…` to the transport as `http(s)://127.0.0.1:<port>/…`, so it reaches the real machine's loopback through the local proxy or Node's `fetch` (through the page's own `fetch`, a request needs CORS from that server and Chrome's Local Network Access permission); other loopback addresses are still refused. This is HTTP and HTTPS through the proxy only: `/etc/hosts` and WASIX name resolution give the name `10.0.2.2`, QEMU's address for its host, so a program that connects a socket to it directly fails with `ENETUNREACH` instead of reaching the kernel's loopback.
 - **Without a transport**, every request is answered `502` with `slicc-kernel: no network transport (createKernel({ network: { transport } }))`.
 
 Process workers can also use the transport directly, without a socket: the `net-request`, `net-read` and `net-close` syscalls (`Module.sliccKernel.http` in a process) open a request and read its body in pieces. Requests are per process and closed when it exits, and a request body has no size limit of its own (the transport's `maxRequestBody` applies to the proxy). `net-traits` answers the transport's `traits` with `crossOrigin`: `'cors'` when it is bound by CORS like `fetchTransport()`, `'any'` (the default for a transport that does not say) when it is not; without a transport it fails with `ENETUNREACH`.
@@ -176,7 +186,7 @@ The page and the proxy speak raw mode of SLICC's `/api/fetch-proxy`. Every reque
 - `X-Slicc-Raw-Request: <json>`, the head to send upstream: `{ "url", "method", "headers": [[name, value], …] }`, with the headers in order and repeats kept apart. Characters past U+007E are `\u`-escaped so the value is a valid header.
 - The request body, if any, as the hop's body.
 
-The proxy answers `200` with `Content-Type: application/vnd.slicc.raw-fetch`: a big-endian `u32` length, that many bytes of UTF-8 JSON `{ "status", "statusText", "headers": [[name, value], …], "url" }`, then the upstream body until the end of the hop. Any other answer is a refusal with a JSON `{ "error" }` body and `X-Proxy-Error: 1`; programs see its status (`400` malformed head, `403` wrong origin or key, `413` body too large, `502` upstream unreachable). A probe is a `POST` with the key and `X-Slicc-Raw-Probe: 1` instead of a head; it is answered with JSON `{ "rawFetch": 1, "requestBodyStreaming", "maxRequestBodyBytes" }` and fetches nothing. The server side (origin allowlist, CORS, the Private Network Access preflight) is specified in [slicc-node's README](https://github.com/ai-ecoverse/slicc-node#protocol).
+The proxy answers `200` with `Content-Type: application/vnd.slicc.raw-fetch`: a big-endian `u32` length, that many bytes of UTF-8 JSON `{ "status", "statusText", "headers": [[name, value], …], "url" }`, then the upstream body until the end of the hop. Any other answer is a refusal with a JSON `{ "error" }` body and `X-Proxy-Error: 1`; programs see its status (`400` malformed head, `403` wrong origin or key, `413` body too large, `502` upstream unreachable). A probe is a `POST` with the key and `X-Slicc-Raw-Probe: 1` instead of a head; it is answered with JSON `{ "rawFetch": 1, "requestBodyStreaming", "maxRequestBodyBytes" }` and fetches nothing. A proxy that tunnels to the kernel for `<port>.kernel.localhost` adds `"kernelTunnel"` and `"kernelPort"`, which `checkLocalProxy` and `probeLocalProxy` pass on when they are numbers. The server side (origin allowlist, CORS, the Private Network Access preflight) is specified in [slicc-node's README](https://github.com/ai-ecoverse/slicc-node#protocol).
 
 ## Filesystem
 

package/dist/driver-worker.js

@@ -110,8 +110,247 @@ var RemoteTransport = class {
   }
 };
 
+// src/client/dial-stream.ts
+function errorWithCode(message) {
+  const code = /^E[A-Z]+/.exec(message)?.[0];
+  return Object.assign(new Error(message), code ? { code } : {});
+}
+function dialStream(port, onClose) {
+  let closed = false;
+  let reader;
+  const acks = [];
+  const readable = new ReadableStream({
+    start(controller) {
+      reader = controller;
+    },
+    pull() {
+      if (!closed) port.postMessage({ more: true });
+    },
+    cancel() {
+      reader = void 0;
+      close();
+    }
+  });
+  const finish = (fail2) => {
+    const controller = reader;
+    reader = void 0;
+    for (const ack of acks.splice(0)) {
+      if (fail2) ack.reject(fail2);
+      else ack.resolve();
+    }
+    if (!controller) return;
+    if (fail2) controller.error(fail2);
+    else controller.close();
+  };
+  function shut() {
+    if (closed) return;
+    closed = true;
+    port.close();
+    onClose?.();
+  }
+  function close() {
+    if (!closed) port.postMessage({ close: true });
+    shut();
+    finish();
+  }
+  port.addEventListener("message", ({ data }) => {
+    if (data instanceof Uint8Array) reader?.enqueue(data);
+    else if ("ack" in data) acks.shift()?.resolve();
+    else if ("end" in data) finish();
+    else finish(errorWithCode(data.error));
+  });
+  port.start?.();
+  const writable = new WritableStream({
+    write(chunk) {
+      if (closed) throw errorWithCode("EPIPE: the connection is closed");
+      const copy = chunk.slice();
+      const acked = new Promise((resolve, reject) => acks.push({ resolve, reject }));
+      port.postMessage(copy, [copy.buffer]);
+      return acked;
+    },
+    close() {
+      if (!closed) port.postMessage({ end: true });
+    },
+    abort() {
+      close();
+    }
+  });
+  const fail = (error) => {
+    shut();
+    finish(error);
+  };
+  return { readable, writable, close, fail };
+}
+
+// src/client/loopback-fetch.ts
+var OWN_HEADERS = /* @__PURE__ */ new Set(["host", "connection", "content-length", "transfer-encoding"]);
+var encoder = new TextEncoder();
+var latin1 = new TextDecoder("latin1");
+var ByteSource = class {
+  buffered = new Uint8Array(0);
+  reader;
+  constructor(reader) {
+    this.reader = reader;
+  }
+  async more() {
+    const { done, value } = await this.reader.read();
+    if (done) return false;
+    const joined = new Uint8Array(this.buffered.length + value.length);
+    joined.set(this.buffered);
+    joined.set(value, this.buffered.length);
+    this.buffered = joined;
+    return true;
+  }
+  async until(marker) {
+    const needle = encoder.encode(marker);
+    for (; ; ) {
+      const at = indexOf(this.buffered, needle);
+      if (at >= 0) return this.take(at + needle.length);
+      if (!await this.more()) return void 0;
+    }
+  }
+  async some(max) {
+    if (this.buffered.length === 0 && !await this.more()) return void 0;
+    return this.take(Math.min(max, this.buffered.length));
+  }
+  async exactly(count) {
+    while (this.buffered.length < count) if (!await this.more()) return void 0;
+    return this.take(count);
+  }
+  take(count) {
+    const out = this.buffered.slice(0, count);
+    this.buffered = this.buffered.slice(count);
+    return out;
+  }
+};
+function indexOf(haystack, needle) {
+  outer: for (let i = 0; i + needle.length <= haystack.length; i++) {
+    for (let j = 0; j < needle.length; j++) if (haystack[i + j] !== needle[j]) continue outer;
+    return i;
+  }
+  return -1;
+}
+function parseHead(bytes) {
+  const [statusLine = "", ...lines] = latin1.decode(bytes).split("\r\n");
+  const match = /^HTTP\/1\.[01] (\d{3})(?: (.*))?$/.exec(statusLine);
+  if (!match) throw errorWithCode(`EPROTO: not an HTTP/1.x response: ${statusLine.slice(0, 80)}`);
+  const headers = [];
+  for (const line of lines) {
+    const colon = line.indexOf(":");
+    if (colon > 0) headers.push([line.slice(0, colon).trim(), line.slice(colon + 1).trim()]);
+  }
+  return { status: Number(match[1]), statusText: match[2] ?? "", headers };
+}
+function header(head, name) {
+  return head.headers.find(([key]) => key.toLowerCase() === name)?.[1];
+}
+async function* chunked(source) {
+  for (; ; ) {
+    const line = await source.until("\r\n");
+    if (!line) throw errorWithCode("EPROTO: the chunked body ended early");
+    const size = Number.parseInt(latin1.decode(line).trim().split(";")[0] ?? "", 16);
+    if (!Number.isFinite(size)) throw errorWithCode("EPROTO: a malformed chunk size");
+    if (size === 0) return;
+    const data = await source.exactly(size);
+    if (!data || !await source.exactly(2)) throw errorWithCode("EPROTO: a chunk ended early");
+    yield data;
+  }
+}
+async function* counted(source, length) {
+  let left = length;
+  while (left > 0) {
+    const data = await source.some(left);
+    if (!data) throw errorWithCode("EPROTO: the body ended early");
+    left -= data.length;
+    yield data;
+  }
+}
+async function* untilClose(source) {
+  for (let data = await source.some(65536); data; data = await source.some(65536)) yield data;
+}
+function bodyOf(source, head, method) {
+  if (method === "HEAD" || head.status === 204 || head.status === 304) return void 0;
+  if (header(head, "transfer-encoding")?.toLowerCase().includes("chunked")) return chunked(source);
+  const length = header(head, "content-length");
+  if (length === void 0) return untilClose(source);
+  const size = Number(length);
+  if (!/^\d+$/.test(length.trim()) || !Number.isSafeInteger(size)) {
+    throw errorWithCode(`EPROTO: a malformed Content-Length: ${length}`);
+  }
+  return counted(source, size);
+}
+function stream(chunks, socket, signal) {
+  return new ReadableStream({
+    start(controller) {
+      signal?.addEventListener("abort", () => controller.error(signal.reason), { once: true });
+    },
+    async pull(controller) {
+      try {
+        const next = await chunks.next();
+        if (next.done) {
+          controller.close();
+          socket.close();
+        } else controller.enqueue(next.value);
+      } catch (err) {
+        controller.error(err);
+        socket.close();
+      }
+    },
+    cancel() {
+      socket.close();
+    }
+  });
+}
+async function requestBytes(request) {
+  const url = new URL(request.url);
+  const body = request.body ? new Uint8Array(await request.arrayBuffer()) : void 0;
+  const lines = [`${request.method} ${url.pathname}${url.search} HTTP/1.1`, `Host: ${url.host}`];
+  request.headers.forEach((value, name) => {
+    if (!OWN_HEADERS.has(name)) lines.push(`${name}: ${value}`);
+  });
+  if (body) lines.push(`Content-Length: ${body.length}`);
+  lines.push("Connection: close", "", "");
+  const head = encoder.encode(lines.join("\r\n"));
+  const out = new Uint8Array(head.length + (body?.length ?? 0));
+  out.set(head);
+  if (body) out.set(body, head.length);
+  return out;
+}
+async function loopbackFetch(dial, input, options) {
+  options.signal?.throwIfAborted();
+  const request = input instanceof Request ? input : new Request(input);
+  const bytes = await requestBytes(request);
+  const socket = await dial({
+    port: options.port,
+    ...options.host ? { host: options.host } : {}
+  });
+  options.signal?.addEventListener("abort", () => socket.close(), { once: true });
+  try {
+    const writer = socket.writable.getWriter();
+    await writer.write(bytes);
+    writer.releaseLock();
+    const source = new ByteSource(socket.readable.getReader());
+    let head;
+    do {
+      const raw = await source.until("\r\n\r\n");
+      if (!raw) throw errorWithCode("ECONNRESET: the connection closed before a response");
+      head = parseHead(raw);
+    } while (head.status >= 100 && head.status < 200);
+    const chunks = bodyOf(source, head, request.method);
+    if (!chunks) socket.close();
+    return new Response(chunks ? stream(chunks, socket, options.signal) : null, {
+      status: head.status,
+      statusText: head.statusText,
+      headers: head.headers
+    });
+  } catch (err) {
+    socket.close();
+    throw err;
+  }
+}
+
 // src/client/protocol.ts
-var PROTOCOL = [1, 2];
+var PROTOCOL = [1, 4];
 function locksOf(scope = globalThis) {
   const locks = scope.navigator?.locks;
   return typeof locks?.request === "function" ? locks : void 0;
@@ -138,8 +377,8 @@ var KernelCallError = class extends Error {
     this.code = code;
   }
 };
-var encoder = new TextEncoder();
-var bytesOf = (data) => typeof data === "string" ? encoder.encode(data) : data;
+var encoder2 = new TextEncoder();
+var bytesOf = (data) => typeof data === "string" ? encoder2.encode(data) : data;
 var UNAVAILABLE = {
   manualRedirects: false,
   encodedBodies: false,
@@ -247,8 +486,15 @@ var Channel = class {
     else call.resolve(reply.result);
   }
 };
+async function dialOn(channel, { port, host }) {
+  const req = { op: "dial", port, ...host ? { host } : {} };
+  return dialStream(await since(channel, 4, "dial", () => channel.call(req)));
+}
 async function spawnOn(channel, argv, opts) {
   const { onStdout, onStderr, stdin, group: _, ...rest } = opts;
+  if (opts.pgid !== void 0)
+    await since(channel, 3, "spawn into a process group", async () => {
+    });
   const started = Promise.withResolvers();
   const options = { ...rest, ...stdin !== void 0 ? { stdin: bytesOf(stdin) } : {} };
   const { done } = channel.request(
@@ -260,11 +506,12 @@ async function spawnOn(channel, argv, opts) {
   );
   done.catch(started.reject);
   const pid = await started.promise;
+  const pgid = opts.pgid ?? pid;
   return {
     pid,
-    pgid: pid,
+    pgid,
     exited: done,
-    signal: async (name = "SIGTERM") => void await channel.call({ op: "kill", pid: -pid, signal: name })
+    signal: async (name = "SIGTERM") => void await channel.call({ op: "kill", pid: -pgid, signal: name })
   };
 }
 async function runOn(channel, argv, opts) {
@@ -421,6 +668,8 @@ async function attachKernel(port, options = {}) {
       const { signal, ...rest } = req;
       return transport.fetch({ ...rest, signal: signal ?? new AbortController().signal });
     },
+    dial: (options2) => dialOn(channel, options2),
+    loopbackFetch: (input, options2) => loopbackFetch((o) => dialOn(channel, o), input, options2),
     fs: fsOn(channel),
     async close(o = {}) {
       if (channel.failure) return;