@ai-ecoverse/slicc-kernel 1.170.3 → 1.171.0

package/README.md

@@ -179,7 +179,7 @@ Attaches to a kernel over a port from `kernel.connect()`, in any realm: a worker
 | `ps()` | the kernel's processes: `{ pid, ppid, pgid, sid, argv, tty, started, state, rss, utime, stime, uid, gid }`, `state` being `'S'`, `'T'` (stopped) or `'Z'`, `rss` the process's resident memory in bytes (the size of its wasm memory, what `/proc/<pid>/status` shows as `VmRSS`; absent from kernels before 1.61), `utime`/`stime` the CPU time it has used, in milliseconds, in its program and in the kernel's calls (see [`/proc`](#proc); absent for a JS program and from kernels before 1.63) and `uid`/`gid` the effective ids (absent from a 1.7 kernel), and `tag` the process's tag (see `kernel.run`), when it has one, with `tags`, all of them in order, when a spawn added one (see `kernel.connect`), on rows the client's user may see as root or as the same uid. A client bound to a user sees only that user's processes |
 | `kill(pid, signal?)` | a process, or a process group with a negative pid; no such process rejects with `code: 'ESRCH'`, and one the client's user may not signal with `code: 'EPERM'` |
 | `fs` | `readFile`, `readText`, `writeFile(path, string \| bytes)`, `stat`, `lstat`, `readdir`, `mkdir` (with parents), `rm(path, { force })` (recursive), `rename`, `realpath`, `symlink(target, path)`, `readlink`, `exists`, on the processes' file system, mounts included (a path under a `tmpfs`, `fsa` or `hostfs` mount reads and writes the mounted file system, as `cat` in a process would); failures reject with `KernelCallError` and a POSIX `code` |
-| `fetch({ url, method, headers, body, signal })` | through the kernel's network transport (the one the page passed to `createKernel`), with a streaming body; `transport` is the same as a `NetworkTransport` |
+| `fetch({ url, method, headers, body, redirect, signal })` | through the kernel's network transport (the one the page passed to `createKernel`), with a streaming body; `redirect` is the per-request policy (see [Network](#network)); `transport` is the same as a `NetworkTransport` |
 | `fs.watch(paths, { recursive }, onChange)` | resolves with `{ close() }`; `onChange` gets `{ paths }`, the changed paths at, below (or, without `recursive`, directly in) the watched ones, batched per task, or `{ overflow: true }` when there were too many to list, to rescan; it covers every change made through this kernel, by its processes and its clients, and a removal or rename that failed or found nothing to do is none |
 | `serveCdp(hook, { runtime })` | offers this client's browser to the kernel's programs (protocol 1.5): `hook` is the `cdp` hook of `createKernel`, asked once per program connection; resolves with `{ close() }`, and closing or detaching takes the offer back and closes its connections (see [Browser automation](#browser-automation-cdp)) |
 | `serveFacade(name, hook) → { close() }` | serves a facade that `createKernel({ facades })` declared, the same way as its `hook` (protocol 1.11); the latest registration wins, over the embedder's hook too, and `close()` hands it back. An undeclared name is `EINVAL` |
@@ -479,7 +479,7 @@ The module's default export (or its `main` export) is called with a context and
 - `crontab({ action, user?, text? })`, what the [`crontab`](#cron) command calls: `action` is `'list'`, `'remove'` or `'install'` (with the table as `text`), on the caller's own table or, for an effective uid of 0, on `user`'s. It resolves with `{ text }` (the table for `list`, `''` otherwise) or `{ error }`, the message to print. A malformed call is `EINVAL`;
 - `net`: `listen({ port, host, backlog })` and `connect({ host, port, signal })` (`host` defaults to `127.0.0.1`) on the kernel's loopback, the same sockets WASIX programs use, so a JS server is reached by other processes, by the page as `<port>.kernel.localhost` (`loopbackFetch`, `dial`) and by attached clients. A listener has its `port` (as bound, so port `0` picks a free one), `accept({ signal })`, `close()` and `for await` over its connections. A connection has `local` and `remote` addresses, `readable` and `writable` web streams (closing the writable shuts down the sending side, so the peer reads the end), `close()`, and `reset()`, which closes it with an RST as Node's `resetAndDestroy()` does ([#1530](https://github.com/ai-ecoverse/slicc-kernel/issues/1530)): on the kernel's loopback, the peer first reads the bytes it has already received, as on Linux and BSD ([#1577](https://github.com/ai-ecoverse/slicc-kernel/issues/1577)), and then its reads fail with `ECONNRESET` instead of seeing the end; its writes fail with `ECONNRESET` at once. A TLS connection (`secure()`, a `tls` connect on the loopback) resets the same way. On an uplink connection `reset()` closes at once with a FIN, as the transport can't send an RST: bytes still buffered for the peer are discarded instead of sent. The loopback is IPv4 (`127.0.0.1` by default, or `0.0.0.0`), so an IPv6 host such as `::1` fails with `EADDRNOTAVAIL` for both `listen` and `connect`, and binding and connecting fail as they would for a WASIX program (`EADDRINUSE`, `ECONNREFUSED`, `EADDRNOTAVAIL`). `sync.net.listen` takes the same options and gives the same listener, bound and listening when it returns, so its `port` is known at once, as Node's `listen()` needs ([#1398](https://github.com/ai-ecoverse/slicc-kernel/issues/1398)); it needs the sync lane. A blocked accept, read or write waits without holding anything in the kernel, so aborting it or closing the socket leaves nothing behind. A write to a connection the peer closed fails with `EPIPE` and no `SIGPIPE`;
 - `net.connect({ host, port, tls: { servername, alpn, rejectUnauthorized, ca, path, cert, key, passphrase, sni, checkHostname, keylog } })` takes one of two paths. By default it's **A1, TLS through the transport** ([#914](https://github.com/ai-ecoverse/slicc-kernel/issues/914)), described first below. For a destination the uplink routes it can also be **A2, a TLS handshake in the kernel** ([#922](https://github.com/ai-ecoverse/slicc-kernel/issues/922)), described after it. `conn.tls.path` (`'transport'` or `'kernel'`) and `conn.tls.substitutes` say which one was taken.
-  - **What it does:** there's no handshake in the kernel. The connection carries plain HTTP/1.1 (and WebSocket upgrades) to the realm proxy's HTTP path for the origin `https://<servername or host>[:port]`, exactly as a `CONNECT` through `127.0.0.1:3128` does after the kernel terminates its TLS. The transport (the page's `fetch` or WebSocket, or the uplink's) makes the real TLS connection and verifies it. So `https.request` over a socket, keep-alive agents, `agent-base` and `wss://` clients work with the transport's semantics. That includes sealed placeholders in headers, which reach the transport exactly as from `ctx.fetch`.
+  - **What it does:** there's no handshake in the kernel. The connection carries plain HTTP/1.1 (and WebSocket upgrades) to the realm proxy's HTTP path for the origin `https://<servername or host>[:port]`, exactly as a `CONNECT` through `127.0.0.1:3128` does after the kernel terminates its TLS. The transport (the page's `fetch` or WebSocket, or the uplink's) makes the real TLS connection and verifies it. So `https.request` over a socket, keep-alive agents, `agent-base` and `wss://` clients work with the transport's semantics. A request with a sealed placeholder is refused there as from `ctx.fetch` (see **The transport** under [`/dev/sealed`](#devsealed)).
   - **The `tls` property:** the connection's `tls` is `{ path: 'transport', substitutes: true, protocol: 'transport', alpnProtocol: 'http/1.1', authorized: true, servername }`. `authorized` means the transport verified the server. There's no peer certificate chain and no TLS version of the kernel's own, so don't read it as a real handshake's result.
   - **Options:** the transport always verifies, so `rejectUnauthorized: false` is accepted and changes nothing. Outside the uplink, a custom `ca` is refused (`ENOTSUP`), and so is an `alpn` list without `http/1.1` (`EPROTONOSUPPORT`). The kernel's own loopback never goes to the transport. A `host` of `localhost` or `127.x` makes the handshake in the kernel instead (Loopback, below), so with nothing listening it's `ECONNREFUSED`. The rest of it (`*.localhost`, `0.0.0.0`, `::1`, and any loopback `servername` for another host) is `ENOTSUP` at connect, as `ctx.fetch` refuses `https:` and `wss:` there. `host.slicc.internal` goes to the transport under its policy, as for `ctx.fetch`.
   - **Non-HTTP protocols aren't carried:**
@@ -487,7 +487,7 @@ The module's default export (or its `main` export) is called with a context and
     - non-HTTP first bytes on any other port get a `400` and a close;
     - a server-speaks-first protocol on another port (the client sends nothing) waits for the proxy's 120 s idle limit.
     - For a destination the uplink routes, these cases take A2 instead (below).
-  - **Which path** (A2 needs a destination the uplink routes, `net.destination(host) === 'uplink'`; an uplink-routed host on 443 with default options stays on A1, so its sealed placeholders keep working):
+  - **Which path** (A2 needs a destination the uplink routes, `net.destination(host) === 'uplink'`; an uplink-routed host on 443 with default options stays on A1, where a request with a sealed placeholder is refused):
 
     | The call | Uplink destination | Any other destination |
     |---|---|---|
@@ -637,6 +637,7 @@ WASIX programs get `AF_INET`/`AF_INET6` `SOCK_DGRAM` sockets, and `AF_UNIX` ones
 - **Streamed request bodies** ([#238](https://github.com/ai-ecoverse/slicc-kernel/issues/238)): a request comes with its body whole, as `body` (a `Uint8Array`), unless the transport sets `traits.streamingBodies`. Then a body the kernel doesn't hold whole comes as `bodyStream`, an async iterable of `Uint8Array`, never together with `body`. That's an upload through the proxy (`Content-Length` or chunked, curl's `-T`, git's push) and a JS program's `fetch` with a stream body. The kernel reads from the program only as fast as the transport pulls, applies no `maxRequestBody` and reserves no memory for it. A transport with a limit of its own rejects with `status: 413`. When the transport answers before it has read the whole body, the proxy closes the program's connection after the response, since the rest of the body is still on it. A transport without the trait keeps getting whole bodies, capped at its `maxRequestBody`, so existing transports need no change. A transport that wraps another (a router, an injector, a logger) and passes on the inner `traits` must forward `bodyStream`, or not claim `streamingBodies`: a wrapper that rebuilds the request from `url`, `method`, `headers`, `body` and `signal` would drop a streamed upload and still answer. `nodeTransport()` streams (undici's `fetch` with `duplex: 'half'`). `fetchTransport({ streamingBodies: true })` turns it on for a page that knows its targets speak HTTP/2 or later, since Chrome streams a `fetch` upload only there, so it's off by default. The trait is also how an embedder's own transport, such as the `slicc` CLI link's exit, takes streamed uploads; across the page and the kernel's worker, and to attached clients, the body is pulled one chunk at a time. `fetchTransport()` is the plain `fetch` of the page, bound by CORS; its `hint` option is appended to the message of a request it could not make, so a program's 502 can name a way around CORS (`registry.npmjs.org` and jsDelivr allow it); an embedder with a way around CORS passes its own. A response body that delivers no data for `bodyIdleMs` (default 5 minutes; `0` or `Infinity` turns it off) fails with `ETIMEDOUT` instead of waiting forever, which matters because Chrome under heavy load can stop handing a page the rest of a body. A slow body that keeps delivering is never cut off.
 - **Traits of the embedder's own:** `traits` may carry more fields than the kernel reads. The kernel keeps them, and attached clients get them as they are (`client.transport.traits`, `onTraits`). In TypeScript, `NetworkTraits` types its own fields and leaves any other as `unknown`. One is typed: `exit`, the default exit an embedder routes through, `{ kind, caps }` (`NetworkExit`). `kind` is `'local'`, `'remote'` or `'tailnet'` (or another string), `caps` lists what the exit carries (`['net', 'http', 'ws']`), and further fields are `unknown`. When traits change later, call `kernel.updateTransportTraits()`.
 - **Redirect hops** ([#302](https://github.com/ai-ecoverse/slicc-kernel/issues/302)): a transport that follows redirects but can see them (a browser extension's `webRequest.onBeforeRedirect` can) sets `traits.redirectHops` and answers `redirects: [{ status, url, headers }]`, one entry per 3xx it followed, in order (`NetworkRedirectHop`). The kernel then treats it as one that does not follow. Programs get each 3xx as it was answered: through the proxy, curl, requests and `hf` see the real redirect, `X-Repo-Commit` and `ETag` included, and `ctx.fetch` with `redirect: 'manual'` answers it. The rest of the chain is held for the process that asked, at most 30 s and 8 chains per process (64 in all), with the final body unread. A follow-up from that process, with the method the redirect leads to and the same `Range`, `If-None-Match`, `If-Modified-Since`, `If-Range` and `Accept-Encoding` (and an `Authorization` if the first had one), is answered from it without a second request; any other request goes to the transport. A chain that expires or is pushed out is cancelled, and so is one whose follow-up arrives and does not match. After a redirect to another origin the follow-up is expected without `Authorization`, which HTTP clients drop when they follow one; a `Location` matches without its `#fragment`. The page's own `fetch` cannot show a redirect, so `fetchTransport()` in a page stays a transport that follows, and `redirect: 'manual'` fails there as before. `fetchTransport({ redirect: 'manual' })` hands redirects back unfollowed where the platform's `fetch` shows them (Node; a browser answers an opaque status 0), and `nodeTransport()` does so. A kernel socket also records who connected it (`SO_PEERCRED`): the connecting process's pid, uid and gid on the accepting side, and the listener's on the connecting side, kept when the socket passes to another process. The proxy uses this to keep each process's held chains its own.
+- **Per-request redirect policy:** `client.fetch` and `client.transport.fetch` take `redirect: 'follow' | 'manual' | 'error'` (`NetworkRedirectPolicy`), and it reaches the embedder's transport as `request.redirect`. Without it a transport does what it always did: `request.redirect ?? options.redirect ?? 'follow'`, so `nodeTransport()` and `fetchTransport({ redirect: 'manual' })` stay manual. The field is the caller's: requests from programs (`ctx.fetch`, the proxy) carry none. A `manual` or `error` request bypasses held redirect chains: it neither fills nor reads them, and its answer carries no `redirects`. An explicit `follow` is treated like no policy there, so a `redirectHops` transport still answers it one hop at a time. `error` turns a redirect into a transport error, as `fetch` does. `nodeTransport()` follows an explicit `follow` itself, hop by hop, as `fetch` would, and each hop passes the same loopback refusal as the first request, on a pooled connection too. A transport sets `traits.redirectPolicy` to say it honors the field, as `fetchTransport()` does. `traits.manualRedirects` keeps its meaning. A `manual` or `error` request to a transport that doesn't advertise it is refused before it is sent (`status` 502, `detail: 'transport'`), and so is one whose answer shows the transport followed anyway. The exception is `manual` on a transport with `manualRedirects` and no `redirectHops`, which never follows. An explicit `follow` to a transport that can't is ignored. Where a platform hides a manual redirect (a browser's opaque response), the response has `opaqueRedirect: true`, status 0 and no headers. It crosses to the client unchanged, and is never turned into a success or a 3xx. A response's `url` and `redirected` also cross as the transport gave them.
 - **WebSockets:** a `GET` with `Upgrade: websocket`, in absolute form or inside a tunnel, is bridged to the transport's optional `websocket({ url, protocols, headers, signal })`. It answers `{ protocol, buffered, send(data), messages, close(code, reason), closed }`, with `messages` an async iterable of strings and `Uint8Array`s and `closed` resolving with `{ code, reason }`; a transport that has it says so with `traits.websocket`. The program gets `101 Switching Protocols` with the far end's protocol, then its frames go out as messages and the far end's come back as frames. Pings are answered by the proxy, and close codes pass both ways (a code that may not go on the wire becomes 1000 or 1011). A message over 64 MiB, or more than 16 MiB of payload from the far end waiting for the program, closes with 1009. The proxy stops reading the program while 1 MiB of its messages wait to be sent. A refused connection is `502` with the reason. A transport without WebSockets answers `501` with a message naming the option below, and the kernel logs a warning once, so a program never hangs. `fetchTransport({ webSocket: true })` opens them with the page's `WebSocket` (or pass a constructor); they are off by default because a browser WebSocket carries the user's cookies for the target site, which `fetch` here never sends (`credentials: 'omit'`), so a program could reach a site as the user. A page also cannot send headers on a WebSocket, and an `https:` page cannot open `ws:` to anything but loopback, which is a `502` that says so. `nodeTransport()` has WebSockets on (Node keeps no cookies) and sends the request's headers. An open fails as soon as Node's `WebSocket` reports an error, also on Node 22, which never follows the error with a close. A transport's `websocket()` may reject with an `Error` whose `code` is an errno, and the kernel hands that on to the program. Node's `WebSocket` doesn't say why an open failed, so when one fails, `nodeTransport()` finds out: it looks up the host's name (`EAI_NONAME` when that fails), then makes one TCP connection to the target, or a TLS one for `wss:`, for at most 3 s, and closes it at once. A refusal there is `ECONNREFUSED`, an unreachable host `EHOSTUNREACH` or `ENETUNREACH`, no answer `ETIMEDOUT`, and a TLS failure (a bad certificate) or a server that takes the connection but refused the upgrade `EPROTO`. A server that refused the upgrade therefore sees a second connection. The diagnosis never runs when an open succeeds, and never in a page (`fetchTransport()`), whose failed opens stay `ECONNREFUSED`. The credential-free path for a page is a transport that opens them outside the page, such as one over a linked slicc CLI (`npx sliccy@next`). After the program sends its close frame, data the far end sends before its own close still reaches the program, and the far end's close frame follows once it has drained, as RFC 6455 §5.5.1 has it; a far end that doesn't close within 5 s gets the program's own close echoed back ([#1471](https://github.com/ai-ecoverse/slicc-kernel/issues/1471)). The same holds when the kernel runs in a worker and the page's transport carries the socket ([#1506](https://github.com/ai-ecoverse/slicc-kernel/issues/1506)).
 - **The host machine** is `host.slicc.internal` (`SLICC_HOST_LOOPBACK`), as `host.docker.internal` is in Docker, while `localhost` and `127.0.0.1` stay the kernel's own loopback. The proxy forwards `http(s)://host.slicc.internal:<port>/…` to the transport as `http(s)://127.0.0.1:<port>/…`, so it reaches the real machine's loopback through Node's `fetch` or a transport that runs on that machine (through the page's own `fetch`, a request needs CORS from that server and Chrome's Local Network Access permission); other loopback addresses are still refused. This is HTTP and HTTPS through the proxy only: `/etc/hosts` and WASIX name resolution give the name `10.0.2.2`, QEMU's address for its host, so a program that connects a socket to it directly fails with `ENETUNREACH` instead of reaching the kernel's loopback.
 - **Without a transport**, every request is answered `502` with `slicc-kernel: no network transport (createKernel({ network: { transport } }))`.
@@ -1133,6 +1134,12 @@ The kernel then asks the hook `{ v: 1, op: 'materializeConsume', ticket, stamp,
 
 The kernel never asks for a session that isn't `verified`, whose handshake is more than 60 s old (or in the future), or whose fields aren't of that shape. The reply is `{ values: { [placeholder]: value } }` within 10 s; the kernel takes only the placeholders it asked for, with a non-empty value free of CR, LF and NUL, and anything else (an error such as `{ v: 1, error: { code } }`, a timeout, a malformed reply, no hook) means nothing is substituted: the placeholders go out as written.
 
+**The transport.** Nothing fills a placeholder in a request that leaves through the transport, so the kernel refuses that request itself and never hands it to the transport. This covers the realm proxy (`curl`, `CONNECT`, TLS on A1), `ctx.fetch`, `ctx.websocket` and a client's `fetch` and `transport.websocket`. The check needs no vault and no `sealed` hook: anything of the form `fnox_proxy_<24 hex>` is refused, whether or not the vault knows it.
+
+- **Where it looks:** the URL (host, user info, path, query and fragment, percent-encoded or not), header names and values, WebSocket subprotocols, and `Basic` credentials, decoded for the check (the user, the password, or a credential that is malformed or not UTF-8). A `Basic` credential without a placeholder goes out byte for byte. Bodies aren't read: a placeholder in a body goes out as written.
+- **What a refusal looks like:** the realm proxy answers `403` with the reason in the body, for a WebSocket upgrade too. `ctx.fetch` and `ctx.websocket` reject with a `TypeError` whose `cause.code` is `EACCES` and, for `ctx.fetch`, whose `cause.detail` is the reason. A client's `fetch` and `transport.websocket` reject with `code` `EACCES` and `status` `403`. Nothing is sent, retried or sent another way, and a held redirect isn't replayed for it.
+- **What stays as it was:** the kernel's own loopback, A2 sessions (above) and raw sockets never reach this check.
+
 ### `/dev/clipboard`
 
 A character device (`0666`), there in every `users` mode, through which programs read and write the embedder's clipboard (`createKernel({ clipboard: { read, write } })`; on the page, bios answers with `navigator.clipboard`). Emscripten programs (`cat /dev/clipboard`, `cat > /dev/clipboard`, see it in `/dev`, reach it by any path that leads to it, and see it as a character device after fork and exec), WASI programs (`std::fs`) and JS programs (`ctx.fs.readFile`, `ctx.fs.writeFile`, `ctx.open`) all open it.

package/dist/driver-worker.js

@@ -302,6 +302,18 @@ var TransportError = class extends Error {
     this.detail = detail === "transport" ? detail : void 0;
   }
 };
+function unfollowed(redirect) {
+  return redirect !== void 0 && redirect !== "follow";
+}
+function honors(traits, redirect) {
+  if (traits.redirectPolicy === true) return true;
+  const hops = traits.redirectHops;
+  const raw = traits.manualRedirects === true && (hops === void 0 || hops === false);
+  return redirect === "manual" && raw;
+}
+function refused(message) {
+  return new TransportError(message, 502, void 0, "transport");
+}
 var RemoteTransport = class {
   traits;
   port;
@@ -387,8 +399,11 @@ var RemoteTransport = class {
   }
   async fetch(request) {
     const nid = ++this.nextId;
-    const { signal } = request;
+    const { signal, redirect } = request;
     signal.throwIfAborted();
+    if (unfollowed(redirect) && !honors(this.traits, redirect)) {
+      throw refused(`this transport cannot honor redirect: ${redirect}`);
+    }
     let finished = false;
     const abort = () => {
       finished = true;
@@ -411,13 +426,14 @@ var RemoteTransport = class {
         url: request.url,
         method: request.method,
         headers: request.headers,
-        ...request.bodyStream ? { streamed: true } : request.body ? { body: request.body } : {}
+        ...request.bodyStream ? { streamed: true } : request.body ? { body: request.body } : {},
+        ...redirect !== void 0 ? { redirect } : {}
       });
     } catch (e) {
       finish();
       throw e;
     }
-    const { status, statusText, headers, url, redirected, redirects } = head;
+    const { status, statusText, headers, url, redirected, redirects, opaqueRedirect } = head;
     const read = async () => {
       if (finished) return void 0;
       const reply = await this.ask({ net: "read", nid }).catch((e) => {
@@ -433,9 +449,15 @@ var RemoteTransport = class {
       finish();
       this.cancel(nid);
     };
+    if (unfollowed(redirect) && (redirected || redirects?.length)) {
+      await cancel();
+      throw refused(`this transport followed a redirect despite redirect: ${redirect}`);
+    }
     return {
-      ...redirected ? { url, redirected } : {},
+      ...url !== void 0 ? { url } : {},
+      ...redirected !== void 0 ? { redirected } : {},
       ...redirects ? { redirects } : {},
+      ...opaqueRedirect ? { opaqueRedirect } : {},
       status,
       statusText,
       headers,
@@ -1194,9 +1216,9 @@ async function greet(channel, lock, timeoutMs, requests) {
   } finally {
     clearTimeout(timer);
   }
-  const refused = hello.error ?? versionError(hello.protocol);
-  if (refused) {
-    const error = new Error(refused);
+  const refused2 = hello.error ?? versionError(hello.protocol);
+  if (refused2) {
+    const error = new Error(refused2);
     channel.end(error);
     throw error;
   }
@@ -1343,9 +1365,9 @@ function serveFilesystem(port, handlers, capabilities = {}, options = {}) {
   let open = false;
   const reply = (message, transfer = []) => port.postMessage(message, transfer);
   async function hello(message) {
-    const refused = driverVersionError(message.protocol);
-    if (refused) {
-      reply({ hello: { protocol: DRIVER_PROTOCOL, error: refused, errno: "EPROTO" } });
+    const refused2 = driverVersionError(message.protocol);
+    if (refused2) {
+      reply({ hello: { protocol: DRIVER_PROTOCOL, error: refused2, errno: "EPROTO" } });
       return;
     }
     try {