@ai-ecoverse/slicc-kernel 1.171.0 → 1.172.0

package/README.md

@@ -41,7 +41,7 @@ TypeScript declarations ship in `dist/types/`, and each entry (`.`, `./node`, `.
 | `hostname` | `'slicc'` | the kernel's node name (see [Node name](#node-name)) |
 | `users` | `'enforce'` | `'off'`, `'report'` or `'enforce'`: whether files have owners and their modes are checked (see [File owners](#file-owners) and [Permissions](#permissions)), and whether `sudo` elevates (see [Sudo](#sudo)). `'off'` behaves exactly as kernels before owners did; any other value rejects. `createNodeKernel` takes it too, with the same default. Kernels up to 1.73 defaulted to `'off'` |
 | `onDenial` | none | `({ pid, uid, op, path, errno, tag? }) => void`: called for every permission check that fails, under `'report'` and `'enforce'` (see [Permissions](#permissions)). `createNodeKernel` takes it too |
-| `sealed` | none | `(req, { signal }) => Promise<object>`: answers requests written to [`/dev/sealed`](#devsealed). `createNodeKernel` takes it too |
+| `sealed` | none | `(req, { signal }) => Promise<object>`, or `{ request, fetch? }` with that function as `request`: `request` answers requests written to [`/dev/sealed`](#devsealed), and `fetch` fills requests that carry a sealed placeholder (see **The fetch hook** under [`/dev/sealed`](#devsealed)). `createNodeKernel` takes it too. An exported `SEALED_FETCH_VERSION`, the number `1`, says that this kernel takes the object |
 | `clipboard` | none | `{ read(stamp) => Promise<string>, write(text, stamp) => Promise<void> }`: the clipboard behind [`/dev/clipboard`](#devclipboard). `createNodeKernel` takes it too |
 | `requests` | none | `{ kinds, handle(request) }`: the page's answers to [`/dev/embedder`](#devembedder) for the processes it starts. `createNodeKernel` takes it too |
 | `approve` | none | `(req, { signal }) => 'once' \| 'always' \| 'deny'` (or a promise of one): decides a `sudo` request that needs approval (see [Sudo](#sudo)); without it, such a request is denied. `createNodeKernel` takes it too |
@@ -153,13 +153,14 @@ Reach a server a program runs on the kernel's loopback (a dev server, `python -m
 
 ### `kernel.attest(path)` and `kernel.watch(paths, onChange)`
 
-Page-only: neither is on an attached client (`attachKernel`, `connect()`), so no process or client can ask for them. They are for a page that has to trust a root-owned file, such as a policy file that only `sudo` may change.
+Neither is on an attached client (`attachKernel`, `connect()`), so no process or client can ask for them. `watch` is page-only; `attest` is on `createNodeKernel`'s kernel too. They are for a page that has to trust a root-owned file, such as a policy file that only `sudo` may change.
 
 - **`attest(path)`** resolves with exactly `{ path, bytes, uid, gid, mode, ino, mtimeMs, sha256, users, onRootFs, parents }`.
   - `path` is the canonical absolute path. `bytes` is a `Uint8Array` of its own, holding the file's contents. `sha256` is the lowercase hex digest of exactly those bytes. `mode` includes the type bits. `users` is the kernel's `users` mode. `onRootFs` is always `true`. `parents` lists `{ path, uid, gid, mode }` for every directory from `/` down to the file's own.
   - The kernel answers it itself, over the root file system without permission checks, and lstats every component. It reads the file between two lstats and accepts the read only when the inode, the modification time and the size are unchanged and the bytes match the size. Otherwise it reads once more, then fails with `EAGAIN`.
   - It refuses with an `Error` whose `code` is: `EACCES` for a symlink at the path or in any parent, for a path at or under a mount (any mount, one at `/` included; a mount that failed and isn't installed doesn't count), and for a parent that isn't owned by uid 0 or that group or others may write; `EISDIR` for a directory; `EINVAL` for another kind of file or a path that isn't absolute; `ENOTDIR` for a parent that isn't a directory; `ENOENT` for a missing one; and `EFBIG` for a file over 1 MiB.
   - Under `users: 'off'` there are no owners, so the parent check is skipped and the answer shows what the file system shows (uid and gid 1000, as processes see them). A consumer should accept only `users: 'enforce'`.
+  - **`attest(path, { absent: 'allow' })`** answers the same for a file that exists. When the path itself is missing, and every directory from `/` to its parent exists and passes the checks above (no symlink, owned by uid 0 and not writable by group or others unless `users` is `'off'`, not at or under a mount), it resolves with exactly `{ path, absent: true, users, onRootFs: true, parents }`: a trusted statement that the file isn't there, with no ids, hash or bytes. A missing parent, a symlink, a directory, another kind of file, an untrusted parent, a mount or a file that's too big rejects as without the option, and any other value of `absent` is `EINVAL`.
 - **`watch(paths, onChange)`** resolves with an unsubscribe function. `onChange(path)` is called once per burst of changes, 50 ms after the last one, for each watched file that changed.
   - A change is a write, a truncate, a rename, a removal, or a change of mode, owner or times of the file or of any directory above it. A change to a parent directory's own metadata counts too, and creating or removing a sibling changes it when the change comes from another kernel.
   - It sees changes made through any slicc kernel on the origin, in this tab or another one: the kernels tell each other which files are watched and report writes to those. `watch` resolves once every other live kernel on the origin has the watch (a kernel that doesn't answer within 500 ms is not waited for), so a write made after it resolves is reported. A kernel that ends with watches still on is forgotten (by its Web Lock) at the next reported write or unanswered announcement. This needs the shared metadata (the default): with `metadata: false` a kernel sees only its own changes. It doesn't see writes made with the OPFS API outside a kernel, such as a page script or an installer writing the root directly. After such a write the page should attest again itself (and may also attest on tab focus, as a safety net).
@@ -479,7 +480,7 @@ The module's default export (or its `main` export) is called with a context and
 - `crontab({ action, user?, text? })`, what the [`crontab`](#cron) command calls: `action` is `'list'`, `'remove'` or `'install'` (with the table as `text`), on the caller's own table or, for an effective uid of 0, on `user`'s. It resolves with `{ text }` (the table for `list`, `''` otherwise) or `{ error }`, the message to print. A malformed call is `EINVAL`;
 - `net`: `listen({ port, host, backlog })` and `connect({ host, port, signal })` (`host` defaults to `127.0.0.1`) on the kernel's loopback, the same sockets WASIX programs use, so a JS server is reached by other processes, by the page as `<port>.kernel.localhost` (`loopbackFetch`, `dial`) and by attached clients. A listener has its `port` (as bound, so port `0` picks a free one), `accept({ signal })`, `close()` and `for await` over its connections. A connection has `local` and `remote` addresses, `readable` and `writable` web streams (closing the writable shuts down the sending side, so the peer reads the end), `close()`, and `reset()`, which closes it with an RST as Node's `resetAndDestroy()` does ([#1530](https://github.com/ai-ecoverse/slicc-kernel/issues/1530)): on the kernel's loopback, the peer first reads the bytes it has already received, as on Linux and BSD ([#1577](https://github.com/ai-ecoverse/slicc-kernel/issues/1577)), and then its reads fail with `ECONNRESET` instead of seeing the end; its writes fail with `ECONNRESET` at once. A TLS connection (`secure()`, a `tls` connect on the loopback) resets the same way. On an uplink connection `reset()` closes at once with a FIN, as the transport can't send an RST: bytes still buffered for the peer are discarded instead of sent. The loopback is IPv4 (`127.0.0.1` by default, or `0.0.0.0`), so an IPv6 host such as `::1` fails with `EADDRNOTAVAIL` for both `listen` and `connect`, and binding and connecting fail as they would for a WASIX program (`EADDRINUSE`, `ECONNREFUSED`, `EADDRNOTAVAIL`). `sync.net.listen` takes the same options and gives the same listener, bound and listening when it returns, so its `port` is known at once, as Node's `listen()` needs ([#1398](https://github.com/ai-ecoverse/slicc-kernel/issues/1398)); it needs the sync lane. A blocked accept, read or write waits without holding anything in the kernel, so aborting it or closing the socket leaves nothing behind. A write to a connection the peer closed fails with `EPIPE` and no `SIGPIPE`;
 - `net.connect({ host, port, tls: { servername, alpn, rejectUnauthorized, ca, path, cert, key, passphrase, sni, checkHostname, keylog } })` takes one of two paths. By default it's **A1, TLS through the transport** ([#914](https://github.com/ai-ecoverse/slicc-kernel/issues/914)), described first below. For a destination the uplink routes it can also be **A2, a TLS handshake in the kernel** ([#922](https://github.com/ai-ecoverse/slicc-kernel/issues/922)), described after it. `conn.tls.path` (`'transport'` or `'kernel'`) and `conn.tls.substitutes` say which one was taken.
-  - **What it does:** there's no handshake in the kernel. The connection carries plain HTTP/1.1 (and WebSocket upgrades) to the realm proxy's HTTP path for the origin `https://<servername or host>[:port]`, exactly as a `CONNECT` through `127.0.0.1:3128` does after the kernel terminates its TLS. The transport (the page's `fetch` or WebSocket, or the uplink's) makes the real TLS connection and verifies it. So `https.request` over a socket, keep-alive agents, `agent-base` and `wss://` clients work with the transport's semantics. A request with a sealed placeholder is refused there as from `ctx.fetch` (see **The transport** under [`/dev/sealed`](#devsealed)).
+  - **What it does:** there's no handshake in the kernel. The connection carries plain HTTP/1.1 (and WebSocket upgrades) to the realm proxy's HTTP path for the origin `https://<servername or host>[:port]`, exactly as a `CONNECT` through `127.0.0.1:3128` does after the kernel terminates its TLS. The transport (the page's `fetch` or WebSocket, or the uplink's) makes the real TLS connection and verifies it. So `https.request` over a socket, keep-alive agents, `agent-base` and `wss://` clients work with the transport's semantics. A request with a sealed placeholder goes the same way as from `ctx.fetch` (see **The transport** under [`/dev/sealed`](#devsealed)).
   - **The `tls` property:** the connection's `tls` is `{ path: 'transport', substitutes: true, protocol: 'transport', alpnProtocol: 'http/1.1', authorized: true, servername }`. `authorized` means the transport verified the server. There's no peer certificate chain and no TLS version of the kernel's own, so don't read it as a real handshake's result.
   - **Options:** the transport always verifies, so `rejectUnauthorized: false` is accepted and changes nothing. Outside the uplink, a custom `ca` is refused (`ENOTSUP`), and so is an `alpn` list without `http/1.1` (`EPROTONOSUPPORT`). The kernel's own loopback never goes to the transport. A `host` of `localhost` or `127.x` makes the handshake in the kernel instead (Loopback, below), so with nothing listening it's `ECONNREFUSED`. The rest of it (`*.localhost`, `0.0.0.0`, `::1`, and any loopback `servername` for another host) is `ENOTSUP` at connect, as `ctx.fetch` refuses `https:` and `wss:` there. `host.slicc.internal` goes to the transport under its policy, as for `ctx.fetch`.
   - **Non-HTTP protocols aren't carried:**
@@ -487,7 +488,7 @@ The module's default export (or its `main` export) is called with a context and
     - non-HTTP first bytes on any other port get a `400` and a close;
     - a server-speaks-first protocol on another port (the client sends nothing) waits for the proxy's 120 s idle limit.
     - For a destination the uplink routes, these cases take A2 instead (below).
-  - **Which path** (A2 needs a destination the uplink routes, `net.destination(host) === 'uplink'`; an uplink-routed host on 443 with default options stays on A1, where a request with a sealed placeholder is refused):
+  - **Which path** (A2 needs a destination the uplink routes, `net.destination(host) === 'uplink'`; an uplink-routed host on 443 with default options stays on A1, where a request with a sealed placeholder goes to the fetch hook or is refused):
 
     | The call | Uplink destination | Any other destination |
     |---|---|---|
@@ -780,7 +781,7 @@ interface ConnectorStamp { pid: number; ppid: number; uid: number; gid: number;
 
 - **Declared at boot:** `facades` names each facade (`[a-z][a-z0-9-]*`) and its port, from 1 to 65535. A port may not be 9222 (the CDP facade), 3128 (the realm proxy) or another facade's; anything else throws a `TypeError`. `createNodeKernel` takes `facades` too. The hook may be left out and supplied later by an attached client's `client.serveFacade(name, hook)`, which works only for a declared name.
 - **Programs** find each facade in their environment as `SLICC_<NAME>_URL`, upper-cased with `-` as `_`, e.g. `SLICC_LINK_URL=tcp://127.0.0.1:7400`, and connect to it with an ordinary TCP `connect()`: a JS program's `ctx.net.connect`, an Emscripten program's socket, or a WASIX program's `sock_connect`. Every ABI ends in the same kernel call.
-- **The stamp** is the connecting process's identity, taken by the kernel at `connect()` from its credentials, as Linux's `SO_PEERCRED` is: its pid and parent pid as `ps` shows them, effective uid and gid, user name, groups, the `users` mode and the process's tag (see `kernel.run`), the same fields as a `/dev/sealed` request's `caller`. A program cannot forge it. The hook gets it and `{ name }`, so one function can serve several facades, and decides: it resolves with a connection, whose bytes then pass both ways until either side closes, or it rejects, and the program's connection is closed. A connection while no hook is attached is closed at once, the same as a refusal. Bytes from the hook wait in the kernel until the program reads them, up to 16 MiB per connection: past that, the connection is closed and the hook's `close(reason)` gets `slicc-kernel: more than 16777216 bytes are waiting for the program to read`. A program that connects to anything else pays nothing for this: the stamp is built only for a facade's port.
+- **The stamp** is the connecting process's identity, taken by the kernel at `connect()` from its credentials, as Linux's `SO_PEERCRED` is: its pid and parent pid as `ps` shows them, effective uid and gid, user name, groups, the `users` mode and the process's tag (see `kernel.run`), the same fields as a `/dev/sealed` request's `caller`. A program cannot forge it. The hook gets it and `{ name }`, so one function can serve several facades, and decides: it resolves with a connection, whose bytes then pass both ways until either side closes, or it rejects, and the program's connection is closed. A connection while no hook is attached is closed at once, the same as a refusal. Bytes from the hook wait in the kernel until the program reads them, up to 16 MiB per connection: past that, the connection is closed and the hook's `close(reason)` gets `slicc-kernel: more than 16777216 bytes are waiting for the program to read`. A program that connects to anything else pays nothing for this: the stamp is built only for a facade's port, and for the realm proxy's when the kernel has a fetch hook (see **The fetch hook** under [`/dev/sealed`](#devsealed)).
 - **Handing a connection on hands over the identity:** a facade connection that a process passes to another (a fork, an exec, an inherited descriptor) still speaks as the process that connected, as on Linux. So a privileged process should not pass one to a less trusted one.
 - **Reserved:** a facade's port is reserved from the start. A program's `bind()` to it fails with `EADDRINUSE`, so no program can squat it and impersonate the page, and ephemeral ports skip it. Port 9222 is reserved the same way. Like 9222, a facade's port is kernel-only: the page's `kernel.dial`, a client's `dial` and `loopbackFetch` (and so `<port>.kernel.localhost`) are refused with `ECONNREFUSED`.
 
@@ -1134,12 +1135,26 @@ The kernel then asks the hook `{ v: 1, op: 'materializeConsume', ticket, stamp,
 
 The kernel never asks for a session that isn't `verified`, whose handshake is more than 60 s old (or in the future), or whose fields aren't of that shape. The reply is `{ values: { [placeholder]: value } }` within 10 s; the kernel takes only the placeholders it asked for, with a non-empty value free of CR, LF and NUL, and anything else (an error such as `{ v: 1, error: { code } }`, a timeout, a malformed reply, no hook) means nothing is substituted: the placeholders go out as written.
 
-**The transport.** Nothing fills a placeholder in a request that leaves through the transport, so the kernel refuses that request itself and never hands it to the transport. This covers the realm proxy (`curl`, `CONNECT`, TLS on A1), `ctx.fetch`, `ctx.websocket` and a client's `fetch` and `transport.websocket`. The check needs no vault and no `sealed` hook: anything of the form `fnox_proxy_<24 hex>` is refused, whether or not the vault knows it.
+**The transport.** A request with a sealed placeholder never reaches the transport. The kernel looks for one before anything else, so a held redirect isn't replayed for it either. This covers the realm proxy (`curl`, `CONNECT`, TLS on A1), `ctx.fetch`, `ctx.websocket` and a client's `fetch` and `transport.websocket`. The check needs no vault and no `sealed` hook: anything of the form `fnox_proxy_<24 hex>` counts, whether or not the vault knows it, and so does anything in the list of `kernel.sealed.setPlaceholders`. The kernel then asks the fetch hook (below) or refuses.
 
 - **Where it looks:** the URL (host, user info, path, query and fragment, percent-encoded or not), header names and values, WebSocket subprotocols, and `Basic` credentials, decoded for the check (the user, the password, or a credential that is malformed or not UTF-8). A `Basic` credential without a placeholder goes out byte for byte. Bodies aren't read: a placeholder in a body goes out as written.
-- **What a refusal looks like:** the realm proxy answers `403` with the reason in the body, for a WebSocket upgrade too. `ctx.fetch` and `ctx.websocket` reject with a `TypeError` whose `cause.code` is `EACCES` and, for `ctx.fetch`, whose `cause.detail` is the reason. A client's `fetch` and `transport.websocket` reject with `code` `EACCES` and `status` `403`. Nothing is sent, retried or sent another way, and a held redirect isn't replayed for it.
+- **What a refusal looks like:** the realm proxy answers `403` with the reason in the body, for a WebSocket upgrade too. `ctx.fetch` and `ctx.websocket` reject with a `TypeError` whose `cause.code` is `EACCES` and, for `ctx.fetch`, whose `cause.detail` is the reason. A client's `fetch` and `transport.websocket` reject with `code` `EACCES` and `status` `403`. Nothing is sent, retried or sent another way.
 - **What stays as it was:** the kernel's own loopback, A2 sessions (above) and raw sockets never reach this check.
 
+**The fetch hook.** `createKernel({ sealed: { request, fetch } })`, or `createNodeKernel` with the same option, gives the kernel `fetch(req, { signal })`, which is asked instead of a refusal. A bare function as `sealed` is `{ request }`, so `/dev/sealed`, tickets and A2 sessions work with either form. `SealedFetchHook` and `SealedFetchRequest` are exported from both entries.
+
+- **When it's asked:** only with `users: 'enforce'`, only for a request of a process, and never for a WebSocket. In every other case, and with no hook, the request is refused as above: there's nothing else to fall back on.
+- **`req`** is `{ v: 1, caller, method, url, headers, redirect: 'manual', body?, bodyStream? }`. `headers` is the flat list of pairs of a transport request. `bodyStream` is an async iterable that yields one chunk each time it's asked, so an upload moves at the speed the hook reads it.
+- **`caller`** is the same stamp as a `/dev/sealed` request's, and no program can supply it. For the realm proxy it's the stamp of the process that connected, taken at `connect()`, as `SO_PEERCRED` is. For `ctx.fetch` it's built for each request from the process's current credentials, so it follows a `setresuid`, and a process that has exited has none.
+- **The reply** is a transport response that has been scrubbed: `{ status, statusText, headers, body, cancel, url? }`. The kernel gives it to the program as it is, one hop, and keeps none of it. A redirect is a 3xx for the program to follow, with its original placeholders and a new call to the hook.
+- **A failed call:** a hook that throws, a `signal` that aborts, and a reply that isn't a response end in a refusal with `refused: the request with a sealed placeholder was not completed` and the reason, which the program sees, so a hook's error message must be scrubbed too. So does a reply with `redirects`, even an empty list, or `redirected: true`, which means something followed a redirect with the credential: the kernel cancels it. The request never reaches the transport.
+- **`signal`** aborts when the program drops the request, on `terminate()`, and after 5 minutes with neither a head, an upload chunk nor a response chunk. After the head, a dropped response also calls `cancel()`.
+- **On the page**, the call runs from the kernel worker through a `MessagePort` of its own. Each upload or response chunk is sent when the far side asks for it, so one chunk at most is on the way in each direction. When the port closes or the kernel ends, both sides stop.
+
+The stamp also goes to the transport, as the third argument of `transport.fetch(request, owner, stamp)`, while the kernel has a fetch hook. It's for the transport's own routing: the kernel never takes a stamp or an owner from a client or a process.
+
+**`kernel.sealed.setPlaceholders(list)`** replaces the list of placeholders the check looks for beside the form above, for a vault whose placeholders don't all have that form. `list` is an array of non-empty strings (`EINVAL` otherwise), placeholders and never values. It's on the page's `Kernel` and on `createNodeKernel`'s, and not on an attached client or a process. The form above is always checked, so an empty or outdated list changes nothing about it.
+
 ### `/dev/clipboard`
 
 A character device (`0666`), there in every `users` mode, through which programs read and write the embedder's clipboard (`createKernel({ clipboard: { read, write } })`; on the page, bios answers with `navigator.clipboard`). Emscripten programs (`cat /dev/clipboard`, `cat > /dev/clipboard`, see it in `/dev`, reach it by any path that leads to it, and see it as a character device after fork and exec), WASI programs (`std::fs`) and JS programs (`ctx.fs.readFile`, `ctx.fs.writeFile`, `ctx.open`) all open it.

package/dist/driver-worker.js

@@ -397,7 +397,7 @@ var RemoteTransport = class {
     waiter?.reject(new TransportError("the request was cancelled"));
     this.port.postMessage({ net: "cancel", nid });
   }
-  async fetch(request) {
+  async fetch(request, owner, stamp) {
     const nid = ++this.nextId;
     const { signal, redirect } = request;
     signal.throwIfAborted();
@@ -427,7 +427,9 @@ var RemoteTransport = class {
         method: request.method,
         headers: request.headers,
         ...request.bodyStream ? { streamed: true } : request.body ? { body: request.body } : {},
-        ...redirect !== void 0 ? { redirect } : {}
+        ...redirect !== void 0 ? { redirect } : {},
+        ...owner !== void 0 ? { owner } : {},
+        ...stamp ? { stamp } : {}
       });
     } catch (e) {
       finish();