@ai-ecoverse/slicc-kernel 1.61.1 → 1.62.0

package/README.md

@@ -38,7 +38,8 @@ const { status, stdout, stderr } = await kernel.run(['bash', '-c', 'echo hi > he
 | `onMountPending` | none | `({ target, source, insert }) => void`: an `fsa` mount needs a folder; call `insert()` from a user gesture |
 | `processMounts` | `true` | whether programs may call `mount(2)` and `umount2(2)`: `false`, or `(req) => boolean \| Promise<boolean>` to decide each call (see [Mounting from a program](#mounting-from-a-program)) |
 | `hostname` | `'slicc'` | the kernel's node name (see [Node name](#node-name)) |
-| `users` | `'off'` | `'off'`, `'report'` or `'enforce'`: whether files have owners (see [File owners](#file-owners)). `'off'` behaves exactly as kernels before owners did; any other value rejects. `createNodeKernel` takes it too |
+| `users` | `'off'` | `'off'`, `'report'` or `'enforce'`: whether files have owners and their modes are checked (see [File owners](#file-owners) and [Permissions](#permissions)). `'off'` behaves exactly as kernels before owners did; any other value rejects. `createNodeKernel` takes it too |
+| `onDenial` | none | `({ pid, uid, op, path, errno }) => void`: called for every permission check that fails, under `'report'` and `'enforce'` (see [Permissions](#permissions)). `createNodeKernel` takes it too |
 | `cdp` | none | `({ runtime }) => Promise<CdpConnection>`: a browser-level DevTools connection for programs that drive a browser (see [Browser automation](#browser-automation-cdp)) |
 
 ### `kernel.run(argv, options?) → Promise<{ status, stdout, stderr }>`
@@ -160,7 +161,7 @@ const { status, stdout } = await kernel.run(['bash', '-c', 'echo hi'], { cwd: '/
 kernel.terminate();
 ```
 
-`createNodeKernel({ root, modules, env, network, worker, processMounts, fstabRetries, cdp, hostfs, hostfsOrigin, hostfsFetch })` takes the options of `createKernel` except `metadata`, and `fstabRetries`, the delays in ms between tries of a failing `/etc/fstab` line (default `[1000, 4000, 16000]`). `root` is an in-memory directory by default (`memoryRoot()` makes another), which keeps each file in 1 MiB chunks, and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal`, `users`, `setRoutes` and `terminate` as above, plus `root`, `writeFile(path, data)` (creating the parent directories) and `readFile(path)` to put files in place and read results; like the `fs` of an attached client, they go through the mounts, as the processes see them. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`). `connect({ user })` and `attachKernel` work as in the browser: the port is a `worker_threads` `MessagePort`, which a worker thread can attach with, and `terminate()` ends every attached client.
+`createNodeKernel({ root, modules, env, network, worker, processMounts, fstabRetries, cdp, hostfs, hostfsOrigin, hostfsFetch, users, onDenial })` takes the options of `createKernel` except `metadata`, and `fstabRetries`, the delays in ms between tries of a failing `/etc/fstab` line (default `[1000, 4000, 16000]`). `root` is an in-memory directory by default (`memoryRoot()` makes another), which keeps each file in 1 MiB chunks, and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal`, `users`, `setRoutes` and `terminate` as above, plus `root`, `writeFile(path, data)` (creating the parent directories) and `readFile(path)` to put files in place and read results; like the `fs` of an attached client, they go through the mounts, as the processes see them. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`). `connect({ user })` and `attachKernel` work as in the browser: the port is a `worker_threads` `MessagePort`, which a worker thread can attach with, and `terminate()` ends every attached client.
 
 ## Commands
 
@@ -228,7 +229,8 @@ A WASI command can name an ES module of its package that provides imports of its
 - `spawn({ argv, env, cwd, stdin, stdout, stderr })` starts a command (resolved like `kernel.run`) as a child and returns `{ pid, stdin, stdout, stderr }`. `env` is the child's whole environment (the program's own by default), `cwd` the program's by default. Each stdio is `'inherit'` (the default: the program's own), `'null'`, or `'pipe'`: a kernel pipe whose end the result holds as a descriptor, read and written with `fd-read` (`{ op: 'fd-read', fd, max }`, an empty read at the end), `fd-write` (`{ op: 'fd-write', fd, body }`) and `fd-close`, through `syscall` or `async.submit`; the child's ends are closed in the program. A command that does not exist throws `ENOENT`;
 - `wait(pid)` blocks until a child ends and returns `{ status }`, or `{ status: 128 + signal, signal }` when a signal ended it; `kill(pid, sig)` signals it;
 - `async`: operations of the process that any of its threads can wait for: `submit(req)` runs a kernel syscall without blocking the caller, `resolve(value)` completes at once, `hold()` never completes, and each returns an id; `wait(timeoutMs?)` returns the id of a completed one (0 after the timeout, -1 once `close()` was called), `take(id)` its `{ value }` or `{ error }` (`undefined` while it is pending), and `cancel(id)` drops one;
-- `errno(err)`: the WASI errno of an error.
+- `errno(err)`: the WASI errno of an error;
+- `uid` and `gid`, with `users: 'report'` or `'enforce'`: the process's effective uid and gid, read again at each use, so they follow a `setresuid`. `fstat` then shows a file's real owner, and the process's ids for a descriptor with no path. With `users: 'off'`, `ctx` has no `uid` or `gid`, and `fstat` answers 1000 for both, as before.
 
 ### JS programs
 
@@ -406,7 +408,7 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
 - **File contents:** on a mount with `ranges`, as `tmpfs` is, and on the root file system when it is in OPFS (in a browser with sync access handles) or an in-memory root, a program reads and writes a file in 64 KiB pages: reads come from the driver 1 MiB at a time, and changed pages go back in 1 MiB runs once 8 MiB have changed, at `fsync`, and when the file is closed (from the kernel's own descriptors also 250 ms after a write). A file of any size the mount can hold works, with memory to spare for about one more copy of it. In OPFS a write opens a sync access handle for just that write. An `fsa` mount (a picked folder) has no sync access handles, and Chrome copies the whole file into a swap file for every writable stream, so it writes through one stream per open file: the first write back opens it, later runs go into it at their offsets, and it is committed, one copy of the file, when the last process writing the file closes it (or exits), before another process reads what was written, and before a rename, unlink or unmount. Until then other processes see the new size but read the old bytes outside what was written. Elsewhere (`hostfs` with an older proxy, and a root outside OPFS), file contents are read whole when a program opens a file and written back when it closes it. Before a program's next call into the kernel (any wait, read of a pipe or terminal, sleep or spawn), its changed pages also go back, so other processes and the embedder see a write as soon as the writer does anything but compute: a daemon's pidfile, a log another process follows. On a whole-file mount that holds only for files of up to 64 KiB; a larger one goes back at close, fsync and exit, as before. Beyond that, both give close-to-open consistency, as NFS has. A process that ends without closing its files keeps what it wrote: on exit, on a trap, and when a signal kills it. For a killed WASI program, the kernel lets its worker write its changed pages back before `waitpid` reports the death. The only exception is a worker that makes no call into the kernel for 2 seconds after the signal; it is stopped without that write-back.
 - **Rules on a mount:**
   - a rename across mounts is `EXDEV`, so `mv` copies;
-  - a hard link (`link()`, `linkat()`, Python's `os.link`, `ctx.fs.link`) works within a mount whose driver offers `links`, as `tmpfs` does: both names are one file with one inode, `st_nlink` counts them, and the data goes with the last name and the last open descriptor. Across mounts it's `EXDEV`. It's `EPERM` on the root file system (OPFS or memory), on `fsa` folders, on `hostfs` until slicc-node offers links, and for directories. `link()` links a symlink itself and `linkat(…, AT_SYMLINK_FOLLOW)` links its target, as on Linux. Descriptors the kernel holds (JS programs' files, and a WASI program's files once it forks or spawns with them) share one cache for all names of a file, so a write through one name is read at once through another, also after either name is removed or replaced while open; `link()` writes the source's pending changes back first. Within one WASI program, its own descriptors of a file's names share one buffer too, except after it starts a subprocess: the files it had open are kernel descriptors from then on, and a later open of the same file, under that name or another, is only consistent with them at open and close ([#355](https://github.com/ai-ecoverse/slicc-kernel/issues/355)); between programs, a WASI program's own descriptors see the other names' writes at open, as for one name across processes. A WASI program that removes or renames over the name it has open, while another link remains, keeps writing to the file through its descriptor, and the other names see those writes; `linkat(…, AT_SYMLINK_FOLLOW)` writes back the program's pending changes to the symlink's target first. `st_nlink` leaves out the names the kernel keeps in `.slicc-unlinked` for open descriptors, so `fstat` on a descriptor whose name was removed counts the names left: 0 once the last one is gone. The other names of a linked file keep the attributes the kernel has cached for them for up to the mount's `attrTtl` (`tmpfs` caches none). Emscripten programs (bash, coreutils `ln`) get `EMLINK` from their own C library, which never asks the kernel, but they see links other programs made: their open descriptors of one file's names share one buffer, and one that removes the name it has open while another link remains keeps writing to the file;
+  - a hard link (`link()`, `linkat()`, Python's `os.link`, `ctx.fs.link`) works within a mount whose driver offers `links`, as `tmpfs` does: both names are one file with one inode, `st_nlink` counts them, and the data goes with the last name and the last open descriptor. Across mounts it's `EXDEV`. It's `EPERM` on the root file system (OPFS or memory), on `fsa` folders, on `hostfs` until slicc-node offers links, and for directories. `link()` links a symlink itself and `linkat(…, AT_SYMLINK_FOLLOW)` links its target, as on Linux. Descriptors the kernel holds (JS programs' files, and a WASI program's files once it forks or spawns with them) share one cache for all names of a file, so a write through one name is read at once through another, also after either name is removed or replaced while open; `link()` writes the source's pending changes back first. Within one WASI program, its own descriptors of a file's names share one buffer too. Once it starts a subprocess, the files it had open are kernel descriptors, and a later open of one of them, under that name or another (same inode), goes through the kernel and shares their cache; between programs, a WASI program's own descriptors see the other names' writes at open, as for one name across processes. A WASI program that removes or renames over the name it has open, while another link remains, keeps writing to the file through its descriptor, and the other names see those writes; `linkat(…, AT_SYMLINK_FOLLOW)` writes back the program's pending changes to the symlink's target first. `st_nlink` leaves out the names the kernel keeps in `.slicc-unlinked` for open descriptors, so `fstat` on a descriptor whose name was removed counts the names left: 0 once the last one is gone. The other names of a linked file keep the attributes the kernel has cached for them for up to the mount's `attrTtl` (`tmpfs` caches none). Emscripten programs (bash, coreutils `ln`) get `EMLINK` from their own C library, which never asks the kernel, but they see links other programs made: their open descriptors of one file's names share one buffer, and one that removes the name it has open while another link remains keeps writing to the file;
   - `df` reports each mount from its driver's `statfs`;
   - each mount has its own device number;
   - without `chmod` support, `chmod` succeeds and changes nothing;
@@ -520,7 +522,7 @@ Lowering a soft resource limit (`ulimit -S -n 64`) still fails with `EPERM` in E
 
 ### Users
 
-Every process has credentials in the kernel, as on Linux: real, effective and saved user and group ids, and supplementary groups. A process gets them from the user it was started as (`run`, `openTerminal`, a client's `spawn`, or the client's own user; root by default) and keeps its parent's across fork, spawn and exec. Setuid and setgid bits are never honoured on exec. Files may have owners (see [File owners](#file-owners)) but aren't checked against the credentials yet; they decide who may signal whom, and what `/proc` and `ps` show.
+Every process has credentials in the kernel, as on Linux: real, effective and saved user and group ids, and supplementary groups. A process gets them from the user it was started as (`run`, `openTerminal`, a client's `spawn`, or the client's own user; root by default) and keeps its parent's across fork, spawn and exec. Setuid and setgid bits are never honoured on exec. Files may have owners (see [File owners](#file-owners)), checked against the credentials with `users: 'enforce'` (see [Permissions](#permissions)); the credentials also decide who may signal whom, and what `/proc` and `ps` show.
 
 - **The database:** `/etc/passwd` and `/etc/group` are real files. On first start the kernel writes them when they're missing, with `root` (uid 0, home `/root`, created `0700`) and `nobody` (65534), and the groups `root`, `sudo` (27), `users` (100) and `nogroup` (65534); files already there are kept as they are. The kernel reads them for every lookup, so an edit counts at once. [`kernel.users`](#kernelusers) adds and removes users.
 - **Environment:** a process started as a user gets `HOME`, `USER`, `LOGNAME`, `PNPM_HOME`, `NPM_CONFIG_PREFIX` and `PATH`. A user's `PNPM_HOME` is `~/.local/share/pnpm` and its npm prefix `~/.local`; root's are `/usr/local/share/pnpm` (or the embedder's `env.PNPM_HOME`) and `/usr/local`. `PATH` is `$PNPM_HOME/bin:/usr/local/bin:/usr/bin:/bin`.
@@ -536,7 +538,7 @@ Programs ask the kernel for their ids:
 
 ### File owners
 
-With `createKernel({ users: 'report' })` or `'enforce'`, files have owners. Nothing checks them against a process's credentials yet; `'report'` and `'enforce'` differ once permissions are checked. With `'off'`, the default, nothing below happens: the kernel stores no owner, and every file looks owned by the caller as described under [Metadata](#metadata).
+With `createKernel({ users: 'report' })` or `'enforce'`, files have owners, and [Permissions](#permissions) checks them against a process's credentials. With `'off'`, the default, nothing below happens: the kernel stores no owner, and every file looks owned by the caller as described under [Metadata](#metadata).
 
 - **Where owners live:** on the root file system, in the sidecar record's `uid` and `gid` (see [Metadata](#metadata)). A path with no owner of its own has the owner of its nearest ancestor that has one, and `/` belongs to root, so a root written before owners existed, or by the BIOS through the OPFS API, is root's throughout. A directory whose owner changes keeps the owner its unowned contents had, in `heir`, so `chown cone /old` gives `cone` the directory and leaves the files below it to their old owner. A rename keeps the owner of what it moves.
 - **New files:** a file, directory or symlink a process creates belongs to its effective uid, and to its effective gid, or to the directory's group when the directory has the setgid bit; a new directory in a setgid directory gets the setgid bit too. Every file and directory a process creates gets the mode it asked for (`0666` and `0777` when it can't ask, as a JS program's `ctx.fs` can't) less the process's umask, whichever runtime made it; an Emscripten program's file no longer gets `0644` whatever it asked for. A file written through a client's `fs` belongs to the client's user (root for a page) and gets `0666`, a directory `0777`, less the umask `kernel.users.add` gave that user (`022` for root and by default); the kernel's own files, such as `/etc/passwd`, have no owner of their own.
@@ -545,6 +547,18 @@ With `createKernel({ users: 'report' })` or `'enforce'`, files have owners. Noth
 - **What programs see:** `stat` shows the owners to Emscripten programs, whatever their libc reports for their own ids, so a program built before H1 sees real owners too, and its owner checks can fail. WASI programs read them with `slicc_fs.fd_owner` and `path_owner`. The commands in `/usr/bin` and `/bin` are root's, and a user's own commands in `~/.local/share/pnpm/bin` belong to the owner of the closest real directory above them, normally the user.
 - **Mounts:** the options `uid=` and `gid=` (decimal ids) give a mount its owner; without them it belongs to whoever mounted it: root for `kernel.mount`, `/etc/fstab` and a client, the process's effective ids for `mount(2)`, which then shows them in its options. A mount whose driver keeps owners (`tmpfs`) starts with its root owned so, and its files get owners as on the root file system, `chown` included. On `hostfs`, `fsa` and any driver without `owners`, every file shows as the mount's owner and `chown` fails with `EPERM`, root's too, as on a FAT drive. An embedder that mounts a folder for a user (bios's linked and picked folders) passes that user's `uid` and `gid`, or the user's programs see root's files there, and git calls the repository's ownership dubious.
 
+### Permissions
+
+With `users: 'enforce'` the kernel checks every file call that reaches it against the caller's effective uid, effective gid and supplementary groups, as Linux does: the owner bits for the file's owner, else the group bits for a member of its group, else the other bits. Root (effective uid 0) passes every check except executing a file with no `x` bit at all. A failed check is `EACCES`; changing what only the owner may change is `EPERM`. With `users: 'report'` the kernel makes the same checks and refuses nothing: it logs each denial once per user, operation, errno and path, with `console.warn`, up to 10,000 of them, then says that more were suppressed. With both, `onDenial` gets each denial as `{ pid, uid, op, path, errno }` (`pid` 0 for a client's `fs`); the shape is stable. With `users: 'off'` nothing is checked and no call is added.
+
+- **Paths:** a path needs `x` on every directory it goes through, the targets of symlinks on the way included, resolved as the file system resolves them (a `..` after a symlinked directory goes to the target's parent). `stat`, `lstat`, `readlink` and `chown` need only that; `exists` answers `false` where it's refused. Listing a directory needs `r` and `x` on it.
+- **Creating and removing:** `mkdir`, creating a file and `symlink` need `w` and `x` on the directory; `mkdir -p` needs them on the closest directory that exists, and nothing for one that exists already. Unlinking, `rmdir` and `rename` need `w` and `x` on both directories, and a directory moved to another needs `w` on itself. A recursive remove (`ctx.fs.rm`, a client's `rm`) needs `r`, `w` and `x` on every directory in the tree.
+- **The sticky bit:** in a directory with mode `1000` set, such as `/tmp` at `1777`, only the file's owner, the directory's owner or root may unlink or rename a file (`EPERM` otherwise).
+- **Hard links:** `link` needs `w` and `x` on the target's directory. As with Linux's `protected_hardlinks`, a user who doesn't own a file may link it only when it may read and write it (`EPERM` otherwise); root may link anything.
+- **Modes and times:** only the owner or root may `chmod` (`EPERM`), and a `chmod` by a user that isn't in the file's group clears its setgid bit. Only the owner or root may set a file's times (`EPERM`).
+- **A client's `fs`:** checked as the client's user; a page's client (root) passes.
+- **Not yet checked:** opening a file and reading or writing it, executing it, and `access(2)`; until they are, `users: 'enforce'` isn't a security boundary, and no release turns it on by default before `sudo` exists.
+
 ### `/proc`
 
 Every process of the kernel has `/proc/<pid>/` with `cmdline`, `comm`, `stat`, `statm` and `status`, in the formats procps reads, and `/proc/self` is a link to the `/proc/<pid>` of the process that reads it, as on Linux (`readlink /proc/self` is its pid, the one it was exec'd under included). They come from the kernel's process table when they are opened, so a terminal sees the processes of every client. A process that exits while a program reads the `/proc` directory doesn't fail or cut short the listing: its pid is still listed, and opening its files fails with `ENOENT`, which `ps`, `pgrep` and `pidof` skip, as on Linux; an exec'd program shows under the pid its parent knows. `/proc/uptime`, `/proc/loadavg`, `/proc/stat` and `/proc/meminfo` are there too, and `/proc/mounts`. A process with no parent in the kernel (one a page, a client or the Node entry started) has parent 1, as children of init do, in `getppid()`, `/proc/<pid>/stat` and `ps` alike, and so does a process whose parent has ended, waited for or not (`getppid()` asks the kernel each time; a shell's `$PPID` keeps the value it started with, as on Linux). Pids, parents, process groups, sessions, command lines, terminals, start and boot times are real, and so are the credentials in `status` (`Uid:`, `Gid:`, `Groups:`) and each process's memory: the size of its wasm memory (`VSZ` and `RSS` alike, as wasm has no paging), which `/proc/meminfo` counts as used. CPU times, load and `MemTotal` (from `navigator.deviceMemory`, else 4 GiB) are placeholders. A process that isn't root sees only its own user's processes (see [Users](#users)). WASI and WASIX programs see the same `/proc`, read-only (opening a file in it for writing, or linking one of its files elsewhere, fails with `EROFS`), from every thread, also through a symlink that points into it (where every change, `mkdir`, `rename`, `unlink`, `chmod`, a hard link from it or into it, and so on, fails too: with `EROFS` when the root file system has a physical `/proc`, which stays hidden and untouched, and otherwise with the `ENOENT` of the missing directory, since that check is only paid for when a physical `/proc` exists; with one, a symlink whose target passes `..` after a file or a missing name, such as `/f/../proc`, fails with `ENOTDIR` or `ENOENT`, as on Linux), and `/` lists it, so a WASI program gets a preopen for it; `/proc/self` is the program's own pid there too (the pid an exec'd program took over). Neither has `/proc/<pid>/fd` yet.

package/dist/index.js

@@ -630,8 +630,8 @@ function openWebSocket(Socket, request, withHeaders) {
       reject(error);
     };
     const abort = () => {
-      socket.close(1e3, "aborted");
       fail(signal.reason);
+      socket.close(1e3, "aborted");
     };
     signal.addEventListener("abort", abort, { once: true });
     socket.addEventListener("open", () => {
@@ -655,12 +655,12 @@ function openWebSocket(Socket, request, withHeaders) {
     });
     socket.addEventListener("error", () => {
       if (open || settled) return;
+      fail(unopened(url, 1006));
+      queue.end({ code: 1006, reason: "" });
       try {
         socket.close();
       } catch {
       }
-      queue.end({ code: 1006, reason: "" });
-      fail(unopened(url, 1006));
     });
     socket.addEventListener("close", (event) => {
       queue.end({ code: event.code, reason: event.reason });
@@ -1798,6 +1798,7 @@ function initCall(options, transport) {
     ...options.cdp ? { cdp: true } : {},
     ...options.hostname !== void 0 ? { hostname: options.hostname } : {},
     ...options.users !== void 0 ? { users: options.users } : {},
+    ...options.onDenial ? { denials: true } : {},
     ...options.processMounts !== void 0 ? {
       processMounts: typeof options.processMounts === "function" ? "ask" : options.processMounts
     } : {}
@@ -1843,6 +1844,15 @@ function kernelUsers(call, mode = "off") {
     list: async () => await call({ op: "users-list" })
   };
 }
+async function mountPolicy(options, worker, { id, req }) {
+  const decide = options.processMounts;
+  let allowed = false;
+  try {
+    allowed = typeof decide === "function" ? await decide(req) : decide !== false;
+  } catch {
+  }
+  worker.postMessage({ mountAllowed: { id, allowed } });
+}
 async function createKernel(options = {}) {
   if (!globalThis.crossOriginIsolated) throw new Error(ISOLATION);
   if (options.hostname !== void 0 && !isHostname(options.hostname)) {
@@ -1886,15 +1896,7 @@ async function createKernel(options = {}) {
       worker.postMessage({ hostfsGrant: { id, error: err.message ?? String(err) } });
     }
   }
-  async function policy({ id, req }) {
-    const decide = options.processMounts;
-    let allowed = false;
-    try {
-      allowed = typeof decide === "function" ? await decide(req) : decide !== false;
-    } catch {
-    }
-    worker.postMessage({ mountAllowed: { id, allowed } });
-  }
+  const policy = (asked) => mountPolicy(options, worker, asked);
   const pendingMedium = ({ target, source, handle }) => options.onMountPending?.({
     target,
     source,
@@ -1905,6 +1907,7 @@ async function createKernel(options = {}) {
     ({
       data
     }) => {
+      if ("denial" in data) return void options.onDenial?.(data.denial);
       if ("net" in data) return bridge?.answer(data);
       if ("uplink" in data) return uplinks?.answer(data);
       if ("cdpOpen" in data) return cdp.open(data);