@ai-ecoverse/slicc-kernel 1.141.3 → 1.142.1

package/README.md

@@ -202,7 +202,9 @@ const { status, stdout } = await kernel.run(['bash', '-c', 'echo hi'], { cwd: '/
 kernel.terminate();
 ```
 
-`createNodeKernel({ root, modules, commands, env, network, worker, facades, processMounts, fstabRetries, cdp, hostfs, hostfsOrigin, hostfsFetch, users, onDenial, approve })` takes the options of `createKernel` except `metadata`, and `fstabRetries`, the delays in ms between tries of a failing `/etc/fstab` line (default `[1000, 4000, 16000]`). `root` is an in-memory directory by default (`memoryRoot()` makes another), which keeps each file in 1 MiB chunks, and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal`, `users`, `setRoutes`, `updateTransportTraits` and `terminate` as above, plus `integrity.revoke(path?)`, `root`, `writeFile(path, data, { mode }?)` (creating the parent directories; `mode` sets the permission bits, `0` to `0o7777`, before the data is written), `chmod(path, mode)` and `readFile(path)` to put files in place and read results; like the `fs` of an attached client, they go through the mounts, as the processes see them. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`). `connect({ user })` and `attachKernel` work as in the browser: the port is a `worker_threads` `MessagePort`, which a worker thread can attach with, and `terminate()` ends every attached client.
+`createNodeKernel({ root, modules, commands, env, network, worker, facades, processMounts, fstabRetries, cdp, hostfs, hostfsOrigin, hostfsFetch, users, onDenial, approve, compileCache })` takes the options of `createKernel` except `metadata`, and `fstabRetries`, the delays in ms between tries of a failing `/etc/fstab` line (default `[1000, 4000, 16000]`). `root` is an in-memory directory by default (`memoryRoot()` makes another), which keeps each file in 1 MiB chunks, and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal`, `users`, `setRoutes`, `updateTransportTraits` and `terminate` as above, plus `integrity.revoke(path?)`, `root`, `writeFile(path, data, { mode }?)` (creating the parent directories; `mode` sets the permission bits, `0` to `0o7777`, before the data is written), `chmod(path, mode)` and `readFile(path)` to put files in place and read results; like the `fs` of an attached client, they go through the mounts, as the processes see them. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`). `connect({ user })` and `attachKernel` work as in the browser: the port is a `worker_threads` `MessagePort`, which a worker thread can attach with, and `terminate()` ends every attached client.
+
+`compileCache` (opt-in, off by default; #1285, slicc-jsh#566) is an absolute host directory where each process worker keeps the compiled code of the JS programs it imports (`module.enableCompileCache`), so the next spawn of the same program skips V8's compile. It stores V8's code cache, keyed and checksummed by the module's source, one file per program (about 7 MB for a 2.4 MB bundle), shared by every kernel given the same directory; programs can't reach or write it (the worker hands it to Node before the lockdown, and `node:worker_threads` stays refused). Measured on a 2.4 MB bundle, a spawn went from about 54 to 39 ms; a trivial program costs about 1 ms more. **The directory must be private to the embedder:** V8's code cache is trusted bytecode that Node only checksums, so whoever can write it can plant code that runs inside a process worker past the lockdown. Never make it reachable from inside the kernel (no `hostfs` mount of it or of a parent), and don't share one between kernels of different trust. A host's own `NODE_COMPILE_CACHE` is never handed to the process workers, with or without the option, so programs never land in a host-wide cache. If a host preload (`--import` or `--require`) has already enabled another compile cache in the workers, a kernel given `compileCache` refuses to run programs there instead of caching them in the host's directory.
 
 By default `nodeTransport()` refuses destinations on the host itself, because Node can reach them and a page can't: local DevTools on 9222–9229, dev servers, local admin APIs, and the cloud metadata address.
 - **What it refuses:** loopback (`127.0.0.0/8`, `::1`, and their IPv4-mapped forms), unspecified addresses (`0.0.0.0/8`, `::`), `localhost` and `*.localhost`, and link-local addresses (`169.254.0.0/16`, `fe80::/10`), plus AWS's IPv6 metadata address `fd00:ec2::254`.
@@ -746,7 +748,20 @@ The proxy answers `200` with `Content-Type: application/vnd.slicc.raw-fetch`: a
 
 `/` is the OPFS root. Each process mounts the top-level directories that exist when it starts (plus `/usr` and `/bin`), so files in them are shared by all processes and visible through the OPFS API as soon as the process that wrote them has closed them; `run` resolves only after that. `createKernel` creates `/tmp` and `/home` in OPFS, so they are shared too. Files directly in `/`, and directories created there after a process started, are OPFS as well, the same for Emscripten and WASI programs; only `/dev` and `/proc` live in each process's memory. A program's `mkdir` (WASI `path_create_directory`, Emscripten's `mkdir`) is exclusive, as on Linux: it fails with `EEXIST` when the name exists, and of several processes or threads making one name at once exactly one succeeds, so a directory works as a lock. Under a missing directory it fails with `ENOENT`, as on Linux (`mkdir -p` and `os.makedirs` make the parents in the program); a JS program's `ctx.fs.mkdir` makes parents itself. The kernel makes the system `PNPM_HOME` (`/usr/local/share/pnpm`, or the embedder's) when it starts, since pnpm makes only its `bin` directory there. A full origin storage (`QuotaExceededError`, which also comes early in a browser's private or off-the-record contexts, whatever `navigator.storage.estimate()` says) is `ENOSPC`, never retried. A read that meets `NotFoundError` while a writer outside the kernel swaps the file in with `createWritable` (and a `stat`, `lstat` or `utimes` that meets it) looks the name up again, up to three times, before it fails ([#1075](https://github.com/ai-ecoverse/slicc-kernel/issues/1075)), so a program whose package is overwritten in place keeps starting. An OPFS or IndexedDB failure that has no POSIX counterpart is `EIO`; its message names the DOM error and the operation (`lookup`, `read`, `write`, `commit` or `sidecar`), and the kernel worker logs the first of each kind to its console. File contents are buffered per open file and written back on close, `fsync` and exit, so a refused write (`ENOSPC`, `EIO`) surfaces at that close or `fsync`, and a program that ignores close errors doesn't see it (`echo x > f` can exit 0 with `f` empty, as with any buffered write-back). A write-back error a close has already reported to a WASI program isn't raised again when it exits, so the program keeps its own exit status. Metadata operations (`mkdir`, `rename`, `rm`, …) go straight to OPFS. The kernel worker is the only writer.
 
-**Renaming a directory on the OPFS root** ([#1122](https://github.com/ai-ecoverse/slicc-kernel/issues/1122)): Chrome's OPFS can move a file but not a directory. So the kernel renames a directory by creating the new tree and moving each file into it (copying a file only if it can't move, e.g. while it's open elsewhere), then removing the old tree. Within the kernel the rename is atomic to readers: a lookup or listing under the old or the new name, or of either parent, waits until the whole tree has moved. So a stage-then-rename updater (pnpm's, an atomic config write) is never seen half there. Another tab sharing the root reads OPFS directly and can see the tree mid-move. A root whose directories can move (the Node memory root) renames them in one step. **If the page reloads or crashes mid-rename**, some files are already in the new tree and the rest are still in the old one. So before the first move, the kernel writes the rename's intent (`{ from, to }`) to its metadata store, and clears it after the last. At the next boot, when no other tab shares the root (the same rule as for the boot's other repair of stale metadata), the kernel finishes every recorded rename: it moves what's left into the new tree, removes the old name, and clears the record. The moved directory and its files keep their owner, mode and inode, as after a rename that wasn't stopped. While another tab shares the root, a kernel finishes only renames whose kernel is gone ([#1141](https://github.com/ai-ecoverse/slicc-kernel/issues/1141)): the renaming kernel holds a Web Lock (`slicc-rename:<id>`, named in the record) for the whole rename, and the browser releases it when that tab closes or its worker ends. A kernel that boots beside a peer, and every kernel that sees a peer go, finishes each recorded rename whose lock is free, taking that lock while it does, so two survivors never replay the same rename and a live kernel's rename is never touched. A record without a lock (written by an older kernel, or where Web Locks are missing) waits for a boot that's alone, as before. While another tab finishes a rename, this kernel's readers wait as for its own; the other tabs can see the tree mid-move, as during any rename.
+**Renaming a directory on the OPFS root** ([#1122](https://github.com/ai-ecoverse/slicc-kernel/issues/1122)): Chrome's OPFS can move a file but not a directory. So the kernel renames a directory by creating the new tree and moving each file into it (copying a file only if it can't move, e.g. while it's open elsewhere), then removing the old tree. Within the kernel the rename is atomic to readers: a lookup or listing under the old or the new name, or of either parent, waits until the whole tree has moved. So a stage-then-rename updater (pnpm's, an atomic config write) is never seen half there. Another tab sharing the root sees it all at once too (see below). A root whose directories can move (the Node memory root) renames them in one step. **If the page reloads or crashes mid-rename**, some files are already in the new tree and the rest are still in the old one. So before the first move, the kernel writes the rename's intent (`{ from, to }`) to its metadata store, and clears it after the last. At the next boot, when no other tab shares the root (the same rule as for the boot's other repair of stale metadata), the kernel finishes every recorded rename: it moves what's left into the new tree, removes the old name, and clears the record. The moved directory and its files keep their owner, mode and inode, as after a rename that wasn't stopped. While another tab shares the root, a kernel finishes only renames whose kernel is gone ([#1141](https://github.com/ai-ecoverse/slicc-kernel/issues/1141)): the renaming kernel holds a Web Lock (`slicc-rename:<id>`, named in the record) for the whole rename, and the browser releases it when that tab closes or its worker ends. A kernel that boots beside a peer, and every kernel that sees a peer go, finishes each recorded rename whose lock is free, taking that lock while it does, so two survivors never replay the same rename and a live kernel's rename is never touched. A record without a lock (written by an older kernel, or where Web Locks are missing) waits for a boot that's alone, as before. While another tab finishes a rename, this kernel's readers wait as for its own, and so do the other tabs'. A kernel whose wait for another tab's rename ends with the lock free and the record still there finishes the rename itself, under that lock, and a boot that's alone takes a record's lock too, so a third tab that boots meanwhile waits for it.
+
+**Directory renames across tabs** ([#1140](https://github.com/ai-ecoverse/slicc-kernel/issues/1140)): a tab sharing the root sees a directory rename in another tab all at once, as the renaming kernel's own readers do.
+- **The notice:** before the first file moves, the renaming kernel tells the other kernels on the root, and waits until each has answered. A kernel answers once the calls it has in flight under either name, or on either parent, are done (a file rename into or out of the tree included, a symlink by the link's own path, calls that began before the other kernel joined too, and calls through a symlink into the tree, and an open write session there counts until it's closed). From then on, its calls under either name, and its listings of either parent, wait until the rename's Web Lock is released. So a listing in the other tab shows the old tree or the new one, never a mix, and every write that returned success there is in the new tree.
+- **A missed notice:** a kernel also checks the held Web Locks while it shares the root, and reads the recorded renames when it sees a rename lock it doesn't know. Only a call starts a check: it runs in the background once the last answer is 50 ms old, and a call waits for it only when that answer is 100 ms old. An idle tab makes no checks. A rename lock whose record isn't written yet holds the call until the record appears or the lock is released. A call through a symlink whose target is in the renaming tree waits as one under the tree does. So a kernel that missed the notice, or joined the root in the middle of a rename, trusts "no rename in flight" for at most 100 ms.
+- **Overlapping renames:** of two kernels renaming overlapping trees at once (`/t` and `/t/a`), one goes first and the other retries against what is there by then, so it may fail with `ENOENT`. Renames of unrelated trees, siblings like `/t/a` and `/t/ab` included, don't wait for each other.
+- **A stuck peer:** a peer whose tab is alive (its tab's Web Lock is held) is waited for up to 10 s. If it hasn't answered by then, the rename fails with `EBUSY` and nothing has moved. The kernel logs one line naming the peer and the tree, and `/proc/sys/kernel/slicc_rename_ack_busy` counts it. A peer that's gone isn't waited for.
+- **Counters:** `/proc/sys/kernel/slicc_rename_acks` reads five numbers: the renames that asked, the median and the longest wait in ms, the times this kernel waited for another kernel's rename, and the lock checks it made.
+- **A tab that hasn't finished joining:** a kernel that's alone checks the root's held tab locks once before it renames a directory. If another tab holds one but hasn't answered yet (it answers once its own exclusive creates are done), the rename waits for it to join, up to 10 s, and then fails with `EBUSY` with nothing moved.
+- **Alone on the root,** nothing else runs: no notice, no ack, and a call costs one counter.
+- **Limits:**
+  - A kernel without `/proc/sys/kernel/slicc_rename_acks` (older than #1140) doesn't answer and isn't waited for ([#1309](https://github.com/ai-ecoverse/slicc-kernel/issues/1309)). It can still see a tree mid-move, and a write it has in flight when the rename starts can land in the old copy and be lost.
+  - An open descriptor doesn't follow a rename of its directory by another process, in this tab or another ([#1257](https://github.com/ai-ecoverse/slicc-kernel/issues/1257)). Its next write fails with `ENOENT`, and nothing is lost silently.
+  - A peer's call that is already in flight and then reaches into the renaming tree through a call of its own (a rename removing the name it replaces) waits for the rename, while the rename waits for that peer's answer. After the 10 s cap the rename fails with `EBUSY`, nothing has moved, and the peer's call goes on.
 
 A lookup follows at most 40 symlinks, as Linux does: the 41st is `ELOOP`, for WASI programs, JS programs and the kernel API alike, with or without `..` in a link's target. Emscripten programs resolve paths in their glue, which stops one link earlier, at 40.