@ai-ecoverse/slicc-kernel 1.172.1 → 1.173.0

package/README.md

@@ -25,7 +25,7 @@ TypeScript declarations ship in `dist/types/`, and each entry (`.`, `./node`, `.
 
 | option | default | |
 | --- | --- | --- |
-| `root` | `navigator.storage.getDirectory()` | the `FileSystemDirectoryHandle` that becomes `/` |
+| `root` | `navigator.storage.getDirectory()` | the `FileSystemDirectoryHandle` that becomes `/`. WebKit cannot clone a handle, so there it must lie in OPFS, and the kernel worker reopens it by path |
 | `modules` | `'/node_modules'` | where installed packages are scanned for commands |
 | `commands` | none | `{ append: ['/opt/user/node_modules'] }`: more `node_modules` directories scanned for commands after all the others, so they never shadow one (see [Commands](#commands)) |
 | `env` | `{}` | environment added to every run |
@@ -181,7 +181,7 @@ Attaches to a kernel over a port from `kernel.connect()`, in any realm: a worker
 | `kill(pid, signal?)` | a process, or a process group with a negative pid; no such process rejects with `code: 'ESRCH'`, and one the client's user may not signal with `code: 'EPERM'` |
 | `fs` | `readFile`, `readText`, `writeFile(path, string \| bytes)`, `stat`, `lstat`, `readdir`, `mkdir` (with parents), `rm(path, { force })` (recursive), `rename`, `realpath`, `symlink(target, path)`, `readlink`, `exists`, on the processes' file system, mounts included (a path under a `tmpfs`, `fsa` or `hostfs` mount reads and writes the mounted file system, as `cat` in a process would); failures reject with `KernelCallError` and a POSIX `code` |
 | `fetch({ url, method, headers, body, redirect, signal })` | through the kernel's network transport (the one the page passed to `createKernel`), with a streaming body; `redirect` is the per-request policy (see [Network](#network)); `transport` is the same as a `NetworkTransport` |
-| `fs.watch(paths, { recursive }, onChange)` | resolves with `{ close() }`; `onChange` gets `{ paths }`, the changed paths at, below (or, without `recursive`, directly in) the watched ones, batched per task, or `{ overflow: true }` when there were too many to list, to rescan; it covers every change made through this kernel, by its processes and its clients, and a removal or rename that failed or found nothing to do is none |
+| `fs.watch(paths, { recursive }, onChange)` | resolves with `{ close() }`; `onChange` gets `{ paths }`, the changed paths at, below (or, without `recursive`, directly in) the watched ones, batched per task, or `{ overflow: true }` when there were too many to list, to rescan; it covers every change made through this kernel, by its processes and its clients, and a removal or rename that failed or found nothing to do is none. It also covers what another kernel on the same OPFS root changes (another tab, a worker page) and what a writer outside the kernel reports with [`postFsNotice`](#metadata), as those notices name them: a directory another kernel creates, removes or renames, the file it writes back, and the records of its symlinks, modes, owners and times, and a notice's `trees` as their roots. A peer's notices come only once the kernels share the root (`/proc/sys/kernel/slicc_shared_root` reads `1` from the join handshake on) and only with the shared metadata (not with `metadata: false`), and a writer outside the kernel that posts no notice is never heard. So a client that has to catch a page writing OPFS directly needs that page to call `postFsNotice` after each write. Kernels before this one didn't pass peer and outside changes to a client's watch; they reached only `ctx.fs.watch`. A watch is on names, not files: a watched path need not exist, nor its directories, and one under a regular file is a watch too (every kernel with `watch` does this, protocol 1.1 on), so its creation, by any `mkdir -p` above it, a write or a rename to it, is reported as its path, and so are a removal or rename of it or of a directory above it; the watch goes on hearing the name after that. So a client that waits for a directory to appear watches it, rather than polling `stat` and `readdir`. Under `users`, a watched path the client may not read is `EACCES`, and so is a missing one under a directory the client may not search. Changes are reported by the name they were made under (a file written through a symlink also by its real path), so a client that watches a path under a symlink watches its real path too, for a missing path the real path of its nearest existing directory with the rest of the path after it |
 | `serveCdp(hook, { runtime })` | offers this client's browser to the kernel's programs (protocol 1.5): `hook` is the `cdp` hook of `createKernel`, asked once per program connection; resolves with `{ close() }`, and closing or detaching takes the offer back and closes its connections (see [Browser automation](#browser-automation-cdp)) |
 | `serveFacade(name, hook) → { close() }` | serves a facade that `createKernel({ facades })` declared, the same way as its `hook` (protocol 1.11); the latest registration wins, over the embedder's hook too, and `close()` hands it back. An undeclared name is `EINVAL` |
 | `onTraits(listener) → () => void` | calls `listener(traits)` each time the kernel announces new transport traits (`kernel.updateTransportTraits()`, protocol 1.10), after `client.transport.traits` has been updated; the function it returns removes the listener |
@@ -390,7 +390,7 @@ The module's default export (or its `main` export) is called with a context and
   - **Who counts as a writer:** every process in the kernel, the embedder (`kernel.writeFile` and the like) and the mounts that report changes. A `hostfs` change from the host comes as `'change'`, since the host doesn't say which kind it was.
   - **Another tab:** a kernel sharing the OPFS root reports its changes too ([#1391](https://github.com/ai-ecoverse/slicc-kernel/issues/1391)): a structural change (create, remove, rename, mkdir) as `'rename'`, a content or metadata change as `'change'`. Their order across a name isn't guaranteed, and a new directory may also come with a `'change'`, so a watcher should treat either as "look again". Another tab's change never ends or renames a watch, and a directory watch doesn't hear one at the directory itself; a file watch hears another tab's write as `'change'`.
   - **Lifetime:** a watch ends with its process and doesn't keep the program alive.
-  - **The older watches** (`kernel.watch`, a client's `fs.watch` with `{ paths }`) are unchanged.
+  - **The older watches:** `kernel.watch` is unchanged, and a client's `fs.watch` with `{ paths }` hears another tab's changes too (see [`attachKernel`](#attachkernelport-options--promisekernelclient)).
 - `/dev`, in `open`, `fs` and their `sync` forms, as a WASI or Emscripten program sees it, with the same modes and owners as under `users`: `null`, `zero`, `full`, `random`, `urandom` and `tty` (character devices, `0666`, root's), `stdin`, `stdout` and `stderr` (links to `/proc/self/fd/0`, `1`, `2`), `fd` (a link to `/proc/self/fd`, so `/dev/fd/N` is descriptor N) and the kernel's terminals `/dev/ttyN` and `/dev/pts/N` (with users on, owned by the session's user or the master's opener, `0620`). `/dev/shm` is the kernel's shared tmpfs (see [Mounts](#mounts)), read and written like any other mount; `/dev/ptmx` isn't there. `stat` through `/dev/stdin`, `/dev/stdout` or `/dev/fd/N` describes the program's own descriptor (the file it is redirected to, a pipe, a terminal), `readlink` of `/dev/fd/N` names it (`pipe:[N]` and `socket:[N]` for a pipe or a socket), and a closed descriptor is `ENOENT`. Opening a device gives a kernel descriptor on it; opening `/dev/std*` or `/dev/fd/N` opens the file again with the flags asked for when the descriptor is on a file (as Linux reopens `/proc/self/fd/N`, so `append` appends and `truncate` truncates), and otherwise duplicates the descriptor; `/dev/tty` is the program's controlling terminal (`ENXIO` without one); and `/dev/ttyN` and `/dev/pts/N` are opened by name through the kernel, so a terminal that isn't the user's is `EACCES` under `enforce`. A file on anything but a regular file reads and writes as a stream (the position is ignored) and only in the directions it was opened for (`EBADF` otherwise), its `size()` is 0 and `truncate` is `EINVAL`. `readFile` reads a device or a descriptor to its end, up to 64 MiB, and fails with `EFBIG` past that (where Node's `readFile` throws `ERR_FS_FILE_TOO_LARGE`), so `readFile('/dev/zero')` fails instead of hanging, while a bounded `read` of it works. Other paths pay only a prefix check: they make the same kernel calls as before. Removing a `/dev` entry fails (see [`/dev`](#dev-entries)), and so does a new name directly in `/dev` or `/proc`;
 - `/proc`, in `fs`, `open` and their `sync` forms, the same as a WASI program's (see [`/proc`](#proc)): `stat`, `lstat`, `exists`, `readdir`, `readlink` and `readFile` answer it, and `/proc/self` is the program's own pid. It's read-only, so every change in it is `EROFS`: `writeFile`, `mkdir`, `rm`, `unlink`, `chmod`, a `rename`, `link` or `symlink` into or out of it, and an `open` for writing, creating or truncating. A file opened in it is a snapshot read at the open, with no descriptor (`fd` is `-1`): its `read` and `size` answer from the snapshot until it's closed, and `write` and `truncate` are `EBADF`. Other paths pay only a prefix check;
 - `signals`: `on(signal, handler)` catches a signal (by number or as `'SIGINT'`), `ignore(signal)` ignores it and `reset(signal)` restores its default action. A call blocked in a read or write is restarted after a handler runs; signals without a handler keep their default actions, so an uncaught `SIGINT` ends the program with 130 and `SIGKILL` always does;
@@ -916,6 +916,7 @@ An `fsa` mount is a drive for a local folder, from the File System Access API, a
 - **Insert request:** at the same time the kernel calls the page's `onMountPending({ target, source, insert })`, so the page can show "Insert a folder for /mnt/x" with a button.
 - **Inserting:** calling `insert()` from that button's click picks the folder with `requestDirectory()`, or asks for permission again on a folder the drive had before. The medium goes in without a remount, and the state becomes `ok`. `kernel.insert(target, handle)` inserts a handle the page already has.
 - **Persistence:** the drive's source is `fsa:<id>`, and the kernel keeps the folder's handle under that id in IndexedDB (`<metadata>:media`). Mounting `fsa:<id>` again, after a reboot too, inserts the folder at once while its permission holds, and asks again when it is back to `prompt`. In an Incognito window Chrome crashes a page that reads a folder handle back from IndexedDB ([#95](https://github.com/ai-ecoverse/slicc-kernel/issues/95)). Pass `media: false` there: the kernel then never opens the handle database, and keeps the handles in memory, so a remount finds its folder only within the same kernel, while POSIX metadata still persists. `metadata: false` keeps the handles in memory too.
+- **WebKit:** it has no folder picker and cannot clone a handle or keep one in IndexedDB, so a folder must lie in OPFS. The page sends its path, the kernel worker reopens it, and the media store keeps the path.
 - **Ejecting:** `umount` ejects. So does the permission going away: the next operation fails with `ENOMEDIUM` and the kernel asks again. The folder is untouched either way.
 - **Limits:** the folder has no symlinks or modes, and `df` reports no size for it either.
 
@@ -1249,7 +1250,7 @@ In a browser every process worker of the page shares one renderer. V8 reserves a
 - **Below a learned ceiling,** a process that still runs out of memory is started again once another process exits, and the spawn fails with `EAGAIN` after 10 s with none.
 - **`/proc/sys/kernel/slicc_wasm_memories`** reads `<live> <ceiling> <source>`. The source is `default`, `learned`, or `none` (with the ceiling 0).
 
-`/proc/sys/kernel/slicc_shared_root` reads `1` while another kernel shares the root (a tab, or a second kernel in the page) and `0` otherwise. A peer whose kernel ends (its tab closes or reloads) stops counting at once: each kernel queues a request for every peer's alive Web Lock, and the grant tells it the peer has gone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)). `/proc/sys/kernel/slicc_fs_calls` reads the number of path calls (lookups, stats, opens, creates, removals and the like) the kernel has served its programs since it started. `/proc/sys/kernel/slicc_opfs_calls` counts the OPFS calls the kernel has made to look names up and read stats (`getFile`, `getFileHandle`, `getDirectoryHandle`, `resolve`). `/proc/sys/kernel/slicc_opfs_ops` counts every request the root file system makes of OPFS: those lookups, plus creates, writes (one per write, truncate or write session), reads, listings, renames and removals ([#1017](https://github.com/ai-ecoverse/slicc-kernel/issues/1017)). With an idle peer, a tar unpack makes at most 1.5 times the calls it makes alone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)). `/proc/sys/kernel/slicc_cwd_checks` counts the times a process asked the kernel whether its [working directory](#the-working-directory) is still the one it holds.
+`/proc/sys/kernel/slicc_shared_root` reads `1` while another kernel shares the root (a tab, or a second kernel in the page) and `0` otherwise. A peer whose kernel ends (its tab closes or reloads) stops counting at once: each kernel queues a request for every peer's alive Web Lock, and the grant tells it the peer has gone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)). `/proc/sys/kernel/slicc_fs_calls` reads the number of path calls (lookups, stats, opens, creates, removals and the like) the kernel has served its programs since it started. `/proc/sys/kernel/slicc_opfs_calls` counts the OPFS calls the kernel has made to look names up and read stats (`getFile`, `getFileHandle`, `getDirectoryHandle`, `resolve`). `/proc/sys/kernel/slicc_opfs_ops` counts every request the root file system makes of OPFS: those lookups, plus creates, writes (one per write, truncate or write session), reads, listings, renames and removals ([#1017](https://github.com/ai-ecoverse/slicc-kernel/issues/1017)). `/proc/sys/kernel/slicc_meta_transactions` reads two numbers: the read and the write transactions the kernel has opened on its metadata database (IndexedDB) since it started; it reads `0 0` with `metadata: false`. With an idle peer, a tar unpack makes at most 1.5 times the calls it makes alone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)). `/proc/sys/kernel/slicc_cwd_checks` counts the times a process asked the kernel whether its [working directory](#the-working-directory) is still the one it holds.
 
 ### Metadata
 
@@ -1297,7 +1298,13 @@ Reads retry briefly when a concurrent write has invalidated the OPFS file snapsh
 
 At boot, a kernel drops sidecar records whose file is gone (and symlink records whose name became a real file or whose directory is gone), left behind by a tab that closed mid-change. It does this only when it is the only kernel on the origin (the watch channel's join handshake finds no other), and it drops a record only if it is still the one it found: another tab's live changes made while it looked, such as a placeholder tar replaces with a symlink, are kept. Up to 1.119.2, a tab that booted while another extracted symlinks with tar could drop some of them.
 
-A kernel caches the directory handles it has walked and the sidecar entries it has read. Its own changes keep both caches current, and the other kernels on the origin hear of them through a `BroadcastChannel` named after the database (one for the sidecar, one for directories). A cached directory is checked against its path (`resolve()`) at most once a second before a lookup goes through it, so a directory moved, removed or replaced by another writer, such as a page using the OPFS API, is found at its new place or reported missing within a second. It also keeps the file handles it has looked up (up to 4,096, the least recently used dropped first) for a second after each lookup. Before it uses one, it reads the file through it, which is the read a `stat` needs anyway, and that read serves the handle's uses in the kernel's next 64 lookups, for at most 100 ms ([#1425](https://github.com/ai-ecoverse/slicc-kernel/issues/1425)), so the count doesn't depend on how fast the machine is. So a file another writer removed is missing, and a rewritten one shows its new contents, within 100 ms. The kernel's own writes, truncates, rewrites and renames drop the read at once, and a read that started before an own change isn't kept, so the kernel's own changes always show at once. One replaced by a directory is seen as a directory within a second. Its own changes, and the other kernels' notices, drop the handles they touch at once. A lookup of a directory it has cached and checked within the second is answered from the cache, so a directory replaced by another writer is seen as replaced within a second, as above. A name it has just found missing stays missing for the kernel's next 256 lookups, for at most 100 ms, which serves the several checks a single create makes of a name that isn't there yet (about ten per file in a pnpm install, up to a few hundred lookups apart); its own creates, renames and links in that directory end that at once, so only a file another kernel or a page using the OPFS API makes in that window is seen up to 100 ms late. A listing of the directory (`ls`, a glob, `readdir`) ends it too for every name it shows, so `ls` and then `cat` agree ([#1693](https://github.com/ai-ecoverse/slicc-kernel/issues/1693)). A writer outside the kernel whose file must open in the very next run, such as a page that writes a file and then calls `kernel.run`, posts a notice for it (`postFsNotice`, below) or waits 100 ms. While another kernel shares the root, both short-lived windows follow the contract below. From [#945](https://github.com/ai-ecoverse/slicc-kernel/issues/945) until [#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984) they were off then, which made every stat and every miss an OPFS call.
+A kernel caches the directory handles it has walked and the sidecar entries it has read, and, per directory, the symlinks the sidecar holds there, which every listing of the directory needs. Its own changes keep these caches current, and the other kernels on the origin hear of them through a `BroadcastChannel` named after the database (one for the sidecar, one for directories). For the sidecar caches the rules are:
+- **What drops an entry or a directory's symlinks:** each change to the sidecar, this kernel's own once its transaction has committed and another kernel's (or `writeMetadata`'s) when its notice arrives, drops the entries it names and the symlinks cached for their parent directories; a removal or a rename drops everything under the names too, and the symlinks of every directory there. A notice is posted only after its transaction has committed, so a read that starts once it has arrived sees the change.
+- **A read that a change overtook:** an answer is kept only if no change to its name, to a directory above it, or (for a directory's symlinks) to a name in that directory was heard while it was being read. A change to another name doesn't cost it its place, so a kernel that writes all the time still keeps what it reads. Before, any change anywhere threw away every read in flight, and listings weren't cached at all: an agent's polling file watchers drove one kernel to about 3,000 metadata transactions a second, and Chrome's browser process ran out of memory under them.
+- **Reads share transactions:** reads of names and listings that aren't cached yet go to the database in at most 4 read transactions at a time. Any asked while 4 are open wait, and go together in one when a transaction finishes; a read already on its way for the same name is shared. So a storm of polls costs nothing once the answers are cached, and a burst of new names costs a few transactions, not one each.
+- **The residual window** is the one of every notice: between another kernel's commit and the arrival of its notice, this kernel can answer from what it had, by at most a message's delivery. A lock grant doesn't drop the sidecar caches: a reader under a lock relies on the writer's notice, which the writer posts before its change returns. `meta-listings-page.test.mjs` holds the rows, and counts the transactions through `slicc_meta_transactions`.
+
+The OPFS side: a cached directory is checked against its path (`resolve()`) at most once a second before a lookup goes through it, so a directory moved, removed or replaced by another writer, such as a page using the OPFS API, is found at its new place or reported missing within a second. It also keeps the file handles it has looked up (up to 4,096, the least recently used dropped first) for a second after each lookup. Before it uses one, it reads the file through it, which is the read a `stat` needs anyway, and that read serves the handle's uses in the kernel's next 64 lookups, for at most 100 ms ([#1425](https://github.com/ai-ecoverse/slicc-kernel/issues/1425)), so the count doesn't depend on how fast the machine is. So a file another writer removed is missing, and a rewritten one shows its new contents, within 100 ms. The kernel's own writes, truncates, rewrites and renames drop the read at once, and a read that started before an own change isn't kept, so the kernel's own changes always show at once. One replaced by a directory is seen as a directory within a second. Its own changes, and the other kernels' notices, drop the handles they touch at once. A lookup of a directory it has cached and checked within the second is answered from the cache, so a directory replaced by another writer is seen as replaced within a second, as above. A name it has just found missing stays missing for the kernel's next 256 lookups, for at most 100 ms, which serves the several checks a single create makes of a name that isn't there yet (about ten per file in a pnpm install, up to a few hundred lookups apart); its own creates, renames and links in that directory end that at once, so only a file another kernel or a page using the OPFS API makes in that window is seen up to 100 ms late. A listing of the directory (`ls`, a glob, `readdir`) ends it too for every name it shows, so `ls` and then `cat` agree ([#1693](https://github.com/ai-ecoverse/slicc-kernel/issues/1693)). A writer outside the kernel whose file must open in the very next run, such as a page that writes a file and then calls `kernel.run`, posts a notice for it (`postFsNotice`, below) or waits 100 ms. While another kernel shares the root, both short-lived windows follow the contract below. From [#945](https://github.com/ai-ecoverse/slicc-kernel/issues/945) until [#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984) they were off then, which made every stat and every miss an OPFS call.
 
 **What another tab sees, and when** ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)): one rule covers everything a kernel caches about a shared root.
 - **Notices:** while another kernel shares the root, a kernel posts a notice before each changing call returns. A notice names the sidecar records it changed (owners, modes, links, times, write counts), the directories it created, removed or renamed into, and the files it wrote back ([#962](https://github.com/ai-ecoverse/slicc-kernel/issues/962)).

package/dist/index.js

@@ -546,6 +546,32 @@ function integrityUnknown(options) {
   return unknown;
 }
 
+// src/fs/portable.ts
+var privateTop = () => navigator.storage.getDirectory();
+function clonable(handle) {
+  const { port1, port2 } = new MessageChannel();
+  try {
+    port1.postMessage(handle);
+    return true;
+  } catch (err) {
+    if (err?.name === "DataCloneError") return false;
+    throw err;
+  } finally {
+    port1.close();
+    port2.close();
+  }
+}
+async function portable(handle, top = privateTop) {
+  if (clonable(handle)) return { handle };
+  const path = await (await top()).resolve(handle);
+  if (path === null) {
+    throw new Error(
+      "this browser cannot hand a folder to a worker unless it is in the origin private file system"
+    );
+  }
+  return { path };
+}
+
 // src/integrity-relay.ts
 function integrityAnswerer(port, lookup) {
   const ask = async ({ id, path }) => {
@@ -2448,11 +2474,17 @@ function uplinkInit(uplink) {
   const nameserver = typeof uplink.nameserver === "string" ? { nameserver: uplink.nameserver } : {};
   return { uplink: { traits, ...uplink.routes ? { routes: uplink.routes } : {}, ...nameserver } };
 }
-function initCall(options, transport) {
+function rootField(at) {
+  return "path" in at ? { rootPath: at.path } : { root: at.handle };
+}
+async function insertCall(target, handle, source) {
+  return { op: "insert", target, source, ...await portable(handle) };
+}
+async function initCall(options, transport) {
   const hooks = sealedHooks(options.sealed);
   return {
     op: "init",
-    ...options.root ? { root: options.root } : {},
+    ...options.root ? rootField(await portable(options.root)) : {},
     ...options.modules ? { modules: options.modules } : {},
     ...options.commands ? { commands: options.commands } : {},
     ...options.env ? { env: options.env } : {},
@@ -2748,7 +2780,7 @@ async function createKernel(options = {}) {
   const pendingMedium = ({ target, source, handle }) => options.onMountPending?.({
     target,
     source,
-    insert: async () => void await call({ op: "insert", target, source, handle: await chosen(handle) })
+    insert: async () => void await call(await insertCall(target, await chosen(handle), source))
   });
   const aside = (data) => {
     if (files.event(data)) return true;
@@ -2821,7 +2853,7 @@ async function createKernel(options = {}) {
     };
   }
   const { dial } = dials;
-  await call(initCall(options, transport));
+  await call(await initCall(options, transport));
   return {
     async run(argv, runOptions = {}) {
       const { onStdout, onStderr, stdin, ...rest } = runOptions;
@@ -2847,7 +2879,7 @@ async function createKernel(options = {}) {
     mount: async (spec) => await call({ op: "mount", spec }),
     umount: async (target) => void await call({ op: "umount", target }),
     mounts: async () => await call({ op: "mounts" }),
-    insert: async (target, handle) => void await call({ op: "insert", target, handle }),
+    insert: async (target, handle) => void await call(await insertCall(target, handle)),
     ...pageOnly(call),
     watch: files.watch,
     terminate() {