@ai-ecoverse/slicc-kernel 1.119.1 → 1.120.0

package/README.md

@@ -583,7 +583,7 @@ A lookup follows at most 40 symlinks, as Linux does: the 41st is `ELOOP`, for WA
 Directories are renamed with `FileSystemHandle.move()` where available, else by copy and delete.
 
 **FIFOs** (`mkfifo`, #896):
-- **Creating one:** an Emscripten program's `mkfifo`/`mknod(S_IFIFO)` makes a FIFO node on the OPFS root (in `/tmp`, `/home` and the rest), owned by the caller, with mode `mode & ~umask`. `stat`, `ls -l` (`p`) and `test -p` show it as a FIFO, in Emscripten, WASI and JS programs. WASIX libc has no `mkfifo`, but WASIX programs `stat`, `open`, read and write a FIFO that exists. An existing name is `EEXIST`. tmpfs mounts (`/dev/shm`, a tmpfs `/tmp`) have FIFO nodes too, through the driver's `fifos` capability ([#907](https://github.com/ai-ecoverse/slicc-kernel/issues/907)). hostfs, `fsa` and media mounts have none, so `mkfifo` there is `EPERM`, and `EROFS` on a readonly path.
+- **Creating one:** an Emscripten program's `mkfifo`/`mknod(S_IFIFO)` makes a FIFO node on the OPFS root (in `/tmp`, `/home` and the rest), owned by the caller, with mode `mode & ~umask`. `stat`, `ls -l` (`p`) and `test -p` show it as a FIFO, in Emscripten, WASI and JS programs. A WASIX program makes one through `slicc_fs.path_mkfifo` (see [File modes and the umask](#file-modes-and-the-umask); its libc uses it from wasix-sysroot -33), and `stat`s, `open`s, reads and writes one like any program. An existing name is `EEXIST`. tmpfs mounts (`/dev/shm`, a tmpfs `/tmp`) have FIFO nodes too, through the driver's `fifos` capability ([#907](https://github.com/ai-ecoverse/slicc-kernel/issues/907)). hostfs, `fsa` and media mounts have none, so `mkfifo` there is `EPERM`, and `EROFS` on a readonly path.
 - **Opening one** works as on Linux (fifo(7)). A read-open waits until a writer opens, and a write-open until a reader does. `O_RDWR` doesn't wait. With `O_NONBLOCK` a read-open returns at once, and a write-open with no reader is `ENXIO`. A waiting open is interrupted by a caught signal (`EINTR`), restarts under `SA_RESTART`, and isn't ended by a stop.
 - **The data path** is the kernel's pipe: the same capacity, `EOF` once the last writer closes, and `SIGPIPE`/`EPIPE` once the last reader closes. The data is dropped when no end is open.
 - **The node's identity, not its name, picks the FIFO:** an open FIFO keeps working after an `unlink` or a `rename`, and a new `mkfifo` at the same name is another FIFO. `/proc/<pid>/fd/N` names the FIFO's path, also in children that inherit the end and in a WASIX program's `readlink("/proc/self/fd/N")` of an end it opened itself ([#909](https://github.com/ai-ecoverse/slicc-kernel/issues/909)).
@@ -613,7 +613,7 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
 - **Exclusive create across tabs** ([#912](https://github.com/ai-ecoverse/slicc-kernel/issues/912)): two kernels on one OPFS root (two tabs, or a tab and a worker page) agree on `O_CREAT|O_EXCL` and `set -C`: exactly one wins a name. While another kernel on the root is present, the create runs under a Web Lock for its path and checks the OPFS entry itself. A kernel alone on its root takes no lock. Each kernel holds an alive Web Lock, granted before it first looks for peers. A kernel that finds peers waits for their answer (at most 500 ms) before its first exclusive create, and a kernel answers a newcomer only once its unlocked creates are done. So of two kernels that boot at the same moment, the later one to look sees the other, and no unlocked create overlaps a locked one.
 - **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. The same file under two names is one file too ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)): a write by path through a symlink reaches the file the kernel holds under its real path (resolved without following links in a `sudo` process's no-links view, so an untrusted link is still refused), a WASI program's open file also watches its real path (resolved once, the first time another file changes), and an open file with more than one hard link checks its size, version and mtime when anything changed, so a write through another name is seen. Limits for now: an alias on a mount that has no `realpath` is still only the name it was opened by; another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them. A by-path `stat` or `lstat` of a file the kernel holds open has its current size, writes not yet written back included ([#780](https://github.com/ai-ecoverse/slicc-kernel/issues/780)). Its mtime is the file system's until write-back, though, so a tool that compares times, like `make`, may see a just-written file as older than it is for up to the write-back delay.
 - **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, and so are a WASIX program's through the `slicc.fcntl_lock` import (below), so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` keeps its behaviour (whole file, per process, keyed by path). Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program built against a wasix-sysroot without the import keeps its libc's local answer (every lock granted), so python's sqlite excludes other processes only once wasix-python is rebuilt on the sysroot that has it ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)). An Emscripten program's `flock()` is a stub compiled into its libc that never reaches the kernel either, until a homescoop shim routes it to a kernel import ([#680](https://github.com/ai-ecoverse/slicc-kernel/issues/680)). A lock belongs to the file its descriptor was opened on: a file renamed over the name later, or the name's removal, doesn't move it.
-- **Locks across tabs** ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)): two kernels on one OPFS root (two tabs, or a tab and a worker page) share one table of `fcntl` record locks and `flock` locks, so a sqlite3 in one tab finds the database locked while a transaction is open in another.
+- **Locks across tabs** ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)): two kernels on one OPFS root (two tabs, or a tab and a worker page) share one table of `fcntl` record locks and `flock` locks, so a sqlite3 in one tab finds the database locked while a transaction is open in another. **Known issue, data corruption: lock-protected writes across tabs** ([#949](https://github.com/ai-ecoverse/slicc-kernel/issues/949)). The locks are enforced across tabs, but what one tab writes under a lock isn't yet guaranteed to be what the other tab reads once it holds the lock, so a lock-protected read-modify-write across tabs can lose or corrupt data. Don't rely on cross-tab locking until #949 is fixed. Use within a single tab is unaffected.
   - **One leader:** the first kernel on a root takes a Web Lock and serves the table from its kernel worker. A kernel alone on its root is its own leader, and its lock calls cost what they did. Another kernel's lock calls go to the leader over a BroadcastChannel: about 0.1 ms a call against 0.02 ms on the leader, measured with the leader's tab in the background.
   - **Blocking and deadlocks:** `F_SETLKW` and a blocking `flock` wait at the leader. A signal ends the wait with `EINTR`. `EDEADLK` sees the whole cycle, whichever tabs its processes run in.
   - **Kernels that go away:** each kernel holds a Web Lock of its own for as long as it runs, so a closed or crashed tab's locks are dropped and its waiters woken within about a second.
@@ -734,6 +734,7 @@ WASI and WASIX have no call for file modes, so the kernel adds the import module
 | `path_owner(dirfd, path, path_len, flags, out)` | those of a path, of the link itself with `flags & 1` |
 | `fd_filestat_owner(fd, filestat_out, out)` | `fd_filestat_get` and `fd_owner` in one call |
 | `path_filestat_owner(dirfd, lookupflags, path, path_len, filestat_out, out)` | `path_filestat_get` (with its preview1 lookup flags: `1` follows a symlink) and the owner of the same path, in one call, so a libc's `stat` costs one call either way |
+| `path_mkfifo(dirfd, path, path_len, mode)` | `mkfifo`/`mkfifoat` and `mknod(S_IFIFO)` (homescoop#878): makes a [FIFO](#filesystem) with `mode & ~umask`, relative to `dirfd` (`AT_FDCWD` is `-2`). It doesn't follow a link in the last component, so an existing name, also a FIFO or a dangling link, is `EEXIST`. A mount without FIFOs answers `EPERM` |
 | `path_access(dirfd, path, path_len, amode, flags)` | `faccessat`: `amode` is `R_OK` 4, `W_OK` 2, `X_OK` 1 or `F_OK` 0; the check uses the real uid and gid, or the effective ones with `flags & 2` (`AT_EACCESS`), and the link itself with `flags & 1`. With `users: 'off'` it checks only that the path exists, as a libc without it does (see [Permissions](#permissions)) |
 
 The owner calls follow [File owners](#file-owners): with `users: 'off'`, `fd_chown` and `path_chown` succeed and change nothing, and the four owner calls answer `ENOTSUP`, "owners not tracked", before looking at anything, so a libc can remember that once and take its own `getuid()` and `getgid()` as every file's owner, which is what an Emscripten program built against the credential calls sees. Devices are root's. A pipe's owner is the caller.
@@ -932,7 +933,7 @@ Not covered: hard links (OPFS cannot share contents between names; they work onl
 
 Reads retry briefly when a concurrent write has invalidated the OPFS file snapshot (`NotReadableError`).
 
-A kernel caches the directory handles it has walked and the sidecar entries it has read. Its own changes keep both caches current, and the other kernels on the origin hear of them through a `BroadcastChannel` named after the database (one for the sidecar, one for directories). A cached directory is checked against its path (`resolve()`) at most once a second before a lookup goes through it, so a directory moved, removed or replaced by another writer, such as a page using the OPFS API, is found at its new place or reported missing within a second. It also keeps the file handles it has looked up (up to 4,096, the least recently used dropped first) for a second after each lookup. Before it uses one, it reads the file through it, which is the read a `stat` needs anyway, and that read serves the handle's uses of the next 2 ms. So a file another writer removed is missing, and a rewritten one shows its new contents, within 2 ms. The kernel's own writes, truncates, rewrites and renames drop the read at once, and a read that started before an own change isn't kept, so the kernel's own changes always show at once. One replaced by a directory is seen as a directory within a second. Its own changes, and the other kernels' notices, drop the handles they touch at once. A lookup of a directory it has cached and checked within the second is answered from the cache, so a directory replaced by another writer is seen as replaced within a second, as above. A name it has just found missing stays missing for 2 ms, which serves the several checks a single create makes of a name that isn't there yet (about ten per file in a pnpm install); its own creates, renames and links in that directory end that at once, so only a file another kernel or a page using the OPFS API makes in those 2 ms is seen up to 2 ms late.
+A kernel caches the directory handles it has walked and the sidecar entries it has read. Its own changes keep both caches current, and the other kernels on the origin hear of them through a `BroadcastChannel` named after the database (one for the sidecar, one for directories). A cached directory is checked against its path (`resolve()`) at most once a second before a lookup goes through it, so a directory moved, removed or replaced by another writer, such as a page using the OPFS API, is found at its new place or reported missing within a second. It also keeps the file handles it has looked up (up to 4,096, the least recently used dropped first) for a second after each lookup. Before it uses one, it reads the file through it, which is the read a `stat` needs anyway, and that read serves the handle's uses of the next 2 ms. So a file another writer removed is missing, and a rewritten one shows its new contents, within 2 ms. The kernel's own writes, truncates, rewrites and renames drop the read at once, and a read that started before an own change isn't kept, so the kernel's own changes always show at once. One replaced by a directory is seen as a directory within a second. Its own changes, and the other kernels' notices, drop the handles they touch at once. A lookup of a directory it has cached and checked within the second is answered from the cache, so a directory replaced by another writer is seen as replaced within a second, as above. A name it has just found missing stays missing for 2 ms, which serves the several checks a single create makes of a name that isn't there yet (about ten per file in a pnpm install); its own creates, renames and links in that directory end that at once, so only a file another kernel or a page using the OPFS API makes in those 2 ms is seen up to 2 ms late. Both 2 ms windows are off while another kernel shares the root (a second tab on the origin, found through the watch channel's join handshake, and until that handshake is done), so between tabs every look is fresh, as before 1.113.0; only a page writing through the OPFS API itself can still be seen up to 2 ms late ([#945](https://github.com/ai-ecoverse/slicc-kernel/issues/945)). A single tab keeps them.
 
 The filesystem is our own rather than [ZenFS](https://github.com/zen-fs/core) (which SLICC uses), because OPFS stays the single source of truth: other writers, such as the BIOS installing packages or a page writing files, need no index to stay consistent with, and nothing is preloaded into memory at mount. POSIX metadata lives in the IndexedDB sidecar described above, which has per-entry transactions instead of one JSON file rewritten on every change.
 
@@ -1158,13 +1159,15 @@ A threaded WASIX C program for `test/integration/thread-signal.test.mjs` (#520).
 
 ### `fifotest`
 
-A WASIX C program for `test/integration/fifo.test.mjs` (#896). It opens an existing FIFO; WASIX libc has no `mkfifo`, so coreutils makes it.
+A WASIX C program for `test/integration/fifo.test.mjs` (#896).
+- `fifotest snatch PATH N` tries `open(O_RDONLY|O_NONBLOCK)` in a loop until it opens the FIFO or has been refused (`EACCES`) N times, and prints both counts.
+- `fifotest mkfifo PATH [MODE [DIR]]` makes a FIFO through its own `slicc_fs.path_mkfifo` declaration (`MODE` in octal, `666` by default), relative to the cwd, or to `DIR` opened as a directory fd, and prints the result.
 - `fifotest stat PATH` prints `fifo` or `not a fifo`.
 - `fifotest read PATH` opens it for reading (blocking) and prints what it reads until `EOF`. `fifotest nbread PATH` does the same with `O_NONBLOCK`.
 - `fifotest write PATH [TEXT]` opens it for writing and writes TEXT. `fifotest nbwrite PATH [TEXT]` does so with `O_NONBLOCK`, and prints `open: No such device or address` with no reader.
 - `fifotest alarm PATH [restart]` arms `alarm(1)` with a `SIGALRM` handler (with `SA_RESTART` if asked), then opens the FIFO for reading. It prints `open: Interrupted system call`, or `opened` and what it read. The handler creates the file named by `FIFOTEST_MARK`, so a test can wait for the interrupt instead of a clock.
 
-**Build:** `WASIX_SYSROOT_28=/path/to/wasix-sysroot ./build.sh`, with `wasixcc` 0.4.7 and binaryen's `wasm-opt` on `PATH`. `bin/fifotest.wasm` is 51,468 bytes, sha256 `a233774a219058050d8d793d48b05174cb35b8e571cb60aefa4b8d6612da9a28`.
+**Build:** `WASIX_SYSROOT_28=/path/to/wasix-sysroot ./build.sh`, with `wasixcc` 0.4.7 and binaryen's `wasm-opt` on `PATH`. `bin/fifotest.wasm` is 54,410 bytes, sha256 `bec06cd5cc47e64f1b8871d0c09dd5ca9e679e0239c58825cb94047c34886b40`.
 
 ### `fdalias`
 

package/dist/kernel-worker.js

@@ -872,6 +872,7 @@ var RemoteWatch = class {
   pruned = 0;
   joined;
   met;
+  settled = false;
   quiet;
   constructor(name2, options = {}) {
     this.name = name2;
@@ -887,7 +888,9 @@ var RemoteWatch = class {
       });
       if (!this.locks) resolve2();
     });
-    this.joined = granted2.then(() => this.join());
+    this.joined = granted2.then(() => this.join()).then(() => {
+      this.settled = true;
+    });
     this.channel.postMessage({ hello: this.tab });
   }
   async join() {
@@ -909,6 +912,9 @@ var RemoteWatch = class {
     await this.joined;
     return this.peers.size > 0;
   }
+  crowded() {
+    return !this.settled || this.peers.size > 0;
+  }
   lockOf(tab) {
     return `${this.name}:${tab}`;
   }
@@ -1493,7 +1499,7 @@ var OpfsFs = class {
     const kept = this.dirs.get(path);
     if (kept && await this.current(path, kept)) return kept;
     const gone = this.absent.get(path);
-    if (gone !== void 0 && performance.now() - gone < ABSENT_MS) {
+    if (gone !== void 0 && !this.shared() && performance.now() - gone < ABSENT_MS) {
       throw translate({ name: "NotFoundError" }, path);
     }
     const seen = this.changes;
@@ -1680,7 +1686,11 @@ var OpfsFs = class {
         if (this.dirTimes.get(path) === ms) this.dirTimes.delete(path);
     }
   }
+  shared() {
+    return this.remote?.crowded() === true;
+  }
   glanced(handle) {
+    if (this.shared()) return void 0;
     const seen = this.glances.get(handle);
     const fresh2 = seen && seen.changes === this.changes && performance.now() - seen.at < GLANCE_MS;
     return fresh2 ? seen.file : void 0;
@@ -6366,7 +6376,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.119.1" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.120.0" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC