@ai-ecoverse/slicc-kernel 1.167.1 → 1.167.3

package/README.md

@@ -858,7 +858,7 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
 - **Exclusive create across tabs** ([#912](https://github.com/ai-ecoverse/slicc-kernel/issues/912)): two kernels on one OPFS root (two tabs, or a tab and a worker page) agree on `O_CREAT|O_EXCL` and `set -C`: exactly one wins a name. While another kernel on the root is present, the create runs under a Web Lock for its path and checks the OPFS entry itself. A kernel alone on its root takes no lock. Each kernel holds an alive Web Lock, granted before it first looks for peers. A kernel that finds peers waits for their answer (at most 500 ms) before its first exclusive create, and a kernel answers a newcomer only once its unlocked creates are done. So of two kernels that boot at the same moment, the later one to look sees the other, and no unlocked create overlaps a locked one.
 - **A name another process or tab just removed** ([#950](https://github.com/ai-ecoverse/slicc-kernel/issues/950)): sqlite3 deletes its `-journal` at every commit and the next writer creates it again, so two writers (two processes in one kernel, or two tabs on one OPFS root) keep removing and re-creating one name. Two things used to turn that into `unable to open database file`. The root removes a file's sidecar record a few milliseconds after the file itself (removals are batched), and an exclusive create read that pending record as the name being taken: a plain record whose file is gone now counts as gone, and the new file gets a record of its own. A symlink, which lives only in the sidecar, still takes its name. And an Emscripten program keeps the nodes it has looked up, so after another tab removed a name it still held the old node, and `open(O_CREAT)` opened the gone file instead of creating one. Now, when the kernel answers `ENOENT` for a cached node, for `open`, `stat`, `lstat`, `readlink`, `truncate`, `unlink` or a rename's source, or the program's own lookup answers `EEXIST` to `open(O_CREAT|O_EXCL)`, the node is dropped and the call runs once more, against what is there now. A node the program has open or has unwritten changes in is kept. A rename over a name another tab removed needs nothing: the kernel's rename replaces whatever is there.
 - **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. The same file under two names is one file too ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)): a write by path through a symlink reaches the file the kernel holds under its real path (resolved without following links in a `sudo` process's no-links view, so an untrusted link is still refused), a WASI program's open file also watches its real path (resolved once, the first time another file changes), and an open file with more than one hard link checks its size, version and mtime when anything changed, so a write through another name is seen. Limits for now: an alias on a mount that has no `realpath` is still only the name it was opened by; another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them. A by-path `stat` or `lstat` of a file the kernel holds open has its current size, writes not yet written back included ([#780](https://github.com/ai-ecoverse/slicc-kernel/issues/780)). Its mtime is the file system's until write-back, though, so a tool that compares times, like `make`, may see a just-written file as older than it is for up to the write-back delay.
-- **Appends across tabs** ([#1520](https://github.com/ai-ecoverse/slicc-kernel/issues/1520)): while another kernel shares the OPFS root, an `O_APPEND` write lands at the file's real end under a per-file Web Lock, so two tabs' appends never overwrite each other. A tab that sees another join flushes its buffered appends and waits for the writes it has under way at that moment (later ones don't hold it) before it answers, and the joiner's appends wait for every such answer pending at the time, without a timeout, as long as the other tab lives; an append already under way when a further tab's note arrives isn't held back ([#1546](https://github.com/ai-ecoverse/slicc-kernel/issues/1546)). Alone, an append is buffered as before. **Limit:** a write at the end *without* `O_APPEND` (a seek to the end, then a write) is close-to-open across tabs, as on NFS: it can overwrite another tab's appends it hasn't seen yet ([#1546](https://github.com/ai-ecoverse/slicc-kernel/issues/1546)).
+- **Appends across tabs** ([#1520](https://github.com/ai-ecoverse/slicc-kernel/issues/1520)): while another kernel shares the OPFS root, an `O_APPEND` write lands at the file's real end under a per-file Web Lock, so two tabs' appends never overwrite each other. A tab that sees another join flushes its buffered appends and waits for the writes it has under way at that moment (later ones don't hold it) before it answers, and the joiner's appends wait for every such answer pending at the time, without a timeout, as long as the other tab lives; an append already under way when a further tab's note arrives isn't held back ([#1546](https://github.com/ai-ecoverse/slicc-kernel/issues/1546)). Alone, an append is buffered as before. While another tab holds a file open, a seek to its end or an `fstat` reads the real end, and a write past that end goes through under the same lock before it returns, so a write at the end *without* `O_APPEND` (a seek to the end, then a write) doesn't overwrite another tab's appends either ([#1546](https://github.com/ai-ecoverse/slicc-kernel/issues/1546)). That holds for a WASI program's own files, for descriptors the kernel holds (a JS program's `ctx.sync.open`, a shell's `exec 3<>` once a child has it), and for a file opened through a symlink, which is looked up by its real path ([#1675](https://github.com/ai-ecoverse/slicc-kernel/issues/1675)). A file no other tab holds stays buffered: no lock and no extra call. **Limits:** a write inside the file stays buffered, so it's close-to-open across tabs, as on NFS. Reading to the end, rather than seeking there, still stops at the end this tab last knew. A WASI program that opened a file through a symlink keeps the real path it had at open, so after the target is renamed its writes stay buffered until it opens the file again. A writer that has seen the file held keeps reading its end until it finds the file no longer held, so a peer that appends and closes right away is covered; a peer whose whole open, append and close fall between two of this tab's looks is seen only through its notices.
 - **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, and so are a WASIX program's through the `slicc.fcntl_lock` import (below), so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An Emscripten program's file goes to a kernel descriptor at its first lock. From then on, its `fstat` size is that descriptor's, so on a file that isn't read and written in pages (a root outside OPFS, or `hostfs` without `ranges`), sqlite sees the pages it wrote and the kernel still holds (#1019). An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` (wasi-pnpm's store-operation lock) is a whole-file lock, per process and keyed by path, while the kernel is alone on its root. While another kernel shares the root, it becomes a non-blocking `flock` on the descriptor's open file instead, keyed by the file's identity, so it excludes across kernels and tabs (#983). It's released on `unlock`, on close, and about a second after the holder's tab closes. A tryLock checks both the local table and the flock table, so a lock taken before a peer joins, or while one was there, still excludes after the switch. Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program built against a wasix-sysroot without the import keeps its libc's local answer (every lock granted), so python's sqlite excludes other processes only once wasix-python is rebuilt on the sysroot that has it ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)). An Emscripten program's `flock()` reaches the kernel's `flock` table (per open file description) through homescoop's shim, a strong `flock()` that calls the import `env.__syscall_flock(fd, op)`. The kernel wraps that import by name (#947): `LOCK_SH`/`LOCK_EX`/`LOCK_UN` with `LOCK_NB` (`EWOULDBLOCK` for a conflict), 0 or -errno. A glue without the shim keeps Emscripten's stub, which answers 0. A lock belongs to the file its descriptor was opened on: a file renamed over the name later, or the name's removal, doesn't move it.
 - **Locks across tabs** ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)): two kernels on one OPFS root (two tabs, or a tab and a worker page) share one table of `fcntl` record locks and `flock` locks, so a sqlite3 in one tab finds the database locked while a transaction is open in another. **Known issue, data corruption: lock-protected writes across tabs** ([#949](https://github.com/ai-ecoverse/slicc-kernel/issues/949)). The locks are enforced across tabs, but what one tab writes under a lock isn't yet guaranteed to be what the other tab reads once it holds the lock, so a lock-protected read-modify-write across tabs can lose or corrupt data. Don't rely on cross-tab locking until #949 is fixed. Use within a single tab is unaffected.
 - **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, and so are a WASIX program's through the `slicc.fcntl_lock` import (below), so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` keeps its behaviour (whole file, per process, keyed by path). Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program built against a wasix-sysroot without the import keeps its libc's local answer (every lock granted), so python's sqlite excludes other processes only once wasix-python is rebuilt on the sysroot that has it ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)). An Emscripten program's `flock()` is a stub compiled into its libc that never reaches the kernel either, until a homescoop shim routes it to a kernel import ([#680](https://github.com/ai-ecoverse/slicc-kernel/issues/680)). A lock belongs to the file its descriptor was opened on: a file renamed over the name later, or the name's removal, doesn't move it.

package/dist/kernel-worker.js

@@ -8477,7 +8477,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.167.1" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.167.3" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -15242,7 +15242,7 @@ async function heldOpen(entry, path, real, req, fid) {
   const handle = entry.hold?.(path, true, true, real, void 0, req.ino, fid);
   if (handle === void 0) return null;
   await entry.announceHold?.(real ?? path, fid, ((req.mode ?? 0) & HOLD_CREATED) !== 0);
-  return handle;
+  return real !== void 0 && real !== path ? { handle, real } : handle;
 }
 async function idOf(entry, path) {
   return await entry.fileId?.(path)?.catch(() => void 0) ?? void 0;
@@ -16203,6 +16203,10 @@ var AsyncRangedFile = class {
   truncate(size) {
     this.pages.truncate(size);
   }
+  wrote(at, length2) {
+    this.pages.landed(at, length2);
+    this.version = void 0;
+  }
   async flush() {
     if (this.pinned || !this.pages.changed) return;
     const plan = this.pages.plan();
@@ -24488,6 +24492,7 @@ var VfsNode = class {
   ino;
   onDirty;
   onDirtyState;
+  crowded;
   counted = false;
   recount() {
     if (!this.onDirtyState) return;
@@ -24605,9 +24610,48 @@ var VfsNode = class {
     if (!this.dirty) this.data = void 0;
   }
   async size() {
+    if (this.peerHeld()) {
+      const known = await this.exact();
+      if (known !== void 0) return known;
+    }
     const ranged2 = await this.prepare();
     return ranged2 ? ranged2.size() : this.length;
   }
+  peered = false;
+  peerHeld() {
+    if (this.orphaned || this.revoked || this.pin) return false;
+    if (this.crowded?.(this.path) === true) this.peered = true;
+    return this.peered;
+  }
+  async exact() {
+    await this.flush();
+    const { stat } = this.fs;
+    const io = rangedOps(this.fs);
+    if (!stat || !io) return void 0;
+    const st = await this.followed(() => stat.call(this.fs, this.path));
+    this.peered = this.crowded?.(this.path) === true;
+    const size = st.size ?? 0;
+    if (this.ranged) {
+      if (size !== this.ranged.size()) {
+        this.ranged = new AsyncRangedFile(io, this.path, size, st.version);
+      }
+      return void 0;
+    }
+    if (this.data && size !== this.length) this.data = void 0;
+    return this.data ? void 0 : size;
+  }
+  landed(at, bytes) {
+    if (this.ranged) {
+      this.ranged.wrote(at, bytes.length);
+      return;
+    }
+    if (!this.data) return;
+    const end = at + bytes.length;
+    const buf = this.ensure(end);
+    if (at > this.length) buf.fill(0, this.length, at);
+    buf.set(bytes, at);
+    this.length = Math.max(this.length, end);
+  }
   async pread(max, at) {
     const ranged2 = await this.prepare();
     if (ranged2) return this.followed(() => ranged2.read(at, max));
@@ -24679,6 +24723,13 @@ var VfsNode = class {
     if (bytes.length === 0) return 0;
     await this.settled();
     await this.fits(at + bytes.length);
+    const io = this.peerHeld() ? rangedOps(this.fs) : void 0;
+    if (io && at + bytes.length > await this.size()) {
+      const copy = bytes.slice();
+      await this.followed(() => io.pwrite(this.path, at, copy, true));
+      this.landed(at, bytes);
+      return bytes.length;
+    }
     const ranged2 = await this.prepare();
     if (ranged2) {
       await this.followed(() => ranged2.write(at, bytes));
@@ -24775,6 +24826,7 @@ var VfsNodes = class {
   movedAway;
   opened;
   dirtied;
+  crowded;
   fs;
   entries;
   onClosed;
@@ -24920,6 +24972,7 @@ var VfsNodes = class {
     this.reshaped();
     node.onDirty = () => this.changed?.(node.path);
     node.onDirtyState = (on) => this.dirtied?.(on ? 1 : -1);
+    node.crowded = (at) => this.crowded?.(at) === true;
     node.recount();
     this.byPath.set(path, node);
     if (id2 !== void 0) this.know(node, id2);
@@ -31488,6 +31541,7 @@ var Launcher = class {
   hostname;
   watchers = new FsWatchers();
   changes = new FileChanges();
+  peerHeldPaths = /* @__PURE__ */ new Set();
   renameLog = new RenameLog(this.changes);
   cwds = new CwdTable(async (path) => {
     const { dev, ino } = await this.base.stat(path);
@@ -33701,7 +33755,7 @@ var Launcher = class {
       kept: async (tab, items, paths) => {
         for (const { path, hidden, fid } of items) {
           if (hidden !== void 0 && sharedKept(hidden)) await this.adoptShared(path, hidden, fid);
-          else this.adoptKept(path, hidden, fid, ours);
+          else await this.adoptKept(path, hidden, fid, ours);
         }
         await open2(tab, paths);
       },
@@ -33711,14 +33765,33 @@ var Launcher = class {
       }
     };
   }
-  async adoptShared(path, hidden, fid) {
-    const bound3 = [...this.held].map(
-      (holds2) => [holds2, [...holds2.boundAt(path, fid), ...holds2.boundAt(hidden, fid)]]
-    );
-    const keeping = bound3.filter(([, ids]) => ids.length > 0);
+  async keptIds(hidden, fid) {
+    if (fid !== void 0) return [fid];
+    const st = await this.rootFs.lstat(hidden).catch(() => void 0);
+    return st?.fid === void 0 ? [void 0] : [void 0, st.fid];
+  }
+  boundTo(path, hidden, fids) {
+    const keeping = [...this.held].map(
+      (holds2) => [
+        holds2,
+        fids.flatMap((fid) => [...holds2.boundAt(path, fid), ...holds2.boundAt(hidden, fid)])
+      ]
+    ).filter(([, ids]) => ids.length > 0);
     const nodes = [...this.openFiles].filter(
       (open2) => [path, hidden].some((at) => open2.holdsAt(at))
     );
+    const keep2 = (file) => {
+      const kept = nodes.filter(
+        (open2) => fids.some((fid) => open2.keepBound([hidden, path], fid, file))
+      ).length;
+      for (let i = kept; i < nodes.length; i++) file.release();
+      for (const [holds2, ids] of keeping) holds2.keepBound(ids, path, file);
+      void this.rootFs.announceHolds?.([], [path, hidden], false);
+    };
+    return { keeping, nodes, keep: keep2 };
+  }
+  async adoptShared(path, hidden, fid) {
+    const { keeping, nodes, keep: keep2 } = this.boundTo(path, hidden, await this.keptIds(hidden, fid));
     const refs = keeping.length + nodes.length;
     if (refs === 0) {
       this.held.doubt([path]);
@@ -33733,33 +33806,22 @@ var Launcher = class {
       this.held.doubt([path]);
       return;
     }
-    const kept = nodes.filter((open2) => open2.keepBound([hidden, path], fid, file)).length;
-    for (let i = kept; i < nodes.length; i++) file.release();
-    for (const [holds2, ids] of keeping) holds2.keepBound(ids, path, file);
-    void this.rootFs.announceHolds?.([], [path, hidden], false);
+    keep2(file);
   }
-  adoptKept(path, hidden, fid, ours) {
+  async adoptKept(path, hidden, fid, ours) {
     if (hidden === void 0 || !ours(hidden)) {
       this.held.doubt([path]);
       return;
     }
-    const bound3 = [...this.held].map(
-      (holds2) => [holds2, [...holds2.boundAt(path, fid), ...holds2.boundAt(hidden, fid)]]
-    );
-    const keeping = bound3.filter(([, ids]) => ids.length > 0);
-    const nodes = [...this.openFiles].filter(
-      (open2) => [path, hidden].some((at) => open2.holdsAt(at))
-    );
+    const { keeping, nodes, keep: keep2 } = this.boundTo(path, hidden, await this.keptIds(hidden, fid));
     const file = this.unlinked.adopt(hidden, Math.max(keeping.length + nodes.length, 1));
-    const kept = nodes.filter((open2) => open2.keepBound([hidden, path], fid, file)).length;
-    for (let i = kept; i < nodes.length; i++) file.release();
     if (keeping.length === 0 && nodes.length === 0) file.release();
-    for (const [holds2, ids] of keeping) holds2.keepBound(ids, path, file);
-    void this.rootFs.announceHolds?.([], [path, hidden], false);
+    keep2(file);
   }
   openNodes(fs, entries) {
     const nodes = new VfsNodes(fs, (path) => this.released(path), entries);
     nodes.opened = (node) => this.nodeOpened(node);
+    nodes.crowded = (path) => this.peerHeldPaths.has(path);
     return nodes;
   }
   nodeOpened(node) {
@@ -33781,13 +33843,17 @@ var Launcher = class {
     return this.mounts.locate(path) === void 0 ? this.rootFs.bindId?.(path) : void 0;
   }
   flushingPeers(fs) {
-    fs.peerHolds?.((paths) => this.changes.peerHeld(paths));
+    fs.peerHolds?.((paths) => {
+      this.peerHeldPaths = new Set(paths);
+      this.changes.peerHeld(paths);
+    });
     const peers = fs.peers?.();
     if (!peers) return void 0;
     peers.watch((crowded) => {
       this.changes.flush();
       this.changes.foreign(crowded);
       if (crowded) {
+        void this.announceHeld();
         void this.held.bindIds((path) => this.rootFileId(path)).then(
           () => Promise.all([...this.openFiles].map((open2) => open2.bindIds((p) => this.rootFileId(p))))
         ).then(() => this.announceHeld());