@ai-ecoverse/slicc-kernel 1.98.1 → 1.98.2

package/README.md

@@ -535,7 +535,7 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
 - **Capabilities:** `readonly`, `symlinks`, `links` (hard links: the driver answers `link(from, to)`, reports `nlink` in its attributes, and gives every attribute a stable `ino`, the same for all names of a file; a mount whose driver offers `links` but answers an attribute without `ino` loses them: `link` is `EPERM` there and `st_nlink` is 1), `chmod`, `owners`, `ranges`, `maxIo` (the largest read or write per message), `maxFile`, `listingStats`, `sessions`, `linkTimes`, and `attrTtl`/`entryTtl` (in ms, default 1000). `sessions` promises that a handle opened for writing keeps its writes to itself until it is released, so the kernel keeps one open per file being written instead of one per write (the `fsa` driver has it). `linkTimes` promises that `setattr` with `mtime` on a symlink sets the link's own time. Without it, setting a link's own times (`touch -h`, `utimensat` with `AT_SYMLINK_NOFOLLOW`) fails with `EOPNOTSUPP` and never reaches the target. tmpfs has it; hostfs does not. Setting times through a link follows it in the kernel, so `setattr` gets the target. `ranges` (driver protocol 1.1) promises that a write lands in the file at once, with no copy of the whole file on `open` or `release`, and that `setattr` takes `size`. `owners` (driver protocol 1.2) promises that the driver stores owners: its attributes carry `uid` and `gid`, `open` with `create`, `mkdir` and `symlink` may carry the new path's `uid`, `gid` and `mode` (the `owner` of the open flags, the second or third handler argument), a new path without them takes its directory's owner, and `setattr` takes `uid` and `gid`. tmpfs has it.
 - **Caching:** the kernel caches attributes and listings for those TTLs, and a driver can push `invalidate` for changes made outside.
 - **File contents:** on a mount with `ranges`, as `tmpfs` is, and on the root file system when it is in OPFS (in a browser with sync access handles) or an in-memory root, a program reads and writes a file in 64 KiB pages: reads come from the driver 1 MiB at a time, and changed pages go back in 1 MiB runs once 8 MiB have changed, at `fsync`, and when the file is closed (from the kernel's own descriptors also 250 ms after a write). A file of any size the mount can hold works, with memory to spare for about one more copy of it. In OPFS a write opens a sync access handle for just that write. An `fsa` mount (a picked folder) has no sync access handles, and Chrome copies the whole file into a swap file for every writable stream, so it writes through one stream per open file: the first write back opens it, later runs go into it at their offsets, and it is committed, one copy of the file, when the last process writing the file closes it (or exits), before another process reads what was written, and before a rename, unlink or unmount. Until then other processes see the new size but read the old bytes outside what was written. Elsewhere (`hostfs` with an older proxy, and a root outside OPFS), file contents are read whole when a program opens a file and written back when it closes it. Before a program's next call into the kernel (any wait, read of a pipe or terminal, sleep or spawn), its changed pages also go back, so other processes and the embedder see a write as soon as the writer does anything but compute: a daemon's pidfile, a log another process follows. On a whole-file mount that holds only for files of up to 64 KiB; a larger one goes back at close, fsync and exit, as before. Beyond that, both give close-to-open consistency, as NFS has. A process that ends without closing its files keeps what it wrote: on exit, on a trap, and when a signal kills it. For a killed WASI program, the kernel lets its worker write its changed pages back before `waitpid` reports the death. The only exception is a worker that makes no call into the kernel for 2 seconds after the signal; it is stopped without that write-back.
-- **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. Limits for now: the same file under two names (a symlink in the path, a hard link, an alias on a mount without `realpath`) isn't kept coherent yet, so processes sharing a file should name it by the same path ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)); another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them.
+- **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. The same file under two names is one file too ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)): a write by path through a symlink reaches the file the kernel holds under its real path (resolved without following links in a `sudo` process's no-links view, so an untrusted link is still refused), a WASI program's open file also watches its real path (resolved once, the first time another file changes), and an open file with more than one hard link checks its size, version and mtime when anything changed, so a write through another name is seen. Limits for now: an alias on a mount that has no `realpath` is still only the name it was opened by; another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them.
 - **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` keeps its behaviour (whole file, per process, keyed by path). Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program's `fcntl` locks (python's `sqlite3`) are no-ops in its libc that never reach the kernel, until wasix-libc gets a kernel import for them ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)), so python's sqlite doesn't exclude other processes yet.
 - **Rules on a mount:**
   - a rename across mounts is `EXDEV`, so `mv` copies;

package/dist/kernel-worker.js

@@ -8843,9 +8843,19 @@ function touches(watcher, changed) {
     (watched) => changed === watched || under3(watched, changed) || (watcher.recursive ? under3(changed, watched) : parentOf6(changed) === watched)
   );
 }
+var REALS_KEPT = 4096;
 var FsWatchers = class {
   watchers = /* @__PURE__ */ new Set();
   taps = [];
+  reals = /* @__PURE__ */ new Map();
+  reshapes = [];
+  onReshape(listener) {
+    this.reshapes.push(listener);
+  }
+  reshaped() {
+    this.reals.clear();
+    for (const listener of this.reshapes) listener();
+  }
   tap(listener) {
     this.taps.push(listener);
   }
@@ -8889,8 +8899,41 @@ var FsWatchers = class {
         this.changed(...changed(...args));
       }
     };
+    const reals = this.reals;
+    const realOf = async (path) => {
+      const at = normalizePath(path);
+      const known = reals.get(at);
+      if (known !== void 0) return known;
+      if (!fs.realpath) return at;
+      let real;
+      try {
+        real = await fs.realpath(at, true);
+      } catch {
+        return at;
+      }
+      if (reals.size >= REALS_KEPT) reals.clear();
+      reals.set(at, real);
+      return real;
+    };
+    const content = (run2) => async (...args) => {
+      try {
+        await run2(...args);
+      } finally {
+        const at = normalizePath(args[0]);
+        const real = await realOf(at);
+        this.changed(...real === at ? [at] : [at, real]);
+      }
+    };
+    const reshaping = (run2) => async (...args) => {
+      this.reshaped();
+      try {
+        await run2(...args);
+      } finally {
+        this.reshaped();
+      }
+    };
     const overrides = {
-      writeFile: after(fs.writeFile.bind(fs), (path) => [path]),
+      writeFile: content(fs.writeFile.bind(fs)),
       mkdir: async (path, options) => {
         const created3 = [];
         if (options?.recursive) {
@@ -8906,13 +8949,15 @@ var FsWatchers = class {
           this.changed(...created3, path);
         }
       },
-      rm: after(fs.rm.bind(fs), (path) => [path]),
-      rename: after(fs.rename.bind(fs), (from, to) => [from, to]),
-      link: after(
-        (from, to, follow2) => fs.link(from, to, follow2),
-        (_from, to) => [to]
+      rm: reshaping(after(fs.rm.bind(fs), (path) => [path])),
+      rename: reshaping(after(fs.rename.bind(fs), (from, to) => [from, to])),
+      link: reshaping(
+        after(
+          (from, to, follow2) => fs.link(from, to, follow2),
+          (_from, to) => [to]
+        )
       ),
-      symlink: after(fs.symlink.bind(fs), (_target, path) => [path]),
+      symlink: reshaping(after(fs.symlink.bind(fs), (_target, path) => [path])),
       chmod: after(fs.chmod.bind(fs), (path) => [path]),
       ...fs.chown ? { chown: after(fs.chown.bind(fs), (path) => [path]) } : {},
       utimes: after(fs.utimes.bind(fs), (path) => [path]),
@@ -8923,8 +8968,8 @@ var FsWatchers = class {
     };
     const ranged2 = rangedOps(fs);
     if (ranged2) {
-      overrides.pwrite = after(ranged2.pwrite, (path) => [path]);
-      overrides.truncate = after(ranged2.truncate, (path) => [path]);
+      overrides.pwrite = content(ranged2.pwrite);
+      overrides.truncate = content(ranged2.truncate);
     }
     return new Proxy(fs, {
       get(target, key, receiver) {
@@ -9692,6 +9737,7 @@ var MountTable = class {
     };
     mounted = mount;
     this.mounts.set(target, mount);
+    this.deps.reshaped?.();
     this.notes.delete(target);
     return this.list().find((m) => m.target === target);
   }
@@ -9706,6 +9752,7 @@ var MountTable = class {
     if (!detach && this.deps.busy(at)) throw errnoError("EBUSY", `${at} has open files`);
     this.notes.delete(at);
     this.mounts.delete(at);
+    this.deps.reshaped?.();
     const waits = [...this.sessions].filter(([, s]) => s.mount === mount).map(([real]) => this.commit(real));
     for (const c of this.committing.values()) if (c.mount === mount) waits.push(c.done);
     const close = () => {
@@ -12035,7 +12082,7 @@ var PATH_OPS = new Set(SYNC_FS_OPS);
 var EARLY_WHOLE_FILE = 64 * 1024;
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.98.1" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.98.2" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -14899,6 +14946,7 @@ var VfsNodes = class {
     if (!this.byPath.has(to)) this.byPath.set(to, node);
   }
   know(node, id2) {
+    this.reshaped();
     node.id = id2;
     if (!this.byId.has(id2)) this.byId.set(id2, node);
   }
@@ -14908,6 +14956,7 @@ var VfsNodes = class {
     return out;
   }
   forget(node) {
+    this.reshaped();
     for (const p of this.namesOf(node)) this.byPath.delete(p);
     if (node.id !== void 0 && this.byId.get(node.id) === node) this.byId.delete(node.id);
   }
@@ -14922,6 +14971,38 @@ var VfsNodes = class {
     if (node.path === path) node.renamedTo(others[0]);
     return true;
   }
+  holdsAny() {
+    return this.byPath.size > 0;
+  }
+  epoch = 0;
+  plain = /* @__PURE__ */ new Map();
+  reshaped() {
+    this.epoch++;
+    this.plain.clear();
+  }
+  plainAt(path) {
+    return this.plain.get(path) === this.epoch;
+  }
+  get shape() {
+    return this.epoch;
+  }
+  notePlain(path, shape) {
+    if (shape !== this.epoch) return;
+    if (this.plain.size >= 4096) this.plain.clear();
+    this.plain.set(path, shape);
+  }
+  holdsName(name2) {
+    for (const path of this.byPath.keys())
+      if (path.slice(path.lastIndexOf("/") + 1) === name2) return true;
+    return false;
+  }
+  holdsLinked() {
+    return this.byId.size > 0;
+  }
+  holdingId(id2) {
+    const node = this.byId.get(id2);
+    return node && !node.orphaned ? node : void 0;
+  }
   holding(path) {
     if (this.byPath.size === 0) return void 0;
     const node = this.byPath.get(path);
@@ -14933,6 +15014,7 @@ var VfsNodes = class {
     return node;
   }
   create(path, id2, node) {
+    this.reshaped();
     node.onDirty = () => this.changed?.(node.path);
     this.byPath.set(path, node);
     if (id2 !== void 0) this.know(node, id2);
@@ -15074,11 +15156,35 @@ async function refuseReadonly(fs, path, flags2, opts = {}) {
   const st = own ?? await stat(await targetDir(fs, path));
   if (st?.readonly) throw new KernelError("EROFS");
 }
+var baseName = (path) => path.slice(path.lastIndexOf("/") + 1);
+async function byRealPath(fs, nodes, at) {
+  const real = await fs.realpath?.(at, true).catch(() => void 0);
+  return real === void 0 || real === at ? void 0 : nodes.holding(real);
+}
+async function aliasOf(fs, nodes, at) {
+  if (nodes.holdsName(baseName(at))) {
+    const named = await byRealPath(fs, nodes, at);
+    if (named) return named;
+  }
+  const st = await fs.lstat(at).catch(() => void 0);
+  if (!st) return void 0;
+  if (st.isSymbolicLink) return byRealPath(fs, nodes, at);
+  if (st.ino === void 0 || (st.nlink ?? 1) < 2 || !nodes.holdsLinked()) return void 0;
+  return nodes.holdingId(`${st.dev ?? 0}:${st.ino}`);
+}
+async function heldAt(fs, nodes, at) {
+  const held = nodes.holding(at);
+  if (held || !nodes.holdsAny() || nodes.plainAt(at)) return held;
+  const shape = nodes.shape;
+  const found = await aliasOf(fs, nodes, at);
+  if (!found) nodes.notePlain(at, shape);
+  return found;
+}
 function heldIo(fs, nodes, changed) {
   const ranged2 = rangedOps(fs);
-  const node = (path) => nodes.holding(fs.resolvePath("/", path));
+  const node = (path) => heldAt(fs, nodes, fs.resolvePath("/", path));
   const write = async (path, op, fallback) => {
-    const held = node(path);
+    const held = await node(path);
     if (!held) return fallback();
     const out = await held.serial(async () => {
       const done4 = await op(held);
@@ -15086,6 +15192,7 @@ function heldIo(fs, nodes, changed) {
       return done4;
     });
     changed(fs.resolvePath("/", path));
+    if (held.path !== fs.resolvePath("/", path)) changed(held.path);
     return out;
   };
   return {
@@ -15110,17 +15217,17 @@ function heldIo(fs, nodes, changed) {
     ),
     async stat(path) {
       const st = await fs.stat(path);
-      const held = st.isFile ? node(path) : void 0;
+      const held = st.isFile ? await node(path) : void 0;
       return held ? { ...st, size: await held.serial(() => held.size()) } : st;
     },
     async readFileBuffer(path) {
-      const held = node(path);
+      const held = await node(path);
       if (!held) return fs.readFileBuffer(path);
       return held.serial(async () => held.pread(await held.size(), 0));
     },
     ...ranged2 ? {
-      pread: (path, at, n, version) => {
-        const held = node(path);
+      pread: async (path, at, n, version) => {
+        const held = await node(path);
         return held ? held.serial(() => held.pread(n, at)) : ranged2.pread(path, at, n, version);
       },
       pwrite: (path, at, bytes, transfer) => write(
@@ -15168,9 +15275,9 @@ function keepingOpen(fs, nodes, changed = () => {
         nodes.renamed(a, b);
       });
     },
-    writeFile: (path, content, owner) => {
+    writeFile: async (path, content, owner) => {
       const at = fs.resolvePath("/", path);
-      const held = nodes.holding(at);
+      const held = await heldAt(fs, nodes, at);
       if (!held) return nodes.onEntry(at, () => fs.writeFile(path, content, owner));
       const bytes = typeof content === "string" ? new TextEncoder().encode(content) : content;
       return held.serial(async () => {
@@ -15178,6 +15285,7 @@ function keepingOpen(fs, nodes, changed = () => {
         await held.pwrite(bytes, 0);
         await held.flush();
         changed(at);
+        if (held.path !== at) changed(held.path);
       });
     },
     createExclusive: async (path, owner) => nodes.createExclusive(await nodes.entryKey(fs.resolvePath("/", path)), owner),
@@ -20220,7 +20328,7 @@ function usersMode(mode = "off") {
 function tagOption(tag) {
   if (tag !== void 0 && typeof tag !== "string") throw new TypeError("a tag must be a string");
 }
-function baseName(path) {
+function baseName2(path) {
   return path.slice(path.lastIndexOf("/") + 1);
 }
 function packageRootOf(path, roots) {
@@ -20263,7 +20371,7 @@ function appendedRoots(given) {
 }
 function plainEnv(words2) {
   const next2 = words2[1];
-  if (baseName(words2[0] ?? "") !== "env" || next2 === void 0) return false;
+  if (baseName2(words2[0] ?? "") !== "env" || next2 === void 0) return false;
   return next2 === "-S" || !next2.startsWith("-") && !next2.includes("=");
 }
 function shebangCommand(words2) {
@@ -20279,9 +20387,9 @@ function shebangCommand(words2) {
 function interpreter(head) {
   const words2 = shebang(head);
   if (!words2) return "node";
-  if (baseName(words2[0]) === "env") words2.shift();
+  if (baseName2(words2[0]) === "env") words2.shift();
   if (words2[0] === "-S") words2.shift();
-  return baseName(words2[0] ?? "");
+  return baseName2(words2[0] ?? "");
 }
 function expandDefaults(defaults, env) {
   const out = {};
@@ -20455,6 +20563,9 @@ var Launcher = class {
     this.sealed.userOf = async (uid) => (await this.users.lookup(uid))?.name;
     if (this.owners) options.fs.trackOwners?.();
     this.watchers.tap((paths) => this.changes.publish(...paths));
+    this.watchers.onReshape(() => {
+      for (const nodes of this.openFiles) nodes.reshaped();
+    });
     this.mounts = new MountTable({
       open: (type, spec) => this.driver(type, spec),
       busy: (target) => [...this.openFiles].some((nodes) => nodes.holds(target)) || heldUnder(this.held, target),
@@ -20462,6 +20573,7 @@ var Launcher = class {
         if (paths.some((path) => this.pnpmChain.has(path))) this.pnpmStale();
         this.watchers.changed(...paths);
       },
+      reshaped: () => this.watchers.reshaped(),
       owners: this.owners
     });
     this.mounted = this.mounts.wrap(options.fs);
@@ -20719,22 +20831,22 @@ var Launcher = class {
     const head = await this.head(glue);
     if (!head) return void 0;
     if (isWasm(head)) {
-      const owner2 = await this.wasmOwner(glue, baseName(file).replace(/\.wasm$/, ""), scope);
+      const owner2 = await this.wasmOwner(glue, baseName2(file).replace(/\.wasm$/, ""), scope);
       if (owner2) return { ...targetOf(owner2), ...asked };
       return {
         abi: "wasi",
         glue,
         wasm: glue,
-        argv0: baseName(argv0 || file).replace(/\.wasm$/, ""),
+        argv0: baseName2(argv0 || file).replace(/\.wasm$/, ""),
         ...asked
       };
     }
     if (interpreter(head) !== "node") return void 0;
     const wasm = modulePath(glue);
     if (await this.base.exists(wasm)) {
-      return { abi: "emscripten", glue, wasm, argv0: baseName(argv0 || file), ...asked };
+      return { abi: "emscripten", glue, wasm, argv0: baseName2(argv0 || file), ...asked };
     }
-    const owner = await this.glueOwner(glue, baseName(file), scope);
+    const owner = await this.glueOwner(glue, baseName2(file), scope);
     return owner && { ...targetOf(owner), ...asked };
   }
   async glueOwner(glue, name2, scope) {
@@ -20744,11 +20856,11 @@ var Launcher = class {
     return owners.find((c) => c.name === name2) ?? owners[0];
   }
   async wasmOwner(wasm, name2, scope) {
-    const leaf = baseName(wasm);
+    const leaf = baseName2(wasm);
     const real = await physicalPathAsync(this.base, wasm);
     const owners = [];
     for (const command of (await this.commands(scope)).values()) {
-      if (command.abi !== "wasi" || command.script || baseName(command.wasm) !== leaf) continue;
+      if (command.abi !== "wasi" || command.script || baseName2(command.wasm) !== leaf) continue;
       if (command.wasm === wasm || await physicalPathAsync(this.base, command.wasm) === real)
         owners.push(command);
     }
@@ -21812,7 +21924,7 @@ var Launcher = class {
     const path = normalizePath(given);
     if (!await this.base.exists(path)) return void 0;
     if (path === "/") return path;
-    return joinName(await realDir(this.base, parentOf4(path)), baseName(path));
+    return joinName(await realDir(this.base, parentOf4(path)), baseName2(path));
   }
   pnpmReady() {
     if (this.pnpmMade !== void 0) return this.pnpmMade;