@ai-ecoverse/slicc-kernel 1.133.4 → 1.133.5

package/README.md

@@ -846,6 +846,24 @@ Programs mount and unmount with `mount(2)` and `umount2(2)`, so a `mount`/`umoun
 - **Errors** are Linux's: `ENODEV` for an unknown type (or `hostfs` without a hook), `ENOENT` and `ENOTDIR` for the target, `EBUSY` for a target that is mounted already, and `EINVAL` for unmounting what is not a mount point.
 - **`/proc/mounts`** escapes spaces and backslashes as Linux does, and adds `nomedium` or `failed` to the options of a mount in that state.
 
+### The working directory
+
+A process's working directory is the directory it changed into, not just its path ([#1164](https://github.com/ai-ecoverse/slicc-kernel/issues/1164)). The kernel records the directory's inode when a process starts and at every `chdir`. A forked child shares its parent's, and so does a spawned child unless the spawn changes directory.
+- **When the directory is replaced:** if it's removed, or something else takes its name (`rm -r d; mkdir d`, `rm -r d; mv n d`, or `mv d x; mv n d`, from this kernel or from another tab on the same root), the process keeps the removed directory:
+  - a wasip1 program's `.` (its `.` preopen) can't be opened or listed (`ENOENT`, an empty listing for its preopen descriptor);
+  - its relative lookups and creates are `ENOENT`;
+  - `..` still reaches the parent.
+- **Detection costs nothing in the steady state:**
+  - each held directory has a counter in memory the kernel shares with the process. It moves only when this kernel, another tab, or a `postFsNotice` reports a change at that directory or above it: the kernel's own watcher paths, and every path and tree of a peer change or notice.
+  - a process checks the counter, a memory read, before each relative lookup. Only after it moved does it ask the kernel once (`cwd-check`), and the kernel compares the inode at the path with the one it recorded.
+- **The `rm` record:** so that `rm -r d; mkdir d` can be told apart, removing a directory leaves the same small record a rename leaves at the name it vacates (see [Metadata](#metadata)).
+- **Limits:**
+  - a WASIX program's C library keeps the working directory as a string and turns relative paths into absolute ones before they reach the kernel. So a WASIX program still sees the new directory through relative paths, and `getcwd` still returns the path, until wasix-sysroot resolves relative paths against the working directory's descriptor ([#1164](https://github.com/ai-ecoverse/slicc-kernel/issues/1164) stays open for that).
+  - Emscripten and JS programs: [#1164](https://github.com/ai-ecoverse/slicc-kernel/issues/1164), next.
+  - a removal between the check and the lookup is a race, as with any path, and the lookup sees the new directory.
+  - on a `hostfs` folder the inode is the host's, which a host file system may reuse for the new directory (ext4 can; APFS doesn't). There the replacement isn't noticed.
+  - with more than 1023 distinct working directories held at once, the rest are not tracked.
+
 ### Exec
 
 An exec'd program takes over the pid of the process it replaces, as on Linux: `getpid()` and `$$` in the new image, `/proc`, `ps`, `kill` and the parent's `waitpid` all agree on that pid, and the image's parent pid is the replaced process's. Its own children see that pid as their parent. A file removed or renamed over while programs hold it open keeps its bytes for them, as on Linux, whether the descriptors live in the kernel or a program opened the file in its own worker: every open, read-only ones included, is known to the kernel, and all holders share the one file (writes through one are seen through the others). The kernel does not copy it: as NFS does with its "silly rename", it moves the file into a hidden directory, `.slicc-unlinked/`, at the root of the mount it lives on (or of the root file system), and serves the holders' reads and writes from there (a worker's I/O on the old path while the path is gone). No file data is held in memory for it. The file is deleted when the last holder closes it or exits, and the directory goes as soon as it is empty. A file created at the path afterwards is the new file for everyone, also for the holder; a holder whose file another process renames (rather than removes) still uses the old path. A rename over a held file stays atomic for everyone else, as `rename(2)` is: moving the held file aside and the new one into its place are two steps, so a lookup of that name (`open`, `stat`, a read) that comes while they run waits for both, and one that started before them and missed the name in between looks again (also when it spelled the name through a symlinked parent, which the wait doesn't match by path: a miss is retried whenever any replacement finished meanwhile, up to three tries). Every open finds the old file or the new one, never no file (#903). A read through a held descriptor and the rename take the file's turn in order: lookups start to wait for the rename only once it has its turn on every held file it moves aside, so a read that came first finishes, and a rename over a file other processes keep reading never stalls them or itself (#980). A process that opens the file while the rename waits for its turn is a holder too: the rename looks for holders again once it has its turn, so that process keeps the old file as well, and a holder that closed meanwhile leaves nothing to move aside. The same holds for a directory that is removed or renamed over while programs hold files in it: the removal keeps looking for holders until nothing new appears, moving each held file aside, and lookups inside the directory wait for it. An open, a hand-over of a program's descriptor to the kernel (at a fork) or a hold that was under way while the directory went looks its path up again, so it gets the kept old file or ENOENT, never an empty file at the old name (#1013). An open of a path that is gone, with a handle the process no longer holds, is ENOENT rather than a descriptor for a missing file. The OPFS root does the same for its own two steps: a rename onto an existing name replaces the file with `move()` (or, in a browser without it, a copy into the name and a removal of the source), and a lookup that loses that race to a stale handle looks again. A `tmpfs` mount replaces the name in one step; on `fsa` and `hostfs` mounts it is the real file system's `rename`. Listings in the kernel leave `.slicc-unlinked` out, but on an `fsa` or `hostfs` mount it is visible on the real machine while a removed file is held (add `.slicc-unlinked/` to a `.gitignore` there). Each kernel names its entries after itself and holds a Web Lock for that name while it runs; when a mount is set up (and the root at start), the kernel removes the entries whose owner holds no lock, so a tab that died while holding a file leaves nothing behind for long, and entries of other live tabs on the same origin are left alone. Under the Node entry, which has no Web Locks, entries are named after the kernel's process id instead, and an entry is removed when that process is gone (`kill(pid, 0)` fails with `ESRCH`); a reused pid leaves the entry alone. Signals a program ignores stay ignored in the image it execs and in the children it spawns or forks, as POSIX has it, so `nohup cmd` and `trap '' TERM; exec cmd` keep `cmd` safe from them; caught signals go back to their default. The kernel holds what was handed down, so a new image that sets one of those signals back to the default with `signal(SIG_DFL)` still ignores it, until it installs a handler.
@@ -1087,7 +1105,7 @@ OPFS stores names, bytes, sizes and modification times, nothing else. Everything
 
 - mode bits (`chmod`; a `chmod` to the mode a file or directory already has writes nothing and leaves its ctime as it was, which saves a sidecar write for every installer that sets the mode a file was made with);
 - access and change times, and a modification time set explicitly with `utime` (it holds until the file is written again: any write or truncate moves it to the time of the write, also in Chrome, whose OPFS doesn't move a file's `lastModified` for writes through a sync access handle, so the kernel records the write's time itself, and it reaches the metadata store up to 100 ms after the write, so a tab closed inside that window keeps the old mtime on disk; the time follows a rename and is dropped with the file); a directory's times move whenever an entry is created, removed or renamed in it, as POSIX asks, and a time set explicitly after such a change keeps winning, also when the change was made in another kernel and its write lands later; an Emscripten program reads a directory's times again after any call that can wait on another process (a pipe or socket read, `wait`, `poll`, a sleep), so a program that waits for a child or a peer to fill a directory sees the new times (rsync's final directory times rely on it), while a walk without waits reads each directory once; `chown`/`fchown` leave the times alone (Emscripten stamps the setattr of a chown with the current time, which isn't taken as an mtime);
-- inode numbers that stay stable across renames. A file with no record has an inode derived from its path, and a rename keeps it on the file; the rename leaves a small record at the name it vacated with a fresh inode, so a file, symlink or directory made there later doesn't share the renamed one's ([#954](https://github.com/ai-ecoverse/slicc-kernel/issues/954)). An exclusive create doesn't count that record as a taken name, and the boot reconcile keeps it only while a live record still carries the inode it guards;
+- inode numbers that stay stable across renames. A file with no record has an inode derived from its path, and a rename keeps it on the file; the rename leaves a small record at the name it vacated with a fresh inode, so a file, symlink or directory made there later doesn't share the renamed one's ([#954](https://github.com/ai-ecoverse/slicc-kernel/issues/954)). Removing a directory leaves the same record, so a directory made at its name has a new inode, which is how a process's [working directory](#the-working-directory) notices the swap. The record becomes the new entry's own when something is created there; one with nothing at its name is pruned by the next boot that's alone on the root. An exclusive create doesn't count that record as a taken name, and the boot reconcile keeps it only while a live record still carries the inode it guards;
 - symbolic links (`symlink`, `readlink`, `lstat`, and following them in paths), which exist only in the sidecar, with their own times: `touch -h`, `lutimes` and `utimensat` with `AT_SYMLINK_NOFOLLOW` set the link's, never the target's. A link's own mode can't be changed, as on Linux: `chmod -h`, `lchmod` and `fchmodat` with `AT_SYMLINK_NOFOLLOW` fail with `EOPNOTSUPP` (which `chmod -h` and `tar` ignore for links) and never follow it, and `open` with `O_PATH | O_NOFOLLOW` gives a descriptor for the link itself.
 
 OPFS stays authoritative: an entry is consulted only for a path that exists in OPFS (symlinks only for paths that don't), so files written straight to OPFS, for example by the BIOS installer or a page, get the defaults: directories `0755`, files `0644`, and files under `node_modules/*/bin/` `0755`. A file deleted through the OPFS API disappears at once; its leftover entry is ignored and removed when the next kernel starts. Changes go to OPFS first and then to the sidecar, so a crash in between at worst loses the metadata of that one change. Each change is a read-modify-write inside one IndexedDB transaction, so two kernels (two tabs) on the same origin don't lose each other's updates.
@@ -1235,6 +1253,12 @@ Not part of the run yet: the rest of bios's setup (the `rest` shard, the upgrade
 
 The WASIX fixtures of `test/integration/epoll.test.mjs` (#674), under `test/integration/fixtures/`, are committed with their sources, build scripts and built wasm.
 
+### `heldcwd`
+
+`test/integration/fixtures/heldcwd`: a Rust wasm32-wasip1 program, built by `build.sh` with rustup's wasm32-wasip1 target, for the [working directory](#the-working-directory) rows (#1164).
+- `heldcwd`: lists `.`, waits for a line on stdin, then lists `.` again, tries a relative create and stat, and lists `..`.
+- `heldcwd loop N`: stats `old` N times, reads a line, then stats it N more times, for the count of `cwd-check` calls.
+
 ### `tokiotest`
 
 A small tokio program built for WASIX (`wasm32-wasmer-wasi`), the fixture for `test/integration/epoll.test.mjs` (#674). It runs on tokio's current-thread runtime, or on the multi-thread one (2 workers) with `--multi` first:

package/dist/kernel-worker.js

@@ -2307,6 +2307,9 @@ var OpfsFs = class {
       }
     }
     await this.meta.remove([found.path]);
+    if (found.handle?.kind === "directory") {
+      await this.meta.update(found.path, (entry) => entry ?? vacatedAt(found.path));
+    }
     this.touch(Date.now(), parentOf3(found.path));
   }
   links() {
@@ -7092,7 +7095,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.133.4" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.133.5" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -10785,6 +10788,88 @@ var FifoTable = class {
   }
 };
 
+// src/kernel/held-cwd.ts
+var CWD_SLOTS = 1024;
+var UNTRACKED = CWD_SLOTS - 1;
+var under4 = (path, changed) => changed === "/" || path === changed || path.startsWith(`${changed}/`);
+var CwdTable = class {
+  sab = new SharedArrayBuffer(CWD_SLOTS * 4);
+  view = new Int32Array(this.sab);
+  held = /* @__PURE__ */ new Map();
+  free = [];
+  next = 0;
+  checks = 0;
+  inodeOf;
+  constructor(inodeOf2) {
+    this.inodeOf = inodeOf2;
+  }
+  async hold(path, from) {
+    const shared = from === void 0 ? void 0 : this.held.get(from);
+    if (shared && shared.path === path) return this.share(from);
+    const ino = await this.inodeOf(path).catch(() => void 0);
+    const slot2 = this.free.pop() ?? (this.next < UNTRACKED ? this.next++ : UNTRACKED);
+    if (slot2 === UNTRACKED) return slot2;
+    this.held.set(slot2, {
+      path,
+      ino,
+      refs: 1,
+      dead: false,
+      verified: Atomics.load(this.view, slot2)
+    });
+    return slot2;
+  }
+  share(slot2) {
+    const held = slot2 === void 0 ? void 0 : this.held.get(slot2);
+    if (!held) return UNTRACKED;
+    held.refs++;
+    return slot2;
+  }
+  release(slot2) {
+    const held = slot2 === void 0 ? void 0 : this.held.get(slot2);
+    if (!held || --held.refs > 0) return;
+    this.held.delete(slot2);
+    Atomics.add(this.view, slot2, 1);
+    this.free.push(slot2);
+  }
+  stateOf(slot2) {
+    const held = this.held.get(slot2);
+    return { seen: held?.verified ?? Atomics.load(this.view, slot2), dead: held?.dead === true };
+  }
+  touched(paths, trees = []) {
+    if (this.held.size === 0) return;
+    for (const [slot2, held] of this.held) {
+      if (held.dead) continue;
+      if (paths.some((p) => under4(held.path, p)) || trees.some((t) => under4(held.path, t))) {
+        Atomics.add(this.view, slot2, 1);
+      }
+    }
+  }
+  async check(slot2) {
+    this.checks++;
+    const gen = Atomics.load(this.view, slot2);
+    const held = this.held.get(slot2);
+    if (!held) return { gen, alive: true };
+    if (!held.dead) {
+      const ino = await this.inodeOf(held.path).catch(() => null);
+      held.dead = ino === null || held.ino !== void 0 && ino !== held.ino;
+      held.verified = gen;
+    }
+    return { gen, alive: !held.dead };
+  }
+};
+function heldInit(o) {
+  if (!o.cwds || o.cwdSlot === void 0) return {};
+  const state = o.cwdState ?? { seen: 0, dead: false };
+  return { held: { sab: o.cwds, slot: o.cwdSlot, ...state } };
+}
+function heldHooks(o) {
+  if (!o.cwdCheck || !o.cwdChdir) return {};
+  return { cwdCheck: o.cwdCheck, cwdChdir: o.cwdChdir };
+}
+function inheritedCwd(chdir, parent) {
+  return chdir ? void 0 : parent;
+}
+
 // src/fs/copy.ts
 async function present2(fs, path) {
   return fs.lstat(path).then(
@@ -21142,6 +21227,8 @@ var SYSCALL_OPS = /* @__PURE__ */ new Set([
   "fd-list",
   "proc-alarm",
   "proc-umask",
+  "cwd-check",
+  "cwd-chdir",
   "proc-cred",
   "proc-setcred",
   "dl-log",
@@ -22467,6 +22554,14 @@ var WasmProcess = class {
         return { ok: true, kind: "void" };
       case "proc-umask":
         return { ok: true, kind: "json", json: this.setUmask(req.mask) };
+      case "cwd-check":
+        return {
+          ok: true,
+          kind: "json",
+          json: await this.options.cwdCheck?.() ?? { gen: 0, alive: true }
+        };
+      case "cwd-chdir":
+        return { ok: true, kind: "json", json: await this.options.cwdChdir?.(req.path) ?? {} };
       case "proc-cred":
         return { ok: true, kind: "json", json: copyCred(this.cred) };
       case "proc-setcred":
@@ -23016,6 +23111,7 @@ function processInit(opts, sab, umask, cred, lane) {
     umask,
     cred,
     ...opts.changes ? { changes: opts.changes } : {},
+    ...heldInit(opts),
     ...opts.owners ? { owners: true } : {},
     ...opts.links !== void 0 ? { links: true } : {},
     ...opts.ppid !== void 0 ? { ppid: opts.ppid } : {}
@@ -23089,6 +23185,7 @@ function processOptions(opts, header3, hooks, created4) {
     ...opts.identity ? { identity: opts.identity } : {},
     ...opts.onSyscall ? { onSyscall: opts.onSyscall } : {},
     ...opts.settling ? { settling: opts.settling } : {},
+    ...heldHooks(opts),
     spawner: opts.spawner,
     forker: opts.forker,
     ...opts.fork?.dl ? { dl: opts.fork.dl } : {},
@@ -23139,6 +23236,7 @@ function threadInit(opts, program, sab, cred, thread, main) {
     owners: opts.owners === true,
     ...opts.links !== void 0 ? { links: true } : {},
     ...opts.changes ? { changes: opts.changes } : {},
+    ...heldInit(opts),
     ...opts.ppid !== void 0 ? { ppid: opts.ppid } : {},
     thread,
     signals: main
@@ -26770,6 +26868,11 @@ var Launcher = class {
   hostname;
   watchers = new FsWatchers();
   changes = new FileChanges();
+  cwds = new CwdTable(async (path) => {
+    const { dev, ino } = await this.base.stat(path);
+    return `${dev ?? 0}:${ino}`;
+  });
+  cwdSlots = /* @__PURE__ */ new Map();
   pnpmHome;
   pnpmChain;
   pnpmMade;
@@ -26826,7 +26929,10 @@ var Launcher = class {
     this.sealed.tickets.normalise = (path) => physicalPathAsync(this.base, path);
     this.sealed.userOf = async (uid) => (await this.users.lookup(uid))?.name;
     if (this.owners) options.fs.trackOwners?.();
-    this.watchers.tap((paths) => this.changes.publish(...paths));
+    this.watchers.tap((paths) => {
+      this.changes.publish(...paths);
+      this.cwds.touched(paths);
+    });
     this.watchers.onReshape(() => {
       for (const nodes of this.openFiles) nodes.reshaped();
     });
@@ -26896,6 +27002,7 @@ var Launcher = class {
     options.fs.onPeer?.(() => this.flushOpen());
     options.fs.remoteChanges?.((change) => {
       this.changes.flush();
+      this.cwds.touched(change.paths, change.trees);
       this.rewritten(change.paths, change.trees);
     });
     const holders = [this.held, this.openFiles];
@@ -27678,11 +27785,12 @@ var Launcher = class {
     }
     const done5 = this.heldUrl(program);
     try {
+      const cwdSlot = await this.cwds.hold(cwd, req.cwdFrom);
       const handle = await this.admitted(
         program.abi !== "js",
         req.fds,
-        (fds, quietOom) => this.start({ ...req, fds, quietOom, cwd, program, argv0: target.argv0, args, env })
-      );
+        (fds, quietOom) => this.start({ ...req, fds, quietOom, cwd, program, argv0: target.argv0, args, env, cwdSlot })
+      ).finally(() => this.cwds.release(cwdSlot));
       void handle.exited.finally(done5);
       return handle;
     } catch (err) {
@@ -27769,6 +27877,8 @@ var Launcher = class {
       throw new SpawnError("ESRCH");
     }
     const pid = req.pid ?? this.nextPid++;
+    const cwdSlot = this.cwds.share(req.cwdSlot);
+    this.cwdSlots.set(pid, cwdSlot);
     const scrubs = this.treeScrubs(req);
     const stack = this.stackFor(req.tag);
     const terminal = req.fds.stdioTerminal();
@@ -27803,6 +27913,7 @@ var Launcher = class {
       ...this.dac ? { dac: this.dac } : {},
       program: req.program,
       changes: this.changes.sab,
+      ...this.heldFor(pid, cwdSlot),
       argv0: req.argv0,
       args: req.args,
       env: req.env,
@@ -27877,6 +27988,8 @@ var Launcher = class {
       (state) => state === "stopped" ? this.stopped.add(pid) : this.stopped.delete(pid)
     );
     void handle.exited.then(() => {
+      this.cwds.release(this.cwdSlots.get(pid));
+      this.cwdSlots.delete(pid);
       this.freshModules.forget(pid);
       const spent = handle.times?.();
       if (spent) this.spent.set(pid, spent);
@@ -28108,6 +28221,7 @@ var Launcher = class {
       const handle = await this.launch(planned, {
         ...deciding ? { decided: deciding.promise } : {},
         cwd: req.cwd,
+        cwdFrom: inheritedCwd(req.chdir, this.cwdSlots.get(ppid)),
         ...redeemed,
         fds,
         report,
@@ -28169,6 +28283,24 @@ var Launcher = class {
       for (const resolve2 of waiting) resolve2(true);
     }, 0);
   }
+  heldFor(pid, slot2) {
+    return {
+      cwds: this.cwds.sab,
+      cwdSlot: slot2,
+      cwdState: this.cwds.stateOf(slot2),
+      cwdCheck: async () => {
+        const current = this.cwdSlots.get(pid);
+        return { ...await this.cwds.check(current), slot: current };
+      },
+      cwdChdir: (path) => this.chdirHeld(pid, path)
+    };
+  }
+  async chdirHeld(pid, path) {
+    this.cwds.release(this.cwdSlots.get(pid));
+    const slot2 = await this.cwds.hold(path);
+    this.cwdSlots.set(pid, slot2);
+    return { slot: slot2, ...this.cwds.stateOf(slot2) };
+  }
   forker(ppid, parent) {
     const { exec: _exec, pgid: _pgid, group: _group, pid: _pid, scrub: _scrub, ...image } = parent;
     return async (state, given) => {
@@ -28176,6 +28308,7 @@ var Launcher = class {
         ...image,
         quietOom,
         cwd: state.cwd ?? parent.cwd,
+        cwdSlot: this.cwdSlots.get(ppid),
         fds,
         ppid,
         fork: state,