@ai-ecoverse/slicc-kernel 1.168.0 → 1.168.1

package/README.md

@@ -1260,6 +1260,21 @@ OPFS stays authoritative: an entry is consulted only for a path that exists in O
 - **A holder that ends** (its tab closes or its worker dies) releases its locks, as Web Locks do, so execs go on with the files as it left them, and the next installer run repairs them.
 - **Under the Node entry,** Node's `navigator.locks` (26.11) is per thread, although its documentation says it spans worker threads: a lock taken in one thread neither holds nor shows in another. So call `replacing` on the thread that runs the kernel. A Node kernel names its locks with the default `slicc-kernel`.
 
+**Swapping a package inside the kernel** ([#1705](https://github.com/ai-ecoverse/slicc-kernel/issues/1705)): pnpm, run inside the kernel (`kernel.run(['pnpm', 'install', …])`), replaces a package in a hoisted `node_modules` by staging the new copy in a sibling directory, renaming the old one aside, renaming the new one in and then deleting the old one. For the time between the two renames the package isn't there, and an exec that reads its glue before the rename in and its wasm after would get a mix. So the kernel takes the same `replacing` lock for the swap:
+
+- **The rename aside:** a rename that moves a package directory (the shape an exec locks, above) somewhere else first takes that directory's exclusive `slicc-kernel-replacing:v1:<metadata>:<directory>` lock, so execs that are loading from it finish first. A tab that holds the lock shared parks the swap right there, with the package still whole: an exec started after that queues behind the swap (Web Locks let a waiting exclusive request go before later shared ones, even with `ifAvailable`) and gets the new files. When the moved directory has a `package.json`, the kernel keeps the lock after the rename: the swap is open.
+- **The rename in:** a rename onto the open name runs under that lock and ends the swap; the kernel then releases it. A rename onto it that fails leaves the swap open. A staged copy landing on its name from a sibling named like pnpm's or npm's stages (`<name>_pacquet-stage_…`, `<name>_tmp_…` or `.<name>-…`) takes no lock of its own and opens no swap, so pnpm's first try, a new package's stage and a rename in after the bound cost nothing.
+- **The bound:** a swap with no rename onto its name within 2 s ends then, and execs that waited for it go on (and find the package missing, as they would have at once without the lock).
+- **Execs in the kernel:** `run`, `openTerminal`, every spawn, and a new scan of the command catalog, wait while one of the kernel's own swaps is open, and start once it ends. So a program run by name or by path between the two renames, a pnpm lifecycle script included, gets the new files, never `ENOENT` or the copy that was renamed aside. Other kernels sharing the root plan from the catalog they have and wait on the package's lock.
+- **Cost:** nothing per exec, and per rename two string tests; a rename aside of a package makes one lock request and one `stat` of its `package.json`, the rename in none.
+- **`/proc/sys/kernel/slicc_package_swaps`** reads four numbers: the swaps the kernel held, the longest in ms, the times an exec or a catalog scan waited for one, and the ms they waited in all.
+- **Not covered:**
+  - an exec **by path** in another kernel sharing the root, between the two renames: it doesn't see the swap and can get `ENOENT`;
+  - a `run` or `openTerminal` by name in another kernel between the two renames: it scans the catalog afresh and can run the copy renamed aside (the old files, whole);
+  - a rename of a directory above packages (`mv node_modules x`): it locks nothing;
+  - an installer that removes a package (`rm -r`) and writes it again, instead of swapping it in: its execs can see it missing or half written. One that moves it aside and writes the new one in place holds execs for the 2 s, and after that they can see it half written;
+  - a plain `mv pkg elsewhere` (a package moved away for good): it holds execs in the kernel for up to 2 s, after which that package's are missing.
+
 The database is `slicc-kernel` (or the `metadata` option), schema version 1, with one object store, `entries`, keyed by absolute path (`keyPath: 'path'`) and an index `dir` on the parent directory of symlinks. A record looks like `{ path, mode?, atimeMs?, mtimeMs?, mtimeFor?, ctimeMs?, ino?, link?, dir?, uid?, gid?, heir?, writes?, epoch? }`, where `mtimeFor` is the OPFS `lastModified` an explicit `mtimeMs` belongs to, `uid`, `gid` and `heir` (`[uid, gid]`) are the owners of [File owners](#file-owners), `writes` counts the writes kernels with `integrity` made to a protected file (see [Protected paths](#protected-paths)), whose `lastModified` OPFS leaves alone for a write through a sync access handle, and `epoch`, on the entry of `/`, counts their renames and symlinks at or above a protected path. A future schema bumps the version and migrates in `onupgradeneeded`; an older kernel then closes its connection. The sidecar is keyed by path from the `root` handle, so use a different `metadata` name for each distinct root.
 
 Not covered: hard links (OPFS cannot share contents between names; they work only within a mount whose driver offers `links`, such as `tmpfs`). File owners are stored only with `users: 'report'` or `'enforce'` (see [File owners](#file-owners)). With `users: 'off'`, an Emscripten process built against the kernel's credential calls (its glue imports `slicc_cred_js`, as homescoop's H1 builds do) sees every file as owned by its own effective uid and gid: root's 0 by default, a user's own ids when it runs as one, and the new ids after a `setuid`. Owner checks such as git's `safe.directory`, ssh's key checks and gnupg's home directory check then pass, as WASIX programs already see through their libc. An older program, whose libc says it is uid 1000 whoever runs it, sees every file as owned by 1000:1000 as before, so its own checks pass too. The kernel decides this once, when it loads the program. Emscripten also doesn't enforce permission bits, so a file runs whether or not its exec bit is set; the bits are what `ls`, `stat` and `test` report. With `users: 'off'` a file or directory with no permission bits at all shows as `0666` or `0777` to Emscripten programs; with owners on they see the real mode (`0`), and the kernel, not Emscripten's own checks, decides who may read, write or enter it (`cd` asks the kernel for `x` on the directory), so root still enters a `000` directory.

package/dist/kernel-worker.js

@@ -3082,15 +3082,15 @@ var OpfsFs = class {
     const hooks = {};
     const staging = swap ? `.${name2}.${crypto.randomUUID()}` : name2;
     const copying = sourceStat.isDirectory && !movable(source.handle);
-    const staged = resolvePath(parent.path, staging);
+    const staged2 = resolvePath(parent.path, staging);
     const place = async () => {
       this.forget(destination);
       const copies = /* @__PURE__ */ new Map();
       const root = { ino: inodeOf(source.path), ...moving };
       const owned4 = lock ? { lock } : {};
       hooks.notify?.();
-      const first = copying ? await this.intend({ from: source.path, to: staged, root, ...owned4 }) : void 0;
-      const second = copying && swap ? await this.intend({ from: staged, to: destination, replace: true, root, ...owned4 }) : void 0;
+      const first = copying ? await this.intend({ from: source.path, to: staged2, root, ...owned4 }) : void 0;
+      const second = copying && swap ? await this.intend({ from: staged2, to: destination, replace: true, root, ...owned4 }) : void 0;
       await hooks.started?.().catch(async (err) => {
         for (const intent of [first, second]) {
           if (intent) await this.meta.update(intent, () => void 0);
@@ -3101,11 +3101,11 @@ var OpfsFs = class {
         targetStat !== null,
         () => this.place(source.handle, dir, staging, copies)
       ).catch(async (err) => {
-        if (first) await this.rollBack(staged, source.path, [first, second]);
+        if (first) await this.rollBack(staged2, source.path, [first, second]);
         throw err;
       });
       if (swap) await this.rm(destination, { recursive: true });
-      if (second) await this.intend({ from: staged, to: destination, root, ...owned4 }, second);
+      if (second) await this.intend({ from: staged2, to: destination, root, ...owned4 }, second);
       await this.meta.move(source.path, destination, carry, left);
       this.movePending(source.path, destination);
       if (!moved2) {
@@ -7833,15 +7833,15 @@ var UserDb = class {
         (r) => r !== row && this.fs.resolvePath("/", r.home) === home
       );
       const removeHome = !options.keepHome && !shared2 && HOME_DIR.test(home) && await this.fs.exists(home);
-      const staged = `/home/.${name2}.removed-${Date.now()}`;
+      const staged2 = `/home/.${name2}.removed-${Date.now()}`;
       await this.commit(
         formatPasswd(passwd.filter((r) => r !== row)),
         formatGroup(keptGroups),
-        removeHome ? () => this.fs.rename(home, staged) : void 0
+        removeHome ? () => this.fs.rename(home, staged2) : void 0
       );
       this.umasks.delete(name2);
       if (removeHome)
-        await this.fs.rm(staged, { recursive: true, force: true }).catch(() => void 0);
+        await this.fs.rm(staged2, { recursive: true, force: true }).catch(() => void 0);
       return true;
     });
   }
@@ -8497,7 +8497,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.168.0" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.168.1" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -16299,6 +16299,8 @@ var SYS_TREE = {
     slicc_opfs_ops: (listing) => `${listing.opfsOps ?? 0}
 `,
     slicc_cwd_checks: (listing) => `${listing.cwdChecks ?? 0}
+`,
+    slicc_package_swaps: ({ swaps }) => `${swaps?.swaps ?? 0} ${(swaps?.longestMs ?? 0).toFixed(1)} ${swaps?.waits ?? 0} ${(swaps?.waitedMs ?? 0).toFixed(1)}
 `,
     slicc_rename_ack_busy: (listing) => `${listing.renameAcks?.busy ?? 0}
 `,
@@ -29015,6 +29017,105 @@ function replayingHops(inner, heldMs = HELD_MS) {
   };
 }
 
+// src/kernel/package-swaps.ts
+var SWAP_GAP_MS = 2e3;
+var STAGES = ["_tmp_", "_pacquet-stage_"];
+function staged(source, target) {
+  if (parentOf4(source) !== parentOf4(target)) return false;
+  const name2 = target.slice(target.lastIndexOf("/") + 1);
+  const from = source.slice(source.lastIndexOf("/") + 1);
+  return from.startsWith(`.${name2}-`) || STAGES.some((stage) => from.startsWith(`${name2}${stage}`));
+}
+var PackageSwaps = class {
+  gaps = /* @__PURE__ */ new Map();
+  isRoot;
+  hold;
+  closed;
+  gapMs;
+  tally = { swaps: 0, longestMs: 0, waits: 0, waitedMs: 0 };
+  constructor(isRoot, hold, closed, gapMs = SWAP_GAP_MS) {
+    this.isRoot = isRoot;
+    this.hold = hold;
+    this.closed = closed;
+    this.gapMs = gapMs;
+  }
+  counts() {
+    return { ...this.tally };
+  }
+  settled() {
+    if (this.gaps.size === 0) return Promise.resolve();
+    const started = performance.now();
+    this.tally.waits++;
+    return Promise.all([...this.gaps.values()].map((gap) => gap.settled)).then(() => {
+      this.tally.waitedMs += performance.now() - started;
+    });
+  }
+  wrap(fs) {
+    return { ...fs, rename: (from, to) => this.rename(fs, from, to) };
+  }
+  async rename(fs, from, to) {
+    const source = fs.resolvePath("/", from);
+    const target = fs.resolvePath("/", to);
+    const gap = this.gaps.get(target);
+    if (gap) return this.land(gap, target, () => fs.rename(from, to));
+    if (source === target || this.gaps.has(source) || staged(source, target) || !this.isRoot(source)) {
+      return fs.rename(from, to);
+    }
+    const release = await this.hold(source);
+    let kept = false;
+    try {
+      await fs.rename(from, to);
+      kept = await fs.stat(`${target}/package.json`).then(
+        (st) => st.isFile,
+        () => false
+      );
+      if (kept) this.open(source, release);
+    } finally {
+      if (!kept) release();
+    }
+  }
+  open(root, release) {
+    const { promise, resolve: resolve2 } = Promise.withResolvers();
+    const gap = {
+      release,
+      done: () => resolve2(),
+      settled: promise,
+      timer: void 0,
+      landing: 0,
+      opened: performance.now()
+    };
+    this.gaps.set(root, gap);
+    this.tally.swaps++;
+    this.arm(root, gap);
+  }
+  arm(root, gap) {
+    const timer = setTimeout(() => this.close(root, gap), this.gapMs);
+    timer.unref?.();
+    gap.timer = timer;
+  }
+  close(root, gap) {
+    if (this.gaps.get(root) !== gap) return;
+    clearTimeout(gap.timer);
+    this.gaps.delete(root);
+    this.tally.longestMs = Math.max(this.tally.longestMs, performance.now() - gap.opened);
+    this.closed();
+    gap.done();
+    gap.release();
+  }
+  async land(gap, root, run2) {
+    gap.landing++;
+    clearTimeout(gap.timer);
+    try {
+      await run2();
+    } catch (err) {
+      gap.landing--;
+      if (gap.landing === 0 && this.gaps.get(root) === gap) this.arm(root, gap);
+      throw err;
+    }
+    this.close(root, gap);
+  }
+};
+
 // src/kernel/rename-log.ts
 var RENAME_LOG_SIZE = 4096;
 function within9(path, root) {
@@ -29083,6 +29184,18 @@ var PackageLocks = class {
     const names2 = roots.map((root) => replacingLock(this.metadata, root));
     return shared(this.locks, names2, run2, false);
   }
+  holding(root) {
+    const { locks } = this;
+    if (!locks) return Promise.resolve(() => {
+    });
+    const { promise: held, resolve: release } = Promise.withResolvers();
+    return new Promise((granted2, failed3) => {
+      locks.request(replacingLock(this.metadata, root), { mode: "exclusive" }, () => {
+        granted2(() => release());
+        return held;
+      }).catch(failed3);
+    });
+  }
 };
 
 // src/kernel/settling.ts
@@ -31592,6 +31705,11 @@ var Launcher = class {
   jobs = new JobTable();
   locks;
   packageLocks;
+  swaps = new PackageSwaps(
+    (path) => this.packageRootIs(path),
+    (root) => this.packageLocks.holding(root),
+    () => this.resetCatalogs()
+  );
   fifos = new FifoTable();
   memories;
   sems = new SemTable();
@@ -31882,7 +32000,7 @@ var Launcher = class {
   commands(scope = "") {
     let found = this.catalogs.get(scope);
     if (!found) {
-      found = this.trustedAppended().then(async (appended2) => {
+      found = this.swaps.settled().then(() => this.trustedAppended()).then(async (appended2) => {
         const tiers = await this.tiers(scope, appended2);
         const refused3 = [];
         const commands = await scanCommands(
@@ -32727,6 +32845,13 @@ var Launcher = class {
     if (readsEtcHostname(program)) setOrDrop(env, "HOSTNAME", own);
     return { planned, env, program, exeId };
   }
+  packageRootIs(path) {
+    const at = normalizePath(path);
+    const within10 = `${at}/`;
+    const roots = [this.modulesDir, ...this.appended];
+    const root = PACKAGE_ROOT.exec(within10)?.[1] ?? packageRootOf(within10, roots);
+    return root === at;
+  }
   packageRoots(planned) {
     const packages2 = [this.modulesDir, ...this.appended];
     const roots = /* @__PURE__ */ new Set();
@@ -32912,6 +33037,7 @@ var Launcher = class {
         opfsOps: this.rootFs.opfsOps?.() ?? 0,
         cwdChecks: this.cwds.checks,
         renameAcks: this.rootFs.renameWaits?.(),
+        swaps: this.swaps.counts(),
         staleFds: staleFdCount(),
         fileIds: this.rootFs.fileIdWrites?.() ?? 0,
         holds: this.rootFs.holdCounts?.()
@@ -33168,6 +33294,7 @@ var Launcher = class {
   spawner(ppid, report) {
     return async (req, fds, grant) => {
       const booked = this.takeTicket(req);
+      await this.swaps.settled();
       if (grant && !await this.grantedCwd(grant.caller, req.cwd)) {
         await fds.closeAll();
         throw new SpawnError("EACCES");
@@ -34080,10 +34207,12 @@ var Launcher = class {
     nodes.changed = (path) => this.changes.publish(path);
     nodes.movedAway = (from, to) => this.moved(from, to, nodes);
     nodes.dirtied = (delta) => this.changes.dirtied(delta);
-    return unlinked.wrap(keepingOpen(fs, nodes, (path) => this.changes.publish(path)));
+    const kept = keepingOpen(fs, nodes, (path) => this.changes.publish(path));
+    return this.swaps.wrap(unlinked.wrap(kept));
   }
   async starting(argv, dir, account, tags, base) {
     const tag = tags?.[0];
+    await this.swaps.settled();
     this.resetCatalogs();
     const scope = account.uid === 0 ? "" : pnpmHomeOf(account, this.pnpmHome);
     const start2 = dir ?? "/";