@ai-ecoverse/slicc-kernel 1.134.3 → 1.135.0

package/README.md

@@ -859,13 +859,14 @@ A process's working directory is the directory it changed into, not just its pat
   - a wasip1 program's `.` (its `.` preopen) can't be opened or listed (`ENOENT`, an empty listing for its preopen descriptor);
   - its relative lookups and creates are `ENOENT`;
   - `..` still reaches the parent.
+  - an Emscripten program (bash, coreutils, …) and its forked subshells and children: relative paths resolve against `<path> (deleted)`, so `ls` fails with `cannot open directory '.'`, a glob matches nothing, and a relative create is `ENOENT`. `getcwd` is `ENOENT`, as on Linux, so a shell started there says `shell-init: error retrieving current directory` and its `pwd` fails, while bash's own `$PWD`, `cd ..` and `cd /abs` keep working.
+  - a JS program: `ctx.fs` and `ctx.sync.fs` with relative paths are `ENOENT`, and `../x` reaches the parent. `ctx.cwd` (and `process.cwd()`) keeps the plain path, as a WASIX program's `getcwd` does. This needs the sync lane, which JS programs have where `SharedArrayBuffer` is available.
 - **Detection costs nothing in the steady state:**
   - each held directory has a counter in memory the kernel shares with the process. It moves only when this kernel, another tab, or a `postFsNotice` reports a change at that directory or above it: the kernel's own watcher paths, and every path and tree of a peer change or notice.
   - a process checks the counter, a memory read, before each relative lookup. Only after it moved does it ask the kernel once (`cwd-check`), and the kernel compares the inode at the path with the one it recorded.
 - **The `rm` record:** so that `rm -r d; mkdir d` can be told apart, removing a directory leaves the same small record a rename leaves at the name it vacates (see [Metadata](#metadata)).
 - **Limits:**
   - a WASIX program's C library keeps the working directory as a string and turns relative paths into absolute ones before they reach the kernel. So a WASIX program still sees the new directory through relative paths, and `getcwd` still returns the path, until wasix-sysroot resolves relative paths against the working directory's descriptor ([#1164](https://github.com/ai-ecoverse/slicc-kernel/issues/1164) stays open for that).
-  - Emscripten and JS programs: [#1164](https://github.com/ai-ecoverse/slicc-kernel/issues/1164), next.
   - a removal between the check and the lookup is a race, as with any path, and the lookup sees the new directory.
   - on a `hostfs` folder the inode is the host's, which a host file system may reuse for the new directory (ext4 can; APFS doesn't). There the replacement isn't noticed.
   - with more than 1023 distinct working directories held at once, the rest are not tracked.
@@ -1098,12 +1099,13 @@ The kernel writes `/etc/passwd` and `/etc/group` when it starts (see [Users](#us
 In a browser every process worker of the page shares one renderer. V8 reserves about 8 GiB of address space for each 32-bit wasm memory, whatever its declared maximum, and its sandbox holds about 1 TiB. So a page runs about 125 wasm processes (or programs that own a wasm memory) at once, counting everything in the renderer (#970). Node has no such ceiling, and neither does Linux. The kernel keeps processes inside that ceiling:
 - **The ceiling:** 112 by default in a browser and none in Node (`createNodeKernel({ wasmMemories: { ceiling, idleMs } })` sets one in Node; `idleMs`, the quiet spell before `EAGAIN` below, 10 s by default, is there for tests). It's the kernel's count of live wasm processes, with the kernel's other wasm memories in the page left outside it. When an instantiation runs out of memory anyway, the kernel learns the real ceiling at that moment (the live count minus one), and that learned value wins over the default. A learned ceiling is never below 16, since an OOM at a low count means something outside it (another tab in the same renderer) filled the renderer. It's raised by 8 for each minute without another OOM, back to the default.
 - **Admission:** a spawn, fork, exec or `kernel.run` at the ceiling waits for another process to exit. It keeps waiting as long as processes exit, are admitted or finish starting (a process becoming ready counts, so a slow host that is still booting the processes ahead of it doesn't fail the spawn, #1196), and fails with `EAGAIN` only after 10 s with none of these. A long `make -j` or `xargs -P` therefore queues, and a pipeline that needs more live processes than the ceiling fails cleanly. bash's own retry on a fork's `EAGAIN` (`fork: retry: Resource temporarily unavailable`) handles the rest.
+- **JS workers count too** (#1133): each JS process and each `thread.spawn` thread takes one slot, whether or not it creates wasm memories, so a worker that creates several under-counts within a small factor, and a shared memory posted to threads is one reservation however many threads use it. A JS worker also costs the renderer about 65 MB of RAM, which the same bound keeps in check. `thread.spawn` waits for a slot as a spawn does, and rejects with `EAGAIN` after the same quiet spell. With many JS workers live, wasm processes wait earlier, which is deliberate. Node has no ceiling, so nothing changes there, and the per-process thread cap (`SLICC_JS_THREADS`) still applies on top.
 - **Close to the ceiling** (within 16 of it), the kernel waits for each new process to finish instantiating before the spawn returns. If it ran out of memory, the kernel starts it again, quietly, once there's room, so the caller sees no failure.
 - **The rare residual case:** a process that fails to instantiate with nobody waiting on it, which happens only when something outside the count fills the renderer. It ends with status 126 and `<program>: cannot allocate memory for a new process (too many processes at once)` on its stderr, never a JS `RangeError`.
 - **Below a learned ceiling,** a process that still runs out of memory is started again once another process exits, and the spawn fails with `EAGAIN` after 10 s with none.
 - **`/proc/sys/kernel/slicc_wasm_memories`** reads `<live> <ceiling> <source>`. The source is `default`, `learned`, or `none` (with the ceiling 0).
 
-`/proc/sys/kernel/slicc_shared_root` reads `1` while another kernel shares the root (a tab, or a second kernel in the page) and `0` otherwise. A peer whose kernel ends (its tab closes or reloads) stops counting at once: each kernel queues a request for every peer's alive Web Lock, and the grant tells it the peer has gone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)). `/proc/sys/kernel/slicc_fs_calls` reads the number of path calls (lookups, stats, opens, creates, removals and the like) the kernel has served its programs since it started. `/proc/sys/kernel/slicc_opfs_calls` counts the OPFS calls the kernel has made to look names up and read stats (`getFile`, `getFileHandle`, `getDirectoryHandle`, `resolve`). `/proc/sys/kernel/slicc_opfs_ops` counts every request the root file system makes of OPFS: those lookups, plus creates, writes (one per write, truncate or write session), reads, listings, renames and removals ([#1017](https://github.com/ai-ecoverse/slicc-kernel/issues/1017)). With an idle peer, a tar unpack makes at most 1.5 times the calls it makes alone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)).
+`/proc/sys/kernel/slicc_shared_root` reads `1` while another kernel shares the root (a tab, or a second kernel in the page) and `0` otherwise. A peer whose kernel ends (its tab closes or reloads) stops counting at once: each kernel queues a request for every peer's alive Web Lock, and the grant tells it the peer has gone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)). `/proc/sys/kernel/slicc_fs_calls` reads the number of path calls (lookups, stats, opens, creates, removals and the like) the kernel has served its programs since it started. `/proc/sys/kernel/slicc_opfs_calls` counts the OPFS calls the kernel has made to look names up and read stats (`getFile`, `getFileHandle`, `getDirectoryHandle`, `resolve`). `/proc/sys/kernel/slicc_opfs_ops` counts every request the root file system makes of OPFS: those lookups, plus creates, writes (one per write, truncate or write session), reads, listings, renames and removals ([#1017](https://github.com/ai-ecoverse/slicc-kernel/issues/1017)). With an idle peer, a tar unpack makes at most 1.5 times the calls it makes alone ([#984](https://github.com/ai-ecoverse/slicc-kernel/issues/984)). `/proc/sys/kernel/slicc_cwd_checks` counts the times a process asked the kernel whether its [working directory](#the-working-directory) is still the one it holds.
 
 ### Metadata
 
@@ -1246,7 +1248,7 @@ The tools in `tools/cert-review/`:
   - and record, which re-hashes the candidate's artifact against the pack job's sha256.
 - **The verdict** goes into the commit status on the pull request's head and a comment on the pull request. Certified is green. A FAIL is red with the line, and the pull request needs a fix. **INCOMPLETE** (a job that didn't succeed, no verdict, a suite without results) is red with `re-run the failed jobs`, never a silent pass. Main moving on after a review doesn't trigger another one; a push does.
 - **Merge groups only look up.** On `merge_group`, the job lists the group's pull requests (its merge commits) and requires, for each one, a green status on the head being merged, or no behaviour change. Otherwise it's red, naming the pull request. Then it adds the group's downstream verdict. The group's `ci / test` covers the combined tree.
-- **Downstream, once per merge group:** the `merge group pack` job packs the group's tree with `main`'s tools (`x.y.z-group.<run>`), and `downstream bios canary` calls slicc-bios's `canary.yml` at a pinned full sha (`c2cec127…`, never a tag) with that tarball, bound by artifact id, digest and sha256, on the `sweep`, `guard` and `k4-link` shards, each run once. `downstream.mjs` judges its result (`downstreamOf` in `lib.mjs`), and the lookup requires it. The `merge group` job reads that result as `result.json` from the canary's own artifact, downloaded by the call's `artifact-id` with `digest-mismatch: error`, not from the `result` output: that output can be up to 512 KB, and one environment variable holds 128 KB, so passing it that way stopped the step from starting at all (`Argument list too long`, runs 39166647175 and 39168823799). A missing or unreadable artifact leaves no result, so the group is INCOMPLETE:
+- **Downstream, once per merge group:** the `merge group pack` job packs the group's tree with `main`'s tools (`x.y.z-group.<run>`), and `downstream bios canary` calls slicc-bios's `canary.yml` at a pinned full sha (`1f071319…`, never a tag) with that tarball, bound by artifact id, digest and sha256, on the `sweep`, `guard-prompt`, `guard-lagging`, `k4-upgrade`, `k4-agent-modes` and `link-serve` shards, each run once, on the runner the `canary runner` job picks: `.github/runner.mjs`'s `decide()` from `main`, as `ci.yml` uses it, gives `vars.CERT_CANARY_RUNNER` only to a merge group whose entries are all same-repo PRs, and `ubuntu-latest` otherwise, without the variable, or if the decider fails (bios honours a runner only for merge groups and pushes to main in ai-ecoverse, so rolling back is unsetting the variable). The verdict line names the shards, or their count past 32 characters (`6 shards`), so a status description (140 characters) keeps the reason. `downstream.mjs` judges its result (`downstreamOf` in `lib.mjs`), and the lookup requires it. The `merge group` job reads that result as `result.json` from the canary's own artifact, downloaded by the call's `artifact-id` with `digest-mismatch: error`, not from the `result` output: that output can be up to 512 KB, and one environment variable holds 128 KB, so passing it that way stopped the step from starting at all (`Argument list too long`, runs 39166647175 and 39168823799). A missing or unreadable artifact leaves no result, so the group is INCOMPLETE:
   - **PASS:** the candidate passes, `downstream: PASS bios … N pass`;
   - **FAIL:** a regression, or a failing test (by the counts, even without its rows) is **not certified**, and the group is red;
   - **INCOMPLETE** blocks too: no result, a result from another bios than the pinned one, a missing shard, a shard that exited without a failing test, a refused candidate, 0 tests run, or no test passed;

package/dist/kernel-worker.js

@@ -7098,7 +7098,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.134.3" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.135.0" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -13662,7 +13662,7 @@ var AsyncRangedFile = class {
 };
 
 // src/realm/sync-sab-bridge.ts
-var PATH_OPS = new Set(SYNC_FS_OPS);
+var UNCOUNTED_OPS = /* @__PURE__ */ new Set([...SYNC_FS_OPS, "cwd-check", "cwd-chdir"]);
 var SPIN_CORES = 4;
 var spinning = (globalThis.navigator?.hardwareConcurrency ?? 0) >= SPIN_CORES;
 
@@ -13710,6 +13710,8 @@ var SYS_TREE = {
     slicc_opfs_calls: (listing) => `${listing.opfsCalls ?? 0}
 `,
     slicc_opfs_ops: (listing) => `${listing.opfsOps ?? 0}
+`,
+    slicc_cwd_checks: (listing) => `${listing.cwdChecks ?? 0}
 `
   }
 };
@@ -23098,23 +23100,55 @@ var JsThreads = class {
   get size() {
     return this.live.size;
   }
+  pending = 0;
+  get admission() {
+    return this.deps.admission;
+  }
   spawn(msg) {
-    const { control, port } = msg;
-    if (this.closed || this.live.size >= this.deps.cap) {
-      answer(
-        control,
+    if (this.closed || this.live.size + this.pending >= this.deps.cap) {
+      this.refuse(
+        msg,
         this.closed ? { errno: "ESRCH", message: "the process is ending" } : { errno: "EAGAIN", message: `more than ${this.deps.cap} threads` }
       );
-      control.close();
-      port.close();
       return;
     }
+    this.pending++;
+    this.admission.admit().then(
+      () => {
+        this.pending--;
+        if (!this.closed) {
+          let worker;
+          try {
+            worker = this.deps.createWorker();
+          } catch {
+            this.admission.ended();
+            this.refuse(msg, { errno: "EAGAIN", message: "no worker for the thread" });
+            return;
+          }
+          this.begin(msg, worker);
+          return;
+        }
+        this.admission.ended();
+        this.refuse(msg, { errno: "ESRCH", message: "the process is ending" });
+      },
+      () => {
+        this.pending--;
+        this.refuse(msg, { errno: "EAGAIN", message: "no room for another worker" });
+      }
+    );
+  }
+  refuse(msg, why) {
+    answer(msg.control, why);
+    msg.control.close();
+    msg.port.close();
+  }
+  begin(msg, worker) {
+    const { control, port } = msg;
     const tid = ++this.last;
     const sab = new SharedArrayBuffer(SAB_HEADER_BYTES + SAB_DEFAULT_WINDOW_BYTES);
     const header3 = new Int32Array(sab, 0, SAB_HEADER_I32);
     const owned4 = new SharedArrayBuffer(4 * (1 + JS_THREAD_OWNED_FDS));
     if (this.frozen) Atomics.store(header3, SAB_I_STOP, 1);
-    const worker = this.deps.createWorker();
     const clock = this.deps.clocks.thread(tid + 1, sab);
     let ended = false;
     const end = (code) => {
@@ -23133,6 +23167,7 @@ var JsThreads = class {
       responder.dispose();
       lane.dispose();
       this.deps.clocks.threadEnded(tid + 1);
+      this.admission.ended();
       worker.terminate();
       for (const fd of ownedFds(owned4)) {
         void this.deps.dispatch({ op: "fd-close", fd });
@@ -23184,6 +23219,7 @@ var JsThreads = class {
       end(JS_THREAD_CRASHED);
       return;
     }
+    this.admission.started();
     answer(control, { tid });
   }
   crashed(message3, end) {
@@ -23678,7 +23714,8 @@ function jsThreadsOf(opts, process, token, dispatch2, clocks, header3) {
     openLane: (laneDispatch) => openLane(token, laneDispatch),
     attach: (worker, sab, threadDispatch) => attachSyncSabResponder(worker, sab, token, { dispatch: threadDispatch }),
     dispatch: dispatch2,
-    clocks
+    clocks,
+    admission: opts.admission
   });
   process.onState((state) => threads.stopped(state === "stopped"));
   return threads;
@@ -28080,7 +28117,6 @@ var Launcher = class {
     if (req.links !== void 0) this.links.set(pid, req.links);
   }
   async admitted(wasm, fds, start2) {
-    if (!wasm) return start2(fds);
     for (; ; ) {
       await this.memories.admit().catch(async (err) => {
         await fds.closeAll();
@@ -28091,7 +28127,7 @@ var Launcher = class {
         void Promise.resolve(handle2.ready).then(() => this.memories.started());
         return handle2;
       };
-      if (!this.memories.near()) {
+      if (!wasm || !this.memories.near()) {
         try {
           return counted2(start2(fds));
         } catch (err) {
@@ -28169,6 +28205,7 @@ var Launcher = class {
       noLinks: stack.noLinks,
       ...this.prefixes.length > 0 && !stack.writer ? { readonlyAt: (path) => isProtected(this.prefixes, path) } : {},
       createWorker: this.createWorker,
+      admission: this.memories,
       onError: req.report,
       onOom: () => {
         this.memories.learn();
@@ -28187,7 +28224,8 @@ var Launcher = class {
         peers: Number(this.locks.shared?.() === true),
         fsCalls: servedFsCalls(),
         opfsCalls: this.rootFs.opfsCalls?.() ?? 0,
-        opfsOps: this.rootFs.opfsOps?.() ?? 0
+        opfsOps: this.rootFs.opfsOps?.() ?? 0,
+        cwdChecks: this.cwds.checks
       }),
       sudo: (call, signal) => this.sudo(pid, call, signal),
       crontab: (call) => this.crontab(pid, call),