@ai-ecoverse/slicc-kernel 1.134.4 → 1.135.0

package/README.md

@@ -1099,6 +1099,7 @@ The kernel writes `/etc/passwd` and `/etc/group` when it starts (see [Users](#us
 In a browser every process worker of the page shares one renderer. V8 reserves about 8 GiB of address space for each 32-bit wasm memory, whatever its declared maximum, and its sandbox holds about 1 TiB. So a page runs about 125 wasm processes (or programs that own a wasm memory) at once, counting everything in the renderer (#970). Node has no such ceiling, and neither does Linux. The kernel keeps processes inside that ceiling:
 - **The ceiling:** 112 by default in a browser and none in Node (`createNodeKernel({ wasmMemories: { ceiling, idleMs } })` sets one in Node; `idleMs`, the quiet spell before `EAGAIN` below, 10 s by default, is there for tests). It's the kernel's count of live wasm processes, with the kernel's other wasm memories in the page left outside it. When an instantiation runs out of memory anyway, the kernel learns the real ceiling at that moment (the live count minus one), and that learned value wins over the default. A learned ceiling is never below 16, since an OOM at a low count means something outside it (another tab in the same renderer) filled the renderer. It's raised by 8 for each minute without another OOM, back to the default.
 - **Admission:** a spawn, fork, exec or `kernel.run` at the ceiling waits for another process to exit. It keeps waiting as long as processes exit, are admitted or finish starting (a process becoming ready counts, so a slow host that is still booting the processes ahead of it doesn't fail the spawn, #1196), and fails with `EAGAIN` only after 10 s with none of these. A long `make -j` or `xargs -P` therefore queues, and a pipeline that needs more live processes than the ceiling fails cleanly. bash's own retry on a fork's `EAGAIN` (`fork: retry: Resource temporarily unavailable`) handles the rest.
+- **JS workers count too** (#1133): each JS process and each `thread.spawn` thread takes one slot, whether or not it creates wasm memories, so a worker that creates several under-counts within a small factor, and a shared memory posted to threads is one reservation however many threads use it. A JS worker also costs the renderer about 65 MB of RAM, which the same bound keeps in check. `thread.spawn` waits for a slot as a spawn does, and rejects with `EAGAIN` after the same quiet spell. With many JS workers live, wasm processes wait earlier, which is deliberate. Node has no ceiling, so nothing changes there, and the per-process thread cap (`SLICC_JS_THREADS`) still applies on top.
 - **Close to the ceiling** (within 16 of it), the kernel waits for each new process to finish instantiating before the spawn returns. If it ran out of memory, the kernel starts it again, quietly, once there's room, so the caller sees no failure.
 - **The rare residual case:** a process that fails to instantiate with nobody waiting on it, which happens only when something outside the count fills the renderer. It ends with status 126 and `<program>: cannot allocate memory for a new process (too many processes at once)` on its stderr, never a JS `RangeError`.
 - **Below a learned ceiling,** a process that still runs out of memory is started again once another process exits, and the spawn fails with `EAGAIN` after 10 s with none.
@@ -1247,7 +1248,7 @@ The tools in `tools/cert-review/`:
   - and record, which re-hashes the candidate's artifact against the pack job's sha256.
 - **The verdict** goes into the commit status on the pull request's head and a comment on the pull request. Certified is green. A FAIL is red with the line, and the pull request needs a fix. **INCOMPLETE** (a job that didn't succeed, no verdict, a suite without results) is red with `re-run the failed jobs`, never a silent pass. Main moving on after a review doesn't trigger another one; a push does.
 - **Merge groups only look up.** On `merge_group`, the job lists the group's pull requests (its merge commits) and requires, for each one, a green status on the head being merged, or no behaviour change. Otherwise it's red, naming the pull request. Then it adds the group's downstream verdict. The group's `ci / test` covers the combined tree.
-- **Downstream, once per merge group:** the `merge group pack` job packs the group's tree with `main`'s tools (`x.y.z-group.<run>`), and `downstream bios canary` calls slicc-bios's `canary.yml` at a pinned full sha (`c2cec127…`, never a tag) with that tarball, bound by artifact id, digest and sha256, on the `sweep`, `guard` and `k4-link` shards, each run once. `downstream.mjs` judges its result (`downstreamOf` in `lib.mjs`), and the lookup requires it. The `merge group` job reads that result as `result.json` from the canary's own artifact, downloaded by the call's `artifact-id` with `digest-mismatch: error`, not from the `result` output: that output can be up to 512 KB, and one environment variable holds 128 KB, so passing it that way stopped the step from starting at all (`Argument list too long`, runs 39166647175 and 39168823799). A missing or unreadable artifact leaves no result, so the group is INCOMPLETE:
+- **Downstream, once per merge group:** the `merge group pack` job packs the group's tree with `main`'s tools (`x.y.z-group.<run>`), and `downstream bios canary` calls slicc-bios's `canary.yml` at a pinned full sha (`1f071319…`, never a tag) with that tarball, bound by artifact id, digest and sha256, on the `sweep`, `guard-prompt`, `guard-lagging`, `k4-upgrade`, `k4-agent-modes` and `link-serve` shards, each run once, on the runner the `canary runner` job picks: `.github/runner.mjs`'s `decide()` from `main`, as `ci.yml` uses it, gives `vars.CERT_CANARY_RUNNER` only to a merge group whose entries are all same-repo PRs, and `ubuntu-latest` otherwise, without the variable, or if the decider fails (bios honours a runner only for merge groups and pushes to main in ai-ecoverse, so rolling back is unsetting the variable). The verdict line names the shards, or their count past 32 characters (`6 shards`), so a status description (140 characters) keeps the reason. `downstream.mjs` judges its result (`downstreamOf` in `lib.mjs`), and the lookup requires it. The `merge group` job reads that result as `result.json` from the canary's own artifact, downloaded by the call's `artifact-id` with `digest-mismatch: error`, not from the `result` output: that output can be up to 512 KB, and one environment variable holds 128 KB, so passing it that way stopped the step from starting at all (`Argument list too long`, runs 39166647175 and 39168823799). A missing or unreadable artifact leaves no result, so the group is INCOMPLETE:
   - **PASS:** the candidate passes, `downstream: PASS bios … N pass`;
   - **FAIL:** a regression, or a failing test (by the counts, even without its rows) is **not certified**, and the group is red;
   - **INCOMPLETE** blocks too: no result, a result from another bios than the pinned one, a missing shard, a shard that exited without a failing test, a refused candidate, 0 tests run, or no test passed;

package/dist/kernel-worker.js

@@ -7098,7 +7098,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.134.4" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.135.0" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -23100,23 +23100,55 @@ var JsThreads = class {
   get size() {
     return this.live.size;
   }
+  pending = 0;
+  get admission() {
+    return this.deps.admission;
+  }
   spawn(msg) {
-    const { control, port } = msg;
-    if (this.closed || this.live.size >= this.deps.cap) {
-      answer(
-        control,
+    if (this.closed || this.live.size + this.pending >= this.deps.cap) {
+      this.refuse(
+        msg,
         this.closed ? { errno: "ESRCH", message: "the process is ending" } : { errno: "EAGAIN", message: `more than ${this.deps.cap} threads` }
       );
-      control.close();
-      port.close();
       return;
     }
+    this.pending++;
+    this.admission.admit().then(
+      () => {
+        this.pending--;
+        if (!this.closed) {
+          let worker;
+          try {
+            worker = this.deps.createWorker();
+          } catch {
+            this.admission.ended();
+            this.refuse(msg, { errno: "EAGAIN", message: "no worker for the thread" });
+            return;
+          }
+          this.begin(msg, worker);
+          return;
+        }
+        this.admission.ended();
+        this.refuse(msg, { errno: "ESRCH", message: "the process is ending" });
+      },
+      () => {
+        this.pending--;
+        this.refuse(msg, { errno: "EAGAIN", message: "no room for another worker" });
+      }
+    );
+  }
+  refuse(msg, why) {
+    answer(msg.control, why);
+    msg.control.close();
+    msg.port.close();
+  }
+  begin(msg, worker) {
+    const { control, port } = msg;
     const tid = ++this.last;
     const sab = new SharedArrayBuffer(SAB_HEADER_BYTES + SAB_DEFAULT_WINDOW_BYTES);
     const header3 = new Int32Array(sab, 0, SAB_HEADER_I32);
     const owned4 = new SharedArrayBuffer(4 * (1 + JS_THREAD_OWNED_FDS));
     if (this.frozen) Atomics.store(header3, SAB_I_STOP, 1);
-    const worker = this.deps.createWorker();
     const clock = this.deps.clocks.thread(tid + 1, sab);
     let ended = false;
     const end = (code) => {
@@ -23135,6 +23167,7 @@ var JsThreads = class {
       responder.dispose();
       lane.dispose();
       this.deps.clocks.threadEnded(tid + 1);
+      this.admission.ended();
       worker.terminate();
       for (const fd of ownedFds(owned4)) {
         void this.deps.dispatch({ op: "fd-close", fd });
@@ -23186,6 +23219,7 @@ var JsThreads = class {
       end(JS_THREAD_CRASHED);
       return;
     }
+    this.admission.started();
     answer(control, { tid });
   }
   crashed(message3, end) {
@@ -23680,7 +23714,8 @@ function jsThreadsOf(opts, process, token, dispatch2, clocks, header3) {
     openLane: (laneDispatch) => openLane(token, laneDispatch),
     attach: (worker, sab, threadDispatch) => attachSyncSabResponder(worker, sab, token, { dispatch: threadDispatch }),
     dispatch: dispatch2,
-    clocks
+    clocks,
+    admission: opts.admission
   });
   process.onState((state) => threads.stopped(state === "stopped"));
   return threads;
@@ -28082,7 +28117,6 @@ var Launcher = class {
     if (req.links !== void 0) this.links.set(pid, req.links);
   }
   async admitted(wasm, fds, start2) {
-    if (!wasm) return start2(fds);
     for (; ; ) {
       await this.memories.admit().catch(async (err) => {
         await fds.closeAll();
@@ -28093,7 +28127,7 @@ var Launcher = class {
         void Promise.resolve(handle2.ready).then(() => this.memories.started());
         return handle2;
       };
-      if (!this.memories.near()) {
+      if (!wasm || !this.memories.near()) {
         try {
           return counted2(start2(fds));
         } catch (err) {
@@ -28171,6 +28205,7 @@ var Launcher = class {
       noLinks: stack.noLinks,
       ...this.prefixes.length > 0 && !stack.writer ? { readonlyAt: (path) => isProtected(this.prefixes, path) } : {},
       createWorker: this.createWorker,
+      admission: this.memories,
       onError: req.report,
       onOom: () => {
         this.memories.learn();