@ai-ecoverse/slicc-kernel 1.124.0 → 1.124.2

package/README.md

@@ -735,7 +735,7 @@ Programs mount and unmount with `mount(2)` and `umount2(2)`, so a `mount`/`umoun
 
 ### Exec
 
-An exec'd program takes over the pid of the process it replaces, as on Linux: `getpid()` and `$$` in the new image, `/proc`, `ps`, `kill` and the parent's `waitpid` all agree on that pid, and the image's parent pid is the replaced process's. Its own children see that pid as their parent. A file removed or renamed over while programs hold it open keeps its bytes for them, as on Linux, whether the descriptors live in the kernel or a program opened the file in its own worker: every open, read-only ones included, is known to the kernel, and all holders share the one file (writes through one are seen through the others). The kernel does not copy it: as NFS does with its "silly rename", it moves the file into a hidden directory, `.slicc-unlinked/`, at the root of the mount it lives on (or of the root file system), and serves the holders' reads and writes from there (a worker's I/O on the old path while the path is gone). No file data is held in memory for it. The file is deleted when the last holder closes it or exits, and the directory goes as soon as it is empty. A file created at the path afterwards is the new file for everyone, also for the holder; a holder whose file another process renames (rather than removes) still uses the old path. A rename over a held file stays atomic for everyone else, as `rename(2)` is: moving the held file aside and the new one into its place are two steps, so a lookup of that name (`open`, `stat`, a read) that comes while they run waits for both, and one that started before them and missed the name in between looks again (also when it spelled the name through a symlinked parent, which the wait doesn't match by path: a miss is retried whenever any replacement finished meanwhile, up to three tries). Every open finds the old file or the new one, never no file (#903). The OPFS root does the same for its own two steps: a rename onto an existing name replaces the file with `move()` (or, in a browser without it, a copy into the name and a removal of the source), and a lookup that loses that race to a stale handle looks again. A `tmpfs` mount replaces the name in one step; on `fsa` and `hostfs` mounts it is the real file system's `rename`. Listings in the kernel leave `.slicc-unlinked` out, but on an `fsa` or `hostfs` mount it is visible on the real machine while a removed file is held (add `.slicc-unlinked/` to a `.gitignore` there). Each kernel names its entries after itself and holds a Web Lock for that name while it runs; when a mount is set up (and the root at start), the kernel removes the entries whose owner holds no lock, so a tab that died while holding a file leaves nothing behind for long, and entries of other live tabs on the same origin are left alone. Under the Node entry, which has no Web Locks, entries are named after the kernel's process id instead, and an entry is removed when that process is gone (`kill(pid, 0)` fails with `ESRCH`); a reused pid leaves the entry alone. Signals a program ignores stay ignored in the image it execs and in the children it spawns or forks, as POSIX has it, so `nohup cmd` and `trap '' TERM; exec cmd` keep `cmd` safe from them; caught signals go back to their default. The kernel holds what was handed down, so a new image that sets one of those signals back to the default with `signal(SIG_DFL)` still ignores it, until it installs a handler.
+An exec'd program takes over the pid of the process it replaces, as on Linux: `getpid()` and `$$` in the new image, `/proc`, `ps`, `kill` and the parent's `waitpid` all agree on that pid, and the image's parent pid is the replaced process's. Its own children see that pid as their parent. A file removed or renamed over while programs hold it open keeps its bytes for them, as on Linux, whether the descriptors live in the kernel or a program opened the file in its own worker: every open, read-only ones included, is known to the kernel, and all holders share the one file (writes through one are seen through the others). The kernel does not copy it: as NFS does with its "silly rename", it moves the file into a hidden directory, `.slicc-unlinked/`, at the root of the mount it lives on (or of the root file system), and serves the holders' reads and writes from there (a worker's I/O on the old path while the path is gone). No file data is held in memory for it. The file is deleted when the last holder closes it or exits, and the directory goes as soon as it is empty. A file created at the path afterwards is the new file for everyone, also for the holder; a holder whose file another process renames (rather than removes) still uses the old path. A rename over a held file stays atomic for everyone else, as `rename(2)` is: moving the held file aside and the new one into its place are two steps, so a lookup of that name (`open`, `stat`, a read) that comes while they run waits for both, and one that started before them and missed the name in between looks again (also when it spelled the name through a symlinked parent, which the wait doesn't match by path: a miss is retried whenever any replacement finished meanwhile, up to three tries). Every open finds the old file or the new one, never no file (#903). A read through a held descriptor and the rename take the file's turn in order: lookups start to wait for the rename only once it has its turn on every held file it moves aside, so a read that came first finishes, and a rename over a file other processes keep reading never stalls them or itself (#980). A process that opens the file while the rename waits for its turn is a holder too: the rename looks for holders again once it has its turn, so that process keeps the old file as well, and a holder that closed meanwhile leaves nothing to move aside. The OPFS root does the same for its own two steps: a rename onto an existing name replaces the file with `move()` (or, in a browser without it, a copy into the name and a removal of the source), and a lookup that loses that race to a stale handle looks again. A `tmpfs` mount replaces the name in one step; on `fsa` and `hostfs` mounts it is the real file system's `rename`. Listings in the kernel leave `.slicc-unlinked` out, but on an `fsa` or `hostfs` mount it is visible on the real machine while a removed file is held (add `.slicc-unlinked/` to a `.gitignore` there). Each kernel names its entries after itself and holds a Web Lock for that name while it runs; when a mount is set up (and the root at start), the kernel removes the entries whose owner holds no lock, so a tab that died while holding a file leaves nothing behind for long, and entries of other live tabs on the same origin are left alone. Under the Node entry, which has no Web Locks, entries are named after the kernel's process id instead, and an entry is removed when that process is gone (`kill(pid, 0)` fails with `ESRCH`); a reused pid leaves the entry alone. Signals a program ignores stay ignored in the image it execs and in the children it spawns or forks, as POSIX has it, so `nohup cmd` and `trap '' TERM; exec cmd` keep `cmd` safe from them; caught signals go back to their default. The kernel holds what was handed down, so a new image that sets one of those signals back to the default with `signal(SIG_DFL)` still ignores it, until it installs a handler.
 
 - **Emscripten:** `Module.sliccKernel.execve(file, argv, env, cwd)` runs `file` as this process's new image on its fds 0, 1 and 2, waits for it, and returns its wait status, or a negative errno when it cannot start. `env` and `cwd` may be `null` for this process's own. An exec shim detects it with `typeof Module.sliccKernel.execve === 'function'` and exits with the status it returns, as `execve` never returns on success. Older shims spawn the program and then call `Module.sliccKernel.execWait(pid)`; that works the same way: a new program asks the kernel for its pid before `main`, and the kernel answers once its parent has made its next call, so an image an older shim execs also reports the pid it replaced (the wait starts once the program is loaded; should the parent make no call within 100 ms after that, the kernel answers anyway, and the image may then report its own pid; `launcher.identities` counts the programs that asked and those answered by that timeout).
 - **WASIX:** `proc_exec` does this by itself.
@@ -942,11 +942,14 @@ The kernel writes `/etc/passwd` and `/etc/group` when it starts (see [Users](#us
 ### How many processes at once
 
 In a browser every process worker of the page shares one renderer. V8 reserves about 8 GiB of address space for each 32-bit wasm memory, whatever its declared maximum, and its sandbox holds about 1 TiB. So a page runs about 125 wasm processes (or programs that own a wasm memory) at once, counting everything in the renderer (#970). Node has no such ceiling, and neither does Linux. The kernel keeps processes inside that ceiling:
-- **The ceiling:** 112 by default in a browser and none in Node. It's the kernel's count of live wasm processes, with the kernel's other wasm memories in the page left outside it. When an instantiation runs out of memory anyway, the kernel learns the real ceiling at that moment (the live count minus one), and that learned value wins over the default.
+- **The ceiling:** 112 by default in a browser and none in Node. It's the kernel's count of live wasm processes, with the kernel's other wasm memories in the page left outside it. When an instantiation runs out of memory anyway, the kernel learns the real ceiling at that moment (the live count minus one), and that learned value wins over the default. A learned ceiling is never below 16, since an OOM at a low count means something outside it (another tab in the same renderer) filled the renderer. It's raised by 8 for each minute without another OOM, back to the default.
 - **Admission:** a spawn, fork, exec or `kernel.run` at the ceiling waits for another process to exit. It keeps waiting as long as processes exit or are admitted, and fails with `EAGAIN` only after 10 s with neither. A long `make -j` or `xargs -P` therefore queues, and a pipeline that needs more live processes than the ceiling fails cleanly. bash's own retry on a fork's `EAGAIN` (`fork: retry: Resource temporarily unavailable`) handles the rest.
 - **Close to the ceiling** (within 16 of it), the kernel waits for each new process to finish instantiating before the spawn returns. If it ran out of memory, the kernel starts it again, quietly, once there's room, so the caller sees no failure.
 - **The rare residual case:** a process that fails to instantiate with nobody waiting on it, which happens only when something outside the count fills the renderer. It ends with status 126 and `<program>: cannot allocate memory for a new process (too many processes at once)` on its stderr, never a JS `RangeError`.
-- **`/proc/sys/kernel/slicc_wasm_memories`** reads `<live> <ceiling>` (the ceiling 0 for none).
+- **Below a learned ceiling,** a process that still runs out of memory is started again once another process exits, and the spawn fails with `EAGAIN` after 10 s with none.
+- **`/proc/sys/kernel/slicc_wasm_memories`** reads `<live> <ceiling> <source>`. The source is `default`, `learned`, or `none` (with the ceiling 0).
+
+`/proc/sys/kernel/slicc_shared_root` reads `1` while another kernel shares the root (a tab, or a second kernel in the page) and `0` otherwise. A peer's tab that closes is noticed when this kernel next writes to a shared file, at most every 30 s, so the value can stay `1` for a while after a peer has gone.
 
 ### Metadata
 

package/dist/kernel-worker.js

@@ -6565,7 +6565,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.124.0" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.124.2" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -7468,18 +7468,19 @@ var UnlinkedKeeper = class {
     const keeper = this;
     const { replacing } = this;
     const replace = async (target, op) => {
-      const before = replacing.get(target);
       const { promise, resolve: resolve2 } = Promise.withResolvers();
-      const mine = {
-        done: Promise.all([before?.done, promise]).then(() => void 0)
+      let mine;
+      const gated = () => {
+        const before = replacing.get(target);
+        mine = { done: Promise.all([before?.done, promise]).then(() => void 0) };
+        replacing.set(target, mine);
       };
-      replacing.set(target, mine);
       try {
-        await op();
+        await op(gated);
       } finally {
         keeper.replaced += 1;
         resolve2();
-        if (replacing.get(target) === mine) replacing.delete(target);
+        if (mine && replacing.get(target) === mine) replacing.delete(target);
       }
     };
     const unreplaced = async (path, op) => {
@@ -7524,8 +7525,8 @@ var UnlinkedKeeper = class {
       async rename(from, to) {
         const target = fs.resolvePath("/", to);
         if (target === fs.resolvePath("/", from)) return fs.rename(from, to);
-        await replace(target, async () => {
-          const pinned = await keeper.pin(target);
+        await replace(target, async (gated) => {
+          const pinned = await keeper.pin(target, gated);
           await keeper.settle(pinned, () => fs.rename(from, to));
         });
       },
@@ -7542,7 +7543,7 @@ var UnlinkedKeeper = class {
     }
     pinned.commit();
   }
-  async pin(target) {
+  heldWithin(target) {
     const held = /* @__PURE__ */ new Map();
     for (const holder of this.holders) {
       if (holder.owns(target)) continue;
@@ -7551,11 +7552,19 @@ var UnlinkedKeeper = class {
         held.set(path, [...held.get(path) ?? [], holder]);
       }
     }
-    const paths = [...held.keys()].sort();
+    return held;
+  }
+  async pin(target, gated = () => void 0) {
+    const paths = [...this.heldWithin(target).keys()].sort();
     const releases = [];
     for (const path of paths) releases.push(await this.gate.acquire(path));
+    gated();
+    const now = this.heldWithin(target);
+    const held = new Map(
+      paths.flatMap((path) => now.has(path) ? [[path, now.get(path)]] : [])
+    );
     const hidden = /* @__PURE__ */ new Map();
-    for (const path of paths) {
+    for (const path of held.keys()) {
       const at = await this.hide(path);
       if (at !== void 0) hidden.set(path, at);
     }
@@ -12712,7 +12721,9 @@ var SYS_TREE = {
     osrelease: `${OS_RELEASE}
 `,
     ostype: "Linux\n",
-    slicc_wasm_memories: (listing) => `${listing.wasm?.live ?? 0} ${listing.wasm?.ceiling ?? 0}
+    slicc_wasm_memories: (listing) => `${listing.wasm?.live ?? 0} ${listing.wasm?.ceiling ?? 0} ${listing.wasm?.source ?? "none"}
+`,
+    slicc_shared_root: (listing) => `${listing.peers ?? 0}
 `
   }
 };
@@ -21917,37 +21928,65 @@ var SettlingChildren = class {
 
 // src/kernel/wasm-memories.ts
 var DEFAULT_CEILING = 112;
+var CEILING_FLOOR = 16;
 var IDLE_MS = 1e4;
 var MARGIN = 16;
+var DECAY_MS = 6e4;
+var DECAY_STEP = 8;
 var WasmMemories = class {
   live = 0;
-  ceiling;
+  learned;
   waiters = /* @__PURE__ */ new Set();
+  base;
   idleMs;
-  constructor(ceiling, idleMs = IDLE_MS) {
-    this.ceiling = ceiling;
+  now;
+  constructor(ceiling, idleMs = IDLE_MS, now = Date.now) {
+    this.base = ceiling;
     this.idleMs = idleMs;
+    this.now = now;
+  }
+  ceiling() {
+    const learned = this.learned;
+    if (!learned) return this.base;
+    const steps = Math.floor((this.now() - learned.at) / DECAY_MS);
+    const raised = learned.ceiling + steps * DECAY_STEP;
+    if (raised < (this.base ?? DEFAULT_CEILING)) return raised;
+    this.learned = void 0;
+    return this.base;
   }
   count() {
-    return { live: this.live, ceiling: this.ceiling ?? 0 };
+    const ceiling = this.ceiling();
+    const source = this.learned ? "learned" : ceiling === void 0 ? "none" : "default";
+    return { live: this.live, ceiling: ceiling ?? 0, source };
   }
   ended() {
     this.live = Math.max(0, this.live - 1);
     this.progress();
   }
   learn() {
-    this.ceiling = Math.max(1, this.live - 1);
+    const top = this.base ?? DEFAULT_CEILING;
+    this.learned = {
+      ceiling: Math.min(top, Math.max(CEILING_FLOOR, this.live - 1)),
+      at: this.now()
+    };
   }
   near() {
-    return this.ceiling !== void 0 && this.live + MARGIN >= this.ceiling;
+    const ceiling = this.ceiling();
+    return ceiling !== void 0 && this.live + MARGIN >= ceiling;
   }
   async admit() {
-    while (this.ceiling !== void 0 && this.live >= this.ceiling) {
+    for (let ceiling = this.ceiling(); ceiling !== void 0 && this.live >= ceiling; ) {
       if (!await this.changed()) throw new SpawnError("EAGAIN");
+      ceiling = this.ceiling();
     }
     this.live += 1;
     this.progress();
   }
+  async refused() {
+    const ceiling = this.ceiling();
+    if (ceiling === void 0 || this.live >= ceiling) return;
+    if (!await this.changed()) throw new SpawnError("EAGAIN");
+  }
   progress() {
     const waiters = [...this.waiters];
     this.waiters.clear();
@@ -25208,6 +25247,10 @@ var Launcher = class {
         return handle;
       }
       await handle.exited;
+      await this.memories.refused().catch(async (err) => {
+        await fds.closeAll();
+        throw err;
+      });
     }
   }
   start(req) {
@@ -25273,7 +25316,8 @@ var Launcher = class {
         processes: this.list(this.credOf(pid)),
         load: this.loadAverage(),
         cpu: this.cpuTotal(),
-        wasm: this.memories.count()
+        wasm: this.memories.count(),
+        peers: Number(this.locks.shared?.() === true)
       }),
       sudo: (call, signal) => this.sudo(pid, call, signal),
       crontab: (call) => this.crontab(pid, call),