@ai-ecoverse/slicc-kernel 1.122.2 → 1.123.0

package/README.md

@@ -611,6 +611,7 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
 - **Caching:** the kernel caches attributes and listings for those TTLs, and a driver can push `invalidate` for changes made outside.
 - **File contents:** on a mount with `ranges`, as `tmpfs` is, and on the root file system when it is in OPFS (in a browser with sync access handles) or an in-memory root, a program reads and writes a file in 64 KiB pages: reads come from the driver 1 MiB at a time, and changed pages go back in 1 MiB runs once 8 MiB have changed, at `fsync`, and when the file is closed (from the kernel's own descriptors also 250 ms after a write). A file of any size the mount can hold works, with memory to spare for about one more copy of it. In OPFS a write opens a sync access handle for just that write. An `fsa` mount (a picked folder) has no sync access handles, and Chrome copies the whole file into a swap file for every writable stream, so it writes through one stream per open file: the first write back opens it, later runs go into it at their offsets, and it is committed, one copy of the file, when the last process writing the file closes it (or exits), before another process reads what was written, and before a rename, unlink or unmount. Until then other processes see the new size but read the old bytes outside what was written. Elsewhere (`hostfs` with an older proxy, and a root outside OPFS), file contents are read whole when a program opens a file and written back when it closes it. Before a program's next call into the kernel (any wait, read of a pipe or terminal, sleep or spawn), its changed pages also go back, so other processes and the embedder see a write as soon as the writer does anything but compute: a daemon's pidfile, a log another process follows. On a whole-file mount that holds only for files of up to 64 KiB; a larger one goes back at close, fsync and exit, as before. Beyond that, both give close-to-open consistency, as NFS has. A process that ends without closing its files keeps what it wrote: on exit, on a trap, and when a signal kills it. For a killed WASI program, the kernel lets its worker write its changed pages back before `waitpid` reports the death. The only exception is a worker that makes no call into the kernel for 2 seconds after the signal; it is stopped without that write-back.
 - **Exclusive create across tabs** ([#912](https://github.com/ai-ecoverse/slicc-kernel/issues/912)): two kernels on one OPFS root (two tabs, or a tab and a worker page) agree on `O_CREAT|O_EXCL` and `set -C`: exactly one wins a name. While another kernel on the root is present, the create runs under a Web Lock for its path and checks the OPFS entry itself. A kernel alone on its root takes no lock. Each kernel holds an alive Web Lock, granted before it first looks for peers. A kernel that finds peers waits for their answer (at most 500 ms) before its first exclusive create, and a kernel answers a newcomer only once its unlocked creates are done. So of two kernels that boot at the same moment, the later one to look sees the other, and no unlocked create overlaps a locked one.
+- **A name another process or tab just removed** ([#950](https://github.com/ai-ecoverse/slicc-kernel/issues/950)): sqlite3 deletes its `-journal` at every commit and the next writer creates it again, so two writers (two processes in one kernel, or two tabs on one OPFS root) keep removing and re-creating one name. Two things used to turn that into `unable to open database file`. The root removes a file's sidecar record a few milliseconds after the file itself (removals are batched), and an exclusive create read that pending record as the name being taken: a plain record whose file is gone now counts as gone, and the new file gets a record of its own. A symlink, which lives only in the sidecar, still takes its name. And an Emscripten program keeps the nodes it has looked up, so after another tab removed a name it still held the old node, and `open(O_CREAT)` opened the gone file instead of creating one. Now, when the kernel answers `ENOENT` for a cached node, for `open`, `stat`, `lstat`, `readlink`, `truncate`, `unlink` or a rename's source, or the program's own lookup answers `EEXIST` to `open(O_CREAT|O_EXCL)`, the node is dropped and the call runs once more, against what is there now. A node the program has open or has unwritten changes in is kept. A rename over a name another tab removed needs nothing: the kernel's rename replaces whatever is there.
 - **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. The same file under two names is one file too ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)): a write by path through a symlink reaches the file the kernel holds under its real path (resolved without following links in a `sudo` process's no-links view, so an untrusted link is still refused), a WASI program's open file also watches its real path (resolved once, the first time another file changes), and an open file with more than one hard link checks its size, version and mtime when anything changed, so a write through another name is seen. Limits for now: an alias on a mount that has no `realpath` is still only the name it was opened by; another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them. A by-path `stat` or `lstat` of a file the kernel holds open has its current size, writes not yet written back included ([#780](https://github.com/ai-ecoverse/slicc-kernel/issues/780)). Its mtime is the file system's until write-back, though, so a tool that compares times, like `make`, may see a just-written file as older than it is for up to the write-back delay.
 - **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, and so are a WASIX program's through the `slicc.fcntl_lock` import (below), so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` keeps its behaviour (whole file, per process, keyed by path). Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program built against a wasix-sysroot without the import keeps its libc's local answer (every lock granted), so python's sqlite excludes other processes only once wasix-python is rebuilt on the sysroot that has it ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)). An Emscripten program's `flock()` reaches the kernel's `flock` table (per open file description) through homescoop's shim, a strong `flock()` that calls the import `env.__syscall_flock(fd, op)`. The kernel wraps that import by name (#947): `LOCK_SH`/`LOCK_EX`/`LOCK_UN` with `LOCK_NB` (`EWOULDBLOCK` for a conflict), 0 or -errno. A glue without the shim keeps Emscripten's stub, which answers 0. A lock belongs to the file its descriptor was opened on: a file renamed over the name later, or the name's removal, doesn't move it.
 - **Locks across tabs** ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)): two kernels on one OPFS root (two tabs, or a tab and a worker page) share one table of `fcntl` record locks and `flock` locks, so a sqlite3 in one tab finds the database locked while a transaction is open in another. **Known issue, data corruption: lock-protected writes across tabs** ([#949](https://github.com/ai-ecoverse/slicc-kernel/issues/949)). The locks are enforced across tabs, but what one tab writes under a lock isn't yet guaranteed to be what the other tab reads once it holds the lock, so a lock-protected read-modify-write across tabs can lose or corrupt data. Don't rely on cross-tab locking until #949 is fixed. Use within a single tab is unaffected.
@@ -917,6 +918,15 @@ Every process of the kernel has `/proc/<pid>/` with `cmdline`, `comm`, `stat`, `
 
 The kernel writes `/etc/passwd` and `/etc/group` when it starts (see [Users](#users)). While the root has none of them, or no `/etc/mtab`, reading them gets per-process ones: root is uid 0 and the realm user, uid 1000, is named after `USER`, or `web_user` (Emscripten's default) without one, so it matches `$USER`, with `HOME` as its home, so `id`, `whoami`, `ls -l` and `ps` show names. Nothing is written to OPFS, and a real file there wins.
 
+### How many processes at once
+
+In a browser every process worker of the page shares one renderer. V8 reserves about 8 GiB of address space for each 32-bit wasm memory, whatever its declared maximum, and its sandbox holds about 1 TiB. So a page runs about 125 wasm processes (or programs that own a wasm memory) at once, counting everything in the renderer (#970). Node has no such ceiling, and neither does Linux. The kernel keeps processes inside that ceiling:
+- **The ceiling:** 112 by default in a browser and none in Node. It's the kernel's count of live wasm processes, with the kernel's other wasm memories in the page left outside it. When an instantiation runs out of memory anyway, the kernel learns the real ceiling at that moment (the live count minus one), and that learned value wins over the default.
+- **Admission:** a spawn, fork, exec or `kernel.run` at the ceiling waits for another process to exit. It keeps waiting as long as processes exit or are admitted, and fails with `EAGAIN` only after 10 s with neither. A long `make -j` or `xargs -P` therefore queues, and a pipeline that needs more live processes than the ceiling fails cleanly. bash's own retry on a fork's `EAGAIN` (`fork: retry: Resource temporarily unavailable`) handles the rest.
+- **Close to the ceiling** (within 16 of it), the kernel waits for each new process to finish instantiating before the spawn returns. If it ran out of memory, the kernel starts it again, quietly, once there's room, so the caller sees no failure.
+- **The rare residual case:** a process that fails to instantiate with nobody waiting on it, which happens only when something outside the count fills the renderer. It ends with status 126 and `<program>: cannot allocate memory for a new process (too many processes at once)` on its stderr, never a JS `RangeError`.
+- **`/proc/sys/kernel/slicc_wasm_memories`** reads `<live> <ceiling>` (the ceiling 0 for none).
+
 ### Metadata
 
 OPFS stores names, bytes, sizes and modification times, nothing else. Everything POSIX needs on top of that lives in an IndexedDB sidecar, so it survives reloads and new kernels:

package/dist/kernel-worker.js

@@ -2051,6 +2051,7 @@ var OpfsFs = class {
       if (await this.handleExists(dir, name2)) throw fsError("EEXIST", path);
       const freshMeta = this.meta.fresh ? await this.meta.fresh([path]) : await this.meta.get([path]);
       if (freshMeta[0]?.link !== void 0) throw fsError("EEXIST", path);
+      if (freshMeta[0] !== void 0) await this.meta.remove([path]);
       this.forget(physical);
       this.refuse(physical);
       await dir.getDirectoryHandle(name2, { create: true });
@@ -2070,7 +2071,8 @@ var OpfsFs = class {
     await this.locked(physical, async () => {
       if (await this.handleExists(dir, name2)) throw fsError("EEXIST", path);
       const freshMeta = this.meta.fresh ? await this.meta.fresh([path]) : await this.meta.get([path]);
-      if (freshMeta[0] !== void 0) throw fsError("EEXIST", path);
+      if (freshMeta[0]?.link !== void 0) throw fsError("EEXIST", path);
+      if (freshMeta[0] !== void 0) await this.meta.remove([path]);
       this.forget(physical);
       this.refuse(physical);
       await dir.getFileHandle(name2, { create: true });
@@ -6406,7 +6408,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.122.2" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.123.0" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -12549,17 +12551,28 @@ var FALLBACK_MEMORY = 4 * KIB * KIB * KIB;
 var PID_FILES = ["cmdline", "comm", "stat", "statm", "status"];
 var SYSTEM_FILES = ["loadavg", "meminfo", "stat", "uptime", "version"];
 var SYS_TREE = {
-  kernel: { osrelease: `${OS_RELEASE}
-`, ostype: "Linux\n" }
+  kernel: {
+    osrelease: `${OS_RELEASE}
+`,
+    ostype: "Linux\n",
+    slicc_wasm_memories: (listing) => `${listing.wasm?.live ?? 0} ${listing.wasm?.ceiling ?? 0}
+`
+  }
 };
+function isSysFile(at) {
+  return typeof at === "string" || typeof at === "function";
+}
 function sysEntry(parts2) {
   let at = SYS_TREE;
   for (const part of parts2) {
-    if (typeof at !== "object" || !Object.hasOwn(at, part)) return void 0;
+    if (isSysFile(at) || !Object.hasOwn(at, part)) return void 0;
     at = at[part];
   }
   return at;
 }
+function sysText(file, listing) {
+  return typeof file === "string" ? file : file(listing());
+}
 function commOf(info) {
   const name2 = info.argv[0] ?? "";
   return name2.slice(name2.lastIndexOf("/") + 1).slice(0, 15);
@@ -12795,7 +12808,9 @@ var ProcTree = class {
     }
     if (top === "sys") {
       const entry = sysEntry(parts2.slice(1));
-      if (typeof entry === "string") return { kind: "file", content: () => entry };
+      if (typeof entry === "string" || typeof entry === "function") {
+        return { kind: "file", content: () => sysText(entry, () => this.listing()) };
+      }
       return entry && { kind: "dir", names: () => Object.keys(entry) };
     }
     if (top === "self") {
@@ -19252,6 +19267,7 @@ var WasmProcess = class {
 var WASM_PROCESS_INIT = "wasm-process-init";
 var WASM_PROCESS_EXIT = "wasm-process-exit";
 var WASM_PROCESS_ERROR = "wasm-process-error";
+var WASM_PROCESS_READY = "wasm-process-ready";
 var WASM_THREAD_SPAWN = "wasm-thread-spawn";
 var WASM_THREAD_INIT = "wasm-thread-init";
 var WASM_THREAD_EXIT = "wasm-thread-exit";
@@ -19259,6 +19275,8 @@ var WASM_MAX_THREADS = 256;
 
 // src/kernel/host.ts
 var CRASHED = 70;
+var NO_MEMORY = 126;
+var WASM_OOM = /Cannot allocate Wasm memory|WebAssembly\.Memory\(\): could not allocate memory/;
 var KILL_GRACE_MS = 2e3;
 var KILL_HARD_MS = 5e3;
 var descIds = /* @__PURE__ */ new WeakMap();
@@ -19524,7 +19542,22 @@ function threadInit(opts, program, sab, cred, thread, main) {
 function settleExit(exit, settle3, code) {
   void exit.catch(() => void 0).then(() => settle3(code & 255));
 }
+function readiness(abi) {
+  const ready2 = Promise.withResolvers();
+  if (abi === "js") ready2.resolve(true);
+  return ready2;
+}
+function outOfMemory(opts, data) {
+  if (data?.type !== WASM_PROCESS_ERROR || !WASM_OOM.test(String(data.message))) return false;
+  if (!opts.onOom?.()) {
+    opts.onError?.(
+      `${opts.argv0}: cannot allocate memory for a new process (too many processes at once)`
+    );
+  }
+  return true;
+}
 function spawnWasmProcess(opts) {
+  const { promise: ready2, resolve: started } = readiness(opts.program.abi);
   const sab = new SharedArrayBuffer(SAB_HEADER_BYTES + SAB_DEFAULT_WINDOW_BYTES);
   const header3 = new Int32Array(sab, 0, SAB_HEADER_I32);
   const blocked2 = opts.fork?.wasi?.blocked ?? opts.blocked;
@@ -19577,6 +19610,7 @@ function spawnWasmProcess(opts) {
   const finish = (code, sig) => {
     if (done4) return;
     done4 = true;
+    started(true);
     clocks.freeze();
     dying.clear();
     endedBy = sig ?? process.execTermsig;
@@ -19594,7 +19628,12 @@ function spawnWasmProcess(opts) {
     queueMicrotask(() => finish(...dying.status(code)));
   };
   const handle = (data) => {
-    if (data?.type === WASM_PROCESS_EXIT) {
+    if (data?.type === WASM_PROCESS_READY) {
+      started(true);
+    } else if (outOfMemory(opts, data)) {
+      started(false);
+      finish(NO_MEMORY);
+    } else if (data?.type === WASM_PROCESS_EXIT) {
       finish(...dying.status(data.code));
     } else if (data?.type === WASM_PROCESS_ERROR) {
       if (!dying.ending) opts.onError?.(String(data.message));
@@ -19668,6 +19707,7 @@ function spawnWasmProcess(opts) {
   return {
     pid: opts.pid,
     exited,
+    ready: ready2,
     kill,
     signal,
     termsig: () => endedBy,
@@ -21262,6 +21302,63 @@ var SettlingChildren = class {
   }
 };
 
+// src/kernel/wasm-memories.ts
+var DEFAULT_CEILING = 112;
+var IDLE_MS = 1e4;
+var MARGIN = 16;
+var WasmMemories = class {
+  live = 0;
+  ceiling;
+  waiters = /* @__PURE__ */ new Set();
+  idleMs;
+  constructor(ceiling, idleMs = IDLE_MS) {
+    this.ceiling = ceiling;
+    this.idleMs = idleMs;
+  }
+  count() {
+    return { live: this.live, ceiling: this.ceiling ?? 0 };
+  }
+  ended() {
+    this.live = Math.max(0, this.live - 1);
+    this.progress();
+  }
+  learn() {
+    this.ceiling = Math.max(1, this.live - 1);
+  }
+  near() {
+    return this.ceiling !== void 0 && this.live + MARGIN >= this.ceiling;
+  }
+  async admit() {
+    while (this.ceiling !== void 0 && this.live >= this.ceiling) {
+      if (!await this.changed()) throw new SpawnError("EAGAIN");
+    }
+    this.live += 1;
+    this.progress();
+  }
+  progress() {
+    const waiters = [...this.waiters];
+    this.waiters.clear();
+    for (const wake of waiters) wake();
+  }
+  changed() {
+    return new Promise((resolve2) => {
+      const wake = () => {
+        clearTimeout(timer);
+        resolve2(true);
+      };
+      const timer = setTimeout(() => {
+        this.waiters.delete(wake);
+        resolve2(false);
+      }, this.idleMs);
+      this.waiters.add(wake);
+    });
+  }
+};
+function defaultCeiling() {
+  const node = globalThis.process?.versions?.node;
+  return node === void 0 ? DEFAULT_CEILING : void 0;
+}
+
 // src/mount/driver.ts
 function ownerOf2(req) {
   const owner = {
@@ -23459,6 +23556,9 @@ function withScriptEnv(target, command) {
     ]
   };
 }
+function memoriesOf(given) {
+  return new WasmMemories(given?.ceiling ?? defaultCeiling(), given?.idleMs);
+}
 function childHandle(handle) {
   return {
     pid: handle.pid,
@@ -23530,6 +23630,7 @@ var Launcher = class {
   jobs = new JobTable();
   locks;
   fifos = new FifoTable();
+  memories;
   sems = new SemTable();
   settling = new SettlingChildren((pid) => this.jobs.pgidOf(pid));
   ptys = new PtyTable(
@@ -23583,6 +23684,7 @@ var Launcher = class {
   terminals = 0;
   constructor(options) {
     this.usersMode = usersMode(options.users);
+    this.memories = memoriesOf(options.wasmMemories);
     this.rootFs = options.fs;
     const peers = options.fs.peers?.();
     this.locks = kernelLocks(options.fs.rootName, peers);
@@ -24413,7 +24515,11 @@ var Launcher = class {
     }
     const done4 = this.heldUrl(program);
     try {
-      const handle = this.start({ ...req, cwd, program, argv0: target.argv0, args, env });
+      const handle = await this.admitted(
+        program.abi !== "js",
+        req.fds,
+        (fds, quietOom) => this.start({ ...req, fds, quietOom, cwd, program, argv0: target.argv0, args, env })
+      );
       void handle.exited.finally(done4);
       return handle;
     } catch (err) {
@@ -24457,6 +24563,39 @@ var Launcher = class {
     if (req.tag !== void 0) this.tags.set(pid, req.tag);
     if (req.links !== void 0) this.links.set(pid, req.links);
   }
+  async admitted(wasm, fds, start2) {
+    if (!wasm) return start2(fds);
+    for (; ; ) {
+      await this.memories.admit().catch(async (err) => {
+        await fds.closeAll();
+        throw err;
+      });
+      const counted2 = (handle2) => {
+        void handle2.exited.finally(() => this.memories.ended());
+        return handle2;
+      };
+      if (!this.memories.near()) {
+        try {
+          return counted2(start2(fds));
+        } catch (err) {
+          this.memories.ended();
+          throw err;
+        }
+      }
+      const forked = fds.fork();
+      const handle = await Promise.resolve().then(() => counted2(start2(forked, true))).catch(async (err) => {
+        this.memories.ended();
+        await forked.closeAll();
+        throw err;
+      });
+      const fit = await (handle.ready ?? true);
+      if (fit) {
+        await fds.closeAll();
+        return handle;
+      }
+      await handle.exited;
+    }
+  }
   start(req) {
     this.sampling();
     if (req.pgid !== void 0 && this.jobs.sessionOf(req.pgid) === void 0) {
@@ -24507,6 +24646,10 @@ var Launcher = class {
       ...this.prefixes.length > 0 && !stack.writer ? { readonlyAt: (path) => isProtected(this.prefixes, path) } : {},
       createWorker: this.createWorker,
       onError: req.report,
+      onOom: () => {
+        this.memories.learn();
+        return req.quietOom === true;
+      },
       spawner: this.spawner(pid, req.report),
       forker: this.forker(pid, req),
       kill: (target, sig, info) => this.kill(target, sig, this.credOf(pid), pid, info),
@@ -24515,7 +24658,8 @@ var Launcher = class {
         boot: this.boot,
         processes: this.list(this.credOf(pid)),
         load: this.loadAverage(),
-        cpu: this.cpuTotal()
+        cpu: this.cpuTotal(),
+        wasm: this.memories.count()
       }),
       sudo: (call, signal) => this.sudo(pid, call, signal),
       crontab: (call) => this.crontab(pid, call),
@@ -24852,9 +24996,10 @@ var Launcher = class {
   }
   forker(ppid, parent) {
     const { exec: _exec, pgid: _pgid, group: _group, pid: _pid, scrub: _scrub, ...image } = parent;
-    return async (state, fds) => childHandle(
-      this.start({
+    return async (state, given) => {
+      const fresh2 = (fds, quietOom) => this.start({
         ...image,
+        quietOom,
         cwd: state.cwd ?? parent.cwd,
         fds,
         ppid,
@@ -24865,8 +25010,9 @@ var Launcher = class {
         cred: this.credOf(ppid),
         tag: this.tagOf(ppid),
         links: this.links.get(ppid)
-      })
-    );
+      });
+      return childHandle(await this.admitted(parent.program.abi !== "js", given, fresh2));
+    };
   }
   keySignal(tty, fallback, sig, send) {
     const pgid = this.jobs.tcgetpgrp(tty, fallback);