@ai-ecoverse/slicc-kernel 1.31.1 → 1.32.0

package/README.md

@@ -125,7 +125,7 @@ const { status, stdout } = await kernel.run(['bash', '-c', 'echo hi'], { cwd: '/
 kernel.terminate();
 ```
 
-`createNodeKernel({ root, modules, env, network, worker, processMounts, fstabRetries, cdp, hostfs })` takes the options of `createKernel` except `metadata`, and `fstabRetries`, the delays in ms between tries of a failing `/etc/fstab` line (default `[1000, 4000, 16000]`). `root` is an in-memory directory by default (`memoryRoot()` makes another), which keeps each file in 1 MiB chunks, and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal`, `setRoutes` and `terminate` as above, plus `root`, `writeFile(path, data)` (creating the parent directories) and `readFile(path)` to put files in place and read results. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`). `connect()` and `attachKernel` work as in the browser: the port is a `worker_threads` `MessagePort`, which a worker thread can attach with, and `terminate()` ends every attached client.
+`createNodeKernel({ root, modules, env, network, worker, processMounts, fstabRetries, cdp, hostfs, hostfsOrigin, hostfsFetch })` takes the options of `createKernel` except `metadata`, and `fstabRetries`, the delays in ms between tries of a failing `/etc/fstab` line (default `[1000, 4000, 16000]`). `root` is an in-memory directory by default (`memoryRoot()` makes another), which keeps each file in 1 MiB chunks, and POSIX metadata stays in memory. Processes and threads run on `worker_threads`. The kernel has `run`, `openTerminal`, `setRoutes` and `terminate` as above, plus `root`, `writeFile(path, data)` (creating the parent directories) and `readFile(path)` to put files in place and read results. `nodeTransport()` is `fetchTransport()` with Node's `fetch`, which no CORS binds (`crossOrigin: 'any'`). `connect()` and `attachKernel` work as in the browser: the port is a `worker_threads` `MessagePort`, which a worker thread can attach with, and `terminate()` ends every attached client.
 
 ## Commands
 
@@ -325,7 +325,7 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
   - `@ai-ecoverse/slicc-kernel/driver` exports the types and `fsError`.
 - **Capabilities:** `readonly`, `symlinks`, `chmod`, `ranges`, `maxIo` (the largest read or write per message), `maxFile`, `listingStats`, `linkTimes`, and `attrTtl`/`entryTtl` (in ms, default 1000). `linkTimes` promises that `setattr` with `mtime` on a symlink sets the link's own time. Without it, setting a link's own times (`touch -h`, `utimensat` with `AT_SYMLINK_NOFOLLOW`) fails with `EOPNOTSUPP` and never reaches the target. tmpfs has it; hostfs does not. Setting times through a link follows it in the kernel, so `setattr` gets the target. `ranges` (driver protocol 1.1) promises that a write lands in the file at once, with no copy of the whole file on `open` or `release`, and that `setattr` takes `size`.
 - **Caching:** the kernel caches attributes and listings for those TTLs, and a driver can push `invalidate` for changes made outside.
-- **File contents:** on a mount with `ranges`, as `tmpfs` is, and on the root file system when it is in OPFS (in a browser with sync access handles) or an in-memory root, a program reads and writes a file in 64 KiB pages: reads come from the driver 1 MiB at a time, and changed pages go back in 1 MiB runs once 8 MiB have changed, at `fsync`, and when the file is closed (from the kernel's own descriptors also 250 ms after a write). A file of any size the mount can hold works, with memory to spare for about one more copy of it. In OPFS a write opens a sync access handle for just that write. Elsewhere (`fsa` mounts, `hostfs` with an older proxy, and a root outside OPFS), file contents are read whole when a program opens a file and written back when it closes it. Both give close-to-open consistency, as NFS has.
+- **File contents:** on a mount with `ranges`, as `tmpfs` is, and on the root file system when it is in OPFS (in a browser with sync access handles) or an in-memory root, a program reads and writes a file in 64 KiB pages: reads come from the driver 1 MiB at a time, and changed pages go back in 1 MiB runs once 8 MiB have changed, at `fsync`, and when the file is closed (from the kernel's own descriptors also 250 ms after a write). A file of any size the mount can hold works, with memory to spare for about one more copy of it. In OPFS a write opens a sync access handle for just that write. Elsewhere (`fsa` mounts, `hostfs` with an older proxy, and a root outside OPFS), file contents are read whole when a program opens a file and written back when it closes it. Both give close-to-open consistency, as NFS has. A process that ends without closing its files keeps what it wrote: on exit, on a trap, and when a signal kills it. For a killed WASI program, the kernel lets its worker write its changed pages back before `waitpid` reports the death. The only exception is a worker that makes no call into the kernel for 2 seconds after the signal; it is stopped without that write-back.
 - **Rules on a mount:**
   - a rename across mounts is `EXDEV`, so `mv` copies;
   - `df` reports each mount from its driver's `statfs`;
@@ -373,7 +373,8 @@ A `hostfs` mount is a folder exported by the local proxy ([slicc-node](https://g
 - **Options:** `ro` mounts read-only, and `maxfile` caps file size as on any mount. There is no other size limit: reads and writes go in `maxIo` pieces (16 MiB unless the proxy says otherwise), so a file of any size the host can hold works. A proxy that advertises `ranges` in its grant (slicc-node 2.4, slicc-swift 1.4) makes the mount ranged, so programs read and write its files in pages and truncate through `setattr` `size`. With an older proxy files go whole, and a truncate on a ranged mount whose proxy was replaced by an older one is done by rewriting the file.
 - **Changes on the host** arrive as invalidations on a watch stream (`POST /api/hostfs/watch`), so an edit made outside shows up on the next access without a remount.
 - **Proxy loss:** when the watch stream ends, or is silent for 45 s, the kernel reconnects once at once. If that fails, the mount is `nomedium`, as with [removable media](#removable-media-fsa): operations fail with `ENOMEDIUM` (`ENODEV` for WASI), and the kernel keeps reconnecting with backoff (1 s to 30 s), so the folder comes back by itself when the proxy does. A mount made while the proxy is down starts as `nomedium`, and so does one whose grant or first watch takes more than 15 s; it comes up by itself when they arrive.
-- **Consistency:** a file that changes on the host while a program reads it fails with `ESTALE`, and the kernel restarts the whole read (up to 3 times) instead of stitching two versions together. On a ranged mount an open file keeps the version it was opened at (its ETag) until it writes, also when it is handed to a child process: a page fetched after the host changed the file fails the program's read with `ESTALE` ("Stale file handle") instead of mixing versions.
+- **Consistency:** a file that changes on the host while a program reads it fails with `ESTALE`, and the kernel restarts the whole read (up to 3 times) instead of stitching two versions together. On a ranged mount an open file keeps the version it was opened at (its ETag) until it writes, also when it is handed to a child process: a page fetched after the host changed the file fails the program's read with `ESTALE` ("Stale file handle") instead of mixing versions. A file deleted on the host while a program holds it open fails the next read that goes to the host with `ENOENT`, not with NFS's `ESTALE`.
+- **Node:** a page's requests carry its `Origin`, and slicc-node grants a token to that origin only. Node's `fetch` sends none, so a Node embedder passes `createNodeKernel` the origin its `hostfs` hook asked for the grant with, as `hostfsOrigin`, and the kernel adds it to every hostfs request. Without it the mount stays `nomedium`. `hostfsFetch(url, init)` replaces the `fetch` those requests use (for an agent or a test proxy), and gets the `Origin` too when both are given.
 
 ### Mounting from a program
 

package/dist/kernel-worker.js

@@ -5976,6 +5976,7 @@ var SAB_I_TIMERS = 9;
 var SAB_I_MEMORY = 10;
 var SAB_I_TOTAL_HI = 11;
 var SAB_I_OFFSET_HI = 12;
+var SAB_I_KILLED = 13;
 var U32 = 2 ** 32;
 function storeU53(header2, lo, hi, value) {
   Atomics.store(header2, hi, Math.floor(value / U32));
@@ -7817,6 +7818,7 @@ function isWasmSyscall(req) {
   return typeof op === "string" && SYSCALL_OPS.has(op);
 }
 var MAX_READ = 1024 * 1024;
+var DYING = { ok: false, errno: "EINTR", message: "the process is being killed" };
 var HOSTS_ONLY = new Resolver({ routes: new Routes() });
 var WasmProcess = class {
   exited = false;
@@ -7825,6 +7827,7 @@ var WasmProcess = class {
   ignored = 0;
   inherited = 0;
   interrupt = new AbortController();
+  dying = false;
   execChild;
   execTermsig;
   stopped = 0;
@@ -7857,7 +7860,10 @@ var WasmProcess = class {
   signal(sig) {
     if (this.execChild !== void 0) {
       void Promise.resolve(this.options.kill?.(this.execChild, sig)).catch(() => void 0);
-      return sig === SIG.KILL ? "terminate" : "forward";
+      if (sig !== SIG.KILL) return "forward";
+      if (!this.options.writesBack?.(this.execChild)) return "terminate";
+      this.wake?.();
+      return "forward";
     }
     if (sig === SIG.KILL) return "terminate";
     if (sig === SIG.CONT) this.cont();
@@ -7901,14 +7907,22 @@ var WasmProcess = class {
     this.options.onSyscall?.(req);
     for (; ; ) {
       await this.resumed;
+      if (this.dying) return DYING;
       const stops = this.stops;
       const result = await this.dispatch(req);
+      if (this.dying) return DYING;
       const restart = !result.ok && result.errno === "EINTR" && this.stops !== stops;
       if (restart && !this.options.hasPending?.()) continue;
       await this.resumed;
       return result;
     }
   }
+  die() {
+    if (this.dying) return;
+    this.dying = true;
+    this.interrupt.abort();
+    this.wake?.();
+  }
   async dispatch(req) {
     try {
       if (isFdSyscall(req)) return await this.fdSyscall(req);
@@ -8369,6 +8383,7 @@ var WASM_MAX_THREADS = 256;
 
 // src/kernel/host.ts
 var CRASHED = 70;
+var KILL_GRACE_MS = 2e3;
 var descIds = /* @__PURE__ */ new WeakMap();
 var nextDescId = 1;
 function descId(file) {
@@ -8392,6 +8407,41 @@ function inheritedFds(fds) {
     };
   });
 }
+var Dying = class {
+  ending;
+  timer;
+  busy = 0;
+  finish;
+  start(code, sig, header2, process, finish) {
+    if (this.ending) return;
+    this.ending = { code, sig };
+    this.finish = finish;
+    this.arm();
+    Atomics.store(header2, SAB_I_KILLED, code);
+    process.die();
+  }
+  async serving(work) {
+    this.busy++;
+    clearTimeout(this.timer);
+    try {
+      return await work();
+    } finally {
+      this.busy--;
+      this.arm();
+    }
+  }
+  arm() {
+    clearTimeout(this.timer);
+    if (this.busy === 0 && this.finish) this.timer = setTimeout(this.finish, KILL_GRACE_MS);
+  }
+  clear() {
+    clearTimeout(this.timer);
+  }
+  status(reported) {
+    if (this.ending) return [this.ending.code, this.ending.sig];
+    return [typeof reported === "number" ? reported : CRASHED, void 0];
+  }
+};
 function spawnWasmProcess(opts) {
   const sab = new SharedArrayBuffer(SAB_HEADER_BYTES + SAB_DEFAULT_WINDOW_BYTES);
   const header2 = new Int32Array(sab, 0, SAB_HEADER_I32);
@@ -8403,6 +8453,7 @@ function spawnWasmProcess(opts) {
     forker: opts.forker,
     fs: opts.fs,
     kill: opts.kill,
+    ...opts.writesBack ? { writesBack: opts.writesBack } : {},
     processes: opts.processes,
     openFiles: opts.openFiles,
     ...opts.nodes ? { nodes: opts.nodes } : {},
@@ -8433,18 +8484,16 @@ function spawnWasmProcess(opts) {
     renamed: (from, to) => holds.renamed(from, to)
   });
   const worker = opts.createWorker();
-  const dispatch2 = async (req) => {
-    if (isWasmSyscall(req)) return process.syscall(req);
-    return dispatchSyncFs(req);
-  };
+  const dying = new Dying();
+  const dispatch2 = (req) => dying.serving(() => isWasmSyscall(req) ? process.syscall(req) : dispatchSyncFs(req));
   const responder = attachSyncSabResponder(worker, sab, token, { dispatch: dispatch2 });
-  let settle3;
-  const exited = new Promise((resolve) => settle3 = resolve);
+  const { promise: exited, resolve: settle3 } = Promise.withResolvers();
   let done4 = false;
   let endedBy;
   const finish = (code, sig) => {
     if (done4) return;
     done4 = true;
+    dying.clear();
     endedBy = sig ?? process.execTermsig;
     worker.removeEventListener("message", onMessage);
     worker.removeEventListener("error", onError);
@@ -8460,10 +8509,10 @@ function spawnWasmProcess(opts) {
   };
   const handle = (data) => {
     if (data?.type === WASM_PROCESS_EXIT) {
-      finish(typeof data.code === "number" ? data.code : CRASHED);
+      finish(...dying.status(data.code));
     } else if (data?.type === WASM_PROCESS_ERROR) {
-      opts.onError?.(String(data.message));
-      finish(CRASHED);
+      if (!dying.ending) opts.onError?.(String(data.message));
+      finish(...dying.status(CRASHED));
     } else if (data?.type === WASM_THREAD_SPAWN && data.thread) {
       spawnThread(data.thread);
     }
@@ -8533,7 +8582,10 @@ function spawnWasmProcess(opts) {
   };
   worker.postMessage(init, opts.fork ? [opts.fork.memory.buffer] : []);
   const signal = (sig) => {
-    if (process.signal(sig) === "terminate") finish(sig === SIG.KILL ? 137 : 128 + sig, sig);
+    if (process.signal(sig) !== "terminate") return;
+    const code = sig === SIG.KILL ? 137 : 128 + sig;
+    if (opts.program.abi !== "wasi" || done4) finish(code, sig);
+    else dying.start(code, sig, header2, process, () => finish(code, sig));
   };
   const kill = (code = 137) => finish(code, code > 128 && code < 160 ? code - 128 : void 0);
   return {
@@ -8544,7 +8596,8 @@ function spawnWasmProcess(opts) {
     termsig: () => endedBy,
     onState: (listener) => process.onState(listener),
     memory: () => Atomics.load(header2, SAB_I_MEMORY) * 65536,
-    ignoredSignals: (fork) => process.inheritable(fork)
+    ignoredSignals: (fork) => process.inheritable(fork),
+    writesBack: () => opts.program.abi === "wasi" && !done4
   };
 }
 
@@ -10273,6 +10326,7 @@ var Launcher = class {
       spawner: this.spawner(pid, req.report),
       forker: this.forker(pid, req),
       kill: (target, sig) => this.kill(target, sig),
+      writesBack: (target) => this.processes.get(target)?.writesBack() === true,
       processes: () => ({ boot: this.boot, processes: this.list() }),
       openFiles: this.openFiles,
       nodes: this.nodes,