@ai-ecoverse/slicc-kernel 1.109.0 → 1.110.1

package/README.md

@@ -227,6 +227,8 @@ A command's name is any single path component without whitespace or control char
 
 **More roots:** `createKernel({ commands: { append: ['/opt/user/node_modules'] } })` (and `createNodeKernel`) scans more directories, each absolute and normalised (anything else throws a `TypeError`). A name is looked up in a user's own global packages first, then in `modules`, then in root's global packages, then in the appended roots in the order given, and the first match wins, so an appended root never shadows another command (or `sudo`). Their commands are in `/usr/bin` for every user, like root's global packages, and their `binfmt` and file system entries count too. A package in an appended root also gets a command for each `bin` entry of its `package.json` (a string `bin` is named after the package, without its scope), which runs as a `#!` script; a name its `slicc.commands` gives wins. A `bin` entry is skipped when its name isn't a command name (see above), or its file, with symlinks resolved, isn't a regular file inside the package. `/node_modules` and the global packages keep reading only `slicc.commands`. A missing root counts as empty until it appears, and an install or removal in a root is picked up as it happens. With `users: 'enforce'` a root counts only when it and every directory above it are root's and not writable by group or others; another root is skipped without a word. Under `'report'` such a root is still scanned, and reported once to `onDenial` and the report log as `{ pid: 0, uid: 0, op: 'commands', path, errno: 'EACCES' }`. Running a `bin` command needs `r` and `x` on its file under `users` (see [Permissions](#permissions)), checked when it runs, so a `0644` file is `EACCES` under `'enforce'`.
 
+**Variants by environment** ([homescoop#357](https://github.com/ai-ecoverse/homescoop/issues/357)): one package can carry more than one build of a wasm command and choose between them by the environment. `"variants": [{ "env": { "PYTHON_FORK": "true" }, "wasm": "bin/python-fork.wasm" }]` on a command entry (Emscripten or WASI) is matched in order against the environment the new process gets, at spawn and at exec. Every key must equal its value exactly, and the first match runs its `wasm`; no match runs the command's own `wasm`. This holds however the kernel reaches the entry: by name, by its path (`/usr/bin/python3`, as `sys.executable` gives it to `subprocess` and `multiprocessing`), or through a symlink to that path (a venv's `bin/python`). So `PYTHON_FORK=true python3`, `PYTHON_FORK=true exec /usr/bin/python3` and a child that inherits `PYTHON_FORK=true` all get the variant, and `PYTHON_FORK=1` doesn't. Commands of one package that share a build (`python`, `python3`, `pip`, `pip3`) each carry the same `variants`. A variant replaces only the module: the `abi`, the glue, `env`, `argv0`, `argv0Path` and the `/usr/bin` entry stay the command's. For an Emscripten command that means a variant swaps only the `.wasm` and keeps the command's JS glue, so it must be built against that same glue; a WASI or WASIX command has no glue, so there's no such constraint. Only the chosen module is read, checked and compiled, and compiled modules are cached by its path. A read or `stat` of `/usr/bin/<name>` still shows the default `wasm`. A `variants` that isn't an array, or has an entry without an `env` object of strings and a `wasm` inside the package, is ignored as a whole, with one warning per command, and the command runs its `wasm`. An older kernel ignores the field.
+
 A package can also name a command that runs files by their suffix, as `binfmt_misc` does: `"slicc": { "binfmt": { ".jsh": "jsh" } }`. Executing a file that has no `#!` line and is not a wasm module, whose name ends in a mapped suffix, runs that command with the file's path as invoked and the arguments after it, as a `#!` line naming the command would; a `#!` line still wins. The longest matching suffix wins, the whole name is never a suffix (`.jsh` alone is not one), and of two packages that map one suffix, the one whose name sorts first wins, as for commands; the kernel warns once (`console.warn`) when another package maps that suffix to a different command. A suffix starts with a dot and the command must be a name the catalog resolves when the file runs.
 
 ### WASI programs
@@ -878,7 +880,7 @@ Not covered: hard links (OPFS cannot share contents between names; they work onl
 
 Reads retry briefly when a concurrent write has invalidated the OPFS file snapshot (`NotReadableError`).
 
-A kernel caches the directory handles it has walked and the sidecar entries it has read. Its own changes keep both caches current, and the other kernels on the origin hear of them through a `BroadcastChannel` named after the database (one for the sidecar, one for directories). A cached directory is checked against its path (`resolve()`) at most once a second before a lookup goes through it, so a directory moved, removed or replaced by another writer, such as a page using the OPFS API, is found at its new place or reported missing within a second.
+A kernel caches the directory handles it has walked and the sidecar entries it has read. Its own changes keep both caches current, and the other kernels on the origin hear of them through a `BroadcastChannel` named after the database (one for the sidecar, one for directories). A cached directory is checked against its path (`resolve()`) at most once a second before a lookup goes through it, so a directory moved, removed or replaced by another writer, such as a page using the OPFS API, is found at its new place or reported missing within a second. It also keeps the file handles it has looked up (up to 4,096, the least recently used dropped first) for a second after each lookup. Before it uses one, it reads the file through it, which is the read a `stat` needs anyway, so a file another writer removed is missing at once and a rewritten one shows its new contents. One replaced by a directory is seen as a directory within a second. Its own changes, and the other kernels' notices, drop the handles they touch at once.
 
 The filesystem is our own rather than [ZenFS](https://github.com/zen-fs/core) (which SLICC uses), because OPFS stays the single source of truth: other writers, such as the BIOS installing packages or a page writing files, need no index to stay consistent with, and nothing is preloaded into memory at mount. POSIX metadata lives in the IndexedDB sidecar described above, which has per-entry transactions instead of one JSON file rewritten on every change.
 
@@ -893,6 +895,7 @@ The kernel is ported from SLICC's `packages/webapp/src/kernel/` with the browser
 - `src/launcher.ts`, `src/commands.ts`, `src/serve.ts`, `src/index.ts`, `src/users.ts`: command resolution, the kernel worker protocol, the page API and the user database; `src/node.ts`, `src/node-process-worker.ts` and `src/node/` the headless Node entry. An embedder that uses `Launcher` directly, without `createKernel`, calls `await launcher.prepare()` first: it creates `/tmp` and `/home` and writes the CA certificate.
 - `src/kernel/net/`: the proxy, HTTP/1.1, TLS termination and the local CA, and the bridge to the page's transport; `src/transport.ts` and `src/local-proxy-transport.ts` are the page side.
 - `src/cdp/`: the CDP facade on `127.0.0.1:9222`, its WebSocket framing, the registry of hosts and the `MessagePort` bridge to them.
+- `src/cron/`: the cron service (`service.ts`), the schedule and table parsers, the clock, and the `crontab` command with the kernel side of `ctx.crontab`. An embedder that uses `Launcher` directly passes `cron: { lock }` to share the firing lock among the kernels on one root; the default, `slicc-kernel-cron`, is one lock for the whole origin.
 
 
 ## Installing from git
@@ -954,6 +957,21 @@ Releases are cut by semantic-release on every push to `main`, in two jobs, and o
 
 Moving the `certified` dist-tag stays manual. Automating it needs an npm token that may change dist-tags, in a step that re-hashes the tarball and tags only on a `certified` line.
 
+### Certification in the pull request (`cert-review-pr`)
+
+`.github/workflows/cert-review-pr.yml` certifies a pull request before it merges, so `main`, and with it npm's `latest`, only gets certified changes. For now it reports as the `cert-review-pr` commit status and isn't a required check. It becomes the required `cert-review` once it has proved itself on real pull requests; the post-publish review above stays until then.
+
+- **Which pull requests:** those that change shipped behaviour, by a conservative path filter (`behaviourFiles` in `lib.mjs`): `src/`, `build.mjs`, `package.json` and its lock, `tsconfig*.json`, `.releaserc*`, `tools/release-recover.mjs`, the workflows that build or publish (`release.yml`, `ci.yml`), and the reviewer's own code (`tools/cert-review/`, `cert-review*.yml`, `.github/runner.mjs`). **Main's code decides:** every job checks out the pull request's base and fetches only the trees under test, so the path filter, pack, suite, findings, the instructions and record all run from `main`. A pull request that changes the reviewer is reviewed by `main`'s reviewer. The workflow file of a `pull_request` run still comes from the pull request, so **pull requests that touch the reviewer always go to the Kernel Coordinator before they are queued.** The pull request that adds this workflow can't pass its own check, because its base has no tools yet. Any other pull request gets a green status that says `not behaviour: <files>`, so the skip is recorded.
+- **When:** only for a same-repo pull request with `merge-ok`, when the label is added and on every push while it is set. Each pull request has its own concurrency group, and a newer push cancels the older review. A behaviour change without `merge-ok` gets a red status, `waits for merge-ok to be reviewed`. Other labels don't start or cancel anything. Fork code never reaches the pools or the key.
+- **What:** the checkout is the event's default `refs/pull/N/merge`, the pull request merged onto `main` at review time, so a review is of what would land, without pushing to the branch. Both sides are packed in the same run, on a GitHub-hosted runner, with no secrets: the candidate (the merge ref, `x.y.z-pr.N.<run>`) and the base (its first parent, `main`). Then:
+  - the candidate's whole suite, and the base's;
+  - the pull request's new or changed test files on the base build (the NEGATIVE: a fix's test must fail on `main` and pass on the candidate);
+  - kernel-compat on the candidate's artifact (#812), against npm's `latest`;
+  - the review (Claude through `claude-code-action/base-action`, with the Bedrock key in that step only, no issue filing);
+  - and record, which re-hashes the candidate's artifact against the pack job's sha256.
+- **The verdict** goes into the commit status on the pull request's head and a comment on the pull request. Certified is green. A FAIL is red with the line, and the pull request needs a fix. **INCOMPLETE** (a job that didn't succeed, no verdict, a suite without results) is red with `re-run the failed jobs`, never a silent pass. Main moving on after a review doesn't trigger another one; a push does.
+- **Merge groups only look up.** On `merge_group`, the job lists the group's pull requests (its merge commits) and requires, for each one, a green status on the head being merged, or no behaviour change. Otherwise it's red, naming the pull request. It runs in seconds, with no secrets and no model. The group's `ci / test` covers the combined tree.
+
 Not part of the run yet: bios's real setup (the base-set sweep in bios, the upgrade boot, the sudo rows under `enforce`). It would plug in as one more job before review that runs slicc-bios's `npm run test:cert` with `CERT_COMMANDS` and the release's tarball, and leaves its results next to the others.
 
 ## Test fixtures
@@ -1086,9 +1104,9 @@ A threaded WASIX C program for `test/integration/thread-signal.test.mjs` (#520).
 
 A WASIX C program for `test/integration/long-timers.test.mjs` (#767).
 - `longwait alarm` sets `alarm(3000000)` (about 35 days), sleeps 0.5 s, and prints how many alarms came and how many seconds were left.
-- `longwait select|ppoll|nanosleep` waits 3,000,000 s on a pipe that never gets data, until a 300 ms interval timer interrupts the wait, and prints `not before it` when the wait lasted at least 300 ms (`early` otherwise). It sets no upper bound, so a loaded CI host can't fail the row; a wait that never ends times the test out.
+- `longwait select|ppoll|nanosleep` waits 3,000,000 s on a pipe that never gets data, until a 300 ms interval timer interrupts the wait, and prints `not before it` when the wait lasted at least 300 ms (`early` otherwise), measured from before `setitimer` arms the timer, so the time the arming call takes can't shorten it (#858). It sets no upper bound, so a loaded CI host can't fail the row; a wait that never ends times the test out.
 
-**Build:** `WASIX_SYSROOT_28=/path/to/wasix-sysroot ./build.sh`, with `wasixcc` 0.4.7 and binaryen's `wasm-opt` on `PATH`. `bin/longwait.wasm` is 44,004 bytes, sha256 `e69d03f01479693666078297e781f5373719700c8534d1f2933749f5b3cee903`.
+**Build:** `WASIX_SYSROOT_28=/path/to/wasix-sysroot ./build.sh`, with `wasixcc` 0.4.7 and binaryen's `wasm-opt` on `PATH`. `bin/longwait.wasm` is 44,002 bytes, sha256 `24acd996068ea04c526b74c93b896986eef782ec96230e19fb939d1b00a1bad0`.
 
 ### `epollmask`
 

package/dist/kernel-worker.js

@@ -984,6 +984,8 @@ var S_ISGID2 = 1024;
 var READ_ATTEMPTS = 3;
 var MAX_LINKS2 = 40;
 var CHECK_MS = 1e3;
+var FILE_HANDLES = 4096;
+var STALE = /* @__PURE__ */ new Set(["NotFoundError", "TypeMismatchError"]);
 var EXECUTABLE = /\/node_modules\/(?:@[^/]+\/)?[^/]+\/bin\//;
 var ERRNO_BY_DOM_ERROR = {
   NotFoundError: "ENOENT",
@@ -1109,6 +1111,9 @@ var OpfsFs = class {
   root;
   meta;
   dirs = new PathMap();
+  files = new PathMap();
+  snapshots = /* @__PURE__ */ new WeakMap();
+  found = /* @__PURE__ */ new WeakMap();
   checked = /* @__PURE__ */ new WeakMap();
   channel;
   remote;
@@ -1346,6 +1351,19 @@ var OpfsFs = class {
   }
   forget(path) {
     this.dirs.deleteTree(path);
+    this.files.deleteTree(path);
+  }
+  remember(path, handle, looked = false) {
+    if (handle.kind !== "file") return handle;
+    if (looked) {
+      this.found.set(handle, performance.now());
+      this.snapshots.delete(handle);
+    }
+    this.files.delete(path);
+    if (this.files.size >= FILE_HANDLES)
+      this.files.delete(this.files.keys().next().value);
+    this.files.set(path, handle);
+    return handle;
   }
   async inDir(path, op) {
     const dir = await this.dir(path);
@@ -1369,16 +1387,30 @@ var OpfsFs = class {
   }
   async handle(path) {
     if (split(path).length === 0) return this.root;
+    const cached = this.files.get(path);
+    const fresh2 = cached && performance.now() - (this.found.get(cached) ?? -CHECK_MS) < CHECK_MS;
+    if (cached && !fresh2) this.files.delete(path);
+    if (cached && fresh2) {
+      try {
+        this.snapshots.set(cached, await cached.getFile());
+        return this.remember(path, cached);
+      } catch (err) {
+        if (!STALE.has(err?.name))
+          throw translate(err, path, "read");
+        this.files.delete(path);
+        this.snapshots.delete(cached);
+      }
+    }
     const [dir, name2] = await this.parent(path);
     const known = this.dirs.has(path);
     return this.inDir(dir, async (parent) => {
       const [first, second] = known ? ["directory", "file"] : ["file", "directory"];
       try {
-        return await this.child(parent, name2, first);
+        return this.remember(path, await this.child(parent, name2, first), true);
       } catch (err) {
         if (err?.name !== "TypeMismatchError") throw err;
       }
-      return this.child(parent, name2, second);
+      return this.remember(path, await this.child(parent, name2, second), true);
     }).catch((err) => {
       throw translate(err, path);
     });
@@ -1521,9 +1553,15 @@ var OpfsFs = class {
       if (this.dirTimes.get(path) === ms) this.dirTimes.delete(path);
     }
   }
+  async snapshot(handle) {
+    const taken = this.snapshots.get(handle);
+    if (!taken) return handle.getFile();
+    this.snapshots.delete(handle);
+    return taken;
+  }
   async statOf(path, handle, meta, touched) {
     const directory = handle.kind === "directory";
-    const file = directory ? void 0 : await handle.getFile();
+    const file = directory ? void 0 : await this.snapshot(handle);
     const modified = file?.lastModified ?? 0;
     const kept = meta?.mtimeMs !== void 0 && meta.mtimeFor === modified ? meta.mtimeMs : modified;
     const mtime = touched ?? kept;
@@ -2645,14 +2683,14 @@ var WsReader = class {
   }
 };
 function switchingHead(accept2, protocol = "") {
-  const chosen = protocol ? `Sec-WebSocket-Protocol: ${protocol}\r
+  const chosen2 = protocol ? `Sec-WebSocket-Protocol: ${protocol}\r
 ` : "";
   return latin1Bytes(
     `HTTP/1.1 101 Switching Protocols\r
 Upgrade: websocket\r
 Connection: Upgrade\r
 Sec-WebSocket-Accept: ${accept2}\r
-${chosen}\r
+${chosen2}\r
 `
   );
 }
@@ -6108,7 +6146,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.109.0" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.110.1" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -6156,6 +6194,37 @@ function argsOf(pkg, raw) {
   if (!raw.every((arg) => typeof arg === "string")) return void 0;
   return raw.map((arg) => arg.replace(PACKAGE, pkg));
 }
+var warnedVariants = /* @__PURE__ */ new Set();
+function variantOf(pkg, raw) {
+  if (!raw || typeof raw !== "object" || Array.isArray(raw)) return void 0;
+  const { env, wasm } = raw;
+  const path = inside(pkg, wasm);
+  if (!path || !env || typeof env !== "object" || Array.isArray(env)) return void 0;
+  const pairs = Object.entries(env);
+  if (pairs.length === 0) return void 0;
+  if (!pairs.every(([key, value2]) => ENV_KEY.test(key) && typeof value2 === "string")) {
+    return void 0;
+  }
+  return { env: Object.fromEntries(pairs), wasm: path };
+}
+function variantsOf(pkg, name2, raw) {
+  if (raw === void 0) return void 0;
+  const variants = Array.isArray(raw) ? raw.map((one) => variantOf(pkg, one)) : [void 0];
+  if (variants.every((one) => one !== void 0)) {
+    return variants.length > 0 ? variants : void 0;
+  }
+  const key = `${pkg}:${name2}:${JSON.stringify(raw)}`;
+  if (!warnedVariants.has(key)) {
+    warnedVariants.add(key);
+    console.warn(`slicc-kernel: ${pkg}: ${name2}: ignoring malformed "variants", running its wasm`);
+  }
+  return void 0;
+}
+function variantFor(variants, env) {
+  return variants?.find(
+    (one) => Object.entries(one.env).every(([key, value2]) => env[key] === value2)
+  )?.wasm;
+}
 function abiOf(raw, fallback) {
   const abi = raw ?? fallback;
   return abi === "emscripten" || abi === "wasi" || abi === "js" ? abi : void 0;
@@ -6189,6 +6258,7 @@ function commandOf(pkg, name2, raw, packageAbi, shared) {
   const glue = abi === "wasi" ? wasm : inside(pkg, raw.glue);
   if (!abi || !glue || !wasm) return void 0;
   const imports = abi === "wasi" ? inside(pkg, raw.imports) : void 0;
+  const variants = variantsOf(pkg, name2, raw.variants);
   return {
     name: name2,
     abi,
@@ -6200,7 +6270,8 @@ function commandOf(pkg, name2, raw, packageAbi, shared) {
     ...abi === "wasi" && raw.preopenRoot === true ? { preopenRoot: true } : {},
     ...args ? { args } : {},
     ...withEnv,
-    ...imports ? { imports } : {}
+    ...imports ? { imports } : {},
+    ...variants ? { variants } : {}
   };
 }
 function commandsOf(pkg, manifest) {
@@ -6520,13 +6591,14 @@ var CRONTAB_SOURCE = `export default async function crontab(ctx) {
   let text;
   try {
     if (file === "-") {
+      const stream = new TextDecoder();
       const parts = [];
       for (;;) {
         const chunk = await ctx.read(0);
         if (chunk.length === 0) break;
-        parts.push(decode(chunk));
+        parts.push(stream.decode(chunk, { stream: true }));
       }
-      text = parts.join("");
+      text = parts.join("") + stream.decode();
     } else text = decode(await ctx.fs.readFile(ctx.resolve(file)));
   } catch (err) {
     await say(2, "crontab: " + file + ": " + (err && err.code ? err.code : String(err)) + "\\n");
@@ -7294,7 +7366,10 @@ var CronService = class {
       const account = await this.host.lookup(job.user);
       if (account) out.push({ source: "system", account, job });
     }
-    const names = await fs.readdir(SPOOL_DIR).catch(() => []);
+    const names = await fs.readdir(SPOOL_DIR).catch((err) => {
+      if (err?.code === "ENOENT") return [];
+      throw err;
+    });
     for (const name2 of names.sort()) {
       if (name2.startsWith(".") || name2.endsWith("~")) continue;
       const path = `${SPOOL_DIR}/${name2}`;
@@ -21976,9 +22051,18 @@ function targetOf(command) {
     ...command.env ? { env: command.env } : {},
     ...command.unset ? { unset: command.unset } : {},
     ...command.imports ? { imports: command.imports } : {},
-    ...command.preopenRoot ? { preopenRoot: true } : {}
+    ...command.preopenRoot ? { preopenRoot: true } : {},
+    ...command.variants ? { variants: command.variants } : {}
   };
 }
+function chosen(planned, env) {
+  const { target } = planned;
+  const wasm = variantFor(target.variants, env);
+  if (wasm === void 0) return planned;
+  const { asked: _asked, ...rest2 } = target;
+  const glue = target.abi === "wasi" ? wasm : target.glue;
+  return { ...planned, target: { ...rest2, wasm, glue } };
+}
 function withScriptEnv(target, command) {
   if (!command) return target;
   return {
@@ -22889,14 +22973,15 @@ var Launcher = class {
       ...target.preopenRoot ? { preopenRoot: true } : {}
     };
   }
-  async launch(planned, req) {
-    const { target, args } = planned;
+  async launch(asked, req) {
     const cwd = await physicalPathAsync(this.base, req.cwd);
     const seen = { ...req.env, HOSTNAME: this.hostname, cwd };
-    const env = { ...expandDefaults(target.env, seen), ...req.env };
-    for (const key of target.unset ?? []) delete env[key];
+    const env = { ...expandDefaults(asked.target.env, seen), ...req.env };
+    for (const key of asked.target.unset ?? []) delete env[key];
     delete env[TICKET_ENV];
     env.HOSTNAME = this.hostname;
+    const planned = chosen(asked, env);
+    const { target, args } = planned;
     let program;
     await this.pnpmReady();
     try {