@ai-ecoverse/slicc-kernel 1.170.2 → 1.170.3

package/README.md

@@ -234,7 +234,7 @@ A WASI preview1 or WASIX program (Zig, Rust, Go, wasi-libc or wasix-libc C) has
 
 A command's name is any single path component without whitespace or control characters, so coreutils' `[` is in `/usr/bin` too; a name that is empty, `.` or `..`, or has a `/`, a space, a tab, a newline, NUL or another control character is skipped. Entries may also set `argv0`, `args` and `env`, and a package can set `slicc.env` for all of its commands. A WASI entry with `"preopenRoot": true` also gets `/` as a preopen (see [WASI programs](#wasi-programs)). An entry with `"argv0Path": true` gets the absolute path it was run by as `argv[0]` when that path is a link outside `/bin` and `/usr/bin` to the command, as a venv's `bin/python` is (the `#!` line of a script naming such a link included), so it can find what sits beside that link; other entries keep their `argv0`. In `env`, `${package}` is the package directory, `${cwd}` the process's working directory, and `${NAME}` the caller's `NAME` when the program starts (a default naming an unset variable is left out); a relative path that exists in the package (`lib/python3.14`, `./`) becomes that absolute path. The caller's environment wins over these defaults; a `null` value removes the variable for the command, after the caller's environment is applied (a host-networked program can drop the kernel's proxy settings that way), and a command entry can set a variable its package's `slicc.env` removes. The catalog reads its roots in order, and the first root that provides a name wins it: a user's own global packages (for that user's processes), `/node_modules`, root's global packages, then each appended root. Within a root, packages are read by name and the first wins (compared as strings by UTF-16 code unit, so a scoped `@scope/name` comes before names that start with a letter, but after those that start with a digit), whatever order the file system lists them in; a `PNPM_HOME`'s install groups count as one root, read group by group (by the group directory's name, effectively arbitrary for pnpm's hashed names), each by package name. A variant can take over from the package it extends with `"slicc": { "replaces": ["@ai-ecoverse/wasm-imagemagick"] }`: when it and a package it names are in the same root, it wins every command name they share, whatever the order; names only the other provides stay the other's, a third package that provides the name and sorts first still wins it, and packages that replace each other fall back to the name order. A replace never reaches into an earlier root, so root's global packages can't take a name from `/node_modules` or a user's own packages. The field must be an array of package names; anything else is ignored. `sh` runs bash as `sh` unless a package provides its own. A package's `engines.slicc-kernel` is a floor the kernel holds it to: when the running kernel's version doesn't satisfy that npm range (comparators, `^`, `~`, x-ranges, hyphen ranges, `||`), or the range can't be read, its commands are refused. They stay in `/usr/bin`, and running one by name, by path or as a `#!` interpreter exits 126 with `<cmd>: <pkg>@<version> needs slicc-kernel <range>; this kernel is <version>` on stderr. Each refused package is logged once to the console. A non-string `engines.slicc-kernel`, or none, is no constraint. A development build reports the newest release tag as `<tag>+dev` and checks against that; a build of unknown version (`0.0.0`) checks nothing. The refusal follows the program, not the name: its glue or `.wasm` run by path (directly or through a symlink), and a byte-identical copy of its glue beside it named for one of its commands (#390, #899), exit 126 with the same message ([#1235](https://github.com/ai-ecoverse/slicc-kernel/issues/1235)). Only commands are refused: a package without commands (a `py-*` library, a side module) isn't, and a refused package that replaces base commands makes those commands fail with 126 rather than fall back to the package it replaces. The installed commands appear as executables in a virtual `/usr/bin` and `/bin`, so `PATH` lookups, `command -v` and `ls /usr/bin` work without writing anything to OPFS. An entry reads as its program, as a PATH search that opens what it finds expects (`ruby -S rake`, `perl -S`, Pod::Usage looking itself up, bundler checking that `/usr/bin/ruby` isn't empty). A script command reads as its script, a JS command as its module, and a WASI or Emscripten command as its `.wasm`, so `cat /usr/bin/hello` prints the script and `head -c4 /usr/bin/cat` gives `\0asm`. The entry is a plain file and no link: it stays a root-owned `0755` regular file with its own inode, `readlink` of it is `EINVAL`, and `which` and `command -v` print `/usr/bin/<name>`. No file stands behind it, so unlinking it is `ENOENT`, open or not, for a user's own entry in `~/.local/share/pnpm/bin` too: `rm -f` does nothing, `pnpm remove -g` counts it as gone, and the entry leaves with its package. Its size and times are the program's when the caller may read the program (anyone, for a kernel-internal look or with users off, only when everyone may), and 0 and the epoch otherwise. Running it still runs the catalog command, with its `env`, `args` and `argv0`. The read is the caller's: with users on, reading an entry also needs `r` on its program, and that includes a user's own commands in `~/.local/share/pnpm/bin`, so an entry whose program the caller can't read is `EACCES`. A process can also run a program by the path of its glue when the `.wasm` sits next to it, a wasm module by its own path (as a WASI program; when it is the `wasm` of a WASI command in the caller's catalog, the user's own packages included, it runs as that command, with its `env` defaults, `args` and `argv0`, the command whose name matches the file's, without `.wasm`, if several share it, while a copy elsewhere runs bare), and a script through its `#!` line (including `#!/usr/bin/env name`). Executing any other file fails with `ENOEXEC`, so bash runs it as a shell script, as POSIX shells do. A spawn by a bare name (no `/`) that the catalog doesn't have, and that didn't ask for no search (a WASIX `posix_spawn` or `execve`, as opposed to its `posix_spawnp`/`execvp`, keeps its bare name), searches the `PATH` of the spawn's own environment, in order, with an empty entry meaning the working directory; each candidate is run exactly as a spawn of that path would be. The first that runs wins; if one was found but refused and nothing else matched, the spawn fails with `EACCES` (126), otherwise `ENOENT` (127). A catalog command still wins over the same name on `PATH`, the reverse of a shell's order. A searched bare name is never looked up in the working directory itself: a file of that name there runs only through an empty `PATH` entry, as with `execvp` ([#1234](https://github.com/ai-ecoverse/slicc-kernel/issues/1234)). The `PATH` searched is the new process's, from the spawn's environment; POSIX `posix_spawnp` uses the caller's, which is the same unless the caller hands the child another one. This is `posix_spawnp`/`execvp` semantics for every caller that asks for a search: JS programs' `ctx.spawn`, WASI programs' spawn import, WASIX `posix_spawnp`, and Emscripten programs built on homescoop's shim before its own `PATH` search (gmake, tar, git and others hand the kernel the bare name). wasm-coreutils from 9.12.0-11 searches `PATH` itself and hands the kernel a path ([#1165](https://github.com/ai-ecoverse/slicc-kernel/issues/1165), [#1168](https://github.com/ai-ecoverse/slicc-kernel/issues/1168)). A `..` in a path that is run is resolved physically, as the file system resolves it: `pkg/../tools/x` after a symlinked `pkg` is the `tools/x` beside the link's target, and its exec permission is checked there. The walk is the caller's: with users on, each directory it passes needs search (`x`) permission (`EACCES`, 126 in bash, as on Linux), and a sudo process follows only the links it trusts, as for any other path. Commands also come from pnpm's global directory. Every process gets a `PNPM_HOME` and has `$PNPM_HOME/bin` on `PATH`: root's is `/usr/local/share/pnpm` (unless the embedder sets its own), a user's `~/.local/share/pnpm` (see [Users](#users)). The kernel scans each of pnpm's global install groups (`$PNPM_HOME/global/*/<group>/node_modules`) after `/node_modules`, which wins when both provide a command. The catalog is re-read whenever anything under either changes, so `pnpm add -g @ai-ecoverse/wasi-ripgrep` makes `rg` work in the same shell, and `pnpm remove -g` removes it again.
 
-**More roots:** `createKernel({ commands: { append: ['/opt/user/node_modules'] } })` (and `createNodeKernel`) scans more directories, each absolute and normalised (anything else throws a `TypeError`). A name is looked up in a user's own global packages first, then in `modules`, then in root's global packages, then in the appended roots in the order given, and the first match wins, so an appended root never shadows another command (or `sudo`). Their commands are in `/usr/bin` for every user, like root's global packages, and their `binfmt` and file system entries count too. A package in an appended root also gets a command for each `bin` entry of its `package.json` (a string `bin` is named after the package, without its scope), which runs as a `#!` script; a name its `slicc.commands` gives wins. A `bin` entry is skipped when its name isn't a command name (see above), or its file, with symlinks resolved, isn't a regular file inside the package. `/node_modules` and the global packages keep reading only `slicc.commands`. A missing root counts as empty until it appears, and an install or removal in a root is picked up as it happens. With `users: 'enforce'` a root counts only when it and every directory above it are root's and not writable by group or others; another root is skipped without a word. Under `'report'` such a root is still scanned, and reported once to `onDenial` and the report log as `{ pid: 0, uid: 0, op: 'commands', path, errno: 'EACCES' }`. Running a `bin` command needs `r` and `x` on its file under `users` (see [Permissions](#permissions)), checked when it runs, so a `0644` file is `EACCES` under `'enforce'`.
+**More roots:** `createKernel({ commands: { append: ['/opt/user/node_modules'] } })` (and `createNodeKernel`) scans more directories, each absolute and normalised (anything else throws a `TypeError`). A name is looked up in a user's own global packages first, then in `modules`, then in root's global packages, then in the appended roots in the order given, and the first match wins, so an appended root never shadows another command (or `sudo`). Commands of a package that a global install only brought along as a dependency come after all of these: see [A global package's dependencies](#commands). Their commands are in `/usr/bin` for every user, like root's global packages, and their `binfmt` and file system entries count too. A package in an appended root also gets a command for each `bin` entry of its `package.json` (a string `bin` is named after the package, without its scope), which runs as a `#!` script; a name its `slicc.commands` gives wins. A `bin` entry is skipped when its name isn't a command name (see above), or its file, with symlinks resolved, isn't a regular file inside the package. `/node_modules` and the global packages keep reading only `slicc.commands`. A missing root counts as empty until it appears, and an install or removal in a root is picked up as it happens. With `users: 'enforce'` a root counts only when it and every directory above it are root's and not writable by group or others; another root is skipped without a word. Under `'report'` such a root is still scanned, and reported once to `onDenial` and the report log as `{ pid: 0, uid: 0, op: 'commands', path, errno: 'EACCES' }`. Running a `bin` command needs `r` and `x` on its file under `users` (see [Permissions](#permissions)), checked when it runs, so a `0644` file is `EACCES` under `'enforce'`.
 
 **Running a glue by path:** executing a package's Emscripten glue by its path runs the command that names it (the one named like the file, when several share it), even with no `.wasm` beside it. A byte-identical copy of the glue in the same directory, named for one of the package's commands, runs as that command too ([#899](https://github.com/ai-ecoverse/slicc-kernel/issues/899)). wasm-clang ships `bin/clang`, `bin/wasm-ld` and the rest as copies of `bin/llvm`, and emcc executes `$LLVM_ROOT/clang`. The caller must be able to read the copy. A copy elsewhere, or a same-named file whose bytes differ, is not the command. The comparison is cached by path, size, mtime and inode.
 
@@ -242,6 +242,8 @@ A command's name is any single path component without whitespace or control char
 
 **Variants by environment** ([homescoop#357](https://github.com/ai-ecoverse/homescoop/issues/357)): one package can carry more than one build of a wasm command and choose between them by the environment. `"variants": [{ "env": { "PYTHON_FORK": "true" }, "wasm": "bin/python-fork.wasm" }]` on a command entry (Emscripten or WASI) is matched in order against the environment the new process gets, at spawn and at exec. Every key must equal its value exactly, and the first match runs its `wasm`; no match runs the command's own `wasm`. This holds however the kernel reaches the entry: by name, by its path (`/usr/bin/python3`, as `sys.executable` gives it to `subprocess` and `multiprocessing`), or through a symlink to that path (a venv's `bin/python`). So `PYTHON_FORK=true python3`, `PYTHON_FORK=true exec /usr/bin/python3` and a child that inherits `PYTHON_FORK=true` all get the variant, and `PYTHON_FORK=1` doesn't. Commands of one package that share a build (`python`, `python3`, `pip`, `pip3`) each carry the same `variants`. A variant replaces only the module: the `abi`, the glue, `env`, `argv0`, `argv0Path` and the `/usr/bin` entry stay the command's. For an Emscripten command that means a variant swaps only the `.wasm` and keeps the command's JS glue, so it must be built against that same glue; a WASI or WASIX command has no glue, so there's no such constraint. Only the chosen module is read, checked and compiled, and compiled modules are cached by its path. A read or `stat` of `/usr/bin/<name>` still shows the default `wasm`. A `variants` that isn't an array, or has an entry without an `env` object of strings and a `wasm` inside the package, is ignored as a whole, with one warning per command, and the command runs its `wasm`. An older kernel ignores the field.
 
+**A global package's dependencies** ([slicc-cdp#676](https://github.com/ai-ecoverse/slicc-cdp/issues/676)): pnpm hoists a global install's dependencies into the same `node_modules` as the package itself, so the kernel tells them apart by the `dependencies` of the global project's `package.json` (`global/v11/<project>/package.json`), which names only what was installed directly. A package there that the list doesn't name keeps its commands, but they are looked up last, after every other root, for root's global packages as for a user's. So installing a package never changes which `sed` or `playwright-cli` you run through its dependencies, and a dependency's `perl` still runs when nothing else provides `perl`. A package installed directly in any of a user's global projects counts as direct, even if another one also brings it along as a dependency. A project without a readable `package.json` counts every package as direct. This covers commands only; `binfmt` and file system entries of a dependency count as before.
+
 **A pnpm bin of a kernel command** ([#1453](https://github.com/ai-ecoverse/slicc-kernel/issues/1453)): a `#!/usr/bin/env node` (or `#!/usr/bin/node`) script run by its path, whose package lists the same name in `slicc.commands`, runs that kernel command, even when a `node` command exists. This is the bin shim `pnpm add -g` writes into `$PNPM_HOME/bin`, which names its target in a `// pnpm-wasm-shim={"target":…}` line; a script without that line is run as a script, so a same-named file in a package that differs from its glue still doesn't run as the command (#899). The shim's target, with its links resolved, must lie in a package installed directly under one of the command roots, and the name must be the catalog's command from that same package, so a shim only ever runs a command the caller could run by name, with the same checks; with `integrity` on, the shim itself is verified as a source. So `pnpm add -g <pkg> && <cmd>` works in one line.
 
 A package can also name a command that runs files by their suffix, as `binfmt_misc` does: `"slicc": { "binfmt": { ".jsh": "jsh" } }`. Executing a file that has no `#!` line and is not a wasm module, whose name ends in a mapped suffix, runs that command with the file's path as invoked and the arguments after it, as a `#!` line naming the command would; a `#!` line still wins. The longest matching suffix wins, the whole name is never a suffix (`.jsh` alone is not one), and of two packages that map one suffix, the one whose name sorts first wins, as for commands; the kernel warns once (`console.warn`) when another package maps that suffix to a different command. A suffix starts with a dot and the command must be a name the catalog resolves when the file runs.
@@ -1026,7 +1028,7 @@ Every process has credentials in the kernel, as on Linux: real, effective and sa
 
 - **The database:** `/etc/passwd` and `/etc/group` are real files. On first start the kernel writes them when they're missing, with `root` (uid 0, home `/root`, created `0700`) and `nobody` (65534), and the groups `root`, `sudo` (27), `users` (100) and `nogroup` (65534); files already there are kept as they are. The kernel reads them for every lookup, so an edit counts at once. [`kernel.users`](#kernelusers) adds and removes users.
 - **Environment:** a process started as a user gets `HOME`, `USER`, `LOGNAME`, `PNPM_HOME`, `NPM_CONFIG_PREFIX` and `PATH`. A user's `PNPM_HOME` is `~/.local/share/pnpm` and its npm prefix `~/.local`; root's are `/usr/local/share/pnpm` (or the embedder's `env.PNPM_HOME`) and `/usr/local`. `PATH` is `$PNPM_HOME/bin:/usr/local/bin:/usr/bin:/bin`. pnpm makes only `$PNPM_HOME/bin`, so the kernel makes root's `PNPM_HOME` when it is missing, just before a process starts: once at boot, and again after any mount or unmount, a removable medium inserted, a `hostfs` folder coming back, or a change at that directory or above it (removing it, say), and after a failed try (no medium yet). With owners on, the directories it makes there get root's owner and `0755`, as the boot layout does. A process already running when a medium arrives gets it at its next child launch. Writes inside it cost nothing extra.
-- **Commands per user:** root's global packages (`pnpm add -g` as root) are commands every user finds in `/usr/bin`. A user's own global packages are commands for that user only: they show up in the user's `~/.local/share/pnpm/bin`, first on its `PATH`, and a program the user starts finds them by name before the system's; another user, root included, doesn't. A user's packages can map file suffixes (`binfmt`) for that user too. The kernel picks up an install, a removal and a new user as they happen.
+- **Commands per user:** root's global packages (`pnpm add -g` as root) are commands every user finds in `/usr/bin`. A user's own global packages are commands for that user only: they show up in the user's `~/.local/share/pnpm/bin`, first on its `PATH`, and a program the user starts finds them by name before the system's; another user, root included, doesn't. A command that only a dependency of those packages provides shows up there only when no other root has the name (see [Commands](#commands)). A user's packages can map file suffixes (`binfmt`) for that user too. The kernel picks up an install, a removal and a new user as they happen.
 - **Changing credentials:** POSIX `setresuid`/`setresgid` rules. Root (effective uid 0) may set any id and the group list; anyone else may only swap among their own real, effective and saved ids, and gets `EPERM` otherwise.
 - **Signals:** a process may signal another if its effective uid is 0, or if its real or effective uid is the target's real or saved uid; otherwise `kill` fails with `EPERM`. A group kill signals the members it may, and fails with `EPERM` only when it may signal none.
 - **Visibility:** a process that isn't root sees only processes whose real or effective uid is its effective uid, in `/proc` and in `ps`, like Linux's `hidepid=2`; so does a client bound to a user.

package/dist/kernel-worker.js

@@ -8498,7 +8498,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.170.2" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.170.3" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -8767,9 +8767,19 @@ function replacesOf(manifest) {
   if (!Array.isArray(replaces)) return [];
   return replaces.every((name2) => typeof name2 === "string" && PACKAGE_NAME.test(name2)) ? replaces : [];
 }
-async function tierPackages(fs, physical, roots, bins, kernel) {
+async function directInstalls(fs, root) {
+  try {
+    const project = root.slice(0, root.lastIndexOf("/"));
+    const { dependencies } = JSON.parse(await fs.readFile(`${project}/package.json`));
+    return new Set(Object.keys(dependencies ?? {}));
+  } catch {
+    return void 0;
+  }
+}
+async function tierPackages(fs, physical, roots, bins, kernel, global) {
   const out = [];
   for (const root of roots) {
+    const direct = global.has(root) ? await directInstalls(fs, root) : void 0;
     for (const pkg of await packages(fs, root)) {
       let manifest;
       try {
@@ -8781,12 +8791,15 @@ async function tierPackages(fs, physical, roots, bins, kernel) {
       const named = new Set(own.map((command) => command.name));
       const extra = bins ? await binCommandsOf(physical, pkg, manifest, named) : [];
       const refused3 = engineRefusal(manifest, kernel);
-      const commands = [...own, ...extra];
+      const name2 = pkg.slice(root.length + 1);
+      const dependency = direct?.has(name2) === false;
+      const marks = { ...refused3 ? { refused: refused3 } : {}, ...dependency ? { dependency } : {} };
       out.push({
-        name: pkg.slice(root.length + 1),
+        name: name2,
         pkg,
         replaces: replacesOf(manifest),
-        commands: refused3 ? commands.map((command) => ({ ...command, refused: refused3 })) : commands
+        commands: [...own, ...extra].map((command) => ({ ...command, ...marks })),
+        dependency
       });
     }
   }
@@ -8812,19 +8825,25 @@ function winners(tier) {
   }
   return out;
 }
-async function scanCommands(fs, modules, binsFrom = Number.POSITIVE_INFINITY, kernel = KERNEL_VERSION, refused3) {
+async function scanCommands(fs, modules, binsFrom = Number.POSITIVE_INFINITY, kernel = KERNEL_VERSION, refused3, global = /* @__PURE__ */ new Set()) {
   const found = /* @__PURE__ */ new Map();
   const physical = physicalResolver(fs);
   const tiers = (typeof modules === "string" ? [modules] : modules).map(
     (tier) => typeof tier === "string" ? [tier] : tier
   );
-  for (const [index, roots] of tiers.entries()) {
-    const tier = await tierPackages(fs, physical, roots, index >= binsFrom, kernel);
-    if (refused3) refused3.push(...tier.flatMap((entry) => entry.commands.filter((c) => c.refused)));
+  const take = async (tier) => {
     for (const [name2, { pkg, command }] of winners(tier)) {
       if (!found.has(name2)) found.set(name2, await withPackagePaths(fs, pkg, command));
     }
+  };
+  const dependencies = [];
+  for (const [index, roots] of tiers.entries()) {
+    const tier = await tierPackages(fs, physical, roots, index >= binsFrom, kernel, global);
+    if (refused3) refused3.push(...tier.flatMap((entry) => entry.commands.filter((c) => c.refused)));
+    await take(tier.filter((entry) => !entry.dependency));
+    dependencies.push(tier.filter((entry) => entry.dependency));
   }
+  for (const tier of dependencies) await take(tier);
   if (!found.has("sh") && found.has("bash")) {
     found.set("sh", { ...found.get("bash"), name: "sh", argv0: "sh" });
   }
@@ -32025,7 +32044,8 @@ var Launcher = class {
           tiers,
           tiers.length - appended2.length,
           void 0,
-          refused3
+          refused3,
+          await this.globalRoots(scope)
         );
         this.refusedOf.set(commands, refused3.map(refusing));
         return commands;
@@ -32050,11 +32070,31 @@ var Launcher = class {
   ownCommands(scope) {
     let found = this.ownCatalogs.get(scope);
     if (!found) {
-      found = pnpmGlobalRoots(this.base, scope).then((roots) => scanCommands(this.scanned, roots)).then((commands) => new Set(commands.keys()));
+      found = this.ownCatalog(scope);
       this.ownCatalogs.set(scope, found);
     }
     return found;
   }
+  async ownCatalog(scope) {
+    const roots = await pnpmGlobalRoots(this.base, scope);
+    const own = await scanCommands(
+      this.scanned,
+      roots,
+      void 0,
+      void 0,
+      void 0,
+      new Set(roots)
+    );
+    const system = await this.commands();
+    return new Set(
+      [...own.values()].filter((command) => !command.dependency || !system.has(command.name)).map((command) => command.name)
+    );
+  }
+  async globalRoots(scope) {
+    const homes = scope ? [scope, this.pnpmHome] : [this.pnpmHome];
+    const roots = await Promise.all(homes.map((home) => pnpmGlobalRoots(this.base, home)));
+    return new Set(roots.flat());
+  }
   interpreters(scope = "") {
     let found = this.binfmts.get(scope);
     if (!found) {