@ai-ecoverse/slicc-kernel 1.146.7 → 1.147.1

package/README.md

@@ -239,6 +239,8 @@ A command's name is any single path component without whitespace or control char
 
 **Variants by environment** ([homescoop#357](https://github.com/ai-ecoverse/homescoop/issues/357)): one package can carry more than one build of a wasm command and choose between them by the environment. `"variants": [{ "env": { "PYTHON_FORK": "true" }, "wasm": "bin/python-fork.wasm" }]` on a command entry (Emscripten or WASI) is matched in order against the environment the new process gets, at spawn and at exec. Every key must equal its value exactly, and the first match runs its `wasm`; no match runs the command's own `wasm`. This holds however the kernel reaches the entry: by name, by its path (`/usr/bin/python3`, as `sys.executable` gives it to `subprocess` and `multiprocessing`), or through a symlink to that path (a venv's `bin/python`). So `PYTHON_FORK=true python3`, `PYTHON_FORK=true exec /usr/bin/python3` and a child that inherits `PYTHON_FORK=true` all get the variant, and `PYTHON_FORK=1` doesn't. Commands of one package that share a build (`python`, `python3`, `pip`, `pip3`) each carry the same `variants`. A variant replaces only the module: the `abi`, the glue, `env`, `argv0`, `argv0Path` and the `/usr/bin` entry stay the command's. For an Emscripten command that means a variant swaps only the `.wasm` and keeps the command's JS glue, so it must be built against that same glue; a WASI or WASIX command has no glue, so there's no such constraint. Only the chosen module is read, checked and compiled, and compiled modules are cached by its path. A read or `stat` of `/usr/bin/<name>` still shows the default `wasm`. A `variants` that isn't an array, or has an entry without an `env` object of strings and a `wasm` inside the package, is ignored as a whole, with one warning per command, and the command runs its `wasm`. An older kernel ignores the field.
 
+**A pnpm bin of a kernel command** ([#1453](https://github.com/ai-ecoverse/slicc-kernel/issues/1453)): a `#!/usr/bin/env node` (or `#!/usr/bin/node`) script run by its path, whose package lists the same name in `slicc.commands`, runs that kernel command, even when a `node` command exists. This is the bin shim `pnpm add -g` writes into `$PNPM_HOME/bin`, which names its target in a `// pnpm-wasm-shim={"target":…}` line; a script without that line is run as a script, so a same-named file in a package that differs from its glue still doesn't run as the command (#899). The shim's target, with its links resolved, must lie in a package installed directly under one of the command roots, and the name must be the catalog's command from that same package, so a shim only ever runs a command the caller could run by name, with the same checks; with `integrity` on, the shim itself is verified as a source. So `pnpm add -g <pkg> && <cmd>` works in one line.
+
 A package can also name a command that runs files by their suffix, as `binfmt_misc` does: `"slicc": { "binfmt": { ".jsh": "jsh" } }`. Executing a file that has no `#!` line and is not a wasm module, whose name ends in a mapped suffix, runs that command with the file's path as invoked and the arguments after it, as a `#!` line naming the command would; a `#!` line still wins. The longest matching suffix wins, the whole name is never a suffix (`.jsh` alone is not one), and of two packages that map one suffix, the one whose name sorts first wins, as for commands; the kernel warns once (`console.warn`) when another package maps that suffix to a different command. A suffix starts with a dot and the command must be a name the catalog resolves when the file runs.
 
 ### WASI programs
@@ -273,7 +275,7 @@ An empty path is `ENOENT` in every path call (`path_open`, `path_filestat_get`,
 On top of preview1 a WASI program gets:
 
 - **Threads**: `wasi.thread-spawn` (wasm32-wasip1-threads) and WASIX `thread_spawn_v2` start a worker per thread on the process's shared memory, sized from the module's own import (a 2 GiB maximum when the engine cannot reserve more). The threads share one descriptor table, and a file written, created, renamed or removed in one thread shows its new size and metadata to `stat` in every other thread at once, under any of its names (a symlink, a hard link, a new name after a rename in the same process), also when another thread renames a file over that name meanwhile, since the kernel reports which file an open or a change reached; this holds among the kernel's own processes, not against other tabs, `hostfs` writers or programs that aren't WASI; another process's changes show at close and after a `spawn` or `wait`, as with close-to-open caching, so a file another process renames while this one holds it open may show its old size under the new name until then; `exit` or a trap in any thread ends the process. At most 64 threads per process; `SLICC_WASM_THREADS` sets another cap, up to 256.
-- **WASIX** (a module importing `wasix_32v1`): fork and setjmp/longjmp through Asyncify (the module is built with `wasm-opt --asyncify`), with the unwound frames kept in the unused part of the program's stack, below its stack pointer (for a position-independent program, the stack the kernel's linker gives it), so a fork from deep in the stack leaves the program's data alone, and a forked child's thread-locals (`errno` included) where the parent left them: the kernel reruns the module's `__wasm_init_tls` at the parent's TLS base (and puts back the block it overwrites), which also sets the internal globals that hold thread-local addresses (a position-independent program's, a side module's, and those of a variable in the `global-dynamic` TLS model), exec, `posix_spawn`, `waitpid` (with `WNOHANG`, and `WUNTRACED` for a child that stops; WASIX has no `WCONTINUED`), pipes, `dup2`, signal handlers and interval timers, terminal modes, and dynamic linking: a position-independent main module loads side modules with `dlopen` from `LD_LIBRARY_PATH`, its runtime path, `/lib`, `/usr/lib` and `/usr/local/lib`. Each thread has its own copy of the function table, so the kernel hands out its slots (a side module's range, a `dlsym`'d function, a closure): threads loading or looking up at the same time never get the same slot. As in glibc, one `dlopen` at a time runs in a process, constructors included: a `dlopen` from a library's constructor goes ahead in its own thread, and another thread's `dlopen` waits, then finds the library loaded and gets the same handle. A thread waiting there still ends or stops with its process, but a signal handler it has runs once that `dlopen` returns. The lock is let go when a `dlopen` fails (an exception, a trap or `pthread_exit` out of a constructor included) and when its thread ends; a forked child starts with it free. A forked child also gets the side modules its parent had loaded, at the same addresses and table slots, with the slot and closure counters and the parent's thread-local blocks: they are linked again in the child before it resumes, without running their constructors or their data relocations again, since their data is already in the copied memory. The same goes for the main module of a position-independent program: its data relocations ran once, in the parent, so a pointer the parent changed before the fork (`int *p = &a; p = &b;`) is still changed in the child (#777). A side-module function on the stack when `fork` is called (a callback into the main module that forks) is not instrumented by Asyncify, so it runs again when the stack is rewound, in the parent and in the child, and libffi frames there can trap; side modules would need building with Asyncify for that, which the toolchain does not do yet. In such a program, `dlopen(NULL)`, and `dlopen` of the program's own file (also by a path that resolves to it through symlinks), give a handle for the program that is already running (no second copy is loaded); a program that isn't position-independent gets an error from `dlopen(NULL)`, as from any `dlopen`; `dlsym` on it searches the program, then the loaded libraries, and `dlclose` of it does nothing. A function already in the table (the program's own, or a library's) keeps its slot, so `dlsym(handle, "strlen") == strlen`. Each WASIX thread has its own Asyncify driver too, so a thread can `longjmp` through Asyncify and `fork` (#710). The unwound frames go at the bottom of the thread's own stack (the `stack` and `stack_size` its libc passes to `wasi_thread_start`), never on the main thread's. A thread whose start args don't look like wasix-libc's (a `stack_size` under 4096 or over the memory, a stack top that isn't 16-aligned or not right above the args) gets no driver, so `fork` there is `ENOSYS` as before, rather than an unwind buffer at a bad address. A child forked from a thread runs as that thread: it's rewound through `wasi_thread_start`, can fork again, and ends when the thread's function returns. A program built with exceptions (every position-independent one) does `setjmp` with exceptions rather than Asyncify, so there only `fork` uses the driver. A kernel timer can be longer than `setTimeout`'s 2^31-1 ms (about 24.8 days): an `alarm`, an interval timer, or a `select`, `poll_oneoff` or `epoll_wait` timeout waits its whole time, re-armed in steps of at most 2^31-1 ms, instead of firing at once or spinning (#767). A `poll_oneoff` read event's `nbytes` is the number of bytes a read would get at once, as `FIONREAD` counts them, for a pipe, a FIFO, a socket, a terminal and a pty master; on a hangup it's 0 when nothing is left, so a libc can report `POLLHUP` without `POLLIN`, as Linux does (#1053). Other descriptors, regular files included, answer 1. A WASI or WASIX program that a signal ends (`SIGKILL`, or a signal whose default terminates it) stops at its next kernel call. The kernel first lets the calls it is already serving finish and the program write its files back: it ends the process once the program has made no file call for 2 s, and 5 s after the signal at the latest. A call the program makes after the kill is answered `EINTR` at once and doesn't extend the wait, so a program that swallows the kill and keeps retrying (python writing a fatal error to stderr) still ends (#755).
+- **WASIX** (a module importing `wasix_32v1`): fork and setjmp/longjmp through Asyncify (the module is built with `wasm-opt --asyncify`), with the unwound frames kept in the unused part of the program's stack, below its stack pointer (for a position-independent program, the stack the kernel's linker gives it), so a fork from deep in the stack leaves the program's data alone, and a forked child's thread-locals (`errno` included) where the parent left them: the kernel reruns the module's `__wasm_init_tls` at the parent's TLS base (and puts back the block it overwrites), which also sets the internal globals that hold thread-local addresses (a position-independent program's, a side module's, and those of a variable in the `global-dynamic` TLS model), exec, `posix_spawn`, `waitpid` (with `WNOHANG`, and `WUNTRACED` for a child that stops; WASIX has no `WCONTINUED`), pipes, `dup2`, signal handlers and interval timers, terminal modes, and dynamic linking: a position-independent main module loads side modules with `dlopen` from `LD_LIBRARY_PATH`, its runtime path, `/lib`, `/usr/lib` and `/usr/local/lib`. A side module's data (its `dylink.0` memory size) is allocated with the program's own `calloc`, when the program exports one, as Emscripten's loader does, so a `dlopen` doesn't grow memory while the heap has room ([#1408](https://github.com/ai-ecoverse/slicc-kernel/issues/1408)); without `calloc`, or mid-unwind, memory grows as before. Each thread has its own copy of the function table, so the kernel hands out its slots (a side module's range, a `dlsym`'d function, a closure): threads loading or looking up at the same time never get the same slot. As in glibc, one `dlopen` at a time runs in a process, constructors included: a `dlopen` from a library's constructor goes ahead in its own thread, and another thread's `dlopen` waits, then finds the library loaded and gets the same handle. A thread waiting there still ends or stops with its process, but a signal handler it has runs once that `dlopen` returns. The lock is let go when a `dlopen` fails (an exception, a trap or `pthread_exit` out of a constructor included) and when its thread ends; a forked child starts with it free. A forked child also gets the side modules its parent had loaded, at the same addresses and table slots, with the slot and closure counters and the parent's thread-local blocks: they are linked again in the child before it resumes, without running their constructors or their data relocations again, since their data is already in the copied memory. The same goes for the main module of a position-independent program: its data relocations ran once, in the parent, so a pointer the parent changed before the fork (`int *p = &a; p = &b;`) is still changed in the child (#777). A side-module function on the stack when `fork` is called (a callback into the main module that forks) is not instrumented by Asyncify, so it runs again when the stack is rewound, in the parent and in the child, and libffi frames there can trap; side modules would need building with Asyncify for that, which the toolchain does not do yet. In such a program, `dlopen(NULL)`, and `dlopen` of the program's own file (also by a path that resolves to it through symlinks), give a handle for the program that is already running (no second copy is loaded); a program that isn't position-independent gets an error from `dlopen(NULL)`, as from any `dlopen`; `dlsym` on it searches the program, then the loaded libraries, and `dlclose` of it does nothing. A function already in the table (the program's own, or a library's) keeps its slot, so `dlsym(handle, "strlen") == strlen`. Each WASIX thread has its own Asyncify driver too, so a thread can `longjmp` through Asyncify and `fork` (#710). The unwound frames go at the bottom of the thread's own stack (the `stack` and `stack_size` its libc passes to `wasi_thread_start`), never on the main thread's. A thread whose start args don't look like wasix-libc's (a `stack_size` under 4096 or over the memory, a stack top that isn't 16-aligned or not right above the args) gets no driver, so `fork` there is `ENOSYS` as before, rather than an unwind buffer at a bad address. A child forked from a thread runs as that thread: it's rewound through `wasi_thread_start`, can fork again, and ends when the thread's function returns. A program built with exceptions (every position-independent one) does `setjmp` with exceptions rather than Asyncify, so there only `fork` uses the driver. A kernel timer can be longer than `setTimeout`'s 2^31-1 ms (about 24.8 days): an `alarm`, an interval timer, or a `select`, `poll_oneoff` or `epoll_wait` timeout waits its whole time, re-armed in steps of at most 2^31-1 ms, instead of firing at once or spinning (#767). A `poll_oneoff` read event's `nbytes` is the number of bytes a read would get at once, as `FIONREAD` counts them, for a pipe, a FIFO, a socket, a terminal and a pty master; on a hangup it's 0 when nothing is left, so a libc can report `POLLHUP` without `POLLIN`, as Linux does (#1053). Other descriptors, regular files included, answer 1. A WASI or WASIX program that a signal ends (`SIGKILL`, or a signal whose default terminates it) stops at its next kernel call. The kernel first lets the calls it is already serving finish and the program write its files back: it ends the process once the program has made no file call for 2 s, and 5 s after the signal at the latest. A call the program makes after the kill is answered `EINTR` at once and doesn't extend the wait, so a program that swallows the kill and keeps retrying (python writing a fatal error to stderr) still ends (#755).
 - **epoll and eventfd** (WASIX `epoll_create`, `epoll_ctl`, `epoll_wait` and `fd_event`, which tokio's reactor uses through the WASIX mio fork):
   - **Where they live:** an epoll instance and an eventfd are kernel descriptors, so they survive `dup`, `fork` and `exec` (unless close-on-exec) and are shared by a process's threads.
   - **What epoll watches:** sockets (listening, connecting, connected, loopback and uplink), pipes, terminals, `kernel.run`'s stdin, eventfds and other epoll instances (a cycle of instances is `ELOOP`).

package/dist/kernel-worker.js

@@ -7785,7 +7785,7 @@ function satisfies(version, range2) {
 }
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.146.7" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.147.1" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -28859,6 +28859,8 @@ function linksOf2(req) {
   return req.links === void 0 ? {} : { links: req.links };
 }
 var COMMAND2 = /^\/(?:usr\/)?bin\/([^/]+)$/;
+var SHIM_MARKER = /^\/\/ pnpm-wasm-shim=(\{.*\})$/gm;
+var SHIM_MAX = 64 * 1024;
 var PACKAGE_ROOT = /^(.*\/node_modules\/(?:@[^/]+\/)?[^/]+)\//;
 var ENV_REFERENCE = /\$\{([A-Za-z_][A-Za-z0-9_]*)\}/g;
 var WASM_MAGIC = [0, 97, 115, 109];
@@ -29550,7 +29552,10 @@ var Launcher = class {
       return { abi: "emscripten", glue, wasm, argv0: baseName2(argv0 || file), ...asked };
     }
     const owner = await this.glueOwner(glue, baseName2(file), scope);
-    if (!owner) return void 0;
+    if (!owner) {
+      const shim = await this.shimOwner(glue, baseName2(file), scope);
+      return shim && { ...targetOf(shim), ...asked, shim: glue };
+    }
     return { ...targetOf(owner), ...asked, ...owner.glue === glue ? {} : { copy: glue } };
   }
   async ownOrRefused(path, name2, scope) {
@@ -29586,6 +29591,43 @@ var Launcher = class {
     }
     return void 0;
   }
+  async shimOwner(script, name2, scope) {
+    const command = (await this.commands(scope)).get(name2);
+    if (!command || command.script || command.origin?.script) return void 0;
+    const target = await this.shimTarget(script);
+    if (target === void 0) return void 0;
+    const pkg = await this.installedPackage(await physicalPathAsync(this.base, target), scope);
+    if (!pkg) return void 0;
+    const glue = command.origin?.glue ?? command.glue;
+    const own = await physicalPathAsync(this.base, glue).catch(() => void 0);
+    return own?.startsWith(`${pkg}/`) ? command : void 0;
+  }
+  async shimTarget(script) {
+    const st = await this.base.stat(script).catch(() => void 0);
+    if (!st?.isFile || st.size > SHIM_MAX) return void 0;
+    const text = await this.base.readFile(script).catch(() => "");
+    const marker = [...text.matchAll(SHIM_MARKER)].at(-1)?.[1];
+    if (marker === void 0) return void 0;
+    let target;
+    try {
+      target = JSON.parse(marker).target;
+    } catch {
+      return void 0;
+    }
+    if (typeof target !== "string" || target === "") return void 0;
+    const dir = await physicalPathAsync(this.base, script.slice(0, script.lastIndexOf("/")) || "/");
+    return this.fs.resolvePath(dir, target);
+  }
+  async installedPackage(real, scope) {
+    for (const root of await this.roots(scope)) {
+      const top = await physicalPathAsync(this.base, root).catch(() => void 0);
+      if (!top || !real.startsWith(`${top}/`)) continue;
+      const [first = "", second] = real.slice(top.length + 1).split("/");
+      const pkg = first.startsWith("@") && second ? `${top}/${first}/${second}` : `${top}/${first}`;
+      if (pkg !== real && await this.base.exists(`${pkg}/package.json`)) return pkg;
+    }
+    return void 0;
+  }
   async glueOwner(glue, name2, scope) {
     const commands = [...(await this.commands(scope)).values()];
     const owners = commands.filter((c) => c.abi === "emscripten" && !c.script && c.glue === glue);
@@ -29681,7 +29723,10 @@ var Launcher = class {
     const found = await this.resolve(file, argv[0] ?? file, cwd, scope);
     const named = typeof argv0 === "string" && found && !found.fixed;
     const direct = named ? { ...found, argv0 } : found;
-    if (direct) return { target: direct, args: [...direct.prefix ?? [], ...argv.slice(1)] };
+    if (direct) {
+      const shim = direct.shim ? { sources: [direct.shim], asked: direct.asked ? { [direct.shim]: direct.asked } : {} } : {};
+      return { target: direct, args: [...direct.prefix ?? [], ...argv.slice(1)], ...shim };
+    }
     if (catalogOnly && !await this.scriptCommand(file, cwd, scope)) return void 0;
     const script = await this.interpreted(file, argv, cwd, scope);
     return script && { ...script, script: true };