@ai-ecoverse/slicc-kernel 1.102.2 → 1.102.4

package/README.md

@@ -261,7 +261,7 @@ An empty path is `ENOENT` in every path call (`path_open`, `path_filestat_get`,
 On top of preview1 a WASI program gets:
 
 - **Threads**: `wasi.thread-spawn` (wasm32-wasip1-threads) and WASIX `thread_spawn_v2` start a worker per thread on the process's shared memory, sized from the module's own import (a 2 GiB maximum when the engine cannot reserve more). The threads share one descriptor table, and a file written, created, renamed or removed in one thread shows its new size and metadata to `stat` in every other thread at once, under any of its names (a symlink, a hard link, a new name after a rename in the same process), also when another thread renames a file over that name meanwhile, since the kernel reports which file an open or a change reached; this holds among the kernel's own processes, not against other tabs, `hostfs` writers or programs that aren't WASI; another process's changes show at close and after a `spawn` or `wait`, as with close-to-open caching, so a file another process renames while this one holds it open may show its old size under the new name until then; `exit` or a trap in any thread ends the process. At most 64 threads per process; `SLICC_WASM_THREADS` sets another cap, up to 256.
-- **WASIX** (a module importing `wasix_32v1`): fork and setjmp/longjmp through Asyncify (the module is built with `wasm-opt --asyncify`), with the unwound frames kept in the unused part of the program's stack, below its stack pointer (for a position-independent program, the stack the kernel's linker gives it), so a fork from deep in the stack leaves the program's data alone, and a forked child's thread-locals (`errno` included) where the parent left them: the kernel reruns the module's `__wasm_init_tls` at the parent's TLS base (and puts back the block it overwrites), which also sets the internal globals that hold thread-local addresses (a position-independent program's, a side module's, and those of a variable in the `global-dynamic` TLS model), exec, `posix_spawn`, `waitpid` (with `WNOHANG`, and `WUNTRACED` for a child that stops; WASIX has no `WCONTINUED`), pipes, `dup2`, signal handlers and interval timers, terminal modes, and dynamic linking: a position-independent main module loads side modules with `dlopen` from `LD_LIBRARY_PATH`, its runtime path, `/lib`, `/usr/lib` and `/usr/local/lib`. Each thread has its own copy of the function table, so the kernel hands out its slots (a side module's range, a `dlsym`'d function, a closure): threads loading or looking up at the same time never get the same slot. As in glibc, one `dlopen` at a time runs in a process, constructors included: a `dlopen` from a library's constructor goes ahead in its own thread, and another thread's `dlopen` waits, then finds the library loaded and gets the same handle. A thread waiting there still ends or stops with its process, but a signal handler it has runs once that `dlopen` returns. The lock is let go when a `dlopen` fails (an exception, a trap or `pthread_exit` out of a constructor included) and when its thread ends; a forked child starts with it free. A forked child also gets the side modules its parent had loaded, at the same addresses and table slots, with the slot and closure counters and the parent's thread-local blocks: they are linked again in the child before it resumes, without running their constructors again, since their data is already in the copied memory. A side-module function on the stack when `fork` is called (a callback into the main module that forks) is not instrumented by Asyncify, so it runs again when the stack is rewound, in the parent and in the child, and libffi frames there can trap; side modules would need building with Asyncify for that, which the toolchain does not do yet. In such a program, `dlopen(NULL)`, and `dlopen` of the program's own file (also by a path that resolves to it through symlinks), give a handle for the program that is already running (no second copy is loaded); a program that isn't position-independent gets an error from `dlopen(NULL)`, as from any `dlopen`; `dlsym` on it searches the program, then the loaded libraries, and `dlclose` of it does nothing. A function already in the table (the program's own, or a library's) keeps its slot, so `dlsym(handle, "strlen") == strlen`. Each WASIX thread has its own Asyncify driver too, so a thread can `longjmp` through Asyncify and `fork` (#710). The unwound frames go at the bottom of the thread's own stack (the `stack` and `stack_size` its libc passes to `wasi_thread_start`), never on the main thread's. A thread whose start args don't look like wasix-libc's (a `stack_size` under 4096 or over the memory, a stack top that isn't 16-aligned or not right above the args) gets no driver, so `fork` there is `ENOSYS` as before, rather than an unwind buffer at a bad address. A child forked from a thread runs as that thread: it's rewound through `wasi_thread_start`, can fork again, and ends when the thread's function returns. A program built with exceptions (every position-independent one) does `setjmp` with exceptions rather than Asyncify, so there only `fork` uses the driver. A kernel timer can be longer than `setTimeout`'s 2^31-1 ms (about 24.8 days): an `alarm`, an interval timer, or a `select`, `poll_oneoff` or `epoll_wait` timeout waits its whole time, re-armed in steps of at most 2^31-1 ms, instead of firing at once or spinning (#767).
+- **WASIX** (a module importing `wasix_32v1`): fork and setjmp/longjmp through Asyncify (the module is built with `wasm-opt --asyncify`), with the unwound frames kept in the unused part of the program's stack, below its stack pointer (for a position-independent program, the stack the kernel's linker gives it), so a fork from deep in the stack leaves the program's data alone, and a forked child's thread-locals (`errno` included) where the parent left them: the kernel reruns the module's `__wasm_init_tls` at the parent's TLS base (and puts back the block it overwrites), which also sets the internal globals that hold thread-local addresses (a position-independent program's, a side module's, and those of a variable in the `global-dynamic` TLS model), exec, `posix_spawn`, `waitpid` (with `WNOHANG`, and `WUNTRACED` for a child that stops; WASIX has no `WCONTINUED`), pipes, `dup2`, signal handlers and interval timers, terminal modes, and dynamic linking: a position-independent main module loads side modules with `dlopen` from `LD_LIBRARY_PATH`, its runtime path, `/lib`, `/usr/lib` and `/usr/local/lib`. Each thread has its own copy of the function table, so the kernel hands out its slots (a side module's range, a `dlsym`'d function, a closure): threads loading or looking up at the same time never get the same slot. As in glibc, one `dlopen` at a time runs in a process, constructors included: a `dlopen` from a library's constructor goes ahead in its own thread, and another thread's `dlopen` waits, then finds the library loaded and gets the same handle. A thread waiting there still ends or stops with its process, but a signal handler it has runs once that `dlopen` returns. The lock is let go when a `dlopen` fails (an exception, a trap or `pthread_exit` out of a constructor included) and when its thread ends; a forked child starts with it free. A forked child also gets the side modules its parent had loaded, at the same addresses and table slots, with the slot and closure counters and the parent's thread-local blocks: they are linked again in the child before it resumes, without running their constructors or their data relocations again, since their data is already in the copied memory. The same goes for the main module of a position-independent program: its data relocations ran once, in the parent, so a pointer the parent changed before the fork (`int *p = &a; p = &b;`) is still changed in the child (#777). A side-module function on the stack when `fork` is called (a callback into the main module that forks) is not instrumented by Asyncify, so it runs again when the stack is rewound, in the parent and in the child, and libffi frames there can trap; side modules would need building with Asyncify for that, which the toolchain does not do yet. In such a program, `dlopen(NULL)`, and `dlopen` of the program's own file (also by a path that resolves to it through symlinks), give a handle for the program that is already running (no second copy is loaded); a program that isn't position-independent gets an error from `dlopen(NULL)`, as from any `dlopen`; `dlsym` on it searches the program, then the loaded libraries, and `dlclose` of it does nothing. A function already in the table (the program's own, or a library's) keeps its slot, so `dlsym(handle, "strlen") == strlen`. Each WASIX thread has its own Asyncify driver too, so a thread can `longjmp` through Asyncify and `fork` (#710). The unwound frames go at the bottom of the thread's own stack (the `stack` and `stack_size` its libc passes to `wasi_thread_start`), never on the main thread's. A thread whose start args don't look like wasix-libc's (a `stack_size` under 4096 or over the memory, a stack top that isn't 16-aligned or not right above the args) gets no driver, so `fork` there is `ENOSYS` as before, rather than an unwind buffer at a bad address. A child forked from a thread runs as that thread: it's rewound through `wasi_thread_start`, can fork again, and ends when the thread's function returns. A program built with exceptions (every position-independent one) does `setjmp` with exceptions rather than Asyncify, so there only `fork` uses the driver. A kernel timer can be longer than `setTimeout`'s 2^31-1 ms (about 24.8 days): an `alarm`, an interval timer, or a `select`, `poll_oneoff` or `epoll_wait` timeout waits its whole time, re-armed in steps of at most 2^31-1 ms, instead of firing at once or spinning (#767).
 - **epoll and eventfd** (WASIX `epoll_create`, `epoll_ctl`, `epoll_wait` and `fd_event`, which tokio's reactor uses through the WASIX mio fork):
   - **Where they live:** an epoll instance and an eventfd are kernel descriptors, so they survive `dup`, `fork` and `exec` (unless close-on-exec) and are shared by a process's threads.
   - **What epoll watches:** sockets (listening, connecting, connected, loopback and uplink), pipes, terminals, `kernel.run`'s stdin, eventfds and other epoll instances (a cycle of instances is `ELOOP`).
@@ -907,21 +907,21 @@ Releases are cut by semantic-release on every push to `main`, in two jobs, and o
 `.github/workflows/cert-review.yml` certifies every release from CI. It runs when the Release workflow succeeds on `main`. The version is the tag on the released commit. The delta is read from the newest earlier release whose `cert-review` status is `success` or `pending`, skipping failed and unreviewed ones (or `base.json`'s release, or the tag before), so a release whose review was cancelled or failed is covered by the next one, and the line says `covers <first>–<last>`. It also runs on `workflow_dispatch` with a `version`, optionally with `prev` and `base`, or with `planted` (a branch) and `pr` for proofs. A dispatch off `main`, or a planted one, is a dry run. Each release (its commit) has its own concurrency group, so no pending review is cancelled by a newer one. The tools are in `tools/cert-review/`:
 
 - **plan:** works out the version, the previous tag, the commit and the base (`base.mjs`). The base is the newest earlier release whose `cert-review` commit status is `success`, or, until there is one, the release `base.json` names (the cert lane's last line). CI never reads the npm `certified` dist-tag.
-- **delta** (`delta.mjs`): lists the pull requests merged between the two tags (their merge commits, plus `pr` for a planted branch). Each gets its conventional-commit kind (`fix`, `feat`, `perf` and `revert` are behaviour changes) and the integration test files and titles it adds or changes. The titles are read from the pull request's patch: the added `test(…)` lines and the hunk headers git puts above changed lines. Escapes are decoded as JavaScript does, and a title git cut short, or a template literal, matches as a prefix.
-- **suite** (`suite.mjs`, `reporter.mjs`) runs on the bench pool, without secrets. It takes the tag's tree with its own dependencies and Chromium, and replaces `dist/` with the **published npm tarball** (for a planted branch, its own `npm pack`). The test command runs in a session of its own (`detached`), and `suite.mjs` notes a `SIGINT` (`ignored` in its summary) instead of dying of it. That way a stray signal to the step's process group can't cut the run short, while a `SIGTERM` (a real cancel) still ends it.
+- **delta** (`delta.mjs`): lists the pull requests merged between the two tags (their merge commits, plus `pr` for a planted branch). Each gets its conventional-commit kind (`fix`, `feat`, `perf` and `revert` are behaviour changes) and the integration test files and titles it adds or changes. The titles are read from the pull request's patch: the added `test(…)` lines and the hunk headers git puts above changed lines. Escapes are decoded as JavaScript does. A title git cut short matches as a prefix, and each `${…}` in a template-literal title matches anything there, so `` `a forked child of a ${kind} WASIX program…` `` matches the rows it ran as. A title made only of placeholders is skipped.
+- **suite** (`suite.mjs`, `reporter.mjs`) runs on the bench pool, without secrets. It takes the tag's tree with its own dependencies and Chromium, and replaces `dist/` with the **published npm tarball** (for a planted branch, its own `npm pack`). The test command runs in a session of its own (`detached`), and `suite.mjs` notes a `SIGINT` (`ignored` in its summary) instead of dying of it. That way a stray signal to the step's process group can't cut the run short, while a `SIGTERM` (a real cancel) still ends it. The log is written as the run goes, and a `SIGTERM` still writes the summary with the results so far (`exit: 'signal SIGTERM'`). A run that left only its `.jsonl` (the runner was lost) is read from that, as `exit: 'no summary'`. The NEGATIVE step runs even when the release suite's step failed. Before it reports, `suite.mjs` re-runs once just the test files that left no result (no rows, a cancelled row, or a file-level failure, such as a lost process), with a second `node --test` over those files, and merges their results (`retried`, `firstExit`, `unfinished` in the summary). That holds for the release suite and for each NEGATIVE run. A fix whose test passes on its baseline stays `no-negative`.
   - The release job runs the release's whole `test/integration/` suite, then each fix's test files again on the dist of **the newest published release before the fix was merged** (the NEGATIVE). `delta.mjs` groups the files by that baseline: for most fixes it is the previous release, but a fix for a bug that shipped inside the delta (a regression in v1.99.1 fixed for v1.99.2) is checked against the release that had the bug.
   - The base job runs the base release's own suite on its own published dist.
   - Each run uses the tree's own `npm test` command, with a reporter that records every test and subtest with its status and error.
 - **kernel-compat** (`compat.mjs`) supplies the base-set and catalog evidence: homescoop's kernel-compat run for the release.
-  - Release runs share one group and intermediate ones are cancelled. When there is no finished run for the release commit, it dispatches `kernel-compat.yml` for that version (whose run is named `Kernel compat v<version>`) and waits up to 150 minutes.
-  - It reads the run's report artifact, and **re-runs every regressing package once** (a dispatch with that `set`).
+  - Its reference kernel is the certified base, not slicc-bios's pin. cert-review passes `--reference <base>`, and `kernel-compat.yml` takes a `reference` input that it hands to homescoop's workflow (homescoop#504), recorded in compat.json. So it uses a dispatched run named `Kernel compat v<version> ref <base>`, never the release's own `workflow_run` run, which compares with bios's pin. It dispatches one when none has finished and waits up to 150 minutes; package reruns carry the same reference.
+  - It reads the run's report artifact, and **re-runs every regressing package once** (a dispatch with that `set`). The `set` names each one as `recipe@version`, the exact version that regressed, and compat.json records what the rerun `ran`. A rerun result for another version is a `rerun-mismatch`, counted as reproduced and never as a flake.
   - A regression that doesn't reproduce is a `compat-flake`; one that does is a `compat-regression`. A crash that reproduces and doesn't happen on the reference kernel is a `compat-crash`, which record FAILs in code.
-  - A run that failed with no regression in its report is a `compat-incomplete`, missing evidence and not a regression. It names the shards that left no report (for example, a shard that lost its bench runner), and those shards' failed jobs are re-run once first.
+  - A run that failed with no regression in its report is a `compat-incomplete`, missing evidence and not a regression. It names the shards that left no report (for example, a shard that lost its bench runner), and those shards' failed jobs are re-run once first. The same holds for the package rerun's own run, so a rerun that loses its runner is tried again before its missing result counts as reproduced.
 - **review:** `findings.mjs` decides in code, per pull request:
   - `negative` (a fix's test fails on the previous dist and passes on the release) or `passes` (a feature's test passes);
   - otherwise a finding: `no-test`, `no-negative`, `fails-on-release`, `not-run`, `prev-not-run` or `no-titles`.
 
-  It also diffs the two suites test by test (`regressed`, `newFailing`, `fixed`, `stillFailing`, `removed`), flags a suite run that didn't finish (`suite-incomplete`), and adds the kernel-compat findings. Claude, on Bedrock with the key only in that step, then reads those artifacts and `instructions.txt` (which has the roll-forward FAIL list verbatim) and writes `verdict.json` and `comment.md`. Its tools are reading files, `gh pr view`, and `gh issue list`/`view`. `gh issue create` is allowed only for a release on `main`; otherwise, and for an issue in another repository, it lists the issue under `elsewhere` for the cert lane to file.
+  It also diffs the two suites test by test (`regressed`, `newFailing`, `fixed`, `stillFailing`, `removed`), flags a suite run that didn't finish (`suite-incomplete`), and adds the kernel-compat findings. A release suite that didn't finish is a `missing-evidence` finding that always blocks `certified`, with the remedy `re-run the failed jobs`. That covers a signal, a timeout, a cut-off run, no results, or any test file without a result. Its base tests that didn't run are `unrun`, never `removed`. INCOMPLETE is not FAIL: record writes `cert-ci: INCOMPLETE <name>@<version> (<n> files without results: …; re-run the failed jobs)`, and publish posts the `cert-review` status as `error`, which means re-run, where `failure` means a real defect that rolls forward. A defect found in code (supply chain, a compat crash) still FAILs. base and prev skip `error` releases, as they skip `failure`. A base suite that is missing or didn't finish is a `missing-evidence` finding. It blocks `certified` in code (record FAILs it) unless the release suite is complete and all-pass and the delta removes or renames no test file and drops no test title (`dropped` per row). Either way the line ends with `base suite incomplete (<n> results)`. Claude, on Bedrock with the key only in that step, then reads those artifacts and `instructions.txt` (which has the roll-forward FAIL list verbatim) and writes `verdict.json` and `comment.md`. Its tools are reading files, `gh pr view`, and `gh issue list`/`view`. `gh issue create` is allowed only for a release on `main`; otherwise, and for an issue in another repository, it lists the issue under `elsewhere` for the cert lane to file.
 - **record** (`record.mjs`), in code, with read-only permissions:
   - It re-downloads the tarball and checks `dist.integrity` (sha512) and the sha256, the SLSA v1 provenance (the tarball is its subject; `release.yml`, `refs/heads/main`, a GitHub-hosted builder; commit = `gitHead` = the tag), `npm audit signatures`, and that the commit is on `main`. A problem there, a `compat-crash`, or a planted build is a FAIL whatever the review says.
   - It adds the **known issues as of the release**: every issue labelled `known-issue` (open or closed) and every one in `known.json`, unless it was fixed before the release. That holds when `known.json`'s `fixedIn` names an earlier or equal version, or the merged pull request that closed it is an ancestor of the tag, or it was closed (with no closing pull request) before the release commit. An issue closed as not planned or as a duplicate is dropped.
@@ -1040,6 +1040,12 @@ A WASIX C program for `test/integration/long-timers.test.mjs` (#767).
 
 **Build:** `WASIX_SYSROOT_28=/path/to/wasix-sysroot ./build.sh`, with `wasixcc` 0.4.7 and binaryen's `wasm-opt` on `PATH`. `bin/longwait.wasm` is 44,010 bytes, sha256 `70ae2adfb46864fad3af0241fe26f401a62a5176ccd25ab82fe6588178cfb5c8`.
 
+### `forkrelocs`
+
+A position-independent WASIX C program (`forkrelocs.c`) and a side module (`side.c`, `lib/libforkrelocs.so`), for `test/integration/fork-relocs.test.mjs` (#777). Both have a statically initialized pointer (`int *p = &a`) that the parent changes before it forks (with `dlopen` of the side module when given its path). The child reports whether each pointer kept the parent's value or was reset to its initial one.
+
+**Build:** `WASIX_SYSROOT_28=/path/to/wasix-sysroot ./build.sh`, with homescoop's pinned `wasixcc` 0.4.7 and binaryen's `wasm-opt` on `PATH`. `bin/forkrelocs.wasm` sha256 `7538bc2580d8b4a30912d28a4744d1848544da74d7b47d549a7c80d623a2ed8e`; `lib/libforkrelocs.so` is 753 bytes, sha256 `3dd52124b804670fb22eac46124678d35d8f478da94faf37792406d21e8e9165`.
+
 ### `semtest`
 
 A WASIX C program for `test/integration/semaphore.test.mjs` (#717). It calls the `slicc.sem_*` imports directly (`import_module("slicc")`), so the rows don't wait for the libc side. `semtest MODE` runs one of:

package/dist/kernel-worker.js

@@ -8273,7 +8273,8 @@ var GEN = 0;
 var RING = 1;
 var RING_SIZE = 4096;
 var SLOT = 2;
-var CHANGES_BYTES = (RING + RING_SIZE * SLOT) * 4;
+var DIRTY = RING + RING_SIZE * SLOT;
+var CHANGES_BYTES = (DIRTY + 1) * 4;
 function changeHash(path) {
   let h = 2166136261;
   for (let i = 0; i < path.length; i++) {
@@ -8298,6 +8299,9 @@ var FileChanges = class {
     this.sab = sab;
     this.view = new Int32Array(sab);
   }
+  dirtied(delta) {
+    Atomics.add(this.view, DIRTY, delta);
+  }
   publish(...paths) {
     for (const path of paths) {
       const g = Atomics.add(this.view, GEN, 1) + 1 | 0;
@@ -12136,7 +12140,7 @@ var PATH_OPS = new Set(SYNC_FS_OPS);
 var EARLY_WHOLE_FILE = 64 * 1024;
 
 // src/version.ts
-var KERNEL_VERSION = true ? "1.102.2" : "0.0.0-dev";
+var KERNEL_VERSION = true ? "1.102.4" : "0.0.0-dev";
 var OS_RELEASE = "6.0.0-slicc";
 function procVersion() {
   return `Linux version ${OS_RELEASE} (slicc-kernel ${KERNEL_VERSION}) #1 SMP PREEMPT_DYNAMIC
@@ -15017,6 +15021,15 @@ var VfsNode = class {
   orphaned;
   id;
   onDirty;
+  onDirtyState;
+  counted = false;
+  recount() {
+    if (!this.onDirtyState) return;
+    const on = this.dirty && !this.orphaned;
+    if (on === this.counted) return;
+    this.counted = on;
+    this.onDirtyState?.(on);
+  }
   constructor(fs, path, contents2, orphaned = false, dirty = false, pin) {
     this.fs = fs;
     this.pin = pin;
@@ -15166,6 +15179,7 @@ var VfsNode = class {
   }
   markDirty() {
     this.dirty = true;
+    this.recount();
     if (!this.orphaned) this.onDirty?.();
     if (this.writeBack !== void 0 || this.orphaned) return;
     const delay = Math.max(WRITEBACK_MS, this.writeBackCost * WRITEBACK_COST_FACTOR);
@@ -15193,6 +15207,8 @@ var VfsNode = class {
     } catch (err) {
       this.dirty = true;
       throw err;
+    } finally {
+      this.recount();
     }
     this.writeBackCost = performance.now() - started;
   }
@@ -15203,6 +15219,7 @@ var VfsNodes = class {
   byId = /* @__PURE__ */ new Map();
   keepsOnDisk = false;
   changed;
+  dirtied;
   fs;
   entries;
   onClosed;
@@ -15286,6 +15303,7 @@ var VfsNodes = class {
   }
   orphan(node) {
     node.orphaned = true;
+    node.recount();
     this.forget(node);
   }
   dropName(path, node) {
@@ -15340,6 +15358,8 @@ var VfsNodes = class {
   create(path, id2, node) {
     this.reshaped();
     node.onDirty = () => this.changed?.(node.path);
+    node.onDirtyState = (on) => this.dirtied?.(on ? 1 : -1);
+    node.recount();
     this.byPath.set(path, node);
     if (id2 !== void 0) this.know(node, id2);
     return node;
@@ -22556,6 +22576,7 @@ var Launcher = class {
   }
   processChain(fs, nodes, unlinked) {
     nodes.changed = (path) => this.changes.publish(path);
+    nodes.dirtied = (delta) => this.changes.dirtied(delta);
     return unlinked.wrap(keepingOpen(fs, nodes, (path) => this.changes.publish(path)));
   }
   async starting(argv, dir, account, tag) {