@ai-ecoverse/slicc-kernel 1.127.1 → 1.128.1

package/README.md

@@ -29,7 +29,7 @@ TypeScript declarations ship in `dist/types/`, and each entry (`.`, `./node`, `.
 | `modules` | `'/node_modules'` | where installed packages are scanned for commands |
 | `commands` | none | `{ append: ['/opt/user/node_modules'] }`: more `node_modules` directories scanned for commands after all the others, so they never shadow one (see [Commands](#commands)) |
 | `env` | `{}` | environment added to every run |
-| `metadata` | `'slicc-kernel'` | the IndexedDB database for POSIX metadata (see [Metadata](#metadata)); `false` keeps it in memory for the kernel's lifetime |
+| `metadata` | `'slicc-kernel'` for the origin's OPFS root, `'slicc-kernel:<root id>'` for any other `root` | the IndexedDB database for POSIX metadata (see [Metadata](#metadata)); `false` keeps it in memory for the kernel's lifetime. Kernels share metadata, and count each other as peers on one root, exactly when they use the same database. So the default gives every root its own: the origin's own root (`navigator.storage.getDirectory()`) keeps `slicc-kernel`, and any other directory passed as `root` gets an id, which the kernel keeps with the directory's handle in the IndexedDB database `slicc-kernel-roots` ([#1120](https://github.com/ai-ecoverse/slicc-kernel/issues/1120)). Until #1120 every root used `slicc-kernel`, so records of two roots with a file at the same path were mixed there, and the kernel can't split them again. So that an upgrade loses nothing, a root that isn't empty when it's first registered starts with a copy of the `slicc-kernel` records, if that database exists, and from then on its records are its own: they never mix with another root's. The copy is of the whole `slicc-kernel` store, once per root, so on a long-used origin the first boot of such a root takes longer; it logs one console line naming the new database and how many records it copied (never paths or contents), and later boots log nothing. An empty root starts with no records. A moved root, or one whose registry entry was cleared, is registered again: its old records don't follow it. Registering is done under the Web Lock `slicc-kernel-roots`, so tabs booting one new root at once agree on its id. A name given here is used as is |
 | `ca` | `'<metadata>-ca'`, or `'slicc-kernel-ca'` | the IndexedDB database for the proxy's CA (see [Network](#network)); `false` keeps it in memory |
 | `media` | `'<metadata>:media'` | the IndexedDB database for the folder handles of `fsa` drives (see [Removable media](#removable-media-fsa)); `false` keeps them in memory, as `metadata: false` does |
 | `worker` | `new URL('./kernel-worker.js', import.meta.url)` | the kernel worker script |
@@ -335,6 +335,7 @@ A WASI command can name an ES module of its package that provides imports of its
 
 - `memory()`, `instance()` (after instantiation), `tid` (1 for the main thread), `argv`, `env`, `cwd()`;
 - `fs`: the synchronous filesystem the WASI calls use (absolute paths);
+  - `fs.copyFile(from, to, { mode, excl })` copies a regular file inside the kernel, as `slicc_fs.path_copy_file` does (below): `mode` (default `0666`) less the umask, `excl` for `O_EXCL`. It throws the kernel's fs errors (`code` `EEXIST`, `ENOENT`, `EXDEV`, …). An older kernel's `ctx.fs` has no `copyFile`, so a module checks `typeof ctx.fs.copyFile === 'function'`;
 - `fds`: `open(path, { read, write, append, create, exclusive, truncate, directory, nofollow, mode })` returns a WASI descriptor of the program; `mode` is a new file's mode. With `create` and `exclusive` under users `report` or `enforce`, the file is created with it, less the umask, as `open(2)` does, and the name is only checked by the create itself (a symlink there is `EEXIST`). Otherwise the mode is set exactly once the file is open; `close`, `fstat` (`{ kind, path, mode, uid, gid, size, ino, mtimeMs }`; an open file's size is the descriptor's own, so writes not yet flushed count, while the other fields come from a stat of its path, as WASI's `fd_filestat_get` gives them), `fchmod`, and `tryLock(fd, exclusive)` / `unlock(fd)`: advisory locks held by the kernel across processes and threads, keyed by path, released on unlock, on closing the descriptor and at exit (`tryLock` returns 0, or 6 when another process holds the lock);
 - `syscall(req)`: a kernel syscall, synchronously;
 - `net.connect({ host, port, nonblock })` connects a TCP socket to the kernel's loopback and returns its kernel descriptor. `host` is `localhost` or `127.x.y.z`; anything else, or a port outside 1–65535, is `EINVAL`. A refused connection throws `ECONNREFUSED`. The connection carries the program's own identity, as any connect from its process does, so a [page facade](#page-facades) such as `SLICC_LINK_URL` sees the program's stamp. With `nonblock` it returns at once, and `net.ready(fd)` answers 0 once a connect is complete, `EINPROGRESS` (26) while it isn't, or the connect's WASI errno (`SO_ERROR`). On the loopback a connect completes at once, so `ready` is 0 straight away. `ready` is meant for a descriptor from `net.connect`. The descriptor is read and written with `fd-read` and `fd-write` and closed with `fd-close`, as below, and `fd-select` (`{ op: 'fd-select', read: [fds], write: [fds], timeoutMs }`, answering `{ read, write }` with the ready ones; `timeoutMs: -1` waits without a limit) waits until it's readable or writable;
@@ -832,6 +833,7 @@ WASI and WASIX have no call for file modes, so the kernel adds the import module
 | `path_filestat_owner(dirfd, lookupflags, path, path_len, filestat_out, out)` | `path_filestat_get` (with its preview1 lookup flags: `1` follows a symlink) and the owner of the same path, in one call, so a libc's `stat` costs one call either way |
 | `path_mkfifo(dirfd, path, path_len, mode)` | `mkfifo`/`mkfifoat` and `mknod(S_IFIFO)` (homescoop#878): makes a [FIFO](#filesystem) with `mode & ~umask`, relative to `dirfd` (`AT_FDCWD` is `-2`). It doesn't follow a link in the last component, so an existing name, also a FIFO, a dangling link, a directory or the root, is `EEXIST`, and an empty name is `ENOENT` (#972). A mount without FIFOs answers `EPERM` |
 | `path_access(dirfd, path, path_len, amode, flags)` | `faccessat`: `amode` is `R_OK` 4, `W_OK` 2, `X_OK` 1 or `F_OK` 0; the check uses the real uid and gid, or the effective ones with `flags & 2` (`AT_EACCESS`), and the link itself with `flags & 1`. With `users: 'off'` it checks only that the path exists, as a libc without it does (see [Permissions](#permissions)) |
+| `path_copy_file(src_dirfd, src, src_len, dst_dirfd, dst, dst_len, mode, flags)` | copies a regular file inside the kernel, with no bytes through the program ([#1017](https://github.com/ai-ecoverse/slicc-kernel/issues/1017)). Paths resolve as in `path_open`; an absolute one ignores its dirfd, and `AT_FDCWD` is `-2`. The source is followed through symlinks; the destination's last component never is. The new file gets `mode & ~umask` and the caller as owner, with the permissions of a create: read on the source, write and search on the destination's directory. `flags & 1` (`COPY_EXCL`) makes it `O_EXCL`, so of two copies racing to one name exactly one succeeds and the other gets `EEXIST`. Without it, an existing file is replaced in place, as with `O_TRUNC`. `flags & 2` (`COPY_MODE`) takes the source's permission bits instead of `mode`; other bits are `EINVAL`. A failure after the create removes the new name. Other processes of the kernel that look at the destination wait until the copy has committed, so they see the old file or none, then the whole copy. Another tab sharing the OPFS root can see it empty until the commit, as with any create followed by a write. A source that isn't a regular file is `EINVAL`, and a copy between two mounts is `EXDEV`, so a program falls back to reading and writing. On the OPFS root a copy is one kernel call and about 5 OPFS lookups |
 
 The kernel's device nodes answer the mode and owner an Emscripten program sees for them ([#1028](https://github.com/ai-ecoverse/slicc-kernel/issues/1028)):
 - `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom` and `/dev/tty` are root's character devices with `0666`. `/dev/tty` also stats when the process has no controlling terminal; opening it then is `ENXIO`, as on Linux.
@@ -1165,7 +1167,7 @@ The tools in `tools/cert-review/`:
 `.github/workflows/cert-review-pr.yml` certifies a pull request before it merges, so `main`, and with it npm's `latest`, only gets certified changes. It reports as the `cert-review` commit status, a required check of `main`'s merge queue. Every same-repo pull request gets one (`not behaviour`, `waits for merge-ok`, the review's verdict, or the self-guard's hold). **A pull request from a fork gets none**, because fork code never reaches the pools or the key, so it can't enter the queue: only Lars merges one, after reading it.
 
 - **Which pull requests:** those that change shipped behaviour, by a conservative path filter (`behaviourFiles` in `lib.mjs`): `src/`, `build.mjs`, `package.json` and its lock, `tsconfig*.json`, `.releaserc*`, `tools/release-recover.mjs`, the workflows that build or publish (`release.yml`, `ci.yml`), and the reviewer's own code (`tools/cert-review/`, `cert-review*.yml`, `.github/runner.mjs`). **Main's code decides:** every job checks out the pull request's base and fetches only the trees under test, so the path filter, pack, suite, findings, the instructions and record all run from `main`. The merge group's lookup loads its tools from the protected branch itself (`merge_group.base_ref`), not from the group's `base_sha`, which can be the speculative commit of the group ahead and carry the reviewer changes of the pull requests queued before it; `base_sha` is only the lookup's input. A pull request that changes the reviewer is reviewed by `main`'s reviewer. The workflow file of a `pull_request` run still comes from the pull request, so a change to the reviewer is held by the **certification self-guard**, the shared composite action `slicc-shared-web/.github/actions/cert-self-guard` at a pinned sha. Its light `self-guard` job runs on every pull request event and every merge group. When the diff touches the reviewer (`.github/workflows/cert*.yml`, `tools/cert-review/**`, `.github/runner.mjs` and the guard's own paths), `cert-review` is red with `reviewer change: needs coordinator review (cert-review-approved on this head)` and no review runs. The label is the Kernel Coordinator's alone: they add `cert-review-approved` after reading the change, and the review then starts with `merge-ok`. The approval is bound to the head's arrival on GitHub, and a push removes the label. A merge group that holds an unapproved reviewer change is red too. Node's glob `**` skips dot segments, so the guard's paths list the dot forms of each reviewer directory too (`REVIEWER_GLOBS` in `lib.mjs`), and the job's `cert` step reads the guard's receipt and fails closed: a reviewer path the globs still miss (any path under `scripts/cert/`, `tools/cert-review/` or the guard's directory, by prefix) without the label on that head is a FAIL, and a PASS without a receipt, or with a member that lacks its `paths` or `touched` list, is INCOMPLETE. If the guard doesn't finish, the status is an error that asks to re-run the failed jobs. **The residual:** the pull request's YAML still decides what runs, so the guard catches an accidental edit, not a deliberate rewrite of this workflow. The pull request that adds this workflow can't pass its own check, because its base has no tools yet. Any other pull request gets a green status that says `not behaviour: <files>`, so the skip is recorded.
-- **A gate's silence never reads as a pass:** every tool a workflow gates on (`pr.mjs plan`, `same` and `lookup`, `same.mjs`, `pack.mjs`, `delta.mjs`, `suite.mjs`, `compat.mjs`, `findings.mjs`, `record.mjs`, `verify.mjs`) ends with one verdict line, `<tool>: PASS <what>` or `<tool>: FAIL <why>`, from the shared `verdict()` in `lib.mjs`. `tools/cert-review/gate.sh [--either|--evidence] <tool> <command>` runs a tool, passes its output through, and exits 70 with an error when the tool's last line starting `<tool>: ` isn't a PASS or FAIL verdict, so a tool replaced by `true`, an early exit 0 or an empty run fails its step. By default the step passes only on PASS with exit 0; a PASS with a non-zero exit, or a FAIL with exit 0, fails it. `--either` keeps the tool's own exit, for tools whose FAIL is evidence that record judges later (suite, compat). `--evidence` needs only the verdict, for the NEGATIVE run on main's tree, which is expected to fail. `suite: PASS` means exit 0, no failures and every expected file has results; `compat: PASS` means a run turned up with no regressions. `prompt.mjs` writes the review prompt to stdout, which goes into a step output, so it prints no verdict; a missing prompt leaves the review without `verdict.json`, and record marks that INCOMPLETE. The gate reads the tool's stdout, and no tool passes the pull request's output there: suite pipes the tests' output into its log file, and compat prints only its own lines, so a test that prints `suite: PASS` can't stand in for a silent suite. A tool that let such output through would need its verdict on a separate descriptor. Because the workflow of a pull request runs main's tools, the tools gain their verdicts first, and the workflows call them through `gate.sh` in the pull request after that.
+- **A gate's silence never reads as a pass:** every tool a workflow gates on (`pr.mjs plan`, `same`, `lookup` and `notice`, `downstream.mjs`, `same.mjs`, `pack.mjs`, `delta.mjs`, `suite.mjs`, `compat.mjs`, `findings.mjs`, `record.mjs`, `verify.mjs`) ends with one verdict line, `<tool>: PASS <what>` or `<tool>: FAIL <why>`, from the shared `verdict()` in `lib.mjs`. `tools/cert-review/gate.sh [--either|--evidence] <tool> <command>` runs a tool, passes its output through, and exits 70 with an error when the tool's last line starting `<tool>: ` isn't a PASS or FAIL verdict, so a tool replaced by `true`, an early exit 0 or an empty run fails its step. By default the step passes only on PASS with exit 0; a PASS with a non-zero exit, or a FAIL with exit 0, fails it. `--either` keeps the tool's own exit, for tools whose FAIL is evidence that record judges later (suite, compat). `--evidence` needs only the verdict, for the NEGATIVE run on main's tree, which is expected to fail. `suite: PASS` means exit 0, no failures and every expected file has results; `compat: PASS` means a run turned up with no regressions. `prompt.mjs` writes the review prompt to stdout, which goes into a step output, so it prints no verdict; a missing prompt leaves the review without `verdict.json`, and record marks that INCOMPLETE. The gate reads the tool's stdout, and no tool passes the pull request's output there: suite pipes the tests' output into its log file, and compat prints only its own lines, so a test that prints `suite: PASS` can't stand in for a silent suite. A tool that let such output through would need its verdict on a separate descriptor. Because the workflow of a pull request runs main's tools, the tools gain their verdicts first, and the workflows call them through `gate.sh` in the pull request after that.
 - **Evidence comes by id from the job that made it:** the jobs that run the pull request's code (`pack candidate`, which builds it, and both `suite` legs, which run its tests) share the run's artifact store with the jobs that don't, and code in them can reach the runner's artifact token. So no consumer trusts a name or a pattern. Each producer job hands on its artifact's `artifact-id` and `artifact-digest` as job outputs, consumers download by `artifact-ids` with `digest-mismatch: error`, and no upload uses `overwrite`, so a second upload of a name fails instead of replacing it. The base build has its own job, `pack base`, which runs only main's code, so the candidate's job can't speak for it; it restores no Actions cache either, because the candidate job shares the run's cache scope (npm would refuse a tampered tarball against the lockfile's integrity anyway). `tools/cert-review/artifacts.mjs` lists the run's artifacts and fails when one in our namespaces (`kernel-tgz-`, `cert-review-`) wasn't uploaded by its producer, with that producer's id, name and digest, or when a producer's artifact is gone. `same` runs it on the two tarballs before it compares them, so the "not behaviour (identical dist)" skip can't be bought with a planted base tarball. `record` runs it on every artifact before it writes the line, and a failure there makes the record INCOMPLETE. **What this protects:** artifacts of jobs that run no pull-request code (the base build, the delta, kernel-compat, the same-dist evidence, the review's verdict and evidence) can't be forged or replaced. **What it doesn't:** the candidate's own artifacts, its tarball and both suites' results (the base leg runs the pull request's tests on main's tree, for the NEGATIVE), stay the pull request's by design, because code that runs on a runner can always lie about its own results. They're consumed by their exact names, at most one each. That's the same deliberate-malice residual as the workflow-rewrite one above.
 - **Decide by what ships:** when a pull request's behaviour files are only `package.json` and/or `package-lock.json` (a Renovate devDependency bump, say), both sides are packed as usual, and then the `same dist` job (`same.mjs`, in code) compares the two tarballs before anything else runs. It checks the same paths, entry types (a symlink isn't a file) and modes. `package.json` must be equal once `devDependencies`, `version` and `gitHead` are removed, so `scripts` (they ship, a consumer's install runs `postinstall`, and `prepack`/`prepublishOnly` can make the published build differ from the packed one), runtime dependencies, `exports`, `bin`, `files` and `engines` must all match. Every other file must be byte-identical once each side's **exact** version string (build.mjs bakes it into `dist`) is replaced by one placeholder. A version string in one more place, or any looser match, is a difference. Source maps are compared as JSON without their `mappings`, whose columns the longer or shorter version string shifts; the code itself is compared byte for byte. If all of that matches, `cert-review` is green with `not behaviour (identical dist): <candidate sha256> = <base sha256> after normalising <v1>/<v2>`, and the suites, kernel-compat, the review and the record are skipped. `same.json` (uploaded as an artifact) names the rule and both tarballs' full sha256. Any difference, or a `same dist` job that doesn't finish, means the full review. The kernel has no runtime `dependencies`, since esbuild bundles everything into `dist`, so a devDependency that changes the product shows up as a byte difference. A test-only devDependency is checked by `ci / test`, on the pull request and on the merge group.
 - **When:** only for a same-repo pull request with `merge-ok`, when the label is added and on every push while it is set. Each pull request has its own concurrency group, and a newer push cancels the older review. A behaviour change without `merge-ok` gets a red status, `waits for merge-ok to be reviewed`. Other labels don't start or cancel anything. Fork code never reaches the pools or the key.
@@ -1176,9 +1178,15 @@ The tools in `tools/cert-review/`:
   - the review (Claude through `claude-code-action/base-action`, with the Bedrock key in that step only, no issue filing);
   - and record, which re-hashes the candidate's artifact against the pack job's sha256.
 - **The verdict** goes into the commit status on the pull request's head and a comment on the pull request. Certified is green. A FAIL is red with the line, and the pull request needs a fix. **INCOMPLETE** (a job that didn't succeed, no verdict, a suite without results) is red with `re-run the failed jobs`, never a silent pass. Main moving on after a review doesn't trigger another one; a push does.
-- **Merge groups only look up.** On `merge_group`, the job lists the group's pull requests (its merge commits) and requires, for each one, a green status on the head being merged, or no behaviour change. Otherwise it's red, naming the pull request. It runs in seconds, with no secrets and no model. The group's `ci / test` covers the combined tree.
-
-Not part of the run yet: bios's real setup (the base-set sweep in bios, the upgrade boot, the sudo rows under `enforce`). It would plug in as one more job before review that runs slicc-bios's `npm run test:cert` with `CERT_COMMANDS` and the release's tarball, and leaves its results next to the others.
+- **Merge groups only look up.** On `merge_group`, the job lists the group's pull requests (its merge commits) and requires, for each one, a green status on the head being merged, or no behaviour change. Otherwise it's red, naming the pull request. Then it adds the group's downstream verdict. The group's `ci / test` covers the combined tree.
+- **Downstream, once per merge group:** the `merge group pack` job packs the group's tree with `main`'s tools (`x.y.z-group.<run>`), and `downstream bios canary` calls slicc-bios's `canary.yml` at a pinned full sha (`c2cec127…`, never a tag) with that tarball, bound by artifact id, digest and sha256, on the `sweep`, `guard` and `k4-link` shards, each run once. `downstream.mjs` judges its result (`downstreamOf` in `lib.mjs`), and the lookup requires it. The `merge group` job reads that result as `result.json` from the canary's own artifact, downloaded by the call's `artifact-id` with `digest-mismatch: error`, not from the `result` output: that output can be up to 512 KB, and one environment variable holds 128 KB, so passing it that way stopped the step from starting at all (`Argument list too long`, runs 39166647175 and 39168823799). A missing or unreadable artifact leaves no result, so the group is INCOMPLETE:
+  - **PASS:** the candidate passes, `downstream: PASS bios … N pass`;
+  - **FAIL:** a regression, or a failing test (by the counts, even without its rows) is **not certified**, and the group is red;
+  - **INCOMPLETE** blocks too: no result, a result from another bios than the pinned one, a missing shard, a shard that exited without a failing test, a refused candidate, 0 tests run, or no test passed;
+  - **not judged:** every failing test fails on `main`'s own pins too (`both`), so the canary is broken on `main`, not by this group. That passes with a note, as do `both` rows next to a pass. Either way, `pr notice` opens one issue, `downstream: bios canary failing on main (<bios sha>)`, or comments on the open one, and the next group that passes closes it. Flaky rows are advisory.
+- **`downstream-override`**, next to `cert-review-approved`: a coordinator-only label that waives a downstream FAIL. It is head-bound by the same self-guard (`cert-self-guard` with `approval-label: downstream-override` and every path, dot paths included), so a push drops it. It applies only when **every** pull request in the group carries it on the head being merged: the group reads the guard's receipt and waives only when each member was checked and approved, so a pull request whose paths the guard's globs miss gets no waiver. The group's line then reads `downstream: overridden (downstream-override on the head, by the coordinator) …`, with the failure it waived. It never covers INCOMPLETE. A pull request's own certification line ends `downstream: merge group`, since its downstream verdict comes in the group.
+
+Not part of the run yet: the rest of bios's setup (the `rest` shard, the upgrade boot, the sudo rows under `enforce`). More shards plug in through the canary call's `shards`.
 
 ## Test fixtures