@ai-ecoverse/slicc-kernel 1.96.0 → 1.97.1

package/README.md

@@ -373,6 +373,7 @@ WASIX programs get `AF_INET`/`AF_INET6` `SOCK_DGRAM` sockets (`AF_UNIX` datagram
 - **The proxy** listens on `127.0.0.1:3128` from the first connection on. It takes absolute-form `http:` requests and `CONNECT` tunnels, which it terminates with a certificate for the requested host, issued by the kernel's own CA, then hands each request to the transport. Programs start with `http_proxy`, `https_proxy` (and the upper-case names) pointing at it, `no_proxy` covering loopback and the [node name](#node-name), and `SSL_CERT_FILE`, `CURL_CA_BUNDLE` and `GIT_SSL_CAINFO` pointing at the CA bundle in `/etc/ssl/certs/slicc-kernel-ca.pem`; variables in `env` override them. The CA's key never leaves WebCrypto; it is kept in IndexedDB, in `<metadata>-ca`, or in `slicc-kernel-ca` when `metadata` is the default or `false`, so kernels on one origin share one CA whether or not they keep metadata. `ca` names another database, and `ca: false` keeps a CA in memory for the kernel's lifetime. Each kernel adds its CA to the bundle instead of replacing it, so every kernel on the origin stays trusted; the bundle keeps the newest 16.
 - **TLS** needs [`@ai-ecoverse/wasm-tls-engine`](https://www.npmjs.com/package/@ai-ecoverse/wasm-tls-engine) installed under `modules`, like a command. Without it, `CONNECT` to port 443 is answered `501` with the reason. A tunnel to another port is accepted, and what the program sends first decides: plain HTTP is served as `http://host:port`, without TLS, and a TLS handshake is terminated as above, or the tunnel closes when it cannot be.
 - **The transport** is an object on the page with `traits` (`manualRedirects`, `encodedBodies`, `maxRequestBody`, and `crossOrigin`: `'cors'` or `'any'`) and `fetch(request)`, which answers `{ status, statusText, headers, body, cancel }` with `body` an async iterable of `Uint8Array`, and optionally `url` and `redirected: true` when it followed redirects itself (`fetchTransport()` does), so a JS program's `fetch` reports where it ended. The kernel pulls the body one chunk at a time, so a slow program slows the download. A rejection with a numeric `status` is answered with that status; any other with `502`. `fetchTransport()` is the plain `fetch` of the page, bound by CORS; its `hint` option is appended to the message of a request it could not make, so a program's 502 can name a way around CORS (`registry.npmjs.org` and jsDelivr allow it); an embedder with a way around CORS passes its own. A response body that delivers no data for `bodyIdleMs` (default 5 minutes; `0` or `Infinity` turns it off) fails with `ETIMEDOUT` instead of waiting forever, which matters because Chrome under heavy load can stop handing a page the rest of a body. A slow body that keeps delivering is never cut off.
+- **Traits of the embedder's own:** `traits` may carry more fields than the kernel reads. The kernel keeps them, and attached clients get them as they are (`client.transport.traits`, `onTraits`). In TypeScript, `NetworkTraits` types its own fields and leaves any other as `unknown`. One is typed: `exit`, the default exit an embedder routes through, `{ kind, caps }` (`NetworkExit`). `kind` is `'local'`, `'remote'` or `'tailnet'` (or another string), `caps` lists what the exit carries (`['net', 'http', 'ws']`), and further fields are `unknown`. When traits change later, call `kernel.updateTransportTraits()`.
 - **Redirect hops** ([#302](https://github.com/ai-ecoverse/slicc-kernel/issues/302)): a transport that follows redirects but can see them (a browser extension's `webRequest.onBeforeRedirect` can) sets `traits.redirectHops` and answers `redirects: [{ status, url, headers }]`, one entry per 3xx it followed, in order (`NetworkRedirectHop`). The kernel then treats it as one that does not follow. Programs get each 3xx as it was answered: through the proxy, curl, requests and `hf` see the real redirect, `X-Repo-Commit` and `ETag` included, and `ctx.fetch` with `redirect: 'manual'` answers it. The rest of the chain is held for the process that asked, at most 30 s and 8 chains per process (64 in all), with the final body unread. A follow-up from that process, with the method the redirect leads to and the same `Range`, `If-None-Match`, `If-Modified-Since`, `If-Range` and `Accept-Encoding` (and an `Authorization` if the first had one), is answered from it without a second request; any other request goes to the transport. A chain that expires or is pushed out is cancelled, and so is one whose follow-up arrives and does not match. After a redirect to another origin the follow-up is expected without `Authorization`, which HTTP clients drop when they follow one; a `Location` matches without its `#fragment`. The page's own `fetch` cannot show a redirect, so `fetchTransport()` in a page stays a transport that follows, and `redirect: 'manual'` fails there as before. `fetchTransport({ redirect: 'manual' })` hands redirects back unfollowed where the platform's `fetch` shows them (Node; a browser answers an opaque status 0), and `nodeTransport()` does so. A kernel socket also records who connected it (`SO_PEERCRED`): the connecting process's pid, uid and gid on the accepting side, and the listener's on the connecting side, kept when the socket passes to another process. The proxy uses this to keep each process's held chains its own.
 - **WebSockets:** a `GET` with `Upgrade: websocket`, in absolute form or inside a tunnel, is bridged to the transport's optional `websocket({ url, protocols, headers, signal })`. It answers `{ protocol, buffered, send(data), messages, close(code, reason), closed }`, with `messages` an async iterable of strings and `Uint8Array`s and `closed` resolving with `{ code, reason }`; a transport that has it says so with `traits.websocket`. The program gets `101 Switching Protocols` with the far end's protocol, then its frames go out as messages and the far end's come back as frames. Pings are answered by the proxy, and close codes pass both ways (a code that may not go on the wire becomes 1000 or 1011). A message over 64 MiB, or more than 16 MiB of payload from the far end waiting for the program, closes with 1009. The proxy stops reading the program while 1 MiB of its messages wait to be sent. A refused connection is `502` with the reason. A transport without WebSockets answers `501` with a message naming the option below, and the kernel logs a warning once, so a program never hangs. `fetchTransport({ webSocket: true })` opens them with the page's `WebSocket` (or pass a constructor); they are off by default because a browser WebSocket carries the user's cookies for the target site, which `fetch` here never sends (`credentials: 'omit'`), so a program could reach a site as the user. A page also cannot send headers on a WebSocket, and an `https:` page cannot open `ws:` to anything but loopback, which is a `502` that says so. `nodeTransport()` has WebSockets on (Node keeps no cookies) and sends the request's headers. An open fails as soon as Node's `WebSocket` reports an error, also on Node 22, which never follows the error with a close. A transport's `websocket()` may reject with an `Error` whose `code` is an errno, and the kernel hands that on to the program. Node's `WebSocket` doesn't say why an open failed, so when one fails, `nodeTransport()` finds out: it looks up the host's name (`EAI_NONAME` when that fails), then makes one TCP connection to the target, or a TLS one for `wss:`, for at most 3 s, and closes it at once. A refusal there is `ECONNREFUSED`, an unreachable host `EHOSTUNREACH` or `ENETUNREACH`, no answer `ETIMEDOUT`, and a TLS failure (a bad certificate) or a server that takes the connection but refused the upgrade `EPROTO`. A server that refused the upgrade therefore sees a second connection. The diagnosis never runs when an open succeeds, and never in a page (`fetchTransport()`), whose failed opens stay `ECONNREFUSED`. The credential-free path for a page is the coming slicc-node relay (`localProxyTransport()`, phase 2 of [#204](https://github.com/ai-ecoverse/slicc-kernel/issues/204)), which has no WebSockets yet.
 - **`localProxyTransport({ url, key })`** sends every request to a local proxy on loopback, such as [slicc-node](https://github.com/ai-ecoverse/slicc-node) or slicc-swift, which fetches it without CORS. Redirects reach the program unfollowed, with every `Set-Cookie`; bodies arrive decoded. `probeLocalProxy({ url, key })` resolves with the proxy's capabilities, or `null` when it is gone or refuses the key, so a page can fall back to `fetchTransport()`. `checkLocalProxy({ url, key })` says why: it resolves with `{ state }`, which is `ready` (with `probe`), `blocked` (the user denied Chrome's Local Network Access permission, `loopback-network`), `unreachable` (nothing answered; its `permission` is `granted`, `prompt` or `unknown`, and with `prompt` the page cannot tell a dismissed prompt from a proxy that is not running, since Chrome asks only once it has a connection), `refused` (with `status` and `error`, such as a stale key) or `incompatible`. The permission is not consulted when the page itself is on loopback. Both take a `fetch` option, and the transport `maxRequestBody` (default 64 MiB) and `bodyIdleMs` like `fetchTransport`.
@@ -532,7 +533,8 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
 - **Capabilities:** `readonly`, `symlinks`, `links` (hard links: the driver answers `link(from, to)`, reports `nlink` in its attributes, and gives every attribute a stable `ino`, the same for all names of a file; a mount whose driver offers `links` but answers an attribute without `ino` loses them: `link` is `EPERM` there and `st_nlink` is 1), `chmod`, `owners`, `ranges`, `maxIo` (the largest read or write per message), `maxFile`, `listingStats`, `sessions`, `linkTimes`, and `attrTtl`/`entryTtl` (in ms, default 1000). `sessions` promises that a handle opened for writing keeps its writes to itself until it is released, so the kernel keeps one open per file being written instead of one per write (the `fsa` driver has it). `linkTimes` promises that `setattr` with `mtime` on a symlink sets the link's own time. Without it, setting a link's own times (`touch -h`, `utimensat` with `AT_SYMLINK_NOFOLLOW`) fails with `EOPNOTSUPP` and never reaches the target. tmpfs has it; hostfs does not. Setting times through a link follows it in the kernel, so `setattr` gets the target. `ranges` (driver protocol 1.1) promises that a write lands in the file at once, with no copy of the whole file on `open` or `release`, and that `setattr` takes `size`. `owners` (driver protocol 1.2) promises that the driver stores owners: its attributes carry `uid` and `gid`, `open` with `create`, `mkdir` and `symlink` may carry the new path's `uid`, `gid` and `mode` (the `owner` of the open flags, the second or third handler argument), a new path without them takes its directory's owner, and `setattr` takes `uid` and `gid`. tmpfs has it.
 - **Caching:** the kernel caches attributes and listings for those TTLs, and a driver can push `invalidate` for changes made outside.
 - **File contents:** on a mount with `ranges`, as `tmpfs` is, and on the root file system when it is in OPFS (in a browser with sync access handles) or an in-memory root, a program reads and writes a file in 64 KiB pages: reads come from the driver 1 MiB at a time, and changed pages go back in 1 MiB runs once 8 MiB have changed, at `fsync`, and when the file is closed (from the kernel's own descriptors also 250 ms after a write). A file of any size the mount can hold works, with memory to spare for about one more copy of it. In OPFS a write opens a sync access handle for just that write. An `fsa` mount (a picked folder) has no sync access handles, and Chrome copies the whole file into a swap file for every writable stream, so it writes through one stream per open file: the first write back opens it, later runs go into it at their offsets, and it is committed, one copy of the file, when the last process writing the file closes it (or exits), before another process reads what was written, and before a rename, unlink or unmount. Until then other processes see the new size but read the old bytes outside what was written. Elsewhere (`hostfs` with an older proxy, and a root outside OPFS), file contents are read whole when a program opens a file and written back when it closes it. Before a program's next call into the kernel (any wait, read of a pipe or terminal, sleep or spawn), its changed pages also go back, so other processes and the embedder see a write as soon as the writer does anything but compute: a daemon's pidfile, a log another process follows. On a whole-file mount that holds only for files of up to 64 KiB; a larger one goes back at close, fsync and exit, as before. Beyond that, both give close-to-open consistency, as NFS has. A process that ends without closing its files keeps what it wrote: on exit, on a trap, and when a signal kills it. For a killed WASI program, the kernel lets its worker write its changed pages back before `waitpid` reports the death. The only exception is a worker that makes no call into the kernel for 2 seconds after the signal; it is stopped without that write-back.
-- **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. Limits for now: the same file under two names (a symlink in the path, a hard link, an alias on a mount without `realpath`) isn't kept coherent yet, so processes sharing a file should name it by the same path ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)); another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them; and byte-range locks (`fcntl` `F_SETLK`) are not enforced between processes yet (they come with the next change).
+- **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. Limits for now: the same file under two names (a symlink in the path, a hard link, an alias on a mount without `realpath`) isn't kept coherent yet, so processes sharing a file should name it by the same path ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)); another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them.
+- **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` keeps its behaviour (whole file, per process, keyed by path). Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program's `fcntl` locks (python's `sqlite3`) are no-ops in its libc that never reach the kernel, until wasix-libc gets a kernel import for them ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)), so python's sqlite doesn't exclude other processes yet.
 - **Rules on a mount:**
   - a rename across mounts is `EXDEV`, so `mv` copies;
   - a symlink on the root file system that leads onto a mount, such as `~/proj` → `/mnt/drive/proj`, or a chain from a mount to the root and back, is followed onto the mount by every operation, in every runtime: reads, writes, `stat`, listings, `mkdir`, removals, renames, `chmod`, `chown` and times. Removing the link itself removes the link. Before [#547](https://github.com/ai-ecoverse/slicc-kernel/issues/547) was fixed, the kernel resolved such a link against the directory under the mount point for JS and WASI programs, the embedder's `fs` and attached clients (Emscripten programs weren't affected): reads failed with `ENOENT`, and writes went into that hidden directory, which shows again once the mount is gone;