@ai-ecoverse/slicc-kernel 1.121.0 → 1.122.0

package/README.md

@@ -612,6 +612,8 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
 - **File contents:** on a mount with `ranges`, as `tmpfs` is, and on the root file system when it is in OPFS (in a browser with sync access handles) or an in-memory root, a program reads and writes a file in 64 KiB pages: reads come from the driver 1 MiB at a time, and changed pages go back in 1 MiB runs once 8 MiB have changed, at `fsync`, and when the file is closed (from the kernel's own descriptors also 250 ms after a write). A file of any size the mount can hold works, with memory to spare for about one more copy of it. In OPFS a write opens a sync access handle for just that write. An `fsa` mount (a picked folder) has no sync access handles, and Chrome copies the whole file into a swap file for every writable stream, so it writes through one stream per open file: the first write back opens it, later runs go into it at their offsets, and it is committed, one copy of the file, when the last process writing the file closes it (or exits), before another process reads what was written, and before a rename, unlink or unmount. Until then other processes see the new size but read the old bytes outside what was written. Elsewhere (`hostfs` with an older proxy, and a root outside OPFS), file contents are read whole when a program opens a file and written back when it closes it. Before a program's next call into the kernel (any wait, read of a pipe or terminal, sleep or spawn), its changed pages also go back, so other processes and the embedder see a write as soon as the writer does anything but compute: a daemon's pidfile, a log another process follows. On a whole-file mount that holds only for files of up to 64 KiB; a larger one goes back at close, fsync and exit, as before. Beyond that, both give close-to-open consistency, as NFS has. A process that ends without closing its files keeps what it wrote: on exit, on a trap, and when a signal kills it. For a killed WASI program, the kernel lets its worker write its changed pages back before `waitpid` reports the death. The only exception is a worker that makes no call into the kernel for 2 seconds after the signal; it is stopped without that write-back.
 - **Exclusive create across tabs** ([#912](https://github.com/ai-ecoverse/slicc-kernel/issues/912)): two kernels on one OPFS root (two tabs, or a tab and a worker page) agree on `O_CREAT|O_EXCL` and `set -C`: exactly one wins a name. While another kernel on the root is present, the create runs under a Web Lock for its path and checks the OPFS entry itself. A kernel alone on its root takes no lock. Each kernel holds an alive Web Lock, granted before it first looks for peers. A kernel that finds peers waits for their answer (at most 500 ms) before its first exclusive create, and a kernel answers a newcomer only once its unlocked creates are done. So of two kernels that boot at the same moment, the later one to look sees the other, and no unlocked create overlaps a locked one.
 - **One file, several processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): a process that keeps a file open sees what another process writes to it. Every change the kernel applies to a file (a write back, a truncate, an append, a rename or a removal) bumps a change ring the kernel shares with all its processes. Before each read, write or seek to the end, a process's open file checks the ring (one atomic load) and, if its file changed, drops its clean pages and reads the size again, keeping its own unwritten changes. A write by one process is visible to another one's next read once it has gone back to the kernel: at the latest at the writer's next `fsync`, close, exit or call into the kernel (see above), so sqlite, which `fsync`s every commit, sees another connection's commit at its next transaction. A write through a file the kernel holds (a JS program's `ctx.open`, a WASI program's file after a fork or spawn) is in the kernel already, so other processes see it at their next read, before it is written back. A file the kernel itself holds open (a JS program's file, a WASI program's file after a fork or spawn) is the one copy: reads, writes, truncates and appends by path from other processes go to it, and their writes go through to storage at once, so a commit is in storage (and in `kernel.readFile`) when the committing process returns from `fsync`. `O_APPEND` writes go to the kernel at once, and the kernel picks the offset, so appenders in several processes keep every line. A file system without ranged I/O (`hostfs` with an older proxy, a root outside OPFS) rereads a file another process changed, but writes it back whole, so two writers there still need locks. The same file under two names is one file too ([#669](https://github.com/ai-ecoverse/slicc-kernel/issues/669)): a write by path through a symlink reaches the file the kernel holds under its real path (resolved without following links in a `sudo` process's no-links view, so an untrusted link is still refused), a WASI program's open file also watches its real path (resolved once, the first time another file changes), and an open file with more than one hard link checks its size, version and mtime when anything changed, so a write through another name is seen. Limits for now: an alias on a mount that has no `realpath` is still only the name it was opened by; another tab's writes reach this tab's processes only as far as the root's change notifications (OPFS's `BroadcastChannel` watch) carry them. A by-path `stat` or `lstat` of a file the kernel holds open has its current size, writes not yet written back included ([#780](https://github.com/ai-ecoverse/slicc-kernel/issues/780)). Its mtime is the file system's until write-back, though, so a tool that compares times, like `make`, may see a just-written file as older than it is for up to the write-back delay.
+- **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, and so are a WASIX program's through the `slicc.fcntl_lock` import (below), so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` keeps its behaviour (whole file, per process, keyed by path). Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program built against a wasix-sysroot without the import keeps its libc's local answer (every lock granted), so python's sqlite excludes other processes only once wasix-python is rebuilt on the sysroot that has it ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)). An Emscripten program's `flock()` reaches the kernel's `flock` table (per open file description) through homescoop's shim, a strong `flock()` that calls the import `env.__syscall_flock(fd, op)`. The kernel wraps that import by name (#947): `LOCK_SH`/`LOCK_EX`/`LOCK_UN` with `LOCK_NB` (`EWOULDBLOCK` for a conflict), 0 or -errno. A glue without the shim keeps Emscripten's stub, which answers 0. A lock belongs to the file its descriptor was opened on: a file renamed over the name later, or the name's removal, doesn't move it.
+- **Locks across tabs** ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)): two kernels on one OPFS root (two tabs, or a tab and a worker page) share one table of `fcntl` record locks and `flock` locks, so a sqlite3 in one tab finds the database locked while a transaction is open in another. **Known issue, data corruption: lock-protected writes across tabs** ([#949](https://github.com/ai-ecoverse/slicc-kernel/issues/949)). The locks are enforced across tabs, but what one tab writes under a lock isn't yet guaranteed to be what the other tab reads once it holds the lock, so a lock-protected read-modify-write across tabs can lose or corrupt data. Don't rely on cross-tab locking until #949 is fixed. Use within a single tab is unaffected.
 - **Locks between processes** ([#665](https://github.com/ai-ecoverse/slicc-kernel/issues/665)): an Emscripten program's `fcntl` byte-range locks (`F_SETLK`, `F_SETLKW`, `F_GETLK`) are the kernel's, and so are a WASIX program's through the `slicc.fcntl_lock` import (below), so sqlite3 connections in different processes exclude each other: a second writer gets `database is locked` while the first holds its transaction, or waits with a busy timeout. The kernel keeps one table keyed by file identity (device and inode, so every name of a hard link is the same file), with POSIX semantics: locks belong to the process (an `exec`'d program keeps them under its pid; a forked child doesn't get them), a conflicting `F_SETLK` fails with `EAGAIN`, `F_GETLK` reports the holder's pid and range, `F_SETLKW` waits in the kernel without holding up anything else and a signal ends the wait with `EINTR`, a wait that would close a cycle between waiting processes fails with `EDEADLK`, a write lock on a descriptor not open for writing (or a read lock on one not open for reading) is `EBADF`, closing any descriptor of the file drops the process's locks on it, and exit releases them all. Before an unlock, the process writes back what it changed, so the next holder reads it. An open file's directory entry is checked against the change ring too: a name another process removed or replaced is looked up again, so sqlite's check for a hot journal sees the other connection's commit. A WASI imports module's `tryLock`/`unlock` keeps its behaviour (whole file, per process, keyed by path). Limits: locks are per kernel, not across tabs ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)); and a WASIX program built against a wasix-sysroot without the import keeps its libc's local answer (every lock granted), so python's sqlite excludes other processes only once wasix-python is rebuilt on the sysroot that has it ([homescoop#407](https://github.com/ai-ecoverse/homescoop/issues/407)). An Emscripten program's `flock()` is a stub compiled into its libc that never reaches the kernel either, until a homescoop shim routes it to a kernel import ([#680](https://github.com/ai-ecoverse/slicc-kernel/issues/680)). A lock belongs to the file its descriptor was opened on: a file renamed over the name later, or the name's removal, doesn't move it.
 - **Locks across tabs** ([#675](https://github.com/ai-ecoverse/slicc-kernel/issues/675)): two kernels on one OPFS root (two tabs, or a tab and a worker page) share one table of `fcntl` record locks and `flock` locks, so a sqlite3 in one tab finds the database locked while a transaction is open in another. What one tab writes under a lock is what the other reads once it holds the lock ([#949](https://github.com/ai-ecoverse/slicc-kernel/issues/949)). While another kernel shares the table, a grant drops the clean cached contents of the file (a WASIX program's own copy and the kernel's), so the next read comes from storage. An unlock, `LOCK_UN` and the release at close write the kernel's copy back first. A kernel alone on its root keeps its caches.
   - **One leader:** the first kernel on a root takes a Web Lock and serves the table from its kernel worker. A kernel alone on its root is its own leader, and its lock calls cost what they did. Another kernel's lock calls go to the leader over a BroadcastChannel: about 0.1 ms a call against 0.02 ms on the leader, measured with the leader's tab in the background.
@@ -621,7 +623,7 @@ Directories are renamed with `FileSystemHandle.move()` where available, else by
   - **The same file:** a lock is keyed by the file's device and inode, which both kernels read from the shared sidecar, so a rename by either tab keeps it. A file removed and created again at the same path has the same key, as it already has within one kernel.
   - **Versions:** the channel and Web Lock names carry a protocol version (`v1`), and a kernel ignores messages that aren't v1. So kernels on different protocol versions (two tabs across an update) don't share locks, as before #675.
   - **Not across tabs:** the imports module's whole-file `tryLock` (per kernel), and locks on other mounts.
-- **Another tab's writes** ([#959](https://github.com/ai-ecoverse/slicc-kernel/issues/959)): a program's names and stats are cached in its process, and its kernel refreshes them when the kernel itself changes a file. While another kernel shares the root (a second tab), the kernel marks that view foreign, and a program then looks names up and reads stats again on every look. So a program that keeps checking a file sees what the other tab wrote, synced and closed. A kernel alone caches as before. **Known issue** ([#962](https://github.com/ai-ecoverse/slicc-kernel/issues/962)): a descriptor tab B already has open doesn't see tab A's later writes until B reopens the file. Its contents stay in the process. A reader that takes a lock does see them (see above).
+- **Another tab's writes** ([#959](https://github.com/ai-ecoverse/slicc-kernel/issues/959)): a program's names and stats are cached in its process, and its kernel refreshes them when the kernel itself changes a file. While another kernel shares the root (a second tab), the kernel marks that view foreign, and a program then looks names up and reads stats again on every look. So a program that keeps checking a file sees what the other tab wrote, synced and closed. A kernel alone caches as before. While another kernel shares the root, a kernel also tells it of each file it writes ([#962](https://github.com/ai-ecoverse/slicc-kernel/issues/962)). The other kernel then drops the clean contents of that file, both its own and its programs', so a descriptor tab B already has open reads tab A's new bytes on its next read, as Linux's shared page cache does. The notice is a message, so a read that races it by less than a message's delivery can still see the old bytes, as with any reader that doesn't lock.
 - **`slicc.fcntl_lock(fd, cmd, flock_ptr) -> errno`** ([#722](https://github.com/ai-ecoverse/slicc-kernel/issues/722)): a WASIX program's `fcntl` record locks reach the same table as an Emscripten program's, so the two see each other's locks.
   - **cmd:** `12` `F_GETLK`, `13` `F_SETLK` or `14` `F_SETLKW`. Any other cmd is `EINVAL`.
   - **The struct:** musl's wasm32 `struct flock`, 32 bytes: `l_type` i16 at 0 (`0` read, `1` write, `2` unlock), `l_whence` i16 at 2 (`0` SET, `1` CUR, `2` END, resolved against the descriptor's offset or size), `l_start` i64 at 8, `l_len` i64 at 16 (`0` to EOF and beyond, negative before `l_start`), `l_pid` i32 at 24. The kernel reads and writes only those five fields and never the padding.
@@ -1158,6 +1160,12 @@ A threaded WASIX C program for `test/integration/thread-signal.test.mjs` (#520).
 
 **Build:** `WASIX_SYSROOT_28=/path/to/wasix-sysroot ./build.sh`, with `wasixcc` 0.4.7 (`-pthread`) and binaryen's `wasm-opt` on `PATH`. `bin/sigthread.wasm` is 44,788 bytes, sha256 `5781fa39d0a0032cd9b214b71de1f15f589b40e4cecd6862362048dd27f0db56`.
 
+### `eflocktest`
+
+An Emscripten C program for `test/integration/eflock.test.mjs` (#947), linked with homescoop's `cli` shim profile, whose `slicc_libc_gaps.c` has the strong `flock()` over `env.__syscall_flock` (homescoop#947). `eflocktest hold PATH` takes `LOCK_EX`, prints `held`, waits for EOF on stdin, then prints the `LOCK_UN` result. `eflocktest try|shared PATH` prints the result of `LOCK_EX|LOCK_NB` or `LOCK_SH|LOCK_NB`.
+
+**Build:** `HOMESCOOP_ROOT=/path/to/homescoop ./build.sh`, with emsdk 4.0.23 (homescoop's `scripts/install-emsdk.sh`). It was built at homescoop `bb/973-947-shims` f08e0777. `bin/eflocktest` is 59,121 bytes, sha256 `b4ebcc0b6b53bcb86e78f80a5e97a428b4f3156102d9be21cd96dd57f985285a`; `bin/eflocktest.wasm` is 11,754 bytes, sha256 `a4afd017a7d14bd12579c982e3d865f6f095a1679847508d71781f59fe0a2d33`.
+
 ### `esiginherit`
 
 An Emscripten C program for `test/integration/esiginherit.test.mjs` (#965), linked with homescoop's `cli` shim profile, whose `slicc_signals.c` seeds inherited `SIG_IGN`s in a `constructor(101)` from `Module.sliccKernel.sigInherited()` (homescoop#973). It prints what its own `sigaction(sig, NULL, &old)` reports for `SIGPIPE`, `SIGHUP` and `SIGINT`.