@volter/world-machine 2.0.0

package/README.md

@@ -0,0 +1,339 @@
+# @volter/world-machine
+
+The machine as a World service: a Firecracker microVM run as an ordinary `process` service, whose
+memory-and-disk snapshots are entries in the World's ledger. `mark` and `branch --at machine@N` take
+the machine back exactly as they take a twin back, with no change to the kernel or the runtime.
+It is the machine half of *retake*: an agent's world (machine memory and disk, and its twins' state)
+goes back to a mark while the agent's conversation, running outside the machine, continues.
+
+## The shape
+
+| Part | Where | What |
+|---|---|---|
+| the ledger | the World, state service `machine` | one entry per checkpoint (`snapshot:<id>`: label, snapshot directory, memory/disk/store bytes, pause, created), written through world-core's `applyTwinWrite`; the head a branch restores is the last snapshot in its view (`wholeLog`) |
+| the machine host | Linux with KVM, or a Lima >= 2 VM with `nestedVirtualization: true` on a Mac | Firecracker in one **slot** (fixed tap, guest address and disk path), run in place: a restore replaces the slot's process, so the guest keeps its address and identity |
+| the store | a directory on the machine host on XFS (reflink) or btrfs | `base-<hash>/` warm bases, `snapshots/<id>/{vmstate,mem,disk.ext4}`, `slots/<slot>/`; for the container backend also `tree-<hash>/` (a filled tree, below) and `slots/<slot>/ct-loaded` (what the slot's machine was restored from) |
+| the guest agent | inside the guest, in every snapshot | steps the wall clock, resets the connections a restore left stale, writes `/etc/world.env` |
+
+The service holds one persistent shell on the machine host (a new ssh per operation cost 75-125 ms on
+a loaded Mac), so each operation is one round trip.
+
+- **Checkpoint** flushes the disk's dirty pages, pauses the guest, writes the pages dirtied since the
+  last snapshot or load (a Firecracker `Diff` snapshot) and the device state to `/dev/shm`,
+  reflink-clones the disk, resumes it, then materializes one full
+  memory file for the entry (a reflink copy of the parent's with the diff applied by
+  `snapshot-editor edit-memory rebase`), so a restore always loads one file.
+- **Restore** loads this branch's head (or the warm base when the ledger has none) with `mem_backend`
+  `File`, steps the guest's wall clock to the World's clock (`TWIN_WORLD_CLOCK_FILE` when set, else
+  the World host's now), resets every connection the restore left stale, and rebuilds the guest's route out.
+- **Nothing runs until asked.** The service answers at once and boots nothing (`state: idle`). The first
+  `GET /machine` or `POST /machine/checkpoint` brings the machine up (the ledger's head, else the warm base);
+  `POST /machine/restore` loads the head itself. A branch's services start before `branch` forks their
+  ledgers, so a retake is `branch --at` followed by `POST /machine/restore`, and the new branch's machine is
+  loaded once, at its mark (it used to boot its head at start and be replaced by the restore a second later).
+- **The warm base** is the declared guest image cold-booted once, provisioned by the declared script
+  and snapshotted whole, cached in the store under a hash of everything that shapes it. A World's
+  machine never cold-boots (20-60 s measured).
+- **Copy-on-write** is a precondition: the store must reflink (`cp --reflink=always`). There is no
+  fallback to a full copy; a whole-image copy dominated every pause it was part of (1-13 s measured).
+- **Egress**: the guest has no gateway and forwarding from its tap is dropped. Its only way out is
+  one forwarder per twin on the tap's host address, at a stable guest-visible port per vendor, pointed
+  at this World's `*_TWIN_URL` at every restore and start (so a checkout re-points it). The guest
+  reads the stable URLs from `/etc/world.env` (`STRIPE_TWIN_URL=http://172.16.0.1:29772`), with the rest of the World's
+  live env (`VOLTER_WORLD_ENV_FILE`: its fake credentials and app values; its proxy as a forwarder at a stable port,
+  which the twins' own addresses bypass; a certificate file it names (its CA and trust bundle) copied to
+  `/etc/volter-world-<name>`, all written at each start and restore; what only its host can use, a directory there
+  or a loopback URL of its other listeners, is left out and named in the file), and `NODE_OPTIONS` requiring world-core's redirect, written with
+  the files it loads to `/opt/volter-world` at each start and restore too: a real SDK calling its vendor's host reaches the twin, as under `volter world
+  run`. Every login shell exports the file (`/etc/profile.d/volter-world.sh`). A World variable whose value is empty
+  is left unset in the machine and named in the file. `world init --repo` writes a repository's `.env.example` keys
+  that way, and set, even to nothing, they hid the app's own `.env.local` (Next.js and dotenv never override a set
+  variable). A deliberately empty value therefore reads as unset in the machine, as it does on the host: `init` writes
+  no empty placeholders, and `up` leaves an empty World value unset.
+
+## The container backend (`backend: "container"`, not yet run)
+
+On a host where a microVM would be nested (a Mac, Firecracker inside a Lima VM), first-touch guest memory faults through
+two hypervisors, so a real project's install ran 20–80× slower than one level up. The container backend (`container.ts`,
+`ct.sh`) runs the machine as the host's own processes in a crun container instead, marked by CRIU:
+
+| | firecracker | container |
+|---|---|---|
+| the machine | a microVM on KVM | a crun container in a network namespace holding the same guest address |
+| its tree | a reflinked `disk.ext4` | one reflinked ext4 image holding the container's root (so NFS handles on an unchanged file stay valid across a restore) |
+| a mark | a memory diff, the disk reflinked | the container paused, the image reflinked, a full CRIU dump (arm64 has no soft-dirty tracking), resumed; the dump's pages then composed against the parent's (`compose.py`) |
+| a restore | the snapshot loaded, the clock stepped, stale sockets reset by the guest agent | the image reflinked back, CRIU restore, stale sockets reset from the host (`ss -K` in the namespace) |
+| the clock | stepped to the World's | the host kernel's, which is now; a frozen World clock is not followed |
+| the World's env, certificates, redirect | written by the guest agent | written into the tree |
+
+Its config adds:
+- `backend`;
+- `image` (the OCI image the tree is made from) and `imageSize` (default 40G);
+- `binds` (host directories bound read-only into the container);
+- `export` (a path in the tree the host's NFS server v3 serves to one client, unexported and re-exported around every
+  swap of the image, with a fixed fsid);
+- `publishPorts` (`{"ranges": [[3000, 3999]], "except": [...]}`, none by default): the machine's listening ports in those
+  ranges, published on the host's loopback at the same port as the machine's own `localhost` (`publish.py`).
+
+The app in the machine registers `http://localhost:<port>` with a vendor, and a twin on the host delivered its webhooks
+to the host's own loopback, where nothing listened. A Lima host also forwards those ports on to the Mac.
+
+A port is published only when:
+- it is allowed: the machine runs code the user did not write, and a port it held first would be taken from a host
+  service that binds it later;
+- the machine listens on it on a wildcard or loopback address, and on no address of its own (how its doors bind);
+- the host does not listen on it already.
+
+Each connection enters the machine's namespace to its loopback. `setup` adds `skip-in-flight` to CRIU's default config
+(`/etc/criu/default.conf`, which crun's CRIU reads). A connection the kernel had accepted but the paused process had
+not made a mark fail (measured under requests to a published port); skipped, its client sees a reset and retries.
+With it, 8 of 8 marks succeeded under steady requests. `setup` installs
+crun, CRIU and podman and checks CRIU. `base` fills the tree, runs the provisioning once in the container, starts
+`world-machine-init` (which runs each `/etc/world-machine/services.d/*.sh` to completion: a service script starts its
+daemon and returns once it listens) and marks it.
+
+Measured on the pieces, not on this code (Volter Harness `container/retake/README.md`):
+- a `next dev` and Postgres froze 1.28 s for a CRIU dump and restored in place in 1.6 s, warm;
+- consecutive marks share 71% of their pages, and zstd takes a mark 6×;
+- a twin replaced across a restore cost one in-flight request.
+
+The container's processes are told io_uring does not exist (seccomp, `ENOSYS`): CRIU cannot dump one, and Node's libuv
+uses it where the kernel offers it. Both loops (the store's image and the tree's) use direct I/O: through the page cache
+every page was cached once per layer, and a large dump waited on writeback. A store `setup --make-store` creates is
+btrfs with `compress-force=zstd:1` where the host has btrfs (an XFS image made before is kept). After each mark,
+`compose.py` rewrites its CRIU page files so that pages identical to the parent's are reflinked from it and only new ones
+are data. The files stay CRIU's own, byte for byte, and a `next dev` mark of 1.3 GB kept 40–131 MB (mean 89 MB). Every
+mark dumps: whether nothing ran since the parent cannot be told exactly (a freeze and thaw retry the processes'
+interrupted syscalls, as a small real wake-up does, and no counter on a VZ guest tells the two apart: it has no PMU to
+count user instructions; Volter Harness `container/retake/README.md`, "a turn in which nothing ran").
+
+**The container base's identity** is the image by its id on the host (a tag rebuilt or pulled anew is a new image), its
+size, the provisioning, the host functions that make it and its dump (`wm_ct_base_fill` with the init it installs,
+`wm_ct_provision`, `wm_ct_start`, `wm_ct_bundle`, `wm_ct_net`, `wm_ct_mark`, as bash prints them, so a comment is no
+input), the `binds` and the slot's addresses. A change to any other host function keeps the base (it hashed all of
+`ct.sh`, and each such change rebuilt it: 49 s measured). With no image id on the host (podman failing, the image
+removed) there is no base name: a machine with a head restores it anyway, one without reads `error` saying so, and a
+collection stops before deleting anything.
+
+**The filled tree** (`tree-<hash>`: the image's tree after the fill, keyed by the image id, the size, the fill and the
+guest's address) is kept in the store. A base rebuilt for anything but those starts from a reflink of it: 1.5 s where
+filling from the image took 12.6 s (a warm `podman export` of the 1.4 GB, 37,000-file image alone was 8.2 s). `base`
+pulls an image from a registry in the background first, since the base is named by its id; the fill never pulls, so a
+registry tag is refreshed only by a pull on the host.
+
+`POST /machine/prepare {"position": N}` (container backend) is a retake's first half, run by the machine of the World
+being left while the World branches. It restores the head a branch forked at `machine@N` will have, pauses it at once
+with its way out closed, and marks the slot as handed over to that snapshot. When the World stops that machine's
+service, the container is left as it is. The branch's machine, asked to restore, finds its own head waiting there
+paused and adopts it: its egress is pointed at the branch's twins, the World's files are written, stale connections
+are reset, and the processes resume, with no second restore. Anything else (no handover, another snapshot, a running
+container) takes the whole restore as before. With `"live": true` nothing is restored or paused: the running machine
+itself is handed over, and `N` must be this ledger's head (a mark just taken). A partial retake uses it (the twins go
+back, the machine stays): its way out is closed and only its connections through the World's forwarders are reset, so
+its processes and the connections into it (the harness's) run on; its calls to a twin are refused until the branch
+adopts it (about a second). A failure opens the way out again. Measured through supercode's `restore_world`:
+
+| | machine | whole retake |
+|---|---|---|
+| before | 0.57–0.81 s | 2.3–2.6 s |
+| adopted, no dev server | 0.14–0.19 s | 1.17–1.25 s |
+| adopted, `next dev` (1.3 GB) in the machine | 0.32 s | 2.4 s |
+
+The branch itself grew under the concurrent restore (2.0 s against 1.0–1.7 s), which is why the dev-server case
+gained less.
+
+Two services operate the one slot during a retake, the machine being left and the branch's. So on the container
+backend every host operation takes the slot's lock (`slot.lock`, `flock`, up to 120 s), except the store-wide gc. The
+checks behind the handover are made inside the same locked operation as the swap:
+- a prepare refuses a slot that is not its machine's;
+- a prepare that fails once the swap began leaves the machine stopped with its way out closed, never the mark running
+  into the World being left;
+- a failed adopt falls through to the whole restore.
+
+Background processes the host starts (forwarders, the publisher, the composer, the base fill) do not inherit the lock's
+descriptor.
+
+Run end to end on 2026-09-25 (Volter Harness `container/retake/README.md`, "the container machine's first run"). A mark
+paused the machine for 0.2–0.6 s, and a whole retake took 2.4–3.0 s, of which the machine took 0.85 s. Files, process
+memory and a twin came back exactly, and a Mac's NFS handles stayed valid. With an `export`, the host's NFS server stops
+its threads for a restore's swap, so a hard-mounted client waits instead of getting an error. That server is the host's,
+so every export on it waits.
+
+## Declaring it
+
+The machine config (paths are on the machine host; `~/` is that host's home; `provision` is beside
+the config on the World's host):
+
+```json
+{
+  "host": { "kind": "lima", "instance": "tm", "limaHome": "~/.lima-tm" },
+  "store": "~/.cache/volter-world-machine",
+  "kernel": "~/fc/vmlinux",
+  "rootfs": "~/fc/rootfs.ext4",
+  "sshKey": "~/fc/id_rsa",
+  "provision": "provision.sh",
+  "vcpus": 2,
+  "memMib": 1024
+}
+```
+
+Optional: `firecracker` and `snapshotEditor` (default `<store>/bin/…`, installed by `setup`),
+`slot` (`default`), `tap` (`wm0`), `hostIp` (`172.16.0.1`), `guestIp` (`172.16.0.2`), `egressPorts`
+(`{"STRIPE": 21000}` to fix a vendor's port) and `room` (`{"paths": [...], "reserveBytes": 2147483648}`). A mark is
+refused with 507 unless the filesystems under the store have room for it: the store, the file backing it when it is a
+loop image, and each of `paths`. Room means the last mark's growth, at least 1 GiB, plus `reserveBytes`. On a Mac, name
+the home the Lima VM mounts: its free space is the Mac's, beneath the VM's own sparse disk. When the lowest layer
+fills, a write fails as an I/O error in every layer above it, which can corrupt the store; a refused mark only goes
+untaken. Not covered: the warm base's own mark, and what the machine's processes write between marks. On Linux
+`host` is `{"kind":"linux"}` (the default).
+The rootfs must authorize `sshKey`'s public half for root and have `python3` (the guest agent).
+
+The service, declared after the twins it routes to (a service sees the `*_TWIN_URL`s of the
+services started before it):
+
+```json
+{
+  "id": "machine", "type": "process",
+  "execution": { "process": { "command": "bun", "args": ["node_modules/@volter/world-machine/src/cli.ts", "serve", "--config", "machine.json"] }, "controlPlane": true },
+  "endpoint": { "port": "auto" },
+  "bindings": { "injectEnv": "MACHINE_URL" }
+}
+```
+
+## Commands
+
+Every command returns in well under 30 s; the one-time base build is a sequence of short steps.
+
+| Command | Does |
+|---|---|
+| `world-machine setup --config <file> [--make-store <size>]` | on the machine host: the `/dev/kvm` udev rule (udev resets the device's mode), socat, the tap, Firecracker v1.17.0 and `snapshot-editor` when absent, and with `--make-store`, where the store cannot reflink already, a loop-mounted XFS image at the store (in fstab). Idempotent |
+| `world-machine base --config <file>` | advances the warm base one step (copy the rootfs, boot, install the agent and start the provisioning script, then the Full snapshot; for the container backend: pull a registry image, fill the tree or reflink the filled one, provision, mark) and says what it waits on; exit 3 until it prints `"ready": true`. It uses the slot, so a running machine is stopped |
+| `world-machine gc --config <file>` | deletes the snapshots, warm bases and filled trees no registered World's branch names, then commits and trims the store's filesystem; prints what it kept, what it deleted and the bytes freed (the store's `df`, before and after) |
+| `world-machine serve --config <file>` | the service; the World adds `--port` and `--root`. It registers its World in the store at once and collects after the machine's first bring-up (or 10 s after start when nothing brings it up), in the background: a first `GET` waited 1.5 s behind a collection deleting two bases |
+
+## Doors
+
+Every door needs the World's door key: `authorization: Bearer <key>`, the 64 hex digits `serve` writes once to
+`.world-machine-token` beside the configuration it is given (`--config`), mode 0600. That file is on the machine's host
+only (never under the tree the machine runs on or a bind of it), and a caller reads it there: supercode-retake from
+the World's app folder, where its `machine.json` is. Loopback is not a boundary here, since the machine's processes
+reach this host's loopback through the World's egress proxy, which forwards local destinations. Every door also refuses (403) a request carrying `Origin`, naming a `Host` other than this listener's loopback address, or
+POSTing a body that is not `application/json`. The port is loopback, but a VM's loopback is forwarded to its host (Lima),
+where a web page could otherwise send a cross-origin POST, or read an answer through a rebound DNS name.
+
+| Door | Answers |
+|---|---|
+| `GET /machine` | `{ state, head: {id,label}\|null, loaded, guest: {ip, sshPort, sshKey, user}, egress: {<VENDOR>: url}, message? }`; `state` is `running`, `displaced` (another World's machine took the slot; a restore takes it back), `no-base` or `error` (for the container backend also when its image has no id on the host), or `idle` before anything asked for it (a `GET` brings it up, and its answer carries `broughtUp` with the restore's facts when it did) |
+| `POST /machine/checkpoint {label, keep?}` | `{ id, pauseMs, memBytes, diskBytes, storeBytes, memWriteMs, diskCloneMs, materializeMs }`; appends one ledger entry |
+| `POST /machine/restore` | `{ id, label, from, loadMs, restoreMs, clockMs, clockErrorMs, clockSteps, staleReset, resetMs, … , totalMs }` |
+| `POST /machine/present {positions}` | `[{ position, id, present }]`: whether each ledger position's snapshot is still in the store (complete, not `.pending`) |
+| `GET /machine/clock` | `{ errorMs, guestRttMs, hostOffsetMs, hostRttMs }`: the guest's wall clock against the World's, now |
+| `POST /machine/compare {position, path?}` | container backend, read only: what a restore to that position would change, `{ files: {back, gone, changed, counts, skipped, partial}, processes: {restarted, stopped, then, now} }`, from the mark's image mounted read-only (`path` defaults to the exported project; regular files and symlinks, bounded at 30,000 files and 15 s) |
+| `POST /machine/run {argv, user?, timeoutMs?}` | container backend: a command run in the running machine (`crun exec`), as `user` (a name the machine's passwd resolves) or its root, bounded by `timeoutMs` (60 s, at most 300 s) with the machine's own `timeout`, which ends what it started there; `{ code, stdout, stderr, timedOut? }`, each output's last 16,000 bytes. A caller composing a service's restore (stop it, `files`, start it) uses it; an error starting `refused:` ran nothing |
+| `POST /machine/files {position, paths}` | container backend: those paths (absolute in the machine, or relative to the exported project) back as they were at the position, the machine running on, `{ paths: [{path, done, changes, sample}] }`; `done` is `restored` (from the mark's image: a folder as it was, what was made in it since removed; a file by its content, never its time; owners, modes and times), `removed` (the mark did not have it) or `absent`. Every path is checked before any is touched (an error starting `refused:` changed nothing); with `noLinks`, a path holding a symlink anywhere under it, in the mark or now, is refused (a data directory whose tablespace or log lives elsewhere); the live tree is written component by component without following a symlink, and a path whose folders are a symlink in either tree is refused. Hard links, ACLs and extended attributes are not carried |
+
+`memBytes` is the dirty memory written, `storeBytes` the store's `df` growth (read after a syncfs:
+XFS frees an unlinked file's blocks in the background) since the last restore or checkpoint, and
+`diskBytes` that growth less `memBytes` (the disk's copy-on-write blocks).
+
+## A retake
+
+At a mark: `POST /machine/checkpoint`, then the World's positions (`volter world log --json`, the
+highest position per service). To go back: `volter world branch <name> --at machine@N,stripe@M`,
+then `POST /machine/restore` on the new branch's machine (its URL is `MACHINE_URL` in the World's
+env). With a frozen World clock, carry the parent's instant with `volter world clock set` first.
+
+## The clock
+
+A restored guest wakes at its snapshot's wall time. The restore moves it through the guest agent: the
+machine host's clock is measured against the World host's (the shortest of five round trips over the
+persistent shell); after one warm-up request (the first request after a load waits on the guest waking, its
+pages faulted in from the snapshot file), three probes measure the guest's error against the target and the
+agent moves its clock by that difference against its own time (`shift`), so the step does not depend on
+when it arrives; probe and shift repeat (at most four times) until the guest reads within 3 ms. With a
+frozen World clock the guest is moved to the frozen instant and runs on from it, so `GET /machine/clock`
+then reads the time since the restore.
+
+## Stale connections
+
+Every established TCP connection between the guest and the host on its link is stale after a restore: its
+far end belongs to the host as it was at the snapshot. A server listening in the guest (a harness's executor,
+sshd) comes back holding the mark's client, which is gone, and refuses or waits on a new one that resumes the
+same session; a client in the guest waits out its own timeout. Once the clock is moved, the guest agent resets
+all of them except its own connection (the kernel's socket destroy over sock_diag netlink, what `ss -K`
+does, in the agent's own process: the guest kernel needs `CONFIG_INET_DIAG_DESTROY`), so each peer sees a reset at once and reconnects. The restore's answer counts
+them (`staleReset`, `resetMs`).
+
+## Retention
+
+A snapshot is kept while any existing branch's machine ledger names it, through the branch's whole view
+(the parent's history up to the fork included); a purged branch's own snapshots go. Every snapshot
+directory is self-contained (its memory and disk are reflink copies, never references into another
+directory), so deleting one never breaks another. The store is shared by every World whose machine runs
+on the host, so each `serve` registers its World (`<store>/roots.json`: the World's `.volter/worlds`
+directory) and a collection reads every registered World's branches, dropping registrations whose
+directory is gone. It deletes snapshot directories no ledger names, except one a checkpoint is still
+writing (`.pending` until the ledger entry is written) and one a running slot's machine was loaded from
+(`slots/<slot>/loaded`, read only while that slot's Firecracker runs; a container slot's `ct-loaded`, written before a
+restore reads it and removed when its machine's service stops); and warm bases no kept snapshot was built on, except
+this configuration's base, a loaded one and one being built. On the container backend a snapshot never reads its base,
+so there only a branch's head keeps the base it was taken on, and a filled tree stays while it is this configuration's
+or a kept base (or base build) was made from it. Then the store is committed and `fstrim` gives the space back to the
+host (a trim before btrfs committed the deletes gave nothing back; on a Mac the Lima VM's root discards online, so the
+space reaches the Mac's sparse disk file at the root's next commit). A ledger it cannot read stops it before anything is deleted. A checkpoint that
+failed half-way leaves its `.pending` directory behind, which a collection keeps.
+
+`retain` (a number, optional) bounds one World's own marks: a collection run by that World's `serve` keeps each
+machine ledger's head snapshot, and of the World's other snapshots the newest `retain` by the host time in their
+id; the rest go even though a ledger names them, and each is listed in `<store>/expired.list`. Other Worlds'
+snapshots are left to their own `serve`. A snapshot another registration's ledger also names is kept, so a mark shared with a branch
+still registered is not counted against `retain`. With `retain` set, `serve` collects at start and after every tenth
+checkpoint. A snapshot being restored (`slots/<slot>/restoring`, written before its presence is checked and
+removed once the restore returns) is kept like a loaded one. A restore or prepare of a snapshot that is gone is
+refused before anything stops, naming whether it was let go past `retain` or is missing; `POST /machine/present`
+tells a caller beforehand which positions can still be restored.
+
+`thin` (true or false, optional, needs `retain`) thins with age instead of cutting at `retain`: past the newest
+`retain`, one snapshot of each calendar hour stays for a day and one of each UTC day for a month (`src/thin.ts`). The
+one kept is each bucket's oldest, so a newer mark never displaces it: a snapshot that stayed stays until its tier
+ends, and an hour's keeper goes only as it crosses into the day tier, unless it is its day's oldest. A checkpoint taken
+with `keep` (`POST /machine/checkpoint {label, keep: true}`: a mark the agent pinned) is never let go, the newest 20
+such; they count among the newest `retain` like any other.
+
+## Not done
+
+- One slot per machine host: two Worlds' machines share it (the later restore wins; the other reads
+  `displaced`).
+- `diskBytes` is derived from `df`, not from the disk's own changed extents.
+
+## Measured (2026-09-24): on a loaded box, so readings, not results
+
+Surface: this package's own doors (`POST /machine/checkpoint`, `POST /machine/restore`,
+`GET /machine/clock`) and the product CLI (`volter world branch --at`, `world log --json`) on a World
+of the Stripe twin and this service. Apple M3 Pro, macOS 26.4; Lima 2.2.0 VM (`vz`, nested
+virtualization, 4 CPU / 6 GiB, Ubuntu, kernel 7.0); Firecracker v1.17.0; CI guest kernel 6.18 and
+Ubuntu 24.04 rootfs (2 vCPU / 1 GiB, 2 GiB ext4 disk); the store a 12 GiB loop-mounted XFS image.
+**The Mac was not quiet during any reading**: load average 33-90 and 80,000-240,000 page
+compressions per 2 s (`vm_stat`), from other work on the box. Every timing below is therefore a
+reading under that load, not a result; the quiet-box numbers the acceptance asks for were not taken.
+
+| What | Target | Read (loaded) |
+|---|---|---|
+| checkpoint pause | < 200 ms | 61-134 ms over six checkpoints (memory write 23-68 ms, disk clone 20-56 ms); 202-604 ms before the diff went to `/dev/shm` |
+| after the pause | | materialize 95-186 ms |
+| storage per checkpoint | ≈ changed disk + dirty memory | 11-13 MB of dirty memory per turn-sized checkpoint, 1.7-2.2 MB after 5-10 s idle; disk 140-200 KiB (store growth after syncfs; `filefrag` counted 284 KiB of changed disk blocks in one) |
+| restore load | < 100 ms | `snapshot/load` 17-69 ms; the whole in-place replacement (process, disk clone, load) 67-250 ms |
+| restore, answered | | 0.72-1.6 s; the guest's first reply after the load (memory faulted in from the file, through nested virtualization, on a compressing host) takes 0.2-1.7 s of it |
+| guest clock after restore | within ~50 ms | -1.0 to -2.9 ms (World host to machine host round trip 4-14 ms bounds the offset to half of it) |
+| a whole retake (`world branch --at` + restore) | < 1 s | 6.3 s: branch 4.9 s (the machine's start in it 1.1 s), restore 1.4 s; 8.5-10 s before the restore was trimmed |
+
+A dev server in the guest (pid 364) came back from every restore with the same pid and startup nonce
+and its in-memory notes as of the mark (`['1', '2', 'b1']` after notes `b2`, `b3` were written and a
+retake went to `b1`), and `/work/proof` held the mark's rows. The World branch held the Stripe
+customers up to the mark. The guest reached the twin only through its forwarder (`200`); an outside
+address did not answer and the guest has no default route.
+
+The lab's guest clock came back a constant 6 s short. `ssh guest date -s @$(date)` reads the time
+before the ssh login, so the guest lands behind by the login's duration: here the same command left
+the guest 1,534 ms behind with a 1.5 s login under load, where this service's step had read -1.4 ms
+a moment before. (That the lab's 6 s was its login time is inferred from this, not measured there.)

package/dist/src/agent.py

@@ -0,0 +1,92 @@
+# world-machine-agent: runs inside the guest, started once when the warm base is built, so it is in
+# every snapshot. It answers the machine host on the guest's one link: it steps the wall clock to the
+# instant it is sent (a restored guest wakes at its snapshot's wall time), resets the connections a restore
+# left stale and writes the World's env file. One JSON object per line each way.
+import json, os, socket, socketserver, struct, time
+
+# After a restore every established TCP connection to the host on this link is stale by construction: its far
+# end belongs to the host as it was at the snapshot (a server listening here keeps the mark's client attached, a
+# client here waits out its own timeout). reset_stale resets each (the kernel's sock_destroy, what `ss -K` does:
+# CONFIG_INET_DIAG_DESTROY), the agent's own connection excepted (its port serves one connection at a time), over
+# sock_diag netlink in this process: measured, spawning `ss` from a just-restored guest took 1.0-2.3 s.
+NETLINK_SOCK_DIAG, SOCK_DIAG_BY_FAMILY, SOCK_DESTROY = 4, 20, 21
+NLM_F_REQUEST, NLM_F_ACK, NLM_F_DUMP, NLMSG_ERROR, NLMSG_DONE = 1, 4, 0x300, 2, 3
+TCP_ESTABLISHED = 1
+
+def _diag(family, states, sockid=b"\0" * 48):
+    return struct.pack("=BBBBI", family, socket.IPPROTO_TCP, 0, 0, states) + sockid
+
+def _talk(nl, kind, flags, payload, seq):
+    nl.send(struct.pack("=IHHII", 16 + len(payload), kind, flags, seq, 0) + payload)
+
+def reset_stale(host, own_port):
+    peer = socket.inet_aton(host)
+    nl = socket.socket(socket.AF_NETLINK, socket.SOCK_RAW, NETLINK_SOCK_DIAG)
+    try:
+        _talk(nl, SOCK_DIAG_BY_FAMILY, NLM_F_REQUEST | NLM_F_DUMP, _diag(socket.AF_INET, 1 << TCP_ESTABLISHED), 1)
+        stale, done = [], False
+        while not done:
+            data, off = nl.recv(65536), 0
+            while off + 16 <= len(data):
+                length, kind = struct.unpack_from("=IH", data, off)
+                if kind == NLMSG_DONE: done = True; break
+                if kind == NLMSG_ERROR: raise OSError(-struct.unpack_from("=i", data, off + 16)[0], "sock_diag dump failed")
+                sockid = data[off + 20:off + 68]  # inet_diag_msg: family, state, timer, retrans, then the id
+                sport = struct.unpack_from("!H", sockid, 0)[0]
+                if sockid[20:24] == peer and sport != own_port: stale.append(sockid)
+                off += (length + 3) & ~3
+        for i, sockid in enumerate(stale):
+            _talk(nl, SOCK_DESTROY, NLM_F_REQUEST | NLM_F_ACK, _diag(socket.AF_INET, 1 << TCP_ESTABLISHED, sockid), 2 + i)
+            data = nl.recv(65536)
+            err = -struct.unpack_from("=i", data, 16)[0]
+            if err and err != 2: raise OSError(err, "sock_destroy: %s" % os.strerror(err))  # ENOENT: it closed meanwhile
+        return len(stale)
+    finally:
+        nl.close()
+
+class Handler(socketserver.StreamRequestHandler):
+    def handle(self):
+        accepted = time.perf_counter()
+        setup_ms = round((accepted - Server.accepted_at) * 1000, 2)  # from accept() returning to this handler running
+        for line in self.rfile:
+            got = time.perf_counter()
+            try:
+                req = json.loads(line)
+                op = req.get("op")
+                if op == "clock":
+                    time.clock_settime(time.CLOCK_REALTIME, float(req["t"]))
+                    res = {"t": time.time()}
+                elif op == "shift":
+                    # moved by an amount against its own clock, read here: the step does not depend on when it arrives
+                    time.clock_settime(time.CLOCK_REALTIME, time.time() + float(req["by"]))
+                    res = {"t": time.time()}
+                elif op == "reset":
+                    res = {"reset": reset_stale(self.client_address[0], self.server.server_address[1])}
+                elif op == "now":
+                    res = {"t": time.time()}
+                elif op == "write":
+                    path = req["path"]; tmp = path + ".tmp"
+                    os.makedirs(os.path.dirname(path) or "/", exist_ok=True)
+                    with open(tmp, "w") as f: f.write(req["content"])
+                    os.replace(tmp, path)
+                    res = {"ok": True}
+                else:
+                    res = {"error": "unknown op %r" % op}
+            except Exception as e:
+                res = {"error": str(e)}
+            # where a reply's time goes inside the guest: waiting for the line since the connection was accepted, and
+            # handling it (a first request after a load faults this process's pages in from the snapshot file)
+            res["setupMs"] = setup_ms; setup_ms = 0.0
+            res["waitMs"] = round((got - accepted) * 1000, 2); res["handleMs"] = round((time.perf_counter() - got) * 1000, 2)
+            accepted = time.perf_counter()
+            self.wfile.write((json.dumps(res) + "\n").encode()); self.wfile.flush()
+
+# One connection at a time, on the serving thread: the first request after a load pays for every guest
+# page it touches (faulted in from the snapshot file), and a thread per connection touched more.
+class Server(socketserver.TCPServer):
+    allow_reuse_address = True
+    accepted_at = 0.0
+    def get_request(self):
+        r = super().get_request(); Server.accepted_at = time.perf_counter(); return r
+
+Server(("0.0.0.0", int(os.environ.get("WM_AGENT_PORT", "7070"))), Handler).serve_forever()

package/dist/src/agentc.py

@@ -0,0 +1,81 @@
+# The machine host's client for the guest agent (agent.py). Clock arguments are seconds:
+# OFFSET is this host's clock minus the World host's clock, DELTA the World clock minus the World
+# host's clock (0 unless the World's clock is frozen). The target instant is therefore
+# time.time() - OFFSET + DELTA, read the moment before it is sent.
+import json, socket, sys, time
+
+ip, port, op = sys.argv[1], int(sys.argv[2]), sys.argv[3]
+started = time.time()
+deadline = time.time() + 3
+while True:
+    try:
+        s = socket.create_connection((ip, port), timeout=1); break
+    except OSError as e:
+        if time.time() > deadline: print('@' + json.dumps({"error": "guest agent unreachable at %s:%d: %s" % (ip, port, e)})); sys.exit(1)
+        time.sleep(0.005)
+# the first request after a load can wait on guest memory faulted in from the file
+s.settimeout(10); f = s.makefile("rwb")
+connectMs = round((time.time() - started) * 1000, 1)
+
+def call(req):
+    f.write((json.dumps(req) + "\n").encode()); f.flush()
+    return json.loads(f.readline())
+
+def probe(offset, delta, n=5):
+    best = None
+    for _ in range(n):
+        h0 = time.time(); g = call({"op": "now"})["t"]; h1 = time.time()
+        err = g - ((h0 + h1) / 2 - offset + delta)
+        if best is None or h1 - h0 < best[1]: best = (err, h1 - h0)
+    return best
+
+def write_files(files, out):
+    for path, content in files.items():
+        res = call({"op": "write", "path": path, "content": content})
+        if "error" in res: out["error"] = "guest agent: writing %s: %s" % (path, res["error"]); return False
+    return True
+
+if op in ("clock", "now"):
+    offset, delta = float(sys.argv[4]), float(sys.argv[5])
+    out = {}
+    # files to write on the same connection: one path, its content on stdin, or `-` and a JSON object of path ->
+    # content on stdin
+    files = (json.load(sys.stdin) if sys.argv[6] == "-" else {sys.argv[6]: sys.stdin.read()}) if len(sys.argv) > 6 else {}
+    # the restore id first, the first request after the load: a restored connection the guest drops while the clock
+    # is stepped (a second or more under load) must already find the new id
+    rid = [k for k in files if k == "/run/world-machine/restore-id"]
+    if rid and not write_files({rid[0]: files.pop(rid[0])}, out): print('@' + json.dumps(out)); sys.exit(0)
+    if op == "clock":
+        # The first request after a load waits on the guest waking (measured: its own handling 11 ms of a 288 ms
+        # reply); it is only the warm-up.
+        t0 = time.time(); first = call({"op": "now"}); before = first["t"]
+        out["connectMs"] = connectMs; out["firstReplyMs"] = round((time.time() - t0) * 1000, 1)
+        out["firstHandleMs"] = first.get("handleMs"); out["firstWaitMs"] = first.get("waitMs"); out["firstSetupMs"] = first.get("setupMs")
+        # Measure the guest's clock against the target over warm round trips, then move it by the difference,
+        # relative to its own time (op shift): unlike stepping to an instant sent ahead, the step does not land late
+        # by however long the request takes to be read (the first requests after a load wait on guest pages).
+        steps = 0
+        while True:
+            err, rtt = probe(offset, delta, 3)
+            if abs(err) <= 0.003 or steps == 4: break
+            call({"op": "shift", "by": -err}); steps += 1
+        out["clockMs"] = round((time.time() - t0) * 1000, 1)
+        out["steps"] = steps
+        out["steppedByMs"] = round(((time.time() - offset + delta) - before) * 1000)
+        # the files before the reset: a program whose connection the reset ends reads the new restore id (and the
+        # World's env) when it reconnects
+        if not write_files(files, out): print('@' + json.dumps(out)); sys.exit(0)
+        files = {}
+        # then the connections the restore left stale, once the clock reads now (a reset peer reconnects at once)
+        r0 = time.time(); res = call({"op": "reset"})
+        if "error" in res: print('@' + json.dumps({"error": "guest agent: resetting the stale connections: " + res["error"]})); sys.exit(1)
+        out["staleReset"] = res["reset"]; out["resetMs"] = round((time.time() - r0) * 1000, 1)
+    err, rtt = probe(offset, delta)
+    out.update({"errorMs": round(err * 1000, 2), "probeRttMs": round(rtt * 1000, 2)})
+    write_files(files, out)
+    print('@' + json.dumps(out))
+elif op == "write":
+    res = call({"op": "write", "path": sys.argv[4], "content": sys.stdin.read()})
+    print('@' + json.dumps(res))
+else:
+    print('@' + json.dumps({"error": "unknown op " + op})); sys.exit(1)

package/dist/src/base.d.ts

@@ -0,0 +1,9 @@
+import type { MachineConfig } from './config.js';
+import { type HostSession } from './host.js';
+export type BaseStep = {
+    base: string;
+    stage: string;
+    ready: boolean;
+    detail?: string;
+};
+export declare function stepBase(config: MachineConfig, session: HostSession): Promise<BaseStep>;

package/dist/src/base.js

@@ -0,0 +1,61 @@
+// The warm base: the declared guest image, cold-booted once, provisioned by the declared script, and
+// snapshotted whole, in a store-wide cache keyed by everything that shapes it. A World's machine never
+// cold-boots (20-60 s measured); it loads this or a snapshot. The build is a sequence of short steps,
+// each run by one `world-machine base` call that advances as far as it can without waiting and says
+// what it is waiting on; the caller runs it again.
+import { readFileSync } from 'node:fs';
+import { GUEST_AGENT, GUEST_PROFILE } from "./host.js";
+import { baseHash, bootArgs, hp } from "./machine.js";
+import { q } from "./host.js";
+export async function stepBase(config, session) {
+    await session.call('wm_reflink_check');
+    const hash = await baseHash(config, session);
+    const base = `base-${hash}`;
+    const B = `"$WM_STORE/${base}"`;
+    const W = `"$WM_STORE/${base}.building"`;
+    const has = async (test) => (await session.call(`if ${test}; then echo '@{"ok":true}'; else echo '@{"ok":false}'; fi`)).ok;
+    const provision = config.provision ? readFileSync(config.provision, 'utf8') : '';
+    if (await has(`[ -f ${B}/vmstate ]`))
+        return { base, stage: 'ready', ready: true };
+    // 1. the rootfs, copied once into the store (it may not be on a reflink filesystem), in the background
+    if (!(await has(`[ -f ${W}/copied ]`))) {
+        // a copy that is no longer running (the host restarted, the store was remounted) is started again
+        if (!(await has(`[ ! -f ${W}/copying ] || kill -0 "$(cat ${W}/copy.pid 2>/dev/null)" 2>/dev/null`)))
+            await session.call(`rm -f ${W}/copying ${W}/disk.ext4; echo '@{"ok":true}'`);
+        if (!(await has(`[ -f ${W}/copying ]`))) {
+            // a slot builds one base at a time: another base left half-built by this slot (its inputs moved on) is abandoned,
+            // and goes before this one starts; one with no slot recorded predates the record and is taken as this slot's
+            await session.call(`for d in "$WM_STORE"/base-*.building; do [ -d "$d" ] && [ "$d" != ${W} ] || continue; s=$(cat "$d/slot" 2>/dev/null || echo "$WM_SLOT"); [ "$s" = "$WM_SLOT" ] && rm -rf "$d"; done; echo '@{"ok":true}'`);
+            await session.call(`mkdir -p ${W} && printf '%s' "$WM_SLOT" > ${W}/slot && : > ${W}/copying && (setsid bash -c "echo \\$\\$ > ${W.replace(/"/g, '\\"')}/copy.pid; cp --sparse=always ${hp(config.rootfs).replace(/"/g, '\\"')} ${W.replace(/"/g, '\\"')}/disk.ext4 && : > ${W.replace(/"/g, '\\"')}/copied" > ${W}/copy.log 2>&1 < /dev/null &) ; echo '@{"ok":true}'`);
+            return { base, stage: 'copying the rootfs into the store', ready: false };
+        }
+        return { base, stage: 'copying the rootfs into the store', ready: false, detail: 'run again' };
+    }
+    // 2. a cold boot in the slot (the snapshot records the slot's disk path and tap)
+    if (!(await has(`[ -f ${W}/booted ]`))) {
+        await session.call(`wm_net >/dev/null && wm_boot ${hp(config.kernel)} ${W}/disk.ext4 ${config.vcpus} ${config.memMib} ${q(bootArgs(config))} && wm_own '{"token":"base-build"}' && : > ${W}/booted`);
+        return { base, stage: 'booting the guest', ready: false, detail: 'run again once it answers ssh' };
+    }
+    // 3. the guest agent and the provisioning script, started in the guest
+    if (!(await has(`[ -f ${W}/provisioning ]`))) {
+        if (!(await has('WM_GSSH_TIMEOUT=6 wm_gssh true')))
+            return { base, stage: 'waiting for the guest to answer ssh', ready: false };
+        await session.call([
+            `wm_gssh 'mkdir -p /usr/local/sbin && cat > /usr/local/sbin/world-machine-agent' <<'__WM_AGENT__'\n${GUEST_AGENT}\n__WM_AGENT__`,
+            `wm_gssh 'cat > /etc/profile.d/volter-world.sh' <<'__WM_PROFILE__'\n${GUEST_PROFILE}\n__WM_PROFILE__`,
+            `wm_gssh 'cat > /root/.world-machine-provision.sh' <<'__WM_PROVISION__'\n${provision}\n__WM_PROVISION__`,
+            `wm_gssh 'setsid nohup python3 /usr/local/sbin/world-machine-agent > /var/log/world-machine-agent.log 2>&1 < /dev/null & setsid nohup bash -c "bash /root/.world-machine-provision.sh > /root/.world-machine-provision.log 2>&1; echo \\$? > /root/.world-machine-provision.done" > /dev/null 2>&1 < /dev/null &' || return 1`,
+            `: > ${W}/provisioning; echo '@{"ok":true}'`,
+        ].join('\n'));
+        return { base, stage: 'provisioning', ready: false, detail: 'run again' };
+    }
+    // 4. provisioning finished, and finished well
+    const done = await session.call(`printf '@{"code":"%s","log":%s}\\n' "$(wm_gssh 'cat /root/.world-machine-provision.done 2>/dev/null')" "$(wm_gssh 'tail -c 600 /root/.world-machine-provision.log 2>/dev/null' | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')"`);
+    if (done.code === '')
+        return { base, stage: 'provisioning', ready: false, detail: done.log.trim().split('\n').pop() };
+    if (done.code.trim() !== '0')
+        throw new Error(`world-machine base: the provisioning script exited ${done.code.trim()}:\n${done.log}\n(remove ${config.store}/${base}.building to start over)`);
+    // 5. the Full snapshot: memory, device state and disk, then the slot is freed
+    const snap = await session.call(`wm_agent now 0 0 >/dev/null || return 1\nwm_snapshot ${W} Full || return 1\nwm_fc_stop; wm_release\nrm -f ${W}/copying ${W}/copied ${W}/booted ${W}/provisioning ${W}/copy.log\nmv ${W} ${B}`);
+    return { base, stage: 'ready', ready: true, detail: `full snapshot in ${snap.pauseMs} ms, ${Math.round(snap.memBytes / 1048576)} MiB of memory` };
+}

package/dist/src/cli.d.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+/** The doors' key, a file beside the machine's configuration (serve). */
+export declare const DOOR_TOKEN_FILE = ".world-machine-token";