@tikoci/quickchr 0.4.8 → 0.5.1

package/DESIGN.md

@@ -97,6 +97,19 @@ Modules (src/lib/)      ← qemu, images, versions, network, state, ...
 
     ⚠️ **Testing note.** `Bun.serve` **drops an explicit `content-length` when the body is a `ReadableStream`** and frames the response `transfer-encoding: chunked`, so a stub built on it cannot exercise the size-derived budget at all — every response looks unknown-length. The anchor tests use a raw `Bun.listen` HTTP stub for that reason.
 
+12. **Two locks, one primitive** (`src/lib/lock.ts`; #140, #181, #180).
+    - **Machine lock** — `machines/<name>/.lock/`, so it goes with the machine. One mutex for every lifecycle operation: `add`, `start`, relaunch, `setDeviceMode`, `stop`, `clean`, `remove`, orphan removal. It was `.start-lock` and taken only by starts, so `stop`/`clean`/`remove` raced every relaunch. For one release a live `.start-lock` is still honoured; a dead one is ignored, not deleted.
+    - **Allocation lock** — `<dataDir>/.allocate.lock/`, held across read → probe → claim when a machine is created. Without it two creations probe the same free ports and pick the same block. The claim itself has to outlive the lock, because `start()` persists `machine.json` only after the download and the spawn, so the creator's machine lock carries `{ portBase, macs }` as a reservation and `listReservations()` reads it. A reservation ends with its lock, and dies with a crashed creator's pid.
+    - **Who waits.** `stop`/`clean`/`remove` wait, bounded by the machine's boot budget, so a `stop` issued during a `start` runs after it. A second `start`, `setDeviceMode` and orphan removal fail fast with `MACHINE_LOCKED`: waiting would only boot the machine twice, or wait out a create that proves the directory is not an orphan. Nobody waits for a `foreground` holder, which keeps the lock until its VM exits. `remove` has no override for a live holder: deleting a directory under a live spawn is the bug the lock exists to prevent. A broken machine has no live holder, so the lock never blocks clearing it.
+    - **Not reentrant.** Two async operations in one process conflict as much as two processes do, so a lock held by this pid counts as held. Code inside a locked operation calls `stopUnderLock()`/`removeUnderLock()`/`cleanUnderLock()`, never the public `ChrInstance` methods.
+    - **State is re-read under the lock.** A handle's snapshot predates whatever the previous holder did, and a stale `state.pid` is a process that has exited or one the OS has reused.
+    - **An append-only log, so nothing is deleted from under a holder.** A lock is a directory of numbered records, and the highest decides the state: a live holder means held; a released marker, a dead holder's record or nothing means free. Acquiring creates record top+1 with `link()`, so exactly one contender wins and the file never exists half-written. Releasing appends a released marker, and records below the top are collected. A claim stands only if nothing higher exists and its predecessor is still there, which turns away a contender that slept through other claims, or through a remove and re-create of the machine.
+    - **Why not delete a stale lock?** A path has no compare-and-delete. Every deleting recovery tried here had a window between judging a holder dead and deleting its file, in which a successor could be installed and then deleted: rename-aside-and-compare, then a reap guard recovered by age. The Codex review of #190 reproduced both. Taking over a dead holder is now just claiming the next number. A cross-process stress run (6 processes, 360 critical sections, about 15% simulated crashes taken over mid-run) showed no overlap.
+    - **Records are named for the lock's incarnation, so a log can be deleted.** A machine's lock lives in `machines/<name>/` and goes with it. A machine can be removed and created again under the same name while some process is paused holding paths into the old lock, so every record is named `<id>.<gen>`, with an id published once per lock directory. A stale process cannot reach the new incarnation's records: not with a claim, a collect, a rejected-claim cleanup or a released marker. Codex reproduced two such cases while records were named by number alone: a claim matching the new record 1, and a collect run after the released marker that deleted the new holder's record. Before that, the logs were briefly pushed out to a never-deleted `locks/<name>/`, which kept a file for every name ever used.
+    - **A claim stands only on what it judged, and a holder cleans up while it still holds.** After creating its record, a claimer checks three things: nothing higher exists, its predecessor is byte-for-byte the record it judged free, and its id is still the directory's incarnation. Release collects old records *before* publishing its marker, because from that moment it is no longer the holder and nothing may be deleted by path. A released lock therefore keeps two records: the last holder's record and its marker.
+    - **`remove()` renames, then deletes, with no in-place fallback.** `removeMachine()` moves `machines/<name>/` to `<dataDir>/.removing/` in one step before deleting it. A recursive delete in place has a window where `.lock/` is gone but the directory is not, and a waiter or same-named create could take a fresh lock in a directory being deleted. A rename that fails is retried for about 2 s, which covers a scanner holding a handle on Windows. After that, `remove` fails with `STATE_ERROR` and leaves the directory in place, rather than falling back to the unsafe delete (Codex review of #190). Once the rename succeeds the name is free, so a failed delete only warns, giving the staged path. Each removal sweeps `.removing/` for entries left by a delete that failed or a process that stopped after its rename, skipping any whose pid is still alive in another process. A waiter never re-creates a removed machine's directory, because the lock directory is created with a non-recursive `mkdir`: it gets `MACHINE_NOT_FOUND`. `assertSameMachine()` catches one that waited out a remove and re-create.
+    - **Unreadable means held.** A record or legacy `.start-lock` that cannot be read, or that does not parse and is younger than 500 ms, blocks. It is never treated as absent.
+
 ## Port Layout
 
 | Offset | Service    | Guest Port |
@@ -131,7 +144,8 @@ Current consumers are all first-party tikoci projects (centrs, donny, restraml),
 ├── machines/
 │   └── 7.22.1-arm64-1/
 │       ├── machine.json       # Config + state
-│       ├── disk.img           # Working copy
+│       ├── boot.qcow2         # Boot disk, qcow2 — or disk.img when raw (see below)
+│       ├── disk<N>.qcow2      # Extra disks
 │       ├── efi-vars.qcow2     # UEFI vars (arm64; qcow2 so savevm works — #31)
 │       ├── monitor.sock       # QEMU monitor
 │       ├── serial.sock        # Serial console
@@ -143,12 +157,60 @@ Current consumers are all first-party tikoci projects (centrs, donny, restraml),
     └── boot-failure-<machine>-<iso8601>.json
 ```
 
+**One boot disk per machine.** A raw machine boots `disk.img`, a copy of the cached
+image. A qcow2 machine boots `boot.qcow2`, converted straight from the cached image
+into a `.tmp` file, checked to be a standalone qcow2 (no backing file), then renamed
+into place. Machines created before 0.5.1 also kept the raw `disk.img` the qcow2 came
+from: a full-size copy (128 MiB x86) that nothing reads. Restart no longer requires it
+(#200). `clean` removes it, and nothing else deletes it implicitly (#199). The shared
+cache image stays: `clean` and a failed first boot's rollback rebuild from it.
+
+A missing disk is rebuilt only while the machine has never booted. A booted machine's
+disks hold guest state, so a missing `boot.qcow2`, `disk.img` or extra disk is
+`MACHINE_NOT_FOUND`, never a silent rebuild from factory. The way out is to restore the
+file or run `quickchr clean`.
+
+**Storage admission (#199).** An operation that writes new data (creating a machine,
+an image or package download) is admitted when **each filesystem it writes to** has
+`estimate + storage-reserve` free there. It replaced a flat 10 GiB floor that refused
+small hosts which had room. The operation's writes are parts with paths (boot disk →
+`machines/`, downloads → the cache it writes to), grouped by `stat().dev`. The estimate
+is a per-stage constant (`STORAGE_ESTIMATES` in `storage.ts`), rounded up from one
+host's measurements and written down there. The reserve (default 2 GiB) absorbs the
+estimate's error and keeps the host usable. Restart is never gated. Virtual disk
+capacity is not charged: a sparse disk's growth is the guest's, after admission.
+
+Under pressure, eligible cache entries are evicted, but only from a cache on the short
+filesystem, and only when everything eligible could cover the gap. Free space is then
+**measured again**, in passes, until the measurement admits or nothing more can be
+evicted. Logical sizes are not proof of room (sparse files, APFS clones).
+
+**Coordination** (`storage-lease.ts`, after Codex's review of #203 reproduced both
+races). One data-dir lock, held for milliseconds: a check-and-record, or an eviction
+pass. Under it, an operation records a **lease** naming the cache entries it is about
+to read (pins) and the bytes it is about to write, per filesystem. Admission subtracts
+other live leases' bytes from measured free space, so two creators cannot spend the
+same bytes. Eviction skips pinned entries. A pin is recorded before the reader checks
+the entry exists, so an eviction either sees it or has already finished, and then the
+reader fetches the entry again. A create's lease lasts until its `machine.json` is
+saved, which protects the image from then on. A download's lease lasts the download.
+A dead pid's lease is ignored and deleted. A create's own downloads are checked up
+front, so it refuses early, but are reserved only by the download itself, so they are
+not counted twice. Automatic eviction also skips anything used in the last 30
+minutes (a cache hit bumps its mtime). That rule is not what protects a reader; pins
+are. It spares what a finished `cache add` fetched for a later process.
+
+The cache cap (`cache-max-size`) is retention, separate from admission. It counts
+images, package archives with their extractions (one entry: the extraction is only a
+hit while its archive exists) and interrupted downloads. Files quickchr did not write
+are reported and never deleted. A failed delete is reported as failed, not freed.
+
 **Global settings** live separately, under the XDG **config** tier (not the data tree
 above): `~/.config/quickchr/quickchr.env`, dotenv-style (`QUICKCHR_KEY=value` lines).
 Managed via `quickchr settings get|set|print|reset`. Precedence per key: CLI flag >
-`QUICKCHR_<KEY>` env var > `quickchr.env` > built-in default. The 6 managed keys:
-`default-channel`, `default-arch`, `accel`, `cache-max-size`, `timeout-extra`,
-`secure-login`.
+`QUICKCHR_<KEY>` env var > `quickchr.env` > built-in default. The 7 managed keys:
+`default-channel`, `default-arch`, `accel`, `cache-max-size`, `storage-reserve`,
+`timeout-extra`, `secure-login`.
 See MANUAL.md's CLI reference and environment-variables sections for the full surface.
 
 ## Platform Support
@@ -216,9 +278,10 @@ with a layered design:
 - **`release.yml`** — one-click publish from committed release state: freshness gate
   (no suite re-run — main is kept continuously release-able) + `package.json` version
   with a matching non-empty CHANGELOG section → tag + GitHub Release + `npm publish
-  --provenance` (odd minor → `next`, even → `latest`; a stable release is then also
-  tagged `next`, since there is no separately maintained pre-release line and `next`
-  must not point at something older than `latest`). CI never bumps versions or pushes
+  --provenance` under the dispatch-time `npm-tag` (default `next`; promoted to `latest`
+  after QA — not derived from the version number; a `latest` release is also tagged
+  `next`, since there is no separately maintained pre-release line and `next` must not
+  point at something older than `latest`). CI never bumps versions or pushes
   to protected `main`.
 - **`ros-versions.yml`** — daily: new RouterOS versions (per channel) with no linux-x86
   record in `ci-data/tested-versions.json` ride the `routeros-targets` matrix of a
@@ -579,13 +642,57 @@ Always read back what we write. One extra REST API call after a provisioning act
 
 ### Provisioning Scope
 
-quickchr provisions at first boot and (optionally) on restart:
+quickchr provisions at a machine's **first boot** — see "The provisioning window"
+below for why, and for what happens to an option that arrives later:
 - **User creation** — create user, set password, optionally disable admin
 - **Package install** — SCP `.npk` files, reboot to activate
 - **License** — `/system/license/renew` for trial
 - **Device-mode** — `/system/device-mode/update mode=rose container=yes ...` for restricted features (containers, traffic-gen, routerboard). Opt-in: not configured unless explicitly requested via CLI `--device-mode` or API `deviceMode` option. CHR ships with `mode=advanced` which is sufficient for most use cases. Device-mode requires a hard QEMU power-cycle to confirm changes — this is the MikroTik-mandated confirmation mechanism (physical button press on real hardware, cold reboot on VM). The wizard defaults to `rose` when the user opts in, since it enables containers. See: https://help.mikrotik.com/docs/spaces/ROS/pages/93749258/Device-mode
 - **Config import** — planned: load `.rsc` or `.backup` at creation time
 
+#### The provisioning window — and why `lastStartedAt` could not be the gate
+
+Every provisioning step assumes the guest holds RouterOS' default configuration, so
+provisioning runs in a window that closes at **first boot** — not at `add`. A machine
+created with no provisioning flags can still take them on its first `start`, which is
+the behaviour agents actually rely on.
+
+The gate was `!state.lastStartedAt`, and that field does not mean what the gate needed.
+It is stamped immediately after `spawnQemu`, before the guest is known to have booted
+and before provisioning runs, so it records "QEMU has been launched" rather than "the
+guest has been configured". The two agree most of the time, which is why the
+conflation survived — and come apart in three ways, each a defect (#176):
+
+- **`clean()`** replaces the disk with the factory image but left `lastStartedAt`
+  behind, so a genuinely fresh guest could never be provisioned again. `clean()` now
+  clears the gate along with the credentials it already cleared, keeping `cleanedAt`
+  for the forensic clue. A cleaned machine's next start re-applies the retained
+  provisioning *intent* and costs what a first boot costs.
+- **Provisioning that throws after a successful boot** leaves a running,
+  half-provisioned machine with the gate closed. The record is stamped only on
+  success, so this case is at least *visible* now; making it retryable — re-running
+  only the steps that did not land — is the remaining half of #176 and is what
+  `provisioning.steps` exists to support.
+- **A first start with no provisioning options** closed the gate although nothing had
+  been configured.
+
+`MachineState.provisioning` is the positive record: when provisioning last completed
+and which steps ran. `isProvisioningWindowOpen()` reads it and falls back to
+`lastStartedAt` for machines that predate the field, so no existing machine
+re-provisions itself on its next start. This is the desired-config/runtime-state split
+of #58, applied to the one field where the conflation was already producing bugs
+rather than to `machine.json` wholesale.
+
+**Nothing may drop a provisioning option in silence.** `start()` used to pass
+`undefined` in place of the caller's options on every path but a first boot, so all
+seven were discarded with no warning, no error and a normal-length boot — the only way
+to notice was to read the value back. An option outside the window now throws
+`PROVISIONING_WINDOW_CLOSED`, naming each option and the route that still works
+(`quickchr set <name> --license`, the library's `setDeviceMode()` / `installPackage()`,
+or `clean()` for anything else). An option state already records is treated as a
+no-op, so passing the same flags on every start — the shape a script naturally takes —
+is not an error.
+
 Provisioning via REST API is preferred (simple HTTP calls). Serial console provisioning (prompt detection + buffer tracking, as in chr-armed) is a fallback for locked environments. Key lessons from chr-armed serial work: use `\r` not `\r\n` on PTY; accumulate buffer with offset tracking to prevent re-matching; detect prompts dynamically, don't use fixed delays.
 
 ### Exec Transport Design
@@ -946,12 +1053,88 @@ Detaching does not weaken cleanup, because nothing that sweeps QEMU sweeps by gr
 is worth stating rather than assuming: "the child outlives its parent's signal" and "a
 wedged run leaks VMs" would otherwise look like the same change.
 
-The naming half is not fixed and should not be fixed quietly. `--bg` is already the
-default and only ever meant "serial console goes to a log file instead of stdio" — it does
-not make `start` return early, because `start` waits for REST readiness by contract. Both
-an external agent and the maintainer believed it bought them something. The honest repair
-is a separate `--no-wait` that opts out of the wait, not a redefinition of `--bg` that
-would change what existing scripts get; #159 holds that half.
+The naming half was not fixed quietly. `--bg` is already the default and only ever meant
+"serial console goes to a log file instead of stdio" — it does not make `start` return
+early, because `start` waits for readiness by contract. Both an external agent and the
+maintainer believed it bought them something. The repair (#159, 0.5.0) is a separate flag
+rather than a redefinition of `--bg` that would change what existing scripts get:
+
+- **`start --no-wait`** (`StartOptions.wait: false`) returns once QEMU is confirmed
+  running — past `spawnQemu()`'s 1.5 s early-exit check, with state and ports persisted —
+  and leaves readiness to the caller. Measured on CHR 7.24.5 under HVF: the CLI returns in
+  ~2 s, REST answers ~17 s later on its own.
+- **Spelled `--no-wait`, not `--detach`.** QEMU *is* detached (above), so `--detach` would
+  name something every start already does. What changes is whether the CLI waits, and
+  `--no-wait` is the other half of centrs' `--wait <deadline>`:
+  `centrs retrieve --quickchr <name> /system/resource --wait 5m` is the documented way to
+  own readiness afterwards, and the CLI prints it as a tip.
+- **Provisioning is refused with `--no-wait`, never skipped.** Provisioning runs against
+  the booted guest, so the pair cannot both be honoured. That includes first-boot intent
+  `add` recorded, which the caller did not retype — the case where skipping would be
+  silent. `--fg` is refused too: a foreground start holds the console until QEMU exits.
+- **`--bg` stays accepted as a deprecated no-op** that warns on stderr, so existing scripts
+  keep working and learn the flag was never doing what they thought. The library default
+  is unchanged: `start()` resolves ready.
+
+### RouterOS 6 is a boot-only target, not supported RouterOS (#182)
+
+A 6.x CHR downloads and boots under quickchr, but the REST readiness probe can never
+succeed against it (6.x has no `/rest`), so `start` ran to its 180 s timeout and then
+deleted a guest that had answered www, ssh and api from 15 s in. The decision is a narrow
+**"track and boot"** contract, for one consumer: centrs needs a disposable 6.x router to
+test the operations an agent needs for a v6→v7 migration (tikoci/centrs#386).
+
+- **Readiness is a native-API `/login`** on guest 8728, built as a general alternative
+  probe (`waitForApiReady()`, chosen by `readinessProbeFor(version)`) rather than a v6
+  special case — a 7.x guest with REST disabled has the same problem. Two consecutive
+  answers; a credential `!trap` counts as an answer, as a REST 401 does. Grounded on CHR
+  6.49.23 under HVF (2026-10-21): empty `admin` gets `!done`; a bad password gets
+  `!trap =message=invalid user name or password (6)` then `!done`; before RouterOS is up,
+  slirp accepts and the probe times out (~20 s), then ready at ~22–25 s.
+- **Not a bare TCP accept.** Under user-mode networking slirp accepts on the host side
+  whatever the guest is doing (see `probeTcpPort()`), so a connect proves nothing.
+- **Everything else is refused loudly, before download where possible**: every
+  provisioning option and the post-boot `set` routes (`PROVISIONING_VERSION_UNSUPPORTED`,
+  checked before the provisioning-window refusal, whose `clean` route would only lead to a
+  second refusal), `arm64` (MikroTik publishes no arm64 CHR for 6.x), and every REST
+  operation (`REST_UNAVAILABLE`, instead of a request that times out). The `secure-login`
+  setting is a default, not a request, so it is not applied to 6.x.
+- **CLI only.** The wizard refuses 6.x — every step it offers is provisioning.
+
+### A failed first boot resets an `add()`-created machine, whatever failed (#182)
+
+#187 made a first boot whose *provisioning* threw reset the machine instead of removing
+it: disk back to the factory image, `machine.json` back to the record `add()` wrote. A
+*boot timeout* on the same first boot still called `remove()`, so the two failure modes of
+one boot disagreed, and the timeout threw away the options `add` recorded. Now
+`_launchExisting()` snapshots the record whenever the provisioning window is open — not
+only when there are provisioning options — and a timeout resets from it. A machine
+`start()` created in the same call is still removed, and one that has booted before is
+unchanged. The first-boot cleanup messages name `QUICKCHR_PRESERVE_ON_FAILURE=1`, because an
+auto-removed machine is what hid #182's report.
+
+### A restart timeout preserves the configured guest (#189)
+
+Boot-timeout cleanup carries explicit intent: remove a machine created by this
+`start()`, factory-reset and restore an existing never-booted machine, or stop and
+preserve an existing machine whose provisioning window was closed. `_launchExisting()`
+classifies from the refreshed authoritative record under the lifecycle lock,
+before launch stamps `lastStartedAt`, PID or status. Supplying a restore record
+for every restart would be destructive too: that branch resets the disk.
+
+Preservation retains disks, credentials, provisioning history and persistent
+configuration, while the shared stop path clears PID/status and releases socket
+memberships. Older records with only `lastStartedAt` remain outside the provisioning
+window. Diagnostics survive; recovery is retry or explicit removal, never an
+implicit factory reset. The environment override still suppresses destructive
+first-boot cleanup. Provisioning rollback is unchanged.
+
+Synthetic launch tests cover all three cases, the override, stale snapshots in
+both directions, disk sentinels and credential retention. The focused live
+`restart-timeout.test.ts` injects a zero readiness-probe budget after a real
+configured boot, exercises real launch/diagnostics/cleanup, then restarts and
+reads back the guest identity with retained credentials. Whole-disk equality is
+only asserted using synthetic sentinels; a live guest can write during boot.
 
 ### Out of Scope (decided)
 
@@ -969,3 +1152,25 @@ Explicitly rejected, with rationale, so they aren't re-proposed:
 ### Document Maintenance
 
 DESIGN.md is the living home for design decisions and rationale. At the end of any significant work session, agents should review whether new implementation details, design decisions, or discovered constraints belong here. Open/close work in **GitHub Issues** — not BACKLOG.md (see CONTRIBUTING.md "Tracking work"); record grounded RouterOS/QEMU behaviour facts in the narrowest scoped doc (`.github/instructions/*.md`, `docs/`, or `test/lab/<topic>/REPORT.md`); add a CHANGELOG.md entry for user-facing changes. Treat this as a lightweight checklist, not a gate.
+
+## Explicit guest shutdown (#143)
+
+`stop()` remains a power cut (QEMU SIGTERM, then SIGKILL), the primary teardown
+primitive and the mechanism used for device-mode confirmation. `shutdown()` is a
+separate bounded operation: one `/system/shutdown` via REST and independent process
+exit observation share a 10-second default grace budget, followed by the existing
+hard-stop escalation if needed. CLI seconds and library milliseconds are explicit;
+the library does not resolve the CLI's environment override.
+
+The HTTP result is diagnostic evidence, not the completion signal. On CHR 7.24.5,
+x86/HVF and arm64/TCG returned HTTP 400 `interrupted` while successfully halting
+(see `test/lab/shutdown/REPORT.md`). No QGA/ACPI/console selection framework is
+needed. Request failure still waits out the grace budget and reports forced
+fallback if QEMU survives. `guest-exited` means no host stop signal was required;
+it does not promise ROSE, container or filesystem durability.
+
+The entire operation holds the existing per-machine lock, refreshes the machine
+before acting, and uses shared stopped-state cleanup without recursively taking
+the non-reentrant stop lock. It confirms QEMU exit before clearing the PID or
+releasing named-network membership. Lock wait, hard-stop escalation and final
+exit verification are separate bounded phases outside the guest grace budget.