@tikoci/quickchr 0.4.8 → 0.5.0

package/DESIGN.md

@@ -97,6 +97,19 @@ Modules (src/lib/)      ← qemu, images, versions, network, state, ...
 
     ⚠️ **Testing note.** `Bun.serve` **drops an explicit `content-length` when the body is a `ReadableStream`** and frames the response `transfer-encoding: chunked`, so a stub built on it cannot exercise the size-derived budget at all — every response looks unknown-length. The anchor tests use a raw `Bun.listen` HTTP stub for that reason.
 
+12. **Two locks, one primitive** (`src/lib/lock.ts`; #140, #181, #180).
+    - **Machine lock** — `machines/<name>/.lock/`, so it goes with the machine. One mutex for every lifecycle operation: `add`, `start`, relaunch, `setDeviceMode`, `stop`, `clean`, `remove`, orphan removal. It was `.start-lock` and taken only by starts, so `stop`/`clean`/`remove` raced every relaunch. For one release a live `.start-lock` is still honoured; a dead one is ignored, not deleted.
+    - **Allocation lock** — `<dataDir>/.allocate.lock/`, held across read → probe → claim when a machine is created. Without it two creations probe the same free ports and pick the same block. The claim itself has to outlive the lock, because `start()` persists `machine.json` only after the download and the spawn, so the creator's machine lock carries `{ portBase, macs }` as a reservation and `listReservations()` reads it. A reservation ends with its lock, and dies with a crashed creator's pid.
+    - **Who waits.** `stop`/`clean`/`remove` wait, bounded by the machine's boot budget, so a `stop` issued during a `start` runs after it. A second `start`, `setDeviceMode` and orphan removal fail fast with `MACHINE_LOCKED`: waiting would only boot the machine twice, or wait out a create that proves the directory is not an orphan. Nobody waits for a `foreground` holder, which keeps the lock until its VM exits. `remove` has no override for a live holder: deleting a directory under a live spawn is the bug the lock exists to prevent. A broken machine has no live holder, so the lock never blocks clearing it.
+    - **Not reentrant.** Two async operations in one process conflict as much as two processes do, so a lock held by this pid counts as held. Code inside a locked operation calls `stopUnderLock()`/`removeUnderLock()`/`cleanUnderLock()`, never the public `ChrInstance` methods.
+    - **State is re-read under the lock.** A handle's snapshot predates whatever the previous holder did, and a stale `state.pid` is a process that has exited or one the OS has reused.
+    - **An append-only log, so nothing is deleted from under a holder.** A lock is a directory of numbered records, and the highest decides the state: a live holder means held; a released marker, a dead holder's record or nothing means free. Acquiring creates record top+1 with `link()`, so exactly one contender wins and the file never exists half-written. Releasing appends a released marker, and records below the top are collected. A claim stands only if nothing higher exists and its predecessor is still there, which turns away a contender that slept through other claims, or through a remove and re-create of the machine.
+    - **Why not delete a stale lock?** A path has no compare-and-delete. Every deleting recovery tried here had a window between judging a holder dead and deleting its file, in which a successor could be installed and then deleted: rename-aside-and-compare, then a reap guard recovered by age. The Codex review of #190 reproduced both. Taking over a dead holder is now just claiming the next number. A cross-process stress run (6 processes, 360 critical sections, about 15% simulated crashes taken over mid-run) showed no overlap.
+    - **Records are named for the lock's incarnation, so a log can be deleted.** A machine's lock lives in `machines/<name>/` and goes with it. A machine can be removed and created again under the same name while some process is paused holding paths into the old lock, so every record is named `<id>.<gen>`, with an id published once per lock directory. A stale process cannot reach the new incarnation's records: not with a claim, a collect, a rejected-claim cleanup or a released marker. Codex reproduced two such cases while records were named by number alone: a claim matching the new record 1, and a collect run after the released marker that deleted the new holder's record. Before that, the logs were briefly pushed out to a never-deleted `locks/<name>/`, which kept a file for every name ever used.
+    - **A claim stands only on what it judged, and a holder cleans up while it still holds.** After creating its record, a claimer checks three things: nothing higher exists, its predecessor is byte-for-byte the record it judged free, and its id is still the directory's incarnation. Release collects old records *before* publishing its marker, because from that moment it is no longer the holder and nothing may be deleted by path. A released lock therefore keeps two records: the last holder's record and its marker.
+    - **`remove()` renames, then deletes, with no in-place fallback.** `removeMachine()` moves `machines/<name>/` to `<dataDir>/.removing/` in one step before deleting it. A recursive delete in place has a window where `.lock/` is gone but the directory is not, and a waiter or same-named create could take a fresh lock in a directory being deleted. A rename that fails is retried for about 2 s, which covers a scanner holding a handle on Windows. After that, `remove` fails with `STATE_ERROR` and leaves the directory in place, rather than falling back to the unsafe delete (Codex review of #190). Once the rename succeeds the name is free, so a failed delete only warns, giving the staged path. Each removal sweeps `.removing/` for entries left by a delete that failed or a process that stopped after its rename, skipping any whose pid is still alive in another process. A waiter never re-creates a removed machine's directory, because the lock directory is created with a non-recursive `mkdir`: it gets `MACHINE_NOT_FOUND`. `assertSameMachine()` catches one that waited out a remove and re-create.
+    - **Unreadable means held.** A record or legacy `.start-lock` that cannot be read, or that does not parse and is younger than 500 ms, blocks. It is never treated as absent.
+
 ## Port Layout
 
 | Offset | Service    | Guest Port |
@@ -579,13 +592,57 @@ Always read back what we write. One extra REST API call after a provisioning act
 
 ### Provisioning Scope
 
-quickchr provisions at first boot and (optionally) on restart:
+quickchr provisions at a machine's **first boot** — see "The provisioning window"
+below for why, and for what happens to an option that arrives later:
 - **User creation** — create user, set password, optionally disable admin
 - **Package install** — SCP `.npk` files, reboot to activate
 - **License** — `/system/license/renew` for trial
 - **Device-mode** — `/system/device-mode/update mode=rose container=yes ...` for restricted features (containers, traffic-gen, routerboard). Opt-in: not configured unless explicitly requested via CLI `--device-mode` or API `deviceMode` option. CHR ships with `mode=advanced` which is sufficient for most use cases. Device-mode requires a hard QEMU power-cycle to confirm changes — this is the MikroTik-mandated confirmation mechanism (physical button press on real hardware, cold reboot on VM). The wizard defaults to `rose` when the user opts in, since it enables containers. See: https://help.mikrotik.com/docs/spaces/ROS/pages/93749258/Device-mode
 - **Config import** — planned: load `.rsc` or `.backup` at creation time
 
+#### The provisioning window — and why `lastStartedAt` could not be the gate
+
+Every provisioning step assumes the guest holds RouterOS' default configuration, so
+provisioning runs in a window that closes at **first boot** — not at `add`. A machine
+created with no provisioning flags can still take them on its first `start`, which is
+the behaviour agents actually rely on.
+
+The gate was `!state.lastStartedAt`, and that field does not mean what the gate needed.
+It is stamped immediately after `spawnQemu`, before the guest is known to have booted
+and before provisioning runs, so it records "QEMU has been launched" rather than "the
+guest has been configured". The two agree most of the time, which is why the
+conflation survived — and come apart in three ways, each a defect (#176):
+
+- **`clean()`** replaces the disk with the factory image but left `lastStartedAt`
+  behind, so a genuinely fresh guest could never be provisioned again. `clean()` now
+  clears the gate along with the credentials it already cleared, keeping `cleanedAt`
+  for the forensic clue. A cleaned machine's next start re-applies the retained
+  provisioning *intent* and costs what a first boot costs.
+- **Provisioning that throws after a successful boot** leaves a running,
+  half-provisioned machine with the gate closed. The record is stamped only on
+  success, so this case is at least *visible* now; making it retryable — re-running
+  only the steps that did not land — is the remaining half of #176 and is what
+  `provisioning.steps` exists to support.
+- **A first start with no provisioning options** closed the gate although nothing had
+  been configured.
+
+`MachineState.provisioning` is the positive record: when provisioning last completed
+and which steps ran. `isProvisioningWindowOpen()` reads it and falls back to
+`lastStartedAt` for machines that predate the field, so no existing machine
+re-provisions itself on its next start. This is the desired-config/runtime-state split
+of #58, applied to the one field where the conflation was already producing bugs
+rather than to `machine.json` wholesale.
+
+**Nothing may drop a provisioning option in silence.** `start()` used to pass
+`undefined` in place of the caller's options on every path but a first boot, so all
+seven were discarded with no warning, no error and a normal-length boot — the only way
+to notice was to read the value back. An option outside the window now throws
+`PROVISIONING_WINDOW_CLOSED`, naming each option and the route that still works
+(`quickchr set <name> --license`, the library's `setDeviceMode()` / `installPackage()`,
+or `clean()` for anything else). An option state already records is treated as a
+no-op, so passing the same flags on every start — the shape a script naturally takes —
+is not an error.
+
 Provisioning via REST API is preferred (simple HTTP calls). Serial console provisioning (prompt detection + buffer tracking, as in chr-armed) is a fallback for locked environments. Key lessons from chr-armed serial work: use `\r` not `\r\n` on PTY; accumulate buffer with offset tracking to prevent re-matching; detect prompts dynamically, don't use fixed delays.
 
 ### Exec Transport Design
@@ -946,12 +1003,88 @@ Detaching does not weaken cleanup, because nothing that sweeps QEMU sweeps by gr
 is worth stating rather than assuming: "the child outlives its parent's signal" and "a
 wedged run leaks VMs" would otherwise look like the same change.
 
-The naming half is not fixed and should not be fixed quietly. `--bg` is already the
-default and only ever meant "serial console goes to a log file instead of stdio" — it does
-not make `start` return early, because `start` waits for REST readiness by contract. Both
-an external agent and the maintainer believed it bought them something. The honest repair
-is a separate `--no-wait` that opts out of the wait, not a redefinition of `--bg` that
-would change what existing scripts get; #159 holds that half.
+The naming half was not fixed quietly. `--bg` is already the default and only ever meant
+"serial console goes to a log file instead of stdio" — it does not make `start` return
+early, because `start` waits for readiness by contract. Both an external agent and the
+maintainer believed it bought them something. The repair (#159, 0.5.0) is a separate flag
+rather than a redefinition of `--bg` that would change what existing scripts get:
+
+- **`start --no-wait`** (`StartOptions.wait: false`) returns once QEMU is confirmed
+  running — past `spawnQemu()`'s 1.5 s early-exit check, with state and ports persisted —
+  and leaves readiness to the caller. Measured on CHR 7.24.5 under HVF: the CLI returns in
+  ~2 s, REST answers ~17 s later on its own.
+- **Spelled `--no-wait`, not `--detach`.** QEMU *is* detached (above), so `--detach` would
+  name something every start already does. What changes is whether the CLI waits, and
+  `--no-wait` is the other half of centrs' `--wait <deadline>`:
+  `centrs retrieve --quickchr <name> /system/resource --wait 5m` is the documented way to
+  own readiness afterwards, and the CLI prints it as a tip.
+- **Provisioning is refused with `--no-wait`, never skipped.** Provisioning runs against
+  the booted guest, so the pair cannot both be honoured. That includes first-boot intent
+  `add` recorded, which the caller did not retype — the case where skipping would be
+  silent. `--fg` is refused too: a foreground start holds the console until QEMU exits.
+- **`--bg` stays accepted as a deprecated no-op** that warns on stderr, so existing scripts
+  keep working and learn the flag was never doing what they thought. The library default
+  is unchanged: `start()` resolves ready.
+
+### RouterOS 6 is a boot-only target, not supported RouterOS (#182)
+
+A 6.x CHR downloads and boots under quickchr, but the REST readiness probe can never
+succeed against it (6.x has no `/rest`), so `start` ran to its 180 s timeout and then
+deleted a guest that had answered www, ssh and api from 15 s in. The decision is a narrow
+**"track and boot"** contract, for one consumer: centrs needs a disposable 6.x router to
+test the operations an agent needs for a v6→v7 migration (tikoci/centrs#386).
+
+- **Readiness is a native-API `/login`** on guest 8728, built as a general alternative
+  probe (`waitForApiReady()`, chosen by `readinessProbeFor(version)`) rather than a v6
+  special case — a 7.x guest with REST disabled has the same problem. Two consecutive
+  answers; a credential `!trap` counts as an answer, as a REST 401 does. Grounded on CHR
+  6.49.23 under HVF (2026-10-21): empty `admin` gets `!done`; a bad password gets
+  `!trap =message=invalid user name or password (6)` then `!done`; before RouterOS is up,
+  slirp accepts and the probe times out (~20 s), then ready at ~22–25 s.
+- **Not a bare TCP accept.** Under user-mode networking slirp accepts on the host side
+  whatever the guest is doing (see `probeTcpPort()`), so a connect proves nothing.
+- **Everything else is refused loudly, before download where possible**: every
+  provisioning option and the post-boot `set` routes (`PROVISIONING_VERSION_UNSUPPORTED`,
+  checked before the provisioning-window refusal, whose `clean` route would only lead to a
+  second refusal), `arm64` (MikroTik publishes no arm64 CHR for 6.x), and every REST
+  operation (`REST_UNAVAILABLE`, instead of a request that times out). The `secure-login`
+  setting is a default, not a request, so it is not applied to 6.x.
+- **CLI only.** The wizard refuses 6.x — every step it offers is provisioning.
+
+### A failed first boot resets an `add()`-created machine, whatever failed (#182)
+
+#187 made a first boot whose *provisioning* threw reset the machine instead of removing
+it: disk back to the factory image, `machine.json` back to the record `add()` wrote. A
+*boot timeout* on the same first boot still called `remove()`, so the two failure modes of
+one boot disagreed, and the timeout threw away the options `add` recorded. Now
+`_launchExisting()` snapshots the record whenever the provisioning window is open — not
+only when there are provisioning options — and a timeout resets from it. A machine
+`start()` created in the same call is still removed, and one that has booted before is
+unchanged. The first-boot cleanup messages name `QUICKCHR_PRESERVE_ON_FAILURE=1`, because an
+auto-removed machine is what hid #182's report.
+
+### A restart timeout preserves the configured guest (#189)
+
+Boot-timeout cleanup carries explicit intent: remove a machine created by this
+`start()`, factory-reset and restore an existing never-booted machine, or stop and
+preserve an existing machine whose provisioning window was closed. `_launchExisting()`
+classifies from the refreshed authoritative record under the lifecycle lock,
+before launch stamps `lastStartedAt`, PID or status. Supplying a restore record
+for every restart would be destructive too: that branch resets the disk.
+
+Preservation retains disks, credentials, provisioning history and persistent
+configuration, while the shared stop path clears PID/status and releases socket
+memberships. Older records with only `lastStartedAt` remain outside the provisioning
+window. Diagnostics survive; recovery is retry or explicit removal, never an
+implicit factory reset. The environment override still suppresses destructive
+first-boot cleanup. Provisioning rollback is unchanged.
+
+Synthetic launch tests cover all three cases, the override, stale snapshots in
+both directions, disk sentinels and credential retention. The focused live
+`restart-timeout.test.ts` injects a zero readiness-probe budget after a real
+configured boot, exercises real launch/diagnostics/cleanup, then restarts and
+reads back the guest identity with retained credentials. Whole-disk equality is
+only asserted using synthetic sentinels; a live guest can write during boot.
 
 ### Out of Scope (decided)
 
@@ -969,3 +1102,25 @@ Explicitly rejected, with rationale, so they aren't re-proposed:
 ### Document Maintenance
 
 DESIGN.md is the living home for design decisions and rationale. At the end of any significant work session, agents should review whether new implementation details, design decisions, or discovered constraints belong here. Open/close work in **GitHub Issues** — not BACKLOG.md (see CONTRIBUTING.md "Tracking work"); record grounded RouterOS/QEMU behaviour facts in the narrowest scoped doc (`.github/instructions/*.md`, `docs/`, or `test/lab/<topic>/REPORT.md`); add a CHANGELOG.md entry for user-facing changes. Treat this as a lightweight checklist, not a gate.
+
+## Explicit guest shutdown (#143)
+
+`stop()` remains a power cut (QEMU SIGTERM, then SIGKILL), the primary teardown
+primitive and the mechanism used for device-mode confirmation. `shutdown()` is a
+separate bounded operation: one `/system/shutdown` via REST and independent process
+exit observation share a 10-second default grace budget, followed by the existing
+hard-stop escalation if needed. CLI seconds and library milliseconds are explicit;
+the library does not resolve the CLI's environment override.
+
+The HTTP result is diagnostic evidence, not the completion signal. On CHR 7.24.5,
+x86/HVF and arm64/TCG returned HTTP 400 `interrupted` while successfully halting
+(see `test/lab/shutdown/REPORT.md`). No QGA/ACPI/console selection framework is
+needed. Request failure still waits out the grace budget and reports forced
+fallback if QEMU survives. `guest-exited` means no host stop signal was required;
+it does not promise ROSE, container or filesystem durability.
+
+The entire operation holds the existing per-machine lock, refreshes the machine
+before acting, and uses shared stopped-state cleanup without recursively taking
+the non-reentrant stop lock. It confirms QEMU exit before clearing the PID or
+releasing named-network membership. Lock wait, hard-stop escalation and final
+exit verification are separate bounded phases outside the guest grace budget.