@tikoci/quickchr 0.4.5 → 0.4.7

package/DESIGN.md

@@ -54,6 +54,49 @@ Modules (src/lib/)      ← qemu, images, versions, network, state, ...
 
    **Sister-project routing.** Because `fetchResilient` already absorbs a flaky resolver, a download/version-resolve failure should not be papered over downstream with `/etc/hosts` pins or IPv6 toggles in a consuming repo's workflow — fix it (or extend the failback) here in quickchr. The historical `getent ahostsv4` `/etc/hosts` workaround was both in the wrong layer and non-functional (it hit the same broken stub resolver, returning empty). The failback is best-effort, not a guarantee: it recovers connection-class DNS failures, not arbitrary network breakage. Covered by `test/unit/net.test.ts`.
 
+10. **arm64 CHR is forced onto TCG on all Apple Silicon** — `detectAccel("arm64")` never returns `hvf` on macOS; the launch path surfaces `accelNote()` explaining why (`src/lib/platform.ts`). The cause is an **image artifact**, not an accelerator or CPU-model bug: current arm64 CHR images (7.20.8 – 7.23beta5 verified) ship an AArch64 kernel over a **32-bit ARM userspace** — the appended initramfs `/init` is `ELF 32-bit LSB ARM, EABI5, static`, and the 7.22.1 `system` package holds **101 further ARM32 executables and 18 ARM32 shared objects** (the only AArch64 executables are `kexec` and `vmcore-dmesg`). Apple Silicon implements **no AArch32 at any exception level**: `ID_AA64PFR0_EL1` reports AArch64-only for EL0/EL1, and QEMU under HVF passes that hardware register straight through (`hvf_arch_init_vcpu()` re-reads the live vCPU register and only edits the GIC bit; the `-cpu` model is inert). So the guest kernel never sets `ARM64_HAS_32BIT_EL0`, `compat_elf_check_arch()` rejects the `EM_ARM` `/init` with `-ENOEXEC`, the initramfs has no fallback init, and Linux panics at t≈0.076 s with `No working init found`. TCG's emulated models do provide AArch32 EL0, so the same unmodified image boots to `MikroTik Login:`. Grounded in the guest's own panic-time capability bitmap: the M4/HVF guest printed `0x20012,28000230` (8 caps, `ARM64_HAS_32BIT_EL0` absent) against TCG's `0x20013,28402230` (11 caps, present). Full chain in [`docs/m4-hvf-arm64-investigation.md`](docs/m4-hvf-arm64-investigation.md).
+
+    **Scope is every Apple Silicon generation, not M4+.** The first report came from an M4 (tikoci/mikropkl#11) and the original fix keyed on `hw.optional.arm.FEAT_SSBS == 0`, which M4 is the first Apple chip to report — but SSBS was a **coincident marker of the reporting host, not the mechanism**. Linux 5.6 treats SSBS as an optional mitigation and boots fine without it (`cortex-a53`/`neoverse-n1` lack it), while *no* Apple CPU since 2020 implements AArch32. The SSBS predicate therefore left M1/M2/M3 on HVF and panicking — a live bug, not merely a narrow gate. Ordinary server ARM (Ampere, Graviton) does implement AArch32 EL0, which is why arm64 CHR runs there under KVM; this is an Apple-Silicon-and-CHR-artifact interaction, not a RouterOS-on-ARM defect.
+
+    **No QEMU flag or version fixes this.** Apple's `Hypervisor.framework` exposes only `hv_vcpu_config_get_feature_reg()` — there is no setter — so no macOS VMM can present a feature the silicon lacks (UTM's fork behaves identically). A QEMU version floor is therefore *not* a valid restore signal, and the deferred `getQemuVersion()` guard the SSBS-era decision contemplated has been dropped. **The restore signal is the guest artifact:** a future arm64 CHR whose appended `/init` *and* required system-package executables/shared objects are all AArch64, confirmed by a real HVF boot. That check is mechanical (§"Reproduction" in the investigation doc) and could become a release-time guard.
+
+    **Escape hatch.** Because the fallback is unconditional, `--accel <auto|tcg|hvf|kvm>` (and the `accel` setting / `QUICKCHR_ACCEL` env var) forces the accelerator verbatim and bypasses detection — so a fixed future image can be tested on HVF without a code change, and so forcing TCG stays possible for debugging elsewhere. `--accel hvf` on an arm64 guest still prints the panic caveat rather than silently obeying. This complements, and does not replace, the reactive respawn-once net in #8. Covered by `test/unit/platform.test.ts`. → [#97](https://github.com/tikoci/quickchr/issues/97)
+
+11. **A download is bounded by whether it is *moving*, not by a flat total** — `src/lib/download.ts` is the one download path for both `images.ts` and `packages.ts`. Before it, each caller had its own bound and they disagreed: images aborted at a flat `120_000` per attempt with three retries, packages had **no** deadline and **no** retries, so its only bound was whatever the calling test imposed.
+
+    A total-duration deadline cannot distinguish *slow* from *stuck*. It fires on a healthy transfer whose only sin is being large, and the retry path then re-downloads from zero — one slow transfer becomes three.
+
+    **The deadline sat inside the natural variance of a healthy transfer, which is why it was intermittent.** Measured locally 2026-07-31 against `download.mikrotik.com` — same 52.2 MB `all_packages-arm64-7.22.1.zip`, same link, same client, four consecutive attempts:
+
+    | attempt | elapsed | throughput | under the old flat 120 s? |
+    |---|---|---|---|
+    | 1 | 118.4 s | 0.421 MB/s | yes, by 1.6 s |
+    | 2 | 94.2 s | 0.528 MB/s | yes |
+    | 3 | **123.5 s** | 0.403 MB/s | **no — aborted** |
+    | 4 | 82.2 s | 0.606 MB/s | yes |
+
+    All four completed. **One in four exceeded the deadline**, and a fifth attempt through the new streaming path measured 129.4 s (0.385 MB/s) — so the range on one ordinary link is 82–129 s against a 120 s bound. The deadline was inside the distribution, and an unchanged healthy download therefore passed or failed on link jitter alone.
+
+    That is a sharper statement of the defect than a clean abort would have been, and it is why CI saw "both images needed two retries" *sometimes* rather than always. CI's cold ~0.35 MB/s (→ ~149 s) sits **below** this whole local range, which is why a hosted runner hit it far more reliably than this laptop does — run 30606079288, and B7's local suite where `provisioning.test.ts` spent 619 s of 992 s downloading with zero QEMU processes alive.
+
+    ⚠️ **Do not restate this as "the old code always aborted that download."** An earlier draft of this entry said so on the strength of a single 129.4 s measurement; the next attempt completed in 101.5 s and disproved it as a general claim. One measurement of an intermittent is a signal, not a fact — the same rule the CI program runs on.
+
+    **Two deadlines, and the failure says which fired.** Neither alone is sufficient: stall detection alone would let a transfer trickling at one byte per second run forever, and a size-derived budget alone cannot fail a wedge quickly.
+    - **Stall** (`DOWNLOAD_STALL_MS`, 30 s) — reset on every received chunk. Bounds *silence*, so a moving transfer is never aborted for being slow.
+    - **Transfer budget** — `DOWNLOAD_BUDGET_BASE_MS` (30 s for connect/TLS/headers) plus `content-length ÷ COLD_DOWNLOAD_FLOOR_BYTES_PER_S`, a **floor** throughput of 120 000 B/s ≈ ⅓ of the slowest cold transfer ever measured. The 52.2 MB zip gets 465 s. When the server sends no `content-length` the stated `DOWNLOAD_NO_LENGTH_BUDGET_MS` (15 min) applies and the outcome says so — verified that `download.mikrotik.com` *does* send it on both image and package artifacts, so the size-derived budget is the production path.
+
+    **Retry policy follows from that split.** `DOWNLOAD_STALLED` is retriable — a wedged socket usually moves on the next attempt. `DOWNLOAD_TOO_SLOW` is **terminal on purpose**: the budget is already ~3× the slowest throughput on record, so retrying re-downloads from zero on a link measured slower than the floor, which is the exact behavior this decision removes. It is a signal that the floor is wrong (or the link genuinely is), not something to paper over with another attempt.
+
+    **The floor constant has one home.** It lives here and `test/integration/timeouts.ts` re-exports it; `coldDownloadTestTimeout()` is *derived from* `transferBudgetMs()` rather than recomputing it, which makes one of #106's partial orders (`download deadline < test timeout`) structural instead of coincidental. Two constants asked not to drift are how they drift.
+
+    **Streaming also bounds memory, which the old path did not.** `await response.arrayBuffer()` held the entire artifact in RAM before writing it — 52.2 MB for the largest. Measured: a `Bun.file().writer()` sink writing 40 MiB in 1 MiB chunks keeps the file growing on disk in lockstep with flat RSS (22 MB throughout), so the transient allocation is gone. Recorded as a side effect, **not** as a claim about #76 — a 52 MB transient is not a plausible cause of a runner losing communication, and B8a/B8b should not treat this as having changed that picture.
+
+    **Partial transfers cannot poison the cache.** Downloads stream to a **per-call** `<dest>.<pid>-<uuid>.part` and are renamed only after a verified transfer. Per-call, not a fixed `<dest>.part`, because both callers guard only with `existsSync(destPath)` — two concurrent downloads of one artifact (two `quickchr start`s, parallel tests) would otherwise interleave writes into one file, publish the corrupt result, and delete each other's in-flight file. The unique suffix removes the collision without deduplicating the work; coalescing concurrent downloads needs a lock at both call sites and is a larger change.
+
+    Verification is two checks, because either alone leaves a hole: bytes received must equal `content-length` **when one was sent**, and a **zero-byte body is always a failure** regardless of framing. The second exists because Bun's fetch hands back an *unknown-length* response for a malformed `content-length` — measured against a raw stub, `4096.5` and `-5` are passed through as headers but deliver 0 bytes, while a blank value is stripped to `null` — and an unknown-length response has no length to verify against. Without it the empty file would be renamed into the cache and served as a complete artifact forever. No artifact quickchr downloads is empty, so this costs nothing. Both callers gate on `existsSync(zipPath)`, so a truncated file left at the destination would be served as a complete cached artifact forever. Resuming with a `Range` request would be better than either deadline (MikroTik sends `accept-ranges: bytes`) and is deliberately out of scope — this decision bounds and classifies a transfer, it does not change the transport. Covered by `test/unit/download.test.ts`. → [#116](https://github.com/tikoci/quickchr/issues/116)
+
+    ⚠️ **Testing note.** `Bun.serve` **drops an explicit `content-length` when the body is a `ReadableStream`** and frames the response `transfer-encoding: chunked`, so a stub built on it cannot exercise the size-derived budget at all — every response looks unknown-length. The anchor tests use a raw `Bun.listen` HTTP stub for that reason.
+
 ## Port Layout
 
 | Offset | Service    | Guest Port |
@@ -94,14 +137,18 @@ Current consumers are all first-party tikoci projects (centrs, donny, restraml),
 │       ├── serial.sock        # Serial console
 │       ├── qga.sock           # QGA (x86 only)
 │       ├── qemu.pid           # PID file
-│       └── qemu.log           # Output log
+│       ├── qemu.log           # Output log
+│       └── serial.log         # Serial console tee (only with QUICKCHR_SERIAL_LOG=1)
+└── failures/                  # Boot-failure reports (see below)
+    └── boot-failure-<machine>-<iso8601>.json
 ```
 
 **Global settings** live separately, under the XDG **config** tier (not the data tree
 above): `~/.config/quickchr/quickchr.env`, dotenv-style (`QUICKCHR_KEY=value` lines).
 Managed via `quickchr settings get|set|print|reset`. Precedence per key: CLI flag >
-`QUICKCHR_<KEY>` env var > `quickchr.env` > built-in default. The 5 managed keys:
-`default-channel`, `default-arch`, `cache-max-size`, `timeout-extra`, `secure-login`.
+`QUICKCHR_<KEY>` env var > `quickchr.env` > built-in default. The 6 managed keys:
+`default-channel`, `default-arch`, `accel`, `cache-max-size`, `timeout-extra`,
+`secure-login`.
 See MANUAL.md's CLI reference and environment-variables sections for the full surface.
 
 ## Platform Support
@@ -109,16 +156,20 @@ See MANUAL.md's CLI reference and environment-variables sections for the full su
 | Platform               | x86 CHR | arm64 CHR | Notes |
 |------------------------|---------|-----------|-------|
 | macOS x86_64           | HVF     | TCG       | Intel Mac |
-| macOS arm64 (native)   | HVF     | HVF       | Apple Silicon, bun is arm64 |
-| macOS arm64 (Rosetta)  | HVF     | TCG       | bun is x86_64; arm64 HVF skipped |
+| macOS arm64 (native)   | TCG     | TCG¹     | Apple Silicon, bun is arm64 |
+| macOS arm64 (Rosetta)  | TCG     | TCG¹     | Apple Silicon, bun reports x86_64 |
 | Linux x86_64           | KVM     | TCG       | KVM requires `/dev/kvm` writable |
 | Linux aarch64          | TCG     | KVM       | x86 TCG on arm64 Linux |
 | Windows x86_64         | TCG     | TCG       | HVF/KVM not available |
 
+¹ arm64 CHR is forced to **TCG on every Apple Silicon generation**: the image's required userspace is 32-bit ARM and Apple CPUs implement no AArch32, so an HVF guest panics at init (see Design Decisions #10). x86 CHR also uses TCG because HVF cannot virtualize x86 on an arm64 host. Override with `--accel hvf` to test a future AArch64-only image.
+
 **Acceleration detection** (`detectAccel`):
-- macOS: checks `kern.hv_support` via sysctl; for arm64 guest additionally checks `process.arch === "arm64"` (native bun = Apple Silicon).
+- An explicit override (`--accel` flag > `QUICKCHR_ACCEL` env > `accel` in `quickchr.env`) short-circuits detection entirely; `auto` (the default) runs the checks below.
+- macOS: checks `kern.hv_support` via sysctl. Intel x86 guests get HVF; **all guests on Apple Silicon get TCG** — x86 because HVF cannot virtualize across architectures, arm64 because the CHR image needs AArch32 (Design Decisions #10). A Rosetta process is identified via `sysctl.proc_translated`.
 - Linux: checks `/dev/kvm` writability.
-- Falling back to TCG is always safe, just slower (~20s x86 TCG boot on Apple Silicon; ~2 min arm64 TCG on Intel).
+- Falling back to TCG is safe but can be significantly slower; measure with the
+  target host, guest architecture, and RouterOS version.
 
 ## CI System
 
@@ -176,6 +227,33 @@ incidentally; crediting those suppressed the scheduler for versions no full suit
 targeted). The per-run files are the source of truth; `ci-metrics refold` rebuilds the
 rollup after fold-logic changes.
 
+**What survives losing a runner (#76, #77)**: measured on run
+[30665449265](https://github.com/tikoci/quickchr/actions/runs/30665449265), whose three
+`macos-x86` legs stopped communicating mid-suite. This is the constraint every
+lost-runner instrument has to be designed around, so it is recorded rather than
+re-derived:
+
+| Evidence | Survives? |
+| --- | --- |
+| Uploaded artifacts | **No** — `if: always()` steps never run |
+| The archived job log | **No** — the blob returns `BlobNotFound`, even though a *normally failing* leg in the same run keeps its full log. Failure preserves the log; runner loss destroys it |
+| The jobs API step ledger | **Yes** — the test step stays `in_progress` with a start time, later steps `pending`, and the job carries a `completed_at` |
+| A check run PATCHed during the leg | **Yes** — it is server-side the moment it is written, and stays un-closed |
+
+So a progress marker has to be *pushed out of the runner as it goes*; nothing collected
+at the end can work. That is what `ci-leg-checkpoint.ts` (per-file check-run updates) and
+`ci-leg-ledger.ts` (planned-vs-completed reconciliation into `ci-data/attempted-legs.json`)
+implement. The ledger is a **separate file from `tested-versions.json` by decision**: the
+version scheduler reads that file as a presence test, so recording an aborted run there
+would make the version look tested and silently stop rescheduling it.
+
+The same run also pins down #76's shape: the three legs died **62.0 / 64.1 / 65.0 minutes
+into their own jobs**, staggered with their own start times across 14 minutes of
+wall-clock. So the boundary is per-job elapsed, not a shared external event and not a
+round 60-minute constant. Read `completed_at` as an **upper bound** on the wedge — it is
+when the service gave up on a silent runner, and the interval between the runner going
+quiet and that verdict is uncharacterized.
+
 **Merge policy**: squash-only, PR title → main commit subject (write PR titles as
 conventional commits), PR body → commit body, branches auto-delete. Review threads must
 be resolved before merge (`required_conversation_resolution`) and automated reviews must
@@ -191,6 +269,166 @@ merges (`continue-on-error: true`). Thresholds are overridable via dispatch inpu
 - `{integration|main|sweep}-logs-<platform>` — bun test output + timing + metrics.ndjson +
   machine.json + qemu.log (7 days); durable timing lives on the `ci-data` branch
 
+### Boot-Failure Forensics (`src/lib/diagnostics.ts`)
+
+Discovered constraint from the `ci:slow-platform-flake` series (#76 #79 #80 #91):
+**a failure that cleans up after itself is unfixable.** Both `BOOT_TIMEOUT` paths
+used to `stop()` + `remove()` the machine and embed `qemu.log.slice(-1200)`, which
+on every recorded CI failure contained nothing but the cleanup SIGTERM. Four rules
+came out of that:
+
+1. **`waitForBoot()` classifies, it does not just return `false`.** A boot that
+   never becomes REST-ready is several different bugs wearing the same error:
+   `refused` = QEMU never bound the port (host-side); `probe-timeout` = it bound
+   and nothing behind it answered (guest-side); `reset`/`wrong-body` = the guest
+   answered and RouterOS is still settling (the #69 shape). Collapsing all of
+   them to a boolean is why #79 stayed open across two full CI runs. The tally
+   rides on an optional `BootProbeStats` out-param, so normal boots pay nothing.
+
+   **Do not read a live host port as "the guest is up."** Under the default
+   `user` (slirp) network mode QEMU's hostfwd `listen()`s for the process's whole
+   lifetime and accepts before it tries to reach the guest. Verified locally
+   (2026-07-27) by starting CHR with `mem: 32`: 72/72 probes `probe-timeout`,
+   `hostfwd: accepting`, `info status: running`, empty `serial.log`. So
+   `refused` is much rarer than intuition suggests, and `probe-timeout` — not
+   `refused` — is the ordinary signature of a guest that never booted.
+
+2. **The report is written outside the machine directory**, to
+   `<dataDir>/failures/`. Every integration test wraps its body in
+   `finally { cleanupMachine(name) }` → `remove()`, which deletes the machine
+   dir. A report written next to `qemu.log` is gone before CI can upload it,
+   *regardless* of `QUICKCHR_PRESERVE_ON_FAILURE`. The report therefore embeds
+   log contents rather than pointing at them, and stands alone.
+
+3. **`QUICKCHR_SERIAL_LOG=1` is opt-in, and must stay opt-in.** It adds
+   `logfile=` to the serial chardev, which is the only way to see what RouterOS
+   printed on a boot that never reaches REST — the socket chardev keeps no
+   history, so connecting *after* the failure shows nothing. But serial-console
+   provisioning types the generated user password in cleartext, so the log is
+   secret-bearing. CI turns it on because CI passwords are per-run throwaways.
+
+4. **Timeouts report where they stalled.** `monitorCommand()` records
+   connect → first byte → prompt → command written → response first byte, and
+   renders them into the timeout message. "Monitor command timed out" alone
+   cannot distinguish #80's hypotheses (monitor never greeted us vs. QEMU took
+   the command and went quiet); the phase line can, at the cost of a few
+   `Date.now()` calls on the success path.
+
+Two more came out of #105/#106, once the reports above proved only *that* #79's
+boot went silent:
+
+5. **Evidence has to localize, so the capture is layered by invasiveness.**
+   Host-side per-port classification from `info usernet` (read-only, always on)
+   → read-only guest snapshot over the serial console, which stays reachable
+   while REST is dead (on whenever a serial channel exists) → the counting-rule
+   probe, which writes a mangle rule into the guest and is therefore opt-in via
+   `QUICKCHR_DEEP_BOOT_DIAGNOSTICS=1` and skipped under
+   `QUICKCHR_PRESERVE_ON_FAILURE=1`. Each layer answers a question the one
+   before it cannot: `SYN_SENT` proves a silent drop but not *whose*; only a
+   counter inside the guest separates "RouterOS dropped it" from "it never
+   arrived". The RouterOS-side details (why mangle `passthrough` and not a
+   filter `accept`, why conntrack cannot answer this) live in
+   `provisioning.instructions.md`; the `info usernet` reading in
+   `qemu.instructions.md`.
+
+   The same secrecy rule as (3) applies one level up: guest payloads go to the
+   JSON report, and only credential-free shapes — row counts, booleans, a
+   verdict — reach the thrown error, which CI echoes into public job logs.
+
+6. **A timeout that outlives the forensics is not optional.** Integration tests
+   hardcoding `300_000` against a 480 s same-arch TCG budget meant bun killed
+   the test before `waitForBoot()` gave up, so the capture above never ran —
+   four #79 reproductions arrived bare for exactly this reason. Tests derive
+   their timeout from `defaultBootTimeout()` plus `BOOT_FORENSICS_BUDGET_MS`
+   (`test/integration/timeouts.ts`), and a unit test asserts the invariant
+   across every arch/accel combination. Note this is alignment, not tuning: the
+   4× same-arch TCG factor is still ~11× the measured 44 s worst case, and
+   cutting it needs its own measurement of the package-install and device-mode
+   paths (#106).
+
+Corollary for CI cache keys: `actions/cache` only saves when the primary key
+*missed*, so a static key means a newly-resolved RouterOS version re-downloads
+every run forever (#91). Primary keys must carry a rotating component, with
+`restore-keys` doing the prefix fallback.
+
+7. **The boot probe can break the service it is waiting for.** `restGet()`
+   implements its deadline as `req.destroy()` — a mid-flight TCP teardown. On
+   its own that is harmless. But an aborted request **followed by a completed
+   one** corrupts RouterOS `www`: the guest then resets incoming connections,
+   and a surviving connection is answered with the status computed for the
+   *aborted* request. An `admin:` probe carrying a valid empty password comes
+   back `401`. `waitForBoot()` emits exactly that pattern — overrun a probe,
+   then succeed on the next one 2 s later.
+
+   Reproduced deterministically on RouterOS 7.21.5, x86/HVF **and** arm64
+   cross-arch TCG, with `curl` in a separate process confirming it is
+   guest-side rather than a Bun `node:http` artifact
+   (`test/lab/www-abort-damage/`). Two consequences for design: a slow-but-alive
+   REST layer must be **waited for** rather than aborted, and backing off after
+   consecutive aborts matters because it is the abort/complete *alternation*,
+   not the abort count, that does the damage. It also means a REST status
+   received after any timeout in the same conversation cannot be trusted to
+   belong to the request that received it — the #69 surface.
+
+   Bounds, so this is not over-read: locally `www` recovers in 1.3–2.4 s, so the
+   *permanent* wedge in #79's CI forensics is **not** reproduced, and neither is
+   whatever made CI's very first probe exceed 3 s — locally a failed login
+   answers in ~125 ms even at `cpu-load: 100`. Both remain open, and the entry
+   point to the CI failure is still the unexplained one.
+
+8. **Readiness is not the end of the evidence.** Items 1–7 all instrument a boot
+   that never became REST-ready. #69 is the other shape: the machine boots, it
+   passes `waitForBoot()`, and a *later* request is reset — so every instrument
+   above was skipped and the failure arrived as one line of `ECONNRESET`.
+   `captureRunningFailure()` (`quickchr.ts`) runs the same instrument set against
+   a machine that is still up, and does not stop, remove, or otherwise touch it —
+   the caller owns the lifecycle and may well keep testing against it.
+
+   What that capture has to add is a `trigger` (`FailureTrigger` in
+   `diagnostics.ts`): the failed operation, the error with its `code`/`errno`,
+   `sinceReadyMs`, and the credential transition that preceded it. Without it the
+   report is a healthy machine with no statement of what broke — a boot-failure
+   report explains itself through `restProbe`, and here `restProbe` is null. The
+   transition is **stated by the caller, never inferred**: the capture can see
+   the auth header, but "who this request authenticates as" and "what just
+   changed on the guest" are different facts, and only the caller knows the
+   second. An absent field reads as "not recorded", which is true; a guessed one
+   would send the next reader down a path nobody actually took.
+
+   The two report kinds share one directory and one 20-file cap, which is why
+   pruning orders by the timestamp *inside* the filename rather than by the
+   filename: whole-name sort was chronological only while every report carried
+   one prefix, and would now rank every `post-readiness-failure-` above every
+   `boot-failure-`, deleting the newest of one kind to keep the oldest of the
+   other.
+
+   Building that capture surfaced a defect in the instrument it reuses. The
+   guest snapshot's per-query budget was sized on the ~0.3 s cost of a command on
+   an already-open console session, which left the **~11.4 s serial login**
+   unbudgeted — and `bootFailureGuestExec()` then *divided* that budget across
+   credential candidates, so no attempt could finish logging in. No credential was
+   ever marked working, every later query repeated the same split, and the
+   snapshot spent its full 60 s to report `consoleReachable: false` about a guest
+   answering serial in 10 ms. It read as a broken console, which is why it went
+   unnoticed on the boot path: there the guest usually *is* unreachable, so the
+   wrong answer looked right. Queries now carry a login allowance until one of
+   them answers (`GUEST_LOGIN_ALLOWANCE_MS`), with the per-candidate floor raised
+   to the measured login cost; the 60 s snapshot budget and the 180 s forensics budget
+   are unchanged, so no test timeout moves. Locally this took a capture from
+   61.1 s / 0 of 7 queries to 14.4 s / 7 of 7.
+
+   The general rule, since it is not obvious from the numbers: a console budget
+   has two parts that differ by ~40×, and only one of them is a round-trip.
+   `provisioning.instructions.md` has the phase-by-phase measurements.
+
+   Test-side, the same finding forces a single REST client:
+   `test/integration/chr-rest.ts` is the only way integration tests reach CHR.
+   Tests previously used Bun `fetch()` and, in one file, a hand-rolled `node:http`
+   copy of `restGet` — two extra connection-reuse policies that made "client or
+   guest?" unanswerable on every reset. Collapsing them is **confound removal,
+   not a fix**: run 30507484030 reset through `restGet` too, and
+   `test/lab/bun-pool/` never reproduced the pooling bug it was built to catch.
+
 ## Design Principles
 
 ### Scope Boundary — QEMU Expert, Not Orchestrator
@@ -300,6 +538,21 @@ Provisioning via REST API is preferred (simple HTTP calls). Serial console provi
 
 **Credential resolution** (`src/lib/auth.ts`): Priority order is (1) explicit `--user`/`--password` override, (2) provisioned user from `machine.json` (`state.user`), (3) CHR default `admin:` (empty password). Both `exec()` and `rest()` on ChrInstance use this.
 
+**`machine.json` holds two kinds of fact, and `clean()` is where they diverge.**
+Some fields describe the *guest* — what accounts exist on the disk right now:
+`user`, `managedSshKey`, `disableAdmin`, and the per-instance entry in the secret
+store. Others are *intent* — what to apply the next time provisioning runs:
+`packages`, `deviceMode`, `secureLogin`. `clean()` replaces the disk with a fresh
+image, so every guest fact it recorded is false the moment it returns, and nothing
+restores them: a post-`clean()` `start()` reaches `_launchExisting(…, undefined)`
+because `lastStartedAt` is set, so the erased account is never recreated. `clean()`
+therefore clears the guest facts (and deletes the managed keypair under
+`<machineDir>/ssh/`) and keeps the intent. Without that, credential resolution
+kept preferring `state.user` and authenticated every REST call, exec, and SCP as a
+user RouterOS had deleted — the confound removed ahead of #79's QEMU-version
+experiment. Priority (3) above is what a cleaned machine must land on, and it is
+the truth of a fresh CHR image: `admin` with an empty password.
+
 Output formatting via `--json` flag. RouterOS trick for structured output: wrap commands in `[:serialize to=json [<routeros-cmd>]]` to get JSON from any CLI command. For REST-to-CLI mapping, see tikoci/restraml `lookup.html`.
 
 ### Examples Philosophy

package/MANUAL.md

@@ -199,9 +199,17 @@ credentials from the secret store.
 #### `clean <name>`
 
 Resets the machine to a fresh image: stops QEMU, re-copies the boot
-image from cache, recreates extra disks per the saved config, deletes
-EFI vars (UEFI re-initializes), and clears per-instance credentials.
-Provisioning options remain in `machine.json` and re-run on next `start`.
+image from cache, recreates extra disks per the saved config, and clears
+every credential the old disk held — the per-instance secret-store entry,
+the provisioned `user`, the managed SSH keypair, and the `disableAdmin`
+flag. The next `start` therefore logs in as factory `admin` with an empty
+password. EFI vars are deliberately **kept** (they store boot order, not
+guest state; wiping them costs an arm64 device rescan).
+
+`clean` does not re-provision: the boot options in `machine.json`
+(`packages`, `deviceMode`, `secureLogin`) are re-applied only on a machine
+that has never started, so a cleaned machine comes back factory-fresh. Pass
+the provisioning flags again, or `remove` and re-create, to get them back.
 
 ### Inspection
 
@@ -398,12 +406,18 @@ Precedence per key (highest wins): CLI flag > `QUICKCHR_<KEY>` env var >
 |---|---|---|---|
 | `default-channel` | `add`/`start`'s `--channel`, wizard's channel prompt | `stable`\|`long-term`\|`testing`\|`development` | `stable` |
 | `default-arch` | `add`/`start`'s `--arch`, wizard's arch prompt | `arm64`\|`x86`\|`auto` | `auto` (host native) |
+| `accel` | `add`/`start`'s `--accel`; every `detectAccel()` call in the run | `auto`\|`tcg`\|`hvf`\|`kvm` | `auto` (detect) |
 | `cache-max-size` | Auto-prune cap applied after each successful `start` | size string (e.g. `2G`, `512M`) | `2G` |
 | `timeout-extra` | `start`'s `--timeout-extra`/`-T` when omitted | non-negative integer seconds | `0` |
 | `secure-login` | `add`/`start`'s `--secure-login` when neither `--secure-login` nor `--no-secure-login` is passed | boolean | `unset` (reported as `(unset)` — see note) |
 
+Anything other than `auto` for `accel` is handed to QEMU verbatim and **skips detection
+entirely**, including the arm64-on-Apple-Silicon TCG fallback (§Platform notes) and the
+availability checks — force `kvm` on macOS and QEMU itself reports the error. That is
+deliberate: the override exists to test an accelerator the detector currently refuses.
+
 `secure-login` is the one key `settings print`/`get` report as `(unset)` rather than a
-concrete `false`, unlike the other four: the setup wizard's login prompt needs to tell
+concrete `false`, unlike the others: the setup wizard's login prompt needs to tell
 "not configured" apart from "explicitly set to false" (it only pre-highlights "Keep
 admin with no password" when the setting is concretely `false`; leaving it unset keeps
 the wizard's own recommended "managed login" highlighted). Giving it a real
@@ -412,7 +426,7 @@ the wizard's own recommended "managed login" highlighted). Giving it a real
 the setting resolves to exactly `true`.
 
 ```bash
-quickchr settings print                       # all 5 keys, resolved value + source
+quickchr settings print                       # all 6 keys, resolved value + source
 quickchr settings set default-channel testing # write one key to quickchr.env
 quickchr settings get default-arch            # print one key's resolved value
 quickchr settings reset default-channel       # delete one managed line
@@ -420,7 +434,7 @@ quickchr settings reset                       # clear all managed lines
 ```
 
 Credentials (`--add-user`, `MIKROTIK_WEB_ACCOUNT`/`MIKROTIK_WEB_PASSWORD`) are
-never stored here — `settings` only manages the 5 keys above, and does not
+never stored here — `settings` only manages the 6 keys above, and does not
 mutate any machine's `machine.json`.
 
 ### Global behavior
@@ -431,6 +445,9 @@ mutate any machine's `machine.json`.
 | `QUICKCHR_NO_PROMPT` | Force non-interactive mode (skips all wizards) |
 | `QUICKCHR_DEBUG` | Emit `[debug]` lines from the progress logger |
 | `QUICKCHR_INTEGRATION` | Required to run `test/integration/` |
+| `QUICKCHR_PRESERVE_ON_FAILURE` | `=1` keeps the machine directory after a boot timeout instead of removing it (QEMU is still stopped) |
+| `QUICKCHR_SERIAL_LOG` | `=1` tees the guest serial console to `<machineDir>/serial.log`. **Secret-bearing** — serial provisioning types the generated password in cleartext |
+| `QUICKCHR_DEEP_BOOT_DIAGNOSTICS` | `=1` lets a boot-timeout capture write to the guest (a mangle counting rule + host probes) to localize a silent packet drop. Skipped when `QUICKCHR_PRESERVE_ON_FAILURE=1` |
 | `MIKROTIK_WEB_ACCOUNT`, `MIKROTIK_WEB_PASSWORD` | License credentials fallback |
 | `NO_COLOR` | Disable ANSI styling |
 
@@ -923,16 +940,20 @@ arch:
 |---|---|---|
 | macOS x86_64 | x86 | HVF |
 | macOS x86_64 | arm64 | TCG |
-| macOS arm64 (native bun) | arm64 | HVF |
+| macOS arm64 (native bun) | arm64 | TCG |
 | macOS arm64 (native bun) | x86 | TCG |
 | macOS arm64 (Rosetta bun) | arm64 | TCG |
+| macOS arm64 (Rosetta bun) | x86 | TCG |
 | Linux x86_64 + `/dev/kvm` writable | x86 | KVM |
 | Linux aarch64 + `/dev/kvm` writable | arm64 | KVM |
 | any other combo | any | TCG |
 
 `Bun` running under Rosetta on Apple Silicon reports `process.arch ===
-"x64"`, so HVF for arm64 is skipped (architecture mismatch). Run with
-native arm64 Bun to get arm64 HVF.
+"x64"`, so quickchr checks `sysctl.proc_translated` to identify the physical
+arm64 host. Auto-selection uses TCG for both guest architectures there: HVF
+cannot virtualize x86 across architectures, and current arm64 CHR images need
+AArch32 userspace support that Apple Silicon does not provide. Use `--accel`
+(or `QUICKCHR_ACCEL` / the `accel` setting) only to override this deliberately.
 
 ### Boot timeout
 
@@ -996,7 +1017,7 @@ managed via `quickchr settings get|set|print|reset` (§3, Meta). Atomic writes
 preserve every other line (comments, blank lines, foreign vars) byte-for-byte.
 `reset <key>` deletes the line entirely rather than blanking it, since a blank
 `KEY=` line is still "set" to a bash-sourced file. Never stores credentials —
-the 5 managed keys are all structural preferences, not secrets.
+the 6 managed keys are all structural preferences, not secrets.
 
 ### Environment variables consumed
 
@@ -1006,8 +1027,12 @@ the 5 managed keys are all structural preferences, not secrets.
 | `QUICKCHR_NO_PROMPT` | CLI dispatcher — forces non-interactive mode |
 | `QUICKCHR_DEBUG` | progress logger — emits `[debug]` lines |
 | `QUICKCHR_INTEGRATION` | `bun:test` gate for `test/integration/` |
+| `QUICKCHR_PRESERVE_ON_FAILURE` | `quickchr.ts` — skip `remove()` on `BOOT_TIMEOUT` so the machine dir survives for diagnosis |
+| `QUICKCHR_SERIAL_LOG` | `qemu.ts` — adds `logfile=` to the serial chardev (writes cleartext provisioning passwords) |
+| `QUICKCHR_DEEP_BOOT_DIAGNOSTICS` | `guest-snapshot.ts` — allows the guest-mutating counting-rule probe during boot-failure capture |
 | `QUICKCHR_DEFAULT_CHANNEL` | `settings.ts` — env tier for the `default-channel` setting |
 | `QUICKCHR_DEFAULT_ARCH` | `settings.ts` — env tier for the `default-arch` setting |
+| `QUICKCHR_ACCEL` | `settings.ts` / `platform.ts` — env tier for the `accel` setting; forces `-accel` and bypasses detection |
 | `QUICKCHR_CACHE_MAX_SIZE` | `settings.ts` — env tier for the `cache-max-size` setting |
 | `QUICKCHR_TIMEOUT_EXTRA` | `settings.ts` — env tier for the `timeout-extra` setting |
 | `QUICKCHR_SECURE_LOGIN` | `settings.ts` — env tier for the `secure-login` setting |
@@ -1151,8 +1176,12 @@ state deliberately) if a built-in service needs a different host port.
 
 `quickchr clean <name>` re-copies the boot image from cache and
 recreates extra disks per `machine.json`. Snapshots are gone (they live
-inside the qcow2 file). Per-instance credentials are wiped so the next
-boot re-provisions them.
+inside the qcow2 file). Every credential the old disk held is wiped —
+stored instance credentials, `state.user`, the managed SSH keypair under
+`<machineDir>/ssh/`, and `disableAdmin` — because the accounts they named
+went with the disk. The machine comes back as a factory CHR: `admin` with
+an empty password, which is what `rest()`, `exec()`, and `inspect` resolve
+to afterwards. Nothing is re-provisioned automatically.
 
 ### Snapshots
 
@@ -1255,6 +1284,9 @@ prevents orphan disks accumulating across attempts.
   (MNDP) on the host via a `socket-connect` NIC.
 - **docs/qga-x86-macos-qemu10-investigation.md** — root cause for the
   QGA-on-macOS-arm64 limitation.
+- **docs/m4-hvf-arm64-investigation.md** — **open** investigation into why arm64
+  CHR panics under HVF on Apple M4 (why arm64 falls back to TCG there); evidence
+  ledger, competing theories, and the test plan to settle them.
 - **`.github/instructions/`** — agent-facing rules (testing, qemu,
   routeros-rest, provisioning, ci, bun-http, general).
 - **`examples/`** — runnable scripts (`bun run`): `quickstart` (single-CHR smoke),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tikoci/quickchr",
-  "version": "0.4.5",
+  "version": "0.4.7",
   "description": "CLI and library to download, launch, and manage MikroTik CHR virtual machines via QEMU",
   "module": "src/index.ts",
   "main": "src/index.ts",

package/src/cli/index.ts

@@ -157,6 +157,26 @@ export async function resolveTimeoutExtraMs(flags: Record<string, string | boole
 	return seconds !== undefined ? seconds * 1000 : undefined;
 }
 
+/** Apply an explicit `--accel <auto|tcg|hvf|kvm>` as the process-level accelerator
+ *  override, so every detectAccel() call in this run (QEMU args, boot timeouts,
+ *  respawn logic) agrees on one value. Absent flag → the accel setting's own
+ *  env/file/default tiers apply, so this only installs the top tier. Throws
+ *  INVALID_SETTING_VALUE on an unrecognized mode. Shared by cmdAdd and cmdStart. */
+export async function applyAccelFlag(flags: Record<string, string | boolean | string[]>): Promise<void> {
+	const value = flags.accel;
+	if (value === undefined) return;
+	const raw = Array.isArray(value)
+		? value.at(-1) ?? ""
+		: typeof value === "string"
+			? value
+			: "";
+	const [{ parseAccelMode }, { setAccelOverride }] = await Promise.all([
+		import("../lib/types.ts"),
+		import("../lib/platform.ts"),
+	]);
+	setAccelOverride(parseAccelMode(raw));
+}
+
 /** Resolve --secure-login/--no-secure-login, falling back to the secure-login setting
  *  only when neither flag was passed. Shared by cmdAdd, cmdStart, and cmdStart's
  *  --all bulk-restart path. */
@@ -535,6 +555,7 @@ async function cmdAdd(argv: string[]) {
 	const { QuickCHR } = await import("../lib/quickchr.ts");
 	const { bold, formatPorts, formatNetworks, dim } = await import("./format.ts");
 	const { resolveSetting } = await import("../lib/settings.ts");
+	await applyAccelFlag(flags);
 
 	const opts: StartOptions = {
 		version: flag(flags, "version"),
@@ -1175,6 +1196,8 @@ async function cmdSetup() {
 async function cmdStart(argv: string[]) {
 	const { flags, positional } = parseFlags(argv);
 	applyTimeoutExtraShortFlag(argv, flags, positional);
+	// Before the --all branch: both start paths must see the same override.
+	await applyAccelFlag(flags);
 
 	// --all: start every stopped machine in background
 	if (flagBool(flags, "all")) {
@@ -2720,6 +2743,9 @@ Options:
   --arch <arch>         Architecture: arm64, x86, auto (default: host native)
   --cpu <n>             vCPU count (default: 1)
   --mem <mb>            RAM in MB (default: 512)
+  --accel <mode>        Force QEMU accelerator: auto|tcg|hvf|kvm (default: auto).
+                        Anything but 'auto' bypasses detection, including the
+                        arm64-on-Apple-Silicon TCG fallback (tikoci/quickchr#97).
 	--boot-disk-format <f> Boot disk format: qcow2|raw (default: qcow2)
 	--boot-size <size>    Resize boot disk (e.g., 512M, 2G) — qcow2 only, requires qemu-img
 	--add-disk <size>     Add an extra blank qcow2 disk (repeatable, requires qemu-img)
@@ -2802,6 +2828,9 @@ Options:
   --name <name>         Instance name
   --cpu <n>             vCPU count (default: 1)
   --mem <mb>            RAM in MB (default: 512)
+  --accel <mode>        Force QEMU accelerator: auto|tcg|hvf|kvm (default: auto).
+                        Anything but 'auto' bypasses detection, including the
+                        arm64-on-Apple-Silicon TCG fallback (tikoci/quickchr#97).
 	--boot-disk-format <f> Boot disk format: qcow2|raw (default: qcow2)
 	--boot-size <size>    Resize boot disk (e.g., 512M, 2G) — qcow2 only, requires qemu-img
 	--add-disk <size>     Add an extra blank qcow2 disk (repeatable, requires qemu-img)
@@ -3056,6 +3085,8 @@ Managed keys:
   default-channel   Default --channel for 'add'/'start' and the setup wizard.
                     stable|long-term|testing|development (default: stable)
   default-arch      Default --arch. arm64|x86|auto (default: auto = host native)
+  accel             QEMU accelerator for 'add'/'start'.
+                    auto|tcg|hvf|kvm (default: auto = detect)
   cache-max-size    Auto-prune cap applied after each successful start.
                     Size string (e.g. 2G, 512M). (default: 2G)
   timeout-extra     Default for 'start's --timeout-extra/-T when omitted.

package/src/index.ts

@@ -108,6 +108,7 @@ export type { QgaCommandInfo } from "./lib/qga.ts";
 
 // Console exec utilities
 export { consoleExec, isConsoleReady, stripAnsi } from "./lib/console.ts";
+export type { ConsoleExecResult } from "./lib/console.ts";
 
 // Channel utilities
 export { isQgaReady } from "./lib/channels.ts";

package/src/lib/channels.ts

@@ -15,6 +15,7 @@ import type { Arch, ChannelTcpEndpoint, QgaCommand } from "./types.ts";
 import { QuickCHRError } from "./types.ts";
 import { qgaProbe, qgaRawCommand } from "./qga.ts";
 import { qgaKvmWarning } from "./platform.ts";
+import { newMonitorPhaseTimings, summarizeMonitorPhases } from "./diagnostics.ts";
 
 /**
  * Resolve the IPC path for a QEMU channel (monitor, serial, qga).
@@ -87,9 +88,19 @@ export async function monitorCommand(
 	const endpoint = channelEndpoint(machineDir, "monitor", portBase);
 
 	return new Promise((resolve, reject) => {
+		// Per-phase timings so a timeout says *where* the round-trip stalled rather
+		// than just that it did (#80). Only read on the timeout path — a successful
+		// command pays nothing but a few Date.now() calls.
+		const started = Date.now();
+		const phases = newMonitorPhaseTimings();
+		const since = () => Date.now() - started;
+
 		const timeout = setTimeout(() => {
 			socket.destroy();
-			reject(new QuickCHRError("BOOT_TIMEOUT", "Monitor command timed out"));
+			reject(new QuickCHRError(
+				"BOOT_TIMEOUT",
+				`Monitor command timed out after ${timeoutMs}ms: "${command}" — ${summarizeMonitorPhases(phases)}`,
+			));
 		}, timeoutMs);
 
 		let buffer = "";
@@ -99,13 +110,20 @@ export async function monitorCommand(
 			? connect({ path: endpoint })
 			: connect(endpoint.port, endpoint.host);
 
+		socket.on("connect", () => { phases.connectedAtMs = since(); });
+
 		socket.on("data", (data) => {
+			phases.bytesReceived += data.length;
+			phases.firstByteAtMs ??= since();
+			if (sentCommand) phases.responseFirstByteAtMs ??= since();
 			buffer += data.toString();
 			// Wait for the (qemu) prompt before sending command
 			if (!sentCommand && buffer.includes("(qemu)")) {
 				sentCommand = true;
+				phases.promptAtMs = since();
 				buffer = "";
 				socket.write(command + "\n");
+				phases.commandWrittenAtMs = since();
 			} else if (sentCommand && buffer.includes("(qemu)")) {
 				// Got response — strip the prompt, the command echo, and ANSI noise
 				clearTimeout(timeout);
@@ -122,7 +140,10 @@ export async function monitorCommand(
 			if (sentCommand) {
 				resolve(cleanMonitorResponse(buffer.replace(/\(qemu\)\s*/g, "")));
 			} else {
-				reject(new QuickCHRError("MACHINE_STOPPED", "Monitor socket closed before command could be sent"));
+				reject(new QuickCHRError(
+					"MACHINE_STOPPED",
+					`Monitor socket closed before command could be sent — ${summarizeMonitorPhases(phases)}`,
+				));
 			}
 		});