@tikoci/quickchr 0.4.2 → 0.4.3

package/DESIGN.md

@@ -44,7 +44,7 @@ Modules (src/lib/)      ← qemu, images, versions, network, state, ...
    - **Not applied to `_launchExisting`** (the restart-existing-machine path) — that path also spawns + `waitForBoot`s but has never been observed to flake. Extend the same respawn there only with evidence.
    - **Timeout factor left at 1.5×** (`accelTimeoutFactor`, `src/lib/platform.ts`) rather than inflated further — the respawn is the recovery mechanism, not a bigger ceiling.
 
-   How to tell it's firing: grep CI `qemu.log`/run logs for `respawning QEMU once`. A frequent occurrence, or a `BOOT_TIMEOUT` that *survives* the respawn (or appears on the `_launchExisting` path), is the signal to stop treating it as a flake and find the real cause. Tracked in `BACKLOG.md` (boot-respawn watch-item).
+   How to tell it's firing: grep CI `qemu.log`/run logs for `respawning QEMU once`. A frequent occurrence, or a `BOOT_TIMEOUT` that *survives* the respawn (or appears on the `_launchExisting` path), is the signal to stop treating it as a flake and find the real cause. Tracked as a watch-item in [#45](https://github.com/tikoci/quickchr/issues/45).
 
 9. **Resilient downloads: normal `fetch` first, public-DNS IPv4 as a failback** — All fetches to MikroTik's `upgrade`/`download` hosts go through `fetchResilient()` (`src/lib/net.ts`), never bare `fetch()`. The intent is narrow: a **plain `fetch` is the path** — dual-stack (happy eyeballs), on par with curl/most tools, and honoring local DNS (`/etc/hosts` pins, VPN/split-horizon, mirror redirects, IPv6-only egress) — and we layer one **failback** beneath it to ride out *DNS misconfiguration somewhere in the environment*, wherever it comes from. We deliberately do **not** invert this (public-DNS+IPv4 first) — that would override local DNS on every machine and could introduce its own failures (e.g. an IPv4-only connect on an IPv6-only network). Normal first keeps us on par with most tools; the failback only adds smarts when the normal path actually breaks.
 
@@ -66,6 +66,17 @@ Modules (src/lib/)      ← qemu, images, versions, network, state, ...
 | +5     | WinBox     | 8291       |
 | +6—9   | Custom     | —          |
 
+On Windows, the monitor/serial/QGA channels use TCP localhost at `portBase+6/+7/+8` (Winsock can't bind `\\.\pipe\` paths), so the `+6..+8` "Custom" slots are effectively reserved there.
+
+### Allocation rationale & open questions
+
+Ports are assigned in fixed 10-port blocks from `DEFAULT_PORT_BASE` (9100), scanning existing machines and probe-binding to avoid collisions. Two rough edges are **maintainer decisions, not yet settled** — don't "fix" them unilaterally, since they change the persisted-state / port contract (tracked as `needs-decision` work):
+
+1. **Base 9100 is a poor default** — collides with JetDirect/PDL (printers) and trains agents to memorize "quickchr = 9100 for REST", which only holds for the *first* instance (the second gets 9110). Open: random base in a clean high range (persisted per machine) vs caller-requested range vs both; and whether that's a breaking change or a setting with 9100 as the default. → [#56](https://github.com/tikoci/quickchr/issues/56)
+2. **Fixed blocks are overfit** — extras bolt on at `+6..+9` and collide with the Windows IPC offsets above (one slot before spilling); manual `extraPorts` bypass collision checking. Open: dynamic variable-size blocks vs a fixed core pool + a separate extras pool. → [#57](https://github.com/tikoci/quickchr/issues/57)
+
+Current consumers are all first-party tikoci projects (centrs, donny, restraml), so a port-scheme migration can be coordinated across them — but `@tikoci/quickchr` is published and public, so a change to the persisted port contract still needs an issue, a CHANGELOG/docs note, and (for the `ChrInstance`/`StartOptions` surface) a deprecation/migration path, not a silent break.
+
 ## Storage Layout
 
 ```text
@@ -78,15 +89,21 @@ Modules (src/lib/)      ← qemu, images, versions, network, state, ...
 │   └── 7.22.1-arm64-1/
 │       ├── machine.json       # Config + state
 │       ├── disk.img           # Working copy
-│       ├── efi-vars.fd        # UEFI vars (arm64)
+│       ├── efi-vars.qcow2     # UEFI vars (arm64; qcow2 so savevm works — #31)
 │       ├── monitor.sock       # QEMU monitor
 │       ├── serial.sock        # Serial console
 │       ├── qga.sock           # QGA (x86 only)
 │       ├── qemu.pid           # PID file
 │       └── qemu.log           # Output log
-└── config.json                # Global config
 ```
 
+**Global settings** live separately, under the XDG **config** tier (not the data tree
+above): `~/.config/quickchr/quickchr.env`, dotenv-style (`QUICKCHR_KEY=value` lines).
+Managed via `quickchr settings get|set|print|reset`. Precedence per key: CLI flag >
+`QUICKCHR_<KEY>` env var > `quickchr.env` > built-in default. The 5 managed keys:
+`default-channel`, `default-arch`, `cache-max-size`, `timeout-extra`, `secure-login`.
+See MANUAL.md's CLI reference and environment-variables sections for the full surface.
+
 ## Platform Support
 
 | Platform               | x86 CHR | arm64 CHR | Notes |
@@ -105,31 +122,63 @@ Modules (src/lib/)      ← qemu, images, versions, network, state, ...
 
 ## CI System
 
-**Workflows**: `.github/workflows/{ci,verify-extended,publish}.yml`. Full artifact map and
-failure-diagnosis guide live in `.github/instructions/ci.instructions.md` — this section is
-the high-level rationale only.
-
-**`ci.yml`** — core quality gate on every push/PR to `main`:
-
-```text
-lint (ubuntu)  ──┐
-unit-tests (ubuntu, --coverage) ──┴→ integration-x86 (ubuntu) → windows-unit-tests (windows-latest)
-```
-
-`lint` and `unit-tests` run in parallel. `integration-x86` boots an x86 CHR (KVM if
-available, else TCG) and gates on both. `windows-unit-tests` runs last — if the core is
-broken, Windows runner minutes add no signal. Only x86 integration runs on every push;
-cross-arch and macOS are too slow/fragile for the per-push gate.
-
-**`verify-extended.yml`** — `workflow_dispatch` only. Independent jobs (no cross-`needs:`)
-for the platforms kept out of the per-push gate: linux/aarch64, macOS (arm64 + x86), and
-Windows. Dispatch inputs `arm64` / `macos` / `windows` select platforms; `test-filter`
-narrows to specific test files for fast iteration. Each runner boots a CHR matching its
-native arch — `detectAccel()` selects KVM/HVF/TCG automatically, no per-runner overrides.
-
-**`publish.yml`** — triggers on `v*` tags (or dispatch). Runs lint + unit + x86 integration +
-windows-unit before `npm publish` (`--tag next` for odd/pre-release minors, `--tag latest`
-for even/stable). See "Release Process" in `ci.instructions.md`.
+**Workflows**: `.github/workflows/{ci,main,sweep,integration,release,ros-versions,lint-powershell}.yml`.
+Full artifact map, dispatch recipes, and failure-diagnosis guide live in
+`.github/instructions/ci.instructions.md` — this section is the high-level rationale only.
+
+**The 2026-07 refactor (#29)** replaced the organically-grown scheme (integration on every
+PR push, three parallel copies of the runner logic, `continue-on-error` green-washing)
+with a layered design:
+
+- **`ci.yml`** — fast PR/push gate (~3-5 min, no QEMU): lint ∥ unit+coverage ∥ Windows
+  unit ∥ `Integration freshness` (PR-only, required). Integration left the PR path
+  because a 25-minute CHR suite per PR push bought little signal per minute; the quality
+  bar moved to main.
+- **`main.yml`** — full integration suite + examples smoke (+ PowerShell lint) on
+  linux/x86_64 + linux/aarch64 (KVM when available; hosted arm64 may fall back to TCG)
+  on every push to `main`; examples are part of the
+  default flow, not a weekly extra. The **freshness gate** (`scripts/ci-freshness.ts`) makes this honest:
+  PRs merge only while the latest completed main run is green, so a red main visibly
+  blocks everything instead of rotting in the Actions list. Superseded push runs cancel;
+  cancelled runs carry no signal (the gate skips them).
+- **`sweep.yml`** — weekly all-platform sweep (+ examples smoke). A separate file from
+  main.yml *by design*: its TCG legs (macos-x86, windows-x86 — bounded to the anchor
+  smoke subset via `tcg-smoke: true` to cap weekly cost) may red without blocking PRs,
+  but are never green-washed — red is red. On manual `integration.yml` dispatches TCG
+  legs run the FULL suite by default (`tcg-smoke` defaults off): `platforms=all` means
+  all platforms, full set — smoke is opt-in, never an implicit narrowing.
+- **`integration.yml`** — the single reusable integration unit (`workflow_call` +
+  `workflow_dispatch`). A plan job resolves **`platforms` × `routeros-targets`** (both
+  comma lists; `gating`/`all` platform aliases) into one cross-OS matrix — one dispatch
+  can cover e.g. gating platforms × three RouterOS versions, no wrapper jobs or repeat
+  dispatches. `test-filter` narrows; `run-examples` defaults ON. Each runner boots its
+  native CHR arch — `detectAccel()` picks KVM/HVF/TCG. Agents dispatch this to ground
+  platform hypotheses without waiting for a PR cycle.
+- **`release.yml`** — one-click release: freshness gate (no suite re-run — main is kept
+  continuously release-able) + non-empty CHANGELOG `[Unreleased]` →
+  `scripts/release-prep.ts` bump/rollover → tag + GitHub Release + `npm publish
+  --provenance` (odd minor → `next`, even → `latest`).
+- **`ros-versions.yml`** — daily: new RouterOS versions (per channel) with no linux-x86
+  record in `ci-data/tested-versions.json` ride the `routeros-targets` matrix of a
+  single integration dispatch.
+
+**Metrics as byproduct (#30)**: the library appends every successful boot to
+`<dataDir>/boot-log.ndjson` (and stamps `lastAccel`/`lastBootMs` into machine.json);
+integration jobs assemble that + per-file timing into `metrics.ndjson`
+(`scripts/ci-metrics.ts`); collect-metrics callers push per-run files + the
+`tested-versions.json` rollup to the orphan **`ci-data`** branch. Never a second test
+run; never affects pass/fail — the aggregate job is best-effort by contract (fold/push
+failures warn and stay green, because a red there would trip the PR freshness gate over
+side-band data). A full run marks exactly **its target's resolved version** tested —
+never every version it happened to boot (upgrade/pinned-channel tests boot others
+incidentally; crediting those suppressed the scheduler for versions no full suite ever
+targeted). The per-run files are the source of truth; `ci-metrics refold` rebuilds the
+rollup after fold-logic changes.
+
+**Merge policy**: squash-only, PR title → main commit subject (write PR titles as
+conventional commits), PR body → commit body, branches auto-delete. Review threads must
+be resolved before merge (`required_conversation_resolution`) and automated reviews must
+have actually posted — see CONTRIBUTING.md "Pull Requests & Merging".
 
 **Coverage**: `unit-tests` parses `bun test --coverage` output and compares against thresholds
 (default 75% functions, 60% lines). Failures emit `::warning::` annotations but do NOT block
@@ -138,13 +187,14 @@ merges (`continue-on-error: true`). Thresholds are overridable via dispatch inpu
 
 **Artifacts**:
 - `coverage-report` — full per-file coverage table (14 days)
-- `integration-logs-{platform}` — bun test output + machine.json + qemu.log (7 days)
+- `{integration|main|sweep}-logs-<platform>` — bun test output + timing + metrics.ndjson +
+  machine.json + qemu.log (7 days); durable timing lives on the `ci-data` branch
 
 ## Design Principles
 
 ### Scope Boundary — QEMU Expert, Not Orchestrator
 
-quickchr manages individual CHR instances. Multi-router topologies, test matrices, and workflow orchestration are **out of scope** for the CLI and library. Provide `examples/` with Makefiles and `bun:test` scripts to inspire, but don't build a framework. Users (and their AI agents) compose quickchr instances into whatever topology they need — we give them reliable building blocks.
+quickchr manages individual CHR instances. Multi-router topologies, test matrices, and workflow orchestration are **out of scope** for the CLI and library. Provide `examples/` as runnable scripts (a `bun run`-able `<name>.ts` plus `.sh`/`.ps1`/`.py` siblings; `grounding/` is the one `bun:test` reference) to inspire, but don't build a framework. The example convention lives in `.github/instructions/examples.instructions.md`; Python examples prefer `uv run` over a venv. Users (and their AI agents) compose quickchr instances into whatever topology they need — we give them reliable building blocks.
 
 ### Networking — Discover, Don't Configure
 
@@ -183,6 +233,32 @@ multicast port; two CHRs on one group don't discover each other and host capture
 gets nothing. mcast still works on Linux/CI. Prefer `socket-connect` for host
 capture on any platform. Evidence: `test/lab/mndp/REPORT.md`.
 
+#### Guest→Host UDP via the Gateway — No Forward (discovered 2026-06-25)
+
+The dual of `hostfwd`: SLIRP's gateway `10.0.2.2` *is* the host from inside the
+guest, so guest-originated UDP to `10.0.2.2:<port>` reaches a host socket bound on
+loopback `<port>` with **no forward and no extra NIC**. This generalizes the TZSP
+path (`tzspGatewayIp`/`captureInterface`) — it is not TZSP-specific and reaches an
+ordinary bound socket, not just a `tshark`/pcap capture. The host socket **must be
+unconnected**: SLIRP re-emits from a rewritten loopback source
+(`127.0.0.1:<ephemeral>`), which a `connect()`-ed socket would filter. This closed
+centrs' btest UDP-coverage gap (issue #18) with no new quickchr feature — it was a
+discoverability gap. Recipe: `docs/networking-recipes.md`; evidence:
+`test/lab/gateway-udp/REPORT.md`.
+
+#### Port-Range Forwards — Explicit-Host-Only
+
+`--forward`/`extraPorts` accept a range (`name:hostStart-hostEnd[:guest…][/proto]`)
+that expands to one `PortMapping`/`hostfwd` per port — QEMU has no native range.
+**Design choice:** range host ports must be *explicit*. Auto-allocation draws from
+the per-instance 10-port block, which cannot guarantee a contiguous run; requiring
+explicit host ports keeps the change additive (the existing
+`validateExplicitExtraPorts` collision check covers them) and avoids reworking the
+port-block contract. A 64-port cap bounds the generated `hostfwd` string. For
+guest-chosen *unpredictable* ports, the gateway path (above) is the better fit for
+the guest→host direction. `expandForwardSpec` is the range-aware entry point;
+`parseForwardSpec` stays single-port for backward compatibility.
+
 ### Platform Priority
 
 macOS → Linux → Windows. Mac and Linux share most code paths with minor `#ifdef`-style branches. Windows is tracked but lower priority — larger RouterOS admin audience there, but fewer recipes and harder to test. Windows CI runner planned after the existing macOS/Linux matrix is stable.
@@ -191,6 +267,13 @@ macOS → Linux → Windows. Mac and Linux share most code paths with minor `#if
 
 Tighten before expanding. New subcommands (`logs`, `exec`, `console`) wait for a full command tree review. The CLI should be discoverable without paging `--help` — shell completions help more than a long command list. Use multipass and virsh as reference points for symmetry, not to copy.
 
+**Command-surface principles** (locked — these shape every new subcommand):
+
+- **Interactive prompts are confined to `setup`.** Every other command is non-interactive — no selectors. Without a `<name>` argument, print the list + a tip; don't prompt.
+- **`start`/`stop` are pure operations** — no wizard, no creation. `add` creates; `setup` is the wizard.
+- **`set`/`get` are for machine config, not re-provisioning.** After first provisioning, don't add commands that re-provision — the surface grows and each post-provision capability needs its own RouterOS edge-case testing. A drifted machine is recreated, not mutated (see *Out of Scope*).
+- **`--json` on read commands only** — same content as console output (richer metadata OK), pipe-friendly for `jq`. No `--yaml`/`--serialize`/TSV/CSV; callers pipe `--json` through `jq`/`yq`. For `exec`, `--json` wraps the quickchr response — the RouterOS result stays a string (use `:serialize` in-script for structured RouterOS output).
+
 ### RouterOS Verification
 
 Always read back what we write. One extra REST API call after a provisioning action (license, user, package) catches version-specific command drift early. Surface errors with actionable hints rather than silent failures.
@@ -220,13 +303,54 @@ Output formatting via `--json` flag. RouterOS trick for structured output: wrap
 
 ### Examples Philosophy
 
-Three representations of each scenario, targeting different audiences:
-- **Makefile** — recipe-driven, targets as documentation (tikoci tradition, see tikoci/netinstall). Agents read targets; humans run `make`.
-- **bun:test** — library API, TypeScript. First-class integration tests. The "source of truth."
-- **Python** — subprocess around CLI. The language agents and network engineers both reach for. Demonstrates quickchr is a real tool, not just a library.
-
-Building examples early is a form of "anchor testing" for the CLI surface — it finds ergonomic issues before we commit to new commands.
+`examples/` is **load-bearing agent-onboarding surface** — agents (and humans) open
+it before `src/lib/`, so a *wrong* example is worse than none: it teaches the wrong
+lesson and costs a later code-review to unwind. Examples are therefore held to the
+same bar as the code, and a broken one **gates** extended verification.
+
+Each `examples/<name>/` is a **runnable artifact that does something real** against a
+CHR, in the canonical shape (full rules in
+[`.github/instructions/examples.instructions.md`](.github/instructions/examples.instructions.md)):
+
+- **`<name>.ts`** — the **primary**: a `bun run`-able script using the library API,
+  with `runExample()` guaranteeing teardown (success *or* failure). Not a test.
+- **`<name>.sh` / `<name>.ps1`** — the **CLI** mirror (POSIX `sh` + PowerShell),
+  sourcing `examples/common.{sh,ps1}`. New examples ship both; existing ones add
+  `.ps1` where the CLI flow is simple.
+- **`<name>.py`** — optional CLI driver for a non-TS audience, run with `uv run`.
+- **`grounding/` is the one `bun:test` example** — kept as a test on purpose, because
+  there the *assertions are the documentation* (it's the "how to write CHR
+  integration tests" reference). Everywhere else, an agent can wrap a script in
+  `test()` trivially, so a runnable script is the better teaching surface.
+
+Makefiles were the old convention and are now **disallowed** in `examples/`
+(`scripts/validate-examples.ts` enforces this) — they mixed CLI orchestration with
+raw `scp`/`curl`/`ssh`, the opposite of "one example teaches one capability."
+
+Building examples early is a form of "anchor testing" for the CLI surface — it finds
+ergonomic issues before we commit to new commands, and each gap surfaced gets a
+"friction found" note in the example's README plus a GitHub issue rather than
+being papered over (this pass surfaced the missing `quickchr cp`, #23). Coverage of
+the CLI/library surface is tracked in
+[`examples/COVERAGE.md`](examples/COVERAGE.md).
+
+### Agent-Friendliness — Discoverability Over Features
+
+A recurring lesson from downstream agents (donny, centrs, restraml): the friction is usually **finding** an existing capability, not a missing one. Issue #18 (centrs) nearly rebuilt UDP forwarding, `socket-connect` L2, and the guest→host gateway — all of which already existed — because the only way to choose among them was reading `src/lib/network.ts`. Rule: **every CLI-documented capability also needs a library-facing, by-goal surface** — JSDoc at the call site, a by-goal recipe (`docs/networking-recipes.md`), and coverage in the `routeros-quickchr` skill — or agents won't find it. Connection handoff for harnesses goes through `ChrInstance.descriptor()` / `quickchr inspect` / `quickchr env` (credential-bearing by design), never by reading `machine.json`.
+
+### Out of Scope (decided)
+
+Explicitly rejected, with rationale, so they aren't re-proposed:
+
+- **Cloud deployment** — `tikoci/chr-armed` already does OCI + AWS; revisit only once local CHR is solid and provisioning/image layers can be reused.
+- **Multi-CHR orchestration** — building blocks + `examples/`, not a framework (see *Scope Boundary*). Users/agents compose topologies.
+- **`quickchr upgrade <name>` / post-provisioning mutation** — replaced by a future *config audit/verify* report (flags drift; user recreates). Re-provisioning has too many failure modes (version bumps, rotated creds, changed deps).
+- **Packaging (Homebrew/Deb)** and **service management (launchd/systemd)** — lower priority than core; optional later.
+- **Machine templates** — CLI flags are the template, API objects are reusable, the wizard always prompts. No separate template system.
+- **`--no-ansi` flag** — ANSI is fine in text output (`grep`/`jq` cope); the real discipline (separate presentation from content) belongs in a centralized error-message surface instead.
+- **`machine.json` → YAML** — staying JSON (pretty-printed) for `jq` users; YAML adds complexity without a matching benefit.
+- **Separate multi-version/arch matrix runner** — the CI matrix + `examples/version-matrix` cover it.
 
 ### Document Maintenance
 
-DESIGN.md and BACKLOG.md are living documents. At the end of any significant work session, agents should review whether new implementation details, design decisions, or discovered constraints belong in DESIGN.md, and whether completed/new work should update BACKLOG.md. Treat this as a lightweight checklist, not a gate.
+DESIGN.md is the living home for design decisions and rationale. At the end of any significant work session, agents should review whether new implementation details, design decisions, or discovered constraints belong here. Open/close work in **GitHub Issues** — not BACKLOG.md (see CONTRIBUTING.md "Tracking work"); record grounded RouterOS/QEMU behaviour facts in the narrowest scoped doc (`.github/instructions/*.md`, `docs/`, or `test/lab/<topic>/REPORT.md`); add a CHANGELOG.md entry for user-facing changes. Treat this as a lightweight checklist, not a gate.