@keepsnappy/agent 0.0.0-stage → 0.1.0-alpha.11

package/README.md

@@ -1,3 +1,261 @@
-# Temporary Holding Version
+# @keepsnappy/agent
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+The `keepsnappy` host CLI. It runs the customer's own test command several times, collects the perf-lab data the app wrote during each run, turns it into metrics, and asks `@keepsnappy/stats` for a verdict. v1 is local only and measure-only: no code changes in the customer app, no driver of our own (the test command is Maestro, Appium, XCUITest, anything that exits 0 on success).
+
+## Commands
+
+```sh
+bun install && bunx tsc --noEmit && bunx vitest run
+bun run build                      # dist/src/cli.js (bin: perflab)
+
+perflab measure --repeat 10 \
+  --arms A=/builds/base.app,B=/builds/candidate.app \
+  --platform ios-sim [--device <udid>] [--bundle-id com.x.app] [--out runs/s1] [--timeout 600] \
+  [--ri-capture observations.ndjson] [--mem-window chat.open] \
+  -- maestro test flows/open-chat.yaml
+perflab verdict runs/s1 [--baseline A --candidate B]   # table + verdict.json
+perflab extract runs/s1 --ri-capture observations.ndjson   # re-derive metrics.csv (adds ri.* / mem.*)
+perflab aa runs/s1 [--reps 300 --boot 400]             # noise floor / MDE per metric
+perflab doctor                                         # xcrun, simctl, devicectl, maestro, appium, node, booted sims, devices
+```
+
+- `--arms`: `NAME=<.app/.ipa path>` (installed with `simctl install` / `devicectl device install app`, only when the arm changes) or `NAME=<bundle id>` (already installed). Without `--arms` there is one arm `A`: plain repetition, usable for `keepsnappy aa`.
+- `--arm-env 'A:KEY=VAL'` (repeatable): extra env for the test command on that arm's runs, also exported as `MAESTRO_KEY` (Maestro forwards only `MAESTRO_*` to flows). Used for same-binary arms that differ only in knobs, e.g. `--arms A=dev.perflab.reference,B=dev.perflab.reference --arm-env 'A:REF_KNOBS=…' --arm-env 'B:REF_KNOBS=…'`. The arms with their env are written to `<out>/arms.json`. Order stays ABBA.
+- `--platform ios-device` needs `--device <udid>`. The simulator default target is `booted`.
+- The test command runs as argv (no shell) with `PERFLAB_RUN_ID`, `PERFLAB_ARM`, `PERFLAB_RUN_DIR`, `PERFLAB_MARKER_URL` and `MAESTRO_PERFLAB_MARKER_URL` / `MAESTRO_PERFLAB_RUN_ID` in its environment.
+- Declared steps (Maestro): call `maestro/perflab/step.yaml` with `STEP`/`PHASE` (start|end) from your flow; `measure` adds `step:<name>.*` rows (source=declared). `--no-markers` turns the endpoint off, `--clock-offset-ms` overrides the clock estimate. See docs/research/28-step-markers-prototype.md.
+
+## `keepsnappy run` and `keepsnappy.yml` (customer entry point)
+
+`keepsnappy run` is the wrapper for the e2e command you already have (mode A in docs/research/23-execution-modes-and-data-flow.md). It runs the command N times, drops runs where it failed, pulls the app's perf-lab data after each run, derives metrics and writes the out dir. With `--baseline <earlier out dir>` it also says what changed. Nothing is built or installed by it: your CI supplies the build (with `PERFLAB=1`) and the device.
+
+```sh
+perflab run                                   # command built from keepsnappy.yml
+perflab run -- maestro test --include-tags perf e2e/flows/   # wrap your own command (config still gives bundle id, device, repetitions)
+perflab run --baseline main-out --upload      # compare with the baseline run, then send results to the ingest
+```
+
+Flags: `--config` (default `keepsnappy.yml`), `--repeat N`, `--out DIR`, `--baseline DIR`, `--upload [--ingest-url URL]`, `--wait`, `--fail-on-worse`, `--commit SHA` / `--branch NAME` (default `GITHUB_SHA` / `GITHUB_HEAD_REF` on a pull request, else `GITHUB_REF_NAME`), `--base-commit SHA`, `--device-model` / `--device-os` (default: read from the booted simulator), `--scenario NAME` (flow name for a command given after `--`), `--replay <out dir>[:ARM]` (repeatable: take recorded runs instead of running anything), `--no-markers`, `--clock-offset-ms`.
+
+Commit details: when a run set opens (`keepsnappy run --upload`, `keepsnappy farm open`), the CLI reads the subject, author
+name and committer date of `--commit` and `--base-commit` with `git log` in the checkout (argv only, 5 s timeout) and
+sends them, plus the pull request number and URL from `GITHUB_EVENT_PATH` on a `pull_request` event. The dashboard
+shows commits as title, author and time instead of the bare hash. Without git, a shallow clone that lacks the commit,
+or outside a pull request, the fields are left out and nothing fails. On a GitHub `pull_request` event `GITHUB_SHA` is
+the merge commit, so its title is "Merge … into …"; pass `--commit ${{ github.event.pull_request.head.sha }}` (with the
+head commit fetched) to get the branch's own commit.
+
+Journey titles: on `--upload` the run set carries `journeys[]` — per flow its key (the file stem), `source_path` (the flow file relative to the git repo root) and a title and description: from the `{path, title, description}` form of `flows.paths`, else the flow's own `name:` (when it differs from the stem) and the first paragraph of the comment that opens the file. Given values update the journey on the dashboard; absent ones keep what is stored.
+
+Flows: with `runner: maestro` every flow file is run on its own (one `maestro test` per file per repetition, tag filters applied) and its file stem is the journey name (`meta.scenario`, plan D1). A directory in `flows.paths` means the flow files directly in it (not `config.yaml`). A command after `--` or `flows.command` is one flow, named by `--scenario`, its only `.yaml` argument, or `e2e`. With several flows, run ids get the flow as a suffix and `runs.ndjson` records `scenario`.
+
+Annotated example: `examples/keepsnappy.yml`. Every setting:
+
+| Setting | Meaning |
+|---|---|
+| `project.key_env` | Name of the env var with the project key (default `KEEPSNAPPY_KEY`). Only read with `--upload`. |
+| `project.ingest_url` | KeepSnappy API base URL for `--upload` (or `--ingest-url`, or env `KEEPSNAPPY_API`). Defaults to the hosted KeepSnappy API. |
+| `app.bundle_id` | Required. App whose perf-lab log is read after each run. |
+| `platform`, `device` | `ios-sim` (default; `device` = UDID or `booted`) or `ios-device` (UDID required). One device per invocation in v1. |
+| `repetitions` | Runs of the whole flow set (default 5). |
+| `baseline_branch` | Branch the baseline comes from (default `main`). Recorded for CI; the CLI compares with the directory you pass to `--baseline`. |
+| `flows.runner` | `maestro` (build `maestro test ...`) or `command` (run `flows.command`). |
+| `flows.tags`, `flows.exclude_tags` | `--include-tags=` / `--exclude-tags=`. |
+| `flows.paths` | Files, directories or globs (`*`, `?`, `**`); expanded by perflab (no shell). With only tags the path defaults to `.`. An entry can also be `{path, title, description}` (one flow file): how the dashboard names the journey. |
+| `flows.command` | Any command that exits 0 on success (Appium, Detox, ...); text or list, run without a shell. |
+| `flows.maestro_version` | Note only: printed at the start of the run. |
+| `steps.rename` | `old: new` names applied to metrics (`old.dur_ms` becomes `new.dur_ms`, `step:old.*` too), so history keeps stable names. |
+| `steps.pin` | Names always listed in the summary. |
+| `output.dir` | Out dir (default `keepsnappy-out`): `runs.ndjson`, `metrics.csv`, `<run>/test.log`, `<run>/perf-lab.log`, `<run>/markers.ndjson`, and with a baseline `compare/metrics.csv` + `compare/verdict.json`. |
+| `timeout_seconds` | Per run of the command. |
+
+Unknown settings and wrong types are errors, all listed together.
+
+Results and exit codes: results never change the exit code by default (ruling D9, no blocking PR checks): `0` also for "More work". `--fail-on-worse` (needs `--wait`) makes "More work" exit `3`. The CLI exits `2` for a configuration problem (bad or missing `keepsnappy.yml`, flags, a glob matching nothing, `--wait` without a branch or key) and `1` for an infrastructure problem (cannot read the device log, baseline dir without `metrics.csv`, an API request failing; results stay in the out dir). Failed runs are marked invalid in `runs.ndjson` / `metrics.csv` and ignored by the statistics. The ending summary reads like:
+
+```
+7/8 runs valid (dropped: 1 test-failed: exit 1)
+Compared with main-out: 42 metrics
+Top changes:
+  More work chat.open: renders  +100% (100 -> 200)
+More work: 1  Less work: 0  No change: 38  Not enough data yet: 3
+```
+
+`--upload` (off by default) needs a project key with `run` scope in the env var named by `project.key_env` (default `KEEPSNAPPY_KEY`); the API URL defaults to the hosted KeepSnappy API. With a branch it opens a run set (`POST /v1/run-sets`) before measuring, so a wrong key fails before device time is spent; then sends each valid run: `POST /v1/runs` (header `x-perflab-key`, `meta.scenario`, `meta.run_set_id`), `POST /v1/windows` with the run's own `PERF_LAB window` log lines (run token, 200 per batch, deterministic batch ids so a retry is safe), `POST /v1/markers` with its declared step markers, `POST /v1/runs/:id/finish`. A 429 is retried after `retry-after`. Then it closes the run set:
+
+- with `--wait`: `POST /v1/run-sets/:id/complete` computes the verdicts inline and the CLI prints them in plain words (More work / Less work / No change / Not enough data yet per journey and device, the top changed checks, "Few runs — result may be noisy" when the server flags it) plus a dashboard link that opens the comparison;
+- without `--wait`: the set is closed with `verdicts: on-read` and the CLI exits 0 at once, printing the dashboard link; the verdicts are computed when someone looks.
+
+On the baseline branch (`main` by default, set per app on the server) the runs grow each journey's history; on any other branch they are compared with it. Without a branch the runs are uploaded as before, without a run set and without a verdict. The CLI writes `<out>/perflab-result.json` (`run_set_id`, `dashboard_url`, `state`, summary) and `<out>/perflab-run.json` (device, branch), and on GitHub Actions appends the result to `$GITHUB_STEP_SUMMARY`. Ready workflow: `examples/github-actions.yml`.
+
+`keepsnappy verdict <out> --baseline remote [--branch main] [--limit N]` compares each flow's local runs with the server baseline of the same flow and device (`GET /v1/baselines`: the history of the baseline branch, inside the same window the dashboard uses), for the metrics the server judges.
+
+## `keepsnappy farm` (device farm, the app pushes)
+
+For a farm where the CLI cannot reach the device (BrowserStack App Automate; plan 36). The CLI only opens and closes; the
+app (SDK activated per iteration with a run token) pushes its windows to the API itself, and the step script posts its
+markers there directly. Ready workflow and Maestro flows: `examples/farm/`.
+
+```sh
+perflab farm open --api https://perflab-api-….run.app --branch main --commit "$APP_COMMIT" \
+  --device-model 'iPhone 15 Pro' --device-os 17 --device-provider browserstack \
+  --journeys e2e-community-and-live-chat --repeat 5 \
+  --tokens-js farm-run/perflab-tokens.js --state farm-run/perflab-farm.json
+# ... the farm session runs (browserstack-run.mjs --suite-overlay "flows/perflab/tokens.js:farm-run/perflab-tokens.js")
+perflab farm close --state farm-run/perflab-farm.json --bs-build farm-run/browserstack.ios.json --settle-seconds 60 --wait
+```
+
+`keepsnappy farm run` does the whole night in one command (docs/plans/54a): resolve the app (`--app eas:<id>|file|bs://`), `farm open`, an optional BrowserStack Local tunnel (`--tunnel-for localhost:<port>`, repeatable), the device run (`--driver maestro` with `--suite-dir`/`--execute`, or `--driver appium --journey-module FILE` for the built-in Appium driver, `@keepsnappy/appium-driver`, docs/plans/54b; `--driver-script FILE` still runs your own driver), `farm close`, and cleanup (tunnel stopped, tokens file reset) whatever happened. `--env KEY=VALUE` values may use `{tunnel:<port>}`, `{run_id}`, `{out}`, `{tokens_js}`, `{run_set_id}`. Exit codes as `farm close`; a failed device run is 1. `--dry-run` prints the plan with no request. Needs `KEEPSNAPPY_KEY`, `BROWSERSTACK_USERNAME`, `BROWSERSTACK_ACCESS_KEY` (and `EXPO_TOKEN` for `eas:`).
+
+Setup and completion (plan 51 stage 2): `--completion-marker` declares that the flow posts the step marker `run:complete` (end) at the end of every iteration, from the main flow and not from `onFlowComplete`; the API then marks a run without it invalid (`flow-incomplete`). `--driver-version`, `--app-build-id`, `--harness-version`, `--flag-set` go into every run's setup fingerprint; `farm close` adds the device farm's build id from `--bs-build`.
+
+`farm open` also sends `journeys[]`: with `--config keepsnappy.yml` and/or `--flows <files|dirs|globs,...>` each `--journeys` name is matched to the flow file named like it (title, description, source path as for `keepsnappy run`); without them, keys only.
+
+`farm open` makes a run set (`source: farm`, the device of `--device-model/--device-os`: the journey history is keyed by
+it, not by what the phone reports) and one run per journey per iteration, ahead of any data, with run tokens valid for
+`--ttl-seconds` (default 21600 = 6 h). The key (`KEEPSNAPPY_KEY`, or `--key-env`) needs `run` scope and never leaves the CI
+runner. It writes:
+
+- `--tokens-js`: the only file holding run tokens. BrowserStack caps env values at 100 characters, so the tokens travel
+  inside the uploaded suite; the dispatcher overlays this file at `e2e/flows/perflab/tokens.js` for one upload. ES5,
+  for Maestro's `runScript`:
+  ```js
+  output.perflabApi = "https://perflab-api-….run.app";
+  output.perflabRunSet = "<run set id>";
+  output.perflabTokens = {"<journey>": ["pl_rt_…", …one per iteration]};
+  ```
+  `next-token.js` (examples) hands out `keepsnappyTokens[JOURNEY][i]` as `output.perflabToken` for iteration i.
+- `--state`: JSON for `farm close`, no token and no key (safe to upload as an artifact): `version`, `api`, `run_set_id`,
+  `dashboard_url`, `branch`, `commit`, `base_commit`, `device`, `journeys`, `repeat`, `tokens_js`, `opened_at`,
+  `expires_at`, `runs[]` (`order`, `journey`, `iteration`, `run_id`, `external_id`), then `closed_at` and `result`.
+- On GitHub Actions: `::add-mask::` for every token, and outputs `run_set_id`, `dashboard_url`, `state`, `tokens_js`, `runs`.
+
+`farm close` reads the state and:
+1. waits until no batch arrives for `--settle-seconds` (default 60), at most `--max-wait-seconds` (default 300);
+2. reads `--bs-build` (the App Automate build detail `browserstack-run.mjs` writes). When the session failed, the
+   iteration it stopped in is finished as invalid (`device-session-<status>`): the first run without data after the last
+   run with data. A measured iteration sends nothing until its end-of-iteration flush, so a run with data completed and
+   is never blamed; a Maestro failure stops the session, so later iterations have no data;
+3. finishes every run not yet finished (a run without a single batch is reported as "no data", never judged), closes the
+   set (`--wait`: verdicts inline; else on read) and prints the iterations and the same plain-words verdict as
+   `keepsnappy run --upload --wait`, plus `$GITHUB_STEP_SUMMARY` and `<out>/perflab-result.json` (`--out`, default the
+   state's directory);
+4. puts the empty stub back into the tokens file, also when a step above failed. The example workflow also deletes the
+   file before uploading `$OUT_DIR`.
+
+Running it again is safe: finished runs are left alone, the set is not reprocessed, the same result is printed. A batch
+that arrives after a run was finished is added by the server and the run re-processed at once. Exit codes: `0` for any
+result, `3` with `--fail-on-worse` on "More work", `2` for configuration, `1` when the API fails or nothing could be
+judged ("Blocked": no iteration sent any data).
+
+Slack: set `PERFLAB_SLACK_WEBHOOK` (an incoming-webhook URL kept as a CI secret) and `farm close --wait` posts the
+result: a pull request only when something got worse (a table of the 3 biggest changes, main vs PR), main only when
+something changed (`PERFLAB_SLACK_ALWAYS=1` also posts "no change"), and a measurement that sent no data always. The
+commit's title and author come from `git log` in the checkout. A Slack failure is printed and never changes the exit code.
+
+Declared steps on the farm: `maestro/perflab/step.yaml` / `step.js` / `send-markers.js` (copy them into the suite).
+With a token set, `step.js` only records the marker; `send-markers.js`, run in `end-run.yaml` after each iteration,
+posts them to `<output.perflabApi or KEEPSNAPPY_API_URL>/v1/markers` with `Authorization: Bearer <run token>`, so the test
+never waits on the network between actions (and sends the host's clock round trips, plan 51 G1.4). A failed iteration
+never reaches `end-run.yaml`, so the example `run-perflab.yaml` also runs `send-markers.js` from `onFlowComplete`; such a
+run is invalid through its missing completion marker, and its markers stay for diagnosis. The example folder
+(`examples/farm/flows/perflab/`) ships its own copy of `send-markers.js`. Without a token, `step.js` posts to the local
+CLI endpoint at once; there a step started but never ended invalidates its run (`unclosed-steps`).
+
+Detox (Jest): `packages/detox` (`@keepsnappy/detox`) gives `step(name, fn)` / `stepStart` / `stepEnd`, posting the same markers: to the local endpoint from `PERFLAB_MARKER_URL` + `PERFLAB_RUN_ID` (set by `keepsnappy run`/`measure`), or on the farm to `KEEPSNAPPY_API_URL/v1/markers` with `PERFLAB_RUN_TOKEN` as the bearer token. Optional per-test step names via `@keepsnappy/detox/auto-steps`. See its README.
+
+Farm e2e against a local API: `KEEPSNAPPY_KEY=… PERFLAB_INGEST_KEY=… e2e/farm/run.sh <new out dir>` (fake session using
+the SDK push client, one late batch, one empty iteration).
+
+## `keepsnappy findings` and friends (read-only questions)
+
+Ask KeepSnappy what it found, without running anything (plan 48 §4). Every answer has the same shape: what changed, where
+(journey, phone, branch, commit title and author, PR), how sure (runs per side, session spread, "measured differently —
+not blocking"), why (components that rendered longer and their cause, counts that moved), what to do (a prompt for a
+coding agent), and since when it is seen. The MCP server `@keepsnappy/mcp` gives coding agents the same answers with the
+same words (`packages/mcp/README.md`).
+
+```sh
+export KEEPSNAPPY_KEY=pl_rk_…        # a project key with read scope (a CI key with run scope works too)
+perflab findings --branch feat/perflab-integration     # what got worse or better on a PR, worst first
+perflab findings --main --journey live-ui --phone "iPhone 15"
+perflab findings --state all --json                    # also the resolved and not judged ones, as the API's JSON
+perflab findings --state all --offset 200              # the next page (at most 200 per answer)
+perflab finding f_0123456789abcdef                     # one finding with its evidence
+perflab finding f_0123456789abcdef --prompt | pbcopy   # only the hand-off for a coding agent, ending with its id and page
+perflab compare <runA> <runB>                          # two runs (ids or external ids), window by window, no statistics
+perflab history live-ui "Tap: live chat send"          # one window on main, run by run (id or name)
+perflab commits a1b2c3d c3d4e5f [--branch main]        # the measured commits between two points
+perflab layers                                         # which layer answers which question, and its certificate
+```
+
+Flags: `--api URL` (or `KEEPSNAPPY_API`, default the hosted API), `--key-env NAME` (default `KEEPSNAPPY_KEY`), `--json`.
+A finding's id is stable: the same window, measure, journey and phone model keep the id across commits and PRs, so
+`keepsnappy finding <id>` after a fix tells whether it is resolved. `--state` (default `present`) filters on the state on
+the asked branch: `present` (in the newest run set), `resolved` (the newest run set judged the measure: No change, or the
+other way), `not-judged` (the newest run set did not judge it: not measured, unstable, not enough data; it may still be
+there). A list starts with whether and when the branch was measured ("No KeepSnappy measurement on branch X", or the
+newest measured commit, its word and runs), so an empty list on an unmeasured branch does not read as clean. `commits`
+lists each measured commit once, with how many sessions measured it. Words come from the one word table (Worse / Warning / Better …), "Worse (not blocking)" when the
+branch was measured differently from main.
+
+## Output layout
+
+```
+<out>/arms.json              arms of a `measure` (name, target, per-arm env)
+<out>/runs.ndjson            one record per run: run_id, arm, block, t0, t1, exit_code, valid, invalid_reason
+<out>/<run_id>/test.log      stdout/stderr of the test command
+<out>/<run_id>/perf-lab.log  Documents/perf-lab/log.txt pulled from the app
+<out>/metrics.csv            long format: run_id,arm,block,metric,value,valid,invalid_reason
+<out>/verdict.json           written by verdict / aa
+```
+
+## Architecture
+
+```
+ cli.ts ──► runner.measure ──► schedule (ABBA)
+               │  per run:
+               ├─ AppInstaller ── IosSimulatorInstaller | IosDeviceInstaller
+               ├─ Spawner.run(test command)            t0 .. t1, exit code
+               ├─ DataSource.fetchLog ── IosSimulatorDataSource | IosDeviceDataSource
+               └─ window-log (parse, slice by time) ► metrics (per-window medians) ► metrics.csv
+ cli.ts ──► verdict / aa ──► @keepsnappy/stats (compareRuns / aaRuns) ──► table + verdict.json
+```
+
+Pure core (`schedule`, `window-log`, `metrics`, table formatting) plus thin adapters. Process spawning, clock and filesystem come in through `Deps` (`ports.ts`), so every test runs on fakes and fixtures; nothing touches a simulator or device.
+
+## How measure-only works
+
+1. The customer's build already contains the perf-lab SDK, which appends `<wall ms> PERF_LAB window {json}` lines to `Documents/perf-lab/log.txt` for every window it observes (screen opens, interactions).
+2. After each run the log is pulled and the windows whose start (`wall - durMs`) lies in `[t0, t1]` of that run are kept. Anything older is leftover from earlier runs.
+3. Per window name, the run's value is the median over its cycles of: `dur_ms`, `commits`, `renders`, `mounts`, `longtask_ms`, `gc_count`, `gc_ms`, `alloc_mb`, `touch_max_ms`; plus `n`, `timeouts` (windows ending in `timeout` are excluded from medians) and `retained_mb_per_cycle` (Theil-Sen slope of heap after GC across cycles, needs 3+ cycles). Metric names are `<window>.<field>`, kept as-is (e.g. `screen:(app)/community/[handle]/(tabs).dur_ms`) and quoted in the CSV when needed.
+4. A failed test, an install or log-pull failure, or a run with no windows is written as invalid with a reason. Invalid runs stay in `metrics.csv` (a `run.status` row when they have no metrics) and the stats engine drops them whole.
+5. `block` is the position of the run's pair in the ABBA order, so the stats engine can analyse paired blocks.
+
+## Realtime Inspector and memory metrics
+
+`--ri-capture <path>` (on `measure` and `extract`) points at an RI ingest-sink capture, `observations.ndjson`, one JSON per line (`envelope.batch.observations[]`). A `RiSource` port (`FileRiSource`) re-reads it after every run; `client_health.*` `metricSample`s are sliced to the run's `[t0, t1]`:
+
+- mean: `ri.fps.ui`, `ri.fps.js`, `ri.cpu.process_cores`, `ri.cpu.main_thread_cores`, `ri.cpu.js_thread_cores`, `ri.thread.main_blocked_ratio`, `ri.thread.js_blocked_ratio`, `ri.sdk.collect_ms`
+- delta (last - first, it is a running total): `ri.frames.dropped`
+- max: `ri.memory.footprint_mb`, `ri.memory.used_mb`, `ri.thermal.state`, `ri.thread.main_stall_ms`, `ri.thread.js_stall_ms`
+- `ri.samples`: number of `fps.ui` samples in the interval
+
+Memory slopes (Theil-Sen over the cycle index, 3+ cycles of `--mem-window`, default `chat.open`): `mem.retained_mb_per_cycle` from the window's `heapAfterGc` (`js_allocatedBytes + js_externalBytes`, MiB) and, with a capture, `mem.footprint_mb_per_cycle` from the last footprint sample at or before each window start. On ProMotion iPhones `ri.fps.ui`, `ri.frames.dropped` and `ri.thread.main_blocked_ratio` are artefacts: diagnostic only.
+
+`test/parity.test.ts` runs `tools/analysis/ios_series_metrics.py` and the TS extraction on a synthetic series and requires equal values.
+
+## Limitations
+
+- Simulator runs give mechanism metrics only (renders, commits, mounts, GC, allocations). Times (`dur_ms`, long tasks, touch latency) mean something only on a real device.
+- The test driver's own overhead and timing noise are inside the measurement; A/B order cancels drift, not a driver that behaves differently per arm.
+- Window slicing uses the host clock (`t0`/`t1`) against the app's wall clock; on a device with a skewed clock windows may be missed (the run then becomes `no-windows`).
+- iOS only. Android is not in v1. BrowserStack only through `keepsnappy farm` (the app pushes; the CLI never reads the device).
+- `keepsnappy measure` does not boot devices or launch the app; the test command does that.
+
+## Working on the CLI
+
+`bun run typecheck` and `bun run test` first build the workspace packages the CLI imports (`bun run deps`: stats, derive,
+verdicts, profile-analysis, mcp), so both work from a clean checkout.
+
+The `perflab` command and `perflab.yml` / `perflab.yaml` remain supported. Without `--config`, configuration is read in this order: `keepsnappy.yml`, `keepsnappy.yaml`, `perflab.yml`, `perflab.yaml`. `KEEPSNAPPY_*` variables take precedence over matching `PERFLAB_*` variables, including explicit empty values. Explicit `--key-env` or `project.key_env` names are read exactly.