zeno-mobile-runner 0.1.8 → 0.2.1

package/docs/ai-agents.md

@@ -4,6 +4,24 @@ ZMR is built for external agents. The runner provides device state, typed
 actions, waits, assertions, trace explanation, and trace export; the agent
 decides the next step.
 
+```mermaid
+sequenceDiagram
+    participant Agent as AI agent
+    participant ZMR
+    participant Device as Emulator / simulator
+    Agent->>ZMR: semantic_snapshot
+    ZMR->>Device: capture UI + screenshot
+    ZMR-->>Agent: roles, stable selectors, bounds
+    Agent->>ZMR: tap / type / swipe / open_link
+    ZMR->>Device: execute + settle
+    Agent->>ZMR: wait_visible / assert_visible
+    ZMR-->>Agent: typed result + trace events
+    Agent->>ZMR: trace_discover
+    ZMR-->>Agent: reviewable replay scenario
+    Agent->>ZMR: trace_export --redact
+    ZMR-->>Agent: .zmrtrace evidence bundle
+```
+
 ## Agent Setup Loop
 
 Start inside the app checkout:

package/docs/benchmarking.md

@@ -4,6 +4,39 @@ ZMR benchmark output is intentionally simple: each run appends one JSON object
 to `results.jsonl`, and `zmr report` turns that directory into local HTML and
 optional JUnit XML artifacts.
 
+## Public Evidence
+
+Public-safe benchmark evidence lives in [docs/benchmarks](benchmarks/README.md).
+The first committed pack is
+[2026-06-09 iOS simulator demo](benchmarks/2026-06-09-ios-demo.md): 20 repeated
+runs of the generated iOS smoke scenario with a 100% pass rate. It is a
+single-tool reliability benchmark, not a competitive speed claim.
+
+The first baseline comparison is documented in
+[docs/benchmarks](benchmarks/README.md): 20 ZMR runs and 20 baseline runner
+runs against the same generated iOS demo app.
+
+Additional public-safe packs in that directory include a second baseline
+comparison and a native shim floor. The floor is not a product comparison; it
+shows the warmed platform path ZMR can approach after runner and trace overhead
+are reduced.
+
+A richer iOS workflow pack is also committed there: 20 ZMR rows and 20 baseline
+runner rows against the same generated app build, covering profile entry,
+catalog item selection, save, review, and final-state assertion.
+
+Benchmark Lab v1 is the next public evidence layer. It defines framework
+fixtures, timing modes, runner-adapter labels, and claim rules in a manifest
+that can be validated or rendered with `zmr-benchmark-lab`.
+
+The generated Android workflow now has its first 20-run evidence pack in
+[docs/benchmarks](benchmarks/README.md), using the platform UIAutomator path
+without the optional Android instrumentation shim.
+
+A generated React Native/Expo fixture is now available for the next evidence
+slice. It includes stable `testID` values, accessibility labels, deep-link
+setup, and Android/iOS ZMR workflow scenarios, but no public timing rows yet.
+
 ## Single Tool Benchmark
 
 ```bash
@@ -70,6 +103,10 @@ For `--runs 1`, the script exports normal and redacted `.zmrtrace` bundles.
 For `--runs > 1`, the pilot wrappers and generated app reliability scripts
 write benchmark directories with HTML and JUnit reports.
 
+Apps scaffolded by `zmr-wizard` get matching package scripts, so app-local
+reliability gates run as `bun run zmr:android:reliability` and
+`bun run zmr:ios:reliability` (or the npm equivalents).
+
 The iOS pilot wrapper supports the same repeated-run gates:
 
 ```bash
@@ -138,6 +175,8 @@ Benchmark reports include:
 
 Before making public performance claims, run the same scenario repeatedly on a clean emulator image and include the raw `results.jsonl` plus the redacted trace bundle for any failure.
 
+![ZMR HTML trace report showing the trace summary and per-event timeline](assets/report-html.png)
+
 ## Compare Against A Baseline
 
 Use `zmr-compare-benchmarks` when a private app repo has benchmark rows from

package/docs/benchmarks/2026-06-09-android-workflow.md

@@ -0,0 +1,73 @@
+# 2026-06-09 Android Emulator Workflow
+
+This evidence pack records 20 repeated ZMR runs of the generated public Android
+workflow demo app. The flow launches the app, fills a profile form, scrolls the
+catalog, opens an item detail page, saves the item, reviews the order, and
+asserts the final state.
+
+This is a single-tool reliability and timing pack, not a comparison against
+another runner. Treat it as reproducible evidence for this app, host, emulator,
+app build, and workflow shape.
+
+## Result
+
+| Tool | Runs | Pass rate | Failures | Mean duration | p95 duration |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| ZMR | 20 | 100.00% | 0 | 44134 ms | 46385 ms |
+
+The fastest run was 38627 ms and the slowest run was 49875 ms. The later rows
+clustered around 43 seconds, which points to Android UIAutomator snapshot and
+scroll execution as the next optimization target for this fixture.
+
+## Environment
+
+| Field | Value |
+| --- | --- |
+| ZMR runner | `0.1.8` |
+| ZMR protocol | `2026-04-28` |
+| Host OS | macOS 26.6, arm64 |
+| Android emulator | 36.4.10.0 |
+| ADB | 1.0.41, platform-tools 37.0.0 |
+| Android platform | Android 15, API 35, arm64-v8a |
+| Emulator viewport | 720 x 1280, 320 dpi |
+| App id | `com.example.mobiletest` |
+| App build label | `generated-android-workflow-demo-20260609` |
+| Demo app source | Generated by `scripts/create-android-demo-app.sh` |
+
+Before collection, the emulator was booted fresh, the app was reinstalled, the
+screen was unlocked, and Android window, transition, and animator duration
+scales were set to `0`.
+
+## Command
+
+```bash
+ZMR_BIN="$PWD/zig-out/bin/zmr" scripts/benchmark.sh \
+  --zmr examples/android-workflow.json \
+  --platform android \
+  --device emulator-5554 \
+  --app-id com.example.mobiletest \
+  --app-build generated-android-workflow-demo-20260609 \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-android-workflow/zmr \
+  --results traces/public-benchmarks/20260609-android-workflow/results.jsonl \
+  --replace \
+  --min-pass-rate 100 \
+  --max-failures 0
+```
+
+The ZMR scenario is committed as
+[`examples/android-workflow.json`](../../examples/android-workflow.json).
+
+## Rows
+
+The sanitized result rows are committed in
+[2026-06-09-android-workflow.results.jsonl](2026-06-09-android-workflow.results.jsonl).
+
+Raw local trace and runner logs are not committed because they can include local
+absolute paths.
+
+## Scope
+
+This benchmark uses the platform UIAutomator path without the optional Android
+instrumentation shim. It does not compare cloud execution, React Native, Expo,
+Flutter, Appium, Maestro, Detox, or Android instrumentation-runner baselines.

package/docs/benchmarks/2026-06-09-android-workflow.results.jsonl

@@ -0,0 +1,20 @@
+{"tool":"zmr","run":1,"status":"ok","durationMs":49875,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-1","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":2,"status":"ok","durationMs":45169,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-2","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":3,"status":"ok","durationMs":46093,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-3","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":4,"status":"ok","durationMs":44695,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-4","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":5,"status":"ok","durationMs":44028,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-5","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":6,"status":"ok","durationMs":44821,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-6","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":7,"status":"ok","durationMs":46385,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-7","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":8,"status":"ok","durationMs":45751,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-8","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":9,"status":"ok","durationMs":38627,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-9","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":10,"status":"ok","durationMs":42599,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-10","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":11,"status":"ok","durationMs":42968,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-11","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":12,"status":"ok","durationMs":43299,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-12","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":13,"status":"ok","durationMs":43684,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-13","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":14,"status":"ok","durationMs":43056,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-14","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":15,"status":"ok","durationMs":43418,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-15","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":16,"status":"ok","durationMs":43267,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-16","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":17,"status":"ok","durationMs":43780,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-17","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":18,"status":"ok","durationMs":43371,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-18","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":19,"status":"ok","durationMs":43095,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-19","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":20,"status":"ok","durationMs":44693,"traceDir":"traces/public-benchmarks/20260609-android-workflow/zmr/zmr-20","platform":"android","device":"emulator-5554","appId":"com.example.mobiletest","scenario":"examples/android-workflow.json","appBuild":"generated-android-workflow-demo-20260609","traceStatus":"passed"}

package/docs/benchmarks/2026-06-09-framework-baseline-status.md

@@ -0,0 +1,32 @@
+# 2026-06-09 Framework Baseline Status
+
+This note tracks the requested baseline coverage beyond the committed iOS demo
+comparisons.
+
+## Completed
+
+| Baseline | Status | Evidence |
+| --- | --- | --- |
+| Maestro | Completed | [iOS ZMR vs Maestro comparison](2026-06-09-ios-maestro-comparison.md) |
+| Appium | Completed | [iOS ZMR vs Appium comparison](2026-06-09-ios-appium-comparison.md) |
+| XCTest floor | Completed | [iOS XCTest shim floor](2026-06-09-ios-xctest-floor.md) |
+
+## Not Yet Fair To Publish
+
+| Baseline | Why it needs a fixture first | Next evidence pack |
+| --- | --- | --- |
+| Detox | The CLI requires a project-local `detox` install and a React Native app with Detox configuration, native iOS/Android build targets, and a test file. Running it against the generated Swift demo would not be representative. | React Native fixture with the same launch, deep link, assertion, and warm-suite/cold-command modes. |
+| Flutter | The local machine does not have the Flutter CLI installed, and ZMR should not claim Flutter widget-tree-driver coverage. | Flutter fixture using platform-level labels/deep links plus either Flutter `integration_test` or an external runner baseline. |
+| Espresso | No Android emulator is currently attached in this workspace. Espresso should compare against an Android fixture with an instrumentation target rather than an iOS-only demo. | Android generated demo with ZMR, direct Espresso instrumentation, and Appium UIAutomator2 rows. |
+
+## Speed Work Opened By This Pass
+
+The XCTest floor showed that ZMR can be made faster. The first fix from this
+pass skips the expensive iOS system-open alert probe for custom URL schemes and
+keeps it for `http://` and `https://` links. On the generated iOS demo smoke
+flow, the shim-backed ZMR mean dropped to `2007 ms` while the direct warmed
+XCTest shim floor measured `1004 ms`.
+
+The next speed target is a warm-suite mode where one ZMR process executes many
+iterations in a single device session, avoiding repeated CLI startup and trace
+setup for benchmark loops.

package/docs/benchmarks/2026-06-09-ios-appium-comparison.md

@@ -0,0 +1,115 @@
+# 2026-06-09 iOS Simulator ZMR vs Appium Comparison
+
+This evidence pack compares ZMR with Appium on the generated public iOS demo
+app. It is a command-level benchmark: each row starts one local runner command,
+executes the same smoke flow on the same booted simulator, and records command
+duration.
+
+This is not a universal claim about either project. Treat it as reproducible
+evidence for this app, host, simulator, and flow shape.
+
+## Result
+
+| Tool | Runs | Pass rate | Failures | Mean duration | p95 duration |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| ZMR | 20 | 100.00% | 0 | 2315 ms | 2339 ms |
+| Appium | 20 | 100.00% | 0 | 3837 ms | 3883 ms |
+
+For this run, ZMR was 1.66x faster on mean duration and 1.66x faster on p95
+duration.
+
+## Environment
+
+| Field | Value |
+| --- | --- |
+| ZMR runner | `0.1.8` |
+| ZMR protocol | `2026-04-28` |
+| Appium | `3.5.0` |
+| Appium XCUITest driver | `11.10.0` |
+| Host OS | macOS 26.6, arm64 |
+| Xcode | 26.5, build 17F42 |
+| Simulator | Booted iOS 26.5 simulator, `zmr-iPhone` |
+| App id | `com.example.mobiletest` |
+| App build label | `generated-ios-demo-20260609` |
+| Demo app source | Generated by `scripts/create-ios-demo-app.sh` |
+
+## ZMR Command
+
+```bash
+ZMR_BIN="$PWD/zig-out/bin/zmr" scripts/benchmark.sh \
+  --zmr examples/ios-smoke.json \
+  --platform ios \
+  --ios-device-type simulator \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --xcrun xcrun \
+  --app-build generated-ios-demo-20260609 \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-appium-comparison/zmr \
+  --results traces/public-benchmarks/20260609-ios-appium-comparison/results.jsonl \
+  --replace \
+  --min-pass-rate 100 \
+  --max-failures 0
+```
+
+The ZMR scenario launches the app, opens `exampleapp:///e2e-auth?probe=1`, and
+captures a snapshot.
+
+## Appium Command
+
+The Appium server was started with a temp Appium home:
+
+```bash
+APPIUM_HOME=/tmp/zmr-appium-home-20260609 npx --yes appium@latest --port 4723 --log-level error
+```
+
+The XCUITest driver was installed in that temp home:
+
+```bash
+APPIUM_HOME=/tmp/zmr-appium-home-20260609 npx --yes appium@latest driver install xcuitest
+```
+
+Each measured Appium row created a fresh XCUITest WebDriver session against the
+same bundle id, opened the same deep link with `simctl openurl`, read page
+source, asserted `Deep link opened`, and deleted the session.
+
+```bash
+scripts/benchmark-command.sh \
+  --tool appium \
+  --platform ios \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --scenario examples/ios-smoke.json \
+  --app-build generated-ios-demo-20260609 \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-appium-comparison/appium \
+  --results traces/public-benchmarks/20260609-ios-appium-comparison/results.jsonl \
+  -- node -e '<create session, open deep link, assert page source, delete session>'
+```
+
+## Comparison Report
+
+```text
+# Benchmark Comparison
+
+| Tool | Runs | Pass rate | Failures | Mean ms | P95 ms |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| zmr | 20 | 100.00% | 0 | 2315 | 2339 |
+| appium | 20 | 100.00% | 0 | 3837 | 3883 |
+
+- Mean speedup: 1.66x (-39.7% candidate vs baseline)
+- P95 speedup: 1.66x (-39.8% candidate vs baseline)
+- Same benchmark context: yes
+```
+
+The sanitized result rows are committed in
+[2026-06-09-ios-appium-comparison.results.jsonl](2026-06-09-ios-appium-comparison.results.jsonl).
+
+Raw local trace and runner logs are not committed because they can include local
+absolute paths.
+
+## Scope
+
+This benchmark compares local command surfaces under repeated execution. It
+does not compare cloud execution, long multi-screen flows, Detox, Flutter,
+Espresso, XCTest-only app tests, or app-specific selectors.

package/docs/benchmarks/2026-06-09-ios-appium-comparison.results.jsonl

@@ -0,0 +1,40 @@
+{"tool":"zmr","run":1,"status":"ok","durationMs":2327,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":2,"status":"ok","durationMs":2326,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":3,"status":"ok","durationMs":2312,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":4,"status":"ok","durationMs":2305,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":5,"status":"ok","durationMs":2305,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":6,"status":"ok","durationMs":2295,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":7,"status":"ok","durationMs":2339,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":8,"status":"ok","durationMs":2289,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":9,"status":"ok","durationMs":2292,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":10,"status":"ok","durationMs":2320,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":11,"status":"ok","durationMs":2332,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":12,"status":"ok","durationMs":2333,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":13,"status":"ok","durationMs":2316,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":14,"status":"ok","durationMs":2291,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":15,"status":"ok","durationMs":2342,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":16,"status":"ok","durationMs":2308,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":17,"status":"ok","durationMs":2298,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":18,"status":"ok","durationMs":2297,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":19,"status":"ok","durationMs":2339,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":20,"status":"ok","durationMs":2329,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"appium","run":1,"status":"ok","durationMs":3825,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":2,"status":"ok","durationMs":3779,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":3,"status":"ok","durationMs":3773,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":4,"status":"ok","durationMs":3776,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":5,"status":"ok","durationMs":3790,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":6,"status":"ok","durationMs":3797,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":7,"status":"ok","durationMs":3747,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":8,"status":"ok","durationMs":3799,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":9,"status":"ok","durationMs":4022,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":10,"status":"ok","durationMs":3871,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":11,"status":"ok","durationMs":3864,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":12,"status":"ok","durationMs":3882,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":13,"status":"ok","durationMs":3858,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":14,"status":"ok","durationMs":3883,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":15,"status":"ok","durationMs":3852,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":16,"status":"ok","durationMs":3879,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":17,"status":"ok","durationMs":3871,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":18,"status":"ok","durationMs":3825,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":19,"status":"ok","durationMs":3825,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":20,"status":"ok","durationMs":3818,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}

package/docs/benchmarks/2026-06-09-ios-demo.md

@@ -0,0 +1,90 @@
+# 2026-06-09 iOS Simulator Demo Benchmark
+
+This evidence pack records a repeat-run reliability benchmark for the public
+ZMR iOS demo app. It is a single-tool benchmark, not a comparison against
+Maestro, Appium, Detox, XCTest, or another runner.
+
+## Result
+
+| Metric | Value |
+| --- | --- |
+| Scenario | `examples/ios-smoke.json` |
+| Platform | iOS simulator |
+| Runs | 20 |
+| Passed | 20 |
+| Failed | 0 |
+| Pass rate | 100.00% |
+| Mean duration | 4192 ms |
+| p95 duration | 5409 ms |
+| Min duration | 3065 ms |
+| Max duration | 6395 ms |
+
+## Environment
+
+| Field | Value |
+| --- | --- |
+| ZMR runner | `0.1.8` |
+| Protocol | `2026-04-28` |
+| Host OS | macOS 26.6, arm64 |
+| Xcode | 26.5, build 17F42 |
+| Simulator | Booted iOS 26.5 simulator, `zmr-iPhone` |
+| App id | `com.example.mobiletest` |
+| Demo app source | Generated by `scripts/create-ios-demo-app.sh` |
+
+## Command
+
+```bash
+rm -rf traces/public-benchmarks/20260609-ios-demo /tmp/zmr-public-benchmark-ios-demo-20260609-plain
+
+scripts/create-ios-demo-app.sh \
+  --out /tmp/zmr-public-benchmark-ios-demo-20260609-plain \
+  --name ZMRDemo \
+  --bundle-id com.example.mobiletest \
+  --deployment-target 16.0
+
+xcodebuild \
+  -project /tmp/zmr-public-benchmark-ios-demo-20260609-plain/ios/ZMRDemo.xcodeproj \
+  -scheme ZMRDemo \
+  -destination 'generic/platform=iOS Simulator' \
+  -configuration Debug \
+  -derivedDataPath /tmp/zmr-public-benchmark-ios-demo-20260609-plain/DerivedData \
+  build
+
+ZMR_BIN="$PWD/zig-out/bin/zmr" scripts/run-ios-pilot.sh \
+  --app-root /tmp/zmr-public-benchmark-ios-demo-20260609-plain \
+  --app-path /tmp/zmr-public-benchmark-ios-demo-20260609-plain/DerivedData/Build/Products/Debug-iphonesimulator/ZMRDemo.app \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --xcrun xcrun \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-demo \
+  --min-pass-rate 100 \
+  --max-failures 0
+```
+
+The local run wrote:
+
+- `traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/results.jsonl`
+- `traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/report.html`
+- `traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/junit.xml`
+
+Those raw artifacts are intentionally not committed because generated reports
+and JUnit properties include absolute local paths. The sanitized result rows are
+committed in
+[2026-06-09-ios-demo.results.jsonl](2026-06-09-ios-demo.results.jsonl).
+
+## Scope
+
+This run validates the public iOS simulator smoke path: install, launch, deep
+link, snapshot, trace collection, benchmark gate, HTML report generation, and
+JUnit report generation.
+
+This run does not measure the app-local XCTest shim path. During collection, a
+shim compatibility check found a generated wrapper portability issue in
+one-shot mode: the wrapper used a `mktemp` template with `XXXXXX` before a
+`.log` suffix, which is not portable on macOS. That generator bug was fixed
+before this evidence pack was committed.
+
+Do not use this evidence as a speed comparison against another mobile testing
+framework. Baseline comparisons require equivalent candidate and baseline rows
+from the same app build, device state, and scenario.

package/docs/benchmarks/2026-06-09-ios-demo.results.jsonl

@@ -0,0 +1,20 @@
+{"tool":"zmr","run":1,"status":"ok","durationMs":5409,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":2,"status":"ok","durationMs":3289,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":3,"status":"ok","durationMs":3319,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":4,"status":"ok","durationMs":3323,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":5,"status":"ok","durationMs":3562,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":6,"status":"ok","durationMs":4048,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":7,"status":"ok","durationMs":3606,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":8,"status":"ok","durationMs":3065,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":9,"status":"ok","durationMs":3692,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":10,"status":"ok","durationMs":3514,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":11,"status":"ok","durationMs":3702,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":12,"status":"ok","durationMs":5092,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":13,"status":"ok","durationMs":4929,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":14,"status":"ok","durationMs":4340,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":15,"status":"ok","durationMs":6395,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":16,"status":"ok","durationMs":5128,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":17,"status":"ok","durationMs":3955,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":18,"status":"ok","durationMs":4410,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":19,"status":"ok","durationMs":4634,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":20,"status":"ok","durationMs":4425,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}