zeno-mobile-runner 0.1.3 → 0.2.0

package/docs/benchmarks/2026-06-09-ios-appium-comparison.md

@@ -0,0 +1,115 @@
+# 2026-06-09 iOS Simulator ZMR vs Appium Comparison
+
+This evidence pack compares ZMR with Appium on the generated public iOS demo
+app. It is a command-level benchmark: each row starts one local runner command,
+executes the same smoke flow on the same booted simulator, and records command
+duration.
+
+This is not a universal claim about either project. Treat it as reproducible
+evidence for this app, host, simulator, and flow shape.
+
+## Result
+
+| Tool | Runs | Pass rate | Failures | Mean duration | p95 duration |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| ZMR | 20 | 100.00% | 0 | 2315 ms | 2339 ms |
+| Appium | 20 | 100.00% | 0 | 3837 ms | 3883 ms |
+
+For this run, ZMR was 1.66x faster on mean duration and 1.66x faster on p95
+duration.
+
+## Environment
+
+| Field | Value |
+| --- | --- |
+| ZMR runner | `0.1.8` |
+| ZMR protocol | `2026-04-28` |
+| Appium | `3.5.0` |
+| Appium XCUITest driver | `11.10.0` |
+| Host OS | macOS 26.6, arm64 |
+| Xcode | 26.5, build 17F42 |
+| Simulator | Booted iOS 26.5 simulator, `zmr-iPhone` |
+| App id | `com.example.mobiletest` |
+| App build label | `generated-ios-demo-20260609` |
+| Demo app source | Generated by `scripts/create-ios-demo-app.sh` |
+
+## ZMR Command
+
+```bash
+ZMR_BIN="$PWD/zig-out/bin/zmr" scripts/benchmark.sh \
+  --zmr examples/ios-smoke.json \
+  --platform ios \
+  --ios-device-type simulator \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --xcrun xcrun \
+  --app-build generated-ios-demo-20260609 \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-appium-comparison/zmr \
+  --results traces/public-benchmarks/20260609-ios-appium-comparison/results.jsonl \
+  --replace \
+  --min-pass-rate 100 \
+  --max-failures 0
+```
+
+The ZMR scenario launches the app, opens `exampleapp:///e2e-auth?probe=1`, and
+captures a snapshot.
+
+## Appium Command
+
+The Appium server was started with a temp Appium home:
+
+```bash
+APPIUM_HOME=/tmp/zmr-appium-home-20260609 npx --yes appium@latest --port 4723 --log-level error
+```
+
+The XCUITest driver was installed in that temp home:
+
+```bash
+APPIUM_HOME=/tmp/zmr-appium-home-20260609 npx --yes appium@latest driver install xcuitest
+```
+
+Each measured Appium row created a fresh XCUITest WebDriver session against the
+same bundle id, opened the same deep link with `simctl openurl`, read page
+source, asserted `Deep link opened`, and deleted the session.
+
+```bash
+scripts/benchmark-command.sh \
+  --tool appium \
+  --platform ios \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --scenario examples/ios-smoke.json \
+  --app-build generated-ios-demo-20260609 \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-appium-comparison/appium \
+  --results traces/public-benchmarks/20260609-ios-appium-comparison/results.jsonl \
+  -- node -e '<create session, open deep link, assert page source, delete session>'
+```
+
+## Comparison Report
+
+```text
+# Benchmark Comparison
+
+| Tool | Runs | Pass rate | Failures | Mean ms | P95 ms |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| zmr | 20 | 100.00% | 0 | 2315 | 2339 |
+| appium | 20 | 100.00% | 0 | 3837 | 3883 |
+
+- Mean speedup: 1.66x (-39.7% candidate vs baseline)
+- P95 speedup: 1.66x (-39.8% candidate vs baseline)
+- Same benchmark context: yes
+```
+
+The sanitized result rows are committed in
+[2026-06-09-ios-appium-comparison.results.jsonl](2026-06-09-ios-appium-comparison.results.jsonl).
+
+Raw local trace and runner logs are not committed because they can include local
+absolute paths.
+
+## Scope
+
+This benchmark compares local command surfaces under repeated execution. It
+does not compare cloud execution, long multi-screen flows, Detox, Flutter,
+Espresso, XCTest-only app tests, or app-specific selectors.

package/docs/benchmarks/2026-06-09-ios-appium-comparison.results.jsonl

@@ -0,0 +1,40 @@
+{"tool":"zmr","run":1,"status":"ok","durationMs":2327,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":2,"status":"ok","durationMs":2326,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":3,"status":"ok","durationMs":2312,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":4,"status":"ok","durationMs":2305,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":5,"status":"ok","durationMs":2305,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":6,"status":"ok","durationMs":2295,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":7,"status":"ok","durationMs":2339,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":8,"status":"ok","durationMs":2289,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":9,"status":"ok","durationMs":2292,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":10,"status":"ok","durationMs":2320,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":11,"status":"ok","durationMs":2332,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":12,"status":"ok","durationMs":2333,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":13,"status":"ok","durationMs":2316,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":14,"status":"ok","durationMs":2291,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":15,"status":"ok","durationMs":2342,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":16,"status":"ok","durationMs":2308,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":17,"status":"ok","durationMs":2298,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":18,"status":"ok","durationMs":2297,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":19,"status":"ok","durationMs":2339,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":20,"status":"ok","durationMs":2329,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/zmr/zmr-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"appium","run":1,"status":"ok","durationMs":3825,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":2,"status":"ok","durationMs":3779,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":3,"status":"ok","durationMs":3773,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":4,"status":"ok","durationMs":3776,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":5,"status":"ok","durationMs":3790,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":6,"status":"ok","durationMs":3797,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":7,"status":"ok","durationMs":3747,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":8,"status":"ok","durationMs":3799,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":9,"status":"ok","durationMs":4022,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":10,"status":"ok","durationMs":3871,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":11,"status":"ok","durationMs":3864,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":12,"status":"ok","durationMs":3882,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":13,"status":"ok","durationMs":3858,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":14,"status":"ok","durationMs":3883,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":15,"status":"ok","durationMs":3852,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":16,"status":"ok","durationMs":3879,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":17,"status":"ok","durationMs":3871,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":18,"status":"ok","durationMs":3825,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":19,"status":"ok","durationMs":3825,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"appium","run":20,"status":"ok","durationMs":3818,"traceDir":"traces/public-benchmarks/20260609-ios-appium-comparison/appium/appium-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}

package/docs/benchmarks/2026-06-09-ios-demo.md

@@ -0,0 +1,90 @@
+# 2026-06-09 iOS Simulator Demo Benchmark
+
+This evidence pack records a repeat-run reliability benchmark for the public
+ZMR iOS demo app. It is a single-tool benchmark, not a comparison against
+Maestro, Appium, Detox, XCTest, or another runner.
+
+## Result
+
+| Metric | Value |
+| --- | --- |
+| Scenario | `examples/ios-smoke.json` |
+| Platform | iOS simulator |
+| Runs | 20 |
+| Passed | 20 |
+| Failed | 0 |
+| Pass rate | 100.00% |
+| Mean duration | 4192 ms |
+| p95 duration | 5409 ms |
+| Min duration | 3065 ms |
+| Max duration | 6395 ms |
+
+## Environment
+
+| Field | Value |
+| --- | --- |
+| ZMR runner | `0.1.8` |
+| Protocol | `2026-04-28` |
+| Host OS | macOS 26.6, arm64 |
+| Xcode | 26.5, build 17F42 |
+| Simulator | Booted iOS 26.5 simulator, `zmr-iPhone` |
+| App id | `com.example.mobiletest` |
+| Demo app source | Generated by `scripts/create-ios-demo-app.sh` |
+
+## Command
+
+```bash
+rm -rf traces/public-benchmarks/20260609-ios-demo /tmp/zmr-public-benchmark-ios-demo-20260609-plain
+
+scripts/create-ios-demo-app.sh \
+  --out /tmp/zmr-public-benchmark-ios-demo-20260609-plain \
+  --name ZMRDemo \
+  --bundle-id com.example.mobiletest \
+  --deployment-target 16.0
+
+xcodebuild \
+  -project /tmp/zmr-public-benchmark-ios-demo-20260609-plain/ios/ZMRDemo.xcodeproj \
+  -scheme ZMRDemo \
+  -destination 'generic/platform=iOS Simulator' \
+  -configuration Debug \
+  -derivedDataPath /tmp/zmr-public-benchmark-ios-demo-20260609-plain/DerivedData \
+  build
+
+ZMR_BIN="$PWD/zig-out/bin/zmr" scripts/run-ios-pilot.sh \
+  --app-root /tmp/zmr-public-benchmark-ios-demo-20260609-plain \
+  --app-path /tmp/zmr-public-benchmark-ios-demo-20260609-plain/DerivedData/Build/Products/Debug-iphonesimulator/ZMRDemo.app \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --xcrun xcrun \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-demo \
+  --min-pass-rate 100 \
+  --max-failures 0
+```
+
+The local run wrote:
+
+- `traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/results.jsonl`
+- `traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/report.html`
+- `traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/junit.xml`
+
+Those raw artifacts are intentionally not committed because generated reports
+and JUnit properties include absolute local paths. The sanitized result rows are
+committed in
+[2026-06-09-ios-demo.results.jsonl](2026-06-09-ios-demo.results.jsonl).
+
+## Scope
+
+This run validates the public iOS simulator smoke path: install, launch, deep
+link, snapshot, trace collection, benchmark gate, HTML report generation, and
+JUnit report generation.
+
+This run does not measure the app-local XCTest shim path. During collection, a
+shim compatibility check found a generated wrapper portability issue in
+one-shot mode: the wrapper used a `mktemp` template with `XXXXXX` before a
+`.log` suffix, which is not portable on macOS. That generator bug was fixed
+before this evidence pack was committed.
+
+Do not use this evidence as a speed comparison against another mobile testing
+framework. Baseline comparisons require equivalent candidate and baseline rows
+from the same app build, device state, and scenario.

package/docs/benchmarks/2026-06-09-ios-demo.results.jsonl

@@ -0,0 +1,20 @@
+{"tool":"zmr","run":1,"status":"ok","durationMs":5409,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":2,"status":"ok","durationMs":3289,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":3,"status":"ok","durationMs":3319,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":4,"status":"ok","durationMs":3323,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":5,"status":"ok","durationMs":3562,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":6,"status":"ok","durationMs":4048,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":7,"status":"ok","durationMs":3606,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":8,"status":"ok","durationMs":3065,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":9,"status":"ok","durationMs":3692,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":10,"status":"ok","durationMs":3514,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":11,"status":"ok","durationMs":3702,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":12,"status":"ok","durationMs":5092,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":13,"status":"ok","durationMs":4929,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":14,"status":"ok","durationMs":4340,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":15,"status":"ok","durationMs":6395,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":16,"status":"ok","durationMs":5128,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":17,"status":"ok","durationMs":3955,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":18,"status":"ok","durationMs":4410,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":19,"status":"ok","durationMs":4634,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}
+{"tool":"zmr","run":20,"status":"ok","durationMs":4425,"traceDir":"traces/public-benchmarks/20260609-ios-demo/ios-smoke-benchmark/zmr-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","traceStatus":"passed"}

package/docs/benchmarks/2026-06-09-ios-maestro-comparison.md

@@ -0,0 +1,128 @@
+# 2026-06-09 iOS Simulator ZMR vs Maestro Comparison
+
+This evidence pack compares ZMR with Maestro on the generated public iOS demo
+app. It is a command-level benchmark: each row starts the runner command,
+executes one smoke flow on the same booted simulator, and records the command
+duration.
+
+This is not a universal claim about either project. Treat it as reproducible
+evidence for this app, host, simulator, and flow shape.
+
+## Result
+
+| Tool | Runs | Pass rate | Failures | Mean duration | p95 duration |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| ZMR | 20 | 100.00% | 0 | 2236 ms | 2282 ms |
+| Maestro | 20 | 95.00% | 1 | 7578 ms | 7918 ms |
+
+For this run, ZMR was 3.39x faster on mean duration and 3.47x faster on p95
+duration. Maestro run 8 failed while launching the app with a driver connection
+error: `Failed to connect to /127.0.0.1:7001`.
+
+## Environment
+
+| Field | Value |
+| --- | --- |
+| ZMR runner | `0.1.8` |
+| ZMR protocol | `2026-04-28` |
+| Maestro | `1.40.3` |
+| Java | OpenJDK 21.0.6 |
+| Host OS | macOS 26.6, arm64 |
+| Xcode | 26.5, build 17F42 |
+| Simulator | Booted iOS 26.5 simulator, `zmr-iPhone` |
+| App id | `com.example.mobiletest` |
+| App build label | `generated-ios-demo-20260609` |
+| Demo app source | Generated by `scripts/create-ios-demo-app.sh` |
+
+## ZMR Command
+
+```bash
+ZMR_BIN="$PWD/zig-out/bin/zmr" scripts/benchmark.sh \
+  --zmr examples/ios-smoke.json \
+  --platform ios \
+  --ios-device-type simulator \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --xcrun xcrun \
+  --app-build generated-ios-demo-20260609 \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-maestro-comparison/zmr \
+  --results traces/public-benchmarks/20260609-ios-maestro-comparison/results.jsonl \
+  --replace \
+  --min-pass-rate 100 \
+  --max-failures 0
+```
+
+The ZMR scenario launches the app, opens `exampleapp:///e2e-auth?probe=1`, and
+captures a snapshot:
+
+```json
+{
+  "name": "ZMR iOS simulator smoke demo",
+  "appId": "com.example.mobiletest",
+  "steps": [
+    { "action": "launch" },
+    { "action": "openLink", "url": "exampleapp:///e2e-auth?probe=1" },
+    { "action": "snapshot" }
+  ]
+}
+```
+
+## Maestro Command
+
+```bash
+scripts/benchmark-command.sh \
+  --tool maestro \
+  --platform ios \
+  --device booted \
+  --app-id com.example.mobiletest \
+  --scenario examples/ios-smoke.json \
+  --app-build generated-ios-demo-20260609 \
+  --runs 20 \
+  --trace-root traces/public-benchmarks/20260609-ios-maestro-comparison/maestro \
+  --results traces/public-benchmarks/20260609-ios-maestro-comparison/results.jsonl \
+  --min-pass-rate 100 \
+  --max-failures 0 \
+  -- maestro --platform ios --device F839B002-5388-4814-B64F-DBE5A113EBED test --no-ansi /tmp/zmr-maestro-benchmark-20260609/ios-smoke.yaml
+```
+
+The Maestro flow launches the same app, opens the same deep link, and asserts
+the resulting status text:
+
+```yaml
+appId: com.example.mobiletest
+---
+- launchApp
+- openLink: exampleapp:///e2e-auth?probe=1
+- assertVisible: Deep link opened
+```
+
+## Comparison Report
+
+```text
+# Benchmark Comparison
+
+| Tool | Runs | Pass rate | Failures | Mean ms | P95 ms |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| zmr | 20 | 100.00% | 0 | 2236 | 2282 |
+| maestro | 20 | 95.00% | 1 | 7578 | 7918 |
+
+- Mean speedup: 3.39x (-70.5% candidate vs baseline)
+- P95 speedup: 3.47x (-71.2% candidate vs baseline)
+- Same benchmark context: yes
+```
+
+The sanitized result rows are committed in
+[2026-06-09-ios-maestro-comparison.results.jsonl](2026-06-09-ios-maestro-comparison.results.jsonl).
+
+Raw local trace and runner logs are not committed because they can include local
+absolute paths. The failure detail above was taken from the failed Maestro
+run's local stdout log.
+
+## Scope
+
+This benchmark compares the local command surfaces under repeated execution. It
+does not compare cloud execution, long multi-screen flows, Detox, XCTest-only
+flows, or app-specific selectors. Detox requires a project-specific native test
+harness, so it should get its own evidence pack rather than being forced into
+this demo comparison.

package/docs/benchmarks/2026-06-09-ios-maestro-comparison.results.jsonl

@@ -0,0 +1,40 @@
+{"tool":"zmr","run":1,"status":"ok","durationMs":2347,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":2,"status":"ok","durationMs":2233,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":3,"status":"ok","durationMs":2282,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":4,"status":"ok","durationMs":2204,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":5,"status":"ok","durationMs":2231,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":6,"status":"ok","durationMs":2185,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":7,"status":"ok","durationMs":2208,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":8,"status":"ok","durationMs":2196,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":9,"status":"ok","durationMs":2189,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":10,"status":"ok","durationMs":2211,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":11,"status":"ok","durationMs":2268,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":12,"status":"ok","durationMs":2245,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":13,"status":"ok","durationMs":2244,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":14,"status":"ok","durationMs":2249,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":15,"status":"ok","durationMs":2224,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":16,"status":"ok","durationMs":2236,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":17,"status":"ok","durationMs":2236,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":18,"status":"ok","durationMs":2226,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":19,"status":"ok","durationMs":2262,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"zmr","run":20,"status":"ok","durationMs":2248,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/zmr/zmr-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609","traceStatus":"passed"}
+{"tool":"maestro","run":1,"status":"ok","durationMs":7762,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-1","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":2,"status":"ok","durationMs":7793,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-2","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":3,"status":"ok","durationMs":7771,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-3","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":4,"status":"ok","durationMs":7762,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-4","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":5,"status":"ok","durationMs":7838,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-5","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":6,"status":"ok","durationMs":7918,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-6","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":7,"status":"ok","durationMs":7975,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-7","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":8,"status":"failed","durationMs":5091,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-8","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":9,"status":"ok","durationMs":6764,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-9","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":10,"status":"ok","durationMs":7716,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-10","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":11,"status":"ok","durationMs":7738,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-11","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":12,"status":"ok","durationMs":7738,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-12","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":13,"status":"ok","durationMs":7808,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-13","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":14,"status":"ok","durationMs":7772,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-14","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":15,"status":"ok","durationMs":7681,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-15","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":16,"status":"ok","durationMs":7629,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-16","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":17,"status":"ok","durationMs":7702,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-17","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":18,"status":"ok","durationMs":7661,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-18","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":19,"status":"ok","durationMs":7787,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-19","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}
+{"tool":"maestro","run":20,"status":"ok","durationMs":7658,"traceDir":"traces/public-benchmarks/20260609-ios-maestro-comparison/maestro/maestro-20","platform":"ios","device":"booted","appId":"com.example.mobiletest","scenario":"examples/ios-smoke.json","appBuild":"generated-ios-demo-20260609"}