session-orchestrator 3.16.0

package/skills/test-runner/SKILL.md

@@ -0,0 +1,362 @@
+---
+name: test-runner
+user-invocable: false
+tags: [test, orchestrator, e2e, ux]
+model: sonnet
+model-preference: sonnet
+model-preference-codex: gpt-5.4
+model-preference-cursor: claude-sonnet-4-6
+description: >
+  Use this skill when orchestrating agentic end-to-end tests. Resolves target + profile, dispatches
+  the right driver(s) (playwright for web today, peekaboo for macOS (issue #381)), invokes the ux-evaluator agent (opus, read-only) against
+  driver artifacts, reconciles findings with the open issue tracker via
+  scripts/lib/test-runner/issue-reconcile.mjs, and writes report.md +
+  JSONL roll-up. Wraps upstream tools (no forks). Hard-gates Playwright MCP
+  for browser drive (4× token cost vs CLI per Microsoft's own benchmark).
+---
+
+# Test Runner Skill
+
+> Project-instruction file resolution: `CLAUDE.md` and `AGENTS.md` (Codex CLI) are transparent aliases — see [skills/_shared/instruction-file-resolution.md](../_shared/instruction-file-resolution.md). Wherever this skill mentions `CLAUDE.md`, the alias rule applies.
+
+## Soul
+
+Before anything else, read and internalize `soul.md` in this skill directory. It defines WHO you are — your role as an orchestrator, your delegation boundaries, and your non-negotiable constraints.
+
+## Phase 0: Bootstrap Gate
+
+Read `skills/_shared/bootstrap-gate.md` and execute the gate check. If GATE_CLOSED, invoke `skills/bootstrap/SKILL.md` and wait for completion. If GATE_OPEN, continue to Phase 1.
+
+<HARD-GATE>
+Do NOT proceed past Phase 0 if GATE_CLOSED. There is no bypass. Refer to `skills/_shared/bootstrap-gate.md` for the full HARD-GATE constraints.
+</HARD-GATE>
+
+## Phase 1: Read Session Config + Resolve Target / Profile
+
+Read and parse Session Config per `skills/_shared/config-reading.md`. Store result as `$CONFIG`.
+
+Test-runner specific fields (parse these specifically):
+- `test-runner.default-profile` (default: `smoke`)
+- `test-runner.retention-days` (default: `30`)
+- `test-command`, `typecheck-command`, `lint-command` (used for context only — not driven here)
+
+### Target / Profile Resolution
+
+Resolution order (first match wins):
+
+1. **CLI argument** `--target <name> --profile <name>` (explicit, highest priority)
+2. **Policy file lookup** — `.orchestrator/policy/test-profiles.json` by target name (if present)
+3. **Convention-based detection** (marker files):
+   - `playwright.config.{ts,js}` present → target type `web`, dispatch `playwright-driver`
+   - `Package.swift` present → target type `mac`, dispatch `peekaboo-driver` (see `skills/peekaboo-driver/SKILL.md`)
+4. **Fallback** → emit error and halt:
+   ```
+   Error: Cannot resolve target — provide --target or add .orchestrator/policy/test-profiles.json
+   ```
+
+### Run ID
+
+Generate a run ID immediately after target resolution:
+
+```js
+import { makeRunId } from 'scripts/lib/test-runner/artifact-paths.mjs';
+const runId = makeRunId(); // e.g. "your-target-app-1715688000123"
+```
+
+All artifact paths in subsequent phases derive from this run ID. Never use ad-hoc paths.
+
+### Status Report
+
+After resolution, emit: `Test Runner: target=[name] profile=[name] run_id=[runId] driver=[driver]`
+
+## Phase 2: Driver Dispatch
+
+Determine `${RUN_DIR}` from `artifact-paths.mjs:runDirPath(runId)` before dispatching any driver. All drivers write artifacts under `${RUN_DIR}/`.
+
+### --since Filtering (when `since_ref` is provided)
+
+When `since_ref` is set (passed from the `/test --since <git-ref>` handoff contract):
+
+1. Import and call `changedFilesSince(since_ref)` from `scripts/lib/discovery/helpers.mjs`.
+2. If the helper throws (ref unresolvable), surface the error to the user and halt.
+3. If the result is `[]` (no files changed since the ref), emit:
+   ```
+   No files changed since <since_ref>. Skipping test run.
+   ```
+   and exit with status 0. Do NOT fall back to a full-repo test run.
+4. If the result is a non-empty array, JSON-stringify it and set `TEST_CHANGED_FILES` in the driver subprocess environment (see driver invocations below). Driver-side filtering is deferred — drivers receive the env var but do not yet filter by it in this wave.
+
+For each resolved driver:
+
+### Web (playwright-driver)
+
+Dispatch via Bash per `skills/playwright-driver/SKILL.md`. Pass `${RUN_DIR}` so the driver writes all artifacts (screenshots, AX dumps, HAR) under it.
+
+```bash
+# Example invocation shape (exact flags defined by playwright-driver SKILL.md)
+TEST_CHANGED_FILES="${CHANGED_FILES_JSON}" node scripts/lib/playwright-driver/runner.mjs \
+  --run-dir "${RUN_DIR}" \
+  --profile "${PROFILE}" \
+  --target "${TARGET}"
+```
+
+Where `${CHANGED_FILES_JSON}` is `JSON.stringify(changedFiles)` when `--since` was provided, or an empty string otherwise.
+
+Capture exit code. A non-zero exit from Playwright means test failures — these become findings for the UX evaluator. They are NOT a fatal error for the orchestrator. Continue to Phase 3 regardless of exit code.
+
+Log: `playwright-driver exited [code] — [N] test files captured under ${RUN_DIR}`
+
+### macOS (peekaboo-driver)
+
+> See `skills/peekaboo-driver/SKILL.md` for the full dispatch contract, permission probe, and artifact layout.
+
+**Pre-dispatch platform check:** The driver's Phase 1 gate handles the platform and version checks (`darwin` + macOS 15.0+) and exits 0 (non-fatal skip) on incompatible systems. The orchestrator does not need to replicate these checks.
+
+**Permission probe:** The driver runs its own Phase 2 permission probe via `peekaboo permissions status --json`. If required permissions (Screen Recording, Accessibility) are not granted, the driver surfaces an AUQ and exits 2 on failure. The orchestrator treats exit 2 as a driver-framework error, not a test failure.
+
+**Invocation:**
+
+```bash
+# All inputs via environment variables — no positional arguments
+RUN_DIR="${RUN_DIR}" TARGET="${TARGET}" PROFILE="${PROFILE}" bash skills/peekaboo-driver/SKILL.md
+```
+
+Any bash-driver shim or report helper you write for this dispatch is subject to `.claude/rules/bash-harness-pitfalls.md` (`grep -c` double-print, stdout-capture pollution, file-based aggregate verdicts, `perl -pi` script-surgery corruption).
+
+**Outputs the orchestrator must parse:**
+
+| Artifact | Description |
+|---|---|
+| `${RUN_DIR}/exit_code` | Plain integer file written by driver before exit |
+| `${RUN_DIR}/results.json` | Driver summary: `exit_code`, `scenarios_attempted`, `scenarios_passed`, `scenarios_failed` |
+| `${RUN_DIR}/ax-snapshots/<scenario>.json` | peekaboo AX-tree output per scenario |
+| `${RUN_DIR}/ax-snapshots/glass-modifiers-<ts>.json` | Liquid Glass conformance artifact (consumed by ux-evaluator Check 4) |
+| `${RUN_DIR}/screenshots/<step>-<ts>.png` | Per-step screenshots (evidence for ux-evaluator findings) |
+| `${RUN_DIR}/console.ndjson` | Driver log events as NDJSON |
+
+**Exit-code semantics:**
+
+| Code | Meaning | Orchestrator Action |
+|---|---|---|
+| 0 | All captures succeeded (or platform skip) | Record pass or skip, continue to Phase 3 |
+| 1 | At least one capture failed | Failures become findings (non-fatal) — continue to Phase 3 |
+| 2 | Framework error (missing binary, permission denied, OS mismatch) | Surface as driver error; still continue to Phase 3 with available artifacts |
+
+Capture exit code. Exit 1 (capture failures) produces findings for the UX evaluator — it is NOT fatal for the orchestrator. Exit 2 (framework error) is surfaced in the report but does not halt Phase 3. Continue to Phase 3 regardless of exit code.
+
+Log: `peekaboo-driver exited [code] — [N] scenarios captured under ${RUN_DIR}`
+
+## Phase 3: UX Evaluator Dispatch
+
+Invoke the `ux-evaluator` agent (`agents/ux-evaluator.md`) via the Agent tool. The agent reads driver artifacts under `${RUN_DIR}/` and applies `skills/test-runner/rubric-v1.md` (4 checks). The agent writes `findings.jsonl` directly to `${RUN_DIR}/findings.jsonl` — the coordinator does NOT need to forward findings through prompt context.
+
+```js
+Agent({
+  description: `UX evaluate run ${runId}`,
+  prompt: `<scope: ${RUN_DIR}, rubric: skills/test-runner/rubric-v1.md, output: ${RUN_DIR}/findings.jsonl>`,
+  subagent_type: "ux-evaluator",
+  run_in_background: false
+})
+```
+
+`run_in_background: false` is mandatory — Phase 4 depends on `findings.jsonl` being fully written before reconciliation begins.
+
+After the agent completes, verify `${RUN_DIR}/findings.jsonl` exists. If missing, emit a warning and skip Phase 4 (no findings to reconcile).
+
+## Phase 4: Issue Reconciliation
+
+Read `${RUN_DIR}/findings.jsonl`. Use the helpers in `scripts/lib/test-runner/issue-reconcile.mjs` for all glab/gh interactions — never call `glab` or `gh` directly.
+
+### Available Helper Functions (issue-reconcile.mjs)
+
+| Function | Purpose |
+|---|---|
+| `listExistingFindings({glabPath, project, label, maxBuffer})` | Query the tracker for all open `from:test-runner` issues; returns `{ok, issues[], fingerprints: Set}` |
+| `createFinding({glabPath, project, fingerprint, title, body, labels, dryRun, maxBuffer})` | Create a new issue; returns `{ok, action: 'create', iid?, command?}` |
+| `updateFinding({glabPath, project, iid, comment, dryRun, maxBuffer})` | Add a comment to an existing issue; returns `{ok, action: 'comment', command?}` |
+| `triageDecision(finding, candidates)` | Pure decision: fingerprint-exact → `ignore`; Levenshtein ≤ 2 on title → `update`; else → `create` |
+| `reconcileFinding({finding, existingFingerprints, glabPath, dryRun})` | Track-A legacy helper — single-finding create-or-noop using a pre-built fingerprint Set |
+
+Security notes (ADR-364 §C5, #388, #389):
+- All execFile calls use `maxBuffer: 4 MB` and `shell: false`.
+- Bodies > 65 536 bytes are rejected with `{ok: false, error: {code: 'BODY_TOO_LARGE', ...}}`.
+- `**Fingerprint:**` literals in free-text fields are replaced with `__Fingerprint__` before the authoritative sentinel line is appended (sentinel-injection hardening, #388).
+
+### Existing-Fingerprint Dedup
+
+Before iterating findings, build the dedup set once:
+
+```js
+import { listExistingFindings } from 'scripts/lib/test-runner/issue-reconcile.mjs';
+
+const listResult = await listExistingFindings({ glabPath });
+// listResult.ok === false → log and continue with empty Set (conservative)
+const existingFingerprints = listResult.ok ? listResult.fingerprints : new Set();
+const existingIssues = listResult.ok ? listResult.issues : [];
+```
+
+If the glab query fails, log the error and proceed with an empty fingerprint set (conservative: may create duplicate issues, safer than silently skipping).
+
+### Severity Routing
+
+| Severity | Action |
+|----------|--------|
+| `critical` | Auto-create issue — no AUQ. Label: `from:test-runner,priority:critical` |
+| `high` | Auto-create issue — no AUQ. Label: `from:test-runner,priority:high` |
+| `medium` | Batched AUQ triage (see below) |
+| `low` | Batched AUQ triage (see below) |
+
+For `critical` and `high`, call `triageDecision(finding, existingIssues)` first. If `action === 'ignore'`, skip. If `action === 'update'`, call `updateFinding()`. If `action === 'create'`, call `createFinding()`.
+
+### Batched AUQ Triage (medium / low)
+
+Group `medium` and `low` findings and present via a single `AskUserQuestion` call (mirror the discovery skill's Phase 5 Step 3 pattern):
+
+```js
+AskUserQuestion({
+  questions: [{
+    question: `<N> medium/low findings to triage. How to handle?`,
+    header: "Test-runner triage",
+    options: [
+      {
+        label: "Create all (Recommended)",
+        description: "File <N> new issues, all with label from:test-runner"
+      },
+      {
+        label: "Review each",
+        description: "Walk through findings individually with create/update/ignore"
+      },
+      {
+        label: "Skip all",
+        description: "Dismiss medium/low; only critical/high get auto-filed"
+      }
+    ],
+    multiSelect: false
+  }]
+})
+```
+
+**"Create all"**: For each medium/low finding call `triageDecision(finding, existingIssues)`:
+- `ignore` → skip silently.
+- `update` → call `updateFinding({iid: target, comment, ...})`.
+- `create` → call `createFinding({fingerprint, title, body, labels, ...})`.
+
+**"Review each"**: Walk through findings one by one. For each, call `triageDecision` to get the recommended action, then present a per-finding AUQ with three options (Create / Update existing `#<iid>` / Skip). Honour the user's selection.
+
+**"Skip all"**: Dismiss all medium/low findings without filing. Log `skipped: N medium/low findings (user dismissed)`.
+
+### Reconcile Outcomes
+
+Collect outcomes from reconciliation calls into three buckets:
+- `created[]` — issue IIDs of newly created issues
+- `commented[]` — issue IIDs of existing issues that received a new comment
+- `noop[]` — fingerprints/IIDs where no action was taken (dedup hit or user dismissed)
+
+Pass these buckets to Phase 5.
+
+## Phase 5: Report + JSONL Roll-up
+
+### report.md
+
+Write `${RUN_DIR}/report.md` — a markdown summary of the run. Minimum content:
+
+```markdown
+# Test Run Report
+
+**Run ID:** <runId>
+**Target:** <target>
+**Profile:** <profile>
+**Timestamp:** <ISO 8601>
+**Duration:** <ms>ms
+
+## Findings Summary
+
+| Severity | Count |
+|----------|-------|
+| critical | N |
+| high | N |
+| medium | N |
+| low | N |
+
+## Check Breakdown
+
+| Check | Findings |
+|-------|---------|
+| <check_name> | N |
+
+## Issue Actions
+
+- **Created:** <IIDs>
+- **Commented:** <IIDs>
+- **No-op (dedup):** <IIDs>
+
+## Driver Exit Codes
+
+| Driver | Exit Code |
+|--------|-----------|
+| playwright | N |
+```
+
+### JSONL Roll-up
+
+Append one record to `.orchestrator/metrics/test-runs.jsonl` (path from `artifact-paths.mjs:jsonlRollupPath()`) via `appendJsonlAtomic` from `scripts/lib/autopilot/telemetry.mjs`. Never use `appendFileSync` — POSIX-atomic guarantee is required (per #376).
+
+Schema v1:
+
+```json
+{
+  "schema_version": 1,
+  "run_id": "your-target-app-1715688000123",
+  "timestamp": "2026-05-14T07:30:00Z",
+  "target": "your-target-app",
+  "profile": "smoke",
+  "drivers": ["playwright"],
+  "finding_counts": {"critical": 0, "high": 2, "medium": 5, "low": 3},
+  "issues": {"created": [388, 389], "commented": [], "noop": [380]},
+  "exit_code": 0,
+  "duration_ms": 12345
+}
+```
+
+`exit_code` is the orchestrator's own exit code: `0` if no critical/high findings auto-created, `1` otherwise.
+
+### Final Status Line
+
+Emit to stdout:
+
+```
+Test Runner complete: run_id=<runId> findings=[total] issues_created=[N] duration=[ms]ms
+```
+
+## Anti-Patterns
+
+- **DO NOT** invoke Playwright or Peekaboo directly from the coordinator prompt — always dispatch via the wrapper skills (`skills/playwright-driver/SKILL.md`, `skills/peekaboo-driver/SKILL.md`).
+- **DO NOT** inline AX-tree dumps or screenshots into coordinator or agent prompt context — artifacts go to disk under `${RUN_DIR}`.
+- **DO NOT** re-file existing findings — fingerprint-based dedup via `issue-reconcile.mjs` is mandatory every run.
+- **DO NOT** use `@playwright/mcp` for browser drive — R5 hard-gate enforced by `scripts/lib/validate/check-playwright-mcp-canary.mjs` (4× token cost per Microsoft benchmark).
+- **DO NOT** call `glab` or `gh` directly — always go through `scripts/lib/test-runner/issue-reconcile.mjs` (execFile + binary allowlist + arg-validation, per ADR-364 §C5).
+- **DO NOT** write JSONL via `appendFileSync` — use `appendJsonlAtomic` from `scripts/lib/autopilot/telemetry.mjs`.
+
+## Critical Rules
+
+- **NEVER** auto-merge MRs created by issue-reconcile.
+- **NEVER** persist artifacts outside `${RUN_DIR}` — it is the single source of truth for a run.
+- **NEVER** call `glab`/`gh` directly — always route through `issue-reconcile.mjs`.
+- **ALWAYS** use the orchestrator's run-id everywhere (from `artifact-paths.mjs:makeRunId()`).
+- **ALWAYS** invoke `appendJsonlAtomic` for JSONL roll-up — never `appendFileSync` (POSIX-atomic guarantee, per #376).
+- **ALWAYS** set `run_in_background: false` for the `ux-evaluator` agent dispatch — Phase 4 depends on its output.
+
+## Sub-File Reference
+
+| File | Purpose |
+|------|---------|
+| `soul.md` | Orchestrator identity — read before Phase 0 |
+| `rubric-v1.md` | UX rubric specification (4 checks, fingerprint formula, findings JSON schema) — read by `ux-evaluator` agent |
+
+## Implementation Modules
+
+| File | Purpose |
+|------|---------|
+| `scripts/lib/test-runner/fingerprint.mjs` | Stable 16-hex fingerprint per finding (deterministic, idempotent) |
+| `scripts/lib/test-runner/artifact-paths.mjs` | Run-dir + artifact path builders — pure functions, no side effects |
+| `scripts/lib/test-runner/issue-reconcile.mjs` | glab/gh dispatch via execFile + binary allowlist (ADR-364 §C5) |