@haus-tech/haus-workflow 7.1.0 → 7.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. package/CHANGELOG.md +27 -0
  2. package/dist/cli.js +9712 -9498
  3. package/package.json +1 -1
  4. package/packages/agents/qa-fixer.md +66 -0
  5. package/packages/agents/qa-inventory.md +36 -0
  6. package/packages/agents/qa-reviewer.md +32 -0
  7. package/packages/agents/qa-tester.md +85 -0
  8. package/packages/packages.yaml +8 -0
  9. package/packages/skills/fixing-weak-tests/SKILL.md +89 -0
  10. package/packages/skills/fixing-weak-tests/references/audit-brief.md +57 -0
  11. package/packages/skills/fixing-weak-tests/references/fix-brief.md +55 -0
  12. package/packages/skills/optimizing-test-suites/SKILL.md +157 -0
  13. package/packages/skills/optimizing-test-suites/references/affected-tests.md +46 -0
  14. package/packages/skills/optimizing-test-suites/references/levers.md +79 -0
  15. package/packages/skills/optimizing-test-suites/scripts/slowest.mjs +78 -0
  16. package/packages/skills/qa-genomklickning/SKILL.md +179 -0
  17. package/packages/skills/qa-genomklickning/drivers.md +159 -0
  18. package/packages/skills/qa-genomklickning/report.md +23 -0
  19. package/packages/skills/qa-genomklickning/scripts/android-setup.sh +26 -0
  20. package/packages/skills/qa-genomklickning/scripts/android.sh +62 -0
  21. package/packages/skills/qa-genomklickning/scripts/chrome-tab-guard.py +55 -0
  22. package/packages/skills/qa-genomklickning/scripts/env-check.sh +32 -0
  23. package/packages/skills/qa-genomklickning/scripts/ios-sim.sh +49 -0
  24. package/packages/skills/request/SKILL.md +3 -2
  25. package/packages/skills/request/scripts/compare-scores.mjs +23 -11
  26. package/packages/skills/testsammanfattning/SKILL.md +134 -0
  27. package/packages/skills/testsammanfattning/references/konfiguration.md +110 -0
  28. package/packages/skills/testsammanfattning/scripts/test-log.mjs +218 -0
  29. package/packages/skills/testsammanfattning/scripts/test-summary.mjs +495 -0
  30. package/packages/templates/agentic-workflow-standard.md +3 -3
@@ -0,0 +1,46 @@
1
+ # Running only affected tests: what the tools really do
2
+
3
+ Verified against the source of Vitest 4.1 and Playwright 1.62. Re-check the source when the
4
+ version differs: selection logic changes between releases.
5
+
6
+ ## Vitest `--changed <ref>` and `vitest related <files>`
7
+
8
+ 1. Changed files = `git diff --name-only <ref>...HEAD` plus staged, unstaged, untracked.
9
+ 2. Selects every test file whose **static import graph** (from Vite's transform) contains a
10
+ changed file (`filterTestsBySource`, `getTestDependencies` in `vitest/dist/chunks/cli-api.*.js`).
11
+ 3. Right for unit tests. Wrong for e2e files that all import one app config which imports
12
+ every plugin: any plugin change selects every file. Measure it: build the graph with
13
+ `createVitest` and `getTestDependencies` per file, then count which source files are
14
+ shared by all files. Only changes outside that shared set give a real saving.
15
+ 4. Hooks: `experimental.vcsProvider` (a changed test file always selects itself, so a
16
+ provider returning chosen test files becomes your own selector) and
17
+ `coverage.provider: 'custom'` (per-file coverage).
18
+
19
+ ## Playwright `--only-changed=<ref>`
20
+
21
+ 1. Follows only files the runner loads with a spec (`collectAffectedTestFiles`). Specs that
22
+ reach the app over HTTP never import app source, so an app change selects **nothing**.
23
+ 2. The dependency hook (`populateDependencies`) is private, used by component testing only.
24
+ 3. Usable for helper or fixture changes. For app changes use `--test-list <file>` with your
25
+ own selection.
26
+
27
+ ## Options, cheapest first
28
+
29
+ 1. **Name the files** you touched while iterating; `--last-failed` after a red run;
30
+ `--project <name>` when only one side changed. Full suites once before the PR. Zero code.
31
+ 2. **Static rule:** run everything when a shared file changes, otherwise `vitest related`.
32
+ 2 to 3 hours. Helps only when changes are outside the shared set.
33
+ 3. **Coverage impact map:** a full green run records which lines each test file (or test)
34
+ executes, using `node:inspector` precise coverage plus source maps
35
+ (`@jridgewell/trace-mapping`). The map stores its commit. The selector diffs
36
+ `git diff -U0 <map commit>`, runs tests that executed a changed line, and runs
37
+ **everything** for lockfiles, `package.json`, test config, test helpers and any file the
38
+ map does not know (templates, CSV, CSS). It also prints changed lines no test executes.
39
+ About 1 day per suite plus a half-day spike; every server process in the recording run
40
+ must load the coverage hook; the recording run is slower (no template cache, dev build
41
+ for source maps).
42
+ 4. **Hand-kept spec-to-source map or tags:** not recommended as the gate. Tests reach code
43
+ through HTTP and plugins affect each other at runtime, which is what e2e exists to catch.
44
+
45
+ Recommend by payoff: once the full suites are a few minutes, option 1 is usually enough and
46
+ option 3 is a decision for the user with its cost stated.
@@ -0,0 +1,79 @@
1
+ # Levers, with the gotchas that bite
2
+
3
+ Each lever: when it applies, how, what to check. Examples are Vitest 4 and Playwright 1.5x
4
+ with Postgres; the pattern carries to other stacks.
5
+
6
+ ## 1. Parallel workers for server-booting e2e files (Vitest)
7
+
8
+ When: each file boots the app on a fixed port against one DB, `fileParallelism: false`.
9
+
10
+ 1. Per-worker keys from `process.env.VITEST_POOL_ID` (1..maxWorkers in pool `forks`):
11
+ DB name `${base}_w${id}`, port from a list picked in `globalSetup`.
12
+ 2. `globalSetup` runs once in the main process; `process.env` values it sets reach the
13
+ fork workers. Pick free ports there (probe with `net.createServer().listen`) and pass
14
+ them as one env var, so two worktrees on one machine do not collide.
15
+ 3. Remove `fileParallelism: false`, set `maxWorkers` from an env var (default = measured
16
+ knee, often 4).
17
+ 4. Check shared external resources: object storage buckets (tests that list or empty a
18
+ bucket), mail outboxes, job queues (per DB is fine), caches keyed by name.
19
+ 5. Validate env-derived DB names with a regex before using them in SQL.
20
+
21
+ ## 2. Build once, clone per file (seeded template DB)
22
+
23
+ When: several files run the same expensive seed in `beforeAll`.
24
+
25
+ 1. Name each seed (`hosted_catalogue`, `test_clubs`); the template DB is `${base}_t_${name}`.
26
+ 2. First file to need it, under `pg_advisory_lock(hashtext(templateName))`: init a normal
27
+ DB, run the seed through the app, wait for job queues to go idle, stop the server,
28
+ `CREATE DATABASE tpl TEMPLATE workerDb`. Later files: `CREATE DATABASE workerDb TEMPLATE tpl`
29
+ (about 1 s) and skip populate.
30
+ 3. `CREATE DATABASE ... TEMPLATE` fails if any connection is open on the source: close the
31
+ app first.
32
+ 4. Drop old templates in `globalSetup` so a seed change is never served from a stale template.
33
+ 5. Tests that test the seed itself keep running the seed.
34
+ 6. Other databases: MySQL has no `TEMPLATE`; dump the seeded DB once (`mysqldump`) and
35
+ restore per worker, or `CREATE TABLE w.t LIKE tpl.t` + `INSERT ... SELECT` per table.
36
+ SQLite: copy the file. Any DB: a snapshot of a container volume also works.
37
+ 7. Truncate in `beforeEach` on one shared DB: parallel only after each worker has its own DB
38
+ (lever 1). Wrapping each test in a transaction that rolls back is often faster than
39
+ truncating, when the app under test uses the same connection.
40
+
41
+ ## 3. Parallel browser tests with a stub API (Playwright)
42
+
43
+ When: `workers: 1` because one stub server keeps in-memory state.
44
+
45
+ 1. One stub server and one app server per worker: ports `base + n`, the worker reads
46
+ `process.env.TEST_PARALLEL_INDEX` (0..workers-1, stable per worker slot).
47
+ 2. `webServer` entries: N stubs, build once in the first app entry, `preview` for the others.
48
+ 3. Projects that share a real DB keep `workers: 1` on that project (per-project `workers`)
49
+ and are listed **first**, so their worker starts at once instead of after the parallel
50
+ project drains.
51
+ 4. `fullyParallel: true` on the stub project only.
52
+ 5. Browser tests against one real backend and DB: either every test creates its own data
53
+ (unique email, cart, shop per test, nothing global mutated) and runs in parallel, or the
54
+ backend runs once per worker with its own DB (lever 1 + 2). Tests that change global state
55
+ (settings, statuses) stay in a `workers: 1` project.
56
+ 6. Never `reuseExistingServer` for stub servers: a dev server on that port would make
57
+ isolation tests prove nothing.
58
+
59
+ ## 4. Short waits in the test build only
60
+
61
+ When: tests wait for retry backoff (TanStack Query default 1, 2, 4 s), polling or sleeps.
62
+
63
+ 1. A build-time env var (`VITE_TEST_RETRY_DELAY_MS`), inlined by the bundler, read in one
64
+ small function that only overrides `retryDelay` when set. Keep the retry count, so the
65
+ retry path is still tested.
66
+ 2. Unit test both: unset leaves defaults, set overrides.
67
+ 3. Check every other build (Lighthouse, production, preview deploys) rebuilds without it.
68
+ 4. Fixed sleeps in tests: replace with waiting for a condition (`expect.poll`,
69
+ `waitForResponse`, queue idle).
70
+
71
+ ## 5. Wrong layer
72
+
73
+ A browser or e2e test whose assertion is a formatter, validator or calculation: write the
74
+ unit test, mutate the code to prove the unit test catches it, then remove the slow test.
75
+
76
+ ## Measuring workers
77
+
78
+ Run 1, 4, 6 workers on an idle machine. Pick the knee (in one repo, 6 workers saved 20 s
79
+ over 4 on a 4 min run, so 4 stayed). Postgres `max_connections` and CPU are usual ceilings.
@@ -0,0 +1,78 @@
1
+ #!/usr/bin/env node
2
+ // Usage: node slowest.mjs <report.json> [top=10]
3
+ // Reads a Vitest (--reporter=json) or Playwright (--reporter=json) report and prints
4
+ // wall time, the sum of test time, and the slowest files and tests.
5
+ import { readFileSync } from 'node:fs'
6
+
7
+ const [file, topArg] = process.argv.slice(2)
8
+ if (!file) {
9
+ console.error('usage: node slowest.mjs <report.json> [top=10]')
10
+ process.exit(1)
11
+ }
12
+ const top = Number(topArg ?? 10)
13
+ const report = JSON.parse(readFileSync(file, 'utf8'))
14
+ const tests = [] // { file, title, ms, status }
15
+ let wallMs
16
+
17
+ if (Array.isArray(report.testResults)) {
18
+ // Vitest (Jest-compatible shape)
19
+ const starts = report.testResults.map((f) => f.startTime).filter(Boolean)
20
+ const ends = report.testResults.map((f) => f.endTime).filter(Boolean)
21
+ if (starts.length && ends.length) wallMs = Math.max(...ends) - Math.min(...starts)
22
+ for (const f of report.testResults) {
23
+ for (const t of f.assertionResults ?? []) {
24
+ tests.push({
25
+ file: f.name,
26
+ title: t.fullName ?? t.title,
27
+ ms: t.duration ?? 0,
28
+ status: t.status,
29
+ })
30
+ }
31
+ }
32
+ } else if (Array.isArray(report.suites)) {
33
+ // Playwright
34
+ wallMs = report.stats?.duration
35
+ const walk = (suite, specFile) => {
36
+ const fileName = suite.file ?? specFile
37
+ for (const spec of suite.specs ?? []) {
38
+ for (const t of spec.tests ?? []) {
39
+ const results = t.results ?? []
40
+ const ms = results.reduce((sum, r) => sum + (r.duration ?? 0), 0)
41
+ const status = results.at(-1)?.status ?? t.status
42
+ tests.push({
43
+ file: `${t.projectName ?? ''}:${spec.file ?? fileName}`,
44
+ title: spec.title,
45
+ ms,
46
+ status,
47
+ })
48
+ }
49
+ }
50
+ for (const child of suite.suites ?? []) walk(child, fileName)
51
+ }
52
+ for (const s of report.suites) walk(s, s.file)
53
+ } else {
54
+ console.error('Unknown report shape: expected Vitest testResults[] or Playwright suites[]')
55
+ process.exit(1)
56
+ }
57
+
58
+ const s = (ms) => `${(ms / 1000).toFixed(1)} s`
59
+ const sum = tests.reduce((acc, t) => acc + t.ms, 0)
60
+ const byFile = new Map()
61
+ for (const t of tests) byFile.set(t.file, (byFile.get(t.file) ?? 0) + t.ms)
62
+ const failed = tests.filter((t) => !['passed', 'expected', 'skipped', 'pending'].includes(t.status))
63
+
64
+ console.log(`tests: ${tests.length} failed: ${failed.length}`)
65
+ console.log(
66
+ `wall: ${wallMs ? s(wallMs) : 'n/a'} sum of test time: ${s(sum)}${wallMs ? ` ratio: ${(sum / wallMs).toFixed(2)}` : ''}`,
67
+ )
68
+ console.log('(ratio near 1 = serial; above 1 = parallel; setup in hooks is not in the sum)')
69
+ console.log(`\nslowest ${top} files:`)
70
+ for (const [f, ms] of [...byFile].sort((a, b) => b[1] - a[1]).slice(0, top))
71
+ console.log(` ${s(ms).padStart(8)} ${f}`)
72
+ console.log(`\nslowest ${top} tests:`)
73
+ for (const t of [...tests].sort((a, b) => b.ms - a.ms).slice(0, top))
74
+ console.log(` ${s(t.ms).padStart(8)} ${t.title} (${t.file})`)
75
+ if (failed.length) {
76
+ console.log('\nnot passed:')
77
+ for (const t of failed) console.log(` ${t.status} ${t.title} (${t.file})`)
78
+ }
@@ -0,0 +1,179 @@
1
+ ---
2
+ name: qa-genomklickning
3
+ description: Use when everything built on a branch (a release branch, a feature branch, a PR) must be clicked through and probed in as many combinations as possible before release, with the bugs found fixed test-first and re-verified. Triggers on "/qa-genomklickning", "klicka igenom allt", "QA:a releasebranchen", "testa allt som gjorts på branchen", "genomklickning", "full QA pass", "click through everything on this branch".
4
+ argument-hint: '[branch] [scope-base]'
5
+ ---
6
+
7
+ # QA genomklickning
8
+
9
+ You are the QA orchestrator. You click through and probe EVERYTHING built in the
10
+ scope, in as many relevant combinations as possible, fix what is broken with TDD,
11
+ and verify the fixes. Subagents do the work in parallel; you plan, coordinate,
12
+ merge and report. You run in the main conversation on purpose: only the main
13
+ conversation can ask the user a question, and it keeps the fan-out one level deep.
14
+
15
+ **Violating the letter of these rules is violating the spirit of these rules.**
16
+
17
+ ## Step 0: Which branch
18
+
19
+ Arguments: `$ARGUMENTS` (first word = branch under test, second = optional scope base).
20
+
21
+ 1. No branch given: run `git branch -a --sort=-committerdate | head -15` and
22
+ `gh pr list --limit 10`, then ask with AskUserQuestion: "Which branch should the
23
+ QA pass run against?" Options: the current branch, the 2-3 most recent
24
+ `release/*` branches, the head branch of the newest open PR. Wait for the answer.
25
+ This is the only question you ask before the report.
26
+ 2. Make sure the repo's main checkout has that branch checked out and clean, and
27
+ that the local app runs that code. Never switch branches in a checkout with
28
+ uncommitted work: stop and say so.
29
+ 3. Scope base: the second argument, else the base of the branch's PR
30
+ (`gh pr view <branch> --json baseRefName`), else `git merge-base origin/<default> <branch>`.
31
+
32
+ Everything else (specs, environment, logins, limits) you find yourself: PR body,
33
+ commit messages, ticket references (Asana MCP), `docs/`, ADRs, README,
34
+ `package.json`, `.claude/launch.json`, running ports (`lsof -iTCP -sTCP:LISTEN`),
35
+ seed files and `.env.example`. Record what you found and where in `CONTEXT.md`,
36
+ together with: repo path, branch and HEAD, scope base, running URLs and ports,
37
+ test credentials source, DB access command, the shell quirks of this machine, the
38
+ commit co-author line, and the rule that agents never edit, commit or switch
39
+ branches in the main checkout.
40
+
41
+ ## Work folder
42
+
43
+ `<repo>/.qa/<YYYY-MM-DD>-<branch-slug>/`. Add `.qa/` to `.git/info/exclude` so it
44
+ is never committed. Holds `CONTEXT.md`, `inventory-<area>.md`, `testplan-<area>.md`,
45
+ `results-<area>.md`, `ledger.md`, `auth/`, `shots/`, and the final report.
46
+ `ledger.md` is the single source of truth: one row per test case with id, area,
47
+ priority, driver, status, evidence path, attempts. You work from the ledger, not
48
+ from memory; your context will not hold 200 cases.
49
+
50
+ ## Agents and models
51
+
52
+ | # | Role | Agent | Model / effort |
53
+ | --- | ------------------------------------------- | -------------- | ------------------------------------------------------------------- |
54
+ | 1 | Inventory: git, tickets, routes, test infra | `qa-inventory` | haiku / low |
55
+ | 2 | Test plan + execution per area | `qa-tester` | sonnet / medium |
56
+ | 3 | Root cause + TDD fix | `qa-fixer` | opus / high (sonnet for a local one-file bug: pass `model: sonnet`) |
57
+ | 4 | Second fixer after a failed first one | `qa-fixer` | opus, `effort: xhigh` |
58
+ | 5 | Independent review + final verification | `qa-reviewer` | opus / high |
59
+
60
+ The agent files live in `.claude/agents/`. Each brief you write names the work
61
+ folder, the area, the driver lanes allowed, and the test-data prefix. Briefs never
62
+ mention time, deadlines or that anyone is waiting.
63
+
64
+ ## Drivers (how agents reach the app)
65
+
66
+ Read `drivers.md` in this folder before Phase 3. Summary:
67
+
68
+ | # | Lane | Use for | Parallel limit |
69
+ | --- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
70
+ | 1 | `headless` (playwright-cli, isolated session per agent) | default for everything: desktop, mobile viewports, WebKit, two tabs, double submit, offline, slow or failing API | 6 agents, several sessions each |
71
+ | 2 | `chrome` (Claude in Chrome, the user's logged-in Chrome) | only what needs the user's real session or real Chrome profile | 2 agents, own tab, `tabId` on every call, never resize |
72
+ | 3 | `ios-sim` (cloned iOS Simulator, real Mobile Safari) | mobile Safari smoke and P1 flows on iPhone | 3 simulators |
73
+ | 4 | `android` (emulator, real Chrome on Android driven over DevTools) | Android Chrome smoke and P1 flows; without the SDK: BLOCKED "Android SDK not installed" | 2 emulators (`qa1`, `qa2`) |
74
+ | 5 | `api` / `db` (curl, GraphQL, psql) | verifying data behind every UI result, preconditions, fast probes | unlimited |
75
+
76
+ Before Phase 3: run `bash .claude/skills/qa-genomklickning/scripts/env-check.sh`
77
+ and create a headless auth state once (see `drivers.md`). Test values only: seed or
78
+ example-config credentials on a local host. Never the user's real password.
79
+
80
+ ## Phases
81
+
82
+ 1. **Inventory** (parallel `qa-inventory`, one per area). What was done (`git log`,
83
+ `git diff <base>...<branch>`), tickets and acceptance criteria (also implicit
84
+ ones), affected routes/components/forms/endpoints/jobs/emails/permissions/flags,
85
+ existing test infra. You merge into a traceability matrix in `ledger.md`:
86
+ ticket → criterion → files → UI/endpoint → how it is reached. Also list code
87
+ without spec and spec without code. Split into independent areas. Do not wait
88
+ for the user to approve the matrix; continue.
89
+ 2. **Test plan** (inside each `qa-tester`, first half). Dimensions: data/state,
90
+ input, roles, flows, environment (desktop + 375 px mobile + iOS Safari), errors,
91
+ cross-ticket interaction. Pairwise for breadth, full coverage for payment,
92
+ permissions, data loss and tenant isolation. P1/P2/P3.
93
+ 3. **Execution** (parallel `qa-tester`, second half). Each case: steps, expected
94
+ (with spec source), actual, status, evidence. Data verified in API/DB, not only
95
+ in UI. Every FAIL reproduced and minimised.
96
+ 4. **Fix with TDD** (`qa-fixer`, `isolation: worktree`). Group failures by root
97
+ cause, one issue per root cause. Fixes that touch the same files run one after
98
+ another. Each fixer branches `qa/<slug>` from the branch under test.
99
+ 5. **Review and re-verify**. `qa-reviewer` (never one that wrote a fix) reviews
100
+ each fix. You integrate: create `qa/fixes-<date>` from the branch under test,
101
+ rebase each approved fix branch onto it and fast-forward, run the full suite
102
+ (lint, typecheck, unit, relevant e2e) after EACH one. Then re-run all P1 cases,
103
+ all earlier FAILs, and every PASS whose code a fix touched, on the integrated
104
+ code. New failures go back to Phase 4.
105
+ 6. **Land**: fast-forward the branch under test to `qa/fixes-<date>` locally.
106
+ Before pushing it, check what it deploys (`.haus-workflow/deploy.json`,
107
+ push-triggered workflows, Vercel link). Production or customer-visible: do not
108
+ push, say so in the report and the final message.
109
+ 7. **Report**: `qa-report-<YYYY-MM-DD>.md` in the work folder, structure in
110
+ `report.md`. In English.
111
+
112
+ ## Stubbornness (envishet)
113
+
114
+ The pass is finished when every case is PASS, FAIL with an issue, or BLOCKED with
115
+ proof. Not before. Nobody is waiting; speed is never a reason to stop.
116
+
117
+ 1. **NOT RUN is not a result.** A case may only end BLOCKED after the unblock
118
+ ladder, with each rung's attempt and outcome written in the ledger:
119
+ 1. Switch lane (chrome → headless → ios-sim, or UI → API).
120
+ 2. Create your own data instead of touching shared state (`qa-<area>-` shop,
121
+ product, customer, order). Global toggles that cannot be owned go in the
122
+ serial lane: you run those cases alone at the end of the round.
123
+ 3. Build the missing precondition: seed the customer account through the API,
124
+ start a second server instance on another port against the same database,
125
+ throttle or fail the network with `route`, set the clock.
126
+ 4. Ask a fresh tester (different model or lane) to try it.
127
+ Only prod data, real money, real email to real people, or a third-party
128
+ system outside your control make a case irreducibly BLOCKED.
129
+ 2. **Seen once is a FAIL.** A failure seen once and not on the next try is
130
+ intermittent, not "could not reproduce". Run it at least 10 times, varying:
131
+ click before and after hydration, cold and warm cache, slow and offline
132
+ network, chromium and webkit, with and without parallel load. Record x/N and
133
+ the condition that triggers it. Hand it to a fixer with that condition.
134
+ 3. **Three attempts per hypothesis, not per issue.** A fixer tests each hypothesis
135
+ with evidence (logging, a breakpoint, a network trace, reading the dependency's
136
+ source in `node_modules`, read-only) before changing code. When the first fixer
137
+ reports NOT FIXED, dispatch a second `qa-fixer` with `effort: xhigh`, the first
138
+ one's analysis, and its hypotheses listed as ruled out. Only after two fixers
139
+ with different root-cause hypotheses is an issue NOT FIXED.
140
+ 4. **A workaround in the repo is a fix, not a design decision**, when it makes the
141
+ app do what the spec says (for example handling `data: null` at the call site
142
+ because the library hook does not). It is a product decision only when it
143
+ changes user-visible behaviour the spec does not define. Then: open question,
144
+ not fixed.
145
+ 5. **Rounds until stable.** After Phase 5, re-dispatch every case that is still
146
+ NOT RUN or intermittent. Repeat until a full round changes no status, maximum
147
+ 4 rounds. Write the report only then.
148
+ 6. **PASS needs evidence.** A PASS without a status code, DB row, network line or
149
+ screenshot path is NOT RUN. Spot-check 10 % of PASSes in every area yourself.
150
+
151
+ | # | Excuse | Reality |
152
+ | --- | ---------------------------------------------------- | --------------------------------------------------- |
153
+ | 1 | "The orchestrator wants results soon" | Nobody is waiting. Finish the ladder. |
154
+ | 2 | "Shared Chrome window, can't resize" | Use the headless lane with `--device`. |
155
+ | 3 | "Global state, would disturb other agents" | Own `qa-` entity, or the serial lane. |
156
+ | 4 | "Only one replica / no customer account / no device" | Build it: second port, seed via API, iOS simulator. |
157
+ | 5 | "Saw it once, second try passed" | Intermittent FAIL. 10 runs, vary timing. |
158
+ | 6 | "Rule says stop after three attempts" | Three per hypothesis, then a second fixer. |
159
+ | 7 | "Library code is out of scope" | Reading it is in scope. Fix at the call site. |
160
+ | 8 | "One rerun round, then stop" | Rounds until no status changes, max 4. |
161
+ | 9 | "Covered by existing e2e tests" | Run that e2e test and cite its result, or test it. |
162
+
163
+ **Red flags, stop and go back:** NOT RUN without a ladder; "flaky" as a final
164
+ status; a fixer reply with one hypothesis; a PASS without evidence; a report
165
+ written while the ledger has NOT RUN rows.
166
+
167
+ Stubbornness never overrides safety: never production, never real payments or
168
+ real emails, never weaken, skip or delete a test, never guess at an unclear spec
169
+ (open question, test both readings).
170
+
171
+ ## Hard rules
172
+
173
+ 1. Test data carries the prefix `qa-<area>-`. Clean it up or list what is left.
174
+ 2. Fix only clear deviations from spec or obvious bugs. Design questions are reported.
175
+ 3. Never edit, commit or switch branches in the user's main checkout except the
176
+ final local fast-forward in Phase 6.
177
+ 4. Subagent reports are not visible to the user. Summarise what matters.
178
+ 5. Final message to the user: verdict (ready / not ready), counts, path to the
179
+ report, and only the decisions that are theirs.
@@ -0,0 +1,159 @@
1
+ # Driver lanes
2
+
3
+ Pick the lane per test case and write it in the ledger. Verified on Tim's Mac
4
+ (64 GB, 12 cores, 2026-10-22): two headless sessions (chrome + webkit iPhone) and
5
+ two cloned iOS simulators ran in parallel against one localhost server.
6
+
7
+ ## 1. headless (default)
8
+
9
+ Tool: `playwright-cli` (`@playwright/cli`, installed globally; `npm install -g @playwright/cli@0.1.22`
10
+ if missing). Each session is an isolated in-memory browser. Cheaper in tokens than
11
+ the Playwright MCP: snapshots go to files, not into context.
12
+
13
+ Rules:
14
+
15
+ 1. `cd` into your own folder first (`<work>/shots/<area>/`): the CLI writes
16
+ `.playwright-cli/` snapshots, console logs and screenshots into the cwd.
17
+ 2. Session names `-s=qa-<area>-<n>`. Close your own sessions with
18
+ `playwright-cli -s=<name> close`. Never `close-all` or `kill-all`: they kill
19
+ other agents' browsers.
20
+ 3. Load auth before navigating: `playwright-cli -s=<name> state-load <work>/auth/<role>.json`.
21
+
22
+ Recipes:
23
+
24
+ ```bash
25
+ playwright-cli -s=qa-cart-1 open http://localhost:3012/ # desktop Chrome, headless
26
+ playwright-cli -s=qa-cart-2 open --device "iPhone 15" --browser webkit URL # mobile Safari engine
27
+ playwright-cli -s=qa-cart-3 open --device "Pixel 7" URL # mobile Chrome
28
+ playwright-cli -s=qa-cart-1 snapshot # a11y tree with refs (e12)
29
+ playwright-cli -s=qa-cart-1 find "Lägg i varukorg"
30
+ playwright-cli -s=qa-cart-1 click e12 ; playwright-cli -s=qa-cart-1 dblclick e12
31
+ playwright-cli -s=qa-cart-1 fill e5 "åäö 😀 <img src=x onerror=alert(1)>" --submit
32
+ playwright-cli -s=qa-cart-1 console error # console errors
33
+ playwright-cli -s=qa-cart-1 requests # network list; request <n> for details
34
+ playwright-cli -s=qa-cart-1 tab-new URL ; playwright-cli -s=qa-cart-1 tab-select 0 # two tabs, same session
35
+ playwright-cli -s=qa-cart-1 network-state-set offline
36
+ playwright-cli -s=qa-cart-1 route "**/shop-api" --status 500 --body '{"errors":[{"message":"boom"}]}'
37
+ playwright-cli -s=qa-cart-1 screenshot --filename=<work>/shots/<area>/CART-07-fail.png
38
+ playwright-cli -s=qa-cart-1 eval "getComputedStyle(document.body).overflowX"
39
+ ```
40
+
41
+ Slow network or a race (delay one request, click during it):
42
+
43
+ ```bash
44
+ playwright-cli -s=qa-cart-1 run-code "async page => { await page.route('**/shop-api', async r => { await new Promise(s => setTimeout(s, 4000)); await r.continue(); }); }"
45
+ ```
46
+
47
+ Keyboard and a11y: `press Tab` repeatedly, then `eval "document.activeElement.outerHTML"`.
48
+
49
+ ### Auth state (once per role, by the orchestrator)
50
+
51
+ Log in headlessly with TEST credentials only: from the project's seed,
52
+ `.env.example`, or test accounts created in this run, on a local or named test host.
53
+ Never the user's real password, never production.
54
+
55
+ ```bash
56
+ cd <work>/auth
57
+ playwright-cli -s=qa-auth open http://localhost:3000/dashboard/login
58
+ playwright-cli -s=qa-auth snapshot # find the field refs
59
+ playwright-cli -s=qa-auth fill <userRef> "<seed user>"
60
+ playwright-cli -s=qa-auth fill <passRef> "<seed password>" --submit
61
+ playwright-cli -s=qa-auth state-save admin.json
62
+ playwright-cli -s=qa-auth close
63
+ ```
64
+
65
+ storageState holds cookies and localStorage per origin. Vendure: `tokenMethod`
66
+ defaults to `cookie` (captured); with `bearer` the token sits in the client's
67
+ localStorage for that origin only. A storefront on another origin needs its own
68
+ state. sessionStorage is not saved. No test credentials anywhere: use the chrome
69
+ lane for logged-in cases and note it in CONTEXT.md.
70
+
71
+ ## 2. chrome (the user's logged-in Chrome, Claude in Chrome)
72
+
73
+ Use only for what needs the user's real session or profile. Known defects
74
+ (anthropics/claude-code #101431, #87767): subagents share one tab group, `navigate`
75
+ without `tabId` drives the first tab of the group, and concurrent calls can
76
+ receive each other's results.
77
+
78
+ 1. Load the tools in ONE ToolSearch call: `select:mcp__claude-in-chrome__tabs_context_mcp,mcp__claude-in-chrome__tabs_create_mcp,mcp__claude-in-chrome__tabs_close_mcp,mcp__claude-in-chrome__navigate,mcp__claude-in-chrome__computer,mcp__claude-in-chrome__read_page,mcp__claude-in-chrome__find,mcp__claude-in-chrome__form_input,mcp__claude-in-chrome__get_page_text,mcp__claude-in-chrome__javascript_tool,mcp__claude-in-chrome__read_console_messages,mcp__claude-in-chrome__read_network_requests`.
79
+ 2. `tabs_create_mcp` once, then pass that `tabId` on EVERY call (a hook in
80
+ `qa-tester` blocks calls without it, and blocks `resize_window`).
81
+ 3. After each action, confirm the URL or title in the result is your tab's. A
82
+ mismatch means crosstalk: repeat the action.
83
+ 4. Max 2 agents in this lane at once. No `alert()`/`confirm()` triggers: they
84
+ freeze the tab. Never sign out, never change the user's account settings.
85
+ Vendure's dashboard keeps the active channel in localStorage shared by all tabs:
86
+ do not switch channel in the shared Chrome; use a headless session for that.
87
+ 5. Close your tab with `tabs_close_mcp` when done, and also the empty "New tab"
88
+ that `tabs_context_mcp`/`navigate` may have created for you.
89
+ 6. `get_page_text` can return "No text content found" on small pages; use
90
+ `read_page` or `find` with your `tabId` instead.
91
+
92
+ ## 3. ios-sim (real Mobile Safari)
93
+
94
+ Helper: `bash .claude/skills/qa-genomklickning/scripts/ios-sim.sh`
95
+
96
+ ```bash
97
+ UDID=$(bash ios-sim.sh up qa-cart "iPhone 17") # clone + boot, prints UDID
98
+ bash ios-sim.sh open $UDID http://hacken.localhost:3012/
99
+ bash ios-sim.sh shot $UDID <work>/shots/cart/CART-12-ios.png
100
+ bash ios-sim.sh down $UDID # shutdown + delete the clone
101
+ ```
102
+
103
+ Interact with the iOS Simulator tool (`mcp__Claude_Code_iOS_Simulator__control`,
104
+ `device: <UDID>`): `tap`, `text`, `swipe`, `screenshot`. Coordinates are device
105
+ points from its screenshot. The simulator shares the Mac's network, so `localhost`
106
+ and `*.localhost` work. Max 3 clones at once. Always `down` your clone, also on failure.
107
+ Look at every simulator screenshot: an ✕ in the address bar means the page is
108
+ still loading, so wait and shoot again before judging it.
109
+
110
+ ## 4. android (real Chrome on Android emulator)
111
+
112
+ Installed on Tim's Mac (2026-10-22): AVDs `qa1` and `qa2` (Pixel 8, Android 35,
113
+ Play Store image), Chrome's first-run screen already dismissed on both with Tim's
114
+ OK. Elsewhere: `bash .claude/skills/qa-genomklickning/scripts/android-setup.sh`
115
+ (about 4 GB); the orchestrator does not run it on its own, it lists it in the
116
+ report as the way to unblock Android cases.
117
+
118
+ Helper: `bash .claude/skills/qa-genomklickning/scripts/android.sh` (puts the SDK
119
+ on PATH itself). One AVD per agent, `qa1` on port 5554, `qa2` on 5556.
120
+
121
+ ```bash
122
+ A="$PWD"/.claude/skills/qa-genomklickning/scripts/android.sh
123
+ S=$(bash $A up qa2 5556) # boots headless (~15 s), prints emulator-5556
124
+ bash $A reverse $S 3012 # Mac localhost:3012 inside the emulator
125
+ bash $A open $S http://hacken.localhost:3012/ # opens in Chrome
126
+ CDP=$(bash $A cdp $S 9302) # Chrome DevTools on the Mac
127
+ playwright-cli attach --cdp=$CDP --session=qa-<area>-and
128
+ playwright-cli -s=qa-<area>-and snapshot # then click/fill/console/requests as in lane 1
129
+ playwright-cli -s=qa-<area>-and detach # detach, not close: the browser is the emulator Chrome
130
+ bash $A shot $S <work>/shots/<area>/X-android.png
131
+ bash $A down $S
132
+ ```
133
+
134
+ `playwright-cli attach --cdp` drives the real Chrome on Android: DOM snapshot with
135
+ refs, click, fill, eval, console and network, all verified. Use it for page
136
+ interaction. `bash $A texts $S` and `bash $A tap-text $S "<text>"` (uiautomator)
137
+ work on Android system UI, Chrome's own dialogs and visible page text; use them
138
+ for quick checks, not for page interaction.
139
+
140
+ Before every `attach`, run `bash $A texts $S` and confirm Chrome is in front with
141
+ your URL. Chrome can drop to the home screen; a click then times out as "not
142
+ visible" and the DevTools socket dies (ECONNRESET). Recover with `open`, `cdp`
143
+ and a new `attach`. `attach` sees every Chrome tab on the device, so pick yours
144
+ by URL with `tab-list`/`tab-select`, not by index.
145
+
146
+ Dialogs that can appear: "Chrome notifications make things easier" (tap
147
+ "No thanks"), "Chrome isn't responding" when two emulators boot at once (tap
148
+ "Close app", then `open` again). A new or wiped AVD shows Chrome's Terms of
149
+ Service first-run screen: an agent dismisses it only with the user's recorded OK
150
+ in CONTEXT.md; otherwise the Android cases are BLOCKED with that reason.
151
+
152
+ Chrome resolves `*.localhost` to loopback itself, so `adb reverse` covers shop
153
+ subdomains; confirm with a screenshot on first use.
154
+
155
+ ## 5. api / db
156
+
157
+ Every UI result is checked in data: GraphQL/REST with curl (cookie jar in
158
+ `<work>/auth/`), DB read-only queries (`docker exec <pg> psql ...`). Writes only on
159
+ `qa-` data.
@@ -0,0 +1,23 @@
1
+ # Report structure: qa-report-<YYYY-MM-DD>.md
2
+
3
+ English. Numbered lists, tables get a # column. No em-dash.
4
+
5
+ 1. **Summary**: branch, scope (`<base>...<head>`, commit count), rounds run.
6
+ Counts: cases, PASS, FAIL, fixed, not fixed, intermittent, BLOCKED. Verdict:
7
+ ready / not ready to release, with the one-line reason.
8
+ 2. **Traceability matrix**: ticket → acceptance criterion → files → UI/endpoint →
9
+ cases → status.
10
+ 3. **Fixed defects**: title, ticket, root cause (file:line), test added (file and
11
+ test name), red output in one line, commit SHA, reviewer verdict.
12
+ 4. **Not fixed**, sorted Critical / High / Medium / Low: minimal repro steps,
13
+ expected vs actual (with spec source), evidence (screenshot path, console or
14
+ network lines, DB rows), both fixers' hypotheses and why each was ruled out,
15
+ why it was not fixed.
16
+ 5. **Intermittent**: x/N, trigger condition, status.
17
+ 6. **Spec vs code**: in code but not in spec, in spec but not in code.
18
+ 7. **BLOCKED and why**: each case with the unblock ladder rungs tried, and what
19
+ would unblock it (for example "install Android SDK: scripts/android-setup.sh").
20
+ 8. **Lanes and agents**: every subagent run with agent, model, effort, lane, count.
21
+ 9. **Choices I made**: internal choices taken without asking.
22
+ 10. **Open questions**: numbered, for unclear spec. Both readings tested where possible.
23
+ 11. **Left behind**: `qa-` test data still present, branches, worktrees, simulators.
@@ -0,0 +1,26 @@
1
+ #!/bin/bash
2
+ # One-time Android emulator setup on Apple Silicon without Android Studio.
3
+ # Downloads about 8 GB (JDK, SDK tools, Android 35 arm64 image). Creates AVDs qa1 and qa2.
4
+ set -euo pipefail
5
+ # Brew's automatic cleanup removes other old kegs and can fail on unrelated files.
6
+ export HOMEBREW_NO_INSTALL_CLEANUP=1
7
+
8
+ # The temurin cask is a .pkg that needs an admin password; skip it when a JDK exists.
9
+ /usr/libexec/java_home >/dev/null 2>&1 || brew install --cask temurin
10
+ brew list --cask android-commandlinetools >/dev/null 2>&1 || brew install --cask android-commandlinetools
11
+ export ANDROID_HOME="$(brew --prefix)/share/android-commandlinetools"
12
+ # `yes` dies of SIGPIPE when sdkmanager exits, which pipefail would treat as failure.
13
+ yes | sdkmanager --licenses >/dev/null || true
14
+ # sdkmanager is deprecated in favour of `android sdk` but still works and needs no login.
15
+ sdkmanager "platform-tools" "emulator" "platforms;android-35" \
16
+ "system-images;android-35;google_apis_playstore;arm64-v8a"
17
+ for n in qa1 qa2; do
18
+ echo no | avdmanager create avd -n "$n" \
19
+ -k "system-images;android-35;google_apis_playstore;arm64-v8a" --device pixel_8 --force
20
+ done
21
+
22
+ cat <<EOF
23
+ Done. Add to ~/.zshrc:
24
+ export ANDROID_HOME="$ANDROID_HOME"
25
+ export PATH="\$ANDROID_HOME/platform-tools:\$ANDROID_HOME/emulator:\$PATH"
26
+ EOF