@haus-tech/haus-workflow 7.1.0 → 7.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/dist/cli.js +395 -233
- package/package.json +1 -1
- package/packages/agents/qa-fixer.md +66 -0
- package/packages/agents/qa-inventory.md +36 -0
- package/packages/agents/qa-reviewer.md +32 -0
- package/packages/agents/qa-tester.md +85 -0
- package/packages/packages.yaml +8 -0
- package/packages/skills/fixing-weak-tests/SKILL.md +89 -0
- package/packages/skills/fixing-weak-tests/references/audit-brief.md +57 -0
- package/packages/skills/fixing-weak-tests/references/fix-brief.md +55 -0
- package/packages/skills/optimizing-test-suites/SKILL.md +157 -0
- package/packages/skills/optimizing-test-suites/references/affected-tests.md +46 -0
- package/packages/skills/optimizing-test-suites/references/levers.md +79 -0
- package/packages/skills/optimizing-test-suites/scripts/slowest.mjs +78 -0
- package/packages/skills/qa-genomklickning/SKILL.md +179 -0
- package/packages/skills/qa-genomklickning/drivers.md +159 -0
- package/packages/skills/qa-genomklickning/report.md +23 -0
- package/packages/skills/qa-genomklickning/scripts/android-setup.sh +26 -0
- package/packages/skills/qa-genomklickning/scripts/android.sh +62 -0
- package/packages/skills/qa-genomklickning/scripts/chrome-tab-guard.py +55 -0
- package/packages/skills/qa-genomklickning/scripts/env-check.sh +32 -0
- package/packages/skills/qa-genomklickning/scripts/ios-sim.sh +49 -0
- package/packages/skills/request/SKILL.md +3 -2
- package/packages/skills/request/scripts/compare-scores.mjs +23 -11
- package/packages/skills/testsammanfattning/SKILL.md +134 -0
- package/packages/skills/testsammanfattning/references/konfiguration.md +110 -0
- package/packages/skills/testsammanfattning/scripts/test-log.mjs +218 -0
- package/packages/skills/testsammanfattning/scripts/test-summary.mjs +495 -0
- package/packages/templates/agentic-workflow-standard.md +1 -1
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Running only affected tests: what the tools really do
|
|
2
|
+
|
|
3
|
+
Verified against the source of Vitest 4.1 and Playwright 1.62. Re-check the source when the
|
|
4
|
+
version differs: selection logic changes between releases.
|
|
5
|
+
|
|
6
|
+
## Vitest `--changed <ref>` and `vitest related <files>`
|
|
7
|
+
|
|
8
|
+
1. Changed files = `git diff --name-only <ref>...HEAD` plus staged, unstaged, untracked.
|
|
9
|
+
2. Selects every test file whose **static import graph** (from Vite's transform) contains a
|
|
10
|
+
changed file (`filterTestsBySource`, `getTestDependencies` in `vitest/dist/chunks/cli-api.*.js`).
|
|
11
|
+
3. Right for unit tests. Wrong for e2e files that all import one app config which imports
|
|
12
|
+
every plugin: any plugin change selects every file. Measure it: build the graph with
|
|
13
|
+
`createVitest` and `getTestDependencies` per file, then count which source files are
|
|
14
|
+
shared by all files. Only changes outside that shared set give a real saving.
|
|
15
|
+
4. Hooks: `experimental.vcsProvider` (a changed test file always selects itself, so a
|
|
16
|
+
provider returning chosen test files becomes your own selector) and
|
|
17
|
+
`coverage.provider: 'custom'` (per-file coverage).
|
|
18
|
+
|
|
19
|
+
## Playwright `--only-changed=<ref>`
|
|
20
|
+
|
|
21
|
+
1. Follows only files the runner loads with a spec (`collectAffectedTestFiles`). Specs that
|
|
22
|
+
reach the app over HTTP never import app source, so an app change selects **nothing**.
|
|
23
|
+
2. The dependency hook (`populateDependencies`) is private, used by component testing only.
|
|
24
|
+
3. Usable for helper or fixture changes. For app changes use `--test-list <file>` with your
|
|
25
|
+
own selection.
|
|
26
|
+
|
|
27
|
+
## Options, cheapest first
|
|
28
|
+
|
|
29
|
+
1. **Name the files** you touched while iterating; `--last-failed` after a red run;
|
|
30
|
+
`--project <name>` when only one side changed. Full suites once before the PR. Zero code.
|
|
31
|
+
2. **Static rule:** run everything when a shared file changes, otherwise `vitest related`.
|
|
32
|
+
2 to 3 hours. Helps only when changes are outside the shared set.
|
|
33
|
+
3. **Coverage impact map:** a full green run records which lines each test file (or test)
|
|
34
|
+
executes, using `node:inspector` precise coverage plus source maps
|
|
35
|
+
(`@jridgewell/trace-mapping`). The map stores its commit. The selector diffs
|
|
36
|
+
`git diff -U0 <map commit>`, runs tests that executed a changed line, and runs
|
|
37
|
+
**everything** for lockfiles, `package.json`, test config, test helpers and any file the
|
|
38
|
+
map does not know (templates, CSV, CSS). It also prints changed lines no test executes.
|
|
39
|
+
About 1 day per suite plus a half-day spike; every server process in the recording run
|
|
40
|
+
must load the coverage hook; the recording run is slower (no template cache, dev build
|
|
41
|
+
for source maps).
|
|
42
|
+
4. **Hand-kept spec-to-source map or tags:** not recommended as the gate. Tests reach code
|
|
43
|
+
through HTTP and plugins affect each other at runtime, which is what e2e exists to catch.
|
|
44
|
+
|
|
45
|
+
Recommend by payoff: once the full suites are a few minutes, option 1 is usually enough and
|
|
46
|
+
option 3 is a decision for the user with its cost stated.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Levers, with the gotchas that bite
|
|
2
|
+
|
|
3
|
+
Each lever: when it applies, how, what to check. Examples are Vitest 4 and Playwright 1.5x
|
|
4
|
+
with Postgres; the pattern carries to other stacks.
|
|
5
|
+
|
|
6
|
+
## 1. Parallel workers for server-booting e2e files (Vitest)
|
|
7
|
+
|
|
8
|
+
When: each file boots the app on a fixed port against one DB, `fileParallelism: false`.
|
|
9
|
+
|
|
10
|
+
1. Per-worker keys from `process.env.VITEST_POOL_ID` (1..maxWorkers in pool `forks`):
|
|
11
|
+
DB name `${base}_w${id}`, port from a list picked in `globalSetup`.
|
|
12
|
+
2. `globalSetup` runs once in the main process; `process.env` values it sets reach the
|
|
13
|
+
fork workers. Pick free ports there (probe with `net.createServer().listen`) and pass
|
|
14
|
+
them as one env var, so two worktrees on one machine do not collide.
|
|
15
|
+
3. Remove `fileParallelism: false`, set `maxWorkers` from an env var (default = measured
|
|
16
|
+
knee, often 4).
|
|
17
|
+
4. Check shared external resources: object storage buckets (tests that list or empty a
|
|
18
|
+
bucket), mail outboxes, job queues (per DB is fine), caches keyed by name.
|
|
19
|
+
5. Validate env-derived DB names with a regex before using them in SQL.
|
|
20
|
+
|
|
21
|
+
## 2. Build once, clone per file (seeded template DB)
|
|
22
|
+
|
|
23
|
+
When: several files run the same expensive seed in `beforeAll`.
|
|
24
|
+
|
|
25
|
+
1. Name each seed (`hosted_catalogue`, `test_clubs`); the template DB is `${base}_t_${name}`.
|
|
26
|
+
2. First file to need it, under `pg_advisory_lock(hashtext(templateName))`: init a normal
|
|
27
|
+
DB, run the seed through the app, wait for job queues to go idle, stop the server,
|
|
28
|
+
`CREATE DATABASE tpl TEMPLATE workerDb`. Later files: `CREATE DATABASE workerDb TEMPLATE tpl`
|
|
29
|
+
(about 1 s) and skip populate.
|
|
30
|
+
3. `CREATE DATABASE ... TEMPLATE` fails if any connection is open on the source: close the
|
|
31
|
+
app first.
|
|
32
|
+
4. Drop old templates in `globalSetup` so a seed change is never served from a stale template.
|
|
33
|
+
5. Tests that test the seed itself keep running the seed.
|
|
34
|
+
6. Other databases: MySQL has no `TEMPLATE`; dump the seeded DB once (`mysqldump`) and
|
|
35
|
+
restore per worker, or `CREATE TABLE w.t LIKE tpl.t` + `INSERT ... SELECT` per table.
|
|
36
|
+
SQLite: copy the file. Any DB: a snapshot of a container volume also works.
|
|
37
|
+
7. Truncate in `beforeEach` on one shared DB: parallel only after each worker has its own DB
|
|
38
|
+
(lever 1). Wrapping each test in a transaction that rolls back is often faster than
|
|
39
|
+
truncating, when the app under test uses the same connection.
|
|
40
|
+
|
|
41
|
+
## 3. Parallel browser tests with a stub API (Playwright)
|
|
42
|
+
|
|
43
|
+
When: `workers: 1` because one stub server keeps in-memory state.
|
|
44
|
+
|
|
45
|
+
1. One stub server and one app server per worker: ports `base + n`, the worker reads
|
|
46
|
+
`process.env.TEST_PARALLEL_INDEX` (0..workers-1, stable per worker slot).
|
|
47
|
+
2. `webServer` entries: N stubs, build once in the first app entry, `preview` for the others.
|
|
48
|
+
3. Projects that share a real DB keep `workers: 1` on that project (per-project `workers`)
|
|
49
|
+
and are listed **first**, so their worker starts at once instead of after the parallel
|
|
50
|
+
project drains.
|
|
51
|
+
4. `fullyParallel: true` on the stub project only.
|
|
52
|
+
5. Browser tests against one real backend and DB: either every test creates its own data
|
|
53
|
+
(unique email, cart, shop per test, nothing global mutated) and runs in parallel, or the
|
|
54
|
+
backend runs once per worker with its own DB (lever 1 + 2). Tests that change global state
|
|
55
|
+
(settings, statuses) stay in a `workers: 1` project.
|
|
56
|
+
6. Never `reuseExistingServer` for stub servers: a dev server on that port would make
|
|
57
|
+
isolation tests prove nothing.
|
|
58
|
+
|
|
59
|
+
## 4. Short waits in the test build only
|
|
60
|
+
|
|
61
|
+
When: tests wait for retry backoff (TanStack Query default 1, 2, 4 s), polling or sleeps.
|
|
62
|
+
|
|
63
|
+
1. A build-time env var (`VITE_TEST_RETRY_DELAY_MS`), inlined by the bundler, read in one
|
|
64
|
+
small function that only overrides `retryDelay` when set. Keep the retry count, so the
|
|
65
|
+
retry path is still tested.
|
|
66
|
+
2. Unit test both: unset leaves defaults, set overrides.
|
|
67
|
+
3. Check every other build (Lighthouse, production, preview deploys) rebuilds without it.
|
|
68
|
+
4. Fixed sleeps in tests: replace with waiting for a condition (`expect.poll`,
|
|
69
|
+
`waitForResponse`, queue idle).
|
|
70
|
+
|
|
71
|
+
## 5. Wrong layer
|
|
72
|
+
|
|
73
|
+
A browser or e2e test whose assertion is a formatter, validator or calculation: write the
|
|
74
|
+
unit test, mutate the code to prove the unit test catches it, then remove the slow test.
|
|
75
|
+
|
|
76
|
+
## Measuring workers
|
|
77
|
+
|
|
78
|
+
Run 1, 4, 6 workers on an idle machine. Pick the knee (in one repo, 6 workers saved 20 s
|
|
79
|
+
over 4 on a 4 min run, so 4 stayed). Postgres `max_connections` and CPU are usual ceilings.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Usage: node slowest.mjs <report.json> [top=10]
|
|
3
|
+
// Reads a Vitest (--reporter=json) or Playwright (--reporter=json) report and prints
|
|
4
|
+
// wall time, the sum of test time, and the slowest files and tests.
|
|
5
|
+
import { readFileSync } from 'node:fs'
|
|
6
|
+
|
|
7
|
+
const [file, topArg] = process.argv.slice(2)
|
|
8
|
+
if (!file) {
|
|
9
|
+
console.error('usage: node slowest.mjs <report.json> [top=10]')
|
|
10
|
+
process.exit(1)
|
|
11
|
+
}
|
|
12
|
+
const top = Number(topArg ?? 10)
|
|
13
|
+
const report = JSON.parse(readFileSync(file, 'utf8'))
|
|
14
|
+
const tests = [] // { file, title, ms, status }
|
|
15
|
+
let wallMs
|
|
16
|
+
|
|
17
|
+
if (Array.isArray(report.testResults)) {
|
|
18
|
+
// Vitest (Jest-compatible shape)
|
|
19
|
+
const starts = report.testResults.map((f) => f.startTime).filter(Boolean)
|
|
20
|
+
const ends = report.testResults.map((f) => f.endTime).filter(Boolean)
|
|
21
|
+
if (starts.length && ends.length) wallMs = Math.max(...ends) - Math.min(...starts)
|
|
22
|
+
for (const f of report.testResults) {
|
|
23
|
+
for (const t of f.assertionResults ?? []) {
|
|
24
|
+
tests.push({
|
|
25
|
+
file: f.name,
|
|
26
|
+
title: t.fullName ?? t.title,
|
|
27
|
+
ms: t.duration ?? 0,
|
|
28
|
+
status: t.status,
|
|
29
|
+
})
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
} else if (Array.isArray(report.suites)) {
|
|
33
|
+
// Playwright
|
|
34
|
+
wallMs = report.stats?.duration
|
|
35
|
+
const walk = (suite, specFile) => {
|
|
36
|
+
const fileName = suite.file ?? specFile
|
|
37
|
+
for (const spec of suite.specs ?? []) {
|
|
38
|
+
for (const t of spec.tests ?? []) {
|
|
39
|
+
const results = t.results ?? []
|
|
40
|
+
const ms = results.reduce((sum, r) => sum + (r.duration ?? 0), 0)
|
|
41
|
+
const status = results.at(-1)?.status ?? t.status
|
|
42
|
+
tests.push({
|
|
43
|
+
file: `${t.projectName ?? ''}:${spec.file ?? fileName}`,
|
|
44
|
+
title: spec.title,
|
|
45
|
+
ms,
|
|
46
|
+
status,
|
|
47
|
+
})
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
for (const child of suite.suites ?? []) walk(child, fileName)
|
|
51
|
+
}
|
|
52
|
+
for (const s of report.suites) walk(s, s.file)
|
|
53
|
+
} else {
|
|
54
|
+
console.error('Unknown report shape: expected Vitest testResults[] or Playwright suites[]')
|
|
55
|
+
process.exit(1)
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
const s = (ms) => `${(ms / 1000).toFixed(1)} s`
|
|
59
|
+
const sum = tests.reduce((acc, t) => acc + t.ms, 0)
|
|
60
|
+
const byFile = new Map()
|
|
61
|
+
for (const t of tests) byFile.set(t.file, (byFile.get(t.file) ?? 0) + t.ms)
|
|
62
|
+
const failed = tests.filter((t) => !['passed', 'expected', 'skipped', 'pending'].includes(t.status))
|
|
63
|
+
|
|
64
|
+
console.log(`tests: ${tests.length} failed: ${failed.length}`)
|
|
65
|
+
console.log(
|
|
66
|
+
`wall: ${wallMs ? s(wallMs) : 'n/a'} sum of test time: ${s(sum)}${wallMs ? ` ratio: ${(sum / wallMs).toFixed(2)}` : ''}`,
|
|
67
|
+
)
|
|
68
|
+
console.log('(ratio near 1 = serial; above 1 = parallel; setup in hooks is not in the sum)')
|
|
69
|
+
console.log(`\nslowest ${top} files:`)
|
|
70
|
+
for (const [f, ms] of [...byFile].sort((a, b) => b[1] - a[1]).slice(0, top))
|
|
71
|
+
console.log(` ${s(ms).padStart(8)} ${f}`)
|
|
72
|
+
console.log(`\nslowest ${top} tests:`)
|
|
73
|
+
for (const t of [...tests].sort((a, b) => b.ms - a.ms).slice(0, top))
|
|
74
|
+
console.log(` ${s(t.ms).padStart(8)} ${t.title} (${t.file})`)
|
|
75
|
+
if (failed.length) {
|
|
76
|
+
console.log('\nnot passed:')
|
|
77
|
+
for (const t of failed) console.log(` ${t.status} ${t.title} (${t.file})`)
|
|
78
|
+
}
|
|
@@ -0,0 +1,179 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa-genomklickning
|
|
3
|
+
description: Use when everything built on a branch (a release branch, a feature branch, a PR) must be clicked through and probed in as many combinations as possible before release, with the bugs found fixed test-first and re-verified. Triggers on "/qa-genomklickning", "klicka igenom allt", "QA:a releasebranchen", "testa allt som gjorts på branchen", "genomklickning", "full QA pass", "click through everything on this branch".
|
|
4
|
+
argument-hint: '[branch] [scope-base]'
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# QA genomklickning
|
|
8
|
+
|
|
9
|
+
You are the QA orchestrator. You click through and probe EVERYTHING built in the
|
|
10
|
+
scope, in as many relevant combinations as possible, fix what is broken with TDD,
|
|
11
|
+
and verify the fixes. Subagents do the work in parallel; you plan, coordinate,
|
|
12
|
+
merge and report. You run in the main conversation on purpose: only the main
|
|
13
|
+
conversation can ask the user a question, and it keeps the fan-out one level deep.
|
|
14
|
+
|
|
15
|
+
**Violating the letter of these rules is violating the spirit of these rules.**
|
|
16
|
+
|
|
17
|
+
## Step 0: Which branch
|
|
18
|
+
|
|
19
|
+
Arguments: `$ARGUMENTS` (first word = branch under test, second = optional scope base).
|
|
20
|
+
|
|
21
|
+
1. No branch given: run `git branch -a --sort=-committerdate | head -15` and
|
|
22
|
+
`gh pr list --limit 10`, then ask with AskUserQuestion: "Which branch should the
|
|
23
|
+
QA pass run against?" Options: the current branch, the 2-3 most recent
|
|
24
|
+
`release/*` branches, the head branch of the newest open PR. Wait for the answer.
|
|
25
|
+
This is the only question you ask before the report.
|
|
26
|
+
2. Make sure the repo's main checkout has that branch checked out and clean, and
|
|
27
|
+
that the local app runs that code. Never switch branches in a checkout with
|
|
28
|
+
uncommitted work: stop and say so.
|
|
29
|
+
3. Scope base: the second argument, else the base of the branch's PR
|
|
30
|
+
(`gh pr view <branch> --json baseRefName`), else `git merge-base origin/<default> <branch>`.
|
|
31
|
+
|
|
32
|
+
Everything else (specs, environment, logins, limits) you find yourself: PR body,
|
|
33
|
+
commit messages, ticket references (Asana MCP), `docs/`, ADRs, README,
|
|
34
|
+
`package.json`, `.claude/launch.json`, running ports (`lsof -iTCP -sTCP:LISTEN`),
|
|
35
|
+
seed files and `.env.example`. Record what you found and where in `CONTEXT.md`,
|
|
36
|
+
together with: repo path, branch and HEAD, scope base, running URLs and ports,
|
|
37
|
+
test credentials source, DB access command, the shell quirks of this machine, the
|
|
38
|
+
commit co-author line, and the rule that agents never edit, commit or switch
|
|
39
|
+
branches in the main checkout.
|
|
40
|
+
|
|
41
|
+
## Work folder
|
|
42
|
+
|
|
43
|
+
`<repo>/.qa/<YYYY-MM-DD>-<branch-slug>/`. Add `.qa/` to `.git/info/exclude` so it
|
|
44
|
+
is never committed. Holds `CONTEXT.md`, `inventory-<area>.md`, `testplan-<area>.md`,
|
|
45
|
+
`results-<area>.md`, `ledger.md`, `auth/`, `shots/`, and the final report.
|
|
46
|
+
`ledger.md` is the single source of truth: one row per test case with id, area,
|
|
47
|
+
priority, driver, status, evidence path, attempts. You work from the ledger, not
|
|
48
|
+
from memory; your context will not hold 200 cases.
|
|
49
|
+
|
|
50
|
+
## Agents and models
|
|
51
|
+
|
|
52
|
+
| # | Role | Agent | Model / effort |
|
|
53
|
+
| --- | ------------------------------------------- | -------------- | ------------------------------------------------------------------- |
|
|
54
|
+
| 1 | Inventory: git, tickets, routes, test infra | `qa-inventory` | haiku / low |
|
|
55
|
+
| 2 | Test plan + execution per area | `qa-tester` | sonnet / medium |
|
|
56
|
+
| 3 | Root cause + TDD fix | `qa-fixer` | opus / high (sonnet for a local one-file bug: pass `model: sonnet`) |
|
|
57
|
+
| 4 | Second fixer after a failed first one | `qa-fixer` | opus, `effort: xhigh` |
|
|
58
|
+
| 5 | Independent review + final verification | `qa-reviewer` | opus / high |
|
|
59
|
+
|
|
60
|
+
The agent files live in `.claude/agents/`. Each brief you write names the work
|
|
61
|
+
folder, the area, the driver lanes allowed, and the test-data prefix. Briefs never
|
|
62
|
+
mention time, deadlines or that anyone is waiting.
|
|
63
|
+
|
|
64
|
+
## Drivers (how agents reach the app)
|
|
65
|
+
|
|
66
|
+
Read `drivers.md` in this folder before Phase 3. Summary:
|
|
67
|
+
|
|
68
|
+
| # | Lane | Use for | Parallel limit |
|
|
69
|
+
| --- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
|
|
70
|
+
| 1 | `headless` (playwright-cli, isolated session per agent) | default for everything: desktop, mobile viewports, WebKit, two tabs, double submit, offline, slow or failing API | 6 agents, several sessions each |
|
|
71
|
+
| 2 | `chrome` (Claude in Chrome, the user's logged-in Chrome) | only what needs the user's real session or real Chrome profile | 2 agents, own tab, `tabId` on every call, never resize |
|
|
72
|
+
| 3 | `ios-sim` (cloned iOS Simulator, real Mobile Safari) | mobile Safari smoke and P1 flows on iPhone | 3 simulators |
|
|
73
|
+
| 4 | `android` (emulator, real Chrome on Android driven over DevTools) | Android Chrome smoke and P1 flows; without the SDK: BLOCKED "Android SDK not installed" | 2 emulators (`qa1`, `qa2`) |
|
|
74
|
+
| 5 | `api` / `db` (curl, GraphQL, psql) | verifying data behind every UI result, preconditions, fast probes | unlimited |
|
|
75
|
+
|
|
76
|
+
Before Phase 3: run `bash .claude/skills/qa-genomklickning/scripts/env-check.sh`
|
|
77
|
+
and create a headless auth state once (see `drivers.md`). Test values only: seed or
|
|
78
|
+
example-config credentials on a local host. Never the user's real password.
|
|
79
|
+
|
|
80
|
+
## Phases
|
|
81
|
+
|
|
82
|
+
1. **Inventory** (parallel `qa-inventory`, one per area). What was done (`git log`,
|
|
83
|
+
`git diff <base>...<branch>`), tickets and acceptance criteria (also implicit
|
|
84
|
+
ones), affected routes/components/forms/endpoints/jobs/emails/permissions/flags,
|
|
85
|
+
existing test infra. You merge into a traceability matrix in `ledger.md`:
|
|
86
|
+
ticket → criterion → files → UI/endpoint → how it is reached. Also list code
|
|
87
|
+
without spec and spec without code. Split into independent areas. Do not wait
|
|
88
|
+
for the user to approve the matrix; continue.
|
|
89
|
+
2. **Test plan** (inside each `qa-tester`, first half). Dimensions: data/state,
|
|
90
|
+
input, roles, flows, environment (desktop + 375 px mobile + iOS Safari), errors,
|
|
91
|
+
cross-ticket interaction. Pairwise for breadth, full coverage for payment,
|
|
92
|
+
permissions, data loss and tenant isolation. P1/P2/P3.
|
|
93
|
+
3. **Execution** (parallel `qa-tester`, second half). Each case: steps, expected
|
|
94
|
+
(with spec source), actual, status, evidence. Data verified in API/DB, not only
|
|
95
|
+
in UI. Every FAIL reproduced and minimised.
|
|
96
|
+
4. **Fix with TDD** (`qa-fixer`, `isolation: worktree`). Group failures by root
|
|
97
|
+
cause, one issue per root cause. Fixes that touch the same files run one after
|
|
98
|
+
another. Each fixer branches `qa/<slug>` from the branch under test.
|
|
99
|
+
5. **Review and re-verify**. `qa-reviewer` (never one that wrote a fix) reviews
|
|
100
|
+
each fix. You integrate: create `qa/fixes-<date>` from the branch under test,
|
|
101
|
+
rebase each approved fix branch onto it and fast-forward, run the full suite
|
|
102
|
+
(lint, typecheck, unit, relevant e2e) after EACH one. Then re-run all P1 cases,
|
|
103
|
+
all earlier FAILs, and every PASS whose code a fix touched, on the integrated
|
|
104
|
+
code. New failures go back to Phase 4.
|
|
105
|
+
6. **Land**: fast-forward the branch under test to `qa/fixes-<date>` locally.
|
|
106
|
+
Before pushing it, check what it deploys (`.haus-workflow/deploy.json`,
|
|
107
|
+
push-triggered workflows, Vercel link). Production or customer-visible: do not
|
|
108
|
+
push, say so in the report and the final message.
|
|
109
|
+
7. **Report**: `qa-report-<YYYY-MM-DD>.md` in the work folder, structure in
|
|
110
|
+
`report.md`. In English.
|
|
111
|
+
|
|
112
|
+
## Stubbornness (envishet)
|
|
113
|
+
|
|
114
|
+
The pass is finished when every case is PASS, FAIL with an issue, or BLOCKED with
|
|
115
|
+
proof. Not before. Nobody is waiting; speed is never a reason to stop.
|
|
116
|
+
|
|
117
|
+
1. **NOT RUN is not a result.** A case may only end BLOCKED after the unblock
|
|
118
|
+
ladder, with each rung's attempt and outcome written in the ledger:
|
|
119
|
+
1. Switch lane (chrome → headless → ios-sim, or UI → API).
|
|
120
|
+
2. Create your own data instead of touching shared state (`qa-<area>-` shop,
|
|
121
|
+
product, customer, order). Global toggles that cannot be owned go in the
|
|
122
|
+
serial lane: you run those cases alone at the end of the round.
|
|
123
|
+
3. Build the missing precondition: seed the customer account through the API,
|
|
124
|
+
start a second server instance on another port against the same database,
|
|
125
|
+
throttle or fail the network with `route`, set the clock.
|
|
126
|
+
4. Ask a fresh tester (different model or lane) to try it.
|
|
127
|
+
Only prod data, real money, real email to real people, or a third-party
|
|
128
|
+
system outside your control make a case irreducibly BLOCKED.
|
|
129
|
+
2. **Seen once is a FAIL.** A failure seen once and not on the next try is
|
|
130
|
+
intermittent, not "could not reproduce". Run it at least 10 times, varying:
|
|
131
|
+
click before and after hydration, cold and warm cache, slow and offline
|
|
132
|
+
network, chromium and webkit, with and without parallel load. Record x/N and
|
|
133
|
+
the condition that triggers it. Hand it to a fixer with that condition.
|
|
134
|
+
3. **Three attempts per hypothesis, not per issue.** A fixer tests each hypothesis
|
|
135
|
+
with evidence (logging, a breakpoint, a network trace, reading the dependency's
|
|
136
|
+
source in `node_modules`, read-only) before changing code. When the first fixer
|
|
137
|
+
reports NOT FIXED, dispatch a second `qa-fixer` with `effort: xhigh`, the first
|
|
138
|
+
one's analysis, and its hypotheses listed as ruled out. Only after two fixers
|
|
139
|
+
with different root-cause hypotheses is an issue NOT FIXED.
|
|
140
|
+
4. **A workaround in the repo is a fix, not a design decision**, when it makes the
|
|
141
|
+
app do what the spec says (for example handling `data: null` at the call site
|
|
142
|
+
because the library hook does not). It is a product decision only when it
|
|
143
|
+
changes user-visible behaviour the spec does not define. Then: open question,
|
|
144
|
+
not fixed.
|
|
145
|
+
5. **Rounds until stable.** After Phase 5, re-dispatch every case that is still
|
|
146
|
+
NOT RUN or intermittent. Repeat until a full round changes no status, maximum
|
|
147
|
+
4 rounds. Write the report only then.
|
|
148
|
+
6. **PASS needs evidence.** A PASS without a status code, DB row, network line or
|
|
149
|
+
screenshot path is NOT RUN. Spot-check 10 % of PASSes in every area yourself.
|
|
150
|
+
|
|
151
|
+
| # | Excuse | Reality |
|
|
152
|
+
| --- | ---------------------------------------------------- | --------------------------------------------------- |
|
|
153
|
+
| 1 | "The orchestrator wants results soon" | Nobody is waiting. Finish the ladder. |
|
|
154
|
+
| 2 | "Shared Chrome window, can't resize" | Use the headless lane with `--device`. |
|
|
155
|
+
| 3 | "Global state, would disturb other agents" | Own `qa-` entity, or the serial lane. |
|
|
156
|
+
| 4 | "Only one replica / no customer account / no device" | Build it: second port, seed via API, iOS simulator. |
|
|
157
|
+
| 5 | "Saw it once, second try passed" | Intermittent FAIL. 10 runs, vary timing. |
|
|
158
|
+
| 6 | "Rule says stop after three attempts" | Three per hypothesis, then a second fixer. |
|
|
159
|
+
| 7 | "Library code is out of scope" | Reading it is in scope. Fix at the call site. |
|
|
160
|
+
| 8 | "One rerun round, then stop" | Rounds until no status changes, max 4. |
|
|
161
|
+
| 9 | "Covered by existing e2e tests" | Run that e2e test and cite its result, or test it. |
|
|
162
|
+
|
|
163
|
+
**Red flags, stop and go back:** NOT RUN without a ladder; "flaky" as a final
|
|
164
|
+
status; a fixer reply with one hypothesis; a PASS without evidence; a report
|
|
165
|
+
written while the ledger has NOT RUN rows.
|
|
166
|
+
|
|
167
|
+
Stubbornness never overrides safety: never production, never real payments or
|
|
168
|
+
real emails, never weaken, skip or delete a test, never guess at an unclear spec
|
|
169
|
+
(open question, test both readings).
|
|
170
|
+
|
|
171
|
+
## Hard rules
|
|
172
|
+
|
|
173
|
+
1. Test data carries the prefix `qa-<area>-`. Clean it up or list what is left.
|
|
174
|
+
2. Fix only clear deviations from spec or obvious bugs. Design questions are reported.
|
|
175
|
+
3. Never edit, commit or switch branches in the user's main checkout except the
|
|
176
|
+
final local fast-forward in Phase 6.
|
|
177
|
+
4. Subagent reports are not visible to the user. Summarise what matters.
|
|
178
|
+
5. Final message to the user: verdict (ready / not ready), counts, path to the
|
|
179
|
+
report, and only the decisions that are theirs.
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
# Driver lanes
|
|
2
|
+
|
|
3
|
+
Pick the lane per test case and write it in the ledger. Verified on Tim's Mac
|
|
4
|
+
(64 GB, 12 cores, 2026-10-22): two headless sessions (chrome + webkit iPhone) and
|
|
5
|
+
two cloned iOS simulators ran in parallel against one localhost server.
|
|
6
|
+
|
|
7
|
+
## 1. headless (default)
|
|
8
|
+
|
|
9
|
+
Tool: `playwright-cli` (`@playwright/cli`, installed globally; `npm install -g @playwright/cli@0.1.22`
|
|
10
|
+
if missing). Each session is an isolated in-memory browser. Cheaper in tokens than
|
|
11
|
+
the Playwright MCP: snapshots go to files, not into context.
|
|
12
|
+
|
|
13
|
+
Rules:
|
|
14
|
+
|
|
15
|
+
1. `cd` into your own folder first (`<work>/shots/<area>/`): the CLI writes
|
|
16
|
+
`.playwright-cli/` snapshots, console logs and screenshots into the cwd.
|
|
17
|
+
2. Session names `-s=qa-<area>-<n>`. Close your own sessions with
|
|
18
|
+
`playwright-cli -s=<name> close`. Never `close-all` or `kill-all`: they kill
|
|
19
|
+
other agents' browsers.
|
|
20
|
+
3. Load auth before navigating: `playwright-cli -s=<name> state-load <work>/auth/<role>.json`.
|
|
21
|
+
|
|
22
|
+
Recipes:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
playwright-cli -s=qa-cart-1 open http://localhost:3012/ # desktop Chrome, headless
|
|
26
|
+
playwright-cli -s=qa-cart-2 open --device "iPhone 15" --browser webkit URL # mobile Safari engine
|
|
27
|
+
playwright-cli -s=qa-cart-3 open --device "Pixel 7" URL # mobile Chrome
|
|
28
|
+
playwright-cli -s=qa-cart-1 snapshot # a11y tree with refs (e12)
|
|
29
|
+
playwright-cli -s=qa-cart-1 find "Lägg i varukorg"
|
|
30
|
+
playwright-cli -s=qa-cart-1 click e12 ; playwright-cli -s=qa-cart-1 dblclick e12
|
|
31
|
+
playwright-cli -s=qa-cart-1 fill e5 "åäö 😀 <img src=x onerror=alert(1)>" --submit
|
|
32
|
+
playwright-cli -s=qa-cart-1 console error # console errors
|
|
33
|
+
playwright-cli -s=qa-cart-1 requests # network list; request <n> for details
|
|
34
|
+
playwright-cli -s=qa-cart-1 tab-new URL ; playwright-cli -s=qa-cart-1 tab-select 0 # two tabs, same session
|
|
35
|
+
playwright-cli -s=qa-cart-1 network-state-set offline
|
|
36
|
+
playwright-cli -s=qa-cart-1 route "**/shop-api" --status 500 --body '{"errors":[{"message":"boom"}]}'
|
|
37
|
+
playwright-cli -s=qa-cart-1 screenshot --filename=<work>/shots/<area>/CART-07-fail.png
|
|
38
|
+
playwright-cli -s=qa-cart-1 eval "getComputedStyle(document.body).overflowX"
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Slow network or a race (delay one request, click during it):
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
playwright-cli -s=qa-cart-1 run-code "async page => { await page.route('**/shop-api', async r => { await new Promise(s => setTimeout(s, 4000)); await r.continue(); }); }"
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Keyboard and a11y: `press Tab` repeatedly, then `eval "document.activeElement.outerHTML"`.
|
|
48
|
+
|
|
49
|
+
### Auth state (once per role, by the orchestrator)
|
|
50
|
+
|
|
51
|
+
Log in headlessly with TEST credentials only: from the project's seed,
|
|
52
|
+
`.env.example`, or test accounts created in this run, on a local or named test host.
|
|
53
|
+
Never the user's real password, never production.
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
cd <work>/auth
|
|
57
|
+
playwright-cli -s=qa-auth open http://localhost:3000/dashboard/login
|
|
58
|
+
playwright-cli -s=qa-auth snapshot # find the field refs
|
|
59
|
+
playwright-cli -s=qa-auth fill <userRef> "<seed user>"
|
|
60
|
+
playwright-cli -s=qa-auth fill <passRef> "<seed password>" --submit
|
|
61
|
+
playwright-cli -s=qa-auth state-save admin.json
|
|
62
|
+
playwright-cli -s=qa-auth close
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
storageState holds cookies and localStorage per origin. Vendure: `tokenMethod`
|
|
66
|
+
defaults to `cookie` (captured); with `bearer` the token sits in the client's
|
|
67
|
+
localStorage for that origin only. A storefront on another origin needs its own
|
|
68
|
+
state. sessionStorage is not saved. No test credentials anywhere: use the chrome
|
|
69
|
+
lane for logged-in cases and note it in CONTEXT.md.
|
|
70
|
+
|
|
71
|
+
## 2. chrome (the user's logged-in Chrome, Claude in Chrome)
|
|
72
|
+
|
|
73
|
+
Use only for what needs the user's real session or profile. Known defects
|
|
74
|
+
(anthropics/claude-code #101431, #87767): subagents share one tab group, `navigate`
|
|
75
|
+
without `tabId` drives the first tab of the group, and concurrent calls can
|
|
76
|
+
receive each other's results.
|
|
77
|
+
|
|
78
|
+
1. Load the tools in ONE ToolSearch call: `select:mcp__claude-in-chrome__tabs_context_mcp,mcp__claude-in-chrome__tabs_create_mcp,mcp__claude-in-chrome__tabs_close_mcp,mcp__claude-in-chrome__navigate,mcp__claude-in-chrome__computer,mcp__claude-in-chrome__read_page,mcp__claude-in-chrome__find,mcp__claude-in-chrome__form_input,mcp__claude-in-chrome__get_page_text,mcp__claude-in-chrome__javascript_tool,mcp__claude-in-chrome__read_console_messages,mcp__claude-in-chrome__read_network_requests`.
|
|
79
|
+
2. `tabs_create_mcp` once, then pass that `tabId` on EVERY call (a hook in
|
|
80
|
+
`qa-tester` blocks calls without it, and blocks `resize_window`).
|
|
81
|
+
3. After each action, confirm the URL or title in the result is your tab's. A
|
|
82
|
+
mismatch means crosstalk: repeat the action.
|
|
83
|
+
4. Max 2 agents in this lane at once. No `alert()`/`confirm()` triggers: they
|
|
84
|
+
freeze the tab. Never sign out, never change the user's account settings.
|
|
85
|
+
Vendure's dashboard keeps the active channel in localStorage shared by all tabs:
|
|
86
|
+
do not switch channel in the shared Chrome; use a headless session for that.
|
|
87
|
+
5. Close your tab with `tabs_close_mcp` when done, and also the empty "New tab"
|
|
88
|
+
that `tabs_context_mcp`/`navigate` may have created for you.
|
|
89
|
+
6. `get_page_text` can return "No text content found" on small pages; use
|
|
90
|
+
`read_page` or `find` with your `tabId` instead.
|
|
91
|
+
|
|
92
|
+
## 3. ios-sim (real Mobile Safari)
|
|
93
|
+
|
|
94
|
+
Helper: `bash .claude/skills/qa-genomklickning/scripts/ios-sim.sh`
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
UDID=$(bash ios-sim.sh up qa-cart "iPhone 17") # clone + boot, prints UDID
|
|
98
|
+
bash ios-sim.sh open $UDID http://hacken.localhost:3012/
|
|
99
|
+
bash ios-sim.sh shot $UDID <work>/shots/cart/CART-12-ios.png
|
|
100
|
+
bash ios-sim.sh down $UDID # shutdown + delete the clone
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Interact with the iOS Simulator tool (`mcp__Claude_Code_iOS_Simulator__control`,
|
|
104
|
+
`device: <UDID>`): `tap`, `text`, `swipe`, `screenshot`. Coordinates are device
|
|
105
|
+
points from its screenshot. The simulator shares the Mac's network, so `localhost`
|
|
106
|
+
and `*.localhost` work. Max 3 clones at once. Always `down` your clone, also on failure.
|
|
107
|
+
Look at every simulator screenshot: an ✕ in the address bar means the page is
|
|
108
|
+
still loading, so wait and shoot again before judging it.
|
|
109
|
+
|
|
110
|
+
## 4. android (real Chrome on Android emulator)
|
|
111
|
+
|
|
112
|
+
Installed on Tim's Mac (2026-10-22): AVDs `qa1` and `qa2` (Pixel 8, Android 35,
|
|
113
|
+
Play Store image), Chrome's first-run screen already dismissed on both with Tim's
|
|
114
|
+
OK. Elsewhere: `bash .claude/skills/qa-genomklickning/scripts/android-setup.sh`
|
|
115
|
+
(about 4 GB); the orchestrator does not run it on its own, it lists it in the
|
|
116
|
+
report as the way to unblock Android cases.
|
|
117
|
+
|
|
118
|
+
Helper: `bash .claude/skills/qa-genomklickning/scripts/android.sh` (puts the SDK
|
|
119
|
+
on PATH itself). One AVD per agent, `qa1` on port 5554, `qa2` on 5556.
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
A="$PWD"/.claude/skills/qa-genomklickning/scripts/android.sh
|
|
123
|
+
S=$(bash $A up qa2 5556) # boots headless (~15 s), prints emulator-5556
|
|
124
|
+
bash $A reverse $S 3012 # Mac localhost:3012 inside the emulator
|
|
125
|
+
bash $A open $S http://hacken.localhost:3012/ # opens in Chrome
|
|
126
|
+
CDP=$(bash $A cdp $S 9302) # Chrome DevTools on the Mac
|
|
127
|
+
playwright-cli attach --cdp=$CDP --session=qa-<area>-and
|
|
128
|
+
playwright-cli -s=qa-<area>-and snapshot # then click/fill/console/requests as in lane 1
|
|
129
|
+
playwright-cli -s=qa-<area>-and detach # detach, not close: the browser is the emulator Chrome
|
|
130
|
+
bash $A shot $S <work>/shots/<area>/X-android.png
|
|
131
|
+
bash $A down $S
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
`playwright-cli attach --cdp` drives the real Chrome on Android: DOM snapshot with
|
|
135
|
+
refs, click, fill, eval, console and network, all verified. Use it for page
|
|
136
|
+
interaction. `bash $A texts $S` and `bash $A tap-text $S "<text>"` (uiautomator)
|
|
137
|
+
work on Android system UI, Chrome's own dialogs and visible page text; use them
|
|
138
|
+
for quick checks, not for page interaction.
|
|
139
|
+
|
|
140
|
+
Before every `attach`, run `bash $A texts $S` and confirm Chrome is in front with
|
|
141
|
+
your URL. Chrome can drop to the home screen; a click then times out as "not
|
|
142
|
+
visible" and the DevTools socket dies (ECONNRESET). Recover with `open`, `cdp`
|
|
143
|
+
and a new `attach`. `attach` sees every Chrome tab on the device, so pick yours
|
|
144
|
+
by URL with `tab-list`/`tab-select`, not by index.
|
|
145
|
+
|
|
146
|
+
Dialogs that can appear: "Chrome notifications make things easier" (tap
|
|
147
|
+
"No thanks"), "Chrome isn't responding" when two emulators boot at once (tap
|
|
148
|
+
"Close app", then `open` again). A new or wiped AVD shows Chrome's Terms of
|
|
149
|
+
Service first-run screen: an agent dismisses it only with the user's recorded OK
|
|
150
|
+
in CONTEXT.md; otherwise the Android cases are BLOCKED with that reason.
|
|
151
|
+
|
|
152
|
+
Chrome resolves `*.localhost` to loopback itself, so `adb reverse` covers shop
|
|
153
|
+
subdomains; confirm with a screenshot on first use.
|
|
154
|
+
|
|
155
|
+
## 5. api / db
|
|
156
|
+
|
|
157
|
+
Every UI result is checked in data: GraphQL/REST with curl (cookie jar in
|
|
158
|
+
`<work>/auth/`), DB read-only queries (`docker exec <pg> psql ...`). Writes only on
|
|
159
|
+
`qa-` data.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Report structure: qa-report-<YYYY-MM-DD>.md
|
|
2
|
+
|
|
3
|
+
English. Numbered lists, tables get a # column. No em-dash.
|
|
4
|
+
|
|
5
|
+
1. **Summary**: branch, scope (`<base>...<head>`, commit count), rounds run.
|
|
6
|
+
Counts: cases, PASS, FAIL, fixed, not fixed, intermittent, BLOCKED. Verdict:
|
|
7
|
+
ready / not ready to release, with the one-line reason.
|
|
8
|
+
2. **Traceability matrix**: ticket → acceptance criterion → files → UI/endpoint →
|
|
9
|
+
cases → status.
|
|
10
|
+
3. **Fixed defects**: title, ticket, root cause (file:line), test added (file and
|
|
11
|
+
test name), red output in one line, commit SHA, reviewer verdict.
|
|
12
|
+
4. **Not fixed**, sorted Critical / High / Medium / Low: minimal repro steps,
|
|
13
|
+
expected vs actual (with spec source), evidence (screenshot path, console or
|
|
14
|
+
network lines, DB rows), both fixers' hypotheses and why each was ruled out,
|
|
15
|
+
why it was not fixed.
|
|
16
|
+
5. **Intermittent**: x/N, trigger condition, status.
|
|
17
|
+
6. **Spec vs code**: in code but not in spec, in spec but not in code.
|
|
18
|
+
7. **BLOCKED and why**: each case with the unblock ladder rungs tried, and what
|
|
19
|
+
would unblock it (for example "install Android SDK: scripts/android-setup.sh").
|
|
20
|
+
8. **Lanes and agents**: every subagent run with agent, model, effort, lane, count.
|
|
21
|
+
9. **Choices I made**: internal choices taken without asking.
|
|
22
|
+
10. **Open questions**: numbered, for unclear spec. Both readings tested where possible.
|
|
23
|
+
11. **Left behind**: `qa-` test data still present, branches, worktrees, simulators.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
#!/bin/bash
|
|
2
|
+
# One-time Android emulator setup on Apple Silicon without Android Studio.
|
|
3
|
+
# Downloads about 8 GB (JDK, SDK tools, Android 35 arm64 image). Creates AVDs qa1 and qa2.
|
|
4
|
+
set -euo pipefail
|
|
5
|
+
# Brew's automatic cleanup removes other old kegs and can fail on unrelated files.
|
|
6
|
+
export HOMEBREW_NO_INSTALL_CLEANUP=1
|
|
7
|
+
|
|
8
|
+
# The temurin cask is a .pkg that needs an admin password; skip it when a JDK exists.
|
|
9
|
+
/usr/libexec/java_home >/dev/null 2>&1 || brew install --cask temurin
|
|
10
|
+
brew list --cask android-commandlinetools >/dev/null 2>&1 || brew install --cask android-commandlinetools
|
|
11
|
+
export ANDROID_HOME="$(brew --prefix)/share/android-commandlinetools"
|
|
12
|
+
# `yes` dies of SIGPIPE when sdkmanager exits, which pipefail would treat as failure.
|
|
13
|
+
yes | sdkmanager --licenses >/dev/null || true
|
|
14
|
+
# sdkmanager is deprecated in favour of `android sdk` but still works and needs no login.
|
|
15
|
+
sdkmanager "platform-tools" "emulator" "platforms;android-35" \
|
|
16
|
+
"system-images;android-35;google_apis_playstore;arm64-v8a"
|
|
17
|
+
for n in qa1 qa2; do
|
|
18
|
+
echo no | avdmanager create avd -n "$n" \
|
|
19
|
+
-k "system-images;android-35;google_apis_playstore;arm64-v8a" --device pixel_8 --force
|
|
20
|
+
done
|
|
21
|
+
|
|
22
|
+
cat <<EOF
|
|
23
|
+
Done. Add to ~/.zshrc:
|
|
24
|
+
export ANDROID_HOME="$ANDROID_HOME"
|
|
25
|
+
export PATH="\$ANDROID_HOME/platform-tools:\$ANDROID_HOME/emulator:\$PATH"
|
|
26
|
+
EOF
|