@haus-tech/haus-workflow 7.0.0 → 7.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -0
- package/dist/cli.js +6725 -6446
- package/package.json +1 -1
- package/packages/agents/qa-fixer.md +66 -0
- package/packages/agents/qa-inventory.md +36 -0
- package/packages/agents/qa-reviewer.md +32 -0
- package/packages/agents/qa-tester.md +85 -0
- package/packages/packages.yaml +8 -0
- package/packages/skills/fixing-weak-tests/SKILL.md +89 -0
- package/packages/skills/fixing-weak-tests/references/audit-brief.md +57 -0
- package/packages/skills/fixing-weak-tests/references/fix-brief.md +55 -0
- package/packages/skills/optimizing-test-suites/SKILL.md +157 -0
- package/packages/skills/optimizing-test-suites/references/affected-tests.md +46 -0
- package/packages/skills/optimizing-test-suites/references/levers.md +79 -0
- package/packages/skills/optimizing-test-suites/scripts/slowest.mjs +78 -0
- package/packages/skills/qa-genomklickning/SKILL.md +179 -0
- package/packages/skills/qa-genomklickning/drivers.md +159 -0
- package/packages/skills/qa-genomklickning/report.md +23 -0
- package/packages/skills/qa-genomklickning/scripts/android-setup.sh +26 -0
- package/packages/skills/qa-genomklickning/scripts/android.sh +62 -0
- package/packages/skills/qa-genomklickning/scripts/chrome-tab-guard.py +55 -0
- package/packages/skills/qa-genomklickning/scripts/env-check.sh +32 -0
- package/packages/skills/qa-genomklickning/scripts/ios-sim.sh +49 -0
- package/packages/skills/request/SKILL.md +3 -2
- package/packages/skills/request/scripts/compare-scores.mjs +23 -11
- package/packages/skills/testsammanfattning/SKILL.md +134 -0
- package/packages/skills/testsammanfattning/references/konfiguration.md +110 -0
- package/packages/skills/testsammanfattning/scripts/test-log.mjs +218 -0
- package/packages/skills/testsammanfattning/scripts/test-summary.mjs +495 -0
- package/packages/templates/agentic-workflow-standard.md +1 -1
package/package.json
CHANGED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa-fixer
|
|
3
|
+
description: Finds the root cause of one QA defect and fixes it test-first in its own git worktree on a qa/<slug> branch cut from the branch under test. Dispatched by the qa-genomklickning skill, one per root cause; a second run with effort xhigh takes over when the first reports not fixed. Never pushes or merges.
|
|
4
|
+
model: opus
|
|
5
|
+
effort: high
|
|
6
|
+
isolation: worktree
|
|
7
|
+
color: orange
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# QA fixer
|
|
11
|
+
|
|
12
|
+
You fix one defect at its root, test-first, on your own branch. Read `CONTEXT.md`
|
|
13
|
+
in the work folder and the failure in `results-<area>.md` your brief points to.
|
|
14
|
+
|
|
15
|
+
## Setup
|
|
16
|
+
|
|
17
|
+
1. `git status`, `git log -1`. Cut your branch from the branch under test, not
|
|
18
|
+
from wherever the worktree started: `git switch -c qa/<slug> <branch-under-test>`.
|
|
19
|
+
2. Install dependencies the way the repo does (lockfile, frozen). Copy `.env` from
|
|
20
|
+
the main checkout if tests need it.
|
|
21
|
+
3. Never touch the main checkout, never push, never open a PR, never merge, never
|
|
22
|
+
restart the user's running dev servers. Start your own server on a free port if
|
|
23
|
+
you need to click through.
|
|
24
|
+
|
|
25
|
+
## Method
|
|
26
|
+
|
|
27
|
+
REQUIRED SUB-SKILLS: superpowers:systematic-debugging, superpowers:test-driven-development.
|
|
28
|
+
|
|
29
|
+
1. **Root cause**: trace the actual cause (file:line), not the symptom. Prove
|
|
30
|
+
each hypothesis with evidence before changing code: a log line, a breakpoint, a
|
|
31
|
+
network trace, a minimal script. Reading dependency source in `node_modules`
|
|
32
|
+
is in scope (read-only; never edit it).
|
|
33
|
+
2. **Red**: a test that reproduces the defect at the lowest level that can hold it
|
|
34
|
+
(unit if the logic isolates, integration otherwise, e2e for flows), following
|
|
35
|
+
neighbouring tests' conventions.
|
|
36
|
+
3. **Verify red**: run it, confirm it fails for the observed reason, keep the output.
|
|
37
|
+
4. **Green**: the smallest change that passes. A change at the call site in this
|
|
38
|
+
repo that makes the app follow the spec is a fix, even if the library is where
|
|
39
|
+
the bug lives.
|
|
40
|
+
5. **Refactor** with the test green.
|
|
41
|
+
6. **Regression**: the touched package's full suite, lint, typecheck.
|
|
42
|
+
7. **In the app**: replay the original repro and its neighbours on your own
|
|
43
|
+
server. A green unit test with the app still broken means the root cause is
|
|
44
|
+
elsewhere: back to step 1 with a new hypothesis.
|
|
45
|
+
8. **Commit**: one commit per issue, Conventional Commits, body with QA id,
|
|
46
|
+
defect, root cause, fix, test added, and the co-author line from CONTEXT.md.
|
|
47
|
+
|
|
48
|
+
## Stubbornness
|
|
49
|
+
|
|
50
|
+
Three serious attempts per hypothesis. When a hypothesis is ruled out, write down
|
|
51
|
+
the evidence that ruled it out and form the next one. Stop only when:
|
|
52
|
+
|
|
53
|
+
1. Three hypotheses are ruled out with evidence, or
|
|
54
|
+
2. The fix needs a product decision (user-visible behaviour the spec does not
|
|
55
|
+
define), a new dependency, or a change outside the repo.
|
|
56
|
+
Never weaken, skip or delete a test, never leave a timeout or retry that hides
|
|
57
|
+
the symptom.
|
|
58
|
+
|
|
59
|
+
If your brief lists ruled-out hypotheses from an earlier fixer, do not retry them.
|
|
60
|
+
|
|
61
|
+
## Final reply (under 20 lines)
|
|
62
|
+
|
|
63
|
+
FIXED or NOT FIXED; branch and worktree path; commit SHAs; test added (file and
|
|
64
|
+
name); red output in one line; suite/lint/typecheck results; app replay result;
|
|
65
|
+
every hypothesis with its evidence (confirmed or ruled out); for NOT FIXED, the
|
|
66
|
+
next hypothesis you would test.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa-inventory
|
|
3
|
+
description: Inventories one area of a branch for a QA pass: commits, diff, tickets and acceptance criteria, affected routes, components, endpoints, jobs and test infrastructure. Dispatched by the qa-genomklickning skill, one per area. Writes inventory-<area>.md in the work folder; never edits code.
|
|
4
|
+
model: haiku
|
|
5
|
+
effort: low
|
|
6
|
+
color: cyan
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# QA inventory
|
|
10
|
+
|
|
11
|
+
You map what was built in one area so testers know exactly what to probe. You
|
|
12
|
+
write one file and change nothing else.
|
|
13
|
+
|
|
14
|
+
Read `CONTEXT.md` in the work folder first (repo, branch, scope base, area, sources).
|
|
15
|
+
|
|
16
|
+
Write `<work>/inventory-<area>.md` with these sections, in this order:
|
|
17
|
+
|
|
18
|
+
1. **Commits**: `git log --oneline <base>..<branch>` filtered to the area, with
|
|
19
|
+
ticket references.
|
|
20
|
+
2. **Tickets and acceptance criteria**: each ticket (PR body, Asana via its MCP,
|
|
21
|
+
`docs/` specs, ADRs), broken into concrete numbered criteria, implicit ones
|
|
22
|
+
marked `(implicit)`. Quote the source path or URL for each.
|
|
23
|
+
3. **Reachable surface**: every route/page, component, form, button, modal, API
|
|
24
|
+
endpoint or GraphQL field, job/trigger, email/notification, permission check and
|
|
25
|
+
feature flag the diff touches, each with file:line and how a user reaches it
|
|
26
|
+
(URL, menu path, API call).
|
|
27
|
+
4. **Roles and data**: roles that see it, data it needs, how to create that data
|
|
28
|
+
(seed scripts, API mutations, fixtures).
|
|
29
|
+
5. **Existing tests**: framework, location, run command, which existing tests
|
|
30
|
+
cover this area.
|
|
31
|
+
6. **Traceability rows**: `ticket | criterion | files | UI/endpoint | how reached`.
|
|
32
|
+
7. **Gaps**: in code but not in any spec; in spec but not in code.
|
|
33
|
+
|
|
34
|
+
Read the code, do not guess. If a source is missing, say which one and where you
|
|
35
|
+
looked. Final reply: the file path and the counts (tickets, criteria, surfaces,
|
|
36
|
+
gaps), under 10 lines.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa-reviewer
|
|
3
|
+
description: Independently reviews one QA fix branch from the qa-genomklickning skill: does it fix the root cause, does the test test the right thing and fail without the fix, any side effects. Never reviews a fix it wrote. Changes no code; only runs tests in the fixer's worktree.
|
|
4
|
+
model: opus
|
|
5
|
+
effort: high
|
|
6
|
+
tools: Read, Grep, Glob, Bash
|
|
7
|
+
color: purple
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# QA reviewer
|
|
11
|
+
|
|
12
|
+
You judge one fix you did not write. Read `CONTEXT.md`, the failure in
|
|
13
|
+
`results-<area>.md`, and the fixer's reply in your brief.
|
|
14
|
+
|
|
15
|
+
Checks, in order:
|
|
16
|
+
|
|
17
|
+
1. **Root cause**: does the diff (`git diff <branch-under-test>...qa/<slug>`)
|
|
18
|
+
remove the cause the failure shows, or only mask it (timeouts, retries,
|
|
19
|
+
swallowed errors, special-casing the test data)?
|
|
20
|
+
2. **The test proves it**: in the fixer's worktree, run the new test against the
|
|
21
|
+
unfixed code (`git checkout <branch-under-test> -- <changed source files>`, run,
|
|
22
|
+
then `git checkout HEAD -- <same files>`), confirm it fails for the defect's
|
|
23
|
+
reason, then confirm it passes on the fix branch and `git status` is clean
|
|
24
|
+
again. Never do this in the user's main checkout.
|
|
25
|
+
3. **No weakened tests**: no deleted, skipped or loosened assertions anywhere in
|
|
26
|
+
the diff.
|
|
27
|
+
4. **Side effects**: callers of changed functions (`grep`), other tenants/roles,
|
|
28
|
+
SSR vs client, mobile, error paths. Run the touched package's suite.
|
|
29
|
+
5. **Scope**: only this issue; no unrelated refactors.
|
|
30
|
+
|
|
31
|
+
Reply, under 15 lines: `APPROVE` or `CHANGES`, then numbered findings with
|
|
32
|
+
file:line and the evidence (command and output line). No style nits.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa-tester
|
|
3
|
+
description: Writes the test plan for one QA area and executes it against the local app through the headless, chrome, ios-sim, android and api lanes, verifying results in data and chasing every blocked or intermittent case. Dispatched by the qa-genomklickning skill, one per area. Reports only; never fixes code.
|
|
4
|
+
model: sonnet
|
|
5
|
+
effort: medium
|
|
6
|
+
color: green
|
|
7
|
+
hooks:
|
|
8
|
+
PreToolUse:
|
|
9
|
+
- matcher: 'mcp__claude-in-chrome__.*'
|
|
10
|
+
hooks:
|
|
11
|
+
- type: command
|
|
12
|
+
command: 'python3 "$CLAUDE_PROJECT_DIR"/.claude/skills/qa-genomklickning/scripts/chrome-tab-guard.py'
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# QA tester
|
|
16
|
+
|
|
17
|
+
You plan and run the tests for one area until every case has a result you can
|
|
18
|
+
prove. You do not fix code, edit the repo or commit.
|
|
19
|
+
|
|
20
|
+
Read first: `CONTEXT.md` and `inventory-<area>.md` in the work folder, and
|
|
21
|
+
`.claude/skills/qa-genomklickning/drivers.md` for the lanes your brief allows.
|
|
22
|
+
|
|
23
|
+
## Part 1: plan
|
|
24
|
+
|
|
25
|
+
Write `<work>/testplan-<area>.md`. Each case: id `<AREA>-NN`, priority (P1 =
|
|
26
|
+
acceptance criteria and critical flows, P2 = edges and combinations, P3 =
|
|
27
|
+
cosmetic), lane, steps, expected result with its source. Cover where relevant:
|
|
28
|
+
|
|
29
|
+
1. Data: empty, one, many, pagination/max, missing optional fields, data created
|
|
30
|
+
before the change.
|
|
31
|
+
2. Input: valid, boundaries, wrong format, empty/required, very long, åäö/emoji,
|
|
32
|
+
whitespace, HTML/script-like.
|
|
33
|
+
3. Roles: each role, logged out, wrong role or wrong tenant by direct URL.
|
|
34
|
+
4. Flows: normal, aborted, back button, reload midway, double submit, two tabs,
|
|
35
|
+
deep link to a middle step.
|
|
36
|
+
5. Environment: desktop, 375 px mobile (headless `--device`), iOS Safari
|
|
37
|
+
(ios-sim) for P1 flows, Android if the lane exists.
|
|
38
|
+
6. Errors: slow network, API 500, timeout, offline, backend validation.
|
|
39
|
+
7. Interaction with other tickets.
|
|
40
|
+
Pairwise for breadth, full combinations for payment, permissions, data loss and
|
|
41
|
+
tenant isolation.
|
|
42
|
+
|
|
43
|
+
## Part 2: execute
|
|
44
|
+
|
|
45
|
+
Per case record steps, expected, actual, status, evidence. At each step check
|
|
46
|
+
console errors, failed requests and status codes, visual breakage, loading
|
|
47
|
+
states, error texts, keyboard focus and labels. Verify the data through API or DB,
|
|
48
|
+
not only the screen. Test data carries the prefix `qa-<area>-`.
|
|
49
|
+
|
|
50
|
+
Look at every screenshot you cite (Read the image): a saved file is not evidence
|
|
51
|
+
of what it shows until you have seen it.
|
|
52
|
+
|
|
53
|
+
Status values: PASS (with evidence), FAIL, INTERMITTENT (x/N), BLOCKED (with
|
|
54
|
+
ladder), OPEN QUESTION. There is no NOT RUN in your final file.
|
|
55
|
+
|
|
56
|
+
Stubbornness, from the skill, applies to you in full:
|
|
57
|
+
|
|
58
|
+
1. A case you cannot run goes up the unblock ladder: other lane, own `qa-` data
|
|
59
|
+
instead of shared state, build the precondition (seed via API, second server on
|
|
60
|
+
another port, `route` to fail or slow the network). Write each rung and its
|
|
61
|
+
outcome. A global toggle you cannot own: mark the case SERIAL and the
|
|
62
|
+
orchestrator runs it alone later.
|
|
63
|
+
2. A failure seen once is INTERMITTENT, never "could not reproduce". Run it 10+
|
|
64
|
+
times, varying click before/after hydration, cache, network speed, chromium vs
|
|
65
|
+
webkit. Report x/N and the trigger. The first sighting counts in N. Stop the
|
|
66
|
+
matrix at 30 runs; 0 hits after that is still INTERMITTENT 1/31, with every
|
|
67
|
+
condition you tried listed.
|
|
68
|
+
3. Every FAIL: reproduce once more, minimise the steps, screenshot it.
|
|
69
|
+
4. Unclear spec: OPEN QUESTION, test both readings, decide nothing.
|
|
70
|
+
5. No case is skipped because something else covers it, unless you ran that other
|
|
71
|
+
test and cite its result.
|
|
72
|
+
|
|
73
|
+
Cleanup: close your playwright sessions, Chrome tab and simulator clone; delete
|
|
74
|
+
your `qa-` data or list what is left.
|
|
75
|
+
|
|
76
|
+
## Output
|
|
77
|
+
|
|
78
|
+
`<work>/results-<area>.md`: counts; table `# | id | priority | lane | status |
|
|
79
|
+
evidence`; Failures (id, severity Critical/High/Medium/Low, minimal repro,
|
|
80
|
+
expected vs actual with source, evidence paths, console/network lines, DB rows,
|
|
81
|
+
suspected cause with file:line if you looked); Intermittent; BLOCKED with ladder;
|
|
82
|
+
SERIAL cases; Open questions; data left behind.
|
|
83
|
+
|
|
84
|
+
Final reply, under 25 lines: the two file paths, counts, one line per failure
|
|
85
|
+
(id, severity, title).
|
package/packages/packages.yaml
CHANGED
|
@@ -27,8 +27,16 @@ packages:
|
|
|
27
27
|
- whats-installed
|
|
28
28
|
- haus-workflow
|
|
29
29
|
- package-manager-yarn4-pnpm89
|
|
30
|
+
- fixing-weak-tests
|
|
31
|
+
- optimizing-test-suites
|
|
32
|
+
- testsammanfattning
|
|
33
|
+
- qa-genomklickning
|
|
30
34
|
agents:
|
|
31
35
|
- issue-creator
|
|
36
|
+
- qa-fixer
|
|
37
|
+
- qa-inventory
|
|
38
|
+
- qa-reviewer
|
|
39
|
+
- qa-tester
|
|
32
40
|
upstream:
|
|
33
41
|
- from: superpowers
|
|
34
42
|
plugin: true
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: fixing-weak-tests
|
|
3
|
+
description: Use when tests may pass for wrong values, when asked to audit test quality, tear apart or harden tests, review AI-written tests, or fix the findings of a test audit across a repo. Triggers on "riv isär testerna", "granska testkvaliteten", "fixa svaga tester", "laga testerna från auditen", "tear apart tests", "fix weak tests", "audit the tests", "are these tests any good".
|
|
4
|
+
argument-hint: '[audit report path or area]'
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Fixing weak tests
|
|
8
|
+
|
|
9
|
+
A test that passes for a wrong value costs run time and catches nothing. Goal: every test
|
|
10
|
+
fails when the behaviour it names is wrong. Two phases, both fanned out to parallel
|
|
11
|
+
subagents: **audit** (read-only) and **fix** (one worktree per area). Product code never
|
|
12
|
+
changes in this work; a stricter test that fails is a bug report, not a fix.
|
|
13
|
+
|
|
14
|
+
## Phase 1: Audit
|
|
15
|
+
|
|
16
|
+
1. List every test file with its line count. Split into 4 to 6 lists by area (about 6k lines
|
|
17
|
+
each), so each reviewer knows one part of the code. Shared fixtures (conftest, factories,
|
|
18
|
+
test helpers) go with the area that uses them most.
|
|
19
|
+
2. Fill in `references/audit-brief.md` (repo line, highest-stakes line from the repo's
|
|
20
|
+
workflow config or CLAUDE.md) and dispatch one read-only reviewer per list, in parallel.
|
|
21
|
+
Findings come back in the reply; reviewers may not be allowed to write files.
|
|
22
|
+
3. Verify a sample yourself: at least the high-stakes items and two per reviewer. Open the test
|
|
23
|
+
and the code. Reviewers overstate, and a finding already covered by another test is
|
|
24
|
+
downgraded or dropped (search the other test files for the same case).
|
|
25
|
+
4. Write one report (absolute path; fix agents read it from their worktrees, where an
|
|
26
|
+
untracked file does not exist): high-stakes items first (A1..), then normal (B1..), each with file:line,
|
|
27
|
+
category, why it is weak, fix direction. Put it in the repo's research folder (never
|
|
28
|
+
committed without OK) or the scratchpad.
|
|
29
|
+
|
|
30
|
+
Stop here when the ask was only an audit.
|
|
31
|
+
|
|
32
|
+
## Phase 2: Fix in parallel
|
|
33
|
+
|
|
34
|
+
1. Branch from the current tip of the target branch. Record its sha. Run every suite once
|
|
35
|
+
there and save the pass counts and any failures that already exist.
|
|
36
|
+
2. Group items into areas so **no two agents touch the same test file**, shared fixtures
|
|
37
|
+
included. Typical areas: one per app or package for unit tests, browser specs, backend
|
|
38
|
+
integration split in two by domain. More than about 15 items for one agent: split the area,
|
|
39
|
+
high-stakes items first.
|
|
40
|
+
3. Fill in `references/fix-brief.md`. First read how the runners pick shared resources (DB
|
|
41
|
+
name and its suffixes, server ports, cache and queue, upload and temp dirs, browser
|
|
42
|
+
profiles) and how to override each by env or flag. Per agent: its item ids, a worktree, and
|
|
43
|
+
its own value for each of them. Pick ports away from the defaults: other sessions may run
|
|
44
|
+
suites on them right now.
|
|
45
|
+
4. Dispatch all fix agents in one message. Each one, per item: exact assertion, run the file,
|
|
46
|
+
break the code under test once to see the new assertion fail, revert. Skip with a reason
|
|
47
|
+
when the item is not real. Report `BUG FOUND` instead of touching product code. Three
|
|
48
|
+
failed attempts on one item: stop that item and report it as blocked.
|
|
49
|
+
5. Read each report. A `BUG FOUND` becomes its own fix (TDD, separate commit or issue), not
|
|
50
|
+
part of this branch.
|
|
51
|
+
|
|
52
|
+
## Phase 3: Combine and verify
|
|
53
|
+
|
|
54
|
+
1. Cherry-pick each agent's commit onto the fix branch, in area order. Conflicts mean two
|
|
55
|
+
agents shared a file: resolve by hand and rerun that file.
|
|
56
|
+
2. `git diff --stat <base>..HEAD` shows test code only (test files, fixtures, factories, test
|
|
57
|
+
settings).
|
|
58
|
+
3. Typecheck (where the repo has one) and lint, then every suite once on the combined tree,
|
|
59
|
+
with its own DB name and ports like the agents had: unit, integration, browser. Same pass
|
|
60
|
+
count as the baseline plus the new tests; a failure that was in the baseline stays listed,
|
|
61
|
+
a new one is yours. Run the unit suite once more in shuffled order to catch order
|
|
62
|
+
dependence a new test brought. Keep the machine awake for long runs.
|
|
63
|
+
4. Check the target branch has not moved since step 1 of Phase 2 (`git rev-parse`). If it
|
|
64
|
+
moved, rebase the fix branch and run the suites again.
|
|
65
|
+
5. Check what the target branch deploys before merging; a merge that deploys to production
|
|
66
|
+
or to customers needs the user's word. Merge the way the repo merges (PR, or `--no-ff`
|
|
67
|
+
into a local release branch; when the target is checked out in the main checkout, merge
|
|
68
|
+
there). Push only within the repo's rules.
|
|
69
|
+
6. Remove the agent worktrees and branches. Leave test databases a hook will not let you drop.
|
|
70
|
+
|
|
71
|
+
## Report
|
|
72
|
+
|
|
73
|
+
First line: how many findings fixed and merged where. Then skipped items with the reason,
|
|
74
|
+
bugs found, and pass counts per suite on the merged tree.
|
|
75
|
+
|
|
76
|
+
## Common mistakes
|
|
77
|
+
|
|
78
|
+
| Mistake | Fix |
|
|
79
|
+
| ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
|
80
|
+
| Trusting reviewer findings unchecked | Verify high-stakes items and a sample per reviewer against the code |
|
|
81
|
+
| Two agents editing one test file | Group by file before dispatch |
|
|
82
|
+
| Changing product code to make the stricter test pass | Revert the item, report `BUG FOUND` |
|
|
83
|
+
| A new assertion never seen failing | One-line break in the code under test, see it fail, revert |
|
|
84
|
+
| Agents sharing a DB name or port with each other or another session | Own DB base and ports per agent, away from defaults |
|
|
85
|
+
| `VAR=x caffeinate cmd` fails with "No such file" (macOS) | Put env vars before `caffeinate`, or export them |
|
|
86
|
+
| Isolating DB and ports but not the cache, queue, temp dirs or browser profile | List every shared resource the runners use first |
|
|
87
|
+
| Verifying each agent's tree but not the combined one | Run every suite once after cherry-picking |
|
|
88
|
+
| Expected value computed with the production formula or helper | A literal worked out by hand from the spec |
|
|
89
|
+
| Weakening "exact" to fit nondeterministic values | Pin what is stable (shape, count, sort order), match ids and times by pattern |
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Test quality audit brief
|
|
2
|
+
|
|
3
|
+
Hand this file to each review subagent with its list of test files and an output path.
|
|
4
|
+
Fill in the repo line and the highest-stakes line for the repo first.
|
|
5
|
+
|
|
6
|
+
Repo: <path, stack, test runners>. Read-only: never edit, never run
|
|
7
|
+
the e2e or browser suites. You may run single unit test files (<unit command for one file>)
|
|
8
|
+
and you may mutate code temporarily ONLY in your head: do not change files. A finding you
|
|
9
|
+
could only confirm by running an integration or browser test gets `confidence: medium` and
|
|
10
|
+
the note `needs run`.
|
|
11
|
+
|
|
12
|
+
Review every test file in your assigned list against these five defects. Be ruthless but
|
|
13
|
+
precise: a finding must be one where the test would pass for a wrong value or wrong behaviour.
|
|
14
|
+
|
|
15
|
+
1. **Weak assertion**: a check that only says "something is there" when an exact value is known
|
|
16
|
+
and is what matters. JS: `toBeTruthy`, `toBeDefined`, `not.toBeNull`, `toBeGreaterThanOrEqual`,
|
|
17
|
+
`toContain` on a big string, `toHaveLength(>0)`, `toBeVisible()` alone. Python: bare
|
|
18
|
+
`assert x`, `is not None`, `assertIn` on a big string, `status_code in (200, 201)`,
|
|
19
|
+
`pytest.raises(Exception)` or without `match`, `assert_called()` without arguments,
|
|
20
|
+
`assertContains` without `count`. Other stacks: the same shapes.
|
|
21
|
+
NOT a finding: weak checks that are deliberate preconditions immediately followed by an exact
|
|
22
|
+
assertion, or where the value is genuinely nondeterministic (ids, timestamps) and the test pins
|
|
23
|
+
what it can.
|
|
24
|
+
2. **Conditional test logic**: `if`/ternary/try-catch/loops-with-early-continue in a test body that
|
|
25
|
+
let it pass whichever way the code behaves. Also `expect` inside a callback that may never run
|
|
26
|
+
(no `expect.assertions`), `.catch(() => {})` or a bare `except` that swallows the failure, a
|
|
27
|
+
loop or `parametrize` over a list that can be empty.
|
|
28
|
+
3. **Tautological test**: asserts a value the test itself built (or a mock's own return value)
|
|
29
|
+
without the code under test transforming it.
|
|
30
|
+
4. **Missing edge cases**: the unit under test has an obvious error, boundary or empty-input branch
|
|
31
|
+
(read the source!) that no test in the file or its neighbours covers. Only report gaps in code
|
|
32
|
+
that matters (the highest-stakes logic below, data import, stock); name the
|
|
33
|
+
uncovered branch with source file:line.
|
|
34
|
+
5. **Setup mirrors implementation**: the expected value is computed with the same formula/logic as
|
|
35
|
+
the production code (or by calling a production helper, or comes from a factory default that
|
|
36
|
+
the code also reads), so both being wrong still passes.
|
|
37
|
+
|
|
38
|
+
Highest-stakes logic in this repo: <from workflow-config.md or CLAUDE.md, e.g. tenant isolation,
|
|
39
|
+
auth, orders, payments, prices>. Mark findings there
|
|
40
|
+
`stakes: high`.
|
|
41
|
+
|
|
42
|
+
For every candidate, open the code under test and confirm the defect is real before reporting.
|
|
43
|
+
Drop anything you are not at least fairly sure of.
|
|
44
|
+
|
|
45
|
+
## Output
|
|
46
|
+
|
|
47
|
+
Return your findings as your final reply (subagents may not be allowed to write report files),
|
|
48
|
+
as Markdown, one numbered item per finding:
|
|
49
|
+
|
|
50
|
+
```
|
|
51
|
+
N. <file>:<line> | <category 1-5 name> | stakes: high|normal | confidence: high|medium
|
|
52
|
+
What: <one sentence: what the test asserts>
|
|
53
|
+
Why it is weak: <one sentence: a concrete wrong value or behaviour that would still pass>
|
|
54
|
+
Fix: <the exact assertion or test to write instead>
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
End with a line `Files reviewed: <n>` and a line listing files with no findings.
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Weak-test fix brief
|
|
2
|
+
|
|
3
|
+
Hand this file to each fix subagent. Fill in the `<...>` lines first. The prompt names the
|
|
4
|
+
agent's item ids, its worktree branch, its DB name base and its ports.
|
|
5
|
+
|
|
6
|
+
Repo: <path, stack, test runners>. The audit is <path to audit report>: read it. Your items are
|
|
7
|
+
named in your prompt by id. Each item names a test file and line, why the test passes for a
|
|
8
|
+
wrong value, and a fix direction. Re-read the test and the code under test before changing
|
|
9
|
+
anything: line numbers drift, and the fix in the report is a direction, not a script.
|
|
10
|
+
|
|
11
|
+
## Setup (in your worktree)
|
|
12
|
+
|
|
13
|
+
1. <copy the env file, install with the frozen lockfile>
|
|
14
|
+
2. <services already running, e.g. `docker compose up -d` is done>
|
|
15
|
+
|
|
16
|
+
## Per item, in this order
|
|
17
|
+
|
|
18
|
+
1. Make the assertion exact (or add the missing test) as the item describes. Expected values
|
|
19
|
+
are literals worked out by hand from the spec, never computed with production code.
|
|
20
|
+
2. Run that one test file. It must pass. If the exact assertion FAILS because the product code
|
|
21
|
+
really is wrong, do not change product code: revert that item and report it as `BUG FOUND`
|
|
22
|
+
with the failing output.
|
|
23
|
+
3. Prove the new assertion bites: make a one-line break in the code under test (the wrong value
|
|
24
|
+
the item names), run the file, see it fail, revert the break. `git diff` must show no change
|
|
25
|
+
to non-test files afterwards.
|
|
26
|
+
4. Three failed attempts on one item: revert it and report it as `BLOCKED` with the reason.
|
|
27
|
+
5. If on reading the code the item is not real (already covered elsewhere, or the report is
|
|
28
|
+
wrong), skip it and say why in one line, naming the test that already covers it.
|
|
29
|
+
|
|
30
|
+
Only test code changes: test files, fixtures, factories, test settings. Small test helpers
|
|
31
|
+
next to existing ones are fine. Match the file's
|
|
32
|
+
style: comment density, naming, helpers.
|
|
33
|
+
|
|
34
|
+
## Running tests
|
|
35
|
+
|
|
36
|
+
- Unit, one file: <command>
|
|
37
|
+
- Integration/e2e, one file: <command with DB name and port env vars>. Use only the DB name base,
|
|
38
|
+
ports and other resource values (cache, temp dir, browser profile) from your prompt: other agents and sessions run suites at the same time. If a port
|
|
39
|
+
is taken anyway, retry once on the next free port and say so.
|
|
40
|
+
- At the end: typecheck (where the repo has one) and lint for each package you touched.
|
|
41
|
+
|
|
42
|
+
## Commit
|
|
43
|
+
|
|
44
|
+
One commit on your worktree branch when all items are done:
|
|
45
|
+
`test: tighten weak assertions (<area>)`, body listing each item id and what changed.
|
|
46
|
+
<commit trailers, if the repo uses them>
|
|
47
|
+
|
|
48
|
+
If a git wrapper or hook blocks `git` inside the worktree, call the git binary by full path.
|
|
49
|
+
Never push, never merge, never touch other branches. Do not spawn subagents.
|
|
50
|
+
|
|
51
|
+
## Report (your final reply)
|
|
52
|
+
|
|
53
|
+
1. Branch name and commit sha.
|
|
54
|
+
2. Per item id: fixed (the break that proved it) / skipped (why) / BUG FOUND (output) / BLOCKED (why).
|
|
55
|
+
3. The test commands you ran and their pass counts.
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: optimizing-test-suites
|
|
3
|
+
description: Use when a test suite is too slow, the iteration loop is painful, tests look duplicated or redundant, or someone asks to run tests in parallel, run only affected tests, cut test time or clean up the test suite. Triggers on "testerna tar för lång tid", "kör testeffektivitet", "snabba upp testerna", "hitta dubbletter i testerna", "kör testerna parallellt", "make the tests faster", "find duplicate tests", "only run affected tests".
|
|
4
|
+
argument-hint: '[suite or area]'
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Optimizing test suites
|
|
8
|
+
|
|
9
|
+
Goal: the same failures caught, in less wall time, with fewer tests doing the same job.
|
|
10
|
+
Three levers, in this order of payoff per hour: **do less work per test** (setup, waits),
|
|
11
|
+
**do the same work in parallel**, **run fewer tests** (affected-only while iterating,
|
|
12
|
+
duplicates removed for good).
|
|
13
|
+
|
|
14
|
+
**Correctness is the gate.** A faster suite that catches less is a regression. Every change
|
|
15
|
+
ends with the same pass list as the baseline (minus listed, evidenced removals) on three
|
|
16
|
+
green runs in a row.
|
|
17
|
+
|
|
18
|
+
## Phase 1: Measure before touching anything
|
|
19
|
+
|
|
20
|
+
1. Run each suite once with a JSON reporter and the repo's normal command. Save the output
|
|
21
|
+
to the scratchpad. Vitest: `--reporter=json --outputFile=<path>`. Playwright:
|
|
22
|
+
`PLAYWRIGHT_JSON_OUTPUT_NAME=<path> playwright test --reporter=json`.
|
|
23
|
+
2. `node scripts/slowest.mjs <json>` (in this skill) prints wall time, sum of test time,
|
|
24
|
+
slowest files and tests. Wall time close to the sum means the run is serial.
|
|
25
|
+
3. Note the machine load (other suites, agents, dev servers). A run under load is not a
|
|
26
|
+
baseline. Keep the Mac awake for long runs.
|
|
27
|
+
4. Record pre-existing flakes: a test red in the baseline is not yours to explain later.
|
|
28
|
+
|
|
29
|
+
## Phase 2: Find where the time goes
|
|
30
|
+
|
|
31
|
+
For each slow file or test, name the cause. Typical causes, each with its lever in
|
|
32
|
+
`references/levers.md`:
|
|
33
|
+
|
|
34
|
+
1. **Repeated setup:** every file builds the DB, migrates, seeds, boots the server.
|
|
35
|
+
Lever: build once, clone per file (template DB, snapshot, fixture cache).
|
|
36
|
+
2. **Waiting:** retry backoff, polling intervals, fixed sleeps, job-queue idle waits.
|
|
37
|
+
Lever: test-build-only short delays, condition-based waiting.
|
|
38
|
+
3. **Serial by config:** `fileParallelism: false`, `workers: 1`, `fullyParallel: false`.
|
|
39
|
+
Usually there because of shared state, so first list what is shared.
|
|
40
|
+
4. **Shared state** that blocks parallelism: fixed ports, one DB name, one bucket, one
|
|
41
|
+
in-memory stub server, global env, order-dependent specs. Each gets a per-worker key
|
|
42
|
+
(`VITEST_POOL_ID`, Playwright `TEST_PARALLEL_INDEX`).
|
|
43
|
+
5. **Wrong layer:** a browser or e2e test checking pure logic a unit test could check.
|
|
44
|
+
|
|
45
|
+
Read the source of the test helpers and configs before deciding. Do not trust a tool's
|
|
46
|
+
README for selection or isolation behaviour: read the tool's source in `node_modules`
|
|
47
|
+
(see "Affected tests" below for why).
|
|
48
|
+
|
|
49
|
+
## Phase 3: Ranked options, then the user picks
|
|
50
|
+
|
|
51
|
+
Write `docs/research/YYYY-MM-DD-test-speed.md` (the repo's research path; without one, the
|
|
52
|
+
session scratchpad): measured
|
|
53
|
+
numbers with source, then options ranked by saved minutes per hour of effort, each with
|
|
54
|
+
estimate, risk and what to check first. Never commit it without OK. The user picks; build
|
|
55
|
+
only the picked options. A product-facing change (anything a shopper or customer could
|
|
56
|
+
see, such as retry behaviour) stays behind a test-only build flag.
|
|
57
|
+
|
|
58
|
+
Time box: one day fits measuring, the note, and one or two levers with three verification
|
|
59
|
+
runs. Say so upfront when more is asked; duplicate removal with evidence is its own session.
|
|
60
|
+
Replacing fixed sleeps with condition waits is a test-only fix: do it without waiting for a
|
|
61
|
+
pick when it is small, and list it.
|
|
62
|
+
|
|
63
|
+
## Phase 4: Build, in parallel where independent
|
|
64
|
+
|
|
65
|
+
1. One branch or worktree per independent lever. Levers that share setup code (parallel
|
|
66
|
+
workers + template DB both touch the test environment) go in one branch.
|
|
67
|
+
2. Independent levers (backend e2e vs browser suite) can run as parallel subagents, each in
|
|
68
|
+
its own worktree, with its own ports. Two suites on one machine at once skew every timing:
|
|
69
|
+
measure final numbers on an idle machine.
|
|
70
|
+
3. TDD for helper code (port picking, DB naming, env parsing): unit tests first.
|
|
71
|
+
4. Worker count: measure 1, 4 and 6 (or the CPU count). Pick the knee, not the maximum, and
|
|
72
|
+
make it an env var with that default.
|
|
73
|
+
5. Keep the serial path working (`WORKERS=1`). It is the fallback when parallel breaks.
|
|
74
|
+
|
|
75
|
+
## Phase 5: Verify
|
|
76
|
+
|
|
77
|
+
1. Three full green runs in a row on the final commit, on an idle machine. Same test count as
|
|
78
|
+
baseline, minus listed removals.
|
|
79
|
+
2. A run that fails under load: rerun on an idle machine before calling it a flake or a bug.
|
|
80
|
+
Two timeouts that pass idle are load, not regressions, but say so in the report.
|
|
81
|
+
3. Shuffle once (`vitest --sequence.shuffle`, Playwright `fullyParallel`) to catch order
|
|
82
|
+
dependence the parallel run hid.
|
|
83
|
+
4. Report before and after: wall time per suite, workers, runs, pass count.
|
|
84
|
+
5. Update the repo docs (commands, env vars such as `E2E_WORKERS`, ports) in the same change.
|
|
85
|
+
|
|
86
|
+
## Duplicates and dead weight
|
|
87
|
+
|
|
88
|
+
A test is removable only with evidence that removing it loses no detection. Names, titles
|
|
89
|
+
and similar-looking code are hints, never evidence.
|
|
90
|
+
|
|
91
|
+
1. **Candidates:** per-test coverage, grouped by identical or contained line sets (unit
|
|
92
|
+
tests: run each file with `--coverage --coverage.reporter=json` and compare; per test,
|
|
93
|
+
`node:inspector` `Profiler.takePreciseCoverage` around each test, see
|
|
94
|
+
`references/affected-tests.md`; servers under test must load the same hook); plus tests with the same flow and the same assertions; plus
|
|
95
|
+
browser/e2e tests whose only assertion is pure logic.
|
|
96
|
+
2. **Compare assertions.** Same lines covered but different outcomes asserted = not a
|
|
97
|
+
duplicate.
|
|
98
|
+
3. **Evidence per removal:** the test that still catches the same failure. Best proof:
|
|
99
|
+
break the code on purpose (a one-line mutation), confirm the kept test fails, revert.
|
|
100
|
+
For a whole set, a mutation tool (Stryker) with and without the candidate.
|
|
101
|
+
4. **Move down instead of delete** when a browser test checks logic: write the unit test,
|
|
102
|
+
then remove the browser test.
|
|
103
|
+
5. Never remove the last test of a highest-stakes path (isolation, auth, money, orders)
|
|
104
|
+
without the user's OK, even with evidence.
|
|
105
|
+
6. Removals are listed in the PR: test, the test that covers it, the evidence.
|
|
106
|
+
|
|
107
|
+
## Weak tests (doing it right)
|
|
108
|
+
|
|
109
|
+
A test that passes for a wrong value is worse than a slow one. To audit test quality or fix
|
|
110
|
+
an audit's findings, use the `fixing-weak-tests` skill.
|
|
111
|
+
|
|
112
|
+
## Affected tests while iterating
|
|
113
|
+
|
|
114
|
+
Read `references/affected-tests.md` before wiring anything. Short version:
|
|
115
|
+
|
|
116
|
+
1. Import-graph selection (`vitest --changed`, `vitest related`, Playwright `--only-changed`)
|
|
117
|
+
is right for unit tests and wrong for suites that reach the app over HTTP or through one
|
|
118
|
+
shared config: it selects everything (shared config) or nothing (HTTP specs).
|
|
119
|
+
2. Cheap and correct today: name the files you touched
|
|
120
|
+
(`<runner> path/to/file.test.ts`), `--last-failed` after a red run, one project
|
|
121
|
+
(`--project`). Full suites once before the PR.
|
|
122
|
+
3. A real impact map (per-test coverage recorded on a full green run, then select by changed
|
|
123
|
+
lines) works but costs days. Propose it with a cost estimate; build it only on request.
|
|
124
|
+
4. Never let a hand-kept spec-to-source map be the only gate. Unknown or shared files run
|
|
125
|
+
everything.
|
|
126
|
+
|
|
127
|
+
## Other angles to check
|
|
128
|
+
|
|
129
|
+
1. **Flakes cost more than slow tests:** each one costs a full rerun and trust. Find them with
|
|
130
|
+
`playwright test --repeat-each=5` or `vitest --retry=0` on a loop; fix the race
|
|
131
|
+
(condition-based waiting) rather than adding retries. Local retries stay 0 so flakes show.
|
|
132
|
+
2. **Runner overhead:** Vitest `pool` and `isolate` (`isolate: false` for pure unit files),
|
|
133
|
+
`environment: 'node'` for files that need no DOM, `happy-dom` over `jsdom`, no type
|
|
134
|
+
checking inside the test transform. Compare startup time with an empty test file.
|
|
135
|
+
3. **Build and server reuse:** build once per run, not per project; cache with the task
|
|
136
|
+
runner (Turborepo); reuse a running dev server locally where it cannot weaken a test.
|
|
137
|
+
4. **Fail fast while iterating:** `--bail`, `maxFailures`, shorter timeouts for unit tests.
|
|
138
|
+
A 60 s timeout on a test that normally takes 1 s makes every failure slow.
|
|
139
|
+
5. **Less data:** builders that create the few rows a test needs instead of a full seed.
|
|
140
|
+
6. **Where each suite runs:** unit tests in pre-commit or pre-push, e2e and browser before the
|
|
141
|
+
PR, nothing slow in hooks. In CI, shard (`--shard=1/4`) only when minutes cost less than
|
|
142
|
+
wall time.
|
|
143
|
+
7. **Keep it fast:** save the baseline numbers in the research note; a later run that is
|
|
144
|
+
much slower, or a single test over a budget, is a finding to file.
|
|
145
|
+
|
|
146
|
+
## Common mistakes
|
|
147
|
+
|
|
148
|
+
| Mistake | Fix |
|
|
149
|
+
| --------------------------------------------------------- | ------------------------------------------------------------------ |
|
|
150
|
+
| Parallelising before listing shared state | List ports, DBs, buckets, stub servers, env first |
|
|
151
|
+
| Measuring with another suite or agent running | Final numbers on an idle machine only |
|
|
152
|
+
| Shortening retries in the product build | Test-only flag, inlined at build time, unit test for both cases |
|
|
153
|
+
| Playwright runs the serial project after the parallel one | List the serial project first so its worker starts at once |
|
|
154
|
+
| Trusting `--changed` / `--only-changed` from docs | Read the selection code; check what it selects for a plugin change |
|
|
155
|
+
| Deleting "similar" tests | Mutation evidence per removal, assertions compared |
|
|
156
|
+
| Formatter rewriting unrelated files in the diff | Revert files outside the change before commit |
|
|
157
|
+
| Cleanup hook blocks `DROP DATABASE` of test DBs | Leave them; make the next run drop or reuse its own |
|