qa-engineer 0.9.1 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/COMPATIBILITY.md +50 -26
- package/README.md +40 -32
- package/package.json +12 -20
- package/packages/engine/bin/qa-engine.mjs +336 -0
- package/packages/engine/lib/analysis/branding.mjs +187 -0
- package/packages/engine/lib/analysis/context.mjs +294 -0
- package/packages/{installer/lib/core/schema-validate.mjs → engine/lib/analysis/contracts.mjs} +11 -4
- package/packages/engine/lib/analysis/diff-guard.mjs +425 -0
- package/packages/engine/lib/analysis/discovery.mjs +220 -0
- package/packages/engine/lib/analysis/evidence.mjs +116 -0
- package/packages/engine/lib/analysis/har.mjs +90 -0
- package/packages/engine/lib/analysis/junit.mjs +126 -0
- package/packages/engine/lib/analysis/redaction.mjs +127 -0
- package/packages/engine/lib/analysis/report-html.mjs +763 -0
- package/packages/engine/lib/analysis/taxonomy.mjs +109 -0
- package/packages/engine/lib/analysis/xml.mjs +153 -0
- package/packages/engine/lib/analysis/zip.mjs +107 -0
- package/packages/engine/lib/diagnostics/engine.mjs +165 -0
- package/packages/engine/lib/diagnostics/internal-contracts.mjs +50 -0
- package/packages/engine/lib/diagnostics/prioritization.mjs +99 -0
- package/packages/engine/lib/diagnostics/repair.mjs +73 -0
- package/packages/engine/lib/diagnostics/root-cause.mjs +92 -0
- package/packages/engine/lib/diagnostics/timeline.mjs +101 -0
- package/packages/engine/lib/frameworks/junit-frameworks.mjs +48 -0
- package/packages/engine/lib/frameworks/playwright.mjs +158 -0
- package/packages/engine/package.json +14 -0
- package/packages/installer/lib/agents/registry.mjs +46 -0
- package/packages/installer/lib/commands/doctor.mjs +7 -16
- package/packages/installer/lib/commands/install.mjs +59 -4
- package/packages/installer/lib/core/bundle.mjs +69 -92
- package/packages/installer/lib/core/config.mjs +1 -1
- package/packages/installer/lib/core/lockfile.mjs +1 -1
- package/packages/installer/lib/core/manifest.mjs +48 -54
- package/packages/installer/lib/core/validate-install.mjs +12 -19
- package/packages/installer/package.json +1 -1
- package/shared/tooling/qa-tool.mjs +125 -0
- package/skills/qa-api/SKILL.md +5 -5
- package/skills/qa-api/references/deterministic-tooling.md +34 -30
- package/skills/qa-api/references/evidence-and-reporting.md +29 -4
- package/skills/qa-api/scripts/qa-tool.mjs +126 -0
- package/skills/qa-audit/SKILL.md +5 -5
- package/skills/qa-audit/references/deterministic-tooling.md +34 -30
- package/skills/qa-audit/references/evidence-and-reporting.md +29 -4
- package/skills/qa-audit/scripts/qa-tool.mjs +126 -0
- package/skills/qa-debug/SKILL.md +6 -6
- package/skills/qa-debug/references/deterministic-tooling.md +34 -30
- package/skills/qa-debug/references/diagnostic-engine.md +3 -3
- package/skills/qa-debug/references/evidence-and-reporting.md +29 -4
- package/skills/qa-debug/scripts/qa-tool.mjs +126 -0
- package/skills/qa-explore/SKILL.md +22 -5
- package/skills/qa-explore/contracts/explore-result.schema.json +26 -2
- package/skills/qa-explore/references/deterministic-tooling.md +132 -0
- package/skills/qa-explore/references/evidence-and-reporting.md +29 -4
- package/skills/qa-explore/references/report-pipeline.md +89 -8
- package/skills/qa-explore/scripts/qa-tool.mjs +126 -0
- package/skills/qa-fix/SKILL.md +4 -4
- package/skills/qa-fix/references/deterministic-tooling.md +34 -30
- package/skills/qa-fix/references/diagnostic-engine.md +3 -3
- package/skills/qa-fix/references/evidence-and-reporting.md +29 -4
- package/skills/qa-fix/scripts/qa-tool.mjs +126 -0
- package/skills/qa-flaky/SKILL.md +4 -4
- package/skills/qa-flaky/references/deterministic-tooling.md +34 -30
- package/skills/qa-flaky/references/evidence-and-reporting.md +29 -4
- package/skills/qa-flaky/scripts/qa-tool.mjs +126 -0
- package/skills/qa-generate/references/evidence-and-reporting.md +29 -4
- package/skills/qa-init/SKILL.md +4 -4
- package/skills/qa-init/references/deterministic-tooling.md +34 -30
- package/skills/qa-init/references/evidence-and-reporting.md +29 -4
- package/skills/qa-init/scripts/qa-tool.mjs +126 -0
- package/skills/qa-report/SKILL.md +8 -7
- package/skills/qa-report/references/deterministic-tooling.md +34 -30
- package/skills/qa-report/references/diagnostic-engine.md +3 -3
- package/skills/qa-report/references/evidence-and-reporting.md +29 -4
- package/skills/qa-report/scripts/qa-tool.mjs +126 -0
- package/skills/qa-review/references/evidence-and-reporting.md +29 -4
- package/skills/qa-run/SKILL.md +6 -6
- package/skills/qa-run/references/deterministic-tooling.md +34 -30
- package/skills/qa-run/references/evidence-and-reporting.md +29 -4
- package/skills/qa-run/scripts/qa-tool.mjs +126 -0
- package/shared/analysis/lib/qa_analysis/__init__.py +0 -11
- package/shared/analysis/lib/qa_analysis/branding.py +0 -175
- package/shared/analysis/lib/qa_analysis/cli.py +0 -129
- package/shared/analysis/lib/qa_analysis/context.py +0 -233
- package/shared/analysis/lib/qa_analysis/contracts.py +0 -158
- package/shared/analysis/lib/qa_analysis/diff_guard.py +0 -327
- package/shared/analysis/lib/qa_analysis/discovery.py +0 -113
- package/shared/analysis/lib/qa_analysis/evidence.py +0 -128
- package/shared/analysis/lib/qa_analysis/har.py +0 -58
- package/shared/analysis/lib/qa_analysis/junit.py +0 -80
- package/shared/analysis/lib/qa_analysis/redaction.py +0 -99
- package/shared/analysis/lib/qa_analysis/taxonomy.py +0 -98
- package/shared/diagnostics/lib/qa_diagnostics/__init__.py +0 -13
- package/shared/diagnostics/lib/qa_diagnostics/cli.py +0 -124
- package/shared/diagnostics/lib/qa_diagnostics/engine.py +0 -143
- package/shared/diagnostics/lib/qa_diagnostics/internal_contracts.py +0 -75
- package/shared/diagnostics/lib/qa_diagnostics/prioritization.py +0 -101
- package/shared/diagnostics/lib/qa_diagnostics/repair.py +0 -72
- package/shared/diagnostics/lib/qa_diagnostics/root_cause.py +0 -89
- package/shared/diagnostics/lib/qa_diagnostics/timeline.py +0 -71
- package/shared/frameworks/cypress/lib/cypress_analysis.py +0 -27
- package/shared/frameworks/playwright/lib/playwright_analysis.py +0 -167
- package/shared/frameworks/selenium/lib/selenium_analysis.py +0 -28
- package/shared/frameworks/webdriverio/lib/webdriverio_analysis.py +0 -24
- package/shared/tooling/qa_tool.py +0 -127
- /package/{shared/analysis/lib/qa_analysis → packages/engine/lib/analysis}/branding.json +0 -0
- /package/{shared → packages/engine/lib}/analysis/schemas/context.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/analysis-result.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/diagnosis.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/execution-result-min.schema.json +0 -0
|
@@ -11,6 +11,55 @@ For run id `R`:
|
|
|
11
11
|
- `qa-artifacts/explore-R/explore-result.json`
|
|
12
12
|
- `qa-artifacts/explore-R/screenshots/…`
|
|
13
13
|
|
|
14
|
+
## Write for a reader who has never seen the product
|
|
15
|
+
|
|
16
|
+
The report is forwarded. A founder, a designer, and the developer who owns one
|
|
17
|
+
finding will each open it without the conversation that produced it, and often
|
|
18
|
+
without knowing what the feature does. Write for that reader:
|
|
19
|
+
|
|
20
|
+
- **Name things, do not point at them.** "the Sign in button on /login", not "the
|
|
21
|
+
button". A reader cannot see your screen.
|
|
22
|
+
- **No unexplained jargon.** IDOR, CLS, and *soft assertion* need a clause of
|
|
23
|
+
explanation or a plainer word.
|
|
24
|
+
- **State the consequence.** A reader who does not know the product cannot infer
|
|
25
|
+
why a duplicate request matters; say that it can trip rate limiting and charge
|
|
26
|
+
twice.
|
|
27
|
+
|
|
28
|
+
The renderer supplies the standing furniture — what an exploratory QA pass is, what
|
|
29
|
+
each dimension means, what the severity labels mean, and what could not be run. You
|
|
30
|
+
supply what only this run knows: the `scope` block below, and findings whose
|
|
31
|
+
`actual` and `expected` read as complete sentences.
|
|
32
|
+
|
|
33
|
+
## Scope: what you did, and what you left alone
|
|
34
|
+
|
|
35
|
+
Fill `scope` on every run. It is what makes the report legible to someone who was
|
|
36
|
+
not there, and without it the report opens on a wall of findings.
|
|
37
|
+
|
|
38
|
+
| Field | What goes in it |
|
|
39
|
+
| --- | --- |
|
|
40
|
+
| `objective` | One or two sentences: which feature you exercised and what you were trying to establish. Written for someone who does not know the app. |
|
|
41
|
+
| `covered` | The concrete things you actually touched — screens, controls, flows, request paths. Not dimension names: the renderer already explains those. |
|
|
42
|
+
| `notCovered` | Every boundary of the run, **each with its reason**. A boundary without a reason reads as an omission. |
|
|
43
|
+
|
|
44
|
+
`notCovered` is the field that protects the reader. A login report that does not say
|
|
45
|
+
"signing in successfully was never tested, because a QA run must not enter real
|
|
46
|
+
credentials" invites the conclusion that login works. State the boundary:
|
|
47
|
+
|
|
48
|
+
- what needed access or credentials you did not have, and why you did not ask
|
|
49
|
+
- what a local or development environment cannot tell you (security headers, bundle
|
|
50
|
+
size, real latency) and where it must be re-checked
|
|
51
|
+
- what would have caused damage to test (rate limits, lockout, destructive actions)
|
|
52
|
+
- what was in scope but blocked, naming the case id
|
|
53
|
+
|
|
54
|
+
Blocked cases and unrun dimensions are added by the renderer from the artifact — do
|
|
55
|
+
not repeat them. Write the boundaries only it cannot know.
|
|
56
|
+
|
|
57
|
+
## Order of production
|
|
58
|
+
|
|
59
|
+
`explore-result.json` is written **first** and validated; the Markdown and HTML are
|
|
60
|
+
renderings of it. Producing the prose first and the JSON afterwards is how the two
|
|
61
|
+
drift, and the reader gets the weaker one.
|
|
62
|
+
|
|
14
63
|
## Markdown structure
|
|
15
64
|
|
|
16
65
|
```markdown
|
|
@@ -44,19 +93,51 @@ For run id `R`:
|
|
|
44
93
|
|
|
45
94
|
Keep branding **neutral** — no org-specific colors or hosting assumptions. Local files are the product; optional publish is the user's choice.
|
|
46
95
|
|
|
47
|
-
##
|
|
96
|
+
## JSON
|
|
48
97
|
|
|
49
|
-
|
|
98
|
+
Write `explore-result.json` matching `contracts/explore-result.schema.json`. Validate before completion:
|
|
50
99
|
|
|
51
|
-
|
|
52
|
-
-
|
|
53
|
-
-
|
|
100
|
+
```bash
|
|
101
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis validate \
|
|
102
|
+
qa-artifacts/explore-R/explore-result.json \
|
|
103
|
+
<SKILL_DIR>/contracts/explore-result.schema.json
|
|
104
|
+
```
|
|
54
105
|
|
|
55
|
-
|
|
106
|
+
## HTML
|
|
56
107
|
|
|
57
|
-
|
|
108
|
+
Render it from the validated JSON. Do not write it by hand and do not write a
|
|
109
|
+
throwaway md→html script:
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis report-html \
|
|
113
|
+
qa-artifacts/explore-R/explore-result.json \
|
|
114
|
+
--out qa-artifacts/explore-R/explore-report.html
|
|
115
|
+
```
|
|
58
116
|
|
|
59
|
-
|
|
117
|
+
The renderer produces one self-contained file — no CDN, no external stylesheet, no
|
|
118
|
+
web font — with, for every finding: severity, the defect, **current behaviour**,
|
|
119
|
+
**expected behaviour**, reproduction, fix direction, and each evidence entry
|
|
120
|
+
(screenshots as `<img>`, everything else as a captioned excerpt). Severity ordering,
|
|
121
|
+
the summary counts, the test-case table, data-validation comparisons, fix order, and
|
|
122
|
+
the attribution footer come with it.
|
|
123
|
+
|
|
124
|
+
So the HTML is complete only insofar as the JSON is. A finding whose `expected` says
|
|
125
|
+
"should work" renders a card that says "should work". Write the fields for a reader
|
|
126
|
+
who has never seen the app:
|
|
127
|
+
|
|
128
|
+
| Field | Not this | This |
|
|
129
|
+
| --- | --- | --- |
|
|
130
|
+
| `actual` | "Login fails" | "Two identical `POST /graphql` requests are sent; the second returns 401 and the form clears" |
|
|
131
|
+
| `expected` | "Should not fail" | "One request per submit; the button is disabled while in flight" |
|
|
132
|
+
| `repro` | "Double-click login" | "1. Open /login 2. Enter valid credentials 3. Double-click **Sign in** within 300 ms 4. Watch the Network panel" |
|
|
133
|
+
| `fixDirection` | "Fix the handler" | "Disable the submit button on the first click and re-enable it in the request's finally block" |
|
|
134
|
+
|
|
135
|
+
Screenshots are referenced as paths relative to the HTML file (`screenshots/finding-01.png`),
|
|
136
|
+
so the run folder stays portable as a whole.
|
|
137
|
+
|
|
138
|
+
For an evidence excerpt that could carry a token or credential, run it through
|
|
139
|
+
`qa-tool.mjs analysis redact` before it goes into the JSON — the renderer escapes
|
|
140
|
+
markup, it does not redact secrets.
|
|
60
141
|
|
|
61
142
|
## Versioning
|
|
62
143
|
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
// synced-from: shared/tooling/qa-tool.mjs — do not edit; edit the source and run: node scripts/sync-shared.mjs --write
|
|
2
|
+
// The launcher every skill invokes to reach the deterministic engine.
|
|
3
|
+
//
|
|
4
|
+
// A skill's SKILL.md documents exactly one command shape:
|
|
5
|
+
//
|
|
6
|
+
// node <SKILL_DIR>/scripts/qa-tool.mjs <tool> <subcommand> [args]
|
|
7
|
+
//
|
|
8
|
+
// and this file finds the engine, wherever it happens to be. That indirection
|
|
9
|
+
// exists because the pack is installed three different ways and the engine lands in
|
|
10
|
+
// a different place each time:
|
|
11
|
+
//
|
|
12
|
+
// 1. `qa install` bundles the engine into the skill, at ./lib/. Offline, fastest,
|
|
13
|
+
// and pinned to the version that was installed.
|
|
14
|
+
// 2. `npx skills add <owner>/<repo>` — or any generic file copier — copies the
|
|
15
|
+
// skill directory out of git and bundles nothing. The engine is then resolved
|
|
16
|
+
// from node_modules if the project happens to depend on the pack.
|
|
17
|
+
// 3. Neither: fall back to `npx qa-engineer`, which fetches the published package
|
|
18
|
+
// on first use and is served from the npm cache afterwards.
|
|
19
|
+
//
|
|
20
|
+
// Because this file is committed rather than generated, path 2 works at all — which
|
|
21
|
+
// is what makes the pack installable by the wider Agent Skills ecosystem. The
|
|
22
|
+
// command a skill runs never changes; only where the engine came from does.
|
|
23
|
+
//
|
|
24
|
+
// Exit codes pass through unchanged: 0 success, 1 an invalid contract, 2 unreadable
|
|
25
|
+
// input or bad usage.
|
|
26
|
+
//
|
|
27
|
+
// No shebang: the synced copies carry a provenance marker on line one, which would
|
|
28
|
+
// sit above it and stop the kernel seeing it anyway. Every documented invocation is
|
|
29
|
+
// `node qa-tool.mjs …`, which needs none.
|
|
30
|
+
|
|
31
|
+
import fs from 'node:fs';
|
|
32
|
+
import path from 'node:path';
|
|
33
|
+
import { spawnSync } from 'node:child_process';
|
|
34
|
+
import { fileURLToPath } from 'node:url';
|
|
35
|
+
|
|
36
|
+
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
37
|
+
|
|
38
|
+
const USAGE = `usage: node qa-tool.mjs <tool> <subcommand> [args]
|
|
39
|
+
|
|
40
|
+
analysis parse artifacts, classify errors, validate contracts, diff-guard,
|
|
41
|
+
read .qa/context.md, render an HTML report, print the footer
|
|
42
|
+
diagnostics root cause, timeline, priority, repair plans, release readiness
|
|
43
|
+
playwright normalize a Playwright report or summarize a trace
|
|
44
|
+
|
|
45
|
+
--where print how the engine was resolved, and stop
|
|
46
|
+
|
|
47
|
+
examples:
|
|
48
|
+
node qa-tool.mjs analysis junit test-results/results.xml
|
|
49
|
+
node qa-tool.mjs analysis report-html qa-artifacts/explore-result.json --out report.html
|
|
50
|
+
node qa-tool.mjs diagnostics report --execution-result qa-artifacts/run.json
|
|
51
|
+
`;
|
|
52
|
+
|
|
53
|
+
/**
|
|
54
|
+
* Where the engine is, and how we found it.
|
|
55
|
+
*
|
|
56
|
+
* Ordered by cost: a bundled copy needs no resolution and no network, a
|
|
57
|
+
* node_modules copy needs no network, and npx needs both on first use. Reporting
|
|
58
|
+
* *which* one answered matters when a skill degrades — "the engine is missing" and
|
|
59
|
+
* "the engine is being fetched" are different problems.
|
|
60
|
+
*/
|
|
61
|
+
function resolveEngine() {
|
|
62
|
+
const bundled = path.join(here, 'lib', 'bin', 'qa-engine.mjs');
|
|
63
|
+
if (fs.existsSync(bundled)) return { kind: 'bundled', command: [process.execPath, bundled] };
|
|
64
|
+
|
|
65
|
+
for (const base of candidateModuleRoots()) {
|
|
66
|
+
const installed = path.join(base, 'qa-engineer', 'packages', 'engine', 'bin', 'qa-engine.mjs');
|
|
67
|
+
if (fs.existsSync(installed)) {
|
|
68
|
+
return { kind: 'node_modules', command: [process.execPath, installed] };
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
return {
|
|
73
|
+
kind: 'npx',
|
|
74
|
+
command: [npxCommand(), '--yes', 'qa-engineer', 'engine'],
|
|
75
|
+
};
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
/** node_modules directories worth checking, nearest first. */
|
|
79
|
+
function candidateModuleRoots() {
|
|
80
|
+
const roots = [];
|
|
81
|
+
let dir = here;
|
|
82
|
+
for (let depth = 0; depth < 12; depth += 1) {
|
|
83
|
+
roots.push(path.join(dir, 'node_modules'));
|
|
84
|
+
const parent = path.dirname(dir);
|
|
85
|
+
if (parent === dir) break;
|
|
86
|
+
dir = parent;
|
|
87
|
+
}
|
|
88
|
+
if (process.cwd() !== here) roots.push(path.join(process.cwd(), 'node_modules'));
|
|
89
|
+
return roots;
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// `npx` is a shell script on POSIX and a .cmd shim on Windows; spawnSync needs the
|
|
93
|
+
// exact name, and `shell: true` would put user-supplied arguments through a shell.
|
|
94
|
+
function npxCommand() {
|
|
95
|
+
return process.platform === 'win32' ? 'npx.cmd' : 'npx';
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
function main(argv) {
|
|
99
|
+
if (argv.length === 0 || argv[0] === '--help' || argv[0] === '-h') {
|
|
100
|
+
process.stdout.write(USAGE);
|
|
101
|
+
return argv.length === 0 ? 2 : 0;
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
const engine = resolveEngine();
|
|
105
|
+
|
|
106
|
+
if (argv[0] === '--where') {
|
|
107
|
+
process.stdout.write(`${JSON.stringify({ resolved: engine.kind, command: engine.command }, null, 2)}\n`);
|
|
108
|
+
return 0;
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
const [program, ...prefix] = engine.command;
|
|
112
|
+
const run = spawnSync(program, [...prefix, ...argv], { stdio: 'inherit' });
|
|
113
|
+
|
|
114
|
+
if (run.error) {
|
|
115
|
+
// Say which path was tried and what to do, because a skill's fallback prose
|
|
116
|
+
// cannot diagnose this and the user is the one who has to fix it.
|
|
117
|
+
const advice = engine.kind === 'npx'
|
|
118
|
+
? 'the engine is not bundled and npx is unavailable — run `npx qa-engineer install` in this project, or install Node 18+'
|
|
119
|
+
: `could not execute ${program}`;
|
|
120
|
+
process.stderr.write(`qa-tool: ${advice}\n${run.error.message}\n`);
|
|
121
|
+
return 2;
|
|
122
|
+
}
|
|
123
|
+
return run.status ?? 2;
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
process.exitCode = main(process.argv.slice(2));
|
package/skills/qa-fix/SKILL.md
CHANGED
|
@@ -54,14 +54,14 @@ This milestone plans repairs; it does not write or apply code. It never edits a
|
|
|
54
54
|
|
|
55
55
|
## Tooling
|
|
56
56
|
|
|
57
|
-
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-fix` or `.claude/skills/qa-fix`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe
|
|
57
|
+
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-fix` or `.claude/skills/qa-fix`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe, and it runs under the same Node that installed the pack — there is no second runtime to find.
|
|
58
58
|
|
|
59
59
|
| Tool | Invocation | Output | Fallback |
|
|
60
60
|
| --- | --- | --- | --- |
|
|
61
|
-
| Repair planner | `
|
|
62
|
-
| Diff guard | `
|
|
61
|
+
| Repair planner | `node <SKILL_DIR>/scripts/qa-tool.mjs diagnostics plan-repairs --diagnosis <path>` | A repair plan per diagnosis entry, escalations included | Reason over the repair-strategy module manually and mark the plan degraded |
|
|
62
|
+
| Diff guard | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis diff-guard <diff-file>` | `{issues, safe}` — `safe:false` means the change is unsafe | None: without the guard, record `diffGuardReview.status` as `not-run` and never claim a diff is safe |
|
|
63
63
|
|
|
64
|
-
A missing `
|
|
64
|
+
A missing `qa-tool.mjs` means the engine is not installed.
|
|
65
65
|
|
|
66
66
|
**The diff guard is not advisory.** Whenever a diff exists — drafted here or supplied by the user — run it through the guard and record the verdict in `diffGuardReview`. A `fail` verdict forbids the disposition `repairable`; the contract rejects that combination, so escalate instead. A plan that carries no diff records `not-run` and claims nothing about safety.
|
|
67
67
|
|
|
@@ -17,7 +17,7 @@ resolves its own location. There is nothing to set up and no shell features are
|
|
|
17
17
|
involved:
|
|
18
18
|
|
|
19
19
|
```bash
|
|
20
|
-
|
|
20
|
+
node <skill-dir>/scripts/qa-tool.mjs <tool> <subcommand> [args]
|
|
21
21
|
```
|
|
22
22
|
|
|
23
23
|
`<skill-dir>` is wherever the host installed this skill — usually
|
|
@@ -25,19 +25,21 @@ python3 <skill-dir>/scripts/qa_tool.py <tool> <subcommand> [args]
|
|
|
25
25
|
whichever exists.
|
|
26
26
|
|
|
27
27
|
```bash
|
|
28
|
-
|
|
28
|
+
node .agents/skills/qa-run/scripts/qa-tool.mjs analysis junit test-results/results.xml
|
|
29
29
|
```
|
|
30
30
|
|
|
31
|
-
That line is identical in bash, zsh, PowerShell, and cmd.exe
|
|
32
|
-
|
|
31
|
+
That line is identical in bash, zsh, PowerShell, and cmd.exe, with no platform
|
|
32
|
+
difference at all: it runs under the same Node the user already had to install the
|
|
33
|
+
pack.
|
|
33
34
|
|
|
34
|
-
|
|
35
|
-
(`QA_LIB="$(ls -d … | head -1)"` with a `PYTHONPATH=` prefix)
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
Never reintroduce a shell-dependent
|
|
35
|
+
Two earlier versions of this contract failed the same way. The first was a shell
|
|
36
|
+
recipe (`QA_LIB="$(ls -d … | head -1)"` with a `PYTHONPATH=` prefix), POSIX-only, so
|
|
37
|
+
on Windows every deterministic call failed and each skill fell back to guesswork
|
|
38
|
+
while appearing to have run its tooling. The second was portable but needed a Python
|
|
39
|
+
interpreter the user never agreed to install. Never reintroduce a shell-dependent
|
|
40
|
+
invocation, and never add a runtime the install did not already require.
|
|
39
41
|
|
|
40
|
-
If `
|
|
42
|
+
If `qa-tool.mjs` is missing, the engine is not installed: say so, recommend
|
|
41
43
|
`qa repair`, use the skill's documented fallback, and mark the result degraded.
|
|
42
44
|
|
|
43
45
|
Every tool writes JSON to stdout. Exit `0` means success; exit `1` means an
|
|
@@ -45,33 +47,35 @@ invalid contract; exit `2` means unreadable input, a malformed artifact, or a
|
|
|
45
47
|
payload that failed its seam contract, and the JSON body carries `error` and
|
|
46
48
|
`detail`. Treat a non-zero exit as missing evidence, never as a value to guess.
|
|
47
49
|
|
|
48
|
-
|
|
50
|
+
Dependency-free Node — nothing to install beyond the Node that ran `npx`.
|
|
49
51
|
|
|
50
|
-
## 2. Analysis core — `
|
|
52
|
+
## 2. Analysis core — `qa-tool.mjs analysis`
|
|
51
53
|
|
|
52
54
|
Framework-agnostic parsing, redaction, and validation.
|
|
53
55
|
|
|
54
56
|
| Subcommand | Invocation | Returns |
|
|
55
57
|
| --- | --- | --- |
|
|
56
|
-
| `junit` | `
|
|
57
|
-
| `har` | `
|
|
58
|
-
| `discover` | `
|
|
59
|
-
| `diff-guard` | `
|
|
60
|
-
| `redact` | `
|
|
61
|
-
| `validate` | `
|
|
62
|
-
| `classify` | `
|
|
63
|
-
| `context` | `
|
|
64
|
-
|
|
65
|
-
|
|
58
|
+
| `junit` | `node <skill-dir>/scripts/qa-tool.mjs analysis junit <report.xml>` | `{tests: {...}, executed: [...]}` normalized counts and per-test outcomes |
|
|
59
|
+
| `har` | `node <skill-dir>/scripts/qa-tool.mjs analysis har <file.har> [--slow-ms N]` | Redacted request/response summary, failures, slow calls |
|
|
60
|
+
| `discover` | `node <skill-dir>/scripts/qa-tool.mjs analysis discover [--root DIR] [--path P]` | Artifacts found, by type, with presence flags |
|
|
61
|
+
| `diff-guard` | `node <skill-dir>/scripts/qa-tool.mjs analysis diff-guard <diff-file>` | `{issues: [...], safe: bool}` — `safe:false` blocks the change |
|
|
62
|
+
| `redact` | `node <skill-dir>/scripts/qa-tool.mjs analysis redact <file>` | The file's text with credentials masked |
|
|
63
|
+
| `validate` | `node <skill-dir>/scripts/qa-tool.mjs analysis validate <instance.json> <schema.json>` | `{valid: bool, errors: [...]}`; exit 1 when invalid |
|
|
64
|
+
| `classify` | `node <skill-dir>/scripts/qa-tool.mjs analysis classify "<error message>" [--http-status N]` | `{classification, confidence, reason}` from the shared taxonomy |
|
|
65
|
+
| `context` | `node <skill-dir>/scripts/qa-tool.mjs analysis context [--root DIR] [--path .qa/context.md]` | The parsed, schema-validated project context as JSON |
|
|
66
|
+
| `report-html` | `node <skill-dir>/scripts/qa-tool.mjs analysis report-html <result.json> [--out report.html]` | The result rendered as one self-contained HTML report — every required field, footer included. Run it instead of writing HTML |
|
|
67
|
+
| `branding` | `node <skill-dir>/scripts/qa-tool.mjs analysis branding --format markdown\|html\|text` | The exact attribution footer bytes for a **rendered** report (never for a JSON artifact) |
|
|
68
|
+
|
|
69
|
+
## 3. Diagnostic engine — `qa-tool.mjs diagnostics`
|
|
66
70
|
|
|
67
71
|
One engine, consumed by the diagnostic skills. Reasoning lives here once.
|
|
68
72
|
|
|
69
73
|
| Subcommand | Invocation | Returns |
|
|
70
74
|
| --- | --- | --- |
|
|
71
|
-
| `diagnose` | `
|
|
72
|
-
| `plan-repairs` | `
|
|
73
|
-
| `summarize` | `
|
|
74
|
-
| `report` | `
|
|
75
|
+
| `diagnose` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics diagnose --execution-result <path> [--analysis-result <path>]` | `{entries: [...], timeline: [...], recommendations: [...]}` |
|
|
76
|
+
| `plan-repairs` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics plan-repairs --diagnosis <path>` | `{plans: [...]}` — one plan per entry, escalations included |
|
|
77
|
+
| `summarize` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics summarize --execution-result <path> --diagnosis <path>` | `{totals, byClassification, topPriority, releaseReadiness}` |
|
|
78
|
+
| `report` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics report --execution-result <path> [--analysis-result <path>]` | `{diagnosis, plans, summary}` — all three in one call |
|
|
75
79
|
|
|
76
80
|
**Inputs.** `--execution-result` takes a `qa-run` execution result, or the minimal
|
|
77
81
|
subset (`tests` counts plus `executed[]` entries carrying `status`).
|
|
@@ -99,7 +103,7 @@ contract names:
|
|
|
99
103
|
|
|
100
104
|
This mapping is not busywork: the strictness is what stops a skill from shipping a
|
|
101
105
|
result whose shape nobody checked. Validate before completion —
|
|
102
|
-
`
|
|
106
|
+
`node <skill-dir>/scripts/qa-tool.mjs analysis validate <result.json> <schema.json>` — and fix the
|
|
103
107
|
result, never the claim.
|
|
104
108
|
|
|
105
109
|
## 4. Framework adapters
|
|
@@ -109,11 +113,11 @@ a `--framework` flag.
|
|
|
109
113
|
|
|
110
114
|
| Adapter | Invocation | Returns |
|
|
111
115
|
| --- | --- | --- |
|
|
112
|
-
| Playwright report | `
|
|
113
|
-
| Playwright trace | `
|
|
116
|
+
| Playwright report | `node <skill-dir>/scripts/qa-tool.mjs playwright report <results.json>` | The same `{tests, executed}` shape as `junit` |
|
|
117
|
+
| Playwright trace | `node <skill-dir>/scripts/qa-tool.mjs playwright trace <trace.zip>` | Actions, console/network counts, errors, classification |
|
|
114
118
|
|
|
115
119
|
For Selenium, Cypress, and WebdriverIO, normalize through
|
|
116
|
-
`
|
|
120
|
+
`qa-tool.mjs analysis junit` — those adapters have no richer artifact than JUnit, and
|
|
117
121
|
the skill says so rather than implying trace-grade depth.
|
|
118
122
|
|
|
119
123
|
## 5. Reporting what ran
|
|
@@ -5,11 +5,11 @@ How the one engine composes the other platforms into a diagnosis, and what lives
|
|
|
5
5
|
|
|
6
6
|
## What the engine does, in code
|
|
7
7
|
|
|
8
|
-
The `
|
|
8
|
+
The `diagnostics` package (in lib/) implements the deterministic reasoning, reusing `analysis`:
|
|
9
9
|
|
|
10
10
|
| Step | Module | Reuses |
|
|
11
11
|
| --- | --- | --- |
|
|
12
|
-
| Classify the failure into a root cause | `root_cause` | `
|
|
12
|
+
| Classify the failure into a root cause | `root_cause` | `the shared taxonomy` |
|
|
13
13
|
| Assign severity, priority, impacts, owner, effort | `prioritization` | the taxonomy classes |
|
|
14
14
|
| Reconstruct the ordered timeline | `timeline` | the evidence in findings |
|
|
15
15
|
| Plan a repair (never code) | `repair` | the taxonomy; the diff guard's guarantees |
|
|
@@ -33,4 +33,4 @@ The engine calls the analysis platform for classification and evidence, the exec
|
|
|
33
33
|
|
|
34
34
|
## Bundling
|
|
35
35
|
|
|
36
|
-
Because the diagnostic skills run the engine in a consumer's repository, the `
|
|
36
|
+
Because the diagnostic skills run the engine in a consumer's repository, the `analysis` and `diagnostics` packages are bundled into each skill's `scripts/lib/` from their canonical source in `shared/`. The bundle is a build artifact produced by the bundler; the source of truth is `shared/`. See ADR-0011.
|
|
@@ -27,17 +27,42 @@ Any skill that reaches a conclusion — a detected fact, a chosen strategy, a cl
|
|
|
27
27
|
| Confidence | Calibrated per rule 4; omitted rather than invented when the skill did not weigh alternatives |
|
|
28
28
|
| Recommendations | The next action, named concretely — often the next command and the artifact to feed it |
|
|
29
29
|
|
|
30
|
+
## Rendering the HTML report
|
|
31
|
+
|
|
32
|
+
**Never type the HTML.** When a skill's result conforms to a contract the renderer
|
|
33
|
+
supports, the HTML report is generated from that artifact:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis report-html <result.json> --out <report.html>
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
The reason is not tidiness. A hand-written report is a second, lossy copy of the
|
|
40
|
+
artifact, and it loses exactly what the reader needs: the first live `/qa-explore`
|
|
41
|
+
run wrote findings whose `actual`, `expected`, and `fixDirection` were all required,
|
|
42
|
+
all present in the JSON, and all absent from the page — collapsed into one sentence
|
|
43
|
+
that left the reader to guess what correct behaviour would have been. The renderer
|
|
44
|
+
cannot make that mistake, because the fields are in its template.
|
|
45
|
+
|
|
46
|
+
What the renderer guarantees per finding: severity, the defect, **what happens
|
|
47
|
+
now**, **what should happen instead**, how to reproduce it, the fix direction, and
|
|
48
|
+
every evidence entry — plus the attribution footer, in one self-contained file with
|
|
49
|
+
no external assets.
|
|
50
|
+
|
|
51
|
+
Run `qa-tool.mjs analysis report-html --help` for the supported contracts. For an
|
|
52
|
+
artifact it does not support, write the HTML by hand from the contract's fields —
|
|
53
|
+
and render **every** field a reader needs, in that order.
|
|
54
|
+
|
|
30
55
|
## Attribution on rendered reports
|
|
31
56
|
|
|
32
57
|
A report a person opens carries a product attribution footer, the way a Lighthouse
|
|
33
58
|
or Allure report does. It is rendered, never typed: the exact bytes come from the
|
|
34
59
|
bundled analysis toolkit, so every report is identical and a wording change is a
|
|
35
|
-
one-file edit.
|
|
60
|
+
one-file edit. `report-html` already embeds it; for any other rendering, ask for it:
|
|
36
61
|
|
|
37
62
|
```bash
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
63
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis branding --format markdown
|
|
64
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis branding --format html
|
|
65
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis branding --format text
|
|
41
66
|
```
|
|
42
67
|
|
|
43
68
|
Append the output as the last element of the rendered document: `html` inside
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
// synced-from: shared/tooling/qa-tool.mjs — do not edit; edit the source and run: node scripts/sync-shared.mjs --write
|
|
2
|
+
// The launcher every skill invokes to reach the deterministic engine.
|
|
3
|
+
//
|
|
4
|
+
// A skill's SKILL.md documents exactly one command shape:
|
|
5
|
+
//
|
|
6
|
+
// node <SKILL_DIR>/scripts/qa-tool.mjs <tool> <subcommand> [args]
|
|
7
|
+
//
|
|
8
|
+
// and this file finds the engine, wherever it happens to be. That indirection
|
|
9
|
+
// exists because the pack is installed three different ways and the engine lands in
|
|
10
|
+
// a different place each time:
|
|
11
|
+
//
|
|
12
|
+
// 1. `qa install` bundles the engine into the skill, at ./lib/. Offline, fastest,
|
|
13
|
+
// and pinned to the version that was installed.
|
|
14
|
+
// 2. `npx skills add <owner>/<repo>` — or any generic file copier — copies the
|
|
15
|
+
// skill directory out of git and bundles nothing. The engine is then resolved
|
|
16
|
+
// from node_modules if the project happens to depend on the pack.
|
|
17
|
+
// 3. Neither: fall back to `npx qa-engineer`, which fetches the published package
|
|
18
|
+
// on first use and is served from the npm cache afterwards.
|
|
19
|
+
//
|
|
20
|
+
// Because this file is committed rather than generated, path 2 works at all — which
|
|
21
|
+
// is what makes the pack installable by the wider Agent Skills ecosystem. The
|
|
22
|
+
// command a skill runs never changes; only where the engine came from does.
|
|
23
|
+
//
|
|
24
|
+
// Exit codes pass through unchanged: 0 success, 1 an invalid contract, 2 unreadable
|
|
25
|
+
// input or bad usage.
|
|
26
|
+
//
|
|
27
|
+
// No shebang: the synced copies carry a provenance marker on line one, which would
|
|
28
|
+
// sit above it and stop the kernel seeing it anyway. Every documented invocation is
|
|
29
|
+
// `node qa-tool.mjs …`, which needs none.
|
|
30
|
+
|
|
31
|
+
import fs from 'node:fs';
|
|
32
|
+
import path from 'node:path';
|
|
33
|
+
import { spawnSync } from 'node:child_process';
|
|
34
|
+
import { fileURLToPath } from 'node:url';
|
|
35
|
+
|
|
36
|
+
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
37
|
+
|
|
38
|
+
const USAGE = `usage: node qa-tool.mjs <tool> <subcommand> [args]
|
|
39
|
+
|
|
40
|
+
analysis parse artifacts, classify errors, validate contracts, diff-guard,
|
|
41
|
+
read .qa/context.md, render an HTML report, print the footer
|
|
42
|
+
diagnostics root cause, timeline, priority, repair plans, release readiness
|
|
43
|
+
playwright normalize a Playwright report or summarize a trace
|
|
44
|
+
|
|
45
|
+
--where print how the engine was resolved, and stop
|
|
46
|
+
|
|
47
|
+
examples:
|
|
48
|
+
node qa-tool.mjs analysis junit test-results/results.xml
|
|
49
|
+
node qa-tool.mjs analysis report-html qa-artifacts/explore-result.json --out report.html
|
|
50
|
+
node qa-tool.mjs diagnostics report --execution-result qa-artifacts/run.json
|
|
51
|
+
`;
|
|
52
|
+
|
|
53
|
+
/**
|
|
54
|
+
* Where the engine is, and how we found it.
|
|
55
|
+
*
|
|
56
|
+
* Ordered by cost: a bundled copy needs no resolution and no network, a
|
|
57
|
+
* node_modules copy needs no network, and npx needs both on first use. Reporting
|
|
58
|
+
* *which* one answered matters when a skill degrades — "the engine is missing" and
|
|
59
|
+
* "the engine is being fetched" are different problems.
|
|
60
|
+
*/
|
|
61
|
+
function resolveEngine() {
|
|
62
|
+
const bundled = path.join(here, 'lib', 'bin', 'qa-engine.mjs');
|
|
63
|
+
if (fs.existsSync(bundled)) return { kind: 'bundled', command: [process.execPath, bundled] };
|
|
64
|
+
|
|
65
|
+
for (const base of candidateModuleRoots()) {
|
|
66
|
+
const installed = path.join(base, 'qa-engineer', 'packages', 'engine', 'bin', 'qa-engine.mjs');
|
|
67
|
+
if (fs.existsSync(installed)) {
|
|
68
|
+
return { kind: 'node_modules', command: [process.execPath, installed] };
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
return {
|
|
73
|
+
kind: 'npx',
|
|
74
|
+
command: [npxCommand(), '--yes', 'qa-engineer', 'engine'],
|
|
75
|
+
};
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
/** node_modules directories worth checking, nearest first. */
|
|
79
|
+
function candidateModuleRoots() {
|
|
80
|
+
const roots = [];
|
|
81
|
+
let dir = here;
|
|
82
|
+
for (let depth = 0; depth < 12; depth += 1) {
|
|
83
|
+
roots.push(path.join(dir, 'node_modules'));
|
|
84
|
+
const parent = path.dirname(dir);
|
|
85
|
+
if (parent === dir) break;
|
|
86
|
+
dir = parent;
|
|
87
|
+
}
|
|
88
|
+
if (process.cwd() !== here) roots.push(path.join(process.cwd(), 'node_modules'));
|
|
89
|
+
return roots;
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// `npx` is a shell script on POSIX and a .cmd shim on Windows; spawnSync needs the
|
|
93
|
+
// exact name, and `shell: true` would put user-supplied arguments through a shell.
|
|
94
|
+
function npxCommand() {
|
|
95
|
+
return process.platform === 'win32' ? 'npx.cmd' : 'npx';
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
function main(argv) {
|
|
99
|
+
if (argv.length === 0 || argv[0] === '--help' || argv[0] === '-h') {
|
|
100
|
+
process.stdout.write(USAGE);
|
|
101
|
+
return argv.length === 0 ? 2 : 0;
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
const engine = resolveEngine();
|
|
105
|
+
|
|
106
|
+
if (argv[0] === '--where') {
|
|
107
|
+
process.stdout.write(`${JSON.stringify({ resolved: engine.kind, command: engine.command }, null, 2)}\n`);
|
|
108
|
+
return 0;
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
const [program, ...prefix] = engine.command;
|
|
112
|
+
const run = spawnSync(program, [...prefix, ...argv], { stdio: 'inherit' });
|
|
113
|
+
|
|
114
|
+
if (run.error) {
|
|
115
|
+
// Say which path was tried and what to do, because a skill's fallback prose
|
|
116
|
+
// cannot diagnose this and the user is the one who has to fix it.
|
|
117
|
+
const advice = engine.kind === 'npx'
|
|
118
|
+
? 'the engine is not bundled and npx is unavailable — run `npx qa-engineer install` in this project, or install Node 18+'
|
|
119
|
+
: `could not execute ${program}`;
|
|
120
|
+
process.stderr.write(`qa-tool: ${advice}\n${run.error.message}\n`);
|
|
121
|
+
return 2;
|
|
122
|
+
}
|
|
123
|
+
return run.status ?? 2;
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
process.exitCode = main(process.argv.slice(2));
|
package/skills/qa-flaky/SKILL.md
CHANGED
|
@@ -53,14 +53,14 @@ Do not use it to run tests (`/qa-run`) or to fix a deterministic failure (`/qa-d
|
|
|
53
53
|
|
|
54
54
|
## Tooling
|
|
55
55
|
|
|
56
|
-
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-flaky` or `.claude/skills/qa-flaky`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe
|
|
56
|
+
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-flaky` or `.claude/skills/qa-flaky`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe, and it runs under the same Node that installed the pack — there is no second runtime to find.
|
|
57
57
|
|
|
58
58
|
| Tool | Invocation | Output | Fallback |
|
|
59
59
|
| --- | --- | --- | --- |
|
|
60
|
-
| Diagnostic engine | `
|
|
61
|
-
| JUnit normalizer | `
|
|
60
|
+
| Diagnostic engine | `node <SKILL_DIR>/scripts/qa-tool.mjs diagnostics diagnose --execution-result <path>` | Deterministic flaky classification and prioritized causes (retry counts drive the `flaky` classification) | Reason over the flakiness module manually and mark the verdict degraded |
|
|
61
|
+
| JUnit normalizer | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis junit <report.xml>` | Per-test retries and final status across runs — the input the flake rate is computed from | Read the reporter and state that the flake rate is unmeasured |
|
|
62
62
|
|
|
63
|
-
A missing `
|
|
63
|
+
A missing `qa-tool.mjs` means the engine is not installed. A flake rate is computed from observed runs or reported as `null` — never estimated.
|
|
64
64
|
|
|
65
65
|
## Output
|
|
66
66
|
|