qa-engineer 0.9.2 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/COMPATIBILITY.md +26 -12
- package/README.md +182 -84
- package/package.json +11 -20
- package/packages/engine/bin/qa-engine.mjs +560 -0
- package/packages/engine/lib/analysis/branding.mjs +187 -0
- package/packages/engine/lib/analysis/context.mjs +294 -0
- package/packages/engine/lib/analysis/contracts.mjs +149 -0
- package/packages/engine/lib/analysis/diff-guard.mjs +425 -0
- package/packages/engine/lib/analysis/discovery.mjs +220 -0
- package/packages/engine/lib/analysis/evidence.mjs +116 -0
- package/packages/engine/lib/analysis/har.mjs +124 -0
- package/packages/engine/lib/analysis/junit.mjs +126 -0
- package/packages/engine/lib/analysis/network.mjs +237 -0
- package/packages/engine/lib/analysis/redaction.mjs +127 -0
- package/packages/engine/lib/analysis/report-html.mjs +76 -0
- package/packages/engine/lib/analysis/taxonomy.mjs +109 -0
- package/packages/engine/lib/analysis/xml.mjs +153 -0
- package/packages/engine/lib/analysis/zip.mjs +107 -0
- package/packages/engine/lib/artifacts/manager.mjs +453 -0
- package/packages/engine/lib/artifacts/mime.mjs +109 -0
- package/packages/engine/lib/artifacts/zip-write.mjs +143 -0
- package/packages/engine/lib/diagnostics/engine.mjs +165 -0
- package/packages/engine/lib/diagnostics/internal-contracts.mjs +50 -0
- package/packages/engine/lib/diagnostics/prioritization.mjs +99 -0
- package/packages/engine/lib/diagnostics/repair.mjs +73 -0
- package/packages/engine/lib/diagnostics/root-cause.mjs +92 -0
- package/packages/engine/lib/diagnostics/timeline.mjs +101 -0
- package/packages/engine/lib/frameworks/junit-frameworks.mjs +48 -0
- package/packages/engine/lib/frameworks/playwright.mjs +158 -0
- package/packages/engine/lib/report/components/charts.mjs +424 -0
- package/packages/engine/lib/report/components/evidence.mjs +207 -0
- package/packages/engine/lib/report/components/findings.mjs +258 -0
- package/packages/engine/lib/report/components/nav.mjs +99 -0
- package/packages/engine/lib/report/components/primitives.mjs +246 -0
- package/packages/engine/lib/report/components/runtime.mjs +246 -0
- package/packages/engine/lib/report/components/timeline.mjs +65 -0
- package/packages/engine/lib/report/core/model.mjs +270 -0
- package/packages/engine/lib/report/core/normalize.mjs +226 -0
- package/packages/engine/lib/report/core/sections.mjs +978 -0
- package/packages/engine/lib/report/export/bundle.mjs +293 -0
- package/packages/engine/lib/report/export/html.mjs +183 -0
- package/packages/engine/lib/report/export/machine.mjs +290 -0
- package/packages/engine/lib/report/export/markdown.mjs +323 -0
- package/packages/engine/lib/report/schemas/qa-report.schema.json +555 -0
- package/packages/engine/lib/report/theme/css.mjs +529 -0
- package/packages/engine/lib/report/theme/tokens.mjs +137 -0
- package/packages/engine/lib/report/version.mjs +78 -0
- package/packages/engine/package.json +14 -0
- package/packages/installer/lib/agents/targets.mjs +90 -0
- package/packages/installer/lib/agents/user-level.mjs +80 -0
- package/packages/installer/lib/cli/flags.mjs +20 -2
- package/packages/installer/lib/commands/doctor.mjs +13 -20
- package/packages/installer/lib/commands/install.mjs +160 -91
- package/packages/installer/lib/commands/repair.mjs +10 -6
- package/packages/installer/lib/commands/self-test.mjs +4 -3
- package/packages/installer/lib/commands/uninstall.mjs +14 -8
- package/packages/installer/lib/commands/update.mjs +9 -4
- package/packages/installer/lib/commands/verify.mjs +13 -12
- package/packages/installer/lib/constants.mjs +13 -0
- package/packages/installer/lib/core/bundle.mjs +69 -92
- package/packages/installer/lib/core/conflict.mjs +5 -4
- package/packages/installer/lib/core/fs-safe.mjs +146 -6
- package/packages/installer/lib/core/integrity.mjs +59 -0
- package/packages/installer/lib/core/lockfile.mjs +19 -3
- package/packages/installer/lib/core/manifest.mjs +48 -57
- package/packages/installer/lib/core/plan.mjs +213 -0
- package/packages/installer/lib/core/qa-home.mjs +145 -0
- package/packages/installer/lib/core/scope.mjs +274 -0
- package/packages/installer/lib/core/validate-install.mjs +49 -31
- package/packages/installer/package.json +1 -1
- package/packages/installer/schemas/qa-lock.schema.json +119 -21
- package/shared/tooling/qa-tool.mjs +161 -0
- package/skills/qa-api/SKILL.md +5 -5
- package/skills/qa-api/references/deterministic-tooling.md +34 -32
- package/skills/qa-api/references/evidence-and-reporting.md +5 -5
- package/skills/qa-api/scripts/qa-tool.mjs +162 -0
- package/skills/qa-audit/SKILL.md +5 -5
- package/skills/qa-audit/references/deterministic-tooling.md +34 -32
- package/skills/qa-audit/references/evidence-and-reporting.md +5 -5
- package/skills/qa-audit/scripts/qa-tool.mjs +162 -0
- package/skills/qa-debug/SKILL.md +6 -6
- package/skills/qa-debug/references/deterministic-tooling.md +34 -32
- package/skills/qa-debug/references/diagnostic-engine.md +3 -3
- package/skills/qa-debug/references/evidence-and-reporting.md +5 -5
- package/skills/qa-debug/scripts/qa-tool.mjs +162 -0
- package/skills/qa-explore/SKILL.md +31 -15
- package/skills/qa-explore/contracts/explore-result.schema.json +517 -11
- package/skills/qa-explore/references/api-replay.md +40 -1
- package/skills/qa-explore/references/deterministic-tooling.md +34 -32
- package/skills/qa-explore/references/evidence-and-reporting.md +5 -5
- package/skills/qa-explore/references/report-pipeline.md +266 -96
- package/skills/qa-explore/scripts/qa-tool.mjs +162 -0
- package/skills/qa-fix/SKILL.md +4 -4
- package/skills/qa-fix/references/deterministic-tooling.md +34 -32
- package/skills/qa-fix/references/diagnostic-engine.md +3 -3
- package/skills/qa-fix/references/evidence-and-reporting.md +5 -5
- package/skills/qa-fix/scripts/qa-tool.mjs +162 -0
- package/skills/qa-flaky/SKILL.md +4 -4
- package/skills/qa-flaky/references/deterministic-tooling.md +34 -32
- package/skills/qa-flaky/references/evidence-and-reporting.md +5 -5
- package/skills/qa-flaky/scripts/qa-tool.mjs +162 -0
- package/skills/qa-generate/references/evidence-and-reporting.md +5 -5
- package/skills/qa-init/SKILL.md +4 -4
- package/skills/qa-init/references/deterministic-tooling.md +34 -32
- package/skills/qa-init/references/evidence-and-reporting.md +5 -5
- package/skills/qa-init/scripts/qa-tool.mjs +162 -0
- package/skills/qa-report/SKILL.md +7 -7
- package/skills/qa-report/references/deterministic-tooling.md +34 -32
- package/skills/qa-report/references/diagnostic-engine.md +3 -3
- package/skills/qa-report/references/evidence-and-reporting.md +5 -5
- package/skills/qa-report/scripts/qa-tool.mjs +162 -0
- package/skills/qa-review/references/evidence-and-reporting.md +5 -5
- package/skills/qa-run/SKILL.md +6 -6
- package/skills/qa-run/references/deterministic-tooling.md +34 -32
- package/skills/qa-run/references/evidence-and-reporting.md +5 -5
- package/skills/qa-run/scripts/qa-tool.mjs +162 -0
- package/shared/analysis/lib/qa_analysis/__init__.py +0 -11
- package/shared/analysis/lib/qa_analysis/branding.py +0 -175
- package/shared/analysis/lib/qa_analysis/cli.py +0 -144
- package/shared/analysis/lib/qa_analysis/context.py +0 -233
- package/shared/analysis/lib/qa_analysis/contracts.py +0 -158
- package/shared/analysis/lib/qa_analysis/diff_guard.py +0 -327
- package/shared/analysis/lib/qa_analysis/discovery.py +0 -113
- package/shared/analysis/lib/qa_analysis/evidence.py +0 -128
- package/shared/analysis/lib/qa_analysis/har.py +0 -87
- package/shared/analysis/lib/qa_analysis/junit.py +0 -104
- package/shared/analysis/lib/qa_analysis/redaction.py +0 -107
- package/shared/analysis/lib/qa_analysis/report_html.py +0 -781
- package/shared/analysis/lib/qa_analysis/taxonomy.py +0 -98
- package/shared/diagnostics/lib/qa_diagnostics/__init__.py +0 -13
- package/shared/diagnostics/lib/qa_diagnostics/cli.py +0 -124
- package/shared/diagnostics/lib/qa_diagnostics/engine.py +0 -143
- package/shared/diagnostics/lib/qa_diagnostics/internal_contracts.py +0 -75
- package/shared/diagnostics/lib/qa_diagnostics/prioritization.py +0 -101
- package/shared/diagnostics/lib/qa_diagnostics/repair.py +0 -72
- package/shared/diagnostics/lib/qa_diagnostics/root_cause.py +0 -89
- package/shared/diagnostics/lib/qa_diagnostics/timeline.py +0 -71
- package/shared/frameworks/cypress/lib/cypress_analysis.py +0 -27
- package/shared/frameworks/playwright/lib/playwright_analysis.py +0 -167
- package/shared/frameworks/selenium/lib/selenium_analysis.py +0 -28
- package/shared/frameworks/webdriverio/lib/webdriverio_analysis.py +0 -24
- package/shared/tooling/qa_tool.py +0 -127
- /package/{shared/analysis/lib/qa_analysis → packages/engine/lib/analysis}/branding.json +0 -0
- /package/{shared → packages/engine/lib}/analysis/schemas/context.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/analysis-result.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/diagnosis.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/execution-result-min.schema.json +0 -0
|
@@ -17,7 +17,7 @@ resolves its own location. There is nothing to set up and no shell features are
|
|
|
17
17
|
involved:
|
|
18
18
|
|
|
19
19
|
```bash
|
|
20
|
-
|
|
20
|
+
node <skill-dir>/scripts/qa-tool.mjs <tool> <subcommand> [args]
|
|
21
21
|
```
|
|
22
22
|
|
|
23
23
|
`<skill-dir>` is wherever the host installed this skill — usually
|
|
@@ -25,19 +25,21 @@ python3 <skill-dir>/scripts/qa_tool.py <tool> <subcommand> [args]
|
|
|
25
25
|
whichever exists.
|
|
26
26
|
|
|
27
27
|
```bash
|
|
28
|
-
|
|
28
|
+
node .agents/skills/qa-run/scripts/qa-tool.mjs analysis junit test-results/results.xml
|
|
29
29
|
```
|
|
30
30
|
|
|
31
|
-
That line is identical in bash, zsh, PowerShell, and cmd.exe
|
|
32
|
-
|
|
31
|
+
That line is identical in bash, zsh, PowerShell, and cmd.exe, with no platform
|
|
32
|
+
difference at all: it runs under the same Node the user already had to install the
|
|
33
|
+
pack.
|
|
33
34
|
|
|
34
|
-
|
|
35
|
-
(`QA_LIB="$(ls -d … | head -1)"` with a `PYTHONPATH=` prefix)
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
Never reintroduce a shell-dependent
|
|
35
|
+
Two earlier versions of this contract failed the same way. The first was a shell
|
|
36
|
+
recipe (`QA_LIB="$(ls -d … | head -1)"` with a `PYTHONPATH=` prefix), POSIX-only, so
|
|
37
|
+
on Windows every deterministic call failed and each skill fell back to guesswork
|
|
38
|
+
while appearing to have run its tooling. The second was portable but needed a Python
|
|
39
|
+
interpreter the user never agreed to install. Never reintroduce a shell-dependent
|
|
40
|
+
invocation, and never add a runtime the install did not already require.
|
|
39
41
|
|
|
40
|
-
If `
|
|
42
|
+
If `qa-tool.mjs` is missing, the engine is not installed: say so, recommend
|
|
41
43
|
`qa repair`, use the skill's documented fallback, and mark the result degraded.
|
|
42
44
|
|
|
43
45
|
Every tool writes JSON to stdout. Exit `0` means success; exit `1` means an
|
|
@@ -45,35 +47,35 @@ invalid contract; exit `2` means unreadable input, a malformed artifact, or a
|
|
|
45
47
|
payload that failed its seam contract, and the JSON body carries `error` and
|
|
46
48
|
`detail`. Treat a non-zero exit as missing evidence, never as a value to guess.
|
|
47
49
|
|
|
48
|
-
|
|
50
|
+
Dependency-free Node — nothing to install beyond the Node that ran `npx`.
|
|
49
51
|
|
|
50
|
-
## 2. Analysis core — `
|
|
52
|
+
## 2. Analysis core — `qa-tool.mjs analysis`
|
|
51
53
|
|
|
52
54
|
Framework-agnostic parsing, redaction, and validation.
|
|
53
55
|
|
|
54
56
|
| Subcommand | Invocation | Returns |
|
|
55
57
|
| --- | --- | --- |
|
|
56
|
-
| `junit` | `
|
|
57
|
-
| `har` | `
|
|
58
|
-
| `discover` | `
|
|
59
|
-
| `diff-guard` | `
|
|
60
|
-
| `redact` | `
|
|
61
|
-
| `validate` | `
|
|
62
|
-
| `classify` | `
|
|
63
|
-
| `context` | `
|
|
64
|
-
| `report-html` | `
|
|
65
|
-
| `branding` | `
|
|
66
|
-
|
|
67
|
-
## 3. Diagnostic engine — `
|
|
58
|
+
| `junit` | `node <skill-dir>/scripts/qa-tool.mjs analysis junit <report.xml>` | `{tests: {...}, executed: [...]}` normalized counts and per-test outcomes |
|
|
59
|
+
| `har` | `node <skill-dir>/scripts/qa-tool.mjs analysis har <file.har> [--slow-ms N]` | Redacted request/response summary, failures, slow calls |
|
|
60
|
+
| `discover` | `node <skill-dir>/scripts/qa-tool.mjs analysis discover [--root DIR] [--path P]` | Artifacts found, by type, with presence flags |
|
|
61
|
+
| `diff-guard` | `node <skill-dir>/scripts/qa-tool.mjs analysis diff-guard <diff-file>` | `{issues: [...], safe: bool}` — `safe:false` blocks the change |
|
|
62
|
+
| `redact` | `node <skill-dir>/scripts/qa-tool.mjs analysis redact <file>` | The file's text with credentials masked |
|
|
63
|
+
| `validate` | `node <skill-dir>/scripts/qa-tool.mjs analysis validate <instance.json> <schema.json>` | `{valid: bool, errors: [...]}`; exit 1 when invalid |
|
|
64
|
+
| `classify` | `node <skill-dir>/scripts/qa-tool.mjs analysis classify "<error message>" [--http-status N]` | `{classification, confidence, reason}` from the shared taxonomy |
|
|
65
|
+
| `context` | `node <skill-dir>/scripts/qa-tool.mjs analysis context [--root DIR] [--path .qa/context.md]` | The parsed, schema-validated project context as JSON |
|
|
66
|
+
| `report-html` | `node <skill-dir>/scripts/qa-tool.mjs analysis report-html <result.json> [--out report.html]` | The result rendered as one self-contained HTML report — every required field, footer included. Run it instead of writing HTML |
|
|
67
|
+
| `branding` | `node <skill-dir>/scripts/qa-tool.mjs analysis branding --format markdown\|html\|text` | The exact attribution footer bytes for a **rendered** report (never for a JSON artifact) |
|
|
68
|
+
|
|
69
|
+
## 3. Diagnostic engine — `qa-tool.mjs diagnostics`
|
|
68
70
|
|
|
69
71
|
One engine, consumed by the diagnostic skills. Reasoning lives here once.
|
|
70
72
|
|
|
71
73
|
| Subcommand | Invocation | Returns |
|
|
72
74
|
| --- | --- | --- |
|
|
73
|
-
| `diagnose` | `
|
|
74
|
-
| `plan-repairs` | `
|
|
75
|
-
| `summarize` | `
|
|
76
|
-
| `report` | `
|
|
75
|
+
| `diagnose` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics diagnose --execution-result <path> [--analysis-result <path>]` | `{entries: [...], timeline: [...], recommendations: [...]}` |
|
|
76
|
+
| `plan-repairs` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics plan-repairs --diagnosis <path>` | `{plans: [...]}` — one plan per entry, escalations included |
|
|
77
|
+
| `summarize` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics summarize --execution-result <path> --diagnosis <path>` | `{totals, byClassification, topPriority, releaseReadiness}` |
|
|
78
|
+
| `report` | `node <skill-dir>/scripts/qa-tool.mjs diagnostics report --execution-result <path> [--analysis-result <path>]` | `{diagnosis, plans, summary}` — all three in one call |
|
|
77
79
|
|
|
78
80
|
**Inputs.** `--execution-result` takes a `qa-run` execution result, or the minimal
|
|
79
81
|
subset (`tests` counts plus `executed[]` entries carrying `status`).
|
|
@@ -101,7 +103,7 @@ contract names:
|
|
|
101
103
|
|
|
102
104
|
This mapping is not busywork: the strictness is what stops a skill from shipping a
|
|
103
105
|
result whose shape nobody checked. Validate before completion —
|
|
104
|
-
`
|
|
106
|
+
`node <skill-dir>/scripts/qa-tool.mjs analysis validate <result.json> <schema.json>` — and fix the
|
|
105
107
|
result, never the claim.
|
|
106
108
|
|
|
107
109
|
## 4. Framework adapters
|
|
@@ -111,11 +113,11 @@ a `--framework` flag.
|
|
|
111
113
|
|
|
112
114
|
| Adapter | Invocation | Returns |
|
|
113
115
|
| --- | --- | --- |
|
|
114
|
-
| Playwright report | `
|
|
115
|
-
| Playwright trace | `
|
|
116
|
+
| Playwright report | `node <skill-dir>/scripts/qa-tool.mjs playwright report <results.json>` | The same `{tests, executed}` shape as `junit` |
|
|
117
|
+
| Playwright trace | `node <skill-dir>/scripts/qa-tool.mjs playwright trace <trace.zip>` | Actions, console/network counts, errors, classification |
|
|
116
118
|
|
|
117
119
|
For Selenium, Cypress, and WebdriverIO, normalize through
|
|
118
|
-
`
|
|
120
|
+
`qa-tool.mjs analysis junit` — those adapters have no richer artifact than JUnit, and
|
|
119
121
|
the skill says so rather than implying trace-grade depth.
|
|
120
122
|
|
|
121
123
|
## 5. Reporting what ran
|
|
@@ -5,11 +5,11 @@ How the one engine composes the other platforms into a diagnosis, and what lives
|
|
|
5
5
|
|
|
6
6
|
## What the engine does, in code
|
|
7
7
|
|
|
8
|
-
The `
|
|
8
|
+
The `diagnostics` package (in lib/) implements the deterministic reasoning, reusing `analysis`:
|
|
9
9
|
|
|
10
10
|
| Step | Module | Reuses |
|
|
11
11
|
| --- | --- | --- |
|
|
12
|
-
| Classify the failure into a root cause | `root_cause` | `
|
|
12
|
+
| Classify the failure into a root cause | `root_cause` | `the shared taxonomy` |
|
|
13
13
|
| Assign severity, priority, impacts, owner, effort | `prioritization` | the taxonomy classes |
|
|
14
14
|
| Reconstruct the ordered timeline | `timeline` | the evidence in findings |
|
|
15
15
|
| Plan a repair (never code) | `repair` | the taxonomy; the diff guard's guarantees |
|
|
@@ -33,4 +33,4 @@ The engine calls the analysis platform for classification and evidence, the exec
|
|
|
33
33
|
|
|
34
34
|
## Bundling
|
|
35
35
|
|
|
36
|
-
Because the diagnostic skills run the engine in a consumer's repository, the `
|
|
36
|
+
Because the diagnostic skills run the engine in a consumer's repository, the `analysis` and `diagnostics` packages are bundled into each skill's `scripts/lib/` from their canonical source in `shared/`. The bundle is a build artifact produced by the bundler; the source of truth is `shared/`. See ADR-0011.
|
|
@@ -33,7 +33,7 @@ Any skill that reaches a conclusion — a detected fact, a chosen strategy, a cl
|
|
|
33
33
|
supports, the HTML report is generated from that artifact:
|
|
34
34
|
|
|
35
35
|
```bash
|
|
36
|
-
|
|
36
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis report-html <result.json> --out <report.html>
|
|
37
37
|
```
|
|
38
38
|
|
|
39
39
|
The reason is not tidiness. A hand-written report is a second, lossy copy of the
|
|
@@ -48,7 +48,7 @@ now**, **what should happen instead**, how to reproduce it, the fix direction, a
|
|
|
48
48
|
every evidence entry — plus the attribution footer, in one self-contained file with
|
|
49
49
|
no external assets.
|
|
50
50
|
|
|
51
|
-
Run `
|
|
51
|
+
Run `qa-tool.mjs analysis report-html --help` for the supported contracts. For an
|
|
52
52
|
artifact it does not support, write the HTML by hand from the contract's fields —
|
|
53
53
|
and render **every** field a reader needs, in that order.
|
|
54
54
|
|
|
@@ -60,9 +60,9 @@ bundled analysis toolkit, so every report is identical and a wording change is a
|
|
|
60
60
|
one-file edit. `report-html` already embeds it; for any other rendering, ask for it:
|
|
61
61
|
|
|
62
62
|
```bash
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
63
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis branding --format markdown
|
|
64
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis branding --format html
|
|
65
|
+
node <SKILL_DIR>/scripts/qa-tool.mjs analysis branding --format text
|
|
66
66
|
```
|
|
67
67
|
|
|
68
68
|
Append the output as the last element of the rendered document: `html` inside
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
// synced-from: shared/tooling/qa-tool.mjs — do not edit; edit the source and run: node scripts/sync-shared.mjs --write
|
|
2
|
+
// The launcher every skill invokes to reach the deterministic engine.
|
|
3
|
+
//
|
|
4
|
+
// A skill's SKILL.md documents exactly one command shape:
|
|
5
|
+
//
|
|
6
|
+
// node <SKILL_DIR>/scripts/qa-tool.mjs <tool> <subcommand> [args]
|
|
7
|
+
//
|
|
8
|
+
// and this file finds the engine, wherever it happens to be. That indirection
|
|
9
|
+
// exists because the pack is installed three different ways and the engine lands in
|
|
10
|
+
// a different place each time:
|
|
11
|
+
//
|
|
12
|
+
// 1. `qa install` bundles the engine into the skill, at ./lib/. Offline, fastest,
|
|
13
|
+
// and pinned to the version that was installed.
|
|
14
|
+
// 2. `npx skills add <owner>/<repo>` — or any generic file copier — copies the
|
|
15
|
+
// skill directory out of git and bundles nothing. The engine is then resolved
|
|
16
|
+
// from node_modules if the project happens to depend on the pack.
|
|
17
|
+
// 3. Neither: fall back to `npx qa-engineer`, which fetches the published package
|
|
18
|
+
// on first use and is served from the npm cache afterwards.
|
|
19
|
+
//
|
|
20
|
+
// Because this file is committed rather than generated, path 2 works at all — which
|
|
21
|
+
// is what makes the pack installable by the wider Agent Skills ecosystem. The
|
|
22
|
+
// command a skill runs never changes; only where the engine came from does.
|
|
23
|
+
//
|
|
24
|
+
// Exit codes pass through unchanged: 0 success, 1 an invalid contract, 2 unreadable
|
|
25
|
+
// input or bad usage.
|
|
26
|
+
//
|
|
27
|
+
// No shebang: the synced copies carry a provenance marker on line one, which would
|
|
28
|
+
// sit above it and stop the kernel seeing it anyway. Every documented invocation is
|
|
29
|
+
// `node qa-tool.mjs …`, which needs none.
|
|
30
|
+
|
|
31
|
+
import fs from 'node:fs';
|
|
32
|
+
import os from 'node:os';
|
|
33
|
+
import path from 'node:path';
|
|
34
|
+
import { spawnSync } from 'node:child_process';
|
|
35
|
+
import { fileURLToPath } from 'node:url';
|
|
36
|
+
|
|
37
|
+
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
38
|
+
|
|
39
|
+
const USAGE = `usage: node qa-tool.mjs <tool> <subcommand> [args]
|
|
40
|
+
|
|
41
|
+
analysis parse artifacts, classify errors, validate contracts, diff-guard,
|
|
42
|
+
read .qa/context.md, render an HTML report, print the footer
|
|
43
|
+
diagnostics root cause, timeline, priority, repair plans, release readiness
|
|
44
|
+
playwright normalize a Playwright report or summarize a trace
|
|
45
|
+
|
|
46
|
+
--where print how the engine was resolved, and stop
|
|
47
|
+
|
|
48
|
+
examples:
|
|
49
|
+
node qa-tool.mjs analysis junit test-results/results.xml
|
|
50
|
+
node qa-tool.mjs analysis report-html qa-artifacts/explore-result.json --out report.html
|
|
51
|
+
node qa-tool.mjs diagnostics report --execution-result qa-artifacts/run.json
|
|
52
|
+
`;
|
|
53
|
+
|
|
54
|
+
/**
|
|
55
|
+
* Where the engine is, and how we found it.
|
|
56
|
+
*
|
|
57
|
+
* Ordered by cost: a bundled copy needs no resolution and no network, a shared copy
|
|
58
|
+
* needs one stat per ancestor, a node_modules copy needs no network, and npx needs both
|
|
59
|
+
* on first use. Reporting *which* one answered matters when a skill degrades — "the
|
|
60
|
+
* engine is missing" and "the engine is being fetched" are different problems.
|
|
61
|
+
*
|
|
62
|
+
* The shared lookup is what lets one skill directory serve a project, a workspace, and a
|
|
63
|
+
* machine-wide install without knowing which installed it. Walking up for
|
|
64
|
+
* `.qa-engineer/engine` finds a workspace install from a skill inside the repository,
|
|
65
|
+
* and finds a global install from a skill linked into an agent's user-level directory —
|
|
66
|
+
* `~/.claude/skills/qa-explore` walks up to `~`, where `~/.qa-engineer/engine` is.
|
|
67
|
+
*/
|
|
68
|
+
function resolveEngine() {
|
|
69
|
+
const bundled = path.join(here, 'lib', 'bin', 'qa-engine.mjs');
|
|
70
|
+
if (fs.existsSync(bundled)) return { kind: 'bundled', command: [process.execPath, bundled] };
|
|
71
|
+
|
|
72
|
+
for (const root of candidateSharedRoots()) {
|
|
73
|
+
const shared = path.join(root, 'bin', 'qa-engine.mjs');
|
|
74
|
+
if (fs.existsSync(shared)) return { kind: 'shared', command: [process.execPath, shared] };
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
for (const base of candidateModuleRoots()) {
|
|
78
|
+
const installed = path.join(base, 'qa-engineer', 'packages', 'engine', 'bin', 'qa-engine.mjs');
|
|
79
|
+
if (fs.existsSync(installed)) {
|
|
80
|
+
return { kind: 'node_modules', command: [process.execPath, installed] };
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
return {
|
|
85
|
+
kind: 'npx',
|
|
86
|
+
command: [npxCommand(), '--yes', 'qa-engineer', 'engine'],
|
|
87
|
+
};
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
/** Shared engine directories worth checking, most specific first. */
|
|
91
|
+
function candidateSharedRoots() {
|
|
92
|
+
const roots = [];
|
|
93
|
+
|
|
94
|
+
// An explicit home wins over anything discovered, so a user who moved the install can
|
|
95
|
+
// rely on it rather than on whatever the walk happens to find first.
|
|
96
|
+
const override = process.env.QA_ENGINEER_HOME;
|
|
97
|
+
if (override && override.trim()) roots.push(path.join(path.resolve(override.trim()), 'engine'));
|
|
98
|
+
|
|
99
|
+
let dir = here;
|
|
100
|
+
for (let depth = 0; depth < 12; depth += 1) {
|
|
101
|
+
roots.push(path.join(dir, '.qa-engineer', 'engine'));
|
|
102
|
+
const parent = path.dirname(dir);
|
|
103
|
+
if (parent === dir) break;
|
|
104
|
+
dir = parent;
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
// The default machine home, for the case where the skill lives outside it entirely.
|
|
108
|
+
const home = os.homedir();
|
|
109
|
+
if (home) roots.push(path.join(home, '.qa-engineer', 'engine'));
|
|
110
|
+
|
|
111
|
+
return roots;
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/** node_modules directories worth checking, nearest first. */
|
|
115
|
+
function candidateModuleRoots() {
|
|
116
|
+
const roots = [];
|
|
117
|
+
let dir = here;
|
|
118
|
+
for (let depth = 0; depth < 12; depth += 1) {
|
|
119
|
+
roots.push(path.join(dir, 'node_modules'));
|
|
120
|
+
const parent = path.dirname(dir);
|
|
121
|
+
if (parent === dir) break;
|
|
122
|
+
dir = parent;
|
|
123
|
+
}
|
|
124
|
+
if (process.cwd() !== here) roots.push(path.join(process.cwd(), 'node_modules'));
|
|
125
|
+
return roots;
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
// `npx` is a shell script on POSIX and a .cmd shim on Windows; spawnSync needs the
|
|
129
|
+
// exact name, and `shell: true` would put user-supplied arguments through a shell.
|
|
130
|
+
function npxCommand() {
|
|
131
|
+
return process.platform === 'win32' ? 'npx.cmd' : 'npx';
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
function main(argv) {
|
|
135
|
+
if (argv.length === 0 || argv[0] === '--help' || argv[0] === '-h') {
|
|
136
|
+
process.stdout.write(USAGE);
|
|
137
|
+
return argv.length === 0 ? 2 : 0;
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
const engine = resolveEngine();
|
|
141
|
+
|
|
142
|
+
if (argv[0] === '--where') {
|
|
143
|
+
process.stdout.write(`${JSON.stringify({ resolved: engine.kind, command: engine.command }, null, 2)}\n`);
|
|
144
|
+
return 0;
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
const [program, ...prefix] = engine.command;
|
|
148
|
+
const run = spawnSync(program, [...prefix, ...argv], { stdio: 'inherit' });
|
|
149
|
+
|
|
150
|
+
if (run.error) {
|
|
151
|
+
// Say which path was tried and what to do, because a skill's fallback prose
|
|
152
|
+
// cannot diagnose this and the user is the one who has to fix it.
|
|
153
|
+
const advice = engine.kind === 'npx'
|
|
154
|
+
? 'the engine is not bundled and npx is unavailable — run `npx qa-engineer install` in this project, or install Node 18+'
|
|
155
|
+
: `could not execute ${program}`;
|
|
156
|
+
process.stderr.write(`qa-tool: ${advice}\n${run.error.message}\n`);
|
|
157
|
+
return 2;
|
|
158
|
+
}
|
|
159
|
+
return run.status ?? 2;
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
process.exitCode = main(process.argv.slice(2));
|
|
@@ -58,12 +58,18 @@ Load only what the situation requires:
|
|
|
58
58
|
1. **Intake.** Confirm URL. Parse attached test cases into a numbered checklist ([test-case-intake.md](references/test-case-intake.md)). Note known bugs as hypotheses to validate. Create a run id and artifact directory `qa-artifacts/explore-<run-id>/`.
|
|
59
59
|
2. **Session.** Select a browser adapter ([browser-adapters.md](references/browser-adapters.md)). Navigate to the URL. If a login wall appears, **stop** and ask the user to sign in; never type credentials or OTPs. Baseline console errors and resource entries.
|
|
60
60
|
3. **Functional.** Execute attached cases first, then surface-by-surface exploration per [pipeline.md](references/pipeline.md). After every action: DOM-verify, then capture evidence on failure or notable finding ([evidence-capture.md](references/evidence-capture.md)).
|
|
61
|
-
4. **API audit.**
|
|
61
|
+
4. **API audit.** Capture a HAR, then run `analysis network` on it — **never count requests, time them, or spot duplicates by eye.** Replay exact app URLs in-page where a call needs confirming. The parser says what happened; you decide which of its flags matter to this product and write those as findings ([api-replay.md](references/api-replay.md)).
|
|
62
62
|
5. **Performance.** Measure payloads, cold vs warm, long tasks / vitals signals under controlled conditions ([performance.md](references/performance.md)).
|
|
63
63
|
6. **Security (client).** Run the client-side pass only — token storage, PII in URLs/payloads, error leakage, headers, optional read-only IDOR probe when in scope ([security.md](references/security.md)). No destructive tests.
|
|
64
64
|
7. **UI / UX.** Check empty/loading/error states, consistency, mobile viewport spot-check; optional persona lens if the user named a role.
|
|
65
65
|
8. **Optional DB.** Only if the user provided access: capture UI values with timestamps; query; separate data vs presentation bugs. Skip entirely otherwise and note "DB validation not in scope".
|
|
66
|
-
9. **Report.** Assign stable IDs and severities ([finding-taxonomy.md](references/finding-taxonomy.md)). Every finding must include proof.
|
|
66
|
+
9. **Report.** Assign stable IDs and severities ([finding-taxonomy.md](references/finding-taxonomy.md)). Every finding must include proof. Then, in this order ([report-pipeline.md](references/report-pipeline.md)):
|
|
67
|
+
1. Write `explore-result.json` — `artifacts[]` for every captured file, `scope` (what you set out to check, what you touched, and every boundary of the run with its reason), `executive` when a non-engineer will read it, and findings written for a reader who has never seen the product.
|
|
68
|
+
2. **Validate** it against the contract.
|
|
69
|
+
3. **Verify the artifacts** — `artifacts verify` must pass before anything is rendered. A missing or zero-byte file is reported, never quietly dropped.
|
|
70
|
+
4. **Build the bundle** with `report-bundle` — a portable folder that opens offline, with every link verified. That folder is the deliverable; point the user at its `index.html`. Never type HTML.
|
|
71
|
+
|
|
72
|
+
Include "what works well" and a prioritized fix order. Only report a score for a dimension the run actually measured.
|
|
67
73
|
10. **Iterate.** On user feedback: validate live, add evidence, bump report version, never renumber IDs. After three stuck browser attempts on the same blocker, stop and escalate with findings.
|
|
68
74
|
|
|
69
75
|
## Guardrails
|
|
@@ -79,26 +85,36 @@ Load only what the situation requires:
|
|
|
79
85
|
|
|
80
86
|
## Tooling
|
|
81
87
|
|
|
82
|
-
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-explore` or `.claude/skills/qa-explore`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe
|
|
88
|
+
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-explore` or `.claude/skills/qa-explore`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe, and it runs under the same Node that installed the pack — there is no second runtime to find.
|
|
83
89
|
|
|
84
90
|
| Tool | Invocation | Output | Fallback |
|
|
85
91
|
| --- | --- | --- | --- |
|
|
86
|
-
| Contract self-check | `
|
|
87
|
-
|
|
|
88
|
-
|
|
|
89
|
-
|
|
|
92
|
+
| Contract self-check | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis validate <explore-result.json> <SKILL_DIR>/contracts/explore-result.schema.json` | `{valid, errors}` — run this before rendering | None: an invalid result is not a report |
|
|
93
|
+
| Artifact check | `node <SKILL_DIR>/scripts/qa-tool.mjs artifacts verify <explore-result.json>` | `{ok, stats, missing}`; exit 1 when a file a finding points at is absent or empty. Run it **before** rendering | List the missing files in the report yourself; never delete the evidence entry to make it pass |
|
|
94
|
+
| Portable report bundle | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis report-bundle <explore-result.json> --out report --zip` | **The canonical output.** A folder — `index.html` + `assets/` — that opens offline in any browser, with every link verified to resolve inside it, plus a `.zip` to send. | **None.** Report that the engine is missing and stop |
|
|
95
|
+
| Single-file HTML | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis report-html <explore-result.json> --embed --out explore-report.html` | The complete self-contained report — nav, search, charts, timeline, expandable findings, evidence, attribution. `--embed` inlines images; `--mode full\|executive\|developer\|artifact` picks the audience | **None.** Report that the engine is missing and stop — a hand-written report is the failure this design removes |
|
|
96
|
+
| Other renderings | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis report-export <explore-result.json> --format <markdown\|sarif\|junit\|csv\|json\|bundle> --out <file>` | The same validated result in the format the destination reads | Write the Markdown by hand; skip the machine formats |
|
|
97
|
+
| Canonical schema | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis report-schema` · `… analysis report-versions` | The producer-neutral contract any agent writes to, and the schema/theme/renderer versions in force | Use the skill's own `contracts/explore-result.schema.json` |
|
|
98
|
+
| Network analysis | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis network <capture.har> [--slow-ms N]` | The report's `network` block, counted and flagged by the parser: totals, per-endpoint timing and size, and an `issue` on each — `failed`, `slow`, `polling`, `duplicate`, `n-plus-one`, `large-payload`, `uncached`. Drop it straight into the result | Read `performance.getEntriesByType('resource')` and say in the report that counts were observed rather than parsed |
|
|
99
|
+
| Secret redaction | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis redact <file>` | The file with credentials and tokens masked, for evidence excerpts | Redact by hand before the excerpt is written |
|
|
100
|
+
| Failure classification | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis classify "<error message>"` | `{classification, confidence, reason}` for a console or network error | Classify per [finding-taxonomy.md](references/finding-taxonomy.md) |
|
|
101
|
+
|
|
102
|
+
A missing `qa-tool.mjs` means the engine is not installed; run `qa doctor`.
|
|
90
103
|
|
|
91
|
-
|
|
104
|
+
**You produce structured data; the pack produces every document.** The result JSON is the source of truth and every format is rendered from it. You never write HTML, CSS, or styling of any kind — the contract has no field to carry a presentation hint, which is what makes a report from this skill identical whichever agent ran it. Hand-typing the page is how the first live run silently dropped `actual`, `expected`, and `fixDirection` from every finding while the JSON held all three.
|
|
92
105
|
|
|
93
|
-
**
|
|
106
|
+
**Register artifacts; do not inline paths.** Every captured file gets an `artifacts[]` entry, and evidence points at it by `artifactId`. Paths are relative to the result JSON. That is what makes `artifacts verify` meaningful and what stopped the second live run's screenshots from rendering as broken images.
|
|
94
107
|
|
|
95
108
|
## Output
|
|
96
109
|
|
|
97
|
-
Write under `qa-artifacts/explore-<run-id
|
|
110
|
+
Write under `qa-artifacts/explore-<run-id>/`, so the whole run folder moves as a unit:
|
|
111
|
+
|
|
112
|
+
- `explore-result.json` — machine-readable result conforming to [contracts/explore-result.schema.json](contracts/explore-result.schema.json); written first, and the source of every rendering below
|
|
113
|
+
- `report/` — **the deliverable.** A portable bundle written by `report-bundle`: `index.html`, `report.json`, `report.md`, `manifest.json`, and `assets/`. It opens offline in any browser and every link is verified to resolve inside it. `--zip` adds `report.zip` for sending.
|
|
114
|
+
- `screenshots/`, `network/`, `console/`, `dom/` — proof, registered in `artifacts[]` and referenced by `artifactId`; the bundle copies these into its own `assets/` tree
|
|
115
|
+
|
|
116
|
+
Point the user at `report/index.html`. A hosted preview — a Claude Artifact, a Cursor preview, any cloud viewer — is an optional convenience that expires; it must never be presented as the deliverable.
|
|
98
117
|
|
|
99
|
-
- `
|
|
100
|
-
- `explore-result.json` — machine-readable result conforming to [contracts/explore-result.schema.json](contracts/explore-result.schema.json); written first, and the source of the two renderings below
|
|
101
|
-
- `explore-report.html` — the report a person reads, **rendered** from the JSON by `report-html`
|
|
102
|
-
- `explore-report.md` — the same content as Markdown, with the rendered attribution footer appended
|
|
118
|
+
On request, also `report-html --embed` for one self-contained file, or `report-export` to SARIF (code-scanning viewers), JUnit (CI), or CSV (a tracking sheet). There is no PDF writer: the HTML has a print stylesheet that forces findings open and prints link targets, so **Print → Save as PDF** is the supported path.
|
|
103
119
|
|
|
104
|
-
Validate the JSON
|
|
120
|
+
Validate the JSON and run `artifacts verify` before rendering, and again before declaring completion. Present a short prose verdict (severity counts + top findings) in the conversation, and point to the artifact paths.
|