qa-engineer 0.9.2 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/COMPATIBILITY.md +26 -12
- package/README.md +20 -30
- package/package.json +11 -20
- package/packages/engine/bin/qa-engine.mjs +336 -0
- package/packages/engine/lib/analysis/branding.mjs +187 -0
- package/packages/engine/lib/analysis/context.mjs +294 -0
- package/packages/engine/lib/analysis/contracts.mjs +149 -0
- package/packages/engine/lib/analysis/diff-guard.mjs +425 -0
- package/packages/engine/lib/analysis/discovery.mjs +220 -0
- package/packages/engine/lib/analysis/evidence.mjs +116 -0
- package/packages/engine/lib/analysis/har.mjs +90 -0
- package/packages/engine/lib/analysis/junit.mjs +126 -0
- package/packages/engine/lib/analysis/redaction.mjs +127 -0
- package/packages/engine/lib/analysis/report-html.mjs +763 -0
- package/packages/engine/lib/analysis/taxonomy.mjs +109 -0
- package/packages/engine/lib/analysis/xml.mjs +153 -0
- package/packages/engine/lib/analysis/zip.mjs +107 -0
- package/packages/engine/lib/diagnostics/engine.mjs +165 -0
- package/packages/engine/lib/diagnostics/internal-contracts.mjs +50 -0
- package/packages/engine/lib/diagnostics/prioritization.mjs +99 -0
- package/packages/engine/lib/diagnostics/repair.mjs +73 -0
- package/packages/engine/lib/diagnostics/root-cause.mjs +92 -0
- package/packages/engine/lib/diagnostics/timeline.mjs +101 -0
- package/packages/engine/lib/frameworks/junit-frameworks.mjs +48 -0
- package/packages/engine/lib/frameworks/playwright.mjs +158 -0
- package/packages/engine/package.json +14 -0
- package/packages/installer/lib/commands/doctor.mjs +7 -16
- package/packages/installer/lib/commands/install.mjs +28 -0
- package/packages/installer/lib/core/bundle.mjs +69 -92
- package/packages/installer/lib/core/manifest.mjs +48 -57
- package/packages/installer/lib/core/validate-install.mjs +12 -19
- package/packages/installer/package.json +1 -1
- package/shared/tooling/qa-tool.mjs +125 -0
- package/skills/qa-api/SKILL.md +5 -5
- package/skills/qa-api/references/deterministic-tooling.md +34 -32
- package/skills/qa-api/references/evidence-and-reporting.md +5 -5
- package/skills/qa-api/scripts/qa-tool.mjs +126 -0
- package/skills/qa-audit/SKILL.md +5 -5
- package/skills/qa-audit/references/deterministic-tooling.md +34 -32
- package/skills/qa-audit/references/evidence-and-reporting.md +5 -5
- package/skills/qa-audit/scripts/qa-tool.mjs +126 -0
- package/skills/qa-debug/SKILL.md +6 -6
- package/skills/qa-debug/references/deterministic-tooling.md +34 -32
- package/skills/qa-debug/references/diagnostic-engine.md +3 -3
- package/skills/qa-debug/references/evidence-and-reporting.md +5 -5
- package/skills/qa-debug/scripts/qa-tool.mjs +126 -0
- package/skills/qa-explore/SKILL.md +6 -6
- package/skills/qa-explore/references/deterministic-tooling.md +34 -32
- package/skills/qa-explore/references/evidence-and-reporting.md +5 -5
- package/skills/qa-explore/references/report-pipeline.md +3 -3
- package/skills/qa-explore/scripts/qa-tool.mjs +126 -0
- package/skills/qa-fix/SKILL.md +4 -4
- package/skills/qa-fix/references/deterministic-tooling.md +34 -32
- package/skills/qa-fix/references/diagnostic-engine.md +3 -3
- package/skills/qa-fix/references/evidence-and-reporting.md +5 -5
- package/skills/qa-fix/scripts/qa-tool.mjs +126 -0
- package/skills/qa-flaky/SKILL.md +4 -4
- package/skills/qa-flaky/references/deterministic-tooling.md +34 -32
- package/skills/qa-flaky/references/evidence-and-reporting.md +5 -5
- package/skills/qa-flaky/scripts/qa-tool.mjs +126 -0
- package/skills/qa-generate/references/evidence-and-reporting.md +5 -5
- package/skills/qa-init/SKILL.md +4 -4
- package/skills/qa-init/references/deterministic-tooling.md +34 -32
- package/skills/qa-init/references/evidence-and-reporting.md +5 -5
- package/skills/qa-init/scripts/qa-tool.mjs +126 -0
- package/skills/qa-report/SKILL.md +7 -7
- package/skills/qa-report/references/deterministic-tooling.md +34 -32
- package/skills/qa-report/references/diagnostic-engine.md +3 -3
- package/skills/qa-report/references/evidence-and-reporting.md +5 -5
- package/skills/qa-report/scripts/qa-tool.mjs +126 -0
- package/skills/qa-review/references/evidence-and-reporting.md +5 -5
- package/skills/qa-run/SKILL.md +6 -6
- package/skills/qa-run/references/deterministic-tooling.md +34 -32
- package/skills/qa-run/references/evidence-and-reporting.md +5 -5
- package/skills/qa-run/scripts/qa-tool.mjs +126 -0
- package/shared/analysis/lib/qa_analysis/__init__.py +0 -11
- package/shared/analysis/lib/qa_analysis/branding.py +0 -175
- package/shared/analysis/lib/qa_analysis/cli.py +0 -144
- package/shared/analysis/lib/qa_analysis/context.py +0 -233
- package/shared/analysis/lib/qa_analysis/contracts.py +0 -158
- package/shared/analysis/lib/qa_analysis/diff_guard.py +0 -327
- package/shared/analysis/lib/qa_analysis/discovery.py +0 -113
- package/shared/analysis/lib/qa_analysis/evidence.py +0 -128
- package/shared/analysis/lib/qa_analysis/har.py +0 -87
- package/shared/analysis/lib/qa_analysis/junit.py +0 -104
- package/shared/analysis/lib/qa_analysis/redaction.py +0 -107
- package/shared/analysis/lib/qa_analysis/report_html.py +0 -781
- package/shared/analysis/lib/qa_analysis/taxonomy.py +0 -98
- package/shared/diagnostics/lib/qa_diagnostics/__init__.py +0 -13
- package/shared/diagnostics/lib/qa_diagnostics/cli.py +0 -124
- package/shared/diagnostics/lib/qa_diagnostics/engine.py +0 -143
- package/shared/diagnostics/lib/qa_diagnostics/internal_contracts.py +0 -75
- package/shared/diagnostics/lib/qa_diagnostics/prioritization.py +0 -101
- package/shared/diagnostics/lib/qa_diagnostics/repair.py +0 -72
- package/shared/diagnostics/lib/qa_diagnostics/root_cause.py +0 -89
- package/shared/diagnostics/lib/qa_diagnostics/timeline.py +0 -71
- package/shared/frameworks/cypress/lib/cypress_analysis.py +0 -27
- package/shared/frameworks/playwright/lib/playwright_analysis.py +0 -167
- package/shared/frameworks/selenium/lib/selenium_analysis.py +0 -28
- package/shared/frameworks/webdriverio/lib/webdriverio_analysis.py +0 -24
- package/shared/tooling/qa_tool.py +0 -127
- /package/{shared/analysis/lib/qa_analysis → packages/engine/lib/analysis}/branding.json +0 -0
- /package/{shared → packages/engine/lib}/analysis/schemas/context.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/analysis-result.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/diagnosis.schema.json +0 -0
- /package/{shared → packages/engine/lib}/diagnostics/schemas/internal/execution-result-min.schema.json +0 -0
|
@@ -1,124 +1,101 @@
|
|
|
1
|
-
// Bundling deterministic
|
|
2
|
-
//
|
|
3
|
-
//
|
|
1
|
+
// Bundling the deterministic engine, integrated into installation.
|
|
2
|
+
//
|
|
3
|
+
// Skills that run the engine need it to travel with them (self-containment). The
|
|
4
|
+
// installer materializes it by COPYING the engine from packages/engine/ into the
|
|
4
5
|
// installed skill's scripts/lib/ — a pure file copy, no code executed at install
|
|
5
|
-
// time, honoring the SECURITY.md guarantee.
|
|
6
|
-
// scripts/bundle_python.py does for repository development; users never run it.
|
|
6
|
+
// time, honoring the SECURITY.md guarantee.
|
|
7
7
|
//
|
|
8
|
-
//
|
|
9
|
-
//
|
|
10
|
-
//
|
|
8
|
+
// The engine is Node, and Node is already present: the user ran `npx` to get here.
|
|
9
|
+
// That removes the whole class of failure the Python bundle had, where the files
|
|
10
|
+
// could be copied perfectly and still not run because the interpreter was missing,
|
|
11
|
+
// the wrong version, or named `python` rather than `python3` (ADR-0012).
|
|
11
12
|
|
|
12
13
|
import fs from 'node:fs';
|
|
13
14
|
import path from 'node:path';
|
|
14
15
|
import { spawnSync } from 'node:child_process';
|
|
15
|
-
import {
|
|
16
|
-
BUNDLE_MANIFEST,
|
|
17
|
-
BUNDLE_SOURCES,
|
|
18
|
-
BUNDLE_PACKAGE_DATA,
|
|
19
|
-
BUNDLE_MODULE_SOURCES,
|
|
20
|
-
BUNDLE_LAUNCHER,
|
|
21
|
-
BUNDLE_DEST,
|
|
22
|
-
} from './manifest.mjs';
|
|
16
|
+
import { BUNDLE_MANIFEST, ENGINE_SOURCE, ENGINE_DATA, BUNDLE_DEST } from './manifest.mjs';
|
|
23
17
|
import { listFilesRelative } from './paths.mjs';
|
|
24
18
|
|
|
25
19
|
/** Bundle file entries for one skill, relative to the skill directory. */
|
|
26
20
|
export function bundleFilesForSkill(sourceRoot, skill) {
|
|
27
|
-
|
|
28
|
-
if (!entry) return [];
|
|
21
|
+
if (!BUNDLE_MANIFEST[skill]) return [];
|
|
29
22
|
const entries = [];
|
|
30
23
|
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
}
|
|
24
|
+
const engineRoot = path.join(sourceRoot, ENGINE_SOURCE);
|
|
25
|
+
if (!fs.existsSync(engineRoot)) {
|
|
26
|
+
throw new Error(`engine source missing: ${ENGINE_SOURCE} (needed by ${skill})`);
|
|
27
|
+
}
|
|
28
|
+
for (const rel of listFilesRelative(engineRoot)) {
|
|
29
|
+
// Tests and the package manifest are development files; the bundle is the
|
|
30
|
+
// runtime only.
|
|
31
|
+
if (rel.startsWith('test/') || rel === 'package.json') continue;
|
|
32
|
+
entries.push({
|
|
33
|
+
rel: `${BUNDLE_DEST}/${rel}`,
|
|
34
|
+
content: fs.readFileSync(path.join(engineRoot, rel)),
|
|
35
|
+
});
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
// Data the engine reads at runtime: the context contract, the internal seam
|
|
39
|
+
// schemas, and the branding metadata.
|
|
40
|
+
for (const data of ENGINE_DATA) {
|
|
41
|
+
const from = path.join(sourceRoot, data.from);
|
|
42
|
+
if (!fs.existsSync(from)) {
|
|
43
|
+
throw new Error(`engine data missing: ${data.from} (needed by ${skill})`);
|
|
41
44
|
}
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
for (const data of BUNDLE_PACKAGE_DATA[pkg] ?? []) {
|
|
45
|
-
const dataSrc = path.join(sourceRoot, data.from);
|
|
46
|
-
if (!fs.existsSync(dataSrc)) {
|
|
47
|
-
throw new Error(`bundled package data missing: ${data.from} (needed by ${skill})`);
|
|
48
|
-
}
|
|
49
|
-
for (const rel of listFilesRelative(dataSrc)) {
|
|
45
|
+
if (fs.statSync(from).isDirectory()) {
|
|
46
|
+
for (const rel of listFilesRelative(from)) {
|
|
50
47
|
entries.push({
|
|
51
|
-
rel: `${BUNDLE_DEST}/${
|
|
52
|
-
content: fs.readFileSync(path.join(
|
|
48
|
+
rel: `${BUNDLE_DEST}/${data.to}/${rel}`,
|
|
49
|
+
content: fs.readFileSync(path.join(from, rel)),
|
|
53
50
|
});
|
|
54
51
|
}
|
|
52
|
+
} else {
|
|
53
|
+
entries.push({ rel: `${BUNDLE_DEST}/${data.to}`, content: fs.readFileSync(from) });
|
|
55
54
|
}
|
|
56
55
|
}
|
|
57
56
|
|
|
58
|
-
// The launcher
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
for (const mod of entry.modules) {
|
|
66
|
-
const src = path.join(sourceRoot, BUNDLE_MODULE_SOURCES[mod]);
|
|
67
|
-
if (!fs.existsSync(src)) {
|
|
68
|
-
throw new Error(`bundled module source missing: ${BUNDLE_MODULE_SOURCES[mod]} (needed by ${skill})`);
|
|
69
|
-
}
|
|
70
|
-
entries.push({
|
|
71
|
-
rel: `${BUNDLE_DEST}/${path.basename(src)}`,
|
|
72
|
-
content: fs.readFileSync(src),
|
|
73
|
-
});
|
|
74
|
-
}
|
|
57
|
+
// The launcher is deliberately NOT bundled. It is a *committed* file in each
|
|
58
|
+
// skill (kept in step by sync-shared), because that is what makes a skill work
|
|
59
|
+
// when a generic file copier installs it. Copying it again here produced two
|
|
60
|
+
// different contents for one path — the committed copy carries a provenance
|
|
61
|
+
// marker — and the installer's conflict detector rejected the install outright.
|
|
62
|
+
// One file, one source.
|
|
75
63
|
|
|
76
64
|
return entries;
|
|
77
65
|
}
|
|
78
66
|
|
|
79
|
-
/** True if any skill carries bundled
|
|
67
|
+
/** True if any skill carries the bundled engine (used to gate doctor's deep check). */
|
|
80
68
|
export function packHasBundles() {
|
|
81
69
|
return Object.keys(BUNDLE_MANIFEST).length > 0;
|
|
82
70
|
}
|
|
83
71
|
|
|
84
|
-
/** Locate a Python 3 interpreter, or null. Used only by doctor. */
|
|
85
|
-
export function findPython() {
|
|
86
|
-
for (const candidate of ['python3', 'python']) {
|
|
87
|
-
const probe = spawnSync(candidate, ['--version'], { encoding: 'utf8' });
|
|
88
|
-
if (probe.status === 0) {
|
|
89
|
-
const version = (probe.stdout || probe.stderr).trim();
|
|
90
|
-
return { bin: candidate, version };
|
|
91
|
-
}
|
|
92
|
-
}
|
|
93
|
-
return null;
|
|
94
|
-
}
|
|
95
|
-
|
|
96
|
-
// Runs inside the bundle with only the bundle on PYTHONPATH. Importing is not
|
|
97
|
-
// enough: a missing data file only surfaces when the engine actually runs, which
|
|
98
|
-
// is exactly the failure this check exists to catch.
|
|
99
|
-
const SMOKE = `
|
|
100
|
-
import json, sys
|
|
101
|
-
names = json.loads(sys.argv[1])
|
|
102
|
-
for name in names:
|
|
103
|
-
__import__(name)
|
|
104
|
-
if "qa_diagnostics" in names:
|
|
105
|
-
from qa_diagnostics import engine
|
|
106
|
-
diagnosis = engine.diagnose({
|
|
107
|
-
"tests": {"total": 1, "passed": 0, "failed": 1, "skipped": 0},
|
|
108
|
-
"executed": [{"title": "t", "status": "failed", "message": "no such element: #cart",
|
|
109
|
-
"file": "t.spec.ts", "retries": 0}],
|
|
110
|
-
})
|
|
111
|
-
assert diagnosis["entries"], "engine produced no diagnosis entries"
|
|
112
|
-
`;
|
|
113
|
-
|
|
114
72
|
/**
|
|
115
|
-
* Verify bundled
|
|
116
|
-
*
|
|
73
|
+
* Verify the bundled engine RUNS from its own directory, under the Node that will
|
|
74
|
+
* run it.
|
|
75
|
+
*
|
|
76
|
+
* Copying files correctly is not the same as the result working: a missing data file
|
|
77
|
+
* only surfaces when the code reads it. So this executes a real command whose whole
|
|
78
|
+
* path touches the runtime data — `classify` exercises the taxonomy, and the
|
|
79
|
+
* diagnostics report validates against the bundled internal schemas.
|
|
117
80
|
*/
|
|
118
|
-
export function
|
|
119
|
-
const
|
|
120
|
-
|
|
81
|
+
export function verifyEngine({ libDir }) {
|
|
82
|
+
const cli = path.join(libDir, 'bin', 'qa-engine.mjs');
|
|
83
|
+
if (!fs.existsSync(cli)) {
|
|
84
|
+
return { ok: false, stderr: `bundled engine CLI missing at ${cli}` };
|
|
85
|
+
}
|
|
86
|
+
const probe = spawnSync(process.execPath, [cli, 'analysis', 'classify', 'no such element: #cart'], {
|
|
121
87
|
encoding: 'utf8',
|
|
122
88
|
});
|
|
123
|
-
|
|
89
|
+
if (probe.status !== 0) {
|
|
90
|
+
return { ok: false, stderr: (probe.stderr || probe.stdout || '').trim() };
|
|
91
|
+
}
|
|
92
|
+
try {
|
|
93
|
+
const parsed = JSON.parse(probe.stdout);
|
|
94
|
+
if (parsed.classification !== 'locator-failure') {
|
|
95
|
+
return { ok: false, stderr: `engine classified a missing element as ${parsed.classification}` };
|
|
96
|
+
}
|
|
97
|
+
} catch (error) {
|
|
98
|
+
return { ok: false, stderr: `engine did not emit JSON: ${error.message}` };
|
|
99
|
+
}
|
|
100
|
+
return { ok: true, stderr: '' };
|
|
124
101
|
}
|
|
@@ -1,74 +1,65 @@
|
|
|
1
|
-
// What the pack ships: the set of skills, the files inside each, and which
|
|
2
|
-
//
|
|
3
|
-
// mirrors scripts/bundle_python.py — the two must agree, and a test asserts it.
|
|
1
|
+
// What the pack ships: the set of skills, the files inside each, and which skills
|
|
2
|
+
// carry the bundled deterministic engine.
|
|
4
3
|
|
|
5
4
|
import fs from 'node:fs';
|
|
6
5
|
import path from 'node:path';
|
|
7
6
|
import { listFilesRelative } from './paths.mjs';
|
|
8
7
|
|
|
9
|
-
//
|
|
8
|
+
// Every bundling skill carries the same payload, named once so the map below reads
|
|
9
|
+
// as the list of skills it is.
|
|
10
|
+
const ENGINE = Object.freeze({ engine: true });
|
|
11
|
+
|
|
12
|
+
// Skills that run the deterministic engine in the consumer's repository. The
|
|
10
13
|
// bundle is materialized at install time, never shipped pre-built.
|
|
11
|
-
// qa-review is knowledge-only and bundles nothing.
|
|
12
14
|
//
|
|
13
|
-
//
|
|
14
|
-
//
|
|
15
|
+
// Every one of them bundles the same thing now. Under the Python engine this was a
|
|
16
|
+
// per-skill list of packages and single-file adapters, because a skill that never
|
|
17
|
+
// touched a Playwright trace had no reason to carry the adapter. The Node engine is
|
|
18
|
+
// one package of about 3,000 lines with no dependencies, so splitting it would save
|
|
19
|
+
// nothing measurable and would give nine skills nine slightly different bundles to
|
|
20
|
+
// go wrong in. qa-review, qa-generate, qa-example and the qa router remain
|
|
21
|
+
// knowledge-only and bundle nothing.
|
|
15
22
|
export const BUNDLE_MANIFEST = Object.freeze({
|
|
16
|
-
'qa-init':
|
|
17
|
-
'qa-run':
|
|
18
|
-
'qa-debug':
|
|
19
|
-
'qa-fix':
|
|
20
|
-
'qa-report':
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
'qa-
|
|
24
|
-
'qa-
|
|
25
|
-
'qa-api': Object.freeze({ packages: ['qa_analysis', 'qa_diagnostics'], modules: [] }),
|
|
26
|
-
'qa-audit': Object.freeze({ packages: ['qa_analysis', 'qa_diagnostics'], modules: [] }),
|
|
27
|
-
});
|
|
28
|
-
|
|
29
|
-
// Canonical source of each bundled package, relative to the pack root.
|
|
30
|
-
export const BUNDLE_SOURCES = Object.freeze({
|
|
31
|
-
qa_analysis: 'shared/analysis/lib/qa_analysis',
|
|
32
|
-
qa_diagnostics: 'shared/diagnostics/lib/qa_diagnostics',
|
|
23
|
+
'qa-init': ENGINE,
|
|
24
|
+
'qa-run': ENGINE,
|
|
25
|
+
'qa-debug': ENGINE,
|
|
26
|
+
'qa-fix': ENGINE,
|
|
27
|
+
'qa-report': ENGINE,
|
|
28
|
+
'qa-flaky': ENGINE,
|
|
29
|
+
'qa-api': ENGINE,
|
|
30
|
+
'qa-audit': ENGINE,
|
|
31
|
+
'qa-explore': ENGINE,
|
|
33
32
|
});
|
|
34
33
|
|
|
35
|
-
//
|
|
36
|
-
//
|
|
37
|
-
|
|
38
|
-
export const BUNDLE_PACKAGE_DATA = Object.freeze({
|
|
39
|
-
// The context CLI validates .qa/context.md against the context contract.
|
|
40
|
-
qa_analysis: Object.freeze([
|
|
41
|
-
Object.freeze({ from: 'shared/analysis/schemas', to: 'schemas' }),
|
|
42
|
-
]),
|
|
43
|
-
qa_diagnostics: Object.freeze([
|
|
44
|
-
Object.freeze({ from: 'shared/diagnostics/schemas/internal', to: 'schemas/internal' }),
|
|
45
|
-
]),
|
|
46
|
-
});
|
|
34
|
+
// The engine, copied wholesale: `lib/` for the modules and `bin/` for the CLI the
|
|
35
|
+
// launcher executes.
|
|
36
|
+
export const ENGINE_SOURCE = 'packages/engine';
|
|
47
37
|
|
|
48
|
-
//
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
38
|
+
// The engine keeps the non-code files it reads at runtime — the context contract,
|
|
39
|
+
// the internal seam schemas, the branding metadata — inside its own lib/, so a
|
|
40
|
+
// wholesale copy of the engine directory is a complete engine. Nothing to enumerate
|
|
41
|
+
// here, and no second place for a data file to be forgotten.
|
|
42
|
+
export const ENGINE_DATA = Object.freeze([]);
|
|
52
43
|
|
|
53
|
-
// The launcher every bundling skill carries, one level above lib/.
|
|
54
|
-
//
|
|
55
|
-
//
|
|
44
|
+
// The launcher every bundling skill carries, one level above lib/.
|
|
45
|
+
//
|
|
46
|
+
// It is a *committed* file in each skill rather than something the installer
|
|
47
|
+
// generates — `node scripts/sync-shared.mjs --write` refreshes the copies and
|
|
48
|
+
// `--check` fails on drift. That is what lets a generic file copier (`npx skills
|
|
49
|
+
// add`, or a plain `git clone`) produce a working skill without running this
|
|
50
|
+
// installer at all. Consequently the bundler does not copy it: two sources for one
|
|
51
|
+
// destination path is a conflict, and the installer is right to refuse it.
|
|
56
52
|
export const BUNDLE_LAUNCHER = Object.freeze({
|
|
57
|
-
from: 'shared/tooling/
|
|
58
|
-
to: 'scripts/
|
|
53
|
+
from: 'shared/tooling/qa-tool.mjs',
|
|
54
|
+
to: 'scripts/qa-tool.mjs',
|
|
59
55
|
});
|
|
60
56
|
|
|
61
|
-
// Where bundled
|
|
57
|
+
// Where the bundled engine lands inside an installed skill.
|
|
62
58
|
export const BUNDLE_DEST = 'scripts/lib';
|
|
63
59
|
|
|
64
|
-
/**
|
|
65
|
-
export function
|
|
66
|
-
return BUNDLE_MANIFEST
|
|
67
|
-
}
|
|
68
|
-
|
|
69
|
-
/** Module names bundled into a skill. */
|
|
70
|
-
export function bundleModulesForSkill(skill) {
|
|
71
|
-
return BUNDLE_MANIFEST[skill]?.modules ?? [];
|
|
60
|
+
/** True when this skill bundles the engine. */
|
|
61
|
+
export function bundlesEngine(skill) {
|
|
62
|
+
return Object.prototype.hasOwnProperty.call(BUNDLE_MANIFEST, skill);
|
|
72
63
|
}
|
|
73
64
|
|
|
74
65
|
/** All skill names shipped by the pack (directories under skills/). */
|
|
@@ -82,9 +73,9 @@ export function listSkills(sourceRoot) {
|
|
|
82
73
|
}
|
|
83
74
|
|
|
84
75
|
/**
|
|
85
|
-
* Files that make up a skill, relative to the skill directory. Excludes the
|
|
86
|
-
*
|
|
87
|
-
*
|
|
76
|
+
* Files that make up a skill, relative to the skill directory. Excludes the bundled
|
|
77
|
+
* engine (scripts/lib): it is a generated artifact produced by the bundle step at
|
|
78
|
+
* install time, not copied from source.
|
|
88
79
|
*/
|
|
89
80
|
export function skillFiles(sourceRoot, skill) {
|
|
90
81
|
const dir = path.join(sourceRoot, 'skills', skill);
|
|
@@ -5,8 +5,8 @@ import path from 'node:path';
|
|
|
5
5
|
import { LOCKFILE, SHARED_SKILLS_DIR, CLAUDE_SKILLS_DIR } from '../constants.mjs';
|
|
6
6
|
import { readLock } from './lockfile.mjs';
|
|
7
7
|
import { hashFile } from './hash.mjs';
|
|
8
|
-
import {
|
|
9
|
-
import { BUNDLE_DEST, BUNDLE_MANIFEST
|
|
8
|
+
import { verifyEngine, packHasBundles } from './bundle.mjs';
|
|
9
|
+
import { BUNDLE_DEST, BUNDLE_MANIFEST } from './manifest.mjs';
|
|
10
10
|
|
|
11
11
|
/**
|
|
12
12
|
* @typedef {{ id: string, ok: boolean, hard: boolean, message: string, hint?: string }} CheckResult
|
|
@@ -114,28 +114,21 @@ export function validateInstall(projectRoot) {
|
|
|
114
114
|
hint: bundleOk ? undefined : 'run: qa repair',
|
|
115
115
|
});
|
|
116
116
|
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
const result =
|
|
117
|
+
// The engine runs under the Node that is already here — no second runtime to
|
|
118
|
+
// find, and nothing to skip when it is absent.
|
|
119
|
+
if (bundleOk) {
|
|
120
|
+
const result = verifyEngine({ libDir });
|
|
121
121
|
checks.push({
|
|
122
|
-
id: '
|
|
122
|
+
id: 'engine-runs',
|
|
123
123
|
ok: result.ok,
|
|
124
|
-
hard:
|
|
124
|
+
hard: true,
|
|
125
125
|
message: result.ok
|
|
126
|
-
? `
|
|
127
|
-
: `
|
|
128
|
-
hint: result.ok ? undefined : '
|
|
129
|
-
});
|
|
130
|
-
} else if (bundleOk && !python) {
|
|
131
|
-
checks.push({
|
|
132
|
-
id: 'python-imports',
|
|
133
|
-
ok: false,
|
|
134
|
-
hard: false,
|
|
135
|
-
message: 'Python not found — analysis engine import check skipped',
|
|
136
|
-
hint: 'install Python 3.8+ for full diagnostics',
|
|
126
|
+
? `bundled engine runs (node ${process.versions.node})`
|
|
127
|
+
: `bundled engine failed to run: ${result.stderr || 'unknown error'}`,
|
|
128
|
+
hint: result.ok ? undefined : 'run: qa repair',
|
|
137
129
|
});
|
|
138
130
|
}
|
|
131
|
+
|
|
139
132
|
}
|
|
140
133
|
|
|
141
134
|
const major = Number(process.versions.node.split('.')[0]);
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
// The launcher every skill invokes to reach the deterministic engine.
|
|
2
|
+
//
|
|
3
|
+
// A skill's SKILL.md documents exactly one command shape:
|
|
4
|
+
//
|
|
5
|
+
// node <SKILL_DIR>/scripts/qa-tool.mjs <tool> <subcommand> [args]
|
|
6
|
+
//
|
|
7
|
+
// and this file finds the engine, wherever it happens to be. That indirection
|
|
8
|
+
// exists because the pack is installed three different ways and the engine lands in
|
|
9
|
+
// a different place each time:
|
|
10
|
+
//
|
|
11
|
+
// 1. `qa install` bundles the engine into the skill, at ./lib/. Offline, fastest,
|
|
12
|
+
// and pinned to the version that was installed.
|
|
13
|
+
// 2. `npx skills add <owner>/<repo>` — or any generic file copier — copies the
|
|
14
|
+
// skill directory out of git and bundles nothing. The engine is then resolved
|
|
15
|
+
// from node_modules if the project happens to depend on the pack.
|
|
16
|
+
// 3. Neither: fall back to `npx qa-engineer`, which fetches the published package
|
|
17
|
+
// on first use and is served from the npm cache afterwards.
|
|
18
|
+
//
|
|
19
|
+
// Because this file is committed rather than generated, path 2 works at all — which
|
|
20
|
+
// is what makes the pack installable by the wider Agent Skills ecosystem. The
|
|
21
|
+
// command a skill runs never changes; only where the engine came from does.
|
|
22
|
+
//
|
|
23
|
+
// Exit codes pass through unchanged: 0 success, 1 an invalid contract, 2 unreadable
|
|
24
|
+
// input or bad usage.
|
|
25
|
+
//
|
|
26
|
+
// No shebang: the synced copies carry a provenance marker on line one, which would
|
|
27
|
+
// sit above it and stop the kernel seeing it anyway. Every documented invocation is
|
|
28
|
+
// `node qa-tool.mjs …`, which needs none.
|
|
29
|
+
|
|
30
|
+
import fs from 'node:fs';
|
|
31
|
+
import path from 'node:path';
|
|
32
|
+
import { spawnSync } from 'node:child_process';
|
|
33
|
+
import { fileURLToPath } from 'node:url';
|
|
34
|
+
|
|
35
|
+
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
36
|
+
|
|
37
|
+
const USAGE = `usage: node qa-tool.mjs <tool> <subcommand> [args]
|
|
38
|
+
|
|
39
|
+
analysis parse artifacts, classify errors, validate contracts, diff-guard,
|
|
40
|
+
read .qa/context.md, render an HTML report, print the footer
|
|
41
|
+
diagnostics root cause, timeline, priority, repair plans, release readiness
|
|
42
|
+
playwright normalize a Playwright report or summarize a trace
|
|
43
|
+
|
|
44
|
+
--where print how the engine was resolved, and stop
|
|
45
|
+
|
|
46
|
+
examples:
|
|
47
|
+
node qa-tool.mjs analysis junit test-results/results.xml
|
|
48
|
+
node qa-tool.mjs analysis report-html qa-artifacts/explore-result.json --out report.html
|
|
49
|
+
node qa-tool.mjs diagnostics report --execution-result qa-artifacts/run.json
|
|
50
|
+
`;
|
|
51
|
+
|
|
52
|
+
/**
|
|
53
|
+
* Where the engine is, and how we found it.
|
|
54
|
+
*
|
|
55
|
+
* Ordered by cost: a bundled copy needs no resolution and no network, a
|
|
56
|
+
* node_modules copy needs no network, and npx needs both on first use. Reporting
|
|
57
|
+
* *which* one answered matters when a skill degrades — "the engine is missing" and
|
|
58
|
+
* "the engine is being fetched" are different problems.
|
|
59
|
+
*/
|
|
60
|
+
function resolveEngine() {
|
|
61
|
+
const bundled = path.join(here, 'lib', 'bin', 'qa-engine.mjs');
|
|
62
|
+
if (fs.existsSync(bundled)) return { kind: 'bundled', command: [process.execPath, bundled] };
|
|
63
|
+
|
|
64
|
+
for (const base of candidateModuleRoots()) {
|
|
65
|
+
const installed = path.join(base, 'qa-engineer', 'packages', 'engine', 'bin', 'qa-engine.mjs');
|
|
66
|
+
if (fs.existsSync(installed)) {
|
|
67
|
+
return { kind: 'node_modules', command: [process.execPath, installed] };
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
return {
|
|
72
|
+
kind: 'npx',
|
|
73
|
+
command: [npxCommand(), '--yes', 'qa-engineer', 'engine'],
|
|
74
|
+
};
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
/** node_modules directories worth checking, nearest first. */
|
|
78
|
+
function candidateModuleRoots() {
|
|
79
|
+
const roots = [];
|
|
80
|
+
let dir = here;
|
|
81
|
+
for (let depth = 0; depth < 12; depth += 1) {
|
|
82
|
+
roots.push(path.join(dir, 'node_modules'));
|
|
83
|
+
const parent = path.dirname(dir);
|
|
84
|
+
if (parent === dir) break;
|
|
85
|
+
dir = parent;
|
|
86
|
+
}
|
|
87
|
+
if (process.cwd() !== here) roots.push(path.join(process.cwd(), 'node_modules'));
|
|
88
|
+
return roots;
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
// `npx` is a shell script on POSIX and a .cmd shim on Windows; spawnSync needs the
|
|
92
|
+
// exact name, and `shell: true` would put user-supplied arguments through a shell.
|
|
93
|
+
function npxCommand() {
|
|
94
|
+
return process.platform === 'win32' ? 'npx.cmd' : 'npx';
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
function main(argv) {
|
|
98
|
+
if (argv.length === 0 || argv[0] === '--help' || argv[0] === '-h') {
|
|
99
|
+
process.stdout.write(USAGE);
|
|
100
|
+
return argv.length === 0 ? 2 : 0;
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
const engine = resolveEngine();
|
|
104
|
+
|
|
105
|
+
if (argv[0] === '--where') {
|
|
106
|
+
process.stdout.write(`${JSON.stringify({ resolved: engine.kind, command: engine.command }, null, 2)}\n`);
|
|
107
|
+
return 0;
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
const [program, ...prefix] = engine.command;
|
|
111
|
+
const run = spawnSync(program, [...prefix, ...argv], { stdio: 'inherit' });
|
|
112
|
+
|
|
113
|
+
if (run.error) {
|
|
114
|
+
// Say which path was tried and what to do, because a skill's fallback prose
|
|
115
|
+
// cannot diagnose this and the user is the one who has to fix it.
|
|
116
|
+
const advice = engine.kind === 'npx'
|
|
117
|
+
? 'the engine is not bundled and npx is unavailable — run `npx qa-engineer install` in this project, or install Node 18+'
|
|
118
|
+
: `could not execute ${program}`;
|
|
119
|
+
process.stderr.write(`qa-tool: ${advice}\n${run.error.message}\n`);
|
|
120
|
+
return 2;
|
|
121
|
+
}
|
|
122
|
+
return run.status ?? 2;
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
process.exitCode = main(process.argv.slice(2));
|
package/skills/qa-api/SKILL.md
CHANGED
|
@@ -53,15 +53,15 @@ Do not use it to generate API tests (`/qa-generate`) or to run a suite (`/qa-run
|
|
|
53
53
|
|
|
54
54
|
## Tooling
|
|
55
55
|
|
|
56
|
-
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-api` or `.claude/skills/qa-api`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe
|
|
56
|
+
Invoke the bundled engine through its launcher, as documented in [references/deterministic-tooling.md](references/deterministic-tooling.md). `SKILL_DIR` below is this skill's own directory — `.agents/skills/qa-api` or `.claude/skills/qa-api`, whichever exists. The command shape is the same in bash, zsh, PowerShell, and cmd.exe, and it runs under the same Node that installed the pack — there is no second runtime to find.
|
|
57
57
|
|
|
58
58
|
| Tool | Invocation | Output | Fallback |
|
|
59
59
|
| --- | --- | --- | --- |
|
|
60
|
-
| HAR analyzer | `
|
|
61
|
-
| Error classifier | `
|
|
62
|
-
| Redaction | `
|
|
60
|
+
| HAR analyzer | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis har <file.har> [--slow-ms N]` | Redacted request/response summary, failures, slow calls | Assess the tests without traffic and say so |
|
|
61
|
+
| Error classifier | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis classify "<message>" --http-status <N>` | Taxonomy classification for a failing call | Classify from the REST/GraphQL modules and lower confidence |
|
|
62
|
+
| Redaction | `node <SKILL_DIR>/scripts/qa-tool.mjs analysis redact <file>` | The file's text with credentials masked, before anything is quoted | Do not quote captured traffic at all |
|
|
63
63
|
|
|
64
|
-
A missing `
|
|
64
|
+
A missing `qa-tool.mjs` means the engine is not installed. Never paste raw captured traffic into a report — redaction is deterministic and must not be undone.
|
|
65
65
|
|
|
66
66
|
## Output
|
|
67
67
|
|