qualiow-exploratory-testing 2.0.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +26 -0
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +65 -0
- package/README.md +123 -12
- package/agents/qa-diff-indexer-agent.md +81 -0
- package/agents/qa-gather-agent.md +21 -4
- package/agents/qa-page-mapper-agent.md +74 -0
- package/agents/qa-reporting-agent.md +103 -0
- package/bin/qualiow +30 -0
- package/data/knowledge/manifest.yml +22 -0
- package/dist/cli/index.js +876 -43
- package/dist/index.d.ts +50 -2
- package/dist/index.js +474 -29
- package/docs/GETTING-STARTED.md +127 -8
- package/hooks/hooks.json +27 -0
- package/hooks/scripts/read-guard.mjs +95 -0
- package/hooks/scripts/secret-patterns.d.mts +8 -0
- package/hooks/scripts/secret-patterns.mjs +136 -0
- package/hooks/scripts/write-guard.mjs +87 -0
- package/package.json +3 -2
- package/skills/qa-explore/SKILL.md +2 -1
- package/skills/qa-explore/phases/00-setup.md +16 -8
- package/skills/qa-explore/phases/03-discovery.md +48 -6
- package/skills/qa-explore/phases/07-reporting.md +36 -15
- package/skills/qa-explore/references/delegation-rules.md +82 -0
- package/skills/qa-explore/references/output-contract.md +7 -2
- package/skills/qa-explore/references/paths.md +25 -1
- package/skills/qa-explore/references/security-rules.md +4 -0
- package/skills/qa-explore/references/session-rules.md +2 -1
- package/skills/qa-explore-cleanup/SKILL.md +44 -19
- package/skills/qa-explore-feedback/SKILL.md +5 -0
- package/skills/qa-explore-mobile/SKILL.md +1 -1
- package/skills/qa-explore-mobile/phases/07-reporting.md +33 -11
- package/skills/qa-explore-quick/SKILL.md +12 -2
- package/skills/qa-explore-report/SKILL.md +29 -5
- package/skills/qa-gather/SKILL.md +30 -9
- package/skills/qa-knowledge-add/SKILL.md +10 -5
- package/skills/qa-knowledge-list/SKILL.md +28 -40
- package/skills/qa-verify-backend/SKILL.md +7 -4
- package/skills/qa-verify-backend/phases/02-static-review.md +11 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +3 -1
- package/skills/qa-verify-backend/phases/03b-api-verification.md +3 -1
- package/skills/qa-verify-backend/phases/05-reporting.md +51 -19
package/docs/GETTING-STARTED.md
CHANGED
|
@@ -15,7 +15,7 @@ Step-by-step guide to running your first AI-powered exploratory testing session.
|
|
|
15
15
|
|
|
16
16
|
```bash
|
|
17
17
|
npm install -g qualiow-exploratory-testing
|
|
18
|
-
qualiow --version # prints the installed package version, e.g. 2.
|
|
18
|
+
qualiow --version # prints the installed package version, e.g. 2.1.0
|
|
19
19
|
```
|
|
20
20
|
|
|
21
21
|
### Initialise a project
|
|
@@ -30,7 +30,7 @@ That writes:
|
|
|
30
30
|
| What | Where |
|
|
31
31
|
|------|-------|
|
|
32
32
|
| the 11 skills | `.claude/skills/qa-*/` |
|
|
33
|
-
| the
|
|
33
|
+
| the four sub-agents | `.claude/agents/` |
|
|
34
34
|
| knowledge base, domain profiles, templates, security policy | `data/knowledge/`, `data/domains/`, `data/templates/`, `data/security/` |
|
|
35
35
|
| the `_default` target (add `--include-examples` for the `_example-*` templates and `testers-ai.yml`) | `data/targets/` |
|
|
36
36
|
| the mobile driver plus `setup-mobile.sh` and `doctor-mobile.sh`, executable | `qa/bin/` |
|
|
@@ -46,9 +46,78 @@ cp qa/.env.example qa/.env # credentials go here, never in a YAML fi
|
|
|
46
46
|
npx playwright-cli install --skills # optional: the official Playwright skill alongside
|
|
47
47
|
```
|
|
48
48
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
49
|
+
### Optionally, install the guard hooks
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
qualiow init --hooks
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Optional, and only for npm projects — a plugin install already ships the same two hooks and
|
|
56
|
+
has them on by default, so do not enable both in one project. `--hooks` copies the guard
|
|
57
|
+
scripts to `qa/hooks/` and merges into `.claude/settings.json` two `PreToolUse` entries and
|
|
58
|
+
three `permissions.allow` rules. The merge is by exact string: running it again changes
|
|
59
|
+
nothing, and hooks you already had are kept.
|
|
60
|
+
|
|
61
|
+
What the guards do: `read-guard.mjs` denies a whole-file `Read` of a qualiow file over
|
|
62
|
+
`QUALIOW_READ_MAX_LINES` lines (default 300) — the knowledge manifest, a knowledge release
|
|
63
|
+
file, a session phase file or a raw snapshot — and names the cheaper route in the refusal; a
|
|
64
|
+
`Read` with `offset` or `limit` passes. `write-guard.mjs` refuses a `Write` or `Edit` under
|
|
65
|
+
`output/` whose content matches the redaction list, so a secret never reaches disk. Neither
|
|
66
|
+
one looks at anything else in your project. `QUALIOW_HOOKS=off` disables both.
|
|
67
|
+
|
|
68
|
+
To write it by hand instead, this is the whole of it:
|
|
69
|
+
|
|
70
|
+
```json
|
|
71
|
+
{
|
|
72
|
+
"hooks": {
|
|
73
|
+
"PreToolUse": [
|
|
74
|
+
{
|
|
75
|
+
"matcher": "Read",
|
|
76
|
+
"hooks": [
|
|
77
|
+
{ "type": "command", "command": "node \"$CLAUDE_PROJECT_DIR/qa/hooks/read-guard.mjs\"" }
|
|
78
|
+
]
|
|
79
|
+
},
|
|
80
|
+
{
|
|
81
|
+
"matcher": "Write|Edit|MultiEdit",
|
|
82
|
+
"hooks": [
|
|
83
|
+
{ "type": "command", "command": "node \"$CLAUDE_PROJECT_DIR/qa/hooks/write-guard.mjs\"" }
|
|
84
|
+
]
|
|
85
|
+
}
|
|
86
|
+
]
|
|
87
|
+
},
|
|
88
|
+
"permissions": {
|
|
89
|
+
"allow": [
|
|
90
|
+
"Bash(playwright-cli:*)",
|
|
91
|
+
"Bash(npx playwright-cli:*)",
|
|
92
|
+
"Bash(qualiow:*)"
|
|
93
|
+
]
|
|
94
|
+
},
|
|
95
|
+
"env": {
|
|
96
|
+
"QUALIOW_READ_MAX_LINES": "300"
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### Or install it as a Claude Code plugin
|
|
102
|
+
|
|
103
|
+
The repository is its own Claude Code marketplace, so an install is two commands:
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
claude plugin marketplace add willcoliveira/qualiow-exploratory-testing-skills
|
|
107
|
+
claude plugin install qualiow@qualiow
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Skills then appear namespaced — `/qualiow:qa-explore`, `/qualiow:qa-verify-backend` — and the
|
|
111
|
+
data files resolve from `${CLAUDE_PLUGIN_ROOT}/data/` unless the project has its own `data/`
|
|
112
|
+
or `qa/` directory, which wins. Pointing Claude Code at a local checkout
|
|
113
|
+
(`claude --plugin-dir /path/to/qualiow-exploratory-testing-skills`) is the development path
|
|
114
|
+
and behaves the same, except that `bin/` is not put on `PATH`, so the helpers are called by
|
|
115
|
+
their full path (`${CLAUDE_PLUGIN_ROOT}/bin/mcli`).
|
|
116
|
+
|
|
117
|
+
A plugin install has no `dist/`, so the `qualiow` CLI runs through the `bin/qualiow` launcher
|
|
118
|
+
shim: it uses the local build when the checkout has one, and otherwise fetches the published
|
|
119
|
+
npm package with `npx`, pinned to the version in `.claude-plugin/plugin.json`. The first run
|
|
120
|
+
of that path needs network. A plain git checkout is described in the project README.
|
|
52
121
|
|
|
53
122
|
## 2. Run your first session
|
|
54
123
|
|
|
@@ -66,10 +135,13 @@ Claude will:
|
|
|
66
135
|
1. Resolve the target config and credentials, and create the session directory
|
|
67
136
|
2. Authenticate if the target requires it
|
|
68
137
|
3. Write a charter — what is being tested, why, and what "done" looks like
|
|
69
|
-
4.
|
|
138
|
+
4. Load the heuristics that apply to this domain with `qualiow kb digest --for explore
|
|
139
|
+
--domain <domain>`, rather than reading the manifest and the entry files whole
|
|
70
140
|
5. Explore across eight phases, saving findings to disk between each
|
|
71
141
|
6. File one bug report per finding in `bugs/`
|
|
72
|
-
7. Write `
|
|
142
|
+
7. Write `phase-7-notes.md` — the executive summary, coverage map, recommendations and
|
|
143
|
+
reflection — then hand the assembly of `session-report.md` and the
|
|
144
|
+
`qualiow session finalize <session-dir>` call to the `qa-reporting-agent` sub-agent
|
|
73
145
|
|
|
74
146
|
The eight phases and their share of the 45-minute cap:
|
|
75
147
|
|
|
@@ -84,6 +156,16 @@ The eight phases and their share of the 45-minute cap:
|
|
|
84
156
|
| 6 | Edge cases | 6 | Boundaries, negative input, security, "what is missing?" |
|
|
85
157
|
| 7 | Reporting | 4 | Reflection, coverage map, session report, cleanup |
|
|
86
158
|
|
|
159
|
+
Three points in that run are not done by the session itself. At setup the knowledge base
|
|
160
|
+
arrives as a `qualiow kb digest` call rather than as the manifest and the entry files. In
|
|
161
|
+
discovery, a page whose raw accessibility tree runs past 300 lines is saved to
|
|
162
|
+
`snapshots/` and handed to the `qa-page-mapper-agent` sub-agent, which returns the forms,
|
|
163
|
+
navigation, controls and error text as a short map — the session works from that instead of
|
|
164
|
+
the tree, and keeps driving the page itself. At the end, the session writes its judgements
|
|
165
|
+
into `phase-7-notes.md` and the `qa-reporting-agent` sub-agent assembles `session-report.md`
|
|
166
|
+
around them and runs `finalize`; the bug reports, `stats.json` and every word of the summary
|
|
167
|
+
and reflection are still the session's own.
|
|
168
|
+
|
|
87
169
|
### Option B: quick check on one page
|
|
88
170
|
|
|
89
171
|
```bash
|
|
@@ -99,7 +181,10 @@ session, so reports, feedback and cleanup all work on it.
|
|
|
99
181
|
/qa-gather
|
|
100
182
|
```
|
|
101
183
|
|
|
102
|
-
Point it at a ticket, a PR, a design doc, a URL or pasted text. It
|
|
184
|
+
Point it at a ticket, a PR, a design doc, a URL or pasted text. It runs in the
|
|
185
|
+
`qa-gather-agent` sub-agent, which sees only the invocation, so put the paths, the URLs or the
|
|
186
|
+
pasted text in the same message — it cannot come back with a question, and anything a source
|
|
187
|
+
leaves unsaid is marked `[GAP]` or `[ASSUMPTION]`. It writes
|
|
103
188
|
`output/context/<TICKET>-context.md`, which you then pass to a session:
|
|
104
189
|
|
|
105
190
|
```bash
|
|
@@ -121,11 +206,13 @@ output/sessions/2026-09-08-1813-explore-parabank/
|
|
|
121
206
|
phase-4-journeys.md
|
|
122
207
|
phase-5-features.md
|
|
123
208
|
phase-6-edge-cases.md
|
|
209
|
+
phase-7-notes.md # the session's own summary, coverage map, recommendations
|
|
124
210
|
bugs/
|
|
125
211
|
BUG-001.md # one bug = one report
|
|
126
212
|
BUG-002.md
|
|
127
213
|
screenshots/
|
|
128
214
|
BUG-001.png
|
|
215
|
+
snapshots/ # raw page snapshots — working files, never part of the report
|
|
129
216
|
videos/
|
|
130
217
|
session-report.md # the deliverable
|
|
131
218
|
stats.json # machine-readable session metrics
|
|
@@ -137,6 +224,19 @@ Plus, at the top level:
|
|
|
137
224
|
(`| Date | Kind | Target | Bugs | Duration | Status | Report |`)
|
|
138
225
|
- `output/bugs/all-bugs.md` — every bug across every session
|
|
139
226
|
|
|
227
|
+
Both are written by `qualiow session finalize`, which the session runs as its last step. It
|
|
228
|
+
checks the session against the output contract first — a confidentiality header on every
|
|
229
|
+
markdown file, `stats.json` in the metrics shape, no unredacted secret anywhere — and refuses
|
|
230
|
+
to append rows until the violations it lists are fixed. Run it by hand at any time:
|
|
231
|
+
|
|
232
|
+
```bash
|
|
233
|
+
qualiow session finalize latest --check # report violations, write nothing
|
|
234
|
+
qualiow session finalize latest --redact # rewrite files that still carry a secret
|
|
235
|
+
qualiow session finalize latest # validate, then append the rows
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
Finalizing twice changes nothing — the rows are appended once.
|
|
239
|
+
|
|
140
240
|
### Bug report structure
|
|
141
241
|
|
|
142
242
|
Each report opens with the confidentiality blockquote, then:
|
|
@@ -266,14 +366,33 @@ indexed by `data/knowledge/manifest.yml`.
|
|
|
266
366
|
Entries are YAML files in `data/knowledge/releases/<version>/entries/`. After editing by hand,
|
|
267
367
|
run `qualiow kb sync` to regenerate the manifest and `qualiow kb check` to verify it.
|
|
268
368
|
|
|
369
|
+
A session does not read the manifest or the entry files. It calls `qualiow kb digest`, which
|
|
370
|
+
prints the entries that apply — the always-load set plus whatever the domain, the tags and the
|
|
371
|
+
skill select — as a few lines each:
|
|
372
|
+
|
|
373
|
+
```bash
|
|
374
|
+
qualiow kb digest --for explore --domain fintech # what a session loads at setup
|
|
375
|
+
qualiow kb digest --for backend # /qa-verify-backend's set
|
|
376
|
+
qualiow kb digest --tag security data-integrity # one or more tags
|
|
377
|
+
qualiow kb digest --entry heuristic-test-tours # one entry, in full
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
`--for` is driven by the optional `loading_strategy.by_skill` block in the manifest, so you can
|
|
381
|
+
decide which entries a skill always gets without touching the skill itself.
|
|
382
|
+
|
|
269
383
|
## 6. CLI commands
|
|
270
384
|
|
|
271
385
|
```bash
|
|
272
386
|
qualiow init --dry-run # preview what init would write
|
|
387
|
+
qualiow init --hooks # add the read/write guards to .claude/settings.json
|
|
273
388
|
qualiow validate --all # targets + qa/target.yml + domains + knowledge base
|
|
274
389
|
qualiow list sessions # also: knowledge | targets | domains
|
|
390
|
+
qualiow list knowledge --stats # also: --domain --tag --type --entry <id> --changelog
|
|
275
391
|
qualiow report -s latest -f html -o report.html
|
|
276
392
|
qualiow kb check
|
|
393
|
+
qualiow kb digest --for explore # the knowledge a session loads
|
|
394
|
+
qualiow session finalize latest # validate a finished session and index it
|
|
395
|
+
qualiow session prune --older-than 30 # dry-run; add --yes to remove
|
|
277
396
|
```
|
|
278
397
|
|
|
279
398
|
`qualiow explore [url]` is **pre-flight only**: it validates the inputs, creates the session
|
package/hooks/hooks.json
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
{
|
|
2
|
+
"description": "qualiow routing guards: whole-file reads of qualiow data over the line threshold, and secrets in session output",
|
|
3
|
+
"hooks": {
|
|
4
|
+
"PreToolUse": [
|
|
5
|
+
{
|
|
6
|
+
"matcher": "Read",
|
|
7
|
+
"hooks": [
|
|
8
|
+
{
|
|
9
|
+
"type": "command",
|
|
10
|
+
"command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/scripts/read-guard.mjs\"",
|
|
11
|
+
"timeout": 10
|
|
12
|
+
}
|
|
13
|
+
]
|
|
14
|
+
},
|
|
15
|
+
{
|
|
16
|
+
"matcher": "Write|Edit|MultiEdit",
|
|
17
|
+
"hooks": [
|
|
18
|
+
{
|
|
19
|
+
"type": "command",
|
|
20
|
+
"command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/scripts/write-guard.mjs\"",
|
|
21
|
+
"timeout": 10
|
|
22
|
+
}
|
|
23
|
+
]
|
|
24
|
+
}
|
|
25
|
+
]
|
|
26
|
+
}
|
|
27
|
+
}
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* PreToolUse hook (matcher: Read).
|
|
4
|
+
*
|
|
5
|
+
* Denies a whole-file Read of a qualiow data/session file once it crosses a
|
|
6
|
+
* line threshold, and names the cheaper route instead (`qualiow kb digest`,
|
|
7
|
+
* `qualiow list knowledge --entry`, offset/limit + Grep, or the
|
|
8
|
+
* qa-page-mapper-agent) — see delegation-rules.md. A Read that already
|
|
9
|
+
* carries `offset`/`limit` is always allowed: the caller is already reading
|
|
10
|
+
* a slice.
|
|
11
|
+
*
|
|
12
|
+
* Node built-ins only — a git-sourced plugin install has neither `dist/` nor
|
|
13
|
+
* `node_modules`. Never throws and never blocks on its own failure: any
|
|
14
|
+
* error here allows the read through (exit 0, no stdout) rather than
|
|
15
|
+
* breaking the caller because of a hook bug.
|
|
16
|
+
*/
|
|
17
|
+
import { readFileSync, existsSync } from 'node:fs';
|
|
18
|
+
import { isAbsolute, resolve } from 'node:path';
|
|
19
|
+
|
|
20
|
+
const GUARDED = [
|
|
21
|
+
{
|
|
22
|
+
re: /(^|\/)data\/knowledge\/(manifest\.ya?ml|releases\/)/,
|
|
23
|
+
route:
|
|
24
|
+
'knowledge manifest or release entry \u2192 `qualiow kb digest --for <explore|backend|mobile>` or `qualiow list knowledge --entry <id>`',
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
re: /(^|\/)output\/sessions\/[^/]+\/phase-\d[^/]*\.md/,
|
|
28
|
+
route: "phase file \u2192 Read with offset/limit after Grep '^## '",
|
|
29
|
+
},
|
|
30
|
+
{
|
|
31
|
+
re: /(^|\/)output\/sessions\/[^/]+\/snapshots\//,
|
|
32
|
+
route: 'raw snapshot \u2192 qa-page-mapper-agent',
|
|
33
|
+
},
|
|
34
|
+
];
|
|
35
|
+
|
|
36
|
+
function deny(reason) {
|
|
37
|
+
process.stdout.write(
|
|
38
|
+
JSON.stringify({
|
|
39
|
+
hookSpecificOutput: {
|
|
40
|
+
hookEventName: 'PreToolUse',
|
|
41
|
+
permissionDecision: 'deny',
|
|
42
|
+
permissionDecisionReason: reason,
|
|
43
|
+
},
|
|
44
|
+
}),
|
|
45
|
+
);
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
/** Number of \n plus one if the last line has no trailing newline. */
|
|
49
|
+
function countLines(text) {
|
|
50
|
+
if (text.length === 0) return 0;
|
|
51
|
+
const newlines = (text.match(/\n/g) || []).length;
|
|
52
|
+
return text.endsWith('\n') ? newlines : newlines + 1;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
try {
|
|
56
|
+
if (process.env.QUALIOW_HOOKS === 'off') process.exit(0);
|
|
57
|
+
|
|
58
|
+
let raw = '';
|
|
59
|
+
try {
|
|
60
|
+
raw = readFileSync(0, 'utf-8');
|
|
61
|
+
} catch {
|
|
62
|
+
process.exit(0);
|
|
63
|
+
}
|
|
64
|
+
if (!raw || !raw.trim()) process.exit(0);
|
|
65
|
+
|
|
66
|
+
const input = JSON.parse(raw);
|
|
67
|
+
if (input.tool_name !== 'Read') process.exit(0);
|
|
68
|
+
|
|
69
|
+
const toolInput = input.tool_input || {};
|
|
70
|
+
if (toolInput.offset != null || toolInput.limit != null) process.exit(0);
|
|
71
|
+
|
|
72
|
+
const filePath = toolInput.file_path;
|
|
73
|
+
if (!filePath || typeof filePath !== 'string') process.exit(0);
|
|
74
|
+
|
|
75
|
+
const normalized = filePath.split('\\').join('/');
|
|
76
|
+
const matched = GUARDED.find((g) => g.re.test(normalized));
|
|
77
|
+
if (!matched) process.exit(0);
|
|
78
|
+
|
|
79
|
+
const abs = isAbsolute(filePath) ? filePath : resolve(input.cwd || process.cwd(), filePath);
|
|
80
|
+
if (!existsSync(abs)) process.exit(0);
|
|
81
|
+
|
|
82
|
+
const content = readFileSync(abs, 'utf-8');
|
|
83
|
+
const lineCount = countLines(content);
|
|
84
|
+
const threshold = Number(process.env.QUALIOW_READ_MAX_LINES) || 300;
|
|
85
|
+
|
|
86
|
+
if (lineCount > threshold) {
|
|
87
|
+
deny(
|
|
88
|
+
`${filePath} is ${lineCount} lines (limit ${threshold}). Route: ${matched.route}. ` +
|
|
89
|
+
'Set QUALIOW_HOOKS=off to disable.',
|
|
90
|
+
);
|
|
91
|
+
}
|
|
92
|
+
process.exit(0);
|
|
93
|
+
} catch {
|
|
94
|
+
process.exit(0);
|
|
95
|
+
}
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
export interface SecretPattern {
|
|
2
|
+
category: string;
|
|
3
|
+
re: RegExp;
|
|
4
|
+
filter?: (match: string) => boolean;
|
|
5
|
+
}
|
|
6
|
+
export const SECRET_PATTERNS: SecretPattern[];
|
|
7
|
+
/** Category names of every secret-shaped pattern found in `text`; never the matched text. */
|
|
8
|
+
export function findSecretCategories(text: string): string[];
|
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Secret detection for the write-guard hook.
|
|
3
|
+
*
|
|
4
|
+
* Mirrors the categories and patterns in `src/utils/redact.ts` as closely as
|
|
5
|
+
* possible, so a hook denial matches what `qualiow session finalize --redact`
|
|
6
|
+
* would flag on the same text. Kept as a standalone, dependency-free ES
|
|
7
|
+
* module (Node built-ins only, no TS import) because a git-sourced plugin
|
|
8
|
+
* install has neither `dist/` nor `node_modules` to pull the real util from.
|
|
9
|
+
*
|
|
10
|
+
* `findSecretCategories` never returns the matched text itself — only
|
|
11
|
+
* category names — so the hook that calls it can safely put the result in a
|
|
12
|
+
* denial reason without leaking the secret it found.
|
|
13
|
+
*/
|
|
14
|
+
|
|
15
|
+
const EMAIL_ALLOW = /(?:@|\.)(?:example\.(?:com|org|net)|localhost)$/i;
|
|
16
|
+
|
|
17
|
+
function luhnValid(digits) {
|
|
18
|
+
let sum = 0;
|
|
19
|
+
let alt = false;
|
|
20
|
+
for (let i = digits.length - 1; i >= 0; i--) {
|
|
21
|
+
const d0 = digits.charCodeAt(i) - 48;
|
|
22
|
+
if (d0 < 0 || d0 > 9) return false;
|
|
23
|
+
let d = d0;
|
|
24
|
+
if (alt) {
|
|
25
|
+
d *= 2;
|
|
26
|
+
if (d > 9) d -= 9;
|
|
27
|
+
}
|
|
28
|
+
sum += d;
|
|
29
|
+
alt = !alt;
|
|
30
|
+
}
|
|
31
|
+
return sum % 10 === 0;
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* category: name surfaced in denial reasons (never the matched text).
|
|
36
|
+
* re: detection pattern (mirrors redact.ts; capture groups dropped since
|
|
37
|
+
* only presence, not replacement, is needed here).
|
|
38
|
+
* filter(match): optional extra check a raw regex match can't express
|
|
39
|
+
* (Luhn validity, the example.com/localhost allow-list).
|
|
40
|
+
*/
|
|
41
|
+
export const SECRET_PATTERNS = [
|
|
42
|
+
{
|
|
43
|
+
category: 'Private key',
|
|
44
|
+
re: /-----BEGIN (?:[A-Z0-9 ]+ )?PRIVATE KEY-----[\s\S]*?-----END (?:[A-Z0-9 ]+ )?PRIVATE KEY-----/g,
|
|
45
|
+
},
|
|
46
|
+
{
|
|
47
|
+
category: 'JWT',
|
|
48
|
+
re: /\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{6,}\.[A-Za-z0-9_-]{6,}\b/g,
|
|
49
|
+
},
|
|
50
|
+
{
|
|
51
|
+
category: 'Authorization header',
|
|
52
|
+
re: /authorization\s*[:=]\s*\S[^\n\r]*/gi,
|
|
53
|
+
},
|
|
54
|
+
{
|
|
55
|
+
category: 'Cookie header',
|
|
56
|
+
re: /^(?:set-cookie|cookie)\s*:\s*.+$/gim,
|
|
57
|
+
},
|
|
58
|
+
{
|
|
59
|
+
category: 'AWS access key id',
|
|
60
|
+
re: /\b(?:AKIA|ASIA|AROA|AIDA)[0-9A-Z]{16}\b/g,
|
|
61
|
+
},
|
|
62
|
+
{
|
|
63
|
+
category: 'AWS secret access key',
|
|
64
|
+
re: /aws_secret_access_key\s*[:=]\s*\S+/gi,
|
|
65
|
+
},
|
|
66
|
+
{
|
|
67
|
+
category: 'OpenAI-style key',
|
|
68
|
+
re: /\bsk-(?:proj-)?[A-Za-z0-9_-]{20,}\b/g,
|
|
69
|
+
},
|
|
70
|
+
{
|
|
71
|
+
category: 'GitHub token',
|
|
72
|
+
re: /\b(?:gh[pousr]_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,})\b/g,
|
|
73
|
+
},
|
|
74
|
+
{
|
|
75
|
+
category: 'Slack token',
|
|
76
|
+
re: /\bxox[abposr]-[A-Za-z0-9-]{10,}\b/g,
|
|
77
|
+
},
|
|
78
|
+
{
|
|
79
|
+
category: 'API key',
|
|
80
|
+
re: /(?:x-api-key|api[_-]?key)\s*[:=]\s*["']?[A-Za-z0-9._-]{8,}["']?/gi,
|
|
81
|
+
},
|
|
82
|
+
{
|
|
83
|
+
category: 'Password',
|
|
84
|
+
re: /(?:password|passwd|pwd)\s*[:=]\s*["']?[^\s,}"']+["']?/gi,
|
|
85
|
+
},
|
|
86
|
+
{
|
|
87
|
+
category: 'Token',
|
|
88
|
+
re: /(?:access[_-]?token|refresh[_-]?token|token)\s*[:=]\s*["']?[A-Za-z0-9._-]{8,}["']?/gi,
|
|
89
|
+
},
|
|
90
|
+
{
|
|
91
|
+
category: 'Secret',
|
|
92
|
+
re: /[a-z_]*secret\s*[:=]\s*["']?[^\s,}"']+["']?/gi,
|
|
93
|
+
},
|
|
94
|
+
{
|
|
95
|
+
category: 'SSN',
|
|
96
|
+
re: /\b\d{3}-\d{2}-\d{4}\b/g,
|
|
97
|
+
},
|
|
98
|
+
{
|
|
99
|
+
category: 'Email',
|
|
100
|
+
re: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g,
|
|
101
|
+
filter: (match) => !EMAIL_ALLOW.test(match),
|
|
102
|
+
},
|
|
103
|
+
{
|
|
104
|
+
category: 'Credit card number',
|
|
105
|
+
re: /\b\d(?:[ -]?\d){12,18}\b/g,
|
|
106
|
+
filter: (match) => {
|
|
107
|
+
const digits = match.replace(/\D/g, '');
|
|
108
|
+
if (digits.length < 13 || digits.length > 19) return false;
|
|
109
|
+
if (!/^[3-6]/.test(digits)) return false;
|
|
110
|
+
return luhnValid(digits);
|
|
111
|
+
},
|
|
112
|
+
},
|
|
113
|
+
];
|
|
114
|
+
|
|
115
|
+
/** Returns the category names of every secret-shaped pattern found in `text`. */
|
|
116
|
+
export function findSecretCategories(text) {
|
|
117
|
+
const found = new Set();
|
|
118
|
+
for (const { category, re, filter } of SECRET_PATTERNS) {
|
|
119
|
+
re.lastIndex = 0;
|
|
120
|
+
let match;
|
|
121
|
+
while ((match = re.exec(text)) !== null) {
|
|
122
|
+
// A value that is already a redaction placeholder is clean: redact.ts leaves it
|
|
123
|
+
// untouched, so containsSecrets() is false for it and this must agree.
|
|
124
|
+
if (/REDACTED\]/.test(match[0])) {
|
|
125
|
+
if (match[0].length === 0) re.lastIndex += 1;
|
|
126
|
+
continue;
|
|
127
|
+
}
|
|
128
|
+
if (!filter || filter(match[0])) {
|
|
129
|
+
found.add(category);
|
|
130
|
+
break;
|
|
131
|
+
}
|
|
132
|
+
if (match[0].length === 0) re.lastIndex += 1;
|
|
133
|
+
}
|
|
134
|
+
}
|
|
135
|
+
return [...found];
|
|
136
|
+
}
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* PreToolUse hook (matcher: Write|Edit|MultiEdit).
|
|
4
|
+
*
|
|
5
|
+
* Denies writing a secret-shaped string into anything under `output/`
|
|
6
|
+
* (excluding `output/**\/snapshots/`, which never ships a report and is
|
|
7
|
+
* excluded from the secrets scan the same way `qualiow session finalize`
|
|
8
|
+
* excludes it). Mechanises security rule 3 (redact before disk) instead of
|
|
9
|
+
* relying on the model remembering it.
|
|
10
|
+
*
|
|
11
|
+
* Node built-ins only, plus the sibling `secret-patterns.mjs` — a
|
|
12
|
+
* git-sourced plugin install has neither `dist/` nor `node_modules`. Never
|
|
13
|
+
* throws and never echoes the matched text back: the denial reason names
|
|
14
|
+
* categories only.
|
|
15
|
+
*/
|
|
16
|
+
import { readFileSync } from 'node:fs';
|
|
17
|
+
import { findSecretCategories } from './secret-patterns.mjs';
|
|
18
|
+
|
|
19
|
+
const WRITE_TOOLS = new Set(['Write', 'Edit', 'MultiEdit']);
|
|
20
|
+
|
|
21
|
+
function deny(reason) {
|
|
22
|
+
process.stdout.write(
|
|
23
|
+
JSON.stringify({
|
|
24
|
+
hookSpecificOutput: {
|
|
25
|
+
hookEventName: 'PreToolUse',
|
|
26
|
+
permissionDecision: 'deny',
|
|
27
|
+
permissionDecisionReason: reason,
|
|
28
|
+
},
|
|
29
|
+
}),
|
|
30
|
+
);
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
function collectText(toolName, toolInput) {
|
|
34
|
+
if (!toolInput) return '';
|
|
35
|
+
if (toolName === 'Write') {
|
|
36
|
+
return typeof toolInput.content === 'string' ? toolInput.content : '';
|
|
37
|
+
}
|
|
38
|
+
if (toolName === 'Edit') {
|
|
39
|
+
return typeof toolInput.new_string === 'string' ? toolInput.new_string : '';
|
|
40
|
+
}
|
|
41
|
+
if (toolName === 'MultiEdit') {
|
|
42
|
+
const edits = Array.isArray(toolInput.edits) ? toolInput.edits : [];
|
|
43
|
+
return edits
|
|
44
|
+
.map((e) => (e && typeof e.new_string === 'string' ? e.new_string : ''))
|
|
45
|
+
.join('\n');
|
|
46
|
+
}
|
|
47
|
+
return '';
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
try {
|
|
51
|
+
if (process.env.QUALIOW_HOOKS === 'off') process.exit(0);
|
|
52
|
+
|
|
53
|
+
let raw = '';
|
|
54
|
+
try {
|
|
55
|
+
raw = readFileSync(0, 'utf-8');
|
|
56
|
+
} catch {
|
|
57
|
+
process.exit(0);
|
|
58
|
+
}
|
|
59
|
+
if (!raw || !raw.trim()) process.exit(0);
|
|
60
|
+
|
|
61
|
+
const input = JSON.parse(raw);
|
|
62
|
+
if (!WRITE_TOOLS.has(input.tool_name)) process.exit(0);
|
|
63
|
+
|
|
64
|
+
const toolInput = input.tool_input || {};
|
|
65
|
+
const filePath = toolInput.file_path;
|
|
66
|
+
if (!filePath || typeof filePath !== 'string') process.exit(0);
|
|
67
|
+
|
|
68
|
+
const normalized = filePath.split('\\').join('/');
|
|
69
|
+
if (!/(^|\/)output\//.test(normalized)) process.exit(0);
|
|
70
|
+
if (normalized.includes('/snapshots/')) process.exit(0);
|
|
71
|
+
|
|
72
|
+
const text = collectText(input.tool_name, toolInput);
|
|
73
|
+
if (!text) process.exit(0);
|
|
74
|
+
|
|
75
|
+
const categories = findSecretCategories(text);
|
|
76
|
+
if (categories.length > 0) {
|
|
77
|
+
deny(
|
|
78
|
+
`Refusing to write ${filePath}: it contains ${categories.join(', ')}. ` +
|
|
79
|
+
'Redact per security-rules.md ([REDACTED]) before writing; ' +
|
|
80
|
+
'qualiow session finalize --redact rewrites an existing file. ' +
|
|
81
|
+
'Set QUALIOW_HOOKS=off to disable.',
|
|
82
|
+
);
|
|
83
|
+
}
|
|
84
|
+
process.exit(0);
|
|
85
|
+
} catch {
|
|
86
|
+
process.exit(0);
|
|
87
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "qualiow-exploratory-testing",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.2.0",
|
|
4
4
|
"description": "qualiow — AI-powered exploratory testing skills for Claude Code: web via playwright-cli, iOS/Android simulators via Maestro/mobile-cli, and backend/API acceptance-criteria verification",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -18,6 +18,7 @@
|
|
|
18
18
|
"skills/",
|
|
19
19
|
"agents/",
|
|
20
20
|
"bin/",
|
|
21
|
+
"hooks/",
|
|
21
22
|
".claude-plugin/",
|
|
22
23
|
"data/domains/*.yml",
|
|
23
24
|
"data/knowledge/",
|
|
@@ -45,7 +46,7 @@
|
|
|
45
46
|
"check:mirror": "diff -rq .claude/skills skills && diff -rq .claude/agents agents",
|
|
46
47
|
"check:pack": "node scripts/check-pack.mjs",
|
|
47
48
|
"lint": "tsc --noEmit",
|
|
48
|
-
"version": "node scripts/sync-version.mjs && git add .claude-plugin/plugin.json",
|
|
49
|
+
"version": "node scripts/sync-version.mjs && git add .claude-plugin/plugin.json .claude-plugin/marketplace.json",
|
|
49
50
|
"prepublishOnly": "npm run build && npm run lint && npm test && npm run check:pack"
|
|
50
51
|
},
|
|
51
52
|
"keywords": [
|
|
@@ -8,7 +8,7 @@ description: >
|
|
|
8
8
|
Use when user says: "explore", "test this site", "find bugs", "QA check", "exploratory session",
|
|
9
9
|
or provides a URL to test.
|
|
10
10
|
argument-hint: "<url> [--target <id>] [--context <file>] [--focus <area>] [--session <dir>]"
|
|
11
|
-
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(npx playwright:*), Bash(diff:*), Read, Write, Glob, Grep
|
|
11
|
+
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(npx playwright:*), Bash(qualiow:*), Bash(npx:*), Bash(wc:*), Bash(diff:*), Read, Write, Glob, Grep
|
|
12
12
|
---
|
|
13
13
|
|
|
14
14
|
# Exploratory Testing Session
|
|
@@ -73,5 +73,6 @@ Consult these as needed throughout the session:
|
|
|
73
73
|
- **`${CLAUDE_SKILL_DIR}/references/security-rules.md`** — prompt-injection resistance, production rule, redaction list, session isolation, output classification
|
|
74
74
|
- **`${CLAUDE_SKILL_DIR}/references/severity-guide.md`** — severity definitions, the "when in doubt go LOWER" rule, risk matrix, priority table
|
|
75
75
|
- **`${CLAUDE_SKILL_DIR}/references/session-rules.md`** — 20 testing-discipline rules: 45-min cap, one bug one report, evidence, risk-proportional time, adapt mid-session, log WHY, AI-bias avoidance
|
|
76
|
+
- **`${CLAUDE_SKILL_DIR}/references/delegation-rules.md`** — what the `qualiow` CLI does instead of you, the never-delegate list (severity, impact, bug reports, verdicts, reflection) and the read thresholds
|
|
76
77
|
- **`${CLAUDE_SKILL_DIR}/references/context-integration.md`** — how `--context` changes the session (context file, inline context, blind mode)
|
|
77
78
|
- **`${CLAUDE_SKILL_DIR}/references/playwright-agents-integration.md`** — opt-in handoff to Playwright Test Agents (planner / generator / healer, Playwright 1.56+) when a reproducible bug should become a regression test; requires the optional `@playwright/test` peer dependency
|
|
@@ -58,21 +58,29 @@ CHANGE SCOPE: [what changed or "unknown -- full exploration"]
|
|
|
58
58
|
|
|
59
59
|
## Step 3: Load Knowledge
|
|
60
60
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
2. Domain-specific entries from `loading_strategy.by_domain`
|
|
64
|
-
3. `<data>/knowledge/learned-patterns.md`: the false-positive skip list and must-check patterns
|
|
65
|
-
4. Business-logic and race-condition techniques (if the app has transactions or state changes)
|
|
61
|
+
Run the digest — it selects the always-loaded heuristics, the domain entries, the tagged
|
|
62
|
+
techniques and the learned-pattern lead-ins, and prints a summary of each:
|
|
66
63
|
|
|
67
|
-
|
|
64
|
+
```bash
|
|
65
|
+
qualiow kb digest --for explore --domain <domain>
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Add `--tag security` when the target has authentication, and `--tag data-integrity` when the
|
|
69
|
+
app has transactions or any state change. Resolve the `qualiow` prefix first (`${CLAUDE_SKILL_DIR}/references/paths.md`)
|
|
70
|
+
and write it literally.
|
|
71
|
+
|
|
72
|
+
Read **only the digest**. When you need a heuristic's full question list, fetch that one entry
|
|
73
|
+
— `qualiow list knowledge --entry <id>`, or `Grep` the release file for the heading. Never
|
|
74
|
+
`Read` `<data>/knowledge/manifest.yml` or a release entry whole: see
|
|
75
|
+
`${CLAUDE_SKILL_DIR}/references/delegation-rules.md`.
|
|
68
76
|
|
|
69
77
|
## Step 4: Create the Session Directory
|
|
70
78
|
|
|
71
79
|
Unless `--session <dir>` was given, create
|
|
72
80
|
`output/sessions/<YYYY-MM-DD-HHmm>-explore-<slug>/` (scheme in `paths.md`; `<slug>` is the
|
|
73
81
|
target id without a leading `_`, or the hostname with dots replaced by dashes) containing
|
|
74
|
-
`screenshots/`, `bugs/`, `videos/`, and an empty `session-log.md`. With
|
|
75
|
-
that directory and append to its `session-log.md`.
|
|
82
|
+
`screenshots/`, `bugs/`, `videos/`, `snapshots/`, and an empty `session-log.md`. With
|
|
83
|
+
`--session`, reuse that directory and append to its `session-log.md`.
|
|
76
84
|
|
|
77
85
|
Choose the playwright-cli session id `-s=explore-<HHmm>-<slug>`. **Every `playwright-cli`
|
|
78
86
|
call in this session carries `-s=<sid>`; the examples in the phase files omit the prefix.**
|