qualiow-exploratory-testing 2.0.0 → 2.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +26 -0
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +83 -0
- package/README.md +123 -12
- package/agents/qa-diff-indexer-agent.md +81 -0
- package/agents/qa-gather-agent.md +21 -4
- package/agents/qa-page-mapper-agent.md +74 -0
- package/agents/qa-reporting-agent.md +103 -0
- package/bin/qualiow +49 -0
- package/data/knowledge/manifest.yml +22 -0
- package/dist/cli/index.js +1111 -62
- package/dist/index.d.ts +91 -2
- package/dist/index.js +532 -48
- package/docs/GETTING-STARTED.md +127 -8
- package/hooks/hooks.json +27 -0
- package/hooks/scripts/read-guard.mjs +95 -0
- package/hooks/scripts/secret-patterns.d.mts +8 -0
- package/hooks/scripts/secret-patterns.mjs +136 -0
- package/hooks/scripts/write-guard.mjs +87 -0
- package/package.json +3 -2
- package/skills/qa-explore/SKILL.md +2 -1
- package/skills/qa-explore/phases/00-setup.md +16 -8
- package/skills/qa-explore/phases/03-discovery.md +48 -6
- package/skills/qa-explore/phases/07-reporting.md +36 -15
- package/skills/qa-explore/references/delegation-rules.md +82 -0
- package/skills/qa-explore/references/output-contract.md +7 -2
- package/skills/qa-explore/references/paths.md +25 -1
- package/skills/qa-explore/references/security-rules.md +4 -0
- package/skills/qa-explore/references/session-rules.md +2 -1
- package/skills/qa-explore-cleanup/SKILL.md +44 -19
- package/skills/qa-explore-feedback/SKILL.md +5 -0
- package/skills/qa-explore-mobile/SKILL.md +1 -1
- package/skills/qa-explore-mobile/phases/07-reporting.md +33 -11
- package/skills/qa-explore-quick/SKILL.md +12 -2
- package/skills/qa-explore-report/SKILL.md +29 -5
- package/skills/qa-gather/SKILL.md +30 -9
- package/skills/qa-knowledge-add/SKILL.md +10 -5
- package/skills/qa-knowledge-list/SKILL.md +28 -40
- package/skills/qa-verify-backend/SKILL.md +7 -4
- package/skills/qa-verify-backend/phases/02-static-review.md +11 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +3 -1
- package/skills/qa-verify-backend/phases/03b-api-verification.md +3 -1
- package/skills/qa-verify-backend/phases/05-reporting.md +51 -19
|
@@ -6,21 +6,63 @@
|
|
|
6
6
|
|
|
7
7
|
```bash
|
|
8
8
|
playwright-cli goto <start_url>
|
|
9
|
-
playwright-cli snapshot
|
|
9
|
+
playwright-cli snapshot --depth=3
|
|
10
10
|
playwright-cli console error
|
|
11
11
|
playwright-cli requests
|
|
12
12
|
```
|
|
13
13
|
|
|
14
14
|
For each page:
|
|
15
|
-
1. `playwright-cli snapshot` — note structure, forms, interactive elements.
|
|
15
|
+
1. `playwright-cli snapshot --depth=3` — note structure, forms, interactive elements.
|
|
16
16
|
2. Click through navigation to discover all sections.
|
|
17
17
|
3. Log: `[DISCOVERED] <url> -- <description> -- Risk: <P0/P1/P2/P3>`
|
|
18
18
|
4. Check `playwright-cli console error` and `playwright-cli requests` on each page.
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
20
|
+
## Snapshot Policy
|
|
21
|
+
|
|
22
|
+
A full accessibility tree is the single largest thing this session can read. On discovery
|
|
23
|
+
pages you take the shape, not the tree:
|
|
24
|
+
|
|
25
|
+
- `playwright-cli snapshot --depth=3` — the default. Enough to see sections, forms and nav.
|
|
26
|
+
- `playwright-cli snapshot <ref>` — zoom into one region once `--depth=3` shows you where to look.
|
|
27
|
+
- `playwright-cli find "<text>"` (or `find --regex "<re>"`) — locate a label, message or
|
|
28
|
+
control by its text, without a tree at all.
|
|
29
|
+
|
|
30
|
+
When you genuinely need the full tree of a page, **write it to disk and size it first —
|
|
31
|
+
never read it straight into the session**:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
playwright-cli --raw snapshot > output/sessions/<session-dir>/snapshots/<page>.yml
|
|
35
|
+
wc -l output/sessions/<session-dir>/snapshots/<page>.yml
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
- **At most 300 lines** — `Read` the file.
|
|
39
|
+
- **Over 300 lines** — invoke the `qa-page-mapper-agent` sub-agent
|
|
40
|
+
(`qualiow:qa-page-mapper-agent` under a plugin install) with the file path and the page
|
|
41
|
+
URL, and work from the map it returns: forms (fields, which are required, the submit ref),
|
|
42
|
+
nav (text → ref), interactive controls, visible error and empty-state text, hidden and
|
|
43
|
+
disabled counts. It returns structure only — no test ideas, no findings. If sub-agents are
|
|
44
|
+
unavailable, do the step yourself as in 2.1.0: read the snapshot in windows (`Grep` for the
|
|
45
|
+
region, then `Read` with `offset`/`limit`).
|
|
46
|
+
|
|
47
|
+
`snapshots/` is working material, not session evidence: it is never shipped in the report and
|
|
48
|
+
never read back by the reporting step.
|
|
49
|
+
|
|
50
|
+
Before/after diff — both files go into `snapshots/` the same way:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
playwright-cli --raw snapshot > output/sessions/<session-dir>/snapshots/<page>-before.yml
|
|
54
|
+
# ...act...
|
|
55
|
+
playwright-cli --raw snapshot > output/sessions/<session-dir>/snapshots/<page>-after.yml
|
|
56
|
+
diff output/sessions/<session-dir>/snapshots/<page>-before.yml output/sessions/<session-dir>/snapshots/<page>-after.yml
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
The diff is what you read — it shows exactly what the action changed.
|
|
60
|
+
|
|
61
|
+
Network helper: `playwright-cli requests --filter="/api/"` — only the calls that matter;
|
|
62
|
+
`playwright-cli request <n>` for headers and body of one of them.
|
|
63
|
+
|
|
64
|
+
The interaction phases (4-6) are unchanged: snapshot before every interaction, and keep the
|
|
65
|
+
refs you act on in this session.
|
|
24
66
|
|
|
25
67
|
## Scope Enforcement
|
|
26
68
|
|
|
@@ -70,15 +70,19 @@ One file per bug, `bugs/BUG-NNN.md`, in exactly the format of
|
|
|
70
70
|
|
|
71
71
|
Redact per `security-rules.md` before writing.
|
|
72
72
|
|
|
73
|
-
## Write
|
|
73
|
+
## Write phase-7-notes.md
|
|
74
74
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
75
|
+
Your judgement, in your own words, in `output/sessions/<session-dir>/phase-7-notes.md`. The
|
|
76
|
+
confidentiality header first, then exactly these sections — the report is assembled from
|
|
77
|
+
them, so a section you leave out is a section nobody can write for you:
|
|
78
|
+
|
|
79
|
+
- `## Executive Summary` — 3 sentences: what was tested, what was found, the biggest risk
|
|
80
|
+
- `## Coverage Map` — rows for `| Area | Risk | Status | Bugs | Notes |`
|
|
81
|
+
(Status: tested / partial / not-tested / code-verified-only)
|
|
82
|
+
- `## Observations` — including what's MISSING and the data-integrity results
|
|
83
|
+
- `## Areas Not Tested` — each with its reason
|
|
84
|
+
- `## Recommendations` — for the next session
|
|
85
|
+
- `## Reflection` — the five answers above
|
|
82
86
|
|
|
83
87
|
## Session Stats
|
|
84
88
|
|
|
@@ -86,18 +90,35 @@ Write `output/sessions/<session-dir>/stats.json` exactly as `output-contract.md`
|
|
|
86
90
|
(`session_id`, `kind: "explore"`, `target`, `date`, `duration_min`, `bugs_found`,
|
|
87
91
|
`severity_counts`, `pages_explored`, plus the optional `domain`, `started_at`,
|
|
88
92
|
`completed_at`, `phases_completed`, `total_phases: 8`, `coverage`, `evidence`,
|
|
89
|
-
`areas_not_tested`, `blocked_by`). No other top-level keys.
|
|
93
|
+
`areas_not_tested`, `blocked_by`). No other top-level keys. The `## Session Stats` table in
|
|
94
|
+
the report is rendered from this file.
|
|
95
|
+
|
|
96
|
+
## Assemble and Finalize
|
|
90
97
|
|
|
91
|
-
|
|
98
|
+
The bugs, the stats and the notes are yours; assembling them into the report is not. Invoke
|
|
99
|
+
the `qa-reporting-agent` sub-agent (`qualiow:qa-reporting-agent` under a plugin install) with
|
|
100
|
+
the session directory and `kind: explore`. It reads `charter.md`, `stats.json`, `bugs/*.md`,
|
|
101
|
+
the phase files in windows and `phase-7-notes.md`, writes `session-report.md` in the format
|
|
102
|
+
of `${CLAUDE_SKILL_DIR}/references/output-contract.md` — copying your Executive Summary,
|
|
103
|
+
Recommendations and Reflection verbatim — and then runs `qualiow session finalize`, which
|
|
104
|
+
validates `stats.json` against the strict schema, checks the confidentiality header on every
|
|
105
|
+
artefact, scans the directory against the redaction list, appends the session row to
|
|
106
|
+
`output/sessions/INDEX.md` and one row per bug to `output/bugs/all-bugs.md`, records the
|
|
107
|
+
session metrics and sets `progress.json` to `complete`. It is idempotent.
|
|
92
108
|
|
|
93
|
-
|
|
109
|
+
Read its return (at most 8 lines). If it reports a missing notes section or a violation it
|
|
110
|
+
could not fix, fix that yourself and re-run the check:
|
|
94
111
|
|
|
95
|
-
|
|
112
|
+
```bash
|
|
113
|
+
qualiow session finalize output/sessions/<session-dir> --check
|
|
114
|
+
```
|
|
96
115
|
|
|
97
|
-
|
|
116
|
+
`--check` writes nothing and prints a numbered list of violations naming each file. Resolve
|
|
117
|
+
the `qualiow` prefix per `${CLAUDE_SKILL_DIR}/references/paths.md`; if the CLI is unavailable,
|
|
118
|
+
write the index rows by hand in the column order defined there.
|
|
98
119
|
|
|
99
|
-
|
|
100
|
-
|
|
120
|
+
If sub-agents are unavailable, do the step yourself as in 2.1.0: write `session-report.md`
|
|
121
|
+
from the notes in the contract's format and run `qualiow session finalize output/sessions/<session-dir>`.
|
|
101
122
|
|
|
102
123
|
## Close the Browser Session
|
|
103
124
|
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Delegation Rules
|
|
2
|
+
|
|
3
|
+
What is allowed to leave this context, and what never is. The pack routes bulk input/output
|
|
4
|
+
away from the session and keeps the QA reasoning in it. Read this before reaching for a whole
|
|
5
|
+
file: most large reads have a cheaper route that returns the same facts.
|
|
6
|
+
|
|
7
|
+
## Tiers
|
|
8
|
+
|
|
9
|
+
| Tier | What runs there | Use it for |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| **0 — deterministic code** | the `qualiow` CLI (`${CLAUDE_SKILL_DIR}/references/paths.md` resolves the binary) | anything with a fixed contract: knowledge digests, index rows, `stats.json` validation, the redaction scan, session listing and archival |
|
|
12
|
+
| **1 — cheap-model sub-agents** | `qa-reporting-agent` (sonnet), `qa-page-mapper-agent` (haiku), `qa-diff-indexer-agent` (haiku), `qa-gather-agent` (sonnet) | bounded reads that need light judgement and return a structured digest |
|
|
13
|
+
| **2 — this session** | the model reading these rules | exploration, interaction, bug finding, every judgement below |
|
|
14
|
+
|
|
15
|
+
Tier 0 beats tier 1 whenever the answer is deterministic: a command costs no tokens and
|
|
16
|
+
cannot hallucinate a row.
|
|
17
|
+
|
|
18
|
+
## Delegates
|
|
19
|
+
|
|
20
|
+
Prefix every name with `qualiow:` under a plugin install (`qualiow:qa-reporting-agent`).
|
|
21
|
+
|
|
22
|
+
| Agent | Input | Returns | Used by |
|
|
23
|
+
|---|---|---|---|
|
|
24
|
+
| `qa-reporting-agent` | session directory + kind | `session-report.md` assembled from the notes, bugs and phase files, `qualiow session finalize` run, a summary of at most 8 lines | phase 7 of explore and mobile, phase 5 of backend, `/qa-explore-report` |
|
|
25
|
+
| `qa-page-mapper-agent` | a `snapshots/<page>.yml` over 300 lines + the page URL | forms with field refs, nav text → ref, interactive controls, visible error and empty-state text, hidden/disabled counts | phase 3 discovery |
|
|
26
|
+
| `qa-diff-indexer-agent` | repo, base, branch, the AC list | file → symbols → line ranges → candidate ACs, plus files matching no AC and ACs matching no file | phase 2 static review |
|
|
27
|
+
| `qa-gather-agent` | files, URLs or pasted text in the invocation | the context file under `output/context/`, gaps and assumptions marked | `/qa-gather` (always forks) |
|
|
28
|
+
|
|
29
|
+
If sub-agents are unavailable, do the step yourself as in 2.1.0 — the thresholds below still
|
|
30
|
+
apply; read in windows instead.
|
|
31
|
+
|
|
32
|
+
## Never delegate
|
|
33
|
+
|
|
34
|
+
These stay in this session, always, whatever the tiering:
|
|
35
|
+
|
|
36
|
+
- **Severity and priority** — and the "when in doubt go LOWER" call
|
|
37
|
+
- **Business impact** — revenue, trust, regulatory, data, scale
|
|
38
|
+
- **Bug reports** — the decision that something *is* a bug, and every word of `bugs/BUG-NNN.md`
|
|
39
|
+
- **The charter** and the risk ranking (P0/P1/P2/P3) behind it
|
|
40
|
+
- **What's MISSING** — the negative-space question no extraction pass can ask
|
|
41
|
+
- **Verdicts** — `PASS`, `PARTIAL`, `FAIL`, `BLOCKED`, `NOT-REACHABLE`, `UNVERIFIABLE`
|
|
42
|
+
- **The executive summary** and the recommendations
|
|
43
|
+
- **The reflection** — what worried you, what you did not test, what to do next
|
|
44
|
+
|
|
45
|
+
A delegate that returns any of these has exceeded its brief; discard that part of its answer
|
|
46
|
+
and make the call yourself.
|
|
47
|
+
|
|
48
|
+
## What a delegate must return
|
|
49
|
+
|
|
50
|
+
Bounded (say the maximum size in the request), structured (a table or a fixed heading set),
|
|
51
|
+
cited (`file:line`, a ref, or a probe command), and free of opinions — no "looks fine", no
|
|
52
|
+
"this is probably a bug", no severity. You then reason over its output; you never paste it
|
|
53
|
+
into a report unchanged.
|
|
54
|
+
|
|
55
|
+
## Thresholds
|
|
56
|
+
|
|
57
|
+
| Input | Ceiling | Route instead |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| A qualiow data file (manifest, release entry, learned patterns) | 300 lines | `qualiow kb digest`, `qualiow list knowledge --entry <id>`, or `Grep` |
|
|
60
|
+
| A raw accessibility snapshot | 300 lines | `snapshot --depth=3`, `snapshot <ref>` to zoom, `find "<text>"` for one label |
|
|
61
|
+
| Any other file | 300 lines | `Read` with `offset`/`limit` after a `Grep` for the heading you need |
|
|
62
|
+
| A branch diff | 25 or more files, or 1,500 or more changed lines | `git diff --stat` first, then `git show <branch>:<path>` for the files that map to an AC |
|
|
63
|
+
|
|
64
|
+
Check before you read when you are unsure: `wc -l <file>`.
|
|
65
|
+
|
|
66
|
+
**Never `Read` `manifest.yml` or a knowledge release entry whole.** Use
|
|
67
|
+
`qualiow kb digest --for <explore|backend|mobile>` for the session's working set,
|
|
68
|
+
`qualiow list knowledge --entry <id>` for one entry in full, or `Grep` the release file for
|
|
69
|
+
the one heading you need.
|
|
70
|
+
|
|
71
|
+
## Overrides
|
|
72
|
+
|
|
73
|
+
- `QUALIOW_READ_MAX_LINES=<n>` — moves the 300-line ceiling for a project
|
|
74
|
+
- `QUALIOW_HOOKS=off` — disables the enforcement layer for a project
|
|
75
|
+
|
|
76
|
+
Both are read by the hook layer — `hooks/hooks.json` in the plugin, or `qualiow init --hooks`
|
|
77
|
+
in an npm project — and belong in the `env` block of the project's `.claude/settings.json`.
|
|
78
|
+
|
|
79
|
+
- `CLAUDE_CODE_SUBAGENT_MODEL` — overrides the model of every sub-agent, whatever each agent
|
|
80
|
+
file pins
|
|
81
|
+
- `model: inherit` in a project's own copy of an agent file — that one delegate runs on the
|
|
82
|
+
session's model instead of the cheap one
|
|
@@ -24,8 +24,10 @@ session report, coverage map, AC matrix, expected-behaviour spec):
|
|
|
24
24
|
| `phase-3-discovery.md` … `phase-6-edge-cases.md` | phases 3–6 (file number = phase number) |
|
|
25
25
|
| `screenshots/BUG-NNN.png`, other `screenshots/*.png` | as taken |
|
|
26
26
|
| `videos/*.webm` (web) / `videos/*.mp4` (mobile) | recording |
|
|
27
|
+
| `snapshots/*.yml` | raw accessibility trees from `playwright-cli --raw snapshot`; working files, never shipped, and excluded from the secrets scan |
|
|
28
|
+
| `phase-7-notes.md` | phase 7 (backend: phase 5) — the session's own executive summary, coverage rows, observations, areas not tested, recommendations and reflection; the input the report is assembled from |
|
|
27
29
|
| `bugs/BUG-NNN.md` | phase 7 |
|
|
28
|
-
| `session-report.md` | phase
|
|
30
|
+
| `session-report.md` | the `qa-reporting-agent` sub-agent, from `phase-7-notes.md`, the bugs, `stats.json` and the phase files (quick sessions write it directly) |
|
|
29
31
|
| `stats.json` | phase 7 |
|
|
30
32
|
| mobile: `logs/BUG-NNN.log` · backend: `ac-matrix.md`, `evidence/`, `probes/`, `expected-behaviour.md` | as produced |
|
|
31
33
|
|
|
@@ -176,7 +178,10 @@ verdict counts under `coverage.verdicts`.
|
|
|
176
178
|
## Index rows
|
|
177
179
|
|
|
178
180
|
Every session ends by appending one row to `output/sessions/INDEX.md` and one row per bug
|
|
179
|
-
to `output/bugs/all-bugs.md`.
|
|
181
|
+
to `output/bugs/all-bugs.md`. Both rows are written by `qualiow session finalize <session-dir>`,
|
|
182
|
+
which refuses the session — exit 1, naming the file — when an artefact is missing the
|
|
183
|
+
confidentiality header, an unredacted secret survives the scan, or `stats.json` does not
|
|
184
|
+
conform. The column order (defined once in `paths.md`) is:
|
|
180
185
|
|
|
181
186
|
```
|
|
182
187
|
| Date | Kind | Target | Bugs | Duration | Status | Report |
|
|
@@ -22,7 +22,9 @@ written by `/qa-target-setup`, and is the right place for anything with internal
|
|
|
22
22
|
## Data directory (`<data>`)
|
|
23
23
|
|
|
24
24
|
First existing of: `$PWD/data` → `${CLAUDE_PLUGIN_ROOT}/data` →
|
|
25
|
-
`${CLAUDE_SKILL_DIR}/../../data` → `${CLAUDE_SKILL_DIR}/../../../data`.
|
|
25
|
+
`${CLAUDE_SKILL_DIR}/../../data` → `${CLAUDE_SKILL_DIR}/../../../data`. The `qualiow` CLI
|
|
26
|
+
resolves the same order — an explicit `--data <dir>`, then `$PWD/data`, then
|
|
27
|
+
`$CLAUDE_PLUGIN_ROOT/data`, then the data shipped inside the package.
|
|
26
28
|
|
|
27
29
|
Read from `<data>`: `domains/<domain>.yml`, `knowledge/manifest.yml`,
|
|
28
30
|
`knowledge/releases/<version>/entries/*.yml`, `knowledge/learned-patterns.md`,
|
|
@@ -46,6 +48,24 @@ First existing of: `$PWD/qa/bin/mcli` → `$PWD/bin/mcli` → `mcli` on `PATH`
|
|
|
46
48
|
default after `qualiow init`); in a checkout of this repository it is `bin/mcli`.
|
|
47
49
|
Substitute the prefix you resolved.
|
|
48
50
|
|
|
51
|
+
## CLI
|
|
52
|
+
|
|
53
|
+
`qualiow` is the deterministic half of the pack: knowledge digests, session finalization,
|
|
54
|
+
listings, archival. First working of:
|
|
55
|
+
|
|
56
|
+
1. `qualiow` on `PATH` — a marketplace plugin install (its `bin/` is on `PATH`), or a global
|
|
57
|
+
npm install
|
|
58
|
+
2. `npx -y -p qualiow-exploratory-testing qualiow` — a project that depends on the npm package
|
|
59
|
+
3. `${CLAUDE_PLUGIN_ROOT}/bin/qualiow` — a plugin loaded with `--plugin-dir`
|
|
60
|
+
4. `bin/qualiow` — this repository checked out
|
|
61
|
+
|
|
62
|
+
The npm package is **`qualiow-exploratory-testing`** and the binary it installs is `qualiow`.
|
|
63
|
+
Never write bare `npx qualiow`: that spelling resolves a package that does not exist.
|
|
64
|
+
|
|
65
|
+
Same rule as the mobile driver — resolve once in setup, then write the resolved literal
|
|
66
|
+
prefix as the first token of every command. The phase files show `qualiow <command>` for
|
|
67
|
+
brevity; substitute the prefix you resolved.
|
|
68
|
+
|
|
49
69
|
## Output
|
|
50
70
|
|
|
51
71
|
Always `$PWD/output/`. Never write session output anywhere else.
|
|
@@ -71,6 +91,10 @@ ends with `playwright-cli -s=<sid> close` and then `playwright-cli -s=<sid> dele
|
|
|
71
91
|
|
|
72
92
|
### Index rows
|
|
73
93
|
|
|
94
|
+
`qualiow session finalize <session-dir>` writes both rows (and creates either index file if
|
|
95
|
+
it is missing). The formats below are what it appends — write them by hand only when the CLI
|
|
96
|
+
is unavailable.
|
|
97
|
+
|
|
74
98
|
Append to `output/sessions/INDEX.md` (columns `| Date | Kind | Target | Bugs | Duration | Status | Report |`):
|
|
75
99
|
|
|
76
100
|
```
|
|
@@ -63,6 +63,10 @@ Never write actual `.env` values, storage-state contents, or session cookies to
|
|
|
63
63
|
file. Raw probe output and API response bodies are the most common leak; redact them on the
|
|
64
64
|
way in.
|
|
65
65
|
|
|
66
|
+
Two layers check the same list behind you: `qualiow session finalize` scans every artefact in
|
|
67
|
+
the session directory, and the plugin's write guard applies it before a file under `output/`
|
|
68
|
+
is written (`QUALIOW_HOOKS=off` disables the guard). Neither replaces redacting on the way in.
|
|
69
|
+
|
|
66
70
|
## Session Isolation
|
|
67
71
|
|
|
68
72
|
1. Every `playwright-cli` command in a session carries the session id
|
|
@@ -6,7 +6,8 @@
|
|
|
6
6
|
2. **ONE BUG = ONE REPORT** -- Never combine
|
|
7
7
|
3. **Evidence for every bug** -- Screenshot + context
|
|
8
8
|
4. **Real-time logging** -- Append to session-log.md continuously
|
|
9
|
-
5. **Snapshot before every interaction** -- Always know the page state
|
|
9
|
+
5. **Snapshot before every interaction** -- Always know the page state. `--depth=3` or
|
|
10
|
+
`snapshot <ref>` by default; a full tree goes to `snapshots/` (phase 3 policy)
|
|
10
11
|
|
|
11
12
|
## Quality Thinking
|
|
12
13
|
|
|
@@ -4,13 +4,17 @@ description: >
|
|
|
4
4
|
Manage exploratory testing sessions — list, archive, or delete old sessions.
|
|
5
5
|
Use when user says: "cleanup sessions", "delete old sessions", "archive", "list sessions".
|
|
6
6
|
argument-hint: "[list | archive <session-dir> | delete <session-dir> | delete-older-than <days>]"
|
|
7
|
-
allowed-tools: Read,
|
|
7
|
+
allowed-tools: Read, Grep, Bash(qualiow:*), Bash(npx:*)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# Session Cleanup & Management
|
|
11
11
|
|
|
12
|
+
This skill is a thin wrapper around `qualiow session`. Run it in this context — never hand
|
|
13
|
+
the delete paths to anything that cannot stop and ask.
|
|
14
|
+
|
|
12
15
|
Session directories follow `output/sessions/<YYYY-MM-DD-HHmm>-<kind>-<slug>/`
|
|
13
|
-
(`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md
|
|
16
|
+
(`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`, which also resolves the `qualiow`
|
|
17
|
+
prefix — write it literally). Archives are still CONFIDENTIAL
|
|
14
18
|
(`${CLAUDE_SKILL_DIR}/../qa-explore/references/security-rules.md`): keep them under
|
|
15
19
|
`output/`, never upload them. What a complete session directory contains — and which files
|
|
16
20
|
must survive an archive — is defined in
|
|
@@ -19,32 +23,53 @@ must survive an archive — is defined in
|
|
|
19
23
|
## Commands
|
|
20
24
|
|
|
21
25
|
### List Sessions
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
qualiow session list
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Prints every row of `output/sessions/INDEX.md` and flags directories
|
|
32
|
+
present on disk but missing from the index as `unindexed`. Show the output as it comes.
|
|
27
33
|
|
|
28
34
|
### Archive Session
|
|
35
|
+
|
|
29
36
|
```bash
|
|
30
|
-
|
|
31
|
-
tar -czf output/sessions/<session-dir>.tar.gz -C output/sessions <session-dir>
|
|
37
|
+
qualiow session archive <session-dir>
|
|
32
38
|
```
|
|
33
|
-
|
|
39
|
+
|
|
40
|
+
Writes `output/sessions/<session-dir>.tar.gz`. Add `--remove` to drop the original directory
|
|
41
|
+
afterwards and mark its index row `archived` — offer that, don't assume it.
|
|
34
42
|
|
|
35
43
|
### Delete Session
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
44
|
+
|
|
45
|
+
**Always confirm with the user before deleting.** The command is a dry run until `--yes`:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
qualiow session delete <session-dir>
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Show its output — file count, size, and the index rows that would go — then ask the user
|
|
52
|
+
whether to proceed. Only on an explicit yes:
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
qualiow session delete <session-dir> --yes
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
That removes the directory, its row from `output/sessions/INDEX.md` and its rows from
|
|
59
|
+
`output/bugs/all-bugs.md`.
|
|
41
60
|
|
|
42
61
|
### Delete Old Sessions
|
|
43
|
-
|
|
44
|
-
|
|
62
|
+
|
|
63
|
+
Same two steps, by age:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
qualiow session prune --older-than <days>
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Show the list of sessions it would remove, ask, then run it again with `--yes`.
|
|
45
70
|
|
|
46
71
|
### Rules
|
|
47
72
|
- NEVER auto-delete — always show and confirm
|
|
48
|
-
- Keep INDEX.md updated after any changes
|
|
49
73
|
- Suggest archiving before deleting
|
|
50
|
-
- Remind the user of the 90-day retention default from the security policy when listing
|
|
74
|
+
- Remind the user of the 90-day retention default from the security policy when listing
|
|
75
|
+
sessions older than that
|
|
@@ -27,6 +27,11 @@ to find the most recent session, or take the user-specified directory. Any sessi
|
|
|
27
27
|
(`explore`, `quick`, `mobile`, `backend`) is valid. Read the session's `session-report.md`
|
|
28
28
|
and all `bugs/BUG-*.md` files.
|
|
29
29
|
|
|
30
|
+
The phase files and `snapshots/` are not needed for feedback — the report and the bugs carry
|
|
31
|
+
every claim the user is reviewing. A `session-report.md` over 300 lines is read in windows:
|
|
32
|
+
`Grep '^## '` for the headings, then `Read` with `offset`/`limit` on Coverage Map, Bugs Found,
|
|
33
|
+
Observations and Reflection.
|
|
34
|
+
|
|
30
35
|
### 2. Review Bugs with User
|
|
31
36
|
|
|
32
37
|
For each bug found, ask the user:
|
|
@@ -11,7 +11,7 @@ description: >
|
|
|
11
11
|
"explore the app on iOS/Android", "find bugs on mobile", "QA the mobile build", or provides
|
|
12
12
|
a mobile-sim target (native or web).
|
|
13
13
|
argument-hint: "--target <id> [--context <file>] [--rebuild]"
|
|
14
|
-
allowed-tools: Bash(qa/bin/mcli:*), Bash(bin/mcli:*), Bash(mcli:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/mcli:*), Bash(qa/bin/wadb:*), Bash(bin/wadb:*), Bash(wadb:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/wadb:*), Bash(qa/bin/wk-ios:*), Bash(bin/wk-ios:*), Bash(wk-ios:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/wk-ios:*), Bash(qa/bin/doctor-mobile.sh:*), Bash(bin/doctor-mobile.sh:*), Bash(qualiow-doctor-mobile:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/doctor-mobile.sh:*), Bash(xcrun:*), Bash(adb:*), Bash(maestro:*), Bash(node:*), Bash(sleep:*), Bash(python3:*), Bash(git:*), Read, Write, Glob, Grep
|
|
14
|
+
allowed-tools: Bash(qa/bin/mcli:*), Bash(bin/mcli:*), Bash(mcli:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/mcli:*), Bash(qa/bin/wadb:*), Bash(bin/wadb:*), Bash(wadb:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/wadb:*), Bash(qa/bin/wk-ios:*), Bash(bin/wk-ios:*), Bash(wk-ios:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/wk-ios:*), Bash(qa/bin/doctor-mobile.sh:*), Bash(bin/doctor-mobile.sh:*), Bash(qualiow-doctor-mobile:*), Bash(${CLAUDE_SKILL_DIR}/../../bin/doctor-mobile.sh:*), Bash(xcrun:*), Bash(adb:*), Bash(maestro:*), Bash(node:*), Bash(sleep:*), Bash(python3:*), Bash(git:*), Bash(qualiow:*), Bash(npx:*), Read, Write, Glob, Grep
|
|
15
15
|
---
|
|
16
16
|
|
|
17
17
|
# Mobile Exploratory Testing Session
|
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
# Phase 7: Reflection & Reporting (Mobile)
|
|
2
2
|
|
|
3
3
|
Follow `${CLAUDE_SKILL_DIR}/../qa-explore/phases/07-reporting.md` for the reflection
|
|
4
|
-
questions
|
|
4
|
+
questions, the `phase-7-notes.md` sections and the hand-off to the reporting agent; every
|
|
5
|
+
artefact uses the format in
|
|
5
6
|
`${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md` (confidentiality header
|
|
6
7
|
first). The mobile differences (both modes):
|
|
7
8
|
|
|
@@ -68,9 +69,12 @@ qa/bin/mcli logs --since 30 --errors > output/sessions/<session-dir>/logs/BUG-NN
|
|
|
68
69
|
Redact the log excerpt per `security-rules.md` before it is saved: process logs routinely
|
|
69
70
|
carry tokens and emails.
|
|
70
71
|
|
|
71
|
-
##
|
|
72
|
+
## phase-7-notes.md — mobile additions
|
|
72
73
|
|
|
73
|
-
`
|
|
74
|
+
Write `phase-7-notes.md` with the sections the web phase lists (`## Executive Summary`,
|
|
75
|
+
`## Coverage Map`, `## Observations`, `## Areas Not Tested`, `## Recommendations`,
|
|
76
|
+
`## Reflection`), then these two — they are copied into `session-report.md` after
|
|
77
|
+
`## Session Stats`:
|
|
74
78
|
|
|
75
79
|
```markdown
|
|
76
80
|
## Mobile Context
|
|
@@ -100,9 +104,9 @@ shows up as a known gap.
|
|
|
100
104
|
|
|
101
105
|
## Requirements coverage cross-check (optional)
|
|
102
106
|
|
|
103
|
-
If the session was scoped to gathered requirements / tickets, add a cross-check table
|
|
104
|
-
|
|
105
|
-
designed?":
|
|
107
|
+
If the session was scoped to gathered requirements / tickets, add a cross-check table to the
|
|
108
|
+
notes; it is carried into the report with the other sections. This is **not** test execution;
|
|
109
|
+
it is "did exploration touch the areas those requirements designed?":
|
|
106
110
|
|
|
107
111
|
```markdown
|
|
108
112
|
## Requirements Cross-Check
|
|
@@ -139,11 +143,29 @@ block under `coverage.mobile` (the schema is strict; no other top-level keys):
|
|
|
139
143
|
|
|
140
144
|
(For native mode set `"mode": "native"`, `app_under_test` to the app id, and `target_url` to null.)
|
|
141
145
|
|
|
142
|
-
##
|
|
146
|
+
## Assemble and Finalize
|
|
143
147
|
|
|
144
|
-
|
|
148
|
+
Invoke the `qa-reporting-agent` sub-agent (`qualiow:qa-reporting-agent` under a plugin
|
|
149
|
+
install) with the session directory and `kind: mobile`. It assembles `session-report.md` from
|
|
150
|
+
`phase-7-notes.md`, the bugs, `stats.json` and the phase files — keeping `## Mobile Context`
|
|
151
|
+
and `## Deferred Tests` after `## Session Stats` — and then runs
|
|
152
|
+
`qualiow session finalize`, which validates `stats.json` against the strict schema, checks the
|
|
153
|
+
confidentiality header on every artefact and scans the directory against the redaction list —
|
|
154
|
+
the logcat / simctl excerpts under `logs/` are scanned like everything else — then appends the
|
|
155
|
+
session row to `output/sessions/INDEX.md` (`Kind` = `mobile`) and one row per bug to
|
|
156
|
+
`output/bugs/all-bugs.md`, records the session metrics and sets `progress.json` to
|
|
157
|
+
`complete`. It is idempotent.
|
|
145
158
|
|
|
146
|
-
|
|
159
|
+
Read its return (at most 8 lines). If it reports a missing notes section or a violation it
|
|
160
|
+
could not fix, fix that yourself and re-run the check:
|
|
147
161
|
|
|
148
|
-
|
|
149
|
-
|
|
162
|
+
```bash
|
|
163
|
+
qualiow session finalize output/sessions/<session-dir> --check
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
`--check` writes nothing and prints a numbered list of violations naming each file. Resolve
|
|
167
|
+
the `qualiow` prefix per `${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`; if the CLI
|
|
168
|
+
is unavailable, write the rows by hand in the column order defined there.
|
|
169
|
+
|
|
170
|
+
If sub-agents are unavailable, do the step yourself as in 2.1.0: write `session-report.md`
|
|
171
|
+
from the notes in the contract's format and run `qualiow session finalize output/sessions/<session-dir>`.
|
|
@@ -5,7 +5,7 @@ description: >
|
|
|
5
5
|
No full site mapping — goes directly to the target, applies heuristics, reports findings.
|
|
6
6
|
Use when user says: "quick check", "test this page", "quick explore", or wants a fast review.
|
|
7
7
|
argument-hint: "<url> [--focus <area>]"
|
|
8
|
-
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Read, Write, Glob, Grep
|
|
8
|
+
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx:*), Read, Write, Glob, Grep
|
|
9
9
|
---
|
|
10
10
|
|
|
11
11
|
# Quick Exploratory Testing Session
|
|
@@ -104,7 +104,17 @@ starts with these two lines, verbatim:
|
|
|
104
104
|
`## Session Stats`
|
|
105
105
|
- `stats.json` — `kind: "quick"`, the eight required fields
|
|
106
106
|
|
|
107
|
-
|
|
107
|
+
Then finalize:
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
qualiow session finalize output/sessions/<session-dir>
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
It validates `stats.json`, checks the confidentiality header on every artefact and scans them
|
|
114
|
+
against the redaction list, then appends the session row to `output/sessions/INDEX.md` and one
|
|
115
|
+
row per bug to `output/bugs/all-bugs.md` and records the metrics. On exit 1 it names each
|
|
116
|
+
offending file: fix it and run again (`--check` validates without writing). Resolve the
|
|
117
|
+
`qualiow` prefix per `${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`.
|
|
108
118
|
|
|
109
119
|
Then close the session:
|
|
110
120
|
|
|
@@ -5,7 +5,9 @@ description: >
|
|
|
5
5
|
Use when user says: "show report", "generate report", "reformat bugs", "session summary",
|
|
6
6
|
or wants to review past session findings.
|
|
7
7
|
argument-hint: "[<session-dir> | latest] [--format markdown|summary|bugs-only|coverage-map] [--overwrite]"
|
|
8
|
-
allowed-tools: Read, Write, Glob, Grep
|
|
8
|
+
allowed-tools: Read, Write, Glob, Grep, Bash(qualiow:*), Bash(npx:*), Bash(wc:*)
|
|
9
|
+
context: fork
|
|
10
|
+
agent: qa-reporting-agent
|
|
9
11
|
---
|
|
10
12
|
|
|
11
13
|
# Exploratory Testing Report Generator
|
|
@@ -15,7 +17,10 @@ Reads any session kind (`explore`, `quick`, `mobile`, `backend`) written in the
|
|
|
15
17
|
same format. Session directories, index columns and the data-resolution order come from
|
|
16
18
|
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`. Security:
|
|
17
19
|
`${CLAUDE_SKILL_DIR}/../qa-explore/references/security-rules.md` applies (redact before
|
|
18
|
-
writing; confidentiality header first on every file).
|
|
20
|
+
writing; confidentiality header first on every file). This skill runs in the
|
|
21
|
+
`qa-reporting-agent` sub-agent (`qualiow:qa-reporting-agent` under a plugin install), in a
|
|
22
|
+
forked context: it assembles what the session already wrote and never invents a finding, a
|
|
23
|
+
severity or a verdict.
|
|
19
24
|
|
|
20
25
|
## Input
|
|
21
26
|
|
|
@@ -37,10 +42,16 @@ directory; list the candidates if it is ambiguous.
|
|
|
37
42
|
- `charter.md`
|
|
38
43
|
- `session-log.md`
|
|
39
44
|
- `progress.json`, `stats.json`
|
|
45
|
+
- `phase-7-notes.md` — the session's own summary, coverage rows, observations, recommendations and reflection; copy those sections, never invent them
|
|
40
46
|
- `phase-3-discovery.md`, `phase-4-journeys.md`, `phase-5-features.md`, `phase-6-edge-cases.md` (explore and mobile sessions; quick sessions have none; backend sessions have `ac-matrix.md`, `evidence/`, `expected-behaviour.md` instead)
|
|
41
47
|
- `bugs/BUG-*.md`
|
|
42
48
|
- the existing `session-report.md`, if any (its coverage map is the coverage source)
|
|
43
49
|
|
|
50
|
+
Size a file before you open it (`wc -l <file>`). A phase file or a report over 300 lines is
|
|
51
|
+
read in windows: `Grep '^## ' <file>` for the headings, then `Read` with `offset`/`limit` on
|
|
52
|
+
the sections you need. `snapshots/` is raw page structure, not session evidence — never read
|
|
53
|
+
it here.
|
|
54
|
+
|
|
44
55
|
### 3. Generate the Report
|
|
45
56
|
|
|
46
57
|
**markdown** (default): the full session report per `output-contract.md`
|
|
@@ -63,6 +74,19 @@ from the session report's coverage table.
|
|
|
63
74
|
- `bugs-only` → `bugs-summary.md`
|
|
64
75
|
- `coverage-map` → `coverage-map.md`
|
|
65
76
|
|
|
66
|
-
All in `output/sessions/<session-dir>/`, confidentiality header first, redacted.
|
|
67
|
-
|
|
68
|
-
|
|
77
|
+
All in `output/sessions/<session-dir>/`, confidentiality header first, redacted. Never modify
|
|
78
|
+
`bugs/BUG-*.md` from this skill; `/qa-explore-feedback` owns severity changes.
|
|
79
|
+
|
|
80
|
+
Then check what you wrote:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
qualiow session finalize output/sessions/<session-dir> --check
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
`--check` writes nothing; it validates `stats.json`, the confidentiality header on every
|
|
87
|
+
artefact and the redaction list, and on exit 1 prints a numbered list of violations naming
|
|
88
|
+
each file. Fix header and format violations and run it again; report anything you did not
|
|
89
|
+
fix. Resolve the `qualiow` prefix per
|
|
90
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`.
|
|
91
|
+
|
|
92
|
+
Present the result to the user.
|