qualiow-exploratory-testing 2.3.0 → 2.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +177 -0
- package/README.md +71 -21
- package/agents/qa-bug-judge.md +1 -1
- package/agents/qa-diff-indexer-agent.md +1 -1
- package/agents/qa-gather-agent.md +1 -1
- package/agents/qa-page-mapper-agent.md +3 -1
- package/agents/qa-reporting-agent.md +28 -14
- package/bin/mobile-cli.mjs +74 -26
- package/bin/qualiow +7 -2
- package/bin/wk-ios +60 -6
- package/bin/wkeval.mjs +7 -2
- package/data/security/SECURITY-POLICY.md +22 -3
- package/data/targets/parabank.yml +13 -0
- package/data/targets/saucedemo.yml +13 -0
- package/data/templates/bug-report.md +3 -0
- package/data/templates/session-charter.md +7 -5
- package/data/templates/session-report.md +7 -1
- package/dist/cli/index.js +2060 -361
- package/dist/index.d.ts +813 -26
- package/dist/index.js +2694 -704
- package/docs/GETTING-STARTED.md +71 -17
- package/docs/MOBILE-SETUP.md +31 -0
- package/hooks/hooks.json +12 -2
- package/hooks/scripts/bash-guard.mjs +242 -0
- package/hooks/scripts/secret-patterns.d.mts +4 -1
- package/hooks/scripts/secret-patterns.mjs +126 -34
- package/hooks/scripts/write-guard.mjs +70 -15
- package/package.json +3 -2
- package/skills/qa-explore/SKILL.md +10 -3
- package/skills/qa-explore/phases/00-setup.md +40 -3
- package/skills/qa-explore/phases/02-charter.md +25 -5
- package/skills/qa-explore/phases/04-journeys.md +14 -0
- package/skills/qa-explore/phases/05-features.md +10 -0
- package/skills/qa-explore/phases/06-edge-cases.md +10 -0
- package/skills/qa-explore/phases/07-reporting.md +93 -11
- package/skills/qa-explore/references/delegation-rules.md +12 -1
- package/skills/qa-explore/references/evidence-triage.md +44 -12
- package/skills/qa-explore/references/output-contract.md +69 -7
- package/skills/qa-explore/references/security-rules.md +25 -3
- package/skills/qa-explore-cleanup/SKILL.md +1 -1
- package/skills/qa-explore-mobile/SKILL.md +13 -3
- package/skills/qa-explore-mobile/phases/02-charter.md +2 -1
- package/skills/qa-explore-mobile/phases/07-reporting.md +2 -0
- package/skills/qa-explore-mobile/references/mobile-edges.md +3 -1
- package/skills/qa-explore-quick/SKILL.md +48 -9
- package/skills/qa-explore-report/SKILL.md +6 -2
- package/skills/qa-gather/SKILL.md +1 -1
- package/skills/qa-knowledge-add/SKILL.md +1 -1
- package/skills/qa-knowledge-list/SKILL.md +1 -1
- package/skills/qa-verify-backend/SKILL.md +1 -1
- package/skills/qa-verify-backend/phases/05-reporting.md +2 -1
- package/data/targets/testers-ai.yml +0 -14
|
@@ -5,7 +5,7 @@ description: >
|
|
|
5
5
|
No full site mapping — goes directly to the target, applies heuristics, reports findings.
|
|
6
6
|
Use when user says: "quick check", "test this page", "quick explore", or wants a fast review.
|
|
7
7
|
argument-hint: "<url> [--focus <area>]"
|
|
8
|
-
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx:*), Read, Write, Glob, Grep
|
|
8
|
+
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*), Read, Write, Glob, Grep
|
|
9
9
|
---
|
|
10
10
|
|
|
11
11
|
# Quick Exploratory Testing Session
|
|
@@ -32,7 +32,7 @@ Before opening the browser:
|
|
|
32
32
|
1. Resolve config per `paths.md`: `qa/target.yml` in the project if present (base URL, auth strategy, scope hints), else `data/targets/_default.yml`.
|
|
33
33
|
2. Credentials from `qa/.env`, else `.env`; never log the values.
|
|
34
34
|
3. Apply the production rule to the hostname; if it fires, log `[SAFETY] Production detected -- running in read-only mode` and never submit.
|
|
35
|
-
4. Create `output/sessions/<YYYY-MM-DD-HHmm>-quick-<slug>/` with `screenshots/`, `bugs/`, and a
|
|
35
|
+
4. Create `output/sessions/<YYYY-MM-DD-HHmm>-quick-<slug>/` with `screenshots/`, `bugs/`, `evidence/`, and a short `charter.md` (header, URL, focus, time box 15 min, read-only flag). Start `session-log.md`.
|
|
36
36
|
5. Choose the session id `-s=quick-<HHmm>-<slug>`. **Every `playwright-cli` call below carries `-s=<sid>`; the examples omit it.**
|
|
37
37
|
|
|
38
38
|
```bash
|
|
@@ -54,6 +54,16 @@ From the snapshot, identify:
|
|
|
54
54
|
|
|
55
55
|
`playwright-cli find "<text>"` locates a label without re-reading the whole tree.
|
|
56
56
|
|
|
57
|
+
Then add the areas you will test to `charter.md` — 1 to 3 rows, IDs `A1`, `A2`, `A3`, never
|
|
58
|
+
renumbered or removed:
|
|
59
|
+
|
|
60
|
+
```markdown
|
|
61
|
+
## Feature Risk Ranking
|
|
62
|
+
| ID | Feature | Risk |
|
|
63
|
+
|----|---------|------|
|
|
64
|
+
| A1 | [the page or feature under test] | P1 |
|
|
65
|
+
```
|
|
66
|
+
|
|
57
67
|
### 3. Focused Testing (10 min)
|
|
58
68
|
|
|
59
69
|
Based on what's on the page, apply relevant heuristics:
|
|
@@ -79,6 +89,14 @@ Based on what's on the page, apply relevant heuristics:
|
|
|
79
89
|
|
|
80
90
|
For each bug: `playwright-cli screenshot --filename=output/sessions/<session-dir>/screenshots/BUG-NNN.png` and document immediately.
|
|
81
91
|
|
|
92
|
+
When you are done with an area — fully tested or only partly — capture it under its charter ID
|
|
93
|
+
(lowercase ASCII slug; at least one per area you will report `tested` or `partial`; nothing
|
|
94
|
+
under `snapshots/` counts):
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
playwright-cli screenshot --filename=output/sessions/<session-dir>/screenshots/A<N>-<slug>.png
|
|
98
|
+
```
|
|
99
|
+
|
|
82
100
|
### 4. Report (2 min)
|
|
83
101
|
|
|
84
102
|
A quick session writes the **same** artefacts as a full one — never a quick-only report
|
|
@@ -92,17 +110,37 @@ starts with these two lines, verbatim:
|
|
|
92
110
|
|
|
93
111
|
- `bugs/BUG-NNN.md` — one file per bug, header first, then
|
|
94
112
|
`# BUG-NNN: [Component] fails [Condition] causing [Impact]`, then `**Severity:**`,
|
|
95
|
-
`**Priority:**`, `**Component:**`, `**
|
|
113
|
+
`**Priority:**`, `**Component:**`, `**Area:** A<N>` (the charter row it was found in, or
|
|
114
|
+
`none`; never an area you report `not-tested`, `blocked` or `deferred`), `**URL:**`,
|
|
96
115
|
`**Environment:** Playwright CLI, Chromium, <viewport>`, `**Reproduction rate:**`, then
|
|
97
116
|
`## Summary`, `## Expected Behavior`, `## Actual Behavior`, `## Steps to Reproduce`,
|
|
98
117
|
`## Business Impact` (mandatory: Revenue / Trust / Regulatory / Data / Scale, "none
|
|
99
118
|
identified" where it does not apply), `## Evidence` (`- Screenshot:`, `- Video:`,
|
|
100
119
|
`- Log:`, `- Console errors:`, `- Network failures:`) and `## Recommended Fix Priority`
|
|
101
|
-
- `
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
120
|
+
- `stats.json` — `kind: "quick"`, the eight required fields, `"contract": 2`, and one
|
|
121
|
+
`coverage.areas` entry per charter row: `{ "id": "A1", "status": "tested", "evidence": ["screenshots/A1-<slug>.png"] }`.
|
|
122
|
+
Status is `tested`, `partial`, `blocked`, `not-tested` or `deferred` (not reached in the
|
|
123
|
+
time box); every status but `tested` carries a `reason` of at most 160 characters with no
|
|
124
|
+
URL and no personal data; `evidence` paths sit under `screenshots/`, `videos/`, `traces/`,
|
|
125
|
+
`logs/` or `evidence/`, never `snapshots/`. Quick has no `--continue`, so no `continues`.
|
|
126
|
+
Never write `coverage_level`.
|
|
127
|
+
- then compute the coverage level — it validates the charter table, the areas, the evidence
|
|
128
|
+
and every bug's `**Area:**` line, and on exit 1 names each violation and its fix:
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
qualiow session level output/sessions/<session-dir> --write
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
On exit 0 it writes `evidence-level.md`, `backlog.md` and `stats.json` `coverage_level`.
|
|
135
|
+
Never write or edit those by hand. A quick session's level is never higher than
|
|
136
|
+
`qualified`. The level is a computed coverage fact, never a ship probability or a release
|
|
137
|
+
verdict.
|
|
138
|
+
- `session-report.md` — `Kind: quick`; `## Executive Summary`; `## Coverage Level` (the body
|
|
139
|
+
of `evidence-level.md` after its confidentiality header, copied verbatim — never
|
|
140
|
+
recomputed or reworded); `## Summary Stats`;
|
|
141
|
+
`## Coverage Map` (`| Area | Risk | Status | Bugs | Notes |`, one row per charter area,
|
|
142
|
+
the Area cell starting with its ID); `## Bugs Found`; `## Observations`;
|
|
143
|
+
`## Areas Not Tested`; `## Recommendations`; `## Session Stats`
|
|
106
144
|
|
|
107
145
|
Then finalize:
|
|
108
146
|
|
|
@@ -111,7 +149,8 @@ qualiow session finalize output/sessions/<session-dir>
|
|
|
111
149
|
```
|
|
112
150
|
|
|
113
151
|
It validates `stats.json`, checks the confidentiality header on every artefact and scans them
|
|
114
|
-
against the redaction list,
|
|
152
|
+
against the redaction list, re-checks contract 2 (a level that is missing or stale after a later
|
|
153
|
+
edit: re-run `qualiow session level <dir> --write`), then appends the session row to `output/sessions/INDEX.md` and one
|
|
115
154
|
row per bug to `output/bugs/all-bugs.md` and records the metrics. On exit 1 it names each
|
|
116
155
|
offending file: fix it and run again (`--check` validates without writing). Resolve the
|
|
117
156
|
`qualiow` prefix per `${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`.
|
|
@@ -5,7 +5,7 @@ description: >
|
|
|
5
5
|
Use when user says: "show report", "generate report", "reformat bugs", "session summary",
|
|
6
6
|
or wants to review past session findings.
|
|
7
7
|
argument-hint: "[<session-dir> | latest] [--format markdown|summary|bugs-only|coverage-map] [--overwrite]"
|
|
8
|
-
allowed-tools: Read, Write, Glob, Grep, Bash(qualiow:*), Bash(npx:*), Bash(wc:*)
|
|
8
|
+
allowed-tools: Read, Write, Glob, Grep, Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*), Bash(wc:*)
|
|
9
9
|
context: fork
|
|
10
10
|
agent: qa-reporting-agent
|
|
11
11
|
---
|
|
@@ -45,6 +45,7 @@ directory; list the candidates if it is ambiguous.
|
|
|
45
45
|
- `phase-7-notes.md` — the session's own summary, coverage rows, observations, recommendations and reflection; copy those sections, never invent them
|
|
46
46
|
- `phase-3-discovery.md`, `phase-4-journeys.md`, `phase-5-features.md`, `phase-6-edge-cases.md` (explore and mobile sessions; quick sessions have none; backend sessions have `ac-matrix.md`, `evidence/`, `expected-behaviour.md` instead)
|
|
47
47
|
- `bugs/BUG-*.md`
|
|
48
|
+
- `evidence-level.md`, when it exists (a contract-2 session) — CLI-written; copied, never recomputed
|
|
48
49
|
- the existing `session-report.md`, if any (its coverage map is the coverage source)
|
|
49
50
|
|
|
50
51
|
Size a file before you open it (`wc -l <file>`). A phase file or a report over 300 lines is
|
|
@@ -59,7 +60,10 @@ it here.
|
|
|
59
60
|
Map with `| Area | Risk | Status | Bugs | Notes |`, Bugs Found, Observations, Areas Not
|
|
60
61
|
Tested, Recommendations, Reflection, Session Stats; mobile and backend additions preserved;
|
|
61
62
|
for a session with a `verification/` directory, the `Verification` column in Bugs Found and
|
|
62
|
-
`## Refuted Findings` after Session Stats
|
|
63
|
+
`## Refuted Findings` after Session Stats; for a session whose `evidence-level.md` exists,
|
|
64
|
+
`## Coverage Level` right after Executive Summary holding that file's body verbatim — all as
|
|
65
|
+
`qa-reporting-agent` defines them). Never run `qualiow session level` from here: the level belongs
|
|
66
|
+
to the session that computed it, and the command refuses a finalized session.
|
|
63
67
|
|
|
64
68
|
**summary**: a brief executive summary: target, date, duration; bug count by severity; top 3
|
|
65
69
|
findings; key recommendation.
|
|
@@ -7,7 +7,7 @@ description: >
|
|
|
7
7
|
Use when user says: "gather requirements", "prepare context", "analyze this ticket",
|
|
8
8
|
"what should I test", or provides requirements documents before an explore session.
|
|
9
9
|
argument-hint: "<file | url | text> [--domain <id>] [--output <file>]"
|
|
10
|
-
allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(git:*), Bash(wc:*)
|
|
10
|
+
allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(git diff:*), Bash(git log:*), Bash(git show:*), Bash(wc:*)
|
|
11
11
|
context: fork
|
|
12
12
|
agent: qa-gather-agent
|
|
13
13
|
---
|
|
@@ -6,7 +6,7 @@ description: >
|
|
|
6
6
|
Use when user says: "add knowledge", "add heuristic", "learn this", "save this technique",
|
|
7
7
|
"add to knowledge base", or provides QA documentation to ingest.
|
|
8
8
|
argument-hint: "[<text> | <url> | <file>]"
|
|
9
|
-
allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(
|
|
9
|
+
allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*)
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# Add QA Knowledge
|
|
@@ -6,7 +6,7 @@ description: >
|
|
|
6
6
|
Use when user says: "list knowledge", "show heuristics", "what's in the knowledge base",
|
|
7
7
|
"knowledge stats", "show changelog".
|
|
8
8
|
argument-hint: "[--domain <id>] [--tag <tag>] [--type <type>] [--entry <id>] [--changelog] [--stats]"
|
|
9
|
-
allowed-tools: Read, Grep, Bash(qualiow:*), Bash(npx:*)
|
|
9
|
+
allowed-tools: Read, Grep, Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*)
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# Browse QA Knowledge Base
|
|
@@ -11,7 +11,7 @@ description: >
|
|
|
11
11
|
"test the API", "check the endpoint", or gives a backend/API/infra ticket whose ACs
|
|
12
12
|
cannot be seen in a browser.
|
|
13
13
|
argument-hint: "[--target <id>] [--context <file>] [--static-only] [--api-only] [--no-e2e] [--parity <target-id>]"
|
|
14
|
-
allowed-tools: Read, Write, Glob, Grep, Bash(git:*), Bash(aws:*), Bash(jq:*), Bash(grep:*), Bash(curl:*), Bash(node:*), Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx:*), Bash(terraform validate:*), Bash(terraform fmt:*)
|
|
14
|
+
allowed-tools: Read, Write, Glob, Grep, Bash(git -C:*), Bash(git show:*), Bash(git diff:*), Bash(git merge-base:*), Bash(git rev-parse:*), Bash(aws:*), Bash(jq:*), Bash(grep:*), Bash(curl:*), Bash(node --env-file=qa/.env:*), Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*), Bash(terraform validate:*), Bash(terraform fmt:*)
|
|
15
15
|
---
|
|
16
16
|
|
|
17
17
|
# Backend & Infrastructure AC Verification
|
|
@@ -168,7 +168,8 @@ install) with the session directory and `kind: backend`. It assembles `session-r
|
|
|
168
168
|
`phase-7-notes.md`, `ac-matrix.md`, the bugs and `stats.json`, copying your summary,
|
|
169
169
|
recommendations and reflection verbatim, and then runs `qualiow session finalize`.
|
|
170
170
|
|
|
171
|
-
That scans every
|
|
171
|
+
That scans every text file in the session, whatever its extension (`.txt`, `.har`, `.csv`
|
|
172
|
+
included; only the top-level `snapshots/` is exempt), with the same redaction list `src/utils/redact.ts` implements — the
|
|
172
173
|
list in `security-rules.md`: tokens, keys, cookies, real user emails, consumer PII, account
|
|
173
174
|
ids not already in the target — and the raw probe output under `evidence/` is scanned like
|
|
174
175
|
everything else. It also checks the confidentiality header on every markdown file and
|
|
@@ -1,14 +0,0 @@
|
|
|
1
|
-
id: testers-ai
|
|
2
|
-
name: "Testers.ai Testing Playground"
|
|
3
|
-
base_url: "https://testers.ai/testing/"
|
|
4
|
-
domain: _default
|
|
5
|
-
auth:
|
|
6
|
-
strategy: none
|
|
7
|
-
browser:
|
|
8
|
-
headless: false
|
|
9
|
-
viewport: { width: 1280, height: 720 }
|
|
10
|
-
scope:
|
|
11
|
-
start_pages: ["/testing/"]
|
|
12
|
-
include_patterns: ["/testing/**"]
|
|
13
|
-
max_depth: 3
|
|
14
|
-
notes: "Intentional test pages with 200+ known issues. Perfect for validating exploratory testing skills."
|