qualiow-exploratory-testing 2.3.0 → 2.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. package/.claude-plugin/marketplace.json +1 -1
  2. package/.claude-plugin/plugin.json +1 -1
  3. package/CHANGELOG.md +177 -0
  4. package/README.md +71 -21
  5. package/agents/qa-bug-judge.md +1 -1
  6. package/agents/qa-diff-indexer-agent.md +1 -1
  7. package/agents/qa-gather-agent.md +1 -1
  8. package/agents/qa-page-mapper-agent.md +3 -1
  9. package/agents/qa-reporting-agent.md +28 -14
  10. package/bin/mobile-cli.mjs +74 -26
  11. package/bin/qualiow +7 -2
  12. package/bin/wk-ios +60 -6
  13. package/bin/wkeval.mjs +7 -2
  14. package/data/security/SECURITY-POLICY.md +22 -3
  15. package/data/targets/parabank.yml +13 -0
  16. package/data/targets/saucedemo.yml +13 -0
  17. package/data/templates/bug-report.md +3 -0
  18. package/data/templates/session-charter.md +7 -5
  19. package/data/templates/session-report.md +7 -1
  20. package/dist/cli/index.js +2060 -361
  21. package/dist/index.d.ts +813 -26
  22. package/dist/index.js +2694 -704
  23. package/docs/GETTING-STARTED.md +71 -17
  24. package/docs/MOBILE-SETUP.md +31 -0
  25. package/hooks/hooks.json +12 -2
  26. package/hooks/scripts/bash-guard.mjs +242 -0
  27. package/hooks/scripts/secret-patterns.d.mts +4 -1
  28. package/hooks/scripts/secret-patterns.mjs +126 -34
  29. package/hooks/scripts/write-guard.mjs +70 -15
  30. package/package.json +3 -2
  31. package/skills/qa-explore/SKILL.md +10 -3
  32. package/skills/qa-explore/phases/00-setup.md +40 -3
  33. package/skills/qa-explore/phases/02-charter.md +25 -5
  34. package/skills/qa-explore/phases/04-journeys.md +14 -0
  35. package/skills/qa-explore/phases/05-features.md +10 -0
  36. package/skills/qa-explore/phases/06-edge-cases.md +10 -0
  37. package/skills/qa-explore/phases/07-reporting.md +93 -11
  38. package/skills/qa-explore/references/delegation-rules.md +12 -1
  39. package/skills/qa-explore/references/evidence-triage.md +44 -12
  40. package/skills/qa-explore/references/output-contract.md +69 -7
  41. package/skills/qa-explore/references/security-rules.md +25 -3
  42. package/skills/qa-explore-cleanup/SKILL.md +1 -1
  43. package/skills/qa-explore-mobile/SKILL.md +13 -3
  44. package/skills/qa-explore-mobile/phases/02-charter.md +2 -1
  45. package/skills/qa-explore-mobile/phases/07-reporting.md +2 -0
  46. package/skills/qa-explore-mobile/references/mobile-edges.md +3 -1
  47. package/skills/qa-explore-quick/SKILL.md +48 -9
  48. package/skills/qa-explore-report/SKILL.md +6 -2
  49. package/skills/qa-gather/SKILL.md +1 -1
  50. package/skills/qa-knowledge-add/SKILL.md +1 -1
  51. package/skills/qa-knowledge-list/SKILL.md +1 -1
  52. package/skills/qa-verify-backend/SKILL.md +1 -1
  53. package/skills/qa-verify-backend/phases/05-reporting.md +2 -1
  54. package/data/targets/testers-ai.yml +0 -14
@@ -5,7 +5,7 @@ description: >
5
5
  No full site mapping — goes directly to the target, applies heuristics, reports findings.
6
6
  Use when user says: "quick check", "test this page", "quick explore", or wants a fast review.
7
7
  argument-hint: "<url> [--focus <area>]"
8
- allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx:*), Read, Write, Glob, Grep
8
+ allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*), Read, Write, Glob, Grep
9
9
  ---
10
10
 
11
11
  # Quick Exploratory Testing Session
@@ -32,7 +32,7 @@ Before opening the browser:
32
32
  1. Resolve config per `paths.md`: `qa/target.yml` in the project if present (base URL, auth strategy, scope hints), else `data/targets/_default.yml`.
33
33
  2. Credentials from `qa/.env`, else `.env`; never log the values.
34
34
  3. Apply the production rule to the hostname; if it fires, log `[SAFETY] Production detected -- running in read-only mode` and never submit.
35
- 4. Create `output/sessions/<YYYY-MM-DD-HHmm>-quick-<slug>/` with `screenshots/`, `bugs/`, and a 5-line `charter.md` (header, URL, focus, time box 15 min, read-only flag). Start `session-log.md`.
35
+ 4. Create `output/sessions/<YYYY-MM-DD-HHmm>-quick-<slug>/` with `screenshots/`, `bugs/`, `evidence/`, and a short `charter.md` (header, URL, focus, time box 15 min, read-only flag). Start `session-log.md`.
36
36
  5. Choose the session id `-s=quick-<HHmm>-<slug>`. **Every `playwright-cli` call below carries `-s=<sid>`; the examples omit it.**
37
37
 
38
38
  ```bash
@@ -54,6 +54,16 @@ From the snapshot, identify:
54
54
 
55
55
  `playwright-cli find "<text>"` locates a label without re-reading the whole tree.
56
56
 
57
+ Then add the areas you will test to `charter.md` — 1 to 3 rows, IDs `A1`, `A2`, `A3`, never
58
+ renumbered or removed:
59
+
60
+ ```markdown
61
+ ## Feature Risk Ranking
62
+ | ID | Feature | Risk |
63
+ |----|---------|------|
64
+ | A1 | [the page or feature under test] | P1 |
65
+ ```
66
+
57
67
  ### 3. Focused Testing (10 min)
58
68
 
59
69
  Based on what's on the page, apply relevant heuristics:
@@ -79,6 +89,14 @@ Based on what's on the page, apply relevant heuristics:
79
89
 
80
90
  For each bug: `playwright-cli screenshot --filename=output/sessions/<session-dir>/screenshots/BUG-NNN.png` and document immediately.
81
91
 
92
+ When you are done with an area — fully tested or only partly — capture it under its charter ID
93
+ (lowercase ASCII slug; at least one per area you will report `tested` or `partial`; nothing
94
+ under `snapshots/` counts):
95
+
96
+ ```bash
97
+ playwright-cli screenshot --filename=output/sessions/<session-dir>/screenshots/A<N>-<slug>.png
98
+ ```
99
+
82
100
  ### 4. Report (2 min)
83
101
 
84
102
  A quick session writes the **same** artefacts as a full one — never a quick-only report
@@ -92,17 +110,37 @@ starts with these two lines, verbatim:
92
110
 
93
111
  - `bugs/BUG-NNN.md` — one file per bug, header first, then
94
112
  `# BUG-NNN: [Component] fails [Condition] causing [Impact]`, then `**Severity:**`,
95
- `**Priority:**`, `**Component:**`, `**URL:**`,
113
+ `**Priority:**`, `**Component:**`, `**Area:** A<N>` (the charter row it was found in, or
114
+ `none`; never an area you report `not-tested`, `blocked` or `deferred`), `**URL:**`,
96
115
  `**Environment:** Playwright CLI, Chromium, <viewport>`, `**Reproduction rate:**`, then
97
116
  `## Summary`, `## Expected Behavior`, `## Actual Behavior`, `## Steps to Reproduce`,
98
117
  `## Business Impact` (mandatory: Revenue / Trust / Regulatory / Data / Scale, "none
99
118
  identified" where it does not apply), `## Evidence` (`- Screenshot:`, `- Video:`,
100
119
  `- Log:`, `- Console errors:`, `- Network failures:`) and `## Recommended Fix Priority`
101
- - `session-report.md` — `Kind: quick`; `## Executive Summary`; `## Summary Stats`;
102
- `## Coverage Map` (`| Area | Risk | Status | Bugs | Notes |`, one row for the page or
103
- feature); `## Bugs Found`; `## Observations`; `## Areas Not Tested`; `## Recommendations`;
104
- `## Session Stats`
105
- - `stats.json` — `kind: "quick"`, the eight required fields
120
+ - `stats.json` — `kind: "quick"`, the eight required fields, `"contract": 2`, and one
121
+ `coverage.areas` entry per charter row: `{ "id": "A1", "status": "tested", "evidence": ["screenshots/A1-<slug>.png"] }`.
122
+ Status is `tested`, `partial`, `blocked`, `not-tested` or `deferred` (not reached in the
123
+ time box); every status but `tested` carries a `reason` of at most 160 characters with no
124
+ URL and no personal data; `evidence` paths sit under `screenshots/`, `videos/`, `traces/`,
125
+ `logs/` or `evidence/`, never `snapshots/`. Quick has no `--continue`, so no `continues`.
126
+ Never write `coverage_level`.
127
+ - then compute the coverage level — it validates the charter table, the areas, the evidence
128
+ and every bug's `**Area:**` line, and on exit 1 names each violation and its fix:
129
+
130
+ ```bash
131
+ qualiow session level output/sessions/<session-dir> --write
132
+ ```
133
+
134
+ On exit 0 it writes `evidence-level.md`, `backlog.md` and `stats.json` `coverage_level`.
135
+ Never write or edit those by hand. A quick session's level is never higher than
136
+ `qualified`. The level is a computed coverage fact, never a ship probability or a release
137
+ verdict.
138
+ - `session-report.md` — `Kind: quick`; `## Executive Summary`; `## Coverage Level` (the body
139
+ of `evidence-level.md` after its confidentiality header, copied verbatim — never
140
+ recomputed or reworded); `## Summary Stats`;
141
+ `## Coverage Map` (`| Area | Risk | Status | Bugs | Notes |`, one row per charter area,
142
+ the Area cell starting with its ID); `## Bugs Found`; `## Observations`;
143
+ `## Areas Not Tested`; `## Recommendations`; `## Session Stats`
106
144
 
107
145
  Then finalize:
108
146
 
@@ -111,7 +149,8 @@ qualiow session finalize output/sessions/<session-dir>
111
149
  ```
112
150
 
113
151
  It validates `stats.json`, checks the confidentiality header on every artefact and scans them
114
- against the redaction list, then appends the session row to `output/sessions/INDEX.md` and one
152
+ against the redaction list, re-checks contract 2 (a level that is missing or stale after a later
153
+ edit: re-run `qualiow session level <dir> --write`), then appends the session row to `output/sessions/INDEX.md` and one
115
154
  row per bug to `output/bugs/all-bugs.md` and records the metrics. On exit 1 it names each
116
155
  offending file: fix it and run again (`--check` validates without writing). Resolve the
117
156
  `qualiow` prefix per `${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`.
@@ -5,7 +5,7 @@ description: >
5
5
  Use when user says: "show report", "generate report", "reformat bugs", "session summary",
6
6
  or wants to review past session findings.
7
7
  argument-hint: "[<session-dir> | latest] [--format markdown|summary|bugs-only|coverage-map] [--overwrite]"
8
- allowed-tools: Read, Write, Glob, Grep, Bash(qualiow:*), Bash(npx:*), Bash(wc:*)
8
+ allowed-tools: Read, Write, Glob, Grep, Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*), Bash(wc:*)
9
9
  context: fork
10
10
  agent: qa-reporting-agent
11
11
  ---
@@ -45,6 +45,7 @@ directory; list the candidates if it is ambiguous.
45
45
  - `phase-7-notes.md` — the session's own summary, coverage rows, observations, recommendations and reflection; copy those sections, never invent them
46
46
  - `phase-3-discovery.md`, `phase-4-journeys.md`, `phase-5-features.md`, `phase-6-edge-cases.md` (explore and mobile sessions; quick sessions have none; backend sessions have `ac-matrix.md`, `evidence/`, `expected-behaviour.md` instead)
47
47
  - `bugs/BUG-*.md`
48
+ - `evidence-level.md`, when it exists (a contract-2 session) — CLI-written; copied, never recomputed
48
49
  - the existing `session-report.md`, if any (its coverage map is the coverage source)
49
50
 
50
51
  Size a file before you open it (`wc -l <file>`). A phase file or a report over 300 lines is
@@ -59,7 +60,10 @@ it here.
59
60
  Map with `| Area | Risk | Status | Bugs | Notes |`, Bugs Found, Observations, Areas Not
60
61
  Tested, Recommendations, Reflection, Session Stats; mobile and backend additions preserved;
61
62
  for a session with a `verification/` directory, the `Verification` column in Bugs Found and
62
- `## Refuted Findings` after Session Stats, as `qa-reporting-agent` defines them).
63
+ `## Refuted Findings` after Session Stats; for a session whose `evidence-level.md` exists,
64
+ `## Coverage Level` right after Executive Summary holding that file's body verbatim — all as
65
+ `qa-reporting-agent` defines them). Never run `qualiow session level` from here: the level belongs
66
+ to the session that computed it, and the command refuses a finalized session.
63
67
 
64
68
  **summary**: a brief executive summary: target, date, duration; bug count by severity; top 3
65
69
  findings; key recommendation.
@@ -7,7 +7,7 @@ description: >
7
7
  Use when user says: "gather requirements", "prepare context", "analyze this ticket",
8
8
  "what should I test", or provides requirements documents before an explore session.
9
9
  argument-hint: "<file | url | text> [--domain <id>] [--output <file>]"
10
- allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(git:*), Bash(wc:*)
10
+ allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(git diff:*), Bash(git log:*), Bash(git show:*), Bash(wc:*)
11
11
  context: fork
12
12
  agent: qa-gather-agent
13
13
  ---
@@ -6,7 +6,7 @@ description: >
6
6
  Use when user says: "add knowledge", "add heuristic", "learn this", "save this technique",
7
7
  "add to knowledge base", or provides QA documentation to ingest.
8
8
  argument-hint: "[<text> | <url> | <file>]"
9
- allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(node:*), Bash(npx qualiow:*), Bash(npx:*), Bash(qualiow:*)
9
+ allowed-tools: Read, Write, Glob, Grep, WebFetch, Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*)
10
10
  ---
11
11
 
12
12
  # Add QA Knowledge
@@ -6,7 +6,7 @@ description: >
6
6
  Use when user says: "list knowledge", "show heuristics", "what's in the knowledge base",
7
7
  "knowledge stats", "show changelog".
8
8
  argument-hint: "[--domain <id>] [--tag <tag>] [--type <type>] [--entry <id>] [--changelog] [--stats]"
9
- allowed-tools: Read, Grep, Bash(qualiow:*), Bash(npx:*)
9
+ allowed-tools: Read, Grep, Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*)
10
10
  ---
11
11
 
12
12
  # Browse QA Knowledge Base
@@ -11,7 +11,7 @@ description: >
11
11
  "test the API", "check the endpoint", or gives a backend/API/infra ticket whose ACs
12
12
  cannot be seen in a browser.
13
13
  argument-hint: "[--target <id>] [--context <file>] [--static-only] [--api-only] [--no-e2e] [--parity <target-id>]"
14
- allowed-tools: Read, Write, Glob, Grep, Bash(git:*), Bash(aws:*), Bash(jq:*), Bash(grep:*), Bash(curl:*), Bash(node:*), Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx:*), Bash(terraform validate:*), Bash(terraform fmt:*)
14
+ allowed-tools: Read, Write, Glob, Grep, Bash(git -C:*), Bash(git show:*), Bash(git diff:*), Bash(git merge-base:*), Bash(git rev-parse:*), Bash(aws:*), Bash(jq:*), Bash(grep:*), Bash(curl:*), Bash(node --env-file=qa/.env:*), Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx -y -p qualiow-exploratory-testing qualiow:*), Bash(terraform validate:*), Bash(terraform fmt:*)
15
15
  ---
16
16
 
17
17
  # Backend & Infrastructure AC Verification
@@ -168,7 +168,8 @@ install) with the session directory and `kind: backend`. It assembles `session-r
168
168
  `phase-7-notes.md`, `ac-matrix.md`, the bugs and `stats.json`, copying your summary,
169
169
  recommendations and reflection verbatim, and then runs `qualiow session finalize`.
170
170
 
171
- That scans every artefact with the same redaction list `src/utils/redact.ts` implements — the
171
+ That scans every text file in the session, whatever its extension (`.txt`, `.har`, `.csv`
172
+ included; only the top-level `snapshots/` is exempt), with the same redaction list `src/utils/redact.ts` implements — the
172
173
  list in `security-rules.md`: tokens, keys, cookies, real user emails, consumer PII, account
173
174
  ids not already in the target — and the raw probe output under `evidence/` is scanned like
174
175
  everything else. It also checks the confidentiality header on every markdown file and
@@ -1,14 +0,0 @@
1
- id: testers-ai
2
- name: "Testers.ai Testing Playground"
3
- base_url: "https://testers.ai/testing/"
4
- domain: _default
5
- auth:
6
- strategy: none
7
- browser:
8
- headless: false
9
- viewport: { width: 1280, height: 720 }
10
- scope:
11
- start_pages: ["/testing/"]
12
- include_patterns: ["/testing/**"]
13
- max_depth: 3
14
- notes: "Intentional test pages with 200+ known issues. Perfect for validating exploratory testing skills."