qualiow-exploratory-testing 2.2.0 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/.env.example +6 -0
- package/CHANGELOG.md +121 -0
- package/README.md +90 -14
- package/agents/qa-bug-judge.md +115 -0
- package/agents/qa-reporting-agent.md +14 -3
- package/bin/qualiow +20 -1
- package/data/security/SECURITY-POLICY.md +17 -1
- package/data/templates/bug-report.md +11 -0
- package/data/templates/session-report.md +12 -4
- package/dist/cli/index.js +1293 -48
- package/dist/index.d.ts +619 -1
- package/dist/index.js +1187 -19
- package/docs/GETTING-STARTED.md +25 -3
- package/package.json +2 -2
- package/skills/qa-explore/SKILL.md +8 -4
- package/skills/qa-explore/phases/00-setup.md +9 -1
- package/skills/qa-explore/phases/07-reporting.md +186 -11
- package/skills/qa-explore/references/delegation-rules.md +21 -2
- package/skills/qa-explore/references/evidence-triage.md +181 -0
- package/skills/qa-explore/references/output-contract.md +40 -7
- package/skills/qa-explore/references/security-rules.md +17 -0
- package/skills/qa-explore-report/SKILL.md +3 -1
|
@@ -26,6 +26,8 @@ session report, coverage map, AC matrix, expected-behaviour spec):
|
|
|
26
26
|
| `videos/*.webm` (web) / `videos/*.mp4` (mobile) | recording |
|
|
27
27
|
| `snapshots/*.yml` | raw accessibility trees from `playwright-cli --raw snapshot`; working files, never shipped, and excluded from the secrets scan |
|
|
28
28
|
| `phase-7-notes.md` | phase 7 (backend: phase 5) — the session's own executive summary, coverage rows, observations, areas not tested, recommendations and reflection; the input the report is assembled from |
|
|
29
|
+
| `verification/claims/CLAIM-NNN.md`, `verification/VERDICT-NNN.md`, `verification/proposed-patterns.md` | phase 7 verification (explore) — the claim card the `qa-bug-judge` sub-agent sees, its verdict copied verbatim, and the false-positive patterns it proposed; `verification/drafts/BUG-NNN.md` holds each draft until its verdict sorts it and is then left as a working file. All header-first, all secret-scanned. Under `verification.mode: triage-shadow`, also `verification/JEV-NNN.md` / `LAYA-NNN.md` (the advisory triage block) and `.json` (the exact request sent) |
|
|
30
|
+
| `bugs/refuted/BUG-NNN.md` | phase 7 — a candidate the judge REFUTED or found UNREPRODUCIBLE, with the verdict under `## Refutation (Judge)`. Excluded from `bugs_found`, the index rows and `## Bugs Found`; listed only in `## Refuted Findings` |
|
|
29
31
|
| `bugs/BUG-NNN.md` | phase 7 |
|
|
30
32
|
| `session-report.md` | the `qa-reporting-agent` sub-agent, from `phase-7-notes.md`, the bugs, `stats.json` and the phase files (quick sessions write it directly) |
|
|
31
33
|
| `stats.json` | phase 7 |
|
|
@@ -48,6 +50,7 @@ The coverage map lives inside `session-report.md`. There is no separate session-
|
|
|
48
50
|
**URL:** <exact URL — mobile: screen name or deep link — backend: endpoint or resource>
|
|
49
51
|
**Environment:** <browser + viewport — mobile: mode / platform / device / OS — backend: env kind + build>
|
|
50
52
|
**Reproduction rate:** Always | Intermittent (~X%) | Once
|
|
53
|
+
**Verification:** Verified | Verified (severity adjusted from <X>) | Unverified (<reason>) | Refuted | Unreproducible
|
|
51
54
|
|
|
52
55
|
## Summary
|
|
53
56
|
<two sentences: what breaks and why it matters>
|
|
@@ -77,11 +80,24 @@ The coverage map lives inside `session-report.md`. There is no separate session-
|
|
|
77
80
|
|
|
78
81
|
## Recommended Fix Priority
|
|
79
82
|
<why this priority, relative to the other findings>
|
|
83
|
+
|
|
84
|
+
## Verification
|
|
85
|
+
- Verdict: <CONFIRMED | CONFIRMED-ADJUSTED | UNVERIFIED> (<live-repro | evidence-only>, <confidence>)
|
|
86
|
+
- Judge repro result: <one line>
|
|
87
|
+
- Severity: <original X → final Y — only when adjusted>
|
|
88
|
+
- Full verdict: `../verification/VERDICT-NNN.md`
|
|
89
|
+
- Triage (advisory): <predicted verdict> · <ROUTE> · P(refuted)=<x> — `../verification/JEV-NNN.md` (only under `triage-shadow`)
|
|
80
90
|
```
|
|
81
91
|
|
|
92
|
+
The `**Verification:**` line and the `## Verification` section are present only when the
|
|
93
|
+
session ran the adversarial judge (`/qa-explore` does unless verification is `off`). A bug
|
|
94
|
+
under `bugs/refuted/` carries `**Verification:** Refuted` or `Unreproducible` and, instead of
|
|
95
|
+
`## Verification`, a `## Refutation (Judge)` section holding the verdict block verbatim.
|
|
96
|
+
|
|
82
97
|
Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is mandatory
|
|
83
98
|
(answer at least one bullet, write "none identified" for the rest); severity per
|
|
84
|
-
`severity-guide.md`, and when in doubt go one level lower.
|
|
99
|
+
`severity-guide.md`, and when in doubt go one level lower. Ids are assigned to the drafts
|
|
100
|
+
and never renumbered: a gap in `bugs/` means that candidate moved to `bugs/refuted/`.
|
|
85
101
|
|
|
86
102
|
## Session report — `session-report.md`
|
|
87
103
|
|
|
@@ -124,9 +140,9 @@ Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is
|
|
|
124
140
|
| <area> | P2 | not-tested | — | time ran out |
|
|
125
141
|
|
|
126
142
|
## Bugs Found
|
|
127
|
-
| # | ID | Title | Severity | Report |
|
|
128
|
-
|
|
129
|
-
| 1 | BUG-001 | <title> | High | bugs/BUG-001.md |
|
|
143
|
+
| # | ID | Title | Severity | Report | Verification |
|
|
144
|
+
|---|---|---|---|---|---|
|
|
145
|
+
| 1 | BUG-001 | <title> | High | bugs/BUG-001.md | Verified |
|
|
130
146
|
|
|
131
147
|
## Observations
|
|
132
148
|
## Areas Not Tested
|
|
@@ -138,6 +154,10 @@ Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is
|
|
|
138
154
|
## Session Stats
|
|
139
155
|
| Metric | Value |
|
|
140
156
|
|---|---|
|
|
157
|
+
|
|
158
|
+
## Refuted Findings
|
|
159
|
+
| ID | Claimed Title | Claimed Severity | Verdict | Refutation |
|
|
160
|
+
|---|---|---|---|---|
|
|
141
161
|
```
|
|
142
162
|
|
|
143
163
|
Coverage `Status` ∈ `tested` · `partial` · `not-tested` · `code-verified-only` (believed
|
|
@@ -145,7 +165,11 @@ correct from reading source, never observed: this is UNVERIFIABLE, not a pass, a
|
|
|
145
165
|
be skimmed as green).
|
|
146
166
|
|
|
147
167
|
Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
|
|
148
|
-
`## AC Matrix Summary` (all six verdicts counted) after the executive summary.
|
|
168
|
+
`## AC Matrix Summary` (all six verdicts counted) after the executive summary. A session
|
|
169
|
+
that ran the bug judge (a `verification/` directory exists) carries the sixth
|
|
170
|
+
`Verification` column in `## Bugs Found` and appends `## Refuted Findings` after
|
|
171
|
+
`## Session Stats` — one row per file in `bugs/refuted/`, or the sentence "All candidate
|
|
172
|
+
bugs survived verification." A session without `verification/` has neither.
|
|
149
173
|
|
|
150
174
|
## `stats.json`
|
|
151
175
|
|
|
@@ -164,7 +188,7 @@ Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
|
|
|
164
188
|
"completed_at": "<ISO timestamp>",
|
|
165
189
|
"phases_completed": 8,
|
|
166
190
|
"total_phases": 8,
|
|
167
|
-
"coverage": { "checklist_total": 22, "checklist_verified": 15, "data_integrity_checks": 6, "data_integrity_passed": 5 },
|
|
191
|
+
"coverage": { "checklist_total": 22, "checklist_verified": 15, "data_integrity_checks": 6, "data_integrity_passed": 5, "verification": { "judged": 5, "verified": 3, "unverified": 1, "refuted": 1, "unreproducible": 0, "budget_min": 15 } },
|
|
168
192
|
"evidence": { "screenshots": 9, "console_errors_found": 2, "network_failures_found": 1 },
|
|
169
193
|
"areas_not_tested": ["settings — time ran out"],
|
|
170
194
|
"blocked_by": null
|
|
@@ -173,7 +197,16 @@ Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
|
|
|
173
197
|
|
|
174
198
|
The first eight keys are required; the rest are optional. No other top-level keys (the
|
|
175
199
|
schema is strict). Mobile puts its device block under `coverage.mobile`; backend puts the
|
|
176
|
-
verdict counts under `coverage.verdicts
|
|
200
|
+
verdict counts under `coverage.verdicts`; a session that ran the bug judge puts its counts
|
|
201
|
+
under `coverage.verification` (`bugs_found` and `severity_counts` count shipped bugs only —
|
|
202
|
+
never the refuted ones). Its keys: `judged` = candidate bugs (drafts), whether or not the judge
|
|
203
|
+
reached them; `verified` = CONFIRMED + CONFIRMED-ADJUSTED, including overruled refutations;
|
|
204
|
+
`unverified` = UNVERIFIED plus not judged; `refuted`; `unreproducible` (the five sum to
|
|
205
|
+
`judged`); `budget_min` = the minutes allotted, not spent. The report's `## Session Stats`
|
|
206
|
+
renders them as one row, `| Bugs judged | <judged> (verified <n>, unverified <n>, refuted <n>, unreproducible <n>) |`. Under
|
|
207
|
+
`verification.mode: triage-shadow` the triage counts go under `coverage.verification.triage.<provider>`
|
|
208
|
+
(`triaged`, `refute_risk`, `unclear`, `likely_confirmed`, `agreed_with_judge`, `unavailable`,
|
|
209
|
+
`input_tokens`, `cost_usd`, `ms_total`).
|
|
177
210
|
|
|
178
211
|
## Index rows
|
|
179
212
|
|
|
@@ -86,3 +86,20 @@ The first two lines of every session artefact (see `output-contract.md`):
|
|
|
86
86
|
|
|
87
87
|
Session output stays on the local disk. Never send it to an external service, paste it
|
|
88
88
|
into a ticket without the user asking, or include it in anything published.
|
|
89
|
+
|
|
90
|
+
**One gated exception: the hosted decision-model triage.** `qualiow judge triage` sends data
|
|
91
|
+
off the machine only when the target config sets `verification.mode: triage-shadow`, lists
|
|
92
|
+
`typesafe` as a provider, and the key is present under the env var the config names. Per
|
|
93
|
+
candidate bug it sends one claim card (title, URL path, claimed severity, environment,
|
|
94
|
+
reproduction rate, expected and actual behaviour, steps), the console and network excerpts
|
|
95
|
+
written on it, the text evidence files it lists from inside the session directory (capped per
|
|
96
|
+
file), and the known false-positive pattern lead-ins. It never sends screenshots, video, the
|
|
97
|
+
storage state, the session log, the charter, the drafts, other claims or any verdict. Before
|
|
98
|
+
anything leaves, every string passes the redaction list, then every URL is cut to its path,
|
|
99
|
+
every dotted hostname becomes `[HOST]`, every absolute path under a common root its basename and every `.auth/`
|
|
100
|
+
reference `[AUTH_STATE]`; the exact request is saved as `verification/JEV-NNN.json`. The
|
|
101
|
+
scrubbing is pattern-based and does not remove IP addresses, single-label hostnames, URL paths,
|
|
102
|
+
or free-text product, company or people names. The self-hosted `laya` provider is not an
|
|
103
|
+
exception: its endpoint must be loopback, checked before anything is sent, and redirects are
|
|
104
|
+
refused. Off by default —
|
|
105
|
+
`${CLAUDE_SKILL_DIR}/references/evidence-triage.md`.
|
|
@@ -57,7 +57,9 @@ it here.
|
|
|
57
57
|
**markdown** (default): the full session report per `output-contract.md`
|
|
58
58
|
(`# Session Report — <target>`, Session Metadata, Executive Summary, Summary Stats, Coverage
|
|
59
59
|
Map with `| Area | Risk | Status | Bugs | Notes |`, Bugs Found, Observations, Areas Not
|
|
60
|
-
Tested, Recommendations, Reflection, Session Stats; mobile and backend additions preserved
|
|
60
|
+
Tested, Recommendations, Reflection, Session Stats; mobile and backend additions preserved;
|
|
61
|
+
for a session with a `verification/` directory, the `Verification` column in Bugs Found and
|
|
62
|
+
`## Refuted Findings` after Session Stats, as `qa-reporting-agent` defines them).
|
|
61
63
|
|
|
62
64
|
**summary**: a brief executive summary: target, date, duration; bug count by severity; top 3
|
|
63
65
|
findings; key recommendation.
|