qualiow-exploratory-testing 2.2.0 → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -26,6 +26,8 @@ session report, coverage map, AC matrix, expected-behaviour spec):
26
26
  | `videos/*.webm` (web) / `videos/*.mp4` (mobile) | recording |
27
27
  | `snapshots/*.yml` | raw accessibility trees from `playwright-cli --raw snapshot`; working files, never shipped, and excluded from the secrets scan |
28
28
  | `phase-7-notes.md` | phase 7 (backend: phase 5) — the session's own executive summary, coverage rows, observations, areas not tested, recommendations and reflection; the input the report is assembled from |
29
+ | `verification/claims/CLAIM-NNN.md`, `verification/VERDICT-NNN.md`, `verification/proposed-patterns.md` | phase 7 verification (explore) — the claim card the `qa-bug-judge` sub-agent sees, its verdict copied verbatim, and the false-positive patterns it proposed; `verification/drafts/BUG-NNN.md` holds each draft until its verdict sorts it and is then left as a working file. All header-first, all secret-scanned. Under `verification.mode: triage-shadow`, also `verification/JEV-NNN.md` / `LAYA-NNN.md` (the advisory triage block) and `.json` (the exact request sent) |
30
+ | `bugs/refuted/BUG-NNN.md` | phase 7 — a candidate the judge REFUTED or found UNREPRODUCIBLE, with the verdict under `## Refutation (Judge)`. Excluded from `bugs_found`, the index rows and `## Bugs Found`; listed only in `## Refuted Findings` |
29
31
  | `bugs/BUG-NNN.md` | phase 7 |
30
32
  | `session-report.md` | the `qa-reporting-agent` sub-agent, from `phase-7-notes.md`, the bugs, `stats.json` and the phase files (quick sessions write it directly) |
31
33
  | `stats.json` | phase 7 |
@@ -48,6 +50,7 @@ The coverage map lives inside `session-report.md`. There is no separate session-
48
50
  **URL:** <exact URL — mobile: screen name or deep link — backend: endpoint or resource>
49
51
  **Environment:** <browser + viewport — mobile: mode / platform / device / OS — backend: env kind + build>
50
52
  **Reproduction rate:** Always | Intermittent (~X%) | Once
53
+ **Verification:** Verified | Verified (severity adjusted from <X>) | Unverified (<reason>) | Refuted | Unreproducible
51
54
 
52
55
  ## Summary
53
56
  <two sentences: what breaks and why it matters>
@@ -77,11 +80,24 @@ The coverage map lives inside `session-report.md`. There is no separate session-
77
80
 
78
81
  ## Recommended Fix Priority
79
82
  <why this priority, relative to the other findings>
83
+
84
+ ## Verification
85
+ - Verdict: <CONFIRMED | CONFIRMED-ADJUSTED | UNVERIFIED> (<live-repro | evidence-only>, <confidence>)
86
+ - Judge repro result: <one line>
87
+ - Severity: <original X → final Y — only when adjusted>
88
+ - Full verdict: `../verification/VERDICT-NNN.md`
89
+ - Triage (advisory): <predicted verdict> · <ROUTE> · P(refuted)=<x> — `../verification/JEV-NNN.md` (only under `triage-shadow`)
80
90
  ```
81
91
 
92
+ The `**Verification:**` line and the `## Verification` section are present only when the
93
+ session ran the adversarial judge (`/qa-explore` does unless verification is `off`). A bug
94
+ under `bugs/refuted/` carries `**Verification:** Refuted` or `Unreproducible` and, instead of
95
+ `## Verification`, a `## Refutation (Judge)` section holding the verdict block verbatim.
96
+
82
97
  Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is mandatory
83
98
  (answer at least one bullet, write "none identified" for the rest); severity per
84
- `severity-guide.md`, and when in doubt go one level lower.
99
+ `severity-guide.md`, and when in doubt go one level lower. Ids are assigned to the drafts
100
+ and never renumbered: a gap in `bugs/` means that candidate moved to `bugs/refuted/`.
85
101
 
86
102
  ## Session report — `session-report.md`
87
103
 
@@ -124,9 +140,9 @@ Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is
124
140
  | <area> | P2 | not-tested | — | time ran out |
125
141
 
126
142
  ## Bugs Found
127
- | # | ID | Title | Severity | Report |
128
- |---|---|---|---|---|
129
- | 1 | BUG-001 | <title> | High | bugs/BUG-001.md |
143
+ | # | ID | Title | Severity | Report | Verification |
144
+ |---|---|---|---|---|---|
145
+ | 1 | BUG-001 | <title> | High | bugs/BUG-001.md | Verified |
130
146
 
131
147
  ## Observations
132
148
  ## Areas Not Tested
@@ -138,6 +154,10 @@ Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is
138
154
  ## Session Stats
139
155
  | Metric | Value |
140
156
  |---|---|
157
+
158
+ ## Refuted Findings
159
+ | ID | Claimed Title | Claimed Severity | Verdict | Refutation |
160
+ |---|---|---|---|---|
141
161
  ```
142
162
 
143
163
  Coverage `Status` ∈ `tested` · `partial` · `not-tested` · `code-verified-only` (believed
@@ -145,7 +165,11 @@ correct from reading source, never observed: this is UNVERIFIABLE, not a pass, a
145
165
  be skimmed as green).
146
166
 
147
167
  Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
148
- `## AC Matrix Summary` (all six verdicts counted) after the executive summary.
168
+ `## AC Matrix Summary` (all six verdicts counted) after the executive summary. A session
169
+ that ran the bug judge (a `verification/` directory exists) carries the sixth
170
+ `Verification` column in `## Bugs Found` and appends `## Refuted Findings` after
171
+ `## Session Stats` — one row per file in `bugs/refuted/`, or the sentence "All candidate
172
+ bugs survived verification." A session without `verification/` has neither.
149
173
 
150
174
  ## `stats.json`
151
175
 
@@ -164,7 +188,7 @@ Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
164
188
  "completed_at": "<ISO timestamp>",
165
189
  "phases_completed": 8,
166
190
  "total_phases": 8,
167
- "coverage": { "checklist_total": 22, "checklist_verified": 15, "data_integrity_checks": 6, "data_integrity_passed": 5 },
191
+ "coverage": { "checklist_total": 22, "checklist_verified": 15, "data_integrity_checks": 6, "data_integrity_passed": 5, "verification": { "judged": 5, "verified": 3, "unverified": 1, "refuted": 1, "unreproducible": 0, "budget_min": 15 } },
168
192
  "evidence": { "screenshots": 9, "console_errors_found": 2, "network_failures_found": 1 },
169
193
  "areas_not_tested": ["settings — time ran out"],
170
194
  "blocked_by": null
@@ -173,7 +197,16 @@ Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
173
197
 
174
198
  The first eight keys are required; the rest are optional. No other top-level keys (the
175
199
  schema is strict). Mobile puts its device block under `coverage.mobile`; backend puts the
176
- verdict counts under `coverage.verdicts`.
200
+ verdict counts under `coverage.verdicts`; a session that ran the bug judge puts its counts
201
+ under `coverage.verification` (`bugs_found` and `severity_counts` count shipped bugs only —
202
+ never the refuted ones). Its keys: `judged` = candidate bugs (drafts), whether or not the judge
203
+ reached them; `verified` = CONFIRMED + CONFIRMED-ADJUSTED, including overruled refutations;
204
+ `unverified` = UNVERIFIED plus not judged; `refuted`; `unreproducible` (the five sum to
205
+ `judged`); `budget_min` = the minutes allotted, not spent. The report's `## Session Stats`
206
+ renders them as one row, `| Bugs judged | <judged> (verified <n>, unverified <n>, refuted <n>, unreproducible <n>) |`. Under
207
+ `verification.mode: triage-shadow` the triage counts go under `coverage.verification.triage.<provider>`
208
+ (`triaged`, `refute_risk`, `unclear`, `likely_confirmed`, `agreed_with_judge`, `unavailable`,
209
+ `input_tokens`, `cost_usd`, `ms_total`).
177
210
 
178
211
  ## Index rows
179
212
 
@@ -86,3 +86,20 @@ The first two lines of every session artefact (see `output-contract.md`):
86
86
 
87
87
  Session output stays on the local disk. Never send it to an external service, paste it
88
88
  into a ticket without the user asking, or include it in anything published.
89
+
90
+ **One gated exception: the hosted decision-model triage.** `qualiow judge triage` sends data
91
+ off the machine only when the target config sets `verification.mode: triage-shadow`, lists
92
+ `typesafe` as a provider, and the key is present under the env var the config names. Per
93
+ candidate bug it sends one claim card (title, URL path, claimed severity, environment,
94
+ reproduction rate, expected and actual behaviour, steps), the console and network excerpts
95
+ written on it, the text evidence files it lists from inside the session directory (capped per
96
+ file), and the known false-positive pattern lead-ins. It never sends screenshots, video, the
97
+ storage state, the session log, the charter, the drafts, other claims or any verdict. Before
98
+ anything leaves, every string passes the redaction list, then every URL is cut to its path,
99
+ every dotted hostname becomes `[HOST]`, every absolute path under a common root its basename and every `.auth/`
100
+ reference `[AUTH_STATE]`; the exact request is saved as `verification/JEV-NNN.json`. The
101
+ scrubbing is pattern-based and does not remove IP addresses, single-label hostnames, URL paths,
102
+ or free-text product, company or people names. The self-hosted `laya` provider is not an
103
+ exception: its endpoint must be loopback, checked before anything is sent, and redirects are
104
+ refused. Off by default —
105
+ `${CLAUDE_SKILL_DIR}/references/evidence-triage.md`.
@@ -57,7 +57,9 @@ it here.
57
57
  **markdown** (default): the full session report per `output-contract.md`
58
58
  (`# Session Report — <target>`, Session Metadata, Executive Summary, Summary Stats, Coverage
59
59
  Map with `| Area | Risk | Status | Bugs | Notes |`, Bugs Found, Observations, Areas Not
60
- Tested, Recommendations, Reflection, Session Stats; mobile and backend additions preserved).
60
+ Tested, Recommendations, Reflection, Session Stats; mobile and backend additions preserved;
61
+ for a session with a `verification/` directory, the `Verification` column in Bugs Found and
62
+ `## Refuted Findings` after Session Stats, as `qa-reporting-agent` defines them).
61
63
 
62
64
  **summary**: a brief executive summary: target, date, duration; bug count by severity; top 3
63
65
  findings; key recommendation.