qualiow-exploratory-testing 1.3.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +26 -0
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +259 -0
- package/README.md +472 -204
- package/agents/qa-diff-indexer-agent.md +81 -0
- package/agents/qa-gather-agent.md +41 -0
- package/agents/qa-page-mapper-agent.md +74 -0
- package/agents/qa-reporting-agent.md +103 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/bin/qualiow +30 -0
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/knowledge/changelog.yml +30 -12
- package/data/knowledge/manifest.yml +355 -127
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +3 -3
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +9 -6
- package/data/templates/expected-behaviour.md +3 -2
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +2286 -701
- package/dist/index.d.ts +637 -81
- package/dist/index.js +2058 -228
- package/docs/BACKEND-VERIFICATION.md +39 -9
- package/docs/GETTING-STARTED.md +300 -121
- package/docs/MOBILE-SETUP.md +149 -18
- package/hooks/hooks.json +27 -0
- package/hooks/scripts/read-guard.mjs +95 -0
- package/hooks/scripts/secret-patterns.d.mts +8 -0
- package/hooks/scripts/secret-patterns.mjs +136 -0
- package/hooks/scripts/write-guard.mjs +87 -0
- package/package.json +33 -18
- package/skills/qa-explore/SKILL.md +32 -25
- package/skills/qa-explore/phases/00-setup.md +76 -42
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +93 -27
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +102 -142
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/delegation-rules.md +82 -0
- package/skills/qa-explore/references/output-contract.md +192 -0
- package/skills/qa-explore/references/paths.md +108 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +70 -25
- package/skills/qa-explore/references/session-rules.md +2 -1
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +51 -15
- package/skills/qa-explore-feedback/SKILL.md +23 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +108 -60
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +71 -26
- package/skills/qa-explore-report/SKILL.md +65 -35
- package/skills/qa-gather/SKILL.md +44 -8
- package/skills/qa-knowledge-add/SKILL.md +25 -9
- package/skills/qa-knowledge-list/SKILL.md +37 -40
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +31 -19
- package/skills/qa-verify-backend/phases/00-setup.md +57 -12
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +2 -2
- package/skills/qa-verify-backend/phases/02-static-review.md +11 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +5 -3
- package/skills/qa-verify-backend/phases/03b-api-verification.md +24 -9
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +18 -2
- package/skills/qa-verify-backend/phases/05-reporting.md +72 -12
- package/skills/qa-verify-backend/references/api-probes.md +28 -1
- package/skills/qa-verify-backend/references/payload-verification.md +1 -1
- package/skills/qa-verify-backend/references/release-readiness.md +1 -1
- package/skills/qa-verify-backend/references/safety-rules.md +13 -8
- package/data/domains/_default.md +0 -181
- package/data/domains/ecommerce.md +0 -192
- package/data/domains/fintech.md +0 -200
- package/data/domains/identity.md +0 -88
- package/data/domains/marketing.md +0 -194
- package/data/domains/saas.md +0 -197
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -4,60 +4,72 @@ description: >
|
|
|
4
4
|
Interactive setup for a new target application. Navigates to the URL, detects auth requirements,
|
|
5
5
|
walks through login flow creation, auto-detects scope, and saves target config.
|
|
6
6
|
Use when user says: "setup target", "add target", "configure app", "new target".
|
|
7
|
-
|
|
7
|
+
argument-hint: "<url> [--name <id>] [--shared]"
|
|
8
|
+
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(npx playwright:*), Read, Write, Glob, Grep
|
|
8
9
|
---
|
|
9
10
|
|
|
10
11
|
# Target Application Setup
|
|
11
12
|
|
|
12
|
-
Interactive wizard
|
|
13
|
+
Interactive wizard that writes a target config. By default it writes the **project-local**
|
|
14
|
+
`qa/target.yml` (the file every session skill reads when no `--target` is given; see
|
|
15
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`). With `--shared`, it writes
|
|
16
|
+
`data/targets/<name>.yml` instead, for a target the whole team shares by name. Keep internal
|
|
17
|
+
hostnames in `qa/target.yml` or in `data/targets/local-*.yml` (both gitignored patterns).
|
|
18
|
+
|
|
19
|
+
Security: `${CLAUDE_SKILL_DIR}/../qa-explore/references/security-rules.md` applies (the
|
|
20
|
+
production rule decides the `safety:` block; never write a credential value into YAML).
|
|
21
|
+
The fields written here are what the session skills read to fill `**Environment:**` and the
|
|
22
|
+
session metadata in `${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md`, so
|
|
23
|
+
`id`, `environment.kind` and `base_url` must be the real ones.
|
|
13
24
|
|
|
14
25
|
## Flow
|
|
15
26
|
|
|
16
27
|
### 1. Gather Basic Info
|
|
17
28
|
|
|
18
|
-
Ask user for:
|
|
19
|
-
- **Target name** (slug
|
|
20
|
-
- **Base URL** (e.g
|
|
21
|
-
- **Domain type** (ecommerce
|
|
29
|
+
Ask the user for:
|
|
30
|
+
- **Target name** (slug, e.g. `company-staging`; becomes `id:` and, with `--shared`, the filename)
|
|
31
|
+
- **Base URL** (e.g. `https://staging.company.com`)
|
|
32
|
+
- **Domain type** (`ecommerce`, `saas`, `fintech`, `marketing`, `identity`, or auto-detect against `<data>/domains/*.yml`)
|
|
33
|
+
|
|
34
|
+
Apply the production rule to the hostname now and tell the user what it decided.
|
|
22
35
|
|
|
23
36
|
### 2. Navigate and Analyze
|
|
24
37
|
|
|
38
|
+
Use the session id `-s=setup-<HHmm>-<name>` on every command (omitted below):
|
|
39
|
+
|
|
25
40
|
```bash
|
|
26
41
|
playwright-cli open <base_url>
|
|
27
42
|
playwright-cli snapshot
|
|
28
43
|
```
|
|
29
44
|
|
|
30
45
|
From the snapshot, detect:
|
|
31
|
-
- Is this a login page? (
|
|
46
|
+
- Is this a login page? (password fields, "Sign in" buttons)
|
|
32
47
|
- What's the main navigation structure?
|
|
33
|
-
- What kind of application is this? (helps auto-detect domain)
|
|
48
|
+
- What kind of application is this? (helps auto-detect the domain)
|
|
34
49
|
|
|
35
50
|
### 3. Handle Authentication
|
|
36
51
|
|
|
37
|
-
**If no login detected:**
|
|
38
|
-
- Set `auth.strategy: none`
|
|
39
|
-
- Skip to step 4
|
|
40
|
-
|
|
41
|
-
**If login detected:**
|
|
52
|
+
**If no login detected:** set `auth.strategy: none` and skip to step 4.
|
|
42
53
|
|
|
43
|
-
|
|
54
|
+
**If login detected**, ask which strategy to use:
|
|
44
55
|
|
|
45
56
|
**Option A: Storage State (recommended)**
|
|
46
|
-
1. Ask user
|
|
47
|
-
2.
|
|
57
|
+
1. Ask the user for the env var **names** for username and password (values go in `qa/.env`).
|
|
58
|
+
2. Log in via playwright-cli, reading the values from `qa/.env` (never echo them):
|
|
48
59
|
```bash
|
|
49
|
-
playwright-cli snapshot #
|
|
50
|
-
playwright-cli fill <email_ref> "
|
|
51
|
-
playwright-cli fill <password_ref> "
|
|
60
|
+
playwright-cli snapshot # find form fields
|
|
61
|
+
playwright-cli fill <email_ref> "<username value>"
|
|
62
|
+
playwright-cli fill <password_ref> "<password value>"
|
|
52
63
|
playwright-cli click <submit_ref>
|
|
53
64
|
```
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
65
|
+
If the identity provider needs MFA, reopen headed (`playwright-cli open <login_url> --headed`) and let the user complete it by hand; never automate an MFA challenge.
|
|
66
|
+
3. Verify login succeeded (snapshot shows dashboard/profile).
|
|
67
|
+
4. Save state: `playwright-cli state-save .auth/<name>.json`
|
|
68
|
+
5. Set `auth.strategy: storage_state` with `state_file: ".auth/<name>.json"`, plus `auth.credentials.username` / `auth.credentials.password` naming the env vars so a session can re-login when the state expires.
|
|
57
69
|
|
|
58
70
|
**Option B: Token**
|
|
59
|
-
1. Ask for token env var name and injection method (cookie
|
|
60
|
-
2. Set `auth.strategy: token`
|
|
71
|
+
1. Ask for the token env var name and the injection method (cookie or localStorage).
|
|
72
|
+
2. Set `auth.strategy: token`, `auth.token: <ENV_VAR_NAME>`.
|
|
61
73
|
|
|
62
74
|
### 4. Detect Scope
|
|
63
75
|
|
|
@@ -67,52 +79,62 @@ playwright-cli snapshot
|
|
|
67
79
|
```
|
|
68
80
|
|
|
69
81
|
From the snapshot, identify:
|
|
70
|
-
- Top-level navigation sections → suggest as `start_pages`
|
|
71
|
-
- Suggest `include_patterns` based on URL structure
|
|
72
|
-
- Suggest `exclude_patterns` for common exclusions (
|
|
82
|
+
- Top-level navigation sections → suggest as `scope.start_pages`
|
|
83
|
+
- Suggest `scope.include_patterns` based on the URL structure
|
|
84
|
+
- Suggest `scope.exclude_patterns` for common exclusions (`/admin`, `/api`, `*.pdf`)
|
|
85
|
+
- `scope.max_depth` (default 3)
|
|
73
86
|
|
|
74
|
-
Ask user to confirm/modify scope.
|
|
87
|
+
Ask the user to confirm/modify scope.
|
|
75
88
|
|
|
76
89
|
### 5. Configure Browser Settings
|
|
77
90
|
|
|
78
91
|
Suggest defaults:
|
|
79
|
-
- `headless: false` (so user can watch)
|
|
92
|
+
- `headless: false` (so the user can watch)
|
|
80
93
|
- `viewport: { width: 1280, height: 720 }`
|
|
81
|
-
- Ask
|
|
94
|
+
- Ask whether mobile emulation is needed (`browser.device`, e.g. `iPhone 15`, used with `playwright-cli open --device="iPhone 15"`, or `--mobile` for a generic mobile device — the two flags are mutually exclusive, and `--mobile` is unsupported on Firefox)
|
|
82
95
|
|
|
83
96
|
### 6. Safety Settings
|
|
84
97
|
|
|
85
|
-
|
|
86
|
-
-
|
|
87
|
-
- If no: no safety restrictions
|
|
98
|
+
- If the production rule fired, or the user says this is production: `safety.read_only: true`, `safety.no_form_submit: true`.
|
|
99
|
+
- Otherwise: `safety.read_only: false`, `safety.no_form_submit: false`.
|
|
88
100
|
|
|
89
101
|
### 7. Save Config
|
|
90
102
|
|
|
91
|
-
Write `data/targets/<
|
|
103
|
+
Write `qa/target.yml` (default) or `data/targets/<name>.yml` (`--shared`) with all gathered
|
|
104
|
+
configuration; `id` equals the target name. Then close the session:
|
|
92
105
|
|
|
93
|
-
Close browser:
|
|
94
106
|
```bash
|
|
95
107
|
playwright-cli close
|
|
108
|
+
playwright-cli delete-data
|
|
96
109
|
```
|
|
97
110
|
|
|
98
|
-
Update
|
|
111
|
+
Update `qa/.env.example` (create it if missing) with every env var name the config references,
|
|
112
|
+
each as a commented `# NAME=` line. Never write values.
|
|
99
113
|
|
|
100
114
|
### 8. Confirm
|
|
101
115
|
|
|
102
|
-
Present the saved config to user:
|
|
116
|
+
Present the saved config to the user:
|
|
103
117
|
- Target name and URL
|
|
104
118
|
- Auth strategy configured
|
|
105
119
|
- Scope defined
|
|
106
|
-
- How to use: `/qa-explore --target <
|
|
120
|
+
- How to use: `/qa-explore` (picks up `qa/target.yml`) or `/qa-explore --target <name>` for a shared target
|
|
107
121
|
|
|
108
122
|
### 9. Optional — Offer Playwright Test Agents Bootstrap
|
|
109
123
|
|
|
110
|
-
If the user plans to author full Playwright tests against this target later (not just
|
|
124
|
+
If the user plans to author full Playwright tests against this target later (not just
|
|
125
|
+
exploratory sessions), offer to scaffold the Test Agents workspace now while credentials and
|
|
126
|
+
scope are already loaded:
|
|
111
127
|
|
|
112
128
|
```bash
|
|
113
129
|
npx playwright init-agents --loop=claude
|
|
114
130
|
```
|
|
115
131
|
|
|
116
|
-
|
|
132
|
+
With `--loop=claude` this writes the planner / generator / healer definitions into
|
|
133
|
+
`.claude/agents/`, a `.mcp.json` registering the `playwright-test` server, and a `specs/`
|
|
134
|
+
scaffold with a seed spec if the project has none (Playwright 1.56+; other loops: `codex`,
|
|
135
|
+
`copilot`, `opencode`, `vscode`, `vscode-legacy`). It is **strictly optional**; declining does not affect
|
|
136
|
+
the target config just written. If the user accepts, note in the config's `notes` field that
|
|
137
|
+
Test Agents are scaffolded.
|
|
117
138
|
|
|
118
|
-
This step requires `@playwright/test`
|
|
139
|
+
This step requires `@playwright/test` and a `playwright.config.*` in the target project. If
|
|
140
|
+
either is missing, skip it silently.
|
|
@@ -10,7 +10,8 @@ description: >
|
|
|
10
10
|
"review the infra change", "does this meet the acceptance criteria", "test the backend",
|
|
11
11
|
"test the API", "check the endpoint", or gives a backend/API/infra ticket whose ACs
|
|
12
12
|
cannot be seen in a browser.
|
|
13
|
-
|
|
13
|
+
argument-hint: "[--target <id>] [--context <file>] [--static-only] [--api-only] [--no-e2e] [--parity <target-id>]"
|
|
14
|
+
allowed-tools: Read, Write, Glob, Grep, Bash(git:*), Bash(aws:*), Bash(jq:*), Bash(grep:*), Bash(curl:*), Bash(node:*), Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(qualiow:*), Bash(npx:*), Bash(terraform validate:*), Bash(terraform fmt:*)
|
|
14
15
|
---
|
|
15
16
|
|
|
16
17
|
# Backend & Infrastructure AC Verification
|
|
@@ -70,28 +71,34 @@ Execute in order. Read and follow the linked file.
|
|
|
70
71
|
|
|
71
72
|
| Phase | File | Summary |
|
|
72
73
|
|-------|------|---------|
|
|
73
|
-
| **Setup** |
|
|
74
|
-
| **AC Decomposition** |
|
|
75
|
-
| **Static Review** |
|
|
76
|
-
| **Live Verification** |
|
|
77
|
-
| **API Verification** |
|
|
78
|
-
| **End-to-End Trigger** |
|
|
79
|
-
| **Reporting** |
|
|
74
|
+
| **Setup** | `${CLAUDE_SKILL_DIR}/phases/00-setup.md` | Resolve target, ticket context, repo branch, AWS access. Decide which lanes are runnable. |
|
|
75
|
+
| **AC Decomposition** | `${CLAUDE_SKILL_DIR}/phases/01-ac-decomposition.md` | Turn each AC into a falsifiable check with a named evidence source. Flag untestable ACs. |
|
|
76
|
+
| **Static Review** | `${CLAUDE_SKILL_DIR}/phases/02-static-review.md` | Read the branch diff against every AC. Producer/consumer contract. Spec drift. Negative space. |
|
|
77
|
+
| **Live Verification** | `${CLAUDE_SKILL_DIR}/phases/03-live-verification.md` | Read-only aws-cli probes of the cloud resources. One probe per AC. Capture raw evidence. |
|
|
78
|
+
| **API Verification** | `${CLAUDE_SKILL_DIR}/phases/03b-api-verification.md` | Fingerprint the environment, then call the endpoints directly from the authenticated session. Case matrix, cross-environment parity, UI-vs-API differential. |
|
|
79
|
+
| **End-to-End Trigger** | `${CLAUDE_SKILL_DIR}/phases/04-e2e-trigger.md` | Drive the real write path (UI or API), then re-probe the data layer. Covers "verified in env" ACs. |
|
|
80
|
+
| **Reporting** | `${CLAUDE_SKILL_DIR}/phases/05-reporting.md` | AC traceability matrix, bug reports with business impact, DoD gaps. |
|
|
80
81
|
|
|
81
82
|
## References
|
|
82
83
|
|
|
83
|
-
-
|
|
84
|
-
-
|
|
85
|
-
-
|
|
86
|
-
-
|
|
87
|
-
-
|
|
88
|
-
-
|
|
84
|
+
- **`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`** — target resolution (`--target` → `data/targets/<id>.yml`; none → `qa/target.yml`), data directory, credentials, session-directory scheme, index rows
|
|
85
|
+
- **`${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md`** — bug-report, session-report and `stats.json` format, confidentiality header
|
|
86
|
+
- **`${CLAUDE_SKILL_DIR}/../qa-explore/references/security-rules.md`** — the production rule with its exclude list, the redaction list, output classification; `references/safety-rules.md` below adds the backend-specific rules on top
|
|
87
|
+
- **`${CLAUDE_SKILL_DIR}/references/aws-readonly-probes.md`** — canonical probe commands per resource type, and how to read their output
|
|
88
|
+
- **`${CLAUDE_SKILL_DIR}/references/api-probes.md`** — getting an authenticated request context without handling a token, the case families worth probing on every endpoint, and how to read the results
|
|
89
|
+
- **`${CLAUDE_SKILL_DIR}/references/environment-fingerprinting.md`** — proving which build and which implementation an environment actually runs, before any verdict is written
|
|
90
|
+
- **`${CLAUDE_SKILL_DIR}/references/payload-verification.md`** — recomputing derived values, saying what a single-source check does not prove, and comparing the rendered value against the payload it came from
|
|
91
|
+
- **`${CLAUDE_SKILL_DIR}/references/release-readiness.md`** — verifying many tickets against one build: the deployment table, the result vocabulary, carry-overs, disposition
|
|
92
|
+
- **`${CLAUDE_SKILL_DIR}/references/safety-rules.md`** — read-only discipline, account confirmation, destructive runbooks, session credentials, repo hygiene, untrusted content
|
|
89
93
|
|
|
90
94
|
## Knowledge Base
|
|
91
95
|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
96
|
+
Before the static and live lanes, run `qualiow kb digest --for backend` (resolve the prefix
|
|
97
|
+
per `${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`). It prints a summary of each
|
|
98
|
+
entry below; fetch one in full with `qualiow list knowledge --entry <id>` when you need its
|
|
99
|
+
steps, and never `Read` the manifest or a release entry whole. The table is the reference
|
|
100
|
+
list of what the digest loads — the BE/API verification layer, techniques derived from real
|
|
101
|
+
sessions with this skill, not from published literature.
|
|
95
102
|
|
|
96
103
|
| Entry | Use it when |
|
|
97
104
|
|-------|-------------|
|
|
@@ -109,6 +116,8 @@ not from published literature.
|
|
|
109
116
|
| `technique-expected-behaviour-specification` | The session found behaviour no AC covers — which is most of what an API probe turns up. |
|
|
110
117
|
| `technique-config-surface-verification` | The change under test *is* the configuration mechanism — a flag endpoint, a settings surface, a shared config library. |
|
|
111
118
|
| `technique-release-readiness-verification` | The ask is "is this release good to go", not "does this ticket meet its ACs". |
|
|
119
|
+
| `technique-exactly-once-verification` | Any money movement or ledger write: assert the balance delta, not the status field; attack identity collision and simultaneous redelivery. |
|
|
120
|
+
| `technique-async-callback-contracts` | The change consumes callbacks or webhooks: what the endpoint does with an event it cannot match, a duplicate, or a late one. |
|
|
112
121
|
|
|
113
122
|
The mode selection entry governs the others: it decides *how* an AC gets observed,
|
|
114
123
|
and the rest supply *what to look for* once the channel is chosen.
|
|
@@ -129,7 +138,7 @@ and the rest supply *what to look for* once the channel is chosen.
|
|
|
129
138
|
size limits. Audit and event systems fail silently by default.
|
|
130
139
|
6. **Absence is a finding.** An AC that nothing implements, a cleanup that was written
|
|
131
140
|
but never run, a doc that still describes the deleted architecture.
|
|
132
|
-
7. **One bug = one report**, each with Business Impact,
|
|
141
|
+
7. **One bug = one report**, each with Business Impact, in the format of `${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md` (`bugs/BUG-NNN.md`).
|
|
133
142
|
8. **Name the observation mode in every verdict.** `PASS (direct request, raw
|
|
134
143
|
response attached)` and `PASS (read the diff)` are different claims. A verdict
|
|
135
144
|
whose only evidence is a code reading is `UNVERIFIABLE`, not `PASS` — see
|
|
@@ -160,4 +169,7 @@ and the rest supply *what to look for* once the channel is chosen.
|
|
|
160
169
|
shared table, no destructive cleanup — not even when a runbook in the repo says to.
|
|
161
170
|
Propose it; let a human run it.
|
|
162
171
|
17. **Redact.** Account IDs beyond what the target config already holds, tokens, emails
|
|
163
|
-
of real people, and consumer PII get `[REDACTED]` before anything is written to disk
|
|
172
|
+
of real people, and consumer PII get `[REDACTED]` before anything is written to disk
|
|
173
|
+
(the full list is in `security-rules.md`).
|
|
174
|
+
18. **Session directory and header.** `output/sessions/<YYYY-MM-DD-HHmm>-backend-<ticket>/`,
|
|
175
|
+
and the confidentiality header is the first two lines of every artefact.
|
|
@@ -3,15 +3,18 @@
|
|
|
3
3
|
## Step 1: Resolve Inputs
|
|
4
4
|
|
|
5
5
|
Extract from the user message:
|
|
6
|
-
- **target**: id
|
|
7
|
-
|
|
6
|
+
- **target**: `--target <id>` → `data/targets/<id>.yml` (with a `backend:` and/or `api:`
|
|
7
|
+
block); no `--target` → the project-local `qa/target.yml` (resolution order in
|
|
8
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`) — or inline resource names
|
|
8
9
|
- **context**: path to a gathered ticket context file, or inline ticket text
|
|
9
10
|
- **flags**: `--static-only` (skip live probes), `--api-only` (skip the cloud lane),
|
|
10
11
|
`--no-e2e` (skip phase 4), `--parity <target-id>` (run the API matrix in a second
|
|
11
12
|
environment too)
|
|
12
13
|
|
|
13
|
-
Read the target config. Read
|
|
14
|
-
|
|
14
|
+
Read the target config. Read `<data>/domains/<domain>.yml` for the domain's
|
|
15
|
+
`data_integrity_checks` — they become extra checks beyond the ticket's own ACs. Apply the
|
|
16
|
+
production rule from `security-rules.md` to the account id, profile name, resource names and
|
|
17
|
+
`api.base_url` now; `environment.kind: production` also triggers it.
|
|
15
18
|
|
|
16
19
|
## Step 2: Resolve the Source Branch
|
|
17
20
|
|
|
@@ -39,28 +42,64 @@ Compare the returned account to `backend.account_id`.
|
|
|
39
42
|
| Matches | Live lane is GO |
|
|
40
43
|
| Different account | **STOP the live lane.** Report it. Do not probe an account the ticket does not concern. |
|
|
41
44
|
| No credentials | Live lane is **BLOCKED**, not failed. Continue with static + write the probe commands into the report so a human can run them. |
|
|
42
|
-
| Account looks like production | Read
|
|
45
|
+
| Account looks like production | Read `${CLAUDE_SKILL_DIR}/references/safety-rules.md` and apply the production hard stop. |
|
|
43
46
|
|
|
44
47
|
## Step 3b: Establish the API Lane (skip if the ticket has no HTTP surface)
|
|
45
48
|
|
|
46
|
-
From `target.api`, confirm the endpoints under test
|
|
47
|
-
|
|
49
|
+
From `target.api`, confirm the endpoints under test, the `probe_allowlist` (read-only,
|
|
50
|
+
callable now) and the `write_allowlist` (state-changing, phase 4 only). How you get a
|
|
51
|
+
request context depends on `api.auth`:
|
|
52
|
+
|
|
53
|
+
### `session-cookie` — the service sits behind a UI
|
|
54
|
+
|
|
55
|
+
Open an authenticated session once — headed, against a persistent profile, because an
|
|
48
56
|
interactive identity provider cannot complete headless:
|
|
49
57
|
|
|
50
58
|
```bash
|
|
51
|
-
playwright-cli -s
|
|
59
|
+
playwright-cli -s=backend-<HHmm>-<ticket> open <api.base_url or base_url> --headed --profile <api.browser_profile>
|
|
52
60
|
```
|
|
53
61
|
|
|
62
|
+
`api.browser_profile` (e.g. `.auth/my-service-dev-profile`) is the persistent profile
|
|
63
|
+
directory the target declares; it holds a live session for a real account and is gitignored.
|
|
64
|
+
|
|
65
|
+
The session id follows the one scheme in
|
|
66
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md` — `-s=<kind>-<HHmm>-<slug>`, here
|
|
67
|
+
`-s=backend-<HHmm>-<ticket>`. Every later `playwright-cli` call in this session carries the
|
|
68
|
+
same `-s=` id, and the session ends in phase 5 with `close` then `delete-data` (the profile
|
|
69
|
+
directory itself is kept).
|
|
70
|
+
|
|
54
71
|
Confirm the session is live before building any matrix — a probe run against a logged-out
|
|
55
72
|
page returns a login page with a `200`, which reads exactly like a passing case.
|
|
56
73
|
|
|
57
|
-
|
|
74
|
+
### `api-key-env` / `bearer-env` — the service has no UI
|
|
75
|
+
|
|
76
|
+
**Open no browser.** There is no session to inherit and no profile to persist. The request
|
|
77
|
+
context is an out-of-browser runner under `probes/` that reads the credential from the env
|
|
78
|
+
var named by `api.token_env` and sends it in `api.header_name` (default `X-Api-Key`), or as
|
|
79
|
+
`Authorization: Bearer` for `bearer-env`. The value is read from the environment at run
|
|
80
|
+
time and never written into a script, a report or this repo.
|
|
81
|
+
|
|
82
|
+
Prove the credential before the first probe, and record both calls in
|
|
83
|
+
`evidence/fingerprint.md`:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
curl -s -o /dev/null -w '%{http_code}\n' "$BASE/<a read endpoint>" # expect 401/403
|
|
87
|
+
curl -s -o /dev/null -w '%{http_code}\n' -H "X-Api-Key: $KEY" "$BASE/<same endpoint>" # expect 200
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
A service that answers the same either way is not authenticating, which is a finding
|
|
91
|
+
before it is a setup step. If the key is missing the lane is **BLOCKED**, not failed —
|
|
92
|
+
write the commands into the report and continue.
|
|
93
|
+
|
|
94
|
+
Charter line for this shape: `api: GO (out-of-browser, api-key)`.
|
|
95
|
+
|
|
96
|
+
If the target declares `api.parity_targets` or `--parity` was passed, prime that context
|
|
58
97
|
too. Same matrix, both environments, or the comparison is not one.
|
|
59
98
|
|
|
60
99
|
## Step 3c: Fingerprint the Environment
|
|
61
100
|
|
|
62
101
|
**Before the first probe of any lane.** Read
|
|
63
|
-
|
|
102
|
+
`${CLAUDE_SKILL_DIR}/references/environment-fingerprinting.md` and answer its three questions: which build is
|
|
64
103
|
deployed in every component of the path, whether the changed code path is *selected*
|
|
65
104
|
here, and whether the commit under test is genuinely an ancestor of what is running.
|
|
66
105
|
|
|
@@ -73,8 +112,11 @@ needed a probe becomes `BLOCKED` with the exact command attached.
|
|
|
73
112
|
|
|
74
113
|
## Step 4: Create the Session Directory
|
|
75
114
|
|
|
76
|
-
`output/sessions/<YYYY-MM-DD-HHmm
|
|
77
|
-
|
|
115
|
+
`output/sessions/<YYYY-MM-DD-HHmm>-backend-<ticket>/` (scheme in `paths.md`; `<ticket>` is the
|
|
116
|
+
ticket id, else the target id) containing:
|
|
117
|
+
`charter.md`, `ac-matrix.md`, `evidence/`, `bugs/`, `probes/`, `session-log.md`, `progress.json`
|
|
118
|
+
(`"kind": "backend"`). Every markdown file starts with the confidentiality header from
|
|
119
|
+
`output-contract.md`.
|
|
78
120
|
|
|
79
121
|
`probes/` holds the probe scripts and case files, so the run is repeatable by someone
|
|
80
122
|
else — and by you, against the next environment.
|
|
@@ -82,6 +124,9 @@ else — and by you, against the next environment.
|
|
|
82
124
|
## Step 5: Write the Charter
|
|
83
125
|
|
|
84
126
|
```markdown
|
|
127
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
128
|
+
> and application details. Do not share outside your organization without review.
|
|
129
|
+
|
|
85
130
|
# Charter — <TICKET>
|
|
86
131
|
**Mission:** Verify <n> acceptance criteria for <ticket summary>
|
|
87
132
|
**Target:** <target id> (<environment kind>)
|
|
@@ -43,7 +43,7 @@ An AC is untestable when it has no observable falsifier. Common shapes:
|
|
|
43
43
|
Every claim is a claim about an environment. Write it in: *"in staging, an empty query
|
|
44
44
|
returns 400"*. An AC verified in one environment and asserted for another is the single
|
|
45
45
|
most common false `PASS` this skill exists to prevent — see
|
|
46
|
-
|
|
46
|
+
`${CLAUDE_SKILL_DIR}/references/environment-fingerprinting.md`.
|
|
47
47
|
|
|
48
48
|
Report these as **DoR gaps**, not as failures of the developer. They are ticket-writing
|
|
49
49
|
defects and they belong in the report — an untestable AC will be marked done by whoever
|
|
@@ -51,7 +51,7 @@ is under the least time pressure.
|
|
|
51
51
|
|
|
52
52
|
## Add Domain Checks
|
|
53
53
|
|
|
54
|
-
From
|
|
54
|
+
From `<data>/domains/<domain>.yml`, pull the `data_integrity_checks` and add any that this
|
|
55
55
|
change touches but the ticket never mentions. These become findings in the
|
|
56
56
|
**negative space** — the things the ACs forgot to require.
|
|
57
57
|
|
|
@@ -13,6 +13,17 @@ Group changed files by AC. Any file group that maps to **no** AC is either scope
|
|
|
13
13
|
or an undocumented dependency — both worth a line in the report. Any AC that maps to
|
|
14
14
|
**no** file is a candidate `FAIL` before you have read a single line.
|
|
15
15
|
|
|
16
|
+
**Gate on the size of the change.** 25 or more files, or 1,500 or more changed lines, in
|
|
17
|
+
the `--stat` output: invoke the `qa-diff-indexer-agent` sub-agent
|
|
18
|
+
(`qualiow:qa-diff-indexer-agent` under a plugin install) with the repo path, the base, the
|
|
19
|
+
branch and the AC list from phase 1. It returns an index — file, symbols or resources
|
|
20
|
+
touched, line ranges, candidate AC ids, note — plus the two lists that matter here: files
|
|
21
|
+
matching no AC, and ACs matching no file. It returns no verdicts and no severities; the
|
|
22
|
+
grouping above is still yours. Then read **only the ranges it names**, with
|
|
23
|
+
`git show <branch>:<path>`. Never diff the whole branch into context. If sub-agents are
|
|
24
|
+
unavailable, do the step yourself as in 2.1.0: read the diff by `--stat` first, then per
|
|
25
|
+
file.
|
|
26
|
+
|
|
16
27
|
## Step 2: Read the Implementation Against Each Claim
|
|
17
28
|
|
|
18
29
|
Work AC by AC. For each, open the actual resource or handler and compare it to the
|
|
@@ -3,13 +3,15 @@
|
|
|
3
3
|
Skip if the live lane is BLOCKED — but still write every probe command into
|
|
4
4
|
`evidence/live-probes.md` so a human with credentials can run them unchanged.
|
|
5
5
|
|
|
6
|
-
|
|
7
|
-
|
|
6
|
+
`${CLAUDE_SKILL_DIR}/references/aws-readonly-probes.md` is the command catalogue: Grep it for
|
|
7
|
+
the heading of the resource you need (`Grep '^## ' <file>` lists them) and Read only that
|
|
8
|
+
section with `offset`/`limit`; never the whole file. Read
|
|
9
|
+
`${CLAUDE_SKILL_DIR}/references/safety-rules.md` before the first probe, and confirm the fingerprint from
|
|
8
10
|
phase 0 step 3c is on file — a probe of an environment that does not run the change
|
|
9
11
|
measures something else.
|
|
10
12
|
|
|
11
13
|
This phase covers the **cloud resources**. The service's own HTTP endpoints are phase 3b
|
|
12
|
-
(
|
|
14
|
+
(`${CLAUDE_SKILL_DIR}/phases/03b-api-verification.md`); most tickets need both.
|
|
13
15
|
|
|
14
16
|
## Rules
|
|
15
17
|
|
|
@@ -8,9 +8,11 @@ Run it whenever the ticket touches a request/response contract: a search or list
|
|
|
8
8
|
endpoint, validation rules, pagination, filter values, an error contract, a payload
|
|
9
9
|
shape a consumer reads. Skip it when the change has no HTTP surface at all.
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
11
|
+
`${CLAUDE_SKILL_DIR}/references/api-probes.md` holds the mechanics and the case catalogue:
|
|
12
|
+
Grep it for the heading of the section you need (`Grep '^## ' <file>` lists them) and Read
|
|
13
|
+
only that section with `offset`/`limit`; never the whole file. Read
|
|
14
|
+
`${CLAUDE_SKILL_DIR}/references/environment-fingerprinting.md` before you believe any result, and
|
|
15
|
+
`${CLAUDE_SKILL_DIR}/references/payload-verification.md` once you have one — a well-shaped `200` says nothing
|
|
14
16
|
about whether the numbers in it are right.
|
|
15
17
|
|
|
16
18
|
## Step 1: Prove Which Implementation You Are Probing
|
|
@@ -45,11 +47,17 @@ playwright-cli -s=<session> eval "$(cat probes/<probe>.js)" --raw
|
|
|
45
47
|
cannot complete headless. `--profile` is what makes it stick — a persistent profile
|
|
46
48
|
directory outlives a `storage_state` JSON by a wide margin.
|
|
47
49
|
|
|
48
|
-
Fall back to an out-of-browser runner (a script driving `curl`)
|
|
49
|
-
|
|
50
|
+
Fall back to an out-of-browser runner (a script driving `curl`) when you need volume or a
|
|
51
|
+
machine-readable matrix. Take the credential from the authenticated profile
|
|
50
52
|
(`playwright-cli cookie-list`), keep it in a file outside the repo, and reference it by
|
|
51
53
|
path — never inline it into a script, a report, or this repo.
|
|
52
54
|
|
|
55
|
+
**When the target has no UI** (`api.auth: api-key-env` or `bearer-env`), the out-of-browser
|
|
56
|
+
runner is not a fallback, it is the only mode: there is no page to run inside. Phase 0 step
|
|
57
|
+
3b has already proved the credential works and that the endpoint rejects a request without
|
|
58
|
+
it. Read the key from the env var at run time; the script contains the variable name, never
|
|
59
|
+
the value.
|
|
60
|
+
|
|
53
61
|
Record which mode produced each result. An in-page probe and a bare `curl` do not prove
|
|
54
62
|
the same thing: the first proves the endpoint behaves this way *for a real logged-in
|
|
55
63
|
session*, the second proves it behaves this way *for whoever holds that credential*.
|
|
@@ -58,7 +66,7 @@ session*, the second proves it behaves this way *for whoever holds that credenti
|
|
|
58
66
|
|
|
59
67
|
Write the cases down first, as data, so the run is repeatable and the evidence is a
|
|
60
68
|
table rather than a scroll of output. Derive them from the ACs, then add the standard
|
|
61
|
-
families from
|
|
69
|
+
families from `${CLAUDE_SKILL_DIR}/references/api-probes.md` — the ACs are a floor:
|
|
62
70
|
|
|
63
71
|
boundary lengths · tokenisation and whitespace · metacharacters of whatever query
|
|
64
72
|
language sits underneath · empty, null, absent and whitespace-only · enum values that
|
|
@@ -112,10 +120,16 @@ and add the structural invariants the values must satisfy regardless of magnitud
|
|
|
112
120
|
|
|
113
121
|
Then write down what the check does not prove: when both sides come from one payload you
|
|
114
122
|
have verified the derivation, not the inputs. Name the independent oracle that would close
|
|
115
|
-
it, and say whether you ran it. Full guidance in
|
|
123
|
+
it, and say whether you ran it. Full guidance in `${CLAUDE_SKILL_DIR}/references/payload-verification.md`.
|
|
116
124
|
|
|
117
125
|
## Step 6: The Differential Pass — the Same Matrix at Both Surfaces
|
|
118
126
|
|
|
127
|
+
**Skip this step when the service has no user interface, and say so in the report** — with
|
|
128
|
+
no second surface there is no differential, and every finding is by definition "API only".
|
|
129
|
+
That is not a reason to downgrade one: an endpoint with no browser guard in front of it is
|
|
130
|
+
reachable exactly as probed, by every client there will ever be. Note instead which findings
|
|
131
|
+
a future client could mask, so the eventual UI is not credited with fixing them.
|
|
132
|
+
|
|
119
133
|
If the ticket also has a UI, run the overlapping cases through the screen too, and sort
|
|
120
134
|
every finding into four buckets:
|
|
121
135
|
|
|
@@ -146,7 +160,8 @@ report:
|
|
|
146
160
|
The API lane calls **read** endpoints only. Reads that the ticket's own ACs describe,
|
|
147
161
|
plus the standard families above, against endpoints the target's `api.probe_allowlist`
|
|
148
162
|
declares. Anything that creates, mutates or deletes belongs to phase 4, goes through the
|
|
149
|
-
real write path, and never runs against production.
|
|
163
|
+
real write path, is limited to `api.write_allowlist`, and never runs against production.
|
|
164
|
+
See `${CLAUDE_SKILL_DIR}/references/safety-rules.md`.
|
|
150
165
|
|
|
151
166
|
Response bodies are real records. Redact before they touch the disk.
|
|
152
167
|
|
|
@@ -154,4 +169,4 @@ Response bodies are real records. Redact before they touch the disk.
|
|
|
154
169
|
|
|
155
170
|
Update `ac-matrix.md` as you go, one row per AC, each verdict naming the environment it
|
|
156
171
|
holds in. Write the case table to `evidence/api-probe-matrix.md` using
|
|
157
|
-
|
|
172
|
+
`<data>/templates/api-probe-matrix.md`.
|
|
@@ -3,7 +3,9 @@
|
|
|
3
3
|
Covers the "verified in environment" style of AC: a real change through the real write
|
|
4
4
|
path, then re-probe the data layer.
|
|
5
5
|
|
|
6
|
-
Skip if `--no-e2e`, if the environment
|
|
6
|
+
Skip if `--no-e2e`, if the target declares `environment.kind: production` (or the production
|
|
7
|
+
rule in `security-rules.md` fired in setup), or if the live lane is BLOCKED. Record the skip
|
|
8
|
+
and its reason in the charter and the report.
|
|
7
9
|
|
|
8
10
|
## Step 1: Capture the Before State
|
|
9
11
|
|
|
@@ -25,6 +27,13 @@ If the UI is unavailable, call the write API directly with the same auth — but
|
|
|
25
27
|
in the report, because an API-only trigger does not prove the UI's identity
|
|
26
28
|
propagation works.
|
|
27
29
|
|
|
30
|
+
**For a service that has no UI at all, the write API *is* the real path** and no caveat is
|
|
31
|
+
owed. Restrict every state-changing call to the target's `api.write_allowlist`; an endpoint
|
|
32
|
+
absent from that list stays uncalled even when its verb looks harmless. Where the service
|
|
33
|
+
offers a workspace reset, treat it as destructive: it is the one call that erases the
|
|
34
|
+
evidence for everything already observed, so make it deliberate and never routine cleanup
|
|
35
|
+
between cases.
|
|
36
|
+
|
|
28
37
|
Conversely, if the UI trigger succeeds, that is not evidence the endpoint accepts what
|
|
29
38
|
other clients will send. The client's own guards are still in the path. Phase 3b covers
|
|
30
39
|
what happens without them.
|
|
@@ -41,7 +50,7 @@ state. Check, in this order:
|
|
|
41
50
|
3. **Fidelity** — the stored snapshot matches what the source store now holds, field for
|
|
42
51
|
field. Compare the actual documents; do not eyeball two summaries. Where the record
|
|
43
52
|
carries derived values, recompute them from the raw figures beside them
|
|
44
|
-
(
|
|
53
|
+
(`${CLAUDE_SKILL_DIR}/references/payload-verification.md`) rather than trusting that they were computed
|
|
45
54
|
once and correctly.
|
|
46
55
|
4. **Key shape** — the sort key format matches what the consumer will query and sort by.
|
|
47
56
|
5. **Completeness** — every attribute the ACs require is present, with the required type.
|
|
@@ -61,6 +70,13 @@ and event pipelines actually break:
|
|
|
61
70
|
another.
|
|
62
71
|
- **DLQ after.** Re-check depth. Anything new means the pipeline dropped something
|
|
63
72
|
while you watched.
|
|
73
|
+
- **The replayed event.** Send the same event twice — sequentially, then concurrently, then
|
|
74
|
+
with one field altered. Which field the system deduplicates on is rarely the one the
|
|
75
|
+
documentation implies, and "deduplicated" is not the same claim as "applied exactly once":
|
|
76
|
+
check the effect, not just the status.
|
|
77
|
+
- **The late event.** Deliver an event *after* a timeout, sweeper or retry has already
|
|
78
|
+
resolved the record. A system that closed the record and then still applies the arriving
|
|
79
|
+
event has two truths at once — and in a money system that is the expensive kind.
|
|
64
80
|
|
|
65
81
|
Record each with its own evidence file. Any of these that fails is a bug report,
|
|
66
82
|
whether or not an AC covers it — the ACs are a floor, not a ceiling.
|