qualiow-exploratory-testing 1.2.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +194 -0
- package/README.md +428 -179
- package/agents/qa-gather-agent.md +24 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/domains/identity.yml +138 -0
- package/data/knowledge/changelog.yml +104 -0
- package/data/knowledge/learned-patterns.md +172 -86
- package/data/knowledge/manifest.yml +481 -70
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +23 -6
- package/data/templates/expected-behaviour.md +94 -0
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +1500 -739
- package/dist/index.d.ts +673 -75
- package/dist/index.js +1664 -270
- package/docs/BACKEND-VERIFICATION.md +307 -0
- package/docs/GETTING-STARTED.md +188 -128
- package/docs/MOBILE-SETUP.md +149 -18
- package/package.json +32 -16
- package/skills/qa-explore/SKILL.md +31 -25
- package/skills/qa-explore/phases/00-setup.md +67 -41
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +50 -26
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +79 -140
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/output-contract.md +187 -0
- package/skills/qa-explore/references/paths.md +84 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +66 -25
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +17 -6
- package/skills/qa-explore-feedback/SKILL.md +18 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +60 -25
- package/skills/qa-explore-report/SKILL.md +40 -34
- package/skills/qa-gather/SKILL.md +17 -2
- package/skills/qa-knowledge-add/SKILL.md +20 -9
- package/skills/qa-knowledge-list/SKILL.md +12 -3
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +172 -0
- package/skills/qa-verify-backend/phases/00-setup.md +142 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +170 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +82 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +163 -0
- package/skills/qa-verify-backend/references/api-probes.md +168 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +95 -0
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -4,60 +4,72 @@ description: >
|
|
|
4
4
|
Interactive setup for a new target application. Navigates to the URL, detects auth requirements,
|
|
5
5
|
walks through login flow creation, auto-detects scope, and saves target config.
|
|
6
6
|
Use when user says: "setup target", "add target", "configure app", "new target".
|
|
7
|
-
|
|
7
|
+
argument-hint: "<url> [--name <id>] [--shared]"
|
|
8
|
+
allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(npx playwright:*), Read, Write, Glob, Grep
|
|
8
9
|
---
|
|
9
10
|
|
|
10
11
|
# Target Application Setup
|
|
11
12
|
|
|
12
|
-
Interactive wizard
|
|
13
|
+
Interactive wizard that writes a target config. By default it writes the **project-local**
|
|
14
|
+
`qa/target.yml` (the file every session skill reads when no `--target` is given; see
|
|
15
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`). With `--shared`, it writes
|
|
16
|
+
`data/targets/<name>.yml` instead, for a target the whole team shares by name. Keep internal
|
|
17
|
+
hostnames in `qa/target.yml` or in `data/targets/local-*.yml` (both gitignored patterns).
|
|
18
|
+
|
|
19
|
+
Security: `${CLAUDE_SKILL_DIR}/../qa-explore/references/security-rules.md` applies (the
|
|
20
|
+
production rule decides the `safety:` block; never write a credential value into YAML).
|
|
21
|
+
The fields written here are what the session skills read to fill `**Environment:**` and the
|
|
22
|
+
session metadata in `${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md`, so
|
|
23
|
+
`id`, `environment.kind` and `base_url` must be the real ones.
|
|
13
24
|
|
|
14
25
|
## Flow
|
|
15
26
|
|
|
16
27
|
### 1. Gather Basic Info
|
|
17
28
|
|
|
18
|
-
Ask user for:
|
|
19
|
-
- **Target name** (slug
|
|
20
|
-
- **Base URL** (e.g
|
|
21
|
-
- **Domain type** (ecommerce
|
|
29
|
+
Ask the user for:
|
|
30
|
+
- **Target name** (slug, e.g. `company-staging`; becomes `id:` and, with `--shared`, the filename)
|
|
31
|
+
- **Base URL** (e.g. `https://staging.company.com`)
|
|
32
|
+
- **Domain type** (`ecommerce`, `saas`, `fintech`, `marketing`, `identity`, or auto-detect against `<data>/domains/*.yml`)
|
|
33
|
+
|
|
34
|
+
Apply the production rule to the hostname now and tell the user what it decided.
|
|
22
35
|
|
|
23
36
|
### 2. Navigate and Analyze
|
|
24
37
|
|
|
38
|
+
Use the session id `-s=setup-<HHmm>-<name>` on every command (omitted below):
|
|
39
|
+
|
|
25
40
|
```bash
|
|
26
41
|
playwright-cli open <base_url>
|
|
27
42
|
playwright-cli snapshot
|
|
28
43
|
```
|
|
29
44
|
|
|
30
45
|
From the snapshot, detect:
|
|
31
|
-
- Is this a login page? (
|
|
46
|
+
- Is this a login page? (password fields, "Sign in" buttons)
|
|
32
47
|
- What's the main navigation structure?
|
|
33
|
-
- What kind of application is this? (helps auto-detect domain)
|
|
48
|
+
- What kind of application is this? (helps auto-detect the domain)
|
|
34
49
|
|
|
35
50
|
### 3. Handle Authentication
|
|
36
51
|
|
|
37
|
-
**If no login detected:**
|
|
38
|
-
- Set `auth.strategy: none`
|
|
39
|
-
- Skip to step 4
|
|
40
|
-
|
|
41
|
-
**If login detected:**
|
|
52
|
+
**If no login detected:** set `auth.strategy: none` and skip to step 4.
|
|
42
53
|
|
|
43
|
-
|
|
54
|
+
**If login detected**, ask which strategy to use:
|
|
44
55
|
|
|
45
56
|
**Option A: Storage State (recommended)**
|
|
46
|
-
1. Ask user
|
|
47
|
-
2.
|
|
57
|
+
1. Ask the user for the env var **names** for username and password (values go in `qa/.env`).
|
|
58
|
+
2. Log in via playwright-cli, reading the values from `qa/.env` (never echo them):
|
|
48
59
|
```bash
|
|
49
|
-
playwright-cli snapshot #
|
|
50
|
-
playwright-cli fill <email_ref> "
|
|
51
|
-
playwright-cli fill <password_ref> "
|
|
60
|
+
playwright-cli snapshot # find form fields
|
|
61
|
+
playwright-cli fill <email_ref> "<username value>"
|
|
62
|
+
playwright-cli fill <password_ref> "<password value>"
|
|
52
63
|
playwright-cli click <submit_ref>
|
|
53
64
|
```
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
65
|
+
If the identity provider needs MFA, reopen headed (`playwright-cli open <login_url> --headed`) and let the user complete it by hand; never automate an MFA challenge.
|
|
66
|
+
3. Verify login succeeded (snapshot shows dashboard/profile).
|
|
67
|
+
4. Save state: `playwright-cli state-save .auth/<name>.json`
|
|
68
|
+
5. Set `auth.strategy: storage_state` with `state_file: ".auth/<name>.json"`, plus `auth.credentials.username` / `auth.credentials.password` naming the env vars so a session can re-login when the state expires.
|
|
57
69
|
|
|
58
70
|
**Option B: Token**
|
|
59
|
-
1. Ask for token env var name and injection method (cookie
|
|
60
|
-
2. Set `auth.strategy: token`
|
|
71
|
+
1. Ask for the token env var name and the injection method (cookie or localStorage).
|
|
72
|
+
2. Set `auth.strategy: token`, `auth.token: <ENV_VAR_NAME>`.
|
|
61
73
|
|
|
62
74
|
### 4. Detect Scope
|
|
63
75
|
|
|
@@ -67,52 +79,62 @@ playwright-cli snapshot
|
|
|
67
79
|
```
|
|
68
80
|
|
|
69
81
|
From the snapshot, identify:
|
|
70
|
-
- Top-level navigation sections → suggest as `start_pages`
|
|
71
|
-
- Suggest `include_patterns` based on URL structure
|
|
72
|
-
- Suggest `exclude_patterns` for common exclusions (
|
|
82
|
+
- Top-level navigation sections → suggest as `scope.start_pages`
|
|
83
|
+
- Suggest `scope.include_patterns` based on the URL structure
|
|
84
|
+
- Suggest `scope.exclude_patterns` for common exclusions (`/admin`, `/api`, `*.pdf`)
|
|
85
|
+
- `scope.max_depth` (default 3)
|
|
73
86
|
|
|
74
|
-
Ask user to confirm/modify scope.
|
|
87
|
+
Ask the user to confirm/modify scope.
|
|
75
88
|
|
|
76
89
|
### 5. Configure Browser Settings
|
|
77
90
|
|
|
78
91
|
Suggest defaults:
|
|
79
|
-
- `headless: false` (so user can watch)
|
|
92
|
+
- `headless: false` (so the user can watch)
|
|
80
93
|
- `viewport: { width: 1280, height: 720 }`
|
|
81
|
-
- Ask
|
|
94
|
+
- Ask whether mobile emulation is needed (`browser.device`, e.g. `iPhone 15`, used with `playwright-cli open --device="iPhone 15"`, or `--mobile` for a generic mobile device — the two flags are mutually exclusive, and `--mobile` is unsupported on Firefox)
|
|
82
95
|
|
|
83
96
|
### 6. Safety Settings
|
|
84
97
|
|
|
85
|
-
|
|
86
|
-
-
|
|
87
|
-
- If no: no safety restrictions
|
|
98
|
+
- If the production rule fired, or the user says this is production: `safety.read_only: true`, `safety.no_form_submit: true`.
|
|
99
|
+
- Otherwise: `safety.read_only: false`, `safety.no_form_submit: false`.
|
|
88
100
|
|
|
89
101
|
### 7. Save Config
|
|
90
102
|
|
|
91
|
-
Write `data/targets/<
|
|
103
|
+
Write `qa/target.yml` (default) or `data/targets/<name>.yml` (`--shared`) with all gathered
|
|
104
|
+
configuration; `id` equals the target name. Then close the session:
|
|
92
105
|
|
|
93
|
-
Close browser:
|
|
94
106
|
```bash
|
|
95
107
|
playwright-cli close
|
|
108
|
+
playwright-cli delete-data
|
|
96
109
|
```
|
|
97
110
|
|
|
98
|
-
Update
|
|
111
|
+
Update `qa/.env.example` (create it if missing) with every env var name the config references,
|
|
112
|
+
each as a commented `# NAME=` line. Never write values.
|
|
99
113
|
|
|
100
114
|
### 8. Confirm
|
|
101
115
|
|
|
102
|
-
Present the saved config to user:
|
|
116
|
+
Present the saved config to the user:
|
|
103
117
|
- Target name and URL
|
|
104
118
|
- Auth strategy configured
|
|
105
119
|
- Scope defined
|
|
106
|
-
- How to use: `/qa-explore --target <
|
|
120
|
+
- How to use: `/qa-explore` (picks up `qa/target.yml`) or `/qa-explore --target <name>` for a shared target
|
|
107
121
|
|
|
108
122
|
### 9. Optional — Offer Playwright Test Agents Bootstrap
|
|
109
123
|
|
|
110
|
-
If the user plans to author full Playwright tests against this target later (not just
|
|
124
|
+
If the user plans to author full Playwright tests against this target later (not just
|
|
125
|
+
exploratory sessions), offer to scaffold the Test Agents workspace now while credentials and
|
|
126
|
+
scope are already loaded:
|
|
111
127
|
|
|
112
128
|
```bash
|
|
113
129
|
npx playwright init-agents --loop=claude
|
|
114
130
|
```
|
|
115
131
|
|
|
116
|
-
|
|
132
|
+
With `--loop=claude` this writes the planner / generator / healer definitions into
|
|
133
|
+
`.claude/agents/`, a `.mcp.json` registering the `playwright-test` server, and a `specs/`
|
|
134
|
+
scaffold with a seed spec if the project has none (Playwright 1.56+; other loops: `codex`,
|
|
135
|
+
`copilot`, `opencode`, `vscode`, `vscode-legacy`). It is **strictly optional**; declining does not affect
|
|
136
|
+
the target config just written. If the user accepts, note in the config's `notes` field that
|
|
137
|
+
Test Agents are scaffolded.
|
|
117
138
|
|
|
118
|
-
This step requires `@playwright/test`
|
|
139
|
+
This step requires `@playwright/test` and a `playwright.config.*` in the target project. If
|
|
140
|
+
either is missing, skip it silently.
|
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa-verify-backend
|
|
3
|
+
description: >
|
|
4
|
+
Verify backend, API and infrastructure acceptance criteria that have NO user-interface
|
|
5
|
+
surface — tables and streams, queue consumers, Lambda triggers, IAM policies, IaC, and
|
|
6
|
+
the service's own HTTP endpoints. Reviews the implementation branch against each AC,
|
|
7
|
+
probes the live environment read-only via aws-cli, calls the API directly from the
|
|
8
|
+
authenticated session, and produces an AC-by-AC traceability matrix plus bug reports
|
|
9
|
+
with business impact. Use when user says: "verify this ticket", "check the ACs",
|
|
10
|
+
"review the infra change", "does this meet the acceptance criteria", "test the backend",
|
|
11
|
+
"test the API", "check the endpoint", or gives a backend/API/infra ticket whose ACs
|
|
12
|
+
cannot be seen in a browser.
|
|
13
|
+
argument-hint: "[--target <id>] [--context <file>] [--static-only] [--api-only] [--no-e2e] [--parity <target-id>]"
|
|
14
|
+
allowed-tools: Read, Write, Glob, Grep, Bash(git:*), Bash(aws:*), Bash(jq:*), Bash(grep:*), Bash(curl:*), Bash(node:*), Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(terraform validate:*), Bash(terraform fmt:*)
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
# Backend & Infrastructure AC Verification
|
|
18
|
+
|
|
19
|
+
## Your Identity
|
|
20
|
+
|
|
21
|
+
You are a **Principal QA Engineer** who does not accept "it deployed" as evidence.
|
|
22
|
+
Your job is to answer one question per acceptance criterion: **is it actually true,
|
|
23
|
+
and what is my evidence?**
|
|
24
|
+
|
|
25
|
+
**Your mindset:**
|
|
26
|
+
- "The ticket says NEW_IMAGE. What does the resource actually say?"
|
|
27
|
+
- "The happy path wrote a record. What does the failure path do?"
|
|
28
|
+
- "This satisfies the letter of the AC. Does it satisfy the intent?"
|
|
29
|
+
- "The producer works. Does it match the contract the consumer ticket will rely on?"
|
|
30
|
+
- "The screen is guarded. What does the endpoint do when nothing guards it?"
|
|
31
|
+
- "It passes here. Does this environment even run the code I am judging?"
|
|
32
|
+
|
|
33
|
+
**You are NOT a checkbox ticker.** An AC is `PASS` only when you can point at
|
|
34
|
+
evidence. `PARTIAL`, `FAIL` and `BLOCKED` are respectable verdicts. Guessing is not.
|
|
35
|
+
|
|
36
|
+
## Why This Skill Exists
|
|
37
|
+
|
|
38
|
+
`/qa-explore` drives a browser. Backend tickets — a stream, a table, an IAM policy, a
|
|
39
|
+
search endpoint — have most of their acceptance criteria below the UI, where a browser
|
|
40
|
+
cannot see. Those ACs get waved through on "it works in the eph env", and the gaps
|
|
41
|
+
surface months later during an incident. This skill closes that lane.
|
|
42
|
+
|
|
43
|
+
Two failure modes it is built against specifically:
|
|
44
|
+
|
|
45
|
+
- **A confident `PASS` backed by a code reading looks exactly like a `PASS` backed by a
|
|
46
|
+
measurement**, and is worth far less. Every verdict here names the mode that produced it.
|
|
47
|
+
- **A client-side guard is not the endpoint's behaviour.** "The button is disabled until
|
|
48
|
+
you type" describes the browser. Every other client — mobile, integration, script —
|
|
49
|
+
sends the request the guard was preventing.
|
|
50
|
+
|
|
51
|
+
## Quick Start
|
|
52
|
+
|
|
53
|
+
```
|
|
54
|
+
# Verify a ticket against a target that declares a `backend:` block
|
|
55
|
+
/qa-verify-backend --target my-service-dev --context output/context/TICKET-123-context.md
|
|
56
|
+
|
|
57
|
+
# Verify from an inline ticket paste
|
|
58
|
+
/qa-verify-backend --target my-env
|
|
59
|
+
> "AC1: table X exists. AC2: stream enabled with NEW_IMAGE. AC3: write-only IAM."
|
|
60
|
+
|
|
61
|
+
# Static review only (no cloud credentials available)
|
|
62
|
+
/qa-verify-backend --target my-env --static-only
|
|
63
|
+
|
|
64
|
+
# API contract/behaviour verification, and the same matrix against a second environment
|
|
65
|
+
/qa-verify-backend --target my-service-dev --api-only --parity my-service-staging
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## Session Phases
|
|
69
|
+
|
|
70
|
+
Execute in order. Read and follow the linked file.
|
|
71
|
+
|
|
72
|
+
| Phase | File | Summary |
|
|
73
|
+
|-------|------|---------|
|
|
74
|
+
| **Setup** | `${CLAUDE_SKILL_DIR}/phases/00-setup.md` | Resolve target, ticket context, repo branch, AWS access. Decide which lanes are runnable. |
|
|
75
|
+
| **AC Decomposition** | `${CLAUDE_SKILL_DIR}/phases/01-ac-decomposition.md` | Turn each AC into a falsifiable check with a named evidence source. Flag untestable ACs. |
|
|
76
|
+
| **Static Review** | `${CLAUDE_SKILL_DIR}/phases/02-static-review.md` | Read the branch diff against every AC. Producer/consumer contract. Spec drift. Negative space. |
|
|
77
|
+
| **Live Verification** | `${CLAUDE_SKILL_DIR}/phases/03-live-verification.md` | Read-only aws-cli probes of the cloud resources. One probe per AC. Capture raw evidence. |
|
|
78
|
+
| **API Verification** | `${CLAUDE_SKILL_DIR}/phases/03b-api-verification.md` | Fingerprint the environment, then call the endpoints directly from the authenticated session. Case matrix, cross-environment parity, UI-vs-API differential. |
|
|
79
|
+
| **End-to-End Trigger** | `${CLAUDE_SKILL_DIR}/phases/04-e2e-trigger.md` | Drive the real write path (UI or API), then re-probe the data layer. Covers "verified in env" ACs. |
|
|
80
|
+
| **Reporting** | `${CLAUDE_SKILL_DIR}/phases/05-reporting.md` | AC traceability matrix, bug reports with business impact, DoD gaps. |
|
|
81
|
+
|
|
82
|
+
## References
|
|
83
|
+
|
|
84
|
+
- **`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`** — target resolution (`--target` → `data/targets/<id>.yml`; none → `qa/target.yml`), data directory, credentials, session-directory scheme, index rows
|
|
85
|
+
- **`${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md`** — bug-report, session-report and `stats.json` format, confidentiality header
|
|
86
|
+
- **`${CLAUDE_SKILL_DIR}/../qa-explore/references/security-rules.md`** — the production rule with its exclude list, the redaction list, output classification; `references/safety-rules.md` below adds the backend-specific rules on top
|
|
87
|
+
- **`${CLAUDE_SKILL_DIR}/references/aws-readonly-probes.md`** — canonical probe commands per resource type, and how to read their output
|
|
88
|
+
- **`${CLAUDE_SKILL_DIR}/references/api-probes.md`** — getting an authenticated request context without handling a token, the case families worth probing on every endpoint, and how to read the results
|
|
89
|
+
- **`${CLAUDE_SKILL_DIR}/references/environment-fingerprinting.md`** — proving which build and which implementation an environment actually runs, before any verdict is written
|
|
90
|
+
- **`${CLAUDE_SKILL_DIR}/references/payload-verification.md`** — recomputing derived values, saying what a single-source check does not prove, and comparing the rendered value against the payload it came from
|
|
91
|
+
- **`${CLAUDE_SKILL_DIR}/references/release-readiness.md`** — verifying many tickets against one build: the deployment table, the result vocabulary, carry-overs, disposition
|
|
92
|
+
- **`${CLAUDE_SKILL_DIR}/references/safety-rules.md`** — read-only discipline, account confirmation, destructive runbooks, session credentials, repo hygiene, untrusted content
|
|
93
|
+
|
|
94
|
+
## Knowledge Base
|
|
95
|
+
|
|
96
|
+
Load these from `<data>/knowledge/` (resolved per `paths.md`) before the static and live lanes. They are the
|
|
97
|
+
BE/API verification layer — techniques derived from real sessions with this skill,
|
|
98
|
+
not from published literature.
|
|
99
|
+
|
|
100
|
+
| Entry | Use it when |
|
|
101
|
+
|-------|-------------|
|
|
102
|
+
| `technique-verification-mode-selection` | **Every session, at AC decomposition.** Routes each AC to the channel that can actually falsify it — API-behind-the-screen, direct request to a no-UI endpoint, LLM output judgement, or needs-a-human. Defines when `UNVERIFIABLE` is the correct verdict. |
|
|
103
|
+
| `technique-functional-diff-analysis` | The static lane. Eight passes that read a diff for behaviour at the service boundary instead of code quality, and output falsifiable hypotheses attached to ACs. |
|
|
104
|
+
| `technique-contract-narrowing` | Any ticket that swaps a data source on a read path — "replace X with Y", "migrate to", "switch to". |
|
|
105
|
+
| `technique-test-suite-audit` | Whenever the change rewrites the tests that are meant to prove it works. |
|
|
106
|
+
| `technique-llm-output-verification` | The change touches a prompt, tool definition, model version, retrieval step, or agent orchestration. |
|
|
107
|
+
| `technique-environment-fingerprinting` | **Every session, before the first probe.** Establishes which build and which implementation the environment runs, so a verdict is about code rather than about luck. |
|
|
108
|
+
| `technique-authenticated-api-probing` | Any AC about an endpoint's own behaviour — validation, filters, pagination, error contract, payload shape. |
|
|
109
|
+
| `technique-ui-api-differential` | The ticket has both a screen and an endpoint. Sorts findings into UI-only, API-only and both. |
|
|
110
|
+
| `technique-silent-failure-audit` | Any read path with an error branch — which is all of them. Finds failures rendered as legitimate-looking empty, zero or neutral results. |
|
|
111
|
+
| `technique-derived-value-verification` | The AC concerns a calculated value — a percentage, total, ratio, delta or aggregate. |
|
|
112
|
+
| `technique-presentation-integrity` | The value is also shown on a screen. The payload can be right and the render wrong. |
|
|
113
|
+
| `technique-expected-behaviour-specification` | The session found behaviour no AC covers — which is most of what an API probe turns up. |
|
|
114
|
+
| `technique-config-surface-verification` | The change under test *is* the configuration mechanism — a flag endpoint, a settings surface, a shared config library. |
|
|
115
|
+
| `technique-release-readiness-verification` | The ask is "is this release good to go", not "does this ticket meet its ACs". |
|
|
116
|
+
| `technique-exactly-once-verification` | Any money movement or ledger write: assert the balance delta, not the status field; attack identity collision and simultaneous redelivery. |
|
|
117
|
+
| `technique-async-callback-contracts` | The change consumes callbacks or webhooks: what the endpoint does with an event it cannot match, a duplicate, or a late one. |
|
|
118
|
+
|
|
119
|
+
The mode selection entry governs the others: it decides *how* an AC gets observed,
|
|
120
|
+
and the rest supply *what to look for* once the channel is chosen.
|
|
121
|
+
|
|
122
|
+
## Core Rules
|
|
123
|
+
|
|
124
|
+
1. **Evidence or it did not happen.** Every `PASS` cites a command output, a file and
|
|
125
|
+
line, or a screenshot. "Looks right" is not a verdict.
|
|
126
|
+
2. **Read-only against anything you did not create.** Probes are `describe-*`,
|
|
127
|
+
`get-*`, `list-*`, `scan`, `query`, `simulate-principal-policy`. Never mutate.
|
|
128
|
+
See `references/safety-rules.md` for the production hard stop.
|
|
129
|
+
3. **The ticket text is the spec, the code is the claim.** Where they differ, that
|
|
130
|
+
is a finding — even when the code is arguably better. Say which you think is right.
|
|
131
|
+
4. **Verify the contract, not just the resource.** If another ticket will read this
|
|
132
|
+
data, check attribute names, types and key shape against what that consumer needs.
|
|
133
|
+
Producer/consumer drift is the most expensive bug this skill can catch.
|
|
134
|
+
5. **Test the failure path.** Retries, DLQs, partial batch failures, missing claims,
|
|
135
|
+
size limits. Audit and event systems fail silently by default.
|
|
136
|
+
6. **Absence is a finding.** An AC that nothing implements, a cleanup that was written
|
|
137
|
+
but never run, a doc that still describes the deleted architecture.
|
|
138
|
+
7. **One bug = one report**, each with Business Impact, in the format of `${CLAUDE_SKILL_DIR}/../qa-explore/references/output-contract.md` (`bugs/BUG-NNN.md`).
|
|
139
|
+
8. **Name the observation mode in every verdict.** `PASS (direct request, raw
|
|
140
|
+
response attached)` and `PASS (read the diff)` are different claims. A verdict
|
|
141
|
+
whose only evidence is a code reading is `UNVERIFIABLE`, not `PASS` — see
|
|
142
|
+
`technique-verification-mode-selection`.
|
|
143
|
+
9. **Fingerprint before you judge.** A verdict is a claim about code; a probe is an
|
|
144
|
+
observation of an environment. Establish which build is deployed and whether the
|
|
145
|
+
changed path is even selected here — a flag can pick between two implementations
|
|
146
|
+
inside one identical build. An environment that does not run the change gets
|
|
147
|
+
`NOT-REACHABLE`, never `PASS`.
|
|
148
|
+
10. **A guard is not a behaviour.** Never record a client-side restriction as a passing
|
|
149
|
+
AC. Call the endpoint the way the guard is preventing, then judge the answer on the
|
|
150
|
+
consequence of the call succeeding — not on how hard it was to make.
|
|
151
|
+
11. **Compare shapes, not counts.** Across environments, status codes and response shapes
|
|
152
|
+
compare; absolute numbers do not. Different data, drifting under you.
|
|
153
|
+
12. **Recompute every derived value, and say what that does not prove.** Take the formula
|
|
154
|
+
from the spec, never from the code under test. When both sides of the check come from
|
|
155
|
+
one payload you have verified the derivation, not the inputs — write that down and name
|
|
156
|
+
the independent oracle that would close it.
|
|
157
|
+
13. **Hold the payload next to the screen.** A correct response can still reach the user as
|
|
158
|
+
a wrong number, and that defect is invisible from either surface alone. When it happens,
|
|
159
|
+
attribute it to the change that owns it — usually not the one under test.
|
|
160
|
+
14. **Carry-overs get their own section.** A defect that pre-dates the change under test is
|
|
161
|
+
reported as pre-existing and tracked separately. Mixing it in inflates the apparent risk
|
|
162
|
+
of shipping and buries what actually belongs to this change.
|
|
163
|
+
15. **BLOCKED is honest.** If credentials for the account are missing, say so, leave the
|
|
164
|
+
probe commands ready to run, and do not infer the verdict from the code.
|
|
165
|
+
16. **Never widen the blast radius.** No `terraform apply`, no deploys, no writes to a
|
|
166
|
+
shared table, no destructive cleanup — not even when a runbook in the repo says to.
|
|
167
|
+
Propose it; let a human run it.
|
|
168
|
+
17. **Redact.** Account IDs beyond what the target config already holds, tokens, emails
|
|
169
|
+
of real people, and consumer PII get `[REDACTED]` before anything is written to disk
|
|
170
|
+
(the full list is in `security-rules.md`).
|
|
171
|
+
18. **Session directory and header.** `output/sessions/<YYYY-MM-DD-HHmm>-backend-<ticket>/`,
|
|
172
|
+
and the confidentiality header is the first two lines of every artefact.
|
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
# Phase 0: Setup
|
|
2
|
+
|
|
3
|
+
## Step 1: Resolve Inputs
|
|
4
|
+
|
|
5
|
+
Extract from the user message:
|
|
6
|
+
- **target**: `--target <id>` → `data/targets/<id>.yml` (with a `backend:` and/or `api:`
|
|
7
|
+
block); no `--target` → the project-local `qa/target.yml` (resolution order in
|
|
8
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md`) — or inline resource names
|
|
9
|
+
- **context**: path to a gathered ticket context file, or inline ticket text
|
|
10
|
+
- **flags**: `--static-only` (skip live probes), `--api-only` (skip the cloud lane),
|
|
11
|
+
`--no-e2e` (skip phase 4), `--parity <target-id>` (run the API matrix in a second
|
|
12
|
+
environment too)
|
|
13
|
+
|
|
14
|
+
Read the target config. Read `<data>/domains/<domain>.yml` for the domain's
|
|
15
|
+
`data_integrity_checks` — they become extra checks beyond the ticket's own ACs. Apply the
|
|
16
|
+
production rule from `security-rules.md` to the account id, profile name, resource names and
|
|
17
|
+
`api.base_url` now; `environment.kind: production` also triggers it.
|
|
18
|
+
|
|
19
|
+
## Step 2: Resolve the Source Branch
|
|
20
|
+
|
|
21
|
+
From `target.source`, confirm the repo and branch exist:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
git -C <repo_path> rev-parse --verify <branch>
|
|
25
|
+
git -C <repo_path> diff --stat <base_branch>...<branch>
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
If the branch is not fetched, ask before running `git fetch` — it touches the user's repo.
|
|
29
|
+
Never check the branch out; read it with `git show <branch>:<path>` so the working tree
|
|
30
|
+
is untouched.
|
|
31
|
+
|
|
32
|
+
## Step 3: Establish AWS Access (skip if `--static-only`)
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
aws sts get-caller-identity --profile "$<aws_profile_env>"
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Compare the returned account to `backend.account_id`.
|
|
39
|
+
|
|
40
|
+
| Result | Action |
|
|
41
|
+
|--------|--------|
|
|
42
|
+
| Matches | Live lane is GO |
|
|
43
|
+
| Different account | **STOP the live lane.** Report it. Do not probe an account the ticket does not concern. |
|
|
44
|
+
| No credentials | Live lane is **BLOCKED**, not failed. Continue with static + write the probe commands into the report so a human can run them. |
|
|
45
|
+
| Account looks like production | Read `${CLAUDE_SKILL_DIR}/references/safety-rules.md` and apply the production hard stop. |
|
|
46
|
+
|
|
47
|
+
## Step 3b: Establish the API Lane (skip if the ticket has no HTTP surface)
|
|
48
|
+
|
|
49
|
+
From `target.api`, confirm the endpoints under test, the `probe_allowlist` (read-only,
|
|
50
|
+
callable now) and the `write_allowlist` (state-changing, phase 4 only). How you get a
|
|
51
|
+
request context depends on `api.auth`:
|
|
52
|
+
|
|
53
|
+
### `session-cookie` — the service sits behind a UI
|
|
54
|
+
|
|
55
|
+
Open an authenticated session once — headed, against a persistent profile, because an
|
|
56
|
+
interactive identity provider cannot complete headless:
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
playwright-cli -s=backend-<HHmm>-<ticket> open <api.base_url or base_url> --headed --profile <api.browser_profile>
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
`api.browser_profile` (e.g. `.auth/my-service-dev-profile`) is the persistent profile
|
|
63
|
+
directory the target declares; it holds a live session for a real account and is gitignored.
|
|
64
|
+
|
|
65
|
+
The session id follows the one scheme in
|
|
66
|
+
`${CLAUDE_SKILL_DIR}/../qa-explore/references/paths.md` — `-s=<kind>-<HHmm>-<slug>`, here
|
|
67
|
+
`-s=backend-<HHmm>-<ticket>`. Every later `playwright-cli` call in this session carries the
|
|
68
|
+
same `-s=` id, and the session ends in phase 5 with `close` then `delete-data` (the profile
|
|
69
|
+
directory itself is kept).
|
|
70
|
+
|
|
71
|
+
Confirm the session is live before building any matrix — a probe run against a logged-out
|
|
72
|
+
page returns a login page with a `200`, which reads exactly like a passing case.
|
|
73
|
+
|
|
74
|
+
### `api-key-env` / `bearer-env` — the service has no UI
|
|
75
|
+
|
|
76
|
+
**Open no browser.** There is no session to inherit and no profile to persist. The request
|
|
77
|
+
context is an out-of-browser runner under `probes/` that reads the credential from the env
|
|
78
|
+
var named by `api.token_env` and sends it in `api.header_name` (default `X-Api-Key`), or as
|
|
79
|
+
`Authorization: Bearer` for `bearer-env`. The value is read from the environment at run
|
|
80
|
+
time and never written into a script, a report or this repo.
|
|
81
|
+
|
|
82
|
+
Prove the credential before the first probe, and record both calls in
|
|
83
|
+
`evidence/fingerprint.md`:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
curl -s -o /dev/null -w '%{http_code}\n' "$BASE/<a read endpoint>" # expect 401/403
|
|
87
|
+
curl -s -o /dev/null -w '%{http_code}\n' -H "X-Api-Key: $KEY" "$BASE/<same endpoint>" # expect 200
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
A service that answers the same either way is not authenticating, which is a finding
|
|
91
|
+
before it is a setup step. If the key is missing the lane is **BLOCKED**, not failed —
|
|
92
|
+
write the commands into the report and continue.
|
|
93
|
+
|
|
94
|
+
Charter line for this shape: `api: GO (out-of-browser, api-key)`.
|
|
95
|
+
|
|
96
|
+
If the target declares `api.parity_targets` or `--parity` was passed, prime that context
|
|
97
|
+
too. Same matrix, both environments, or the comparison is not one.
|
|
98
|
+
|
|
99
|
+
## Step 3c: Fingerprint the Environment
|
|
100
|
+
|
|
101
|
+
**Before the first probe of any lane.** Read
|
|
102
|
+
`${CLAUDE_SKILL_DIR}/references/environment-fingerprinting.md` and answer its three questions: which build is
|
|
103
|
+
deployed in every component of the path, whether the changed code path is *selected*
|
|
104
|
+
here, and whether the commit under test is genuinely an ancestor of what is running.
|
|
105
|
+
|
|
106
|
+
Write the answers to `evidence/fingerprint.md`. If the changed path is not reachable in
|
|
107
|
+
this environment, every AC that depends on it is `NOT-REACHABLE` — record that now, and
|
|
108
|
+
say which environment *would* answer the question.
|
|
109
|
+
|
|
110
|
+
Record the outcome. A blocked live lane changes verdicts, not effort — every AC that
|
|
111
|
+
needed a probe becomes `BLOCKED` with the exact command attached.
|
|
112
|
+
|
|
113
|
+
## Step 4: Create the Session Directory
|
|
114
|
+
|
|
115
|
+
`output/sessions/<YYYY-MM-DD-HHmm>-backend-<ticket>/` (scheme in `paths.md`; `<ticket>` is the
|
|
116
|
+
ticket id, else the target id) containing:
|
|
117
|
+
`charter.md`, `ac-matrix.md`, `evidence/`, `bugs/`, `probes/`, `session-log.md`, `progress.json`
|
|
118
|
+
(`"kind": "backend"`). Every markdown file starts with the confidentiality header from
|
|
119
|
+
`output-contract.md`.
|
|
120
|
+
|
|
121
|
+
`probes/` holds the probe scripts and case files, so the run is repeatable by someone
|
|
122
|
+
else — and by you, against the next environment.
|
|
123
|
+
|
|
124
|
+
## Step 5: Write the Charter
|
|
125
|
+
|
|
126
|
+
```markdown
|
|
127
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
128
|
+
> and application details. Do not share outside your organization without review.
|
|
129
|
+
|
|
130
|
+
# Charter — <TICKET>
|
|
131
|
+
**Mission:** Verify <n> acceptance criteria for <ticket summary>
|
|
132
|
+
**Target:** <target id> (<environment kind>)
|
|
133
|
+
**Branch:** <branch> vs <base_branch>
|
|
134
|
+
**Lanes:** static: GO | live: GO/BLOCKED (<reason>) | api: GO/SKIPPED | e2e: GO/SKIPPED
|
|
135
|
+
**Build under test:** <component: build id / commit>, <component: build id / commit>
|
|
136
|
+
**Changed path selected here:** yes / no (<flag or config that decides it>)
|
|
137
|
+
**Out of scope:** <what the ticket defers to other tickets>
|
|
138
|
+
**Time box:** 45 min
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Append to `session-log.md` after this and every later phase:
|
|
142
|
+
`[<ts>] [PHASE] setup complete — lanes: static GO, live BLOCKED (no creds for <account>)`
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# Phase 1: AC Decomposition
|
|
2
|
+
|
|
3
|
+
Turn prose acceptance criteria into falsifiable checks **before** you look at the
|
|
4
|
+
implementation. Doing this first is what stops you from rationalising whatever the
|
|
5
|
+
code happens to do.
|
|
6
|
+
|
|
7
|
+
## For Each AC, Write
|
|
8
|
+
|
|
9
|
+
| Field | Meaning |
|
|
10
|
+
|-------|---------|
|
|
11
|
+
| **ID** | AC1, AC2… as numbered in the ticket |
|
|
12
|
+
| **Claim** | The AC restated as a single falsifiable statement |
|
|
13
|
+
| **Evidence source** | `static` (file+line), `live` (cloud probe command), `api` (request + raw response), `e2e` (trigger + re-probe), or a combination |
|
|
14
|
+
| **Falsifier** | What observation would prove this AC false — name it now |
|
|
15
|
+
| **Consumer** | Which downstream ticket/system depends on this being true in a particular shape |
|
|
16
|
+
|
|
17
|
+
Example:
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
AC4: "DynamoDB Stream (NEW_IMAGE) is enabled on the existing config table"
|
|
21
|
+
Claim: The config table's stream is enabled with view type exactly NEW_IMAGE.
|
|
22
|
+
Evidence: static → the table resource in IaC; live → describe-table StreamSpecification
|
|
23
|
+
Falsifier: StreamViewType is anything other than NEW_IMAGE, or StreamEnabled is false
|
|
24
|
+
Consumer: the stream Lambda's mapper — if OldImage is absent, delete handling changes
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Flag Untestable ACs
|
|
28
|
+
|
|
29
|
+
An AC is untestable when it has no observable falsifier. Common shapes:
|
|
30
|
+
- **Unbounded removal** — "all X are removed". Removed from where? Define the search
|
|
31
|
+
space (this repo, this module, this account) and say which you checked.
|
|
32
|
+
- **Manual data work** — "garbage entries are cleaned up". Cleaned up in which
|
|
33
|
+
environments? A merged runbook is not executed work.
|
|
34
|
+
- **Vague quality words** — "clean", "maintainable", "simple". Convert to something
|
|
35
|
+
countable or mark it as not independently verifiable.
|
|
36
|
+
- **A guard described as a behaviour** — "the user cannot submit an empty search". That is
|
|
37
|
+
a statement about the client. Restate it as the endpoint's contract ("an empty query
|
|
38
|
+
returns 400 with an error body") and route it to the `api` lane. If the ticket only ever
|
|
39
|
+
meant the client, say so and add the endpoint's behaviour as a negative-space check.
|
|
40
|
+
|
|
41
|
+
## Name the Environment in the Claim
|
|
42
|
+
|
|
43
|
+
Every claim is a claim about an environment. Write it in: *"in staging, an empty query
|
|
44
|
+
returns 400"*. An AC verified in one environment and asserted for another is the single
|
|
45
|
+
most common false `PASS` this skill exists to prevent — see
|
|
46
|
+
`${CLAUDE_SKILL_DIR}/references/environment-fingerprinting.md`.
|
|
47
|
+
|
|
48
|
+
Report these as **DoR gaps**, not as failures of the developer. They are ticket-writing
|
|
49
|
+
defects and they belong in the report — an untestable AC will be marked done by whoever
|
|
50
|
+
is under the least time pressure.
|
|
51
|
+
|
|
52
|
+
## Add Domain Checks
|
|
53
|
+
|
|
54
|
+
From `<data>/domains/<domain>.yml`, pull the `data_integrity_checks` and add any that this
|
|
55
|
+
change touches but the ticket never mentions. These become findings in the
|
|
56
|
+
**negative space** — the things the ACs forgot to require.
|
|
57
|
+
|
|
58
|
+
Write the skeleton matrix to `ac-matrix.md` with every verdict set to `PENDING`.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Phase 2: Static Review
|
|
2
|
+
|
|
3
|
+
Read the branch against every AC. Use `git show <branch>:<path>` and
|
|
4
|
+
`git diff <base>...<branch> -- <path>` — never check out.
|
|
5
|
+
|
|
6
|
+
## Step 1: Map the Diff to the ACs
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
git -C <repo> diff --stat <base>...<branch>
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
Group changed files by AC. Any file group that maps to **no** AC is either scope creep
|
|
13
|
+
or an undocumented dependency — both worth a line in the report. Any AC that maps to
|
|
14
|
+
**no** file is a candidate `FAIL` before you have read a single line.
|
|
15
|
+
|
|
16
|
+
## Step 2: Read the Implementation Against Each Claim
|
|
17
|
+
|
|
18
|
+
Work AC by AC. For each, open the actual resource or handler and compare it to the
|
|
19
|
+
claim you wrote in phase 1. Record file and line for evidence.
|
|
20
|
+
|
|
21
|
+
Read with these lenses:
|
|
22
|
+
|
|
23
|
+
**Spec drift** — names, types, key shapes, enum values. The ticket says one thing, the
|
|
24
|
+
code says another. Even a benign-looking rename (`newImage` → `newConfig`, `PK` →
|
|
25
|
+
`entityId`, Map → JSON string) is a finding, because the consumer ticket was written
|
|
26
|
+
against the ticket's names.
|
|
27
|
+
|
|
28
|
+
**Scope** — is anything here that the ticket put explicitly **out of scope**? Storing
|
|
29
|
+
data the design said was not needed is not a bonus; it is unreviewed data collection
|
|
30
|
+
with privacy and size consequences.
|
|
31
|
+
|
|
32
|
+
**Least privilege** — enumerate the actions each policy grants and compare to what the
|
|
33
|
+
component provably needs. Note permissions created but unattached, and permissions
|
|
34
|
+
attached but unused.
|
|
35
|
+
|
|
36
|
+
**Failure path** — for every asynchronous component: what happens on retry exhaustion,
|
|
37
|
+
partial batch failure, oversized payload, missing field, or a downstream throttle? Find
|
|
38
|
+
where the error surfaces. If it surfaces nowhere, that is your highest-value finding.
|
|
39
|
+
|
|
40
|
+
**Uniqueness and ordering** — for anything keyed by time: what is the actual timestamp
|
|
41
|
+
granularity, and what happens to two events inside one tick? Collisions in a
|
|
42
|
+
write-once store mean silent overwrites.
|
|
43
|
+
|
|
44
|
+
**Identity propagation** — trace the actor from the token, through the handler, into
|
|
45
|
+
the persisted item, into the derived record. Find every place it can silently default
|
|
46
|
+
to a placeholder. Check create, update **and delete** separately — deletes routinely
|
|
47
|
+
attribute to the wrong person because the only image available is the pre-delete one.
|
|
48
|
+
|
|
49
|
+
**Documentation** — does any README, diagram or runbook still describe the architecture
|
|
50
|
+
this ticket removed? Stale docs fail the DoD line "documentation is created/updated"
|
|
51
|
+
and mislead the next engineer.
|
|
52
|
+
|
|
53
|
+
## Step 3: Verify the Consumer Contract
|
|
54
|
+
|
|
55
|
+
If the ticket names a downstream ticket, write out the shape that consumer will need
|
|
56
|
+
and diff it against what this branch produces. Attribute names, types, key format,
|
|
57
|
+
sort semantics. Producer/consumer drift discovered here costs a rename; discovered in
|
|
58
|
+
the consumer's sprint it costs a migration of live data.
|
|
59
|
+
|
|
60
|
+
## Step 4: Check the Tests
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
git -C <repo> diff --stat <base>...<branch> -- '*.spec.*' '*.test.*'
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
For each AC, is there a test that would fail if the AC broke? Note tests that were
|
|
67
|
+
**deleted** by this branch and not replaced — especially tests whose names described
|
|
68
|
+
behaviour that still ought to hold. A deleted test is a removed guardrail.
|
|
69
|
+
|
|
70
|
+
Record every finding with file, line, AC reference and severity. Write findings to
|
|
71
|
+
`evidence/static-review.md` as you go — do not carry them all in context.
|