qualiow-exploratory-testing 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +92 -0
- package/data/domains/_default.md +181 -0
- package/data/domains/ecommerce.md +192 -0
- package/data/domains/fintech.md +200 -0
- package/data/domains/identity.md +88 -0
- package/data/domains/identity.yml +138 -0
- package/data/domains/marketing.md +194 -0
- package/data/domains/saas.md +197 -0
- package/data/knowledge/changelog.yml +86 -0
- package/data/knowledge/learned-patterns.md +172 -86
- package/data/knowledge/manifest.yml +211 -6
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/coverage-map.md +14 -0
- package/data/templates/expected-behaviour.md +93 -0
- package/dist/cli/index.js +79 -70
- package/dist/cli/index.js.map +1 -1
- package/dist/index.d.ts +90 -0
- package/dist/index.js +58 -49
- package/dist/index.js.map +1 -1
- package/docs/BACKEND-VERIFICATION.md +277 -0
- package/package.json +5 -3
- package/skills/qa-verify-backend/SKILL.md +163 -0
- package/skills/qa-verify-backend/phases/00-setup.md +97 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +157 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +66 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +135 -0
- package/skills/qa-verify-backend/references/api-probes.md +141 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +90 -0
|
@@ -0,0 +1,277 @@
|
|
|
1
|
+
# Backend & Infrastructure AC Verification
|
|
2
|
+
|
|
3
|
+
A guide to `/qa-verify-backend` — the lane for tickets whose acceptance criteria cannot be
|
|
4
|
+
seen in a browser.
|
|
5
|
+
|
|
6
|
+
## Why this exists
|
|
7
|
+
|
|
8
|
+
`/qa-explore` drives a browser. A backend ticket — a stream, a table, an IAM policy, a queue
|
|
9
|
+
consumer, a webhook — has most of its acceptance criteria below the UI, where a browser cannot
|
|
10
|
+
reach. Those ACs get waved through on "it works in the dev env", and the gaps surface months
|
|
11
|
+
later during an incident.
|
|
12
|
+
|
|
13
|
+
Three failure modes this skill is built against, all subtler than a missed bug:
|
|
14
|
+
|
|
15
|
+
- **A confident `PASS` backed by a code reading** looks exactly like a `PASS` backed by a
|
|
16
|
+
measurement, and is worth far less. Refusing to fake that is what makes the other verdicts
|
|
17
|
+
credible.
|
|
18
|
+
- **A client-side guard is not the endpoint's behaviour.** "The button is disabled until you
|
|
19
|
+
type" describes the browser. Every other client — mobile, integration, script — sends the
|
|
20
|
+
request the guard was preventing, and that request is frequently the one that fails.
|
|
21
|
+
- **A verdict is a claim about code; a probe is an observation of an environment.** They
|
|
22
|
+
coincide only when that environment runs the change. One identical build can hold two
|
|
23
|
+
implementations of one feature with a flag choosing between them.
|
|
24
|
+
|
|
25
|
+
## The workflow
|
|
26
|
+
|
|
27
|
+
### 1. Analyse the ticket → `output/context/<TICKET>-context.md`
|
|
28
|
+
|
|
29
|
+
Run `/qa-gather` on the ticket, or write the context file by hand. The job is to turn prose
|
|
30
|
+
ACs into falsifiable claims **before** reading the implementation — doing it in that order is
|
|
31
|
+
what stops you rationalising whatever the code happens to do.
|
|
32
|
+
|
|
33
|
+
Flag **DoR gaps** while you are here: ACs with no observable falsifier, missing criteria for
|
|
34
|
+
behaviour the change clearly has, internal contradictions. Take each AC literally and ask what
|
|
35
|
+
observation would prove it false. An AC whose spec is wrong is worth finding before anyone
|
|
36
|
+
implements it.
|
|
37
|
+
|
|
38
|
+
### 2. Create a target config
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
cp data/targets/_example-backend.yml data/targets/local-my-service.yml
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Fill in `base_url`, `environment.kind`, the `backend.resources` names, and `source.branch`.
|
|
45
|
+
|
|
46
|
+
Two rules:
|
|
47
|
+
|
|
48
|
+
- **Credentials are referenced by env var NAME, never by value.** `aws_profile_env:
|
|
49
|
+
QA_AWS_PROFILE` — the value lives in `.env`, which is gitignored.
|
|
50
|
+
- **Name a private target `local-*.yml`.** That prefix is gitignored, so internal hostnames,
|
|
51
|
+
account ids and resource names stay on your machine. Add your own prefix to `.gitignore` if
|
|
52
|
+
you prefer a different convention.
|
|
53
|
+
|
|
54
|
+
### 3. Verify
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
/qa-verify-backend --target local-my-service --context output/context/TICKET-123-context.md
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Flags: `--static-only` (skip live probes — use when you have no cloud credentials),
|
|
61
|
+
`--api-only` (skip the cloud lane and verify the HTTP surface), `--no-e2e` (skip the
|
|
62
|
+
write-path phase), `--parity <target-id>` (run the same API matrix in a second environment
|
|
63
|
+
and compare).
|
|
64
|
+
|
|
65
|
+
### 4. Explore the UI, if the ticket has one
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
/qa-explore --target local-my-service --context output/context/TICKET-123-context.md
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
### 5. Hand off
|
|
72
|
+
|
|
73
|
+
`ac-matrix.md` is the headline deliverable. Bug reports land in `bugs/`, one per finding, each
|
|
74
|
+
with a Business Impact written in terms of consequence rather than mechanism — nobody funds a
|
|
75
|
+
fix for "StreamViewType is NEW_AND_OLD_IMAGES".
|
|
76
|
+
|
|
77
|
+
## The four lanes
|
|
78
|
+
|
|
79
|
+
| Lane | Reads | Never |
|
|
80
|
+
|------|-------|-------|
|
|
81
|
+
| **Static** | The implementation branch via `git show <ref>:<path>` | Checks out, fetches without asking, or creates branches — you may have uncommitted work |
|
|
82
|
+
| **Live** | `describe-*`, `get-*`, `list-*`, `query`, `scan`, `simulate-principal-policy` | Mutates anything, or probes an account other than the one the target declares |
|
|
83
|
+
| **API** | The service's own endpoints, from the authenticated session — the cases a browser cannot send | Calls anything outside `api.probe_allowlist`, or anything that creates, mutates or deletes |
|
|
84
|
+
| **End-to-end** | The real write path in a non-production environment, then the data layer again | Runs against production, or writes directly to a data store (that bypasses the path under test and proves nothing) |
|
|
85
|
+
|
|
86
|
+
### Before any lane: fingerprint the environment
|
|
87
|
+
|
|
88
|
+
A green result against an environment that does not run the changed code proves nothing, and a
|
|
89
|
+
red result there is not the ticket's bug. Phase 0 establishes, with evidence, which build is
|
|
90
|
+
deployed in **every** component of the request path, whether the changed path is *selected*
|
|
91
|
+
here (a flag, a config value, a routing rule), and whether the commit under test is genuinely
|
|
92
|
+
an ancestor of what is running.
|
|
93
|
+
|
|
94
|
+
> **Same build, different behaviour ⇒ configuration, not deploy lag.**
|
|
95
|
+
|
|
96
|
+
The consequence is a verdict: an environment that does not run the change gets
|
|
97
|
+
`NOT-REACHABLE`. And a fix that only lands behind a flag which is off in the environments
|
|
98
|
+
users are in has reached nobody — that is a finding in its own right, because flipping the
|
|
99
|
+
flag is a separate decision carrying its own risk.
|
|
100
|
+
|
|
101
|
+
### The API lane, specifically
|
|
102
|
+
|
|
103
|
+
The request runs **inside the already-authenticated page**, so it carries the same session
|
|
104
|
+
cookie, CSRF token and interceptors the UI has. There is no token to obtain, nothing to store,
|
|
105
|
+
and it works with SSO and MFA that no scripted login can pass:
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
playwright-cli -s=my-env open https://service.example.com/ --headed --profile .auth/my-env
|
|
109
|
+
playwright-cli -s=my-env eval "$(cat probes/search-cases.js)" --raw
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Cases come from the ACs plus the standard families — length boundaries, tokenisation,
|
|
113
|
+
metacharacters of whatever query language sits underneath, the four different kinds of
|
|
114
|
+
nothing (`""`, `null`, whitespace-only, field absent), enum values valid and invalid,
|
|
115
|
+
pagination bounds, type confusion, a second identity and a second scope.
|
|
116
|
+
|
|
117
|
+
If the ticket also has a screen, run the overlapping cases through it too and sort every
|
|
118
|
+
finding into **both** (the UI is faithful — fix it in the service), **API only** (a real
|
|
119
|
+
defect the client's guard is hiding, reachable by every other client), **UI only** (the client
|
|
120
|
+
invents or masks behaviour the service does not have) and **neither**.
|
|
121
|
+
|
|
122
|
+
### A well-shaped `200` is not a correct answer
|
|
123
|
+
|
|
124
|
+
Once the contract holds, the values are still an open question. Every derived number — a
|
|
125
|
+
percentage, total, ratio, delta or aggregate — is recomputed from the raw figures in the same
|
|
126
|
+
response, using the formula from the **specification** rather than from the code under test,
|
|
127
|
+
with cases chosen to stress sign, zero, scale and cardinality, plus the structural invariants
|
|
128
|
+
that must hold regardless of magnitude.
|
|
129
|
+
|
|
130
|
+
Then the report says what that does not prove. When both sides of the check come from one
|
|
131
|
+
payload, the *derivation* is verified and the *inputs* are not — so the limitation is written
|
|
132
|
+
down and the **independent oracle** that would close it is named, along with whether it was
|
|
133
|
+
run. "Internal consistency verified; the outstanding oracle is X and has not been run" is an
|
|
134
|
+
honest, useful verdict. "The numbers are correct" is not.
|
|
135
|
+
|
|
136
|
+
And the payload is held next to the screen, because a correct response can still reach the
|
|
137
|
+
user as a wrong number — a formatter that guesses what a value is, a unit applied twice,
|
|
138
|
+
rounding that crosses a threshold, a truncated figure shown as a total. That defect is
|
|
139
|
+
invisible from either surface alone, and it usually belongs to a different change than the one
|
|
140
|
+
under test. See `.claude/skills/qa-verify-backend/references/payload-verification.md`.
|
|
141
|
+
|
|
142
|
+
## Verdicts
|
|
143
|
+
|
|
144
|
+
| Verdict | Means |
|
|
145
|
+
|---------|-------|
|
|
146
|
+
| `PASS` | Verified, with cited evidence |
|
|
147
|
+
| `PARTIAL` | The intent is met but something material deviates from the stated spec |
|
|
148
|
+
| `FAIL` | The AC as written is not true |
|
|
149
|
+
| `BLOCKED` | Not verifiable with available access — the probe command is attached, ready to run |
|
|
150
|
+
| `NOT-REACHABLE` | This environment does not run the changed code path — a flag, a config value, or an undeployed commit. Not a pass and not a failure |
|
|
151
|
+
| `UNVERIFIABLE` | The AC has no falsifier as written — a DoR defect |
|
|
152
|
+
|
|
153
|
+
Every verdict names the observation mode that produced it **and the environment it holds in**.
|
|
154
|
+
`PASS (dev)` and `PASS` are different claims, and only one of them is honest when the change is
|
|
155
|
+
behind a flag that is off everywhere else. A row is never left at `PENDING`: if it was not
|
|
156
|
+
checked, it is `BLOCKED` and says why.
|
|
157
|
+
|
|
158
|
+
## Choosing how to observe
|
|
159
|
+
|
|
160
|
+
Each AC is routed to the channel that can actually falsify it
|
|
161
|
+
(`technique-verification-mode-selection`):
|
|
162
|
+
|
|
163
|
+
- **API-behind-the-screen** — a user flow reaches the changed code, but you assert on the
|
|
164
|
+
network calls underneath, not the rendered screen. The UI is a lossy renderer: it hides
|
|
165
|
+
fields it does not display, coerces types, and swallows partial failures.
|
|
166
|
+
- **Direct request** — webhooks, internal endpoints, queue consumers, scheduled jobs, and the
|
|
167
|
+
inputs a screen's own guards prevent. Nothing in any product flow reaches these, so you fire
|
|
168
|
+
the request yourself and follow through to what it produced
|
|
169
|
+
(`technique-authenticated-api-probing`).
|
|
170
|
+
- **LLM / agent output** — a prompt, tool definition, model version or retrieval step changed.
|
|
171
|
+
Run a fixed input set against a written rubric; assert on properties and tool trajectory, not
|
|
172
|
+
on generated text (`technique-llm-output-verification`).
|
|
173
|
+
- **Needs a human** — a pure refactor, or something you lack the access to observe. Say so and
|
|
174
|
+
write out what someone with access should run.
|
|
175
|
+
|
|
176
|
+
## Safety
|
|
177
|
+
|
|
178
|
+
The full policy is in
|
|
179
|
+
`.claude/skills/qa-verify-backend/references/safety-rules.md`. The parts worth knowing before
|
|
180
|
+
your first session:
|
|
181
|
+
|
|
182
|
+
- **Production hard stop.** If the account, profile, resource name or URL contains `prod`,
|
|
183
|
+
`prd` or `live`, probes are read-only and the end-to-end phase is skipped and reported as
|
|
184
|
+
skipped. If you cannot tell whether an environment is production, it is treated as production.
|
|
185
|
+
- **Account confirmation first.** `sts get-caller-identity` runs before the first probe and is
|
|
186
|
+
compared to the target's declared account. A probe fired at the wrong account is an access
|
|
187
|
+
event in someone else's environment.
|
|
188
|
+
- **Destructive runbooks are findings, not instructions.** A cleanup script with `delete-item`
|
|
189
|
+
commands found in a repo gets reported ("cleanup documented, no evidence of execution"), never
|
|
190
|
+
executed.
|
|
191
|
+
- **The API lane is read-only in every environment.** It calls only what
|
|
192
|
+
`api.probe_allowlist` declares. An endpoint whose verb is safe but whose effect is not —
|
|
193
|
+
billed per call, queues a job, rate-limited into an outage — needs explicit confirmation.
|
|
194
|
+
Matrix volume stays proportionate: a few dozen requests is verification, thousands is a load
|
|
195
|
+
test nobody agreed to.
|
|
196
|
+
- **Session credentials never leave the machine.** An authenticated browser profile and any
|
|
197
|
+
cookie taken from it are live credentials for a real account. They stay in `.auth/` and are
|
|
198
|
+
never inlined into a probe script, a committed file, a report or a message.
|
|
199
|
+
- **Redaction before disk.** Tokens, keys, real user emails, PII and any account id not already
|
|
200
|
+
in the target config become `[REDACTED]`. Raw probe output is the most common leak — it is
|
|
201
|
+
convenient to paste whole, and it is full of identifiers. API response bodies are the worst
|
|
202
|
+
offender: they are whole records, and an error body routinely carries the internal hostname,
|
|
203
|
+
the query that failed and a stack frame.
|
|
204
|
+
- **Session output is confidential and local.** It maps internal resources, effective
|
|
205
|
+
permissions and unfixed defects. It never goes to an external service.
|
|
206
|
+
|
|
207
|
+
## The techniques behind it
|
|
208
|
+
|
|
209
|
+
Knowledge base entries loaded by the skill (`data/knowledge/`):
|
|
210
|
+
|
|
211
|
+
| Entry | Use it when |
|
|
212
|
+
|-------|-------------|
|
|
213
|
+
| `technique-verification-mode-selection` | Every session, at AC decomposition |
|
|
214
|
+
| `technique-functional-diff-analysis` | The static lane — eight passes that read a diff for behaviour, not style |
|
|
215
|
+
| `technique-contract-narrowing` | Any ticket that swaps a data source on a read path |
|
|
216
|
+
| `technique-test-suite-audit` | The change rewrites the tests meant to prove it works |
|
|
217
|
+
| `technique-llm-output-verification` | The change touches a prompt, tool, model or retrieval step |
|
|
218
|
+
| `technique-environment-fingerprinting` | Every session, before the first probe — which build and which implementation is actually running |
|
|
219
|
+
| `technique-authenticated-api-probing` | Any AC about an endpoint's own behaviour: validation, filters, pagination, error contract, payload shape |
|
|
220
|
+
| `technique-ui-api-differential` | The ticket has both a screen and an endpoint |
|
|
221
|
+
| `technique-silent-failure-audit` | Any read path with an error branch — failures rendered as ordinary-looking empty or zero results |
|
|
222
|
+
| `technique-derived-value-verification` | The AC concerns a calculated value — percentage, total, ratio, delta, aggregate |
|
|
223
|
+
| `technique-presentation-integrity` | The value is also shown on a screen |
|
|
224
|
+
| `technique-expected-behaviour-specification` | The session found behaviour no AC covers |
|
|
225
|
+
| `technique-config-surface-verification` | The change under test *is* the configuration mechanism |
|
|
226
|
+
| `technique-release-readiness-verification` | The ask is "is this release good to go" |
|
|
227
|
+
|
|
228
|
+
Browse them with `/qa-knowledge-list`.
|
|
229
|
+
|
|
230
|
+
## Verifying a release rather than a ticket
|
|
231
|
+
|
|
232
|
+
When the ask is "is this release good to go" instead of "does this ticket meet its ACs", follow
|
|
233
|
+
`.claude/skills/qa-verify-backend/references/release-readiness.md`. It changes the shape of
|
|
234
|
+
the session:
|
|
235
|
+
|
|
236
|
+
- **The build table comes first, for everything.** Per ticket, is the commit it depends on
|
|
237
|
+
genuinely an ancestor of what is deployed? This reclassifies half the session before any
|
|
238
|
+
testing happens. A ticket whose backend is not deployed is **not testable here** — and testing
|
|
239
|
+
its UI anyway produces a convincing, meaningless result, because the frontend ships the
|
|
240
|
+
feature, the API does not answer, and it renders as dashes or zeros that look exactly like a
|
|
241
|
+
data bug.
|
|
242
|
+
- **A fixed result vocabulary** — clean pass · pass with caveats · fail · not testable here ·
|
|
243
|
+
not tested — so the last two stay visible instead of vanishing between passed and failed.
|
|
244
|
+
- **A coverage map with four states**, where 🔍 *code-verified only* is marked distinctly
|
|
245
|
+
because it is `UNVERIFIABLE`, not a pass. Plus the one sentence naming which untested item
|
|
246
|
+
carries the most risk.
|
|
247
|
+
- **Carry-overs in their own section** — pre-existing defects mixed into a release's results
|
|
248
|
+
inflate the apparent risk of shipping it and bury what belongs to it.
|
|
249
|
+
- **A disposition with its reversal condition**, and the scope of what was checked stated
|
|
250
|
+
explicitly when it is narrower than the question being asked.
|
|
251
|
+
|
|
252
|
+
## Writing the spec where none exists
|
|
253
|
+
|
|
254
|
+
Most of what an API probe turns up has no acceptance criterion behind it, so there is nothing to
|
|
255
|
+
file it against and it becomes an argument rather than a fix.
|
|
256
|
+
`data/templates/expected-behaviour.md` is the artifact for that: observed against expected,
|
|
257
|
+
grouped by cause rather than by case, with the decisions the fix forces made explicit —
|
|
258
|
+
|
|
259
|
+
- **Reject, or clamp?** Reject. Silently serving something other than what was asked for breaks
|
|
260
|
+
every caller that trusts the value back.
|
|
261
|
+
- **An error, or an empty result?** An error. A silent zero is indistinguishable from a genuine
|
|
262
|
+
no-match and will be read as data.
|
|
263
|
+
- **Where does validation live?** At the layer covering every implementation of the feature —
|
|
264
|
+
never only in the client, which is not in the path for any other caller.
|
|
265
|
+
|
|
266
|
+
— ranked by what real users can reach *today* rather than by how bad each one reads, and closed
|
|
267
|
+
with the same content in plain English, ready to paste into a ticket comment. That last section
|
|
268
|
+
is what gets the work scheduled.
|
|
269
|
+
|
|
270
|
+
## Templates
|
|
271
|
+
|
|
272
|
+
| Template | Use |
|
|
273
|
+
|----------|-----|
|
|
274
|
+
| `data/templates/api-probe-matrix.md` | The environment fingerprint, the case table with a column per environment, the four-bucket findings table, the evidence index. Copy it in at the start of the API lane |
|
|
275
|
+
| `data/templates/expected-behaviour.md` | The specification that should have existed, plus the plain-English reply |
|
|
276
|
+
| `data/templates/coverage-map.md` | Coverage with four states, including code-verified-only |
|
|
277
|
+
| `data/templates/bug-report.md` | One bug per report, with Business Impact |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "qualiow-exploratory-testing",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.3.0",
|
|
4
4
|
"description": "qualiow — AI-powered exploratory testing skills using Claude Code + Playwright CLI (web) and Maestro/mobile-cli (iOS/Android simulators)",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
"bin/",
|
|
15
15
|
"scripts/setup-mobile.sh",
|
|
16
16
|
"scripts/doctor-mobile.sh",
|
|
17
|
-
"data/domains
|
|
17
|
+
"data/domains/",
|
|
18
18
|
"data/knowledge/",
|
|
19
19
|
"data/templates/",
|
|
20
20
|
"data/targets/_default.yml",
|
|
@@ -22,9 +22,11 @@
|
|
|
22
22
|
"data/targets/_example-sim-ios-safari.yml",
|
|
23
23
|
"data/targets/_example-sim-android-chrome.yml",
|
|
24
24
|
"data/targets/_example-mobile-emulation.yml",
|
|
25
|
+
"data/targets/_example-backend.yml",
|
|
25
26
|
"data/targets/testers-ai.yml",
|
|
26
27
|
"docs/GETTING-STARTED.md",
|
|
27
|
-
"docs/MOBILE-SETUP.md"
|
|
28
|
+
"docs/MOBILE-SETUP.md",
|
|
29
|
+
"docs/BACKEND-VERIFICATION.md"
|
|
28
30
|
],
|
|
29
31
|
"scripts": {
|
|
30
32
|
"sync:skills": "rm -rf skills && cp -R .claude/skills skills",
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa-verify-backend
|
|
3
|
+
description: >
|
|
4
|
+
Verify backend, API and infrastructure acceptance criteria that have NO user-interface
|
|
5
|
+
surface — tables and streams, queue consumers, Lambda triggers, IAM policies, IaC, and
|
|
6
|
+
the service's own HTTP endpoints. Reviews the implementation branch against each AC,
|
|
7
|
+
probes the live environment read-only via aws-cli, calls the API directly from the
|
|
8
|
+
authenticated session, and produces an AC-by-AC traceability matrix plus bug reports
|
|
9
|
+
with business impact. Use when user says: "verify this ticket", "check the ACs",
|
|
10
|
+
"review the infra change", "does this meet the acceptance criteria", "test the backend",
|
|
11
|
+
"test the API", "check the endpoint", or gives a backend/API/infra ticket whose ACs
|
|
12
|
+
cannot be seen in a browser.
|
|
13
|
+
allowed-tools: Read, Write, Glob, Grep, Bash(git:*), Bash(aws:*), Bash(jq:*), Bash(curl:*), Bash(playwright-cli:*), Bash(terraform validate:*), Bash(terraform fmt:*)
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
# Backend & Infrastructure AC Verification
|
|
17
|
+
|
|
18
|
+
## Your Identity
|
|
19
|
+
|
|
20
|
+
You are a **Principal QA Engineer** who does not accept "it deployed" as evidence.
|
|
21
|
+
Your job is to answer one question per acceptance criterion: **is it actually true,
|
|
22
|
+
and what is my evidence?**
|
|
23
|
+
|
|
24
|
+
**Your mindset:**
|
|
25
|
+
- "The ticket says NEW_IMAGE. What does the resource actually say?"
|
|
26
|
+
- "The happy path wrote a record. What does the failure path do?"
|
|
27
|
+
- "This satisfies the letter of the AC. Does it satisfy the intent?"
|
|
28
|
+
- "The producer works. Does it match the contract the consumer ticket will rely on?"
|
|
29
|
+
- "The screen is guarded. What does the endpoint do when nothing guards it?"
|
|
30
|
+
- "It passes here. Does this environment even run the code I am judging?"
|
|
31
|
+
|
|
32
|
+
**You are NOT a checkbox ticker.** An AC is `PASS` only when you can point at
|
|
33
|
+
evidence. `PARTIAL`, `FAIL` and `BLOCKED` are respectable verdicts. Guessing is not.
|
|
34
|
+
|
|
35
|
+
## Why This Skill Exists
|
|
36
|
+
|
|
37
|
+
`/qa-explore` drives a browser. Backend tickets — a stream, a table, an IAM policy, a
|
|
38
|
+
search endpoint — have most of their acceptance criteria below the UI, where a browser
|
|
39
|
+
cannot see. Those ACs get waved through on "it works in the eph env", and the gaps
|
|
40
|
+
surface months later during an incident. This skill closes that lane.
|
|
41
|
+
|
|
42
|
+
Two failure modes it is built against specifically:
|
|
43
|
+
|
|
44
|
+
- **A confident `PASS` backed by a code reading looks exactly like a `PASS` backed by a
|
|
45
|
+
measurement**, and is worth far less. Every verdict here names the mode that produced it.
|
|
46
|
+
- **A client-side guard is not the endpoint's behaviour.** "The button is disabled until
|
|
47
|
+
you type" describes the browser. Every other client — mobile, integration, script —
|
|
48
|
+
sends the request the guard was preventing.
|
|
49
|
+
|
|
50
|
+
## Quick Start
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
# Verify a ticket against a target that declares a `backend:` block
|
|
54
|
+
/qa-verify-backend --target my-service-dev --context output/context/TICKET-123-context.md
|
|
55
|
+
|
|
56
|
+
# Verify from an inline ticket paste
|
|
57
|
+
/qa-verify-backend --target my-env
|
|
58
|
+
> "AC1: table X exists. AC2: stream enabled with NEW_IMAGE. AC3: write-only IAM."
|
|
59
|
+
|
|
60
|
+
# Static review only (no cloud credentials available)
|
|
61
|
+
/qa-verify-backend --target my-env --static-only
|
|
62
|
+
|
|
63
|
+
# API contract/behaviour verification, and the same matrix against a second environment
|
|
64
|
+
/qa-verify-backend --target my-service-dev --api-only --parity my-service-staging
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Session Phases
|
|
68
|
+
|
|
69
|
+
Execute in order. Read and follow the linked file.
|
|
70
|
+
|
|
71
|
+
| Phase | File | Summary |
|
|
72
|
+
|-------|------|---------|
|
|
73
|
+
| **Setup** | `phases/00-setup.md` | Resolve target, ticket context, repo branch, AWS access. Decide which lanes are runnable. |
|
|
74
|
+
| **AC Decomposition** | `phases/01-ac-decomposition.md` | Turn each AC into a falsifiable check with a named evidence source. Flag untestable ACs. |
|
|
75
|
+
| **Static Review** | `phases/02-static-review.md` | Read the branch diff against every AC. Producer/consumer contract. Spec drift. Negative space. |
|
|
76
|
+
| **Live Verification** | `phases/03-live-verification.md` | Read-only aws-cli probes of the cloud resources. One probe per AC. Capture raw evidence. |
|
|
77
|
+
| **API Verification** | `phases/03b-api-verification.md` | Fingerprint the environment, then call the endpoints directly from the authenticated session. Case matrix, cross-environment parity, UI-vs-API differential. |
|
|
78
|
+
| **End-to-End Trigger** | `phases/04-e2e-trigger.md` | Drive the real write path (UI or API), then re-probe the data layer. Covers "verified in env" ACs. |
|
|
79
|
+
| **Reporting** | `phases/05-reporting.md` | AC traceability matrix, bug reports with business impact, DoD gaps. |
|
|
80
|
+
|
|
81
|
+
## References
|
|
82
|
+
|
|
83
|
+
- **`references/aws-readonly-probes.md`** — canonical probe commands per resource type, and how to read their output
|
|
84
|
+
- **`references/api-probes.md`** — getting an authenticated request context without handling a token, the case families worth probing on every endpoint, and how to read the results
|
|
85
|
+
- **`references/environment-fingerprinting.md`** — proving which build and which implementation an environment actually runs, before any verdict is written
|
|
86
|
+
- **`references/payload-verification.md`** — recomputing derived values, saying what a single-source check does not prove, and comparing the rendered value against the payload it came from
|
|
87
|
+
- **`references/release-readiness.md`** — verifying many tickets against one build: the deployment table, the result vocabulary, carry-overs, disposition
|
|
88
|
+
- **`references/safety-rules.md`** — read-only discipline, production guardrails, redaction, confidentiality
|
|
89
|
+
|
|
90
|
+
## Knowledge Base
|
|
91
|
+
|
|
92
|
+
Load these from `data/knowledge/` before the static and live lanes. They are the
|
|
93
|
+
BE/API verification layer — techniques derived from real sessions with this skill,
|
|
94
|
+
not from published literature.
|
|
95
|
+
|
|
96
|
+
| Entry | Use it when |
|
|
97
|
+
|-------|-------------|
|
|
98
|
+
| `technique-verification-mode-selection` | **Every session, at AC decomposition.** Routes each AC to the channel that can actually falsify it — API-behind-the-screen, direct request to a no-UI endpoint, LLM output judgement, or needs-a-human. Defines when `UNVERIFIABLE` is the correct verdict. |
|
|
99
|
+
| `technique-functional-diff-analysis` | The static lane. Eight passes that read a diff for behaviour at the service boundary instead of code quality, and output falsifiable hypotheses attached to ACs. |
|
|
100
|
+
| `technique-contract-narrowing` | Any ticket that swaps a data source on a read path — "replace X with Y", "migrate to", "switch to". |
|
|
101
|
+
| `technique-test-suite-audit` | Whenever the change rewrites the tests that are meant to prove it works. |
|
|
102
|
+
| `technique-llm-output-verification` | The change touches a prompt, tool definition, model version, retrieval step, or agent orchestration. |
|
|
103
|
+
| `technique-environment-fingerprinting` | **Every session, before the first probe.** Establishes which build and which implementation the environment runs, so a verdict is about code rather than about luck. |
|
|
104
|
+
| `technique-authenticated-api-probing` | Any AC about an endpoint's own behaviour — validation, filters, pagination, error contract, payload shape. |
|
|
105
|
+
| `technique-ui-api-differential` | The ticket has both a screen and an endpoint. Sorts findings into UI-only, API-only and both. |
|
|
106
|
+
| `technique-silent-failure-audit` | Any read path with an error branch — which is all of them. Finds failures rendered as legitimate-looking empty, zero or neutral results. |
|
|
107
|
+
| `technique-derived-value-verification` | The AC concerns a calculated value — a percentage, total, ratio, delta or aggregate. |
|
|
108
|
+
| `technique-presentation-integrity` | The value is also shown on a screen. The payload can be right and the render wrong. |
|
|
109
|
+
| `technique-expected-behaviour-specification` | The session found behaviour no AC covers — which is most of what an API probe turns up. |
|
|
110
|
+
| `technique-config-surface-verification` | The change under test *is* the configuration mechanism — a flag endpoint, a settings surface, a shared config library. |
|
|
111
|
+
| `technique-release-readiness-verification` | The ask is "is this release good to go", not "does this ticket meet its ACs". |
|
|
112
|
+
|
|
113
|
+
The mode selection entry governs the others: it decides *how* an AC gets observed,
|
|
114
|
+
and the rest supply *what to look for* once the channel is chosen.
|
|
115
|
+
|
|
116
|
+
## Core Rules
|
|
117
|
+
|
|
118
|
+
1. **Evidence or it did not happen.** Every `PASS` cites a command output, a file and
|
|
119
|
+
line, or a screenshot. "Looks right" is not a verdict.
|
|
120
|
+
2. **Read-only against anything you did not create.** Probes are `describe-*`,
|
|
121
|
+
`get-*`, `list-*`, `scan`, `query`, `simulate-principal-policy`. Never mutate.
|
|
122
|
+
See `references/safety-rules.md` for the production hard stop.
|
|
123
|
+
3. **The ticket text is the spec, the code is the claim.** Where they differ, that
|
|
124
|
+
is a finding — even when the code is arguably better. Say which you think is right.
|
|
125
|
+
4. **Verify the contract, not just the resource.** If another ticket will read this
|
|
126
|
+
data, check attribute names, types and key shape against what that consumer needs.
|
|
127
|
+
Producer/consumer drift is the most expensive bug this skill can catch.
|
|
128
|
+
5. **Test the failure path.** Retries, DLQs, partial batch failures, missing claims,
|
|
129
|
+
size limits. Audit and event systems fail silently by default.
|
|
130
|
+
6. **Absence is a finding.** An AC that nothing implements, a cleanup that was written
|
|
131
|
+
but never run, a doc that still describes the deleted architecture.
|
|
132
|
+
7. **One bug = one report**, each with Business Impact, per `data/templates/bug-report.md`.
|
|
133
|
+
8. **Name the observation mode in every verdict.** `PASS (direct request, raw
|
|
134
|
+
response attached)` and `PASS (read the diff)` are different claims. A verdict
|
|
135
|
+
whose only evidence is a code reading is `UNVERIFIABLE`, not `PASS` — see
|
|
136
|
+
`technique-verification-mode-selection`.
|
|
137
|
+
9. **Fingerprint before you judge.** A verdict is a claim about code; a probe is an
|
|
138
|
+
observation of an environment. Establish which build is deployed and whether the
|
|
139
|
+
changed path is even selected here — a flag can pick between two implementations
|
|
140
|
+
inside one identical build. An environment that does not run the change gets
|
|
141
|
+
`NOT-REACHABLE`, never `PASS`.
|
|
142
|
+
10. **A guard is not a behaviour.** Never record a client-side restriction as a passing
|
|
143
|
+
AC. Call the endpoint the way the guard is preventing, then judge the answer on the
|
|
144
|
+
consequence of the call succeeding — not on how hard it was to make.
|
|
145
|
+
11. **Compare shapes, not counts.** Across environments, status codes and response shapes
|
|
146
|
+
compare; absolute numbers do not. Different data, drifting under you.
|
|
147
|
+
12. **Recompute every derived value, and say what that does not prove.** Take the formula
|
|
148
|
+
from the spec, never from the code under test. When both sides of the check come from
|
|
149
|
+
one payload you have verified the derivation, not the inputs — write that down and name
|
|
150
|
+
the independent oracle that would close it.
|
|
151
|
+
13. **Hold the payload next to the screen.** A correct response can still reach the user as
|
|
152
|
+
a wrong number, and that defect is invisible from either surface alone. When it happens,
|
|
153
|
+
attribute it to the change that owns it — usually not the one under test.
|
|
154
|
+
14. **Carry-overs get their own section.** A defect that pre-dates the change under test is
|
|
155
|
+
reported as pre-existing and tracked separately. Mixing it in inflates the apparent risk
|
|
156
|
+
of shipping and buries what actually belongs to this change.
|
|
157
|
+
15. **BLOCKED is honest.** If credentials for the account are missing, say so, leave the
|
|
158
|
+
probe commands ready to run, and do not infer the verdict from the code.
|
|
159
|
+
16. **Never widen the blast radius.** No `terraform apply`, no deploys, no writes to a
|
|
160
|
+
shared table, no destructive cleanup — not even when a runbook in the repo says to.
|
|
161
|
+
Propose it; let a human run it.
|
|
162
|
+
17. **Redact.** Account IDs beyond what the target config already holds, tokens, emails
|
|
163
|
+
of real people, and consumer PII get `[REDACTED]` before anything is written to disk.
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# Phase 0: Setup
|
|
2
|
+
|
|
3
|
+
## Step 1: Resolve Inputs
|
|
4
|
+
|
|
5
|
+
Extract from the user message:
|
|
6
|
+
- **target**: id under `data/targets/` (with a `backend:` and/or `api:` block) — or inline
|
|
7
|
+
resource names
|
|
8
|
+
- **context**: path to a gathered ticket context file, or inline ticket text
|
|
9
|
+
- **flags**: `--static-only` (skip live probes), `--api-only` (skip the cloud lane),
|
|
10
|
+
`--no-e2e` (skip phase 4), `--parity <target-id>` (run the API matrix in a second
|
|
11
|
+
environment too)
|
|
12
|
+
|
|
13
|
+
Read the target config. Read `data/domains/<domain>.md` for the domain's data-integrity
|
|
14
|
+
checks — they become extra checks beyond the ticket's own ACs.
|
|
15
|
+
|
|
16
|
+
## Step 2: Resolve the Source Branch
|
|
17
|
+
|
|
18
|
+
From `target.source`, confirm the repo and branch exist:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
git -C <repo_path> rev-parse --verify <branch>
|
|
22
|
+
git -C <repo_path> diff --stat <base_branch>...<branch>
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
If the branch is not fetched, ask before running `git fetch` — it touches the user's repo.
|
|
26
|
+
Never check the branch out; read it with `git show <branch>:<path>` so the working tree
|
|
27
|
+
is untouched.
|
|
28
|
+
|
|
29
|
+
## Step 3: Establish AWS Access (skip if `--static-only`)
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
aws sts get-caller-identity --profile "$<aws_profile_env>"
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Compare the returned account to `backend.account_id`.
|
|
36
|
+
|
|
37
|
+
| Result | Action |
|
|
38
|
+
|--------|--------|
|
|
39
|
+
| Matches | Live lane is GO |
|
|
40
|
+
| Different account | **STOP the live lane.** Report it. Do not probe an account the ticket does not concern. |
|
|
41
|
+
| No credentials | Live lane is **BLOCKED**, not failed. Continue with static + write the probe commands into the report so a human can run them. |
|
|
42
|
+
| Account looks like production | Read `references/safety-rules.md` and apply the production hard stop. |
|
|
43
|
+
|
|
44
|
+
## Step 3b: Establish the API Lane (skip if the ticket has no HTTP surface)
|
|
45
|
+
|
|
46
|
+
From `target.api`, confirm the endpoints under test and the probe allowlist. Open an
|
|
47
|
+
authenticated session once — headed, against a persistent profile, because an
|
|
48
|
+
interactive identity provider cannot complete headless:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
playwright-cli -s=<target-id> open <base_url> --headed --profile .auth/<profile>
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Confirm the session is live before building any matrix — a probe run against a logged-out
|
|
55
|
+
page returns a login page with a `200`, which reads exactly like a passing case.
|
|
56
|
+
|
|
57
|
+
If the target declares `api.parity_targets` or `--parity` was passed, prime that session
|
|
58
|
+
too. Same matrix, both environments, or the comparison is not one.
|
|
59
|
+
|
|
60
|
+
## Step 3c: Fingerprint the Environment
|
|
61
|
+
|
|
62
|
+
**Before the first probe of any lane.** Read
|
|
63
|
+
`references/environment-fingerprinting.md` and answer its three questions: which build is
|
|
64
|
+
deployed in every component of the path, whether the changed code path is *selected*
|
|
65
|
+
here, and whether the commit under test is genuinely an ancestor of what is running.
|
|
66
|
+
|
|
67
|
+
Write the answers to `evidence/fingerprint.md`. If the changed path is not reachable in
|
|
68
|
+
this environment, every AC that depends on it is `NOT-REACHABLE` — record that now, and
|
|
69
|
+
say which environment *would* answer the question.
|
|
70
|
+
|
|
71
|
+
Record the outcome. A blocked live lane changes verdicts, not effort — every AC that
|
|
72
|
+
needed a probe becomes `BLOCKED` with the exact command attached.
|
|
73
|
+
|
|
74
|
+
## Step 4: Create the Session Directory
|
|
75
|
+
|
|
76
|
+
`output/sessions/<YYYY-MM-DD-HHmm>-<ticket>-backend/` containing:
|
|
77
|
+
`charter.md`, `ac-matrix.md`, `evidence/`, `bugs/`, `probes/`, `session-log.md`
|
|
78
|
+
|
|
79
|
+
`probes/` holds the probe scripts and case files, so the run is repeatable by someone
|
|
80
|
+
else — and by you, against the next environment.
|
|
81
|
+
|
|
82
|
+
## Step 5: Write the Charter
|
|
83
|
+
|
|
84
|
+
```markdown
|
|
85
|
+
# Charter — <TICKET>
|
|
86
|
+
**Mission:** Verify <n> acceptance criteria for <ticket summary>
|
|
87
|
+
**Target:** <target id> (<environment kind>)
|
|
88
|
+
**Branch:** <branch> vs <base_branch>
|
|
89
|
+
**Lanes:** static: GO | live: GO/BLOCKED (<reason>) | api: GO/SKIPPED | e2e: GO/SKIPPED
|
|
90
|
+
**Build under test:** <component: build id / commit>, <component: build id / commit>
|
|
91
|
+
**Changed path selected here:** yes / no (<flag or config that decides it>)
|
|
92
|
+
**Out of scope:** <what the ticket defers to other tickets>
|
|
93
|
+
**Time box:** 45 min
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Append to `session-log.md` after this and every later phase:
|
|
97
|
+
`[<ts>] [PHASE] setup complete — lanes: static GO, live BLOCKED (no creds for <account>)`
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# Phase 1: AC Decomposition
|
|
2
|
+
|
|
3
|
+
Turn prose acceptance criteria into falsifiable checks **before** you look at the
|
|
4
|
+
implementation. Doing this first is what stops you from rationalising whatever the
|
|
5
|
+
code happens to do.
|
|
6
|
+
|
|
7
|
+
## For Each AC, Write
|
|
8
|
+
|
|
9
|
+
| Field | Meaning |
|
|
10
|
+
|-------|---------|
|
|
11
|
+
| **ID** | AC1, AC2… as numbered in the ticket |
|
|
12
|
+
| **Claim** | The AC restated as a single falsifiable statement |
|
|
13
|
+
| **Evidence source** | `static` (file+line), `live` (cloud probe command), `api` (request + raw response), `e2e` (trigger + re-probe), or a combination |
|
|
14
|
+
| **Falsifier** | What observation would prove this AC false — name it now |
|
|
15
|
+
| **Consumer** | Which downstream ticket/system depends on this being true in a particular shape |
|
|
16
|
+
|
|
17
|
+
Example:
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
AC4: "DynamoDB Stream (NEW_IMAGE) is enabled on the existing config table"
|
|
21
|
+
Claim: The config table's stream is enabled with view type exactly NEW_IMAGE.
|
|
22
|
+
Evidence: static → the table resource in IaC; live → describe-table StreamSpecification
|
|
23
|
+
Falsifier: StreamViewType is anything other than NEW_IMAGE, or StreamEnabled is false
|
|
24
|
+
Consumer: the stream Lambda's mapper — if OldImage is absent, delete handling changes
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Flag Untestable ACs
|
|
28
|
+
|
|
29
|
+
An AC is untestable when it has no observable falsifier. Common shapes:
|
|
30
|
+
- **Unbounded removal** — "all X are removed". Removed from where? Define the search
|
|
31
|
+
space (this repo, this module, this account) and say which you checked.
|
|
32
|
+
- **Manual data work** — "garbage entries are cleaned up". Cleaned up in which
|
|
33
|
+
environments? A merged runbook is not executed work.
|
|
34
|
+
- **Vague quality words** — "clean", "maintainable", "simple". Convert to something
|
|
35
|
+
countable or mark it as not independently verifiable.
|
|
36
|
+
- **A guard described as a behaviour** — "the user cannot submit an empty search". That is
|
|
37
|
+
a statement about the client. Restate it as the endpoint's contract ("an empty query
|
|
38
|
+
returns 400 with an error body") and route it to the `api` lane. If the ticket only ever
|
|
39
|
+
meant the client, say so and add the endpoint's behaviour as a negative-space check.
|
|
40
|
+
|
|
41
|
+
## Name the Environment in the Claim
|
|
42
|
+
|
|
43
|
+
Every claim is a claim about an environment. Write it in: *"in staging, an empty query
|
|
44
|
+
returns 400"*. An AC verified in one environment and asserted for another is the single
|
|
45
|
+
most common false `PASS` this skill exists to prevent — see
|
|
46
|
+
`references/environment-fingerprinting.md`.
|
|
47
|
+
|
|
48
|
+
Report these as **DoR gaps**, not as failures of the developer. They are ticket-writing
|
|
49
|
+
defects and they belong in the report — an untestable AC will be marked done by whoever
|
|
50
|
+
is under the least time pressure.
|
|
51
|
+
|
|
52
|
+
## Add Domain Checks
|
|
53
|
+
|
|
54
|
+
From `data/domains/<domain>.md`, pull the data-integrity checks and add any that this
|
|
55
|
+
change touches but the ticket never mentions. These become findings in the
|
|
56
|
+
**negative space** — the things the ACs forgot to require.
|
|
57
|
+
|
|
58
|
+
Write the skeleton matrix to `ac-matrix.md` with every verdict set to `PENDING`.
|