@brunosps00/dev-workflow 0.8.0 → 0.8.1

package/scaffold/skills/api-testing-recipes/references/log-conventions.md

@@ -0,0 +1,117 @@
+# Log conventions — request/response evidence as JSONL
+
+In API mode, **logs replace screenshots** as the primary QA evidence. Every request/response pair the QA suite makes is captured as one JSONL line so the bug report links back to a reproducible event.
+
+## File location
+
+`{{PRD_PATH}}/QA/logs/api/<scope>.log`
+
+Where `<scope>` is one of:
+
+- `RF-XX-[slug].log` — log for a single requirement run (1 file per RF).
+- `BUG-NN-retest.log` — log for a fix retest (1 file per bug retest cycle).
+- `run-<YYYY-MM-DD>.log` — global run log (full QA pass).
+
+## Line shape (JSONL — one JSON object per line)
+
+```json
+{
+  "ts": 1715000000000,
+  "rf": "RF-03",
+  "case": "happy-path",
+  "method": "POST",
+  "url": "http://localhost:3000/users",
+  "request_headers": {
+    "authorization": "Bearer <redacted>",
+    "content-type": "application/json"
+  },
+  "request_body": {
+    "email": "qa-1@example.com",
+    "name": "QA"
+  },
+  "status": 201,
+  "response_headers": {
+    "content-type": "application/json",
+    "location": "/users/12345"
+  },
+  "response_body": {
+    "id": "12345",
+    "email": "qa-1@example.com",
+    "name": "QA",
+    "created_at": "2026-05-06T12:00:00Z"
+  },
+  "ms": 47,
+  "verdict": "PASS",
+  "assertion_failures": []
+}
+```
+
+## Required fields
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `ts` | int (epoch ms, UTC) | When the request was sent |
+| `rf` | string | Which `RF-XX` this request belongs to (or `"BUG-NN"` for retests) |
+| `case` | string | One of `happy-path`, `validation`, `auth-missing`, `auth-expired`, `authz-cross-tenant`, `not-found`, `conflict`, `server-error`, `contract` |
+| `method` | string | HTTP method |
+| `url` | string | Full URL including query string |
+| `status` | int | HTTP status code |
+| `ms` | int | Elapsed milliseconds |
+| `verdict` | string | `"PASS"` or `"FAIL"` |
+| `assertion_failures` | array of strings | Each failed assertion as a one-line description (empty array on PASS) |
+
+## Optional fields
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `request_headers` | object | Map of header name → value |
+| `request_body` | any | Parsed JSON if `Content-Type: application/json`; raw string otherwise |
+| `response_headers` | object | Same shape as request_headers |
+| `response_body` | any | Parsed JSON if `Content-Type: application/json`; raw string otherwise |
+| `err` | string | Network/runtime error message (if no response was received at all) |
+
+## Redaction rules
+
+The log goes to `QA/logs/api/` which **may end up in artifacts uploaded to CI** or attached to bug reports. Redact:
+
+- **`Authorization` header** → `"Bearer <redacted>"` or `"Basic <redacted>"`. The token's presence is logged; the value never is.
+- **`Cookie` header** → `"<redacted>"`. Same reasoning.
+- **`X-API-Key` header** → `"<redacted>"`.
+- **Response fields named `password*`, `secret*`, `*_hash`, `token*`, `apiKey*`** → `"<redacted>"`. These should never be in a response anyway; if they are, the log redacts AND the QA report flags the leak.
+- **Free-form `request_body` fields named `password`** → `"<redacted>"`.
+
+The redaction is applied at log-write time, never on read; even a leaked log file should not expose secrets.
+
+## Why JSONL (not pretty-printed JSON)
+
+- **Append-friendly**: each request is one line; concurrent runs append safely without parsing the whole file.
+- **Greppable**: `grep '"verdict":"FAIL"' QA/logs/api/RF-03.log` shows every failed case in one shot.
+- **Queryable**: `jq -c 'select(.status >= 500)' QA/logs/api/run-*.log | jq -s 'group_by(.url) | map({url: .[0].url, count: length})'` finds the most-failing URLs.
+- **Diffable across runs**: `diff <(jq -c 'del(.ts, .ms)' RF-03.log) <(jq -c 'del(.ts, .ms)' RF-03.log.prev)` shows behavior changes free of timing noise.
+
+## Per-recipe writers
+
+Every recipe in `recipes/` includes a small writer helper in its example:
+
+- `.http` — the agent writes via `Bash` after each `curl` invocation.
+- `pytest+httpx` — `LoggingClient` subclass overriding `request`.
+- `supertest` — small `logRequest` helper imported by tests.
+- `.NET WebApplicationFactory` — `DelegatingHandler` registered on the test client.
+- `reqwest` — wrapper function around `client.execute(req)`.
+
+All of them produce the same JSONL shape so downstream tooling (the QA report renderer, the bug retest loop) doesn't care which recipe was used.
+
+## How `dw-run-qa` reads logs back
+
+When generating the QA report (Step 8 in `dw-run-qa`), the agent reads each `RF-XX-[slug].log`, computes:
+
+- **Total requests** per RF
+- **Pass count vs fail count**
+- **Failing cases** with the assertion message
+- **Tail latency** (p99 if there are ≥10 requests, max otherwise)
+
+These land in the report's "Verified Requirements" table and feed the bug entries (with `evidence_path: QA/logs/api/RF-03.log#L42` pointing to the failing line).
+
+## How `dw-fix-qa` consumes them
+
+The retest loop reads `QA/bugs.md` for each open bug, finds the corresponding log line via `evidence_path`, replays the request via the same recipe + assertions, and writes a new line to `BUG-NN-retest.log` with `verdict: "PASS"` (closing the bug) or `verdict: "FAIL"` (cycling through the fix-retest loop again, max 5 cycles).

package/scaffold/skills/api-testing-recipes/references/matrix-conventions.md

@@ -0,0 +1,68 @@
+# Matrix conventions — deriving tests from a PRD requirement
+
+Every API requirement (`RF-XX`) gets a structured matrix of test cases. The matrix is the bridge between "the PRD says this endpoint must exist" and "we have evidence it works under the cases that matter."
+
+## The five tiers
+
+For each `RF-XX`, generate at least one test per tier that applies:
+
+| Tier | Goal | When to skip |
+|------|------|--------------|
+| **200 happy path** | Prove the endpoint accepts the documented input and returns the documented output. | Never — every RF needs at least one happy path. |
+| **4xx — validation** | Prove input validation rejects malformed payloads with a useful error. | Skip only for endpoints with no body (`GET` without query params). |
+| **4xx — auth (401)** | Prove missing/expired/invalid credentials return 401. | Skip for endpoints documented as anonymous. |
+| **4xx — authorization (403)** | Prove valid credentials without the required role/scope return 403. | Skip if the endpoint is open to any authenticated user. |
+| **4xx — not found (404)** | Prove non-existent IDs return 404, not 500. | Skip for endpoints that don't take an ID. |
+| **4xx — conflict (409)** | Prove duplicates / version mismatches return 409. | Skip if the endpoint is idempotent and conflict-free by design. |
+| **5xx — server error** | Prove the system fails gracefully (no leaked stack trace, no half-write). | Skip if no synthetic failure is reproducible without invasive infrastructure changes. |
+| **Contract drift** | Prove the response shape matches the documented spec (OpenAPI, TS types, README examples). | Never — this is the cheapest way to catch silent breakage. |
+| **Authorization cross-tenant** | Prove tokens from org A cannot access data of org B. | Skip only for single-tenant systems (rare in practice). |
+
+## Why the cross-tenant test is mandatory
+
+Cross-tenant data leakage is the most damaging API bug class — it's silent (no error), undetected by happy-path tests, and lethal in B2B SaaS. Every endpoint that returns or mutates tenant-scoped data must have a cross-tenant denial test. If the project is single-tenant, mark the test `pytest.skip` / `it.skip` / `[Fact(Skip="single-tenant")]` instead of omitting — the explicit skip is a record of the decision.
+
+## How to enumerate inputs per tier
+
+For each tier, ask:
+
+- **200**: what's the minimum valid payload? Build the test around that. Add 2-3 variations only if the endpoint has interesting branching (nullable fields, enum variants, optional sections).
+- **4xx validation**: what fields are required? Drop each one. What types are constrained? Send the wrong type. What ranges? Test min-1 and max+1. Don't test all combinations — one per kind of constraint is enough.
+- **4xx auth**: 3 variants — no token, expired token, malformed token. One test for each is enough.
+- **4xx authorization**: identify role boundaries (admin vs user vs guest, owner vs member). One test per boundary.
+- **4xx not found**: 1 test with a syntactically-valid-but-nonexistent ID (UUID, integer, etc.).
+- **4xx conflict**: 1 test that triggers the documented conflict (duplicate email, race on version).
+- **5xx**: skip if not reproducible. If the project has a way to inject failures (chaos hooks, dev-only error endpoints), use them.
+- **Contract drift**: 1 test that asserts every documented field is present AND no leaked internal field is.
+- **Cross-tenant**: 1 test per tenant-scoped endpoint with a token from a different tenant.
+
+## Example expansion: `POST /users`
+
+PRD says: "RF-03 — admins can create users. Validation: email is required and must be unique. Returns 201 with the new user."
+
+Matrix:
+
+| # | Tier | Case | Expected |
+|---|------|------|----------|
+| 1 | 200 | admin creates user with valid payload | 201, body has id |
+| 2 | 4xx validation | missing email | 422, error mentions email |
+| 3 | 4xx validation | invalid email format | 422 |
+| 4 | 4xx auth | no token | 401 |
+| 5 | 4xx auth | expired token | 401 |
+| 6 | 4xx authorization | regular user (not admin) | 403 |
+| 7 | 4xx conflict | email already taken | 409 |
+| 8 | Contract | all required fields present, no `password_hash` | matches spec |
+| 9 | Cross-tenant | admin from another org tries to fetch this user | 403 or 404 |
+
+That's 9 test cases for one RF — the floor for a real API surface, not the ceiling.
+
+## What NOT to do
+
+- **Don't test every combination** of validation failures. The framework already enforces type + presence; one test per kind of constraint is the signal.
+- **Don't test the framework**. `Content-Type: application/json` parsing, default routing, etc. — those belong to FastAPI / Fastify / ASP.NET, not to your QA suite.
+- **Don't write tests for endpoints with no PRD reference**. If a route exists but no RF describes it, that's a documentation gap to flag, not a test to add.
+- **Don't skip 5xx because "it shouldn't happen"**. If you have a way to reproduce, do it. If you genuinely can't, document the skip in the QA report so the gap is visible.
+
+## How `dw-run-qa` uses this
+
+When in API mode, `/dw-run-qa` walks each `RF-XX` in the PRD, runs through this matrix, and emits PASS/FAIL per RF — not per test case. A single FAIL in any tier marks the RF as FAIL and lands a `BUG-NN` entry pointing to the failing log line.

package/scaffold/skills/api-testing-recipes/references/openapi-driven.md

@@ -0,0 +1,97 @@
+# OpenAPI-driven mode — generating tests from a spec
+
+When the project exposes an OpenAPI spec (static `openapi.yaml`/`openapi.json`, or dynamic `/openapi.json` for FastAPI), `/dw-run-qa` can derive a baseline test suite directly from it. This catches contract drift between code and spec for free.
+
+## When to use this mode
+
+- The project already maintains an OpenAPI spec — either hand-written, generated from code annotations (FastAPI, NestJS + `@nestjs/swagger`, dotnet Swashbuckle), or synced via a code generator.
+- You want a quick "is this endpoint reachable AND does its response shape match the spec?" check.
+- You want to detect when code drifts ahead of (or behind) the spec.
+
+## What it generates
+
+For each path × method in the spec:
+
+1. A **happy-path test** using the spec's `requestBody` example (or schema-derived sample).
+2. A **contract-shape test** asserting the response matches the documented schema.
+3. Skips paths/methods marked with the `x-internal: true` extension or those without examples.
+
+The generated tests live alongside hand-written ones in `{{PRD_PATH}}/QA/scripts/api/`. Filename pattern: `openapi-RF-XX-[path-slug].http` (or stack-specific extension).
+
+## How to run it
+
+`/dw-run-qa --from-openapi <spec-path-or-url>` — explicit. The `<spec-path-or-url>` can be:
+
+- `./openapi.yaml`
+- `http://localhost:3000/openapi.json` (FastAPI default)
+- `http://localhost:3000/swagger/v1/swagger.json` (ASP.NET Core default)
+
+Without the flag, `/dw-run-qa` auto-detects:
+
+- File at repo root: `openapi.yaml`, `openapi.json`, `swagger.yaml`, `swagger.json`.
+- Project running locally: `GET /openapi.json`, `GET /swagger/v1/swagger.json`, `GET /api-docs`.
+
+If found, the agent asks: "OpenAPI spec detected at `<path>`. Generate baseline tests from it? [y/n]". On `y`, the baseline is added on top of the PRD-derived matrix.
+
+## Mapping spec endpoints to RFs
+
+The PRD names requirements (`RF-01`, `RF-02`); the spec names paths (`/users`, `/orders/{id}`). Two conventions for cross-referencing:
+
+- **By tag**: tests for a path tagged `users` map to the PRD section also tagged `users`. Cleanest if the project keeps tags consistent.
+- **By summary keyword**: the spec's `summary` field is matched against PRD requirement titles. Less reliable; only use as a fallback.
+
+If neither matches, the test lands as `openapi-no-rf-[slug].http` and the QA report flags it as "spec endpoint not mapped to any RF — possible documentation gap."
+
+## Contract drift detection
+
+For each response from a generated test, compare:
+
+1. **Status code** — does the actual status appear in the spec's `responses` block?
+2. **Required fields** — every field marked `required: true` in the schema must be present.
+3. **Type matching** — `email: string` in spec, but actual is `email: null`? Fail.
+4. **No leaked fields** — fields NOT in the spec but present in the response are flagged as **drift forward** (code added a field; spec is stale).
+5. **Sensitive defaults** — fields named `password*`, `secret*`, `token*`, `*_hash` in the response trigger an immediate FAIL with severity HIGH, even if they're "documented."
+
+## Generating example payloads
+
+If the spec has `example` or `examples`, use them verbatim. If only schemas exist, sample using a deterministic strategy:
+
+| JSON Schema type | Sample value |
+|-------|-------|
+| `string` | `"qa-string"` (or `"qa@example.com"` if `format: email`, ISO date if `format: date-time`, UUID v4 if `format: uuid`) |
+| `integer` | `1` (or value within `minimum`/`maximum` if set) |
+| `number` | `1.0` |
+| `boolean` | `true` |
+| `array` | one element of the inner type |
+| `object` | recurse on `properties`; only include `required` fields |
+| `enum` | first value |
+| `oneOf`/`anyOf` | first variant |
+
+Skip endpoints whose request shape can't be sampled deterministically (e.g., free-form JSON without schema, file uploads requiring real binary data).
+
+## What NOT to do
+
+- **Don't replace the PRD-derived matrix with OpenAPI-only tests.** OpenAPI tells you what the code claims to do; the PRD tells you what the product needs. Both matter. Keep both.
+- **Don't trust the spec implicitly.** If `dw-run-qa` finds 0 drift on day 1 and the team has been shipping for 6 months, the spec is probably stale, not the code. Flag the suspicion in the QA report.
+- **Don't generate tests for `x-internal: true` endpoints.** Those are behind an internal-network boundary; QA on them needs different credentials and risk profile.
+
+## Limitations
+
+- Doesn't generate authorization tests automatically (the spec doesn't say "this endpoint should reject other-tenant tokens"). Hand-write those per the cross-tenant pattern in `matrix-conventions.md`.
+- Doesn't generate state-mutating sequences (create → update → delete). Those need PRD context to know what state matters.
+- Treats the spec as authoritative for contract drift, but not for behavior. A spec that's wrong is still going to fail tests against correct code — and that's the right outcome. Update the spec.
+
+## What `dw-run-qa` produces
+
+When OpenAPI mode runs, the QA report gains a section:
+
+```markdown
+## OpenAPI baseline
+
+- Spec source: openapi.yaml (53 paths, 121 operations)
+- Endpoints sampled: 89 (32 skipped: missing examples, file uploads, `x-internal`)
+- Drift detected: 4 endpoints (see RF-12, RF-15, RF-22, openapi-no-rf-internal-metrics)
+- Contract issues:
+  - RF-12 — `email` documented as required, response returns null
+  - openapi-no-rf-internal-metrics — endpoint exists in spec but no PRD reference
+```