@llipe.com/dev-tasks 0.10.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/developer.md +11 -0
- package/.claude/agents/qa-engineer.md +31 -2
- package/.claude/agents/researcher.md +96 -0
- package/.claude/commands/planner.md +22 -7
- package/.claude/commands/product-engineer.md +37 -10
- package/.claude/commands/researcher.md +28 -0
- package/.claude/skills/activity-codebase-research/SKILL.md +158 -0
- package/.claude/skills/activity-contract-validation/SKILL.md +167 -0
- package/.claude/skills/activity-e2e-test-implementation/SKILL.md +156 -0
- package/.claude/skills/activity-integration-test-implementation/SKILL.md +156 -0
- package/.claude/skills/activity-test-standards/SKILL.md +41 -0
- package/.github/agents/developer.agent.md +11 -0
- package/.github/agents/planner.agent.md +22 -7
- package/.github/agents/product-engineer.agent.md +37 -10
- package/.github/agents/qa-engineer.agent.md +30 -3
- package/.github/agents/researcher.agent.md +96 -0
- package/.github/prompts/researcher.prompt.md +29 -0
- package/.github/skills/activity-codebase-research/SKILL.md +158 -0
- package/.github/skills/activity-contract-validation/SKILL.md +167 -0
- package/.github/skills/activity-e2e-test-implementation/SKILL.md +156 -0
- package/.github/skills/activity-integration-test-implementation/SKILL.md +156 -0
- package/.github/skills/activity-test-standards/SKILL.md +41 -0
- package/.kiro/agents/developer.md +11 -0
- package/.kiro/agents/planner.md +22 -7
- package/.kiro/agents/product-engineer.md +37 -10
- package/.kiro/agents/qa-engineer.md +30 -3
- package/.kiro/agents/researcher.md +99 -0
- package/.kiro/agents/verifier.md +1 -1
- package/.kiro/skills/activity-codebase-research/SKILL.md +158 -0
- package/.kiro/skills/activity-contract-validation/SKILL.md +167 -0
- package/.kiro/skills/activity-e2e-test-implementation/SKILL.md +156 -0
- package/.kiro/skills/activity-integration-test-implementation/SKILL.md +156 -0
- package/.kiro/skills/activity-test-standards/SKILL.md +41 -0
- package/AGENTS.md +91 -215
- package/CHANGELOG.md +33 -0
- package/CLAUDE.md +3 -2
- package/README.md +1 -0
- package/TESTING.md +19 -11
- package/package.json +1 -1
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: activity-contract-validation
|
|
3
|
+
description: "Validate API contracts against implementation using dt verify (contract-diff, impact, drift). Detects breaking changes, consumer impact, and spec-to-code staleness. Use when checking API boundary integrity."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Activity: Contract Validation
|
|
7
|
+
|
|
8
|
+
Detect API spec drift, breaking changes, and consumer impact using the `dt verify` family. This skill wires existing extraction and verification tooling into the QA path.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
> **RFC 2119 Notice:** The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **SHOULD**, **SHOULD NOT**, **RECOMMENDED**, **MAY**, and **OPTIONAL** in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
|
|
13
|
+
|
|
14
|
+
## Goal
|
|
15
|
+
|
|
16
|
+
Detect misalignment between API specifications and implementation before it reaches consumers. Contract validation checks the boundary — it does not test internal behavior.
|
|
17
|
+
|
|
18
|
+
Consult `/TESTING.md` first. Its Contract Validation layer boundary governs.
|
|
19
|
+
|
|
20
|
+
## Contract Validation Boundary
|
|
21
|
+
|
|
22
|
+
- **MUST** validate the API boundary (endpoints, schemas, events) against the specification.
|
|
23
|
+
- **MUST NOT** test internal business logic — that belongs at Layer 1 or 2.
|
|
24
|
+
- **MUST NOT** require a running server — operates on static specs and source analysis.
|
|
25
|
+
- **MAY** complement live testing but is not a substitute for E2E.
|
|
26
|
+
|
|
27
|
+
## Part 1 — Prerequisite Detection
|
|
28
|
+
|
|
29
|
+
### Check `dt` availability
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
which dt && dt --version
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
- If `dt` is not found: emit `SKIPPED(dt not installed)` and provide manual instructions.
|
|
36
|
+
- If `dt` is found but not configured for the repo: emit `SKIPPED(dt not configured — run dt init)`.
|
|
37
|
+
|
|
38
|
+
### Detect API specifications
|
|
39
|
+
|
|
40
|
+
Search the repository for:
|
|
41
|
+
|
|
42
|
+
- `openapi.yaml`, `openapi.json`, `swagger.yaml`, `swagger.json` (or under `docs/`, `api/`, `specs/`)
|
|
43
|
+
- `asyncapi.yaml`, `asyncapi.json`
|
|
44
|
+
- `component.json` (dt-managed extraction output)
|
|
45
|
+
|
|
46
|
+
If no specs found: emit `SKIPPED(no OpenAPI or AsyncAPI spec found in repository)`.
|
|
47
|
+
|
|
48
|
+
## Part 2 — Contract Diff (Breaking Change Detection)
|
|
49
|
+
|
|
50
|
+
Run against the base branch to detect breaking changes introduced by the current work:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
dt verify contract-diff --base origin/main --head HEAD
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### Findings classification
|
|
57
|
+
|
|
58
|
+
| Exit code | Meaning | Severity |
|
|
59
|
+
| --------- | ------------------------- | -------- |
|
|
60
|
+
| 0 | No breaking changes | — |
|
|
61
|
+
| 8 | Breaking changes detected | Critical |
|
|
62
|
+
| 1 | Execution error | Blocked |
|
|
63
|
+
|
|
64
|
+
### Report format
|
|
65
|
+
|
|
66
|
+
For each breaking change found:
|
|
67
|
+
|
|
68
|
+
```markdown
|
|
69
|
+
| Finding | Severity | Path | Description |
|
|
70
|
+
| ------- | -------- | ------------------------------- | ------------------------------------------ |
|
|
71
|
+
| BREAK-1 | critical | POST /api/orders → request body | Required field `customerId` removed |
|
|
72
|
+
| BREAK-2 | major | GET /api/users → response | Field `email` type changed string → object |
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Part 3 — Impact Analysis (Consumer Enumeration)
|
|
76
|
+
|
|
77
|
+
When breaking changes are detected, identify affected consumers:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
dt verify impact --contract <contract-id>
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Report:
|
|
84
|
+
|
|
85
|
+
- List of consumers that depend on the changed contract.
|
|
86
|
+
- Per consumer: which endpoints/events they consume.
|
|
87
|
+
- Recommendation: notify consumers, or validate with consumer-driven tests.
|
|
88
|
+
|
|
89
|
+
## Part 4 — Drift Detection (Spec-to-Implementation Staleness)
|
|
90
|
+
|
|
91
|
+
Check whether the specification and implementation have diverged:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
dt verify drift [--id <component-id>]
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
### Drift types
|
|
98
|
+
|
|
99
|
+
| Type | Description | Severity |
|
|
100
|
+
| --------------- | ----------------------------------------------- | -------- |
|
|
101
|
+
| Spec-ahead | Spec documents endpoints not in code | Major |
|
|
102
|
+
| Code-ahead | Code has endpoints not in spec | Major |
|
|
103
|
+
| Stale | Spec not updated in >90 days while code changed | Minor |
|
|
104
|
+
| Schema-mismatch | Response shape differs from spec | Critical |
|
|
105
|
+
|
|
106
|
+
### Report format
|
|
107
|
+
|
|
108
|
+
Same risk-ranked table format as `activity-coverage-gap-analysis`:
|
|
109
|
+
|
|
110
|
+
```markdown
|
|
111
|
+
| Finding | Severity | Component | Description | Days since spec update |
|
|
112
|
+
| ------- | -------- | ------------ | ---------------------------------------------- | ---------------------- |
|
|
113
|
+
| DRIFT-1 | critical | api-gateway | Response schema mismatch: /api/orders | — |
|
|
114
|
+
| DRIFT-2 | major | api-gateway | 12 endpoints in code, 8 in spec | — |
|
|
115
|
+
| DRIFT-3 | minor | auth-service | Spec unchanged 120 days, 14 code commits since | 120 |
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
## Part 5 — Integration with Coverage Gate
|
|
119
|
+
|
|
120
|
+
Contract validation findings **MUST** be reported alongside coverage in the `coverage_gate` path:
|
|
121
|
+
|
|
122
|
+
- Breaking changes → block completion (critical findings).
|
|
123
|
+
- Drift findings → report but do not block (route to follow-up or drift-reconciliation).
|
|
124
|
+
- No spec found → skip with reason (does not count as a gap).
|
|
125
|
+
|
|
126
|
+
### Aggregated output
|
|
127
|
+
|
|
128
|
+
```markdown
|
|
129
|
+
## Contract Validation Summary
|
|
130
|
+
|
|
131
|
+
- **Spec detected:** OpenAPI 3.1 at `api/openapi.yaml`
|
|
132
|
+
- **Breaking changes:** 0 | **Drift findings:** 2 | **Impact:** N/A
|
|
133
|
+
- **Status:** PASS (no breaking changes)
|
|
134
|
+
|
|
135
|
+
### Drift Findings (non-blocking)
|
|
136
|
+
|
|
137
|
+
| ... |
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
## Fallback — Manual Validation
|
|
141
|
+
|
|
142
|
+
When `dt` is not installed, provide these manual steps:
|
|
143
|
+
|
|
144
|
+
1. **Diff specs manually:** compare `api/openapi.yaml` across branches.
|
|
145
|
+
2. **Check for undocumented endpoints:** search route handlers not in the spec.
|
|
146
|
+
3. **Schema validation:** run `npx @redocly/cli lint api/openapi.yaml` or equivalent.
|
|
147
|
+
4. **Consumer notification:** manually identify consumers from project documentation.
|
|
148
|
+
|
|
149
|
+
## Output
|
|
150
|
+
|
|
151
|
+
- `dt` availability status
|
|
152
|
+
- Spec detection result (type, path, version)
|
|
153
|
+
- Breaking changes (count, details, severity)
|
|
154
|
+
- Consumer impact (when applicable)
|
|
155
|
+
- Drift findings (count, details, severity, days since update)
|
|
156
|
+
- Aggregated status for `coverage_gate` integration
|
|
157
|
+
- Manual fallback instructions (when `dt` unavailable)
|
|
158
|
+
|
|
159
|
+
## Final Instructions
|
|
160
|
+
|
|
161
|
+
1. You **MUST** check `dt` availability before attempting verification.
|
|
162
|
+
2. You **MUST** emit `SKIPPED` with a reason when prerequisites are missing.
|
|
163
|
+
3. You **MUST** classify findings by severity using the tables above.
|
|
164
|
+
4. You **MUST** report breaking changes as critical findings that block completion.
|
|
165
|
+
5. You **MUST** report drift as non-blocking findings for follow-up.
|
|
166
|
+
6. You **MUST NOT** test internal business logic — contract validation is boundary-only.
|
|
167
|
+
7. You **MUST NOT** install `dt` or any dependency — detect and recommend only.
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: activity-e2e-test-implementation
|
|
3
|
+
description: "Author Playwright end-to-end tests from verifier scenario tables. Covers auth, state reset, CI config, and scenario-to-spec traceability. Use when writing or backfilling E2E tests."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Activity: E2E Test Implementation (Playwright)
|
|
7
|
+
|
|
8
|
+
Convert `verifier` Design Mode scenario tables into executable Playwright specs. This skill is the execution counterpart to `activity-e2e-test-design`, which produces the scenario tables.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
> **RFC 2119 Notice:** The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **SHOULD**, **SHOULD NOT**, **RECOMMENDED**, **MAY**, and **OPTIONAL** in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
|
|
13
|
+
|
|
14
|
+
## Goal
|
|
15
|
+
|
|
16
|
+
Produce Playwright E2E specs that fail when user-facing behavior is wrong. E2E tests assert on observable outcomes — never on internal state, database rows, or implementation details.
|
|
17
|
+
|
|
18
|
+
Consult `/TESTING.md` first. Its E2E layer boundary governs; this skill supplies the method.
|
|
19
|
+
|
|
20
|
+
## E2E Layer Boundary
|
|
21
|
+
|
|
22
|
+
- **MUST** assert on user-visible behavior (page content, navigation, HTTP responses).
|
|
23
|
+
- **MUST NOT** assert on internal state, database contents, or implementation details.
|
|
24
|
+
- **MUST NOT** mock the application under test. The full stack runs.
|
|
25
|
+
- **MAY** seed test data via API or database before the test begins.
|
|
26
|
+
|
|
27
|
+
## Part 1 — Playwright Prerequisite Contract
|
|
28
|
+
|
|
29
|
+
Before authoring specs, verify or establish these prerequisites:
|
|
30
|
+
|
|
31
|
+
### Authentication strategy
|
|
32
|
+
|
|
33
|
+
| Strategy | When to Use | How |
|
|
34
|
+
| ---------------------------------------------- | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
|
|
35
|
+
| **`storageState` via setup project** (default) | Multi-test suites sharing a logged-in session | A `global-setup.ts` or setup project authenticates once, saves `storageState.json`, and all specs reuse it. |
|
|
36
|
+
| **Per-test programmatic login** | Tests that need different users or roles | Each spec calls login API directly in `beforeEach`. Slower but flexible. |
|
|
37
|
+
| **API-token seeding** | API-only E2E, no browser UI | Set auth headers directly via `request.newContext()`. No browser needed. |
|
|
38
|
+
|
|
39
|
+
Default to `storageState` unless the test requires a different user or role per test.
|
|
40
|
+
|
|
41
|
+
### Seeded test users
|
|
42
|
+
|
|
43
|
+
- Create dedicated test users (not production accounts).
|
|
44
|
+
- Credentials reach CI via environment variables or secrets — never hardcoded in the repository.
|
|
45
|
+
- Use a deterministic naming convention: `e2e-user-admin@test.local`, `e2e-user-member@test.local`.
|
|
46
|
+
- Document how to create/reset test users in the project's test setup README.
|
|
47
|
+
|
|
48
|
+
### Base URL and environment resolution
|
|
49
|
+
|
|
50
|
+
- Use `baseURL` in `playwright.config.ts`, resolved from `process.env.BASE_URL` or `process.env.PLAYWRIGHT_BASE_URL`.
|
|
51
|
+
- Never hardcode `localhost:3000` — CI and staging use different URLs.
|
|
52
|
+
- Provide a `.env.test.example` showing required variables.
|
|
53
|
+
|
|
54
|
+
### Database state reset between runs
|
|
55
|
+
|
|
56
|
+
Choose one:
|
|
57
|
+
|
|
58
|
+
- **Migration + seed** (preferred): drop, migrate, seed before the test suite.
|
|
59
|
+
- **Transaction rollback**: wrap each test in a transaction (complex for E2E).
|
|
60
|
+
- **Snapshot restore**: restore a known-good database snapshot before runs.
|
|
61
|
+
|
|
62
|
+
State reset **MUST** run in `globalSetup` or a setup project, not inside individual specs.
|
|
63
|
+
|
|
64
|
+
### Trace, screenshot, and video retention
|
|
65
|
+
|
|
66
|
+
- **Default:** `on-first-retry` for traces, `only-on-failure` for screenshots, `off` for video.
|
|
67
|
+
- **CI:** retain artifacts as test attachments. Configure `outputDir` and artifact upload in CI workflow.
|
|
68
|
+
- Keep retention bounded — do not retain video for every passing test.
|
|
69
|
+
|
|
70
|
+
### Browser install in CI
|
|
71
|
+
|
|
72
|
+
- CI **MUST** install browsers explicitly: `npx playwright install --with-deps chromium`.
|
|
73
|
+
- Locally: `npx playwright install` on first run.
|
|
74
|
+
- Pin Playwright version in `package.json` — do not use `latest`.
|
|
75
|
+
|
|
76
|
+
### Sharding (large suites)
|
|
77
|
+
|
|
78
|
+
- For suites >50 specs: shard across CI workers using `--shard=1/4`, `--shard=2/4`, etc.
|
|
79
|
+
- Configure in CI matrix, not in `playwright.config.ts`.
|
|
80
|
+
|
|
81
|
+
## Part 2 — Scenario-to-Spec Mapping
|
|
82
|
+
|
|
83
|
+
### Traceability convention
|
|
84
|
+
|
|
85
|
+
Every Playwright spec **MUST** be traceable to a `verifier` Design Mode scenario.
|
|
86
|
+
|
|
87
|
+
**Annotation format:** Include `@scenario SC-{n}` in the test title or as a comment:
|
|
88
|
+
|
|
89
|
+
```typescript
|
|
90
|
+
// @scenario SC-1
|
|
91
|
+
test("SC-1: Successful login with valid credentials", async ({ page }) => {
|
|
92
|
+
// ...
|
|
93
|
+
});
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Or in the test title directly:
|
|
97
|
+
|
|
98
|
+
```typescript
|
|
99
|
+
test("[SC-1] User logs in with valid credentials", async ({ page }) => {
|
|
100
|
+
// ...
|
|
101
|
+
});
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Traceability chain
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
AC-{n} → SC-{n} (test plan) → .spec.ts file / test block (@scenario SC-{n}) → Pass/Fail
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
The `verifier` audit resolves scenario IDs to spec files by searching for `@scenario SC-{n}` or `SC-{n}:` patterns. Orphaned scenarios (in the plan but not in specs) are reported as uncovered.
|
|
111
|
+
|
|
112
|
+
### File naming
|
|
113
|
+
|
|
114
|
+
- One spec file per feature or user journey: `login.spec.ts`, `checkout.spec.ts`.
|
|
115
|
+
- Group related scenarios in one file. Do not create one file per scenario unless they are independent flows.
|
|
116
|
+
|
|
117
|
+
## Part 3 — Converting Scenario Tables to Specs
|
|
118
|
+
|
|
119
|
+
Given a scenario table from `activity-e2e-test-design`:
|
|
120
|
+
|
|
121
|
+
1. **Read the scenario.** Extract: preconditions, steps, expected result, pass criteria.
|
|
122
|
+
2. **Write the spec.** Map steps to Playwright actions. Map pass criteria to assertions.
|
|
123
|
+
3. **Annotate.** Add `@scenario SC-{n}` to the test.
|
|
124
|
+
4. **Handle preconditions.** Seed required state in `beforeAll`/`beforeEach` or a fixture.
|
|
125
|
+
5. **Assert observable outcomes.** Use `expect(page)` or `expect(response)` — never internal state.
|
|
126
|
+
|
|
127
|
+
### Retry and flakiness
|
|
128
|
+
|
|
129
|
+
- Use Playwright's built-in retry (`retries: 2` in config) for CI.
|
|
130
|
+
- Prefer explicit `await expect(locator).toBeVisible()` over arbitrary `waitForTimeout`.
|
|
131
|
+
- If a test is flaky, fix the root cause (race condition, state leak) rather than increasing retries.
|
|
132
|
+
- A flaky test that occasionally passes is worse than no test.
|
|
133
|
+
|
|
134
|
+
### Test isolation
|
|
135
|
+
|
|
136
|
+
- Each spec **MUST** be independent — runnable in any order, in parallel.
|
|
137
|
+
- Never rely on state from a previous test. Use fixtures or setup projects.
|
|
138
|
+
- Use `test.describe.configure({ mode: 'serial' })` only when tests represent a true sequential flow.
|
|
139
|
+
|
|
140
|
+
## Output
|
|
141
|
+
|
|
142
|
+
- Specs written or updated, by path
|
|
143
|
+
- Scenario-to-spec mapping table: `SC-{n}` → file path and test name
|
|
144
|
+
- Uncovered scenarios (in the plan but not yet implemented)
|
|
145
|
+
- Prerequisites missing (base URL, auth setup, browser install, etc.)
|
|
146
|
+
- Stated limitations (e.g., "no staging environment — E2E limited to localhost")
|
|
147
|
+
|
|
148
|
+
## Final Instructions
|
|
149
|
+
|
|
150
|
+
1. You **MUST** consult `/TESTING.md` and respect its E2E layer boundary.
|
|
151
|
+
2. You **MUST** annotate every spec with its scenario ID (`@scenario SC-{n}`).
|
|
152
|
+
3. You **MUST NOT** assert on internal state or implementation details.
|
|
153
|
+
4. You **MUST** verify Playwright prerequisites before authoring specs.
|
|
154
|
+
5. You **MUST** report uncovered scenarios from the test plan.
|
|
155
|
+
6. You **MUST** address flakiness causes, not mask them with retries.
|
|
156
|
+
7. You **MUST NOT** install dependencies — detect and recommend only.
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: activity-integration-test-implementation
|
|
3
|
+
description: "Author integration tests (Layer 2.5) against real databases — local via testcontainers/docker-compose/Supabase CLI, remote via testing environments. Covers migrations, RLS, pgTAP, fixtures, and fallback paths. Use when writing or backfilling integration tests."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Activity: Integration Test Implementation (Layer 2.5)
|
|
7
|
+
|
|
8
|
+
Author integration tests that exercise code against a real database. Layer 2.5 exists because mocking the data layer makes SQL errors, type mismatches, NULL handling, malformed joins, and RLS policies structurally invisible.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
> **RFC 2119 Notice:** The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **SHOULD**, **SHOULD NOT**, **RECOMMENDED**, **MAY**, and **OPTIONAL** in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
|
|
13
|
+
|
|
14
|
+
## Goal
|
|
15
|
+
|
|
16
|
+
Produce integration tests that fail when the data layer is wrong. If the database is mocked, the test belongs at Layer 2, not here.
|
|
17
|
+
|
|
18
|
+
Consult `/TESTING.md` first. Its Layer 2.5 boundary governs; this skill supplies the method.
|
|
19
|
+
|
|
20
|
+
## Layer 2.5 Boundary
|
|
21
|
+
|
|
22
|
+
- **MUST** use a real database (Postgres, SQLite, etc.) — not a mock or in-memory fake.
|
|
23
|
+
- **MUST NOT** mock the data layer. If you mock `sql`, `prisma`, `drizzle`, or the ORM client, the test is Layer 2.
|
|
24
|
+
- **MUST NOT** assert on SQL text or internal query shape. Assert on persisted state or returned data.
|
|
25
|
+
- **MAY** mock third-party external services (payment gateways, email providers) at the HTTP boundary.
|
|
26
|
+
|
|
27
|
+
## Part 1 — Local Integration Environment Detection
|
|
28
|
+
|
|
29
|
+
Detect the easiest applicable local option. Prefer in this order:
|
|
30
|
+
|
|
31
|
+
### 1. Testcontainers (preferred for isolation)
|
|
32
|
+
|
|
33
|
+
- Look for `@testcontainers/postgresql` or equivalent in `package.json`/`pom.xml`/`Cargo.toml`.
|
|
34
|
+
- Requires Docker. Check: `docker info` succeeds.
|
|
35
|
+
- Each test suite gets an ephemeral container — full isolation, no cleanup conflicts.
|
|
36
|
+
|
|
37
|
+
### 2. Docker Compose
|
|
38
|
+
|
|
39
|
+
- Look for `docker-compose.yml` or `compose.yml` with a database service.
|
|
40
|
+
- Requires Docker. Shared instance across tests — needs explicit state reset.
|
|
41
|
+
|
|
42
|
+
### 3. Supabase Local CLI
|
|
43
|
+
|
|
44
|
+
- Look for `supabase/config.toml` or `.supabase/` directory.
|
|
45
|
+
- Check: `supabase status` succeeds.
|
|
46
|
+
- Provides real Postgres with auth, storage, and edge functions locally.
|
|
47
|
+
|
|
48
|
+
### 4. Fallback — No Local Environment
|
|
49
|
+
|
|
50
|
+
- If Docker is unavailable and Supabase CLI is not installed, report the limitation.
|
|
51
|
+
- Emit: `SKIPPED(no local database environment — Docker unavailable, Supabase CLI not installed)`.
|
|
52
|
+
- Recommend environment setup as a follow-up. Do not silently omit.
|
|
53
|
+
|
|
54
|
+
## Part 2 — Fixtures, Seeding, and Lifecycle
|
|
55
|
+
|
|
56
|
+
### Test isolation
|
|
57
|
+
|
|
58
|
+
- Each test **SHOULD** run in a transaction that rolls back on completion (preferred).
|
|
59
|
+
- Alternative: truncate affected tables in `beforeEach`/`afterEach`.
|
|
60
|
+
- Alternative for testcontainers: fresh container per suite (slower but fully isolated).
|
|
61
|
+
|
|
62
|
+
### Seeding
|
|
63
|
+
|
|
64
|
+
- Shared seed data lives in one location per package (e.g., `test/fixtures/seed.sql` or `test/helpers/seed.ts`).
|
|
65
|
+
- Seeds **MUST** be idempotent — running twice produces the same state.
|
|
66
|
+
- Seeds **MUST NOT** depend on auto-increment IDs. Use deterministic UUIDs or named references.
|
|
67
|
+
|
|
68
|
+
### Deterministic teardown
|
|
69
|
+
|
|
70
|
+
- Every integration test file **MUST** clean up after itself or use rollback isolation.
|
|
71
|
+
- Tests **MUST NOT** depend on execution order.
|
|
72
|
+
- Tests **MUST NOT** leave residual data that affects other test files.
|
|
73
|
+
|
|
74
|
+
## Part 3 — Migration Clean-Apply Tests
|
|
75
|
+
|
|
76
|
+
- **MUST** test that all migrations apply cleanly from an empty database.
|
|
77
|
+
- **MUST** verify apply order (no duplicate prefixes, no gaps).
|
|
78
|
+
- **MUST** verify idempotency: running migrations twice does not error.
|
|
79
|
+
- For large stacks (50+ migrations): run full clean-apply on CI/scheduled; run latest N on PR.
|
|
80
|
+
- Report duplicate migration prefixes as a finding.
|
|
81
|
+
|
|
82
|
+
## Part 4 — Row-Level Security (RLS) Policy Tests
|
|
83
|
+
|
|
84
|
+
RLS tests **MUST** exercise the real database policy, not application-level tenant checks in JavaScript.
|
|
85
|
+
|
|
86
|
+
### Multi-session setup
|
|
87
|
+
|
|
88
|
+
- Create at least two database roles or authenticated sessions (Tenant A, Tenant B).
|
|
89
|
+
- Use `SET ROLE` or separate connections per tenant.
|
|
90
|
+
- For Supabase: use `supabase.auth.admin.createUser()` for test users, then authenticate with their JWTs.
|
|
91
|
+
|
|
92
|
+
### Required assertions
|
|
93
|
+
|
|
94
|
+
| Case | Assertion |
|
|
95
|
+
| ---------------------------- | ----------------------------------- |
|
|
96
|
+
| Tenant A reads own data | Returns expected rows |
|
|
97
|
+
| Tenant A reads Tenant B data | Returns empty set or error |
|
|
98
|
+
| Tenant A writes to Tenant B | Insert/update fails or is invisible |
|
|
99
|
+
| Unauthenticated access | Denied by policy |
|
|
100
|
+
| Service role bypasses RLS | Returns all rows (when expected) |
|
|
101
|
+
|
|
102
|
+
### Record limitations
|
|
103
|
+
|
|
104
|
+
If isolation is asserted against application-level checks rather than the real RLS policy, you **MUST** record: "Production RLS policy unverified — tested against application-layer tenant filtering only."
|
|
105
|
+
|
|
106
|
+
## Part 5 — pgTAP / Schema Contract Tests
|
|
107
|
+
|
|
108
|
+
Use pgTAP (or equivalent: `pg_prove`, raw SQL assertions) for:
|
|
109
|
+
|
|
110
|
+
- Table and column existence.
|
|
111
|
+
- Constraint verification (NOT NULL, UNIQUE, CHECK, FK).
|
|
112
|
+
- Function and trigger existence and signatures.
|
|
113
|
+
- Permission grants per role.
|
|
114
|
+
- Index existence for query-critical paths.
|
|
115
|
+
|
|
116
|
+
When pgTAP is unavailable, use raw `SELECT` assertions against `information_schema` or `pg_catalog`.
|
|
117
|
+
|
|
118
|
+
## Part 6 — Remote Integration Testing
|
|
119
|
+
|
|
120
|
+
### Default: read-only
|
|
121
|
+
|
|
122
|
+
- Inspect schema, permissions, RLS policies, and data shape without mutation.
|
|
123
|
+
- Use `EXPLAIN` for query plan verification without executing writes.
|
|
124
|
+
- No approval required for read-only operations.
|
|
125
|
+
|
|
126
|
+
### Testing environment writes
|
|
127
|
+
|
|
128
|
+
- **REQUIRE** explicit user approval before any write or migration apply.
|
|
129
|
+
- Connect only to a dedicated testing environment — never to production for writes.
|
|
130
|
+
- After writes: verify applied state, then clean up test data.
|
|
131
|
+
- Record: environment name, operation performed, approval evidence.
|
|
132
|
+
|
|
133
|
+
### No testing environment available
|
|
134
|
+
|
|
135
|
+
- Report: "Remote integration limited to read-only inspection. No testing environment configured."
|
|
136
|
+
- Recommend environment setup. Do not fabricate a pass.
|
|
137
|
+
|
|
138
|
+
## Output
|
|
139
|
+
|
|
140
|
+
- Tests written or updated, by path and layer
|
|
141
|
+
- Local environment detection result (which option selected, or fallback triggered)
|
|
142
|
+
- Migration clean-apply result
|
|
143
|
+
- RLS policy test coverage per table with RLS enabled
|
|
144
|
+
- Schema contract coverage (tables, constraints, functions)
|
|
145
|
+
- Remote integration status (read-only / testing-env / unavailable)
|
|
146
|
+
- Stated limitations where real-policy testing was not possible
|
|
147
|
+
|
|
148
|
+
## Final Instructions
|
|
149
|
+
|
|
150
|
+
1. You **MUST** consult `/TESTING.md` and respect its Layer 2.5 boundary.
|
|
151
|
+
2. You **MUST NOT** mock the data layer. If the database is mocked, the test is Layer 2.
|
|
152
|
+
3. You **MUST** detect the local environment and recommend the easiest applicable option.
|
|
153
|
+
4. You **MUST** report a limitation when no local or remote environment is available.
|
|
154
|
+
5. You **MUST** test RLS against the real policy layer, not application-level checks.
|
|
155
|
+
6. You **MUST** require explicit approval before any remote write operation.
|
|
156
|
+
7. You **MUST NOT** install dependencies — detect and recommend only.
|
|
@@ -92,6 +92,47 @@ A monorepo defines `test:node` as `app && scraper && db` while the `api` package
|
|
|
92
92
|
|
|
93
93
|
Correct output: **defect** — `api` is unreachable from the aggregate; the CI gate and the deploy gate both let untested backend code through. Reporting "all script names conform" here is a false pass.
|
|
94
94
|
|
|
95
|
+
## Part 4 — Integration, E2E, and Contract Infrastructure Detection
|
|
96
|
+
|
|
97
|
+
Detect the presence and configuration state of higher-layer test infrastructure. Report findings as **informational** — missing infrastructure is not a defect, but it informs the gap analysis.
|
|
98
|
+
|
|
99
|
+
### Integration testing infrastructure
|
|
100
|
+
|
|
101
|
+
| Check | What to look for |
|
|
102
|
+
| ----------------------- | ------------------------------------------------------------- |
|
|
103
|
+
| **Testcontainers** | `@testcontainers/postgresql` or similar in dependencies |
|
|
104
|
+
| **Docker Compose** | `docker-compose.yml` or `compose.yml` with a database service |
|
|
105
|
+
| **Supabase local CLI** | `supabase/config.toml` or `.supabase/` directory |
|
|
106
|
+
| **Docker availability** | `docker info` succeeds (report if unavailable) |
|
|
107
|
+
|
|
108
|
+
### E2E testing infrastructure
|
|
109
|
+
|
|
110
|
+
| Check | What to look for |
|
|
111
|
+
| ------------------------- | -------------------------------------------------------------------------------- |
|
|
112
|
+
| **Playwright config** | `playwright.config.ts` or `playwright.config.js` present |
|
|
113
|
+
| **Base URL configured** | Config references `process.env.BASE_URL` or equivalent (not hardcoded localhost) |
|
|
114
|
+
| **Auth setup project** | A `global-setup` or setup project defined in Playwright config |
|
|
115
|
+
| **Browser install in CI** | CI workflow includes `playwright install` step |
|
|
116
|
+
| **`test:e2e` script** | Present in `package.json` and reachable from aggregate |
|
|
117
|
+
|
|
118
|
+
### Contract validation infrastructure
|
|
119
|
+
|
|
120
|
+
| Check | What to look for |
|
|
121
|
+
| -------------------------- | ------------------------------------------------------ |
|
|
122
|
+
| **OpenAPI spec** | `openapi.yaml`, `openapi.json`, `swagger.*` in repo |
|
|
123
|
+
| **AsyncAPI spec** | `asyncapi.yaml`, `asyncapi.json` in repo |
|
|
124
|
+
| **`dt` CLI** | `which dt` succeeds |
|
|
125
|
+
| **`test:contract` script** | Present in `package.json` and reachable from aggregate |
|
|
126
|
+
|
|
127
|
+
### Reporting
|
|
128
|
+
|
|
129
|
+
Report each finding as:
|
|
130
|
+
|
|
131
|
+
- `[INFO] <infrastructure> detected: <path or status>`
|
|
132
|
+
- `[INFO] <infrastructure> not found — Layer {X} tests require this`
|
|
133
|
+
|
|
134
|
+
These are informational findings for the gap analysis. They do **NOT** block or fail the standards check.
|
|
135
|
+
|
|
95
136
|
## Failure Modes
|
|
96
137
|
|
|
97
138
|
| Condition | Required behavior |
|
|
@@ -120,12 +120,23 @@ Follow `.github/instructions/implement.instructions.md`:
|
|
|
120
120
|
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
121
121
|
| `product-engineer` | Produces the task lists and refined issues that `developer` executes |
|
|
122
122
|
| `planner` | Orchestrates multi-story runs — delegates each story to `developer` in Execute Mode with an integration branch override |
|
|
123
|
+
| `researcher` | May be invoked by `developer` for troubleshooting/diagnosis when a bug or regression is unclear; conditional, never mandatory |
|
|
123
124
|
| `verifier` | Invoked by `developer` in `audit` mode, post-implementation and pre-PR-ready, mandatory and non-skippable — reports fidelity/drift; findings route to `product-engineer`'s `activity-drift-reconciliation` for remediation |
|
|
124
125
|
| `qa-engineer` | Invoked by `developer` at the completion gate, before the `verifier` audit — owns the testing standard, missing test harnesses, and coverage/gap reporting; `developer` still authors feature tests under rule 19 |
|
|
125
126
|
| `technical-writer` | Invoked by `developer` before PR is marked ready — updates `/docs` |
|
|
126
127
|
| `housekeeping` | Can be invoked during implementation for lint/type/test-wiring fixes |
|
|
127
128
|
| `github-ops` | Defines conventions for all GitHub artifacts — `developer` follows these rules |
|
|
128
129
|
|
|
130
|
+
### Codebase Research for Troubleshooting (Conditional)
|
|
131
|
+
|
|
132
|
+
When diagnosing a bug, regression, or unclear failure during implementation, you **SHOULD** invoke `researcher` if:
|
|
133
|
+
|
|
134
|
+
- The failure spans multiple modules and the root cause is not immediately apparent.
|
|
135
|
+
- The area is unfamiliar or lacks documentation.
|
|
136
|
+
- Multiple hypotheses need codebase evidence to disambiguate.
|
|
137
|
+
|
|
138
|
+
**Skip when:** the failure is localized to a single file or function, or the root cause is already identified. This is a recommended diagnostic tool, not a mandatory gate.
|
|
139
|
+
|
|
129
140
|
---
|
|
130
141
|
|
|
131
142
|
## Autonomous Behavior Contract
|
|
@@ -47,6 +47,20 @@ If any required input is missing, ask one focused question with a default option
|
|
|
47
47
|
|
|
48
48
|
---
|
|
49
49
|
|
|
50
|
+
## Codebase Research for Pre-Orchestration (Conditional)
|
|
51
|
+
|
|
52
|
+
Before parsing stories and building the dependency graph, you **SHOULD** invoke `researcher` if:
|
|
53
|
+
|
|
54
|
+
- The stories span multiple modules or packages whose dependencies are not yet established.
|
|
55
|
+
- The orchestration involves unfamiliar areas of the codebase.
|
|
56
|
+
- Story ordering requires understanding cross-module data flow or contract boundaries.
|
|
57
|
+
|
|
58
|
+
**Skip when:** story dependencies are already explicit in the task file, the codebase is small and well-understood, or a non-stale research artifact already covers the scope.
|
|
59
|
+
|
|
60
|
+
When triggered, delegate to `researcher` with the orchestration question (e.g., "What are the dependency relationships between modules X, Y, and Z?"). Consume the resulting `/workstream/research-*.md` artifact to inform dependency ordering in Phase 1.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
50
64
|
## Phase 0 - Discover Task Source
|
|
51
65
|
|
|
52
66
|
### Option A - File in /workstream
|
|
@@ -415,15 +429,16 @@ Planner **MUST NOT** allow multiple open story PRs to merge concurrently.
|
|
|
415
429
|
After all stories are merged into integration:
|
|
416
430
|
|
|
417
431
|
1. Run full integration test suite on integration branch.
|
|
418
|
-
2. Invoke `
|
|
419
|
-
3. Invoke `
|
|
420
|
-
4.
|
|
421
|
-
5.
|
|
422
|
-
6.
|
|
423
|
-
7.
|
|
432
|
+
2. Invoke `qa-engineer` at **PRD scope** (all packages affected by the merged stories, all layers) for a coverage and gap rollup. Report a PRD-level `coverage_gate` that aggregates per-story gates. Scope to packages affected by the merged stories, not the entire workspace.
|
|
433
|
+
3. Invoke `verifier` in `audit` mode for a **PRD-level rollup audit** against the full integrated scope (all merged stories on the integration branch, cross-checked against the source PRD/spec/milestone). This is mandatory and non-skippable — it runs whether or not any story-level drift was previously reported. Post the resulting human-readable summary to the plan/milestone issue via `github-ops` comment conventions. Rollup drift findings **MUST NOT** block the consolidated PR handoff; Unintended/Intended drift routes to `product-engineer`'s `activity-drift-reconciliation` flow.
|
|
434
|
+
4. Invoke `technical-writer` for a planner-level drift/stale-doc validation pass against integrated changes; unresolved drift **MUST** block handoff.
|
|
435
|
+
5. Open one consolidated PR from integration branch to `main`.
|
|
436
|
+
6. **Do NOT merge.** Notify the user that the consolidated PR is ready for their review.
|
|
437
|
+
7. Wait for the user to approve and merge the PR into `main`.
|
|
438
|
+
8. Before final handoff, **MUST** ensure the local working branch is the integration branch used for this run:
|
|
424
439
|
- Preferred: run `git checkout integration/<plan-id>-<short-description>`.
|
|
425
440
|
- Alternative (if checkout is not possible in the current runtime): explicitly verify and report current branch, and provide the exact checkout command the user can run.
|
|
426
|
-
|
|
441
|
+
9. Final user response **MUST** include a `PR Directives (User Action Required)` section with:
|
|
427
442
|
|
|
428
443
|
- consolidated PR URL
|
|
429
444
|
- current CI/check status
|