@brunosps00/dev-workflow 0.10.0 → 0.13.0

package/scaffold/skills/dw-testing-discipline/references/flaky-discipline.md

@@ -0,0 +1,163 @@
+# Flaky discipline — taxonomy, quarantine, SLOs
+
+A flaky test is one that produces different verdicts (pass/fail) on the same code across runs. They corrode trust in the suite faster than any other category of test debt.
+
+## The four root causes (in order of frequency)
+
+### Cause 1: Race conditions (concurrency)
+
+**Tells:**
+- Test passes locally, fails in CI (or vice versa).
+- Failure rate correlates with CI machine load.
+- Adding `await page.waitForTimeout(100)` "fixes" it.
+
+**Common scenarios:**
+- Async operation completes after test moves on (missing `await`).
+- Two requests sent simultaneously, response order matters.
+- DOM update happens after assertion runs.
+- Database write not yet committed when read fires.
+
+**Fix:**
+- Replace wall-clock waits with condition-based waits (`waitFor`, `toBeVisible`, `expect.poll`).
+- Add proper `await` on every async operation.
+- Use transaction boundaries explicitly when test reads its own write.
+
+### Cause 2: Test order dependency
+
+**Tells:**
+- Test passes when suite runs in order, fails with `--shuffle`.
+- Test fails when run with `.only` in isolation.
+- Failures cluster on first run after CI restart but not afterwards.
+
+**Common scenarios:**
+- `beforeAll` populates shared state; second test mutates it; third test fails.
+- Test A creates a global mock; Test B inherits it unexpectedly.
+- Database row persists across tests because cleanup is in `afterEach` but a test threw mid-execution.
+
+**Fix:**
+- Move state creation from `beforeAll` to `beforeEach`.
+- Reset shared state in `beforeEach` (clean slate every test).
+- Avoid global mocks; scope mocks to the test that needs them.
+- Run with `--shuffle` in CI to catch new order dependencies.
+
+### Cause 3: Non-deterministic inputs
+
+**Tells:**
+- Test fails at month boundary, year boundary, DST change.
+- Test fails based on hostname, locale, timezone.
+- Test fails when a flaky RNG produces edge values.
+
+**Common scenarios:**
+- `new Date()` in production code, tested without clock fake.
+- `Math.random()` for IDs, tested without seed.
+- `Intl.DateTimeFormat` rendering based on system locale.
+- File paths with timestamps, hash IDs based on time.
+
+**Fix:**
+- Mock the clock (`vi.useFakeTimers`, `freezegun`).
+- Seed RNG explicitly in tests (`Math.random = () => 0.5` or via DI).
+- Pin locale and timezone in CI environment AND in test setup.
+
+### Cause 4: External dependencies
+
+**Tells:**
+- Test fails when a third-party service has an outage.
+- Test fails when CI runs against a real API and hits rate limits.
+- Test fails differently for different geographic CI runners.
+
+**Common scenarios:**
+- Direct call to external API in unit tests.
+- DNS lookup baked into test execution path.
+- CDN-hosted resources in E2E tests.
+
+**Fix:**
+- Mock external services at unit/integration layers.
+- Use contract tests instead of live calls.
+- For E2E, use a sandbox account / dedicated test environment.
+
+## Quarantine workflow
+
+When a test flakes:
+
+### Within 1 hour of detection
+
+1. **Quarantine the test.** Add `.skip` or equivalent. Add a comment:
+
+   ```javascript
+   test.skip('FLAKY-2026-05-12: race condition in checkout flow — owner: bruno, fix-by: 2026-05-19', () => {
+     // ...
+   });
+   ```
+
+2. **File a tracking issue.** Title: `FLAKY: <test name>`. Body includes:
+   - Test name and file
+   - Failure mode observed (race? order-dependency? non-determinism?)
+   - First detection: CI run URL, timestamp
+   - Hypothesis (if any)
+   - Owner and fix-by date
+
+3. **Note in CI.** The next CI run shows "1 quarantined" — make this visible on the dashboard.
+
+### Within 24 hours
+
+1. **Named owner assigned.** Not "team X" — a person.
+2. **Fix-by date set.** Default 5 business days. Major flake (production-path test): 2 days.
+
+### When fix-by passes without fix
+
+Escalate:
+- Pair the owner with someone for a debug session.
+- If still unfixed after 2× the fix-by window, the test is removed (not skipped). A failing un-skipped test is better than a perpetually skipped test.
+
+## SLOs
+
+### `flaky_rate` (first-class metric)
+
+- Definition: `(tests that pass on retry but fail on first run) / (total test runs)`.
+- Target: < 1–2% per week.
+- Alert at: > 5% on any given day.
+
+### `time-to-fix-flaky`
+
+- Definition: hours from quarantine to fix-merged.
+- Target: median < 24 hours; p95 < 7 days.
+
+### `quarantine inventory`
+
+- Definition: count of currently-skipped tests with `FLAKY-*` markers.
+- Target: < 10 at any time.
+- Alert at: > 25 (the quarantine has become a cemetery — emergency cleanup).
+
+## What NOT to do
+
+- **Auto-retry as fix.** `retries: 3` in CI config is hiding flakes, not fixing them. The 4th run that finally passes still validated nothing.
+- **Increase timeouts indefinitely.** A timeout that grows from 5s to 30s "to make CI pass" means the test isn't waiting on the right condition.
+- **Remove the test without investigation.** "It's been flaky forever, delete it" — sometimes correct, but make sure the underlying invariant is captured elsewhere.
+- **Mark skip without owner.** A skip is a debt. An unowned debt is a perpetual liability.
+
+## When a test should be permanently removed
+
+A flaky test should be DELETED (not just skipped) when:
+
+1. The invariant it tests is covered elsewhere (duplicate per A23).
+2. The invariant it tests is no longer a real requirement.
+3. The test was always probabilistic by design and never had value.
+
+Deletion is acceptable; abandonment-by-skip is not.
+
+## Real-systems-at-final-gate principle
+
+Many flakes come from mocks drifting from reality. The defense:
+
+- **Unit:** mock the world; fast feedback; flake budget tiny here.
+- **Integration:** real DB (testcontainers); mock external services with contract validation.
+- **Contract:** Pact / schemathesis verifying producer-consumer agreement.
+- **E2E:** real services in a preview environment; near-zero mocks.
+
+When CI is wired this way, a flake at unit usually = race or order-dependency (fixable). A flake at E2E usually = real environment issue (fix the environment, not the test).
+
+## Integration with dev-workflow
+
+- `/dw-fix-qa` uses this taxonomy when retest cycles produce inconsistent results: classify the flake, apply the right fix, document.
+- `/dw-code-review` flags tests being modified that have a `FLAKY-*` marker — review must verify the flake is now actually fixed, not just made less likely.
+- `/dw-run-qa` weekly summary includes the `flaky_rate` metric.

package/scaffold/skills/dw-testing-discipline/references/iron-laws.md

@@ -0,0 +1,128 @@
+# Six Iron Laws — expanded with examples
+
+The laws are short for memorization. Each carries nuance that matters in practice.
+
+## Law 1: Test the behavior, never the mock
+
+**What it means:** Your test asserts what the system DOES from the caller's perspective. It does not assert that internal call X was made with internal argument Y.
+
+**Why it matters:** A test bound to internal calls breaks the day you refactor — even when behavior didn't change. The "test is red, behavior is fine" experience erodes trust in the suite. Soon no one runs the suite.
+
+**Violation example:**
+
+```javascript
+// BAD — asserting on mock internals
+test('createOrder calls inventory.reserve', () => {
+  const inventory = { reserve: vi.fn() };
+  createOrder({ items: [...] }, inventory);
+  expect(inventory.reserve).toHaveBeenCalledWith(items, 'reserve');
+});
+```
+
+You've asserted that `createOrder` USES the inventory adapter in a specific way. Now the refactor that consolidates `reserve` into `commit-with-reservation` breaks this test even though the order still gets created.
+
+**Correct version:**
+
+```javascript
+// GOOD — asserting behavior
+test('createOrder reserves inventory before confirming', async () => {
+  const result = await createOrder({ items: [...] });
+  expect(result.status).toBe('confirmed');
+  expect(await getInventoryFor(items[0].sku)).toBe(originalStock - 1);
+});
+```
+
+Now the test cares about the OUTCOME (inventory decremented, order confirmed), not the path.
+
+## Law 2: Push every test to the lowest layer that can detect the failure
+
+**What it means:** If a unit test can catch a bug, use it. If only an integration test can catch it, integration. If only an end-to-end run can catch it, E2E. Don't write E2E for what a unit can prove.
+
+**Why it matters:** Tests at lower layers run faster, fail more precisely, isolate the cause better. A bug in pure logic caught at unit takes 50ms and tells you the exact function. The same bug caught at E2E takes 30 seconds and tells you "checkout failed."
+
+**The pyramid resolved:**
+
+| Layer | Catches | Speed | Cost |
+|-------|---------|-------|------|
+| Unit | Pure logic, math, parsing, formatters | <100ms | low |
+| Integration | Module composition, DB queries, HTTP handlers | 500ms–5s | medium |
+| Contract | Producer/consumer agreement at API boundary | 1–10s | medium |
+| E2E | User journey across multiple services | 10s–60s | high |
+
+**Rule of thumb:**
+- If you can write a unit test for it, do so.
+- If unit can't reach it (needs DB, queue, real HTTP), write integration.
+- E2E only for journeys that NO lower layer can detect (browser-renders-correctly, third-party-callback-arrives, multi-step session state).
+
+## Law 3: When a test fails, fix production first — change the test only after writing why
+
+**What it means:** A red test is a signal. The first question is "what's wrong with production?" Not "why is the test wrong?"
+
+**Why it matters:** Tests are weakened to pass FAR more often than they should be. "The behavior is fine; the test is too strict" is the slippery slope that leaves you with a green suite full of meaningless assertions.
+
+**Process when a test goes red:**
+
+1. **Read the failure message.** What invariant did the test claim, and what did it observe?
+2. **Read production code** in the path that produces the observation.
+3. **Decide which is wrong.** If production violates the invariant, fix production. If the test mis-states the invariant, document WHY before relaxing.
+4. **Commit the analysis** in the test's commit message or PR body. "Relaxed assertion from X to Y because <reason>" is auditable; "fix test" is not.
+
+**Anti-pattern:** Re-run the test until green. Auto-retry on flake. Add `.only` to skip the rest.
+
+## Law 4: Real systems gate the merge. Mocks isolate; they do not validate.
+
+**What it means:** Before code merges to main, at least ONE test path exercised real systems (real DB, real route, real external integration in a sandbox or test account). Mocks are fine for fast unit feedback; they cannot decide "safe to ship."
+
+**Why it matters:** Mock drift is real. The mocked HTTP response from 3 months ago no longer matches the actual API. Tests pass; production fails on first real call.
+
+**Practical pattern:**
+
+- Unit tests: mock the world; run on every keystroke / on every commit.
+- Integration tests: real local DB (testcontainers, in-memory if equivalent); run on every PR.
+- Contract tests: real producer/consumer agreement check; run on every PR.
+- E2E: real preview environment with real services; run on PRs before merge to main.
+
+The discipline: no merge without a green E2E (or equivalent real-system check) for the touched path.
+
+## Law 5: Coverage is a flashlight. Mutation score is a quality probe. Neither is a target.
+
+**What it means:**
+- **Coverage** tells you what lines executed. Useful as a NEGATIVE signal: 30% coverage = lots of dark code. Useless as a positive signal: 95% coverage with weak assertions is decorative.
+- **Mutation score** introduces small bugs (mutations) and measures whether tests catch them. A high mutation score means tests are actually probing behavior, not just executing lines.
+- Neither should be a number you optimize for. They're diagnostics.
+
+**Anti-pattern:** "We need 90% coverage to merge." Coverage as a gate produces tests written to pass the gate, not to find bugs.
+
+**Healthier framing:** "What lines in the touched diff are NOT covered? Why?" Sometimes the answer is "we don't care, it's logging." Sometimes it's "actually that's a critical branch, add a test."
+
+## Law 6: No test-only methods, branches, or flags leak into production code
+
+**What it means:** Production code should not have `if (process.env.NODE_ENV === 'test') { ... }` branches. Should not have `// for testing only` methods exposed on classes. Should not export internals just for assertions.
+
+**Why it matters:** Production code carrying test-only logic is testing decorations leaking into the artifact users run. Bug surface grows; the test environment diverges from production.
+
+**Correct patterns:**
+
+- Need to inject a dependency for testing? Use constructor injection / dependency injection.
+- Need to assert on internal state? Add a logging hook or event emission that production also benefits from.
+- Need to bypass auth in tests? Use a dedicated test environment with test credentials, not a backdoor flag.
+
+**Tell tales:**
+- `// only used in tests` comments.
+- `*ForTesting` suffix on methods.
+- `vi.spyOn(module, '_internal')` accessing things prefixed with underscore.
+- `process.env.E2E_MODE` reaching into production runtime decisions.
+
+If you see these, the test design is wrong. Refactor production to be testable, don't add backdoors.
+
+## Putting the laws together
+
+A healthy test:
+1. Asserts behavior visible to a caller (Law 1).
+2. Sits at the lowest layer that can prove that behavior (Law 2).
+3. When red, sends you to read production code (Law 3).
+4. Has a sibling that exercises real systems somewhere in the pipeline (Law 4).
+5. Survives a mutation in the code it claims to cover (Law 5).
+6. Has zero footprint in production code (Law 6).
+
+Any test that fails ≥2 of these is technical debt accumulating. `/dw-code-review` flags them.

package/scaffold/skills/dw-testing-discipline/references/playwright-recipes.md

@@ -0,0 +1,282 @@
+# Playwright recipes — concrete tactical patterns
+
+Practical Playwright code for the common scenarios. Use this when `/dw-run-qa` runs in UI mode or when `/dw-functional-doc` needs E2E coverage.
+
+> These recipes ASSUME the doctrine in this skill (Iron Laws, positive patterns) has already been applied. Recipes are the HOW once the WHY is settled.
+
+## Basic Navigation
+
+```javascript
+import { test, expect } from '@playwright/test';
+
+test('homepage loads and shows hero', async ({ page }) => {
+  await page.goto('http://localhost:3000');
+  await expect(page).toHaveTitle(/Welcome/);
+  await expect(page.getByRole('heading', { level: 1 })).toBeVisible();
+});
+```
+
+Key practices:
+- Use `expect(page).toHaveTitle(/pattern/)` instead of `await page.title()` + manual assertion (waits for title to settle).
+- Use `getByRole`, not selectors (Positive Pattern #1).
+
+## Form interaction
+
+```javascript
+test('login redirects to dashboard with user info', async ({ page }) => {
+  await page.goto('/login');
+
+  // Use labels, not selectors
+  await page.getByLabel('Email').fill('user@example.com');
+  await page.getByLabel('Password').fill('secret123');
+  await page.getByRole('button', { name: 'Sign in' }).click();
+
+  // Wait on observable outcome, not wall-clock
+  await expect(page).toHaveURL(/\/dashboard/);
+  await expect(page.getByText('user@example.com')).toBeVisible();
+});
+```
+
+## Screenshot capture (debugging + visual evidence)
+
+```javascript
+// For debugging
+test('checkout flow', async ({ page }) => {
+  try {
+    await page.goto('/checkout');
+    // ... interaction
+  } catch (err) {
+    await page.screenshot({ path: `failure-${Date.now()}.png`, fullPage: true });
+    throw err;
+  }
+});
+
+// For visual regression (only on stable surfaces)
+test('header looks correct', async ({ page }) => {
+  await page.goto('/');
+  await expect(page.getByRole('banner')).toHaveScreenshot('header.png');
+});
+```
+
+Warning: visual regression tests need a baseline. Apply the snapshot classification gate from `ai-agent-gates.md` (Gate 5).
+
+## Browser console logs (capture for debug)
+
+```javascript
+test('no console errors during checkout', async ({ page }) => {
+  const errors = [];
+  page.on('console', msg => {
+    if (msg.type() === 'error') errors.push(msg.text());
+  });
+  page.on('pageerror', err => errors.push(err.message));
+
+  await page.goto('/checkout');
+  await page.getByRole('button', { name: 'Complete order' }).click();
+  await expect(page.getByText('Order placed')).toBeVisible();
+
+  expect(errors).toEqual([]);
+});
+```
+
+## Network interception (mock or inspect)
+
+```javascript
+test('handles API failure gracefully', async ({ page }) => {
+  // Intercept the orders API and return 500
+  await page.route('**/api/orders', route =>
+    route.fulfill({ status: 500, body: 'Server error' })
+  );
+
+  await page.goto('/orders');
+
+  // The UI should show a recoverable error state, not crash
+  await expect(page.getByText('Could not load orders')).toBeVisible();
+  await expect(page.getByRole('button', { name: 'Retry' })).toBeVisible();
+});
+```
+
+```javascript
+test('order list calls API with correct params', async ({ page }) => {
+  let capturedUrl;
+  page.on('request', req => {
+    if (req.url().includes('/api/orders')) capturedUrl = req.url();
+  });
+
+  await page.goto('/orders?filter=overdue');
+
+  await expect.poll(() => capturedUrl).toContain('filter=overdue');
+});
+```
+
+## Wait for conditions (NOT wall-clock)
+
+```javascript
+// GOOD — wait on observable condition
+await expect(page.getByText(/order #\d+ confirmed/i)).toBeVisible({ timeout: 10000 });
+
+// GOOD — wait on URL change
+await page.waitForURL(/\/dashboard/);
+
+// GOOD — wait on network response
+const responsePromise = page.waitForResponse(resp => resp.url().includes('/api/orders') && resp.status() === 200);
+await page.getByRole('button', { name: 'Load orders' }).click();
+await responsePromise;
+
+// GOOD — poll a custom condition
+await expect.poll(async () => {
+  const count = await page.getByRole('listitem').count();
+  return count;
+}, { timeout: 5000 }).toBeGreaterThan(0);
+
+// BAD — wall-clock
+await page.waitForTimeout(3000);  // ← Anti-pattern A7
+```
+
+## Mobile viewport testing
+
+```javascript
+test('mobile menu works', async ({ browser }) => {
+  const context = await browser.newContext({
+    viewport: { width: 375, height: 812 },  // iPhone X
+    userAgent: 'Mozilla/5.0 (iPhone; ...)',
+  });
+  const page = await context.newPage();
+
+  await page.goto('/');
+  await page.getByRole('button', { name: /menu/i }).click();
+  await expect(page.getByRole('navigation')).toBeVisible();
+});
+```
+
+For dev-workflow's redesign-ui flow, capture screenshots at TWO viewports: 375px (mobile) and 1440px (desktop). This is documented in `/dw-redesign-ui` step 7.
+
+## Multi-step user journey (Page Object Model)
+
+For >3 tests sharing the same flow, POM pays off:
+
+```javascript
+// checkout-page.ts
+export class CheckoutPage {
+  constructor(private page) {}
+
+  async goto() {
+    await this.page.goto('/checkout');
+  }
+
+  async fillShipping(addr) {
+    await this.page.getByLabel('Address').fill(addr.street);
+    await this.page.getByLabel('City').fill(addr.city);
+    await this.page.getByLabel('ZIP').fill(addr.zip);
+  }
+
+  async selectPayment(method) {
+    await this.page.getByLabel(method).check();
+  }
+
+  async place() {
+    await this.page.getByRole('button', { name: 'Place order' }).click();
+  }
+
+  async expectConfirmed() {
+    await expect(this.page.getByText(/order #\d+ confirmed/i)).toBeVisible();
+  }
+}
+
+// checkout.spec.ts
+test('checkout end-to-end', async ({ page }) => {
+  const checkout = new CheckoutPage(page);
+  await checkout.goto();
+  await checkout.fillShipping(defaultAddress);
+  await checkout.selectPayment('Card ending in 4242');
+  await checkout.place();
+  await checkout.expectConfirmed();
+});
+```
+
+## Helper utilities
+
+For small projects, a couple of helpers reduce boilerplate. Drop these in `test-helpers.ts`:
+
+```typescript
+import type { Page } from '@playwright/test';
+
+/**
+ * Capture browser console logs as the page runs.
+ * Pass the returned array to assert on logs at the end of the test.
+ */
+export function captureConsoleLogs(page: Page) {
+  const logs: Array<{ type: string; text: string; ts: string }> = [];
+  page.on('console', msg => {
+    logs.push({ type: msg.type(), text: msg.text(), ts: new Date().toISOString() });
+  });
+  return logs;
+}
+
+/**
+ * Take a timestamped screenshot for debugging or evidence collection.
+ */
+export async function captureScreenshot(page: Page, name: string) {
+  const stamp = new Date().toISOString().replace(/[:.]/g, '-');
+  const filename = `${name}-${stamp}.png`;
+  await page.screenshot({ path: filename, fullPage: true });
+  return filename;
+}
+```
+
+## Persistent session (auth state)
+
+When tests need to start logged in, save auth state ONCE in setup:
+
+```typescript
+// global-setup.ts
+import { chromium } from '@playwright/test';
+
+export default async () => {
+  const browser = await chromium.launch();
+  const page = await browser.newPage();
+
+  await page.goto('http://localhost:3000/login');
+  await page.getByLabel('Email').fill('e2e-user@example.com');
+  await page.getByLabel('Password').fill(process.env.E2E_PASSWORD);
+  await page.getByRole('button', { name: 'Sign in' }).click();
+  await page.waitForURL(/\/dashboard/);
+
+  await page.context().storageState({ path: '.auth/user.json' });
+  await browser.close();
+};
+
+// playwright.config.ts
+export default defineConfig({
+  globalSetup: require.resolve('./global-setup'),
+  use: {
+    storageState: '.auth/user.json',
+  },
+});
+```
+
+Tests now start authenticated. No login loop in every test.
+
+## Common pitfalls (cross-reference to anti-patterns)
+
+| Pitfall | Anti-pattern | Fix |
+|---------|--------------|-----|
+| `await page.click('.btn-primary')` | A1 (implementation selectors) | `getByRole('button', { name: ... })` |
+| `await page.waitForTimeout(3000)` | A7 (static sleeps) | Wait on observable condition |
+| `expect(button).toBeTruthy()` | A5 (vague existence) | Assert what you actually want |
+| Tests pass solo, fail in suite | A8 (order dependency) | Setup in beforeEach, not beforeAll |
+| `await page.goto('https://www.google.com')` | A20 (third-party site) | Mock or skip |
+
+## Limitations
+
+- Playwright cannot test native mobile apps (use React Native Testing Library or Detox).
+- Some authentication flows (Google Sign-In, MFA hardware keys) cannot be automated; use test-mode bypasses with a dedicated test account.
+- Visual regression tests are sensitive to font rendering across OSes; pin to a CI runner OS.
+
+## Cross-skill integration
+
+When running these recipes, the doctrine in this skill applies:
+- Apply Iron Laws (especially Law 2: lowest layer first — many E2E tests should be integration tests instead).
+- Run the 7 AI Agent Gates if a coding agent is producing this test.
+- Check for Anti-Patterns 1, 5, 7, 20 in the diff.
+- For browser security scenarios (auth bypass, XSS, CSRF), see `security-boundary.md`.
+- For picking which workflow (UI / network / perf) applies, see `three-workflow-patterns.md`.