@brunosps00/dev-workflow 0.11.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (127) hide show
  1. package/README.md +54 -5
  2. package/lib/constants.js +20 -20
  3. package/lib/init.js +24 -1
  4. package/lib/migrate-skills.js +129 -0
  5. package/lib/removed-bundled-skills.js +16 -0
  6. package/lib/uninstall.js +6 -2
  7. package/lib/utils.js +43 -1
  8. package/package.json +1 -1
  9. package/scaffold/en/agent-instructions.md +68 -0
  10. package/scaffold/en/commands/dw-autopilot.md +1 -1
  11. package/scaffold/en/commands/dw-brainstorm.md +1 -1
  12. package/scaffold/en/commands/dw-bugfix.md +4 -3
  13. package/scaffold/en/commands/dw-code-review.md +1 -0
  14. package/scaffold/en/commands/dw-create-tasks.md +6 -0
  15. package/scaffold/en/commands/dw-create-techspec.md +1 -1
  16. package/scaffold/en/commands/dw-deps-audit.md +1 -1
  17. package/scaffold/en/commands/dw-fix-qa.md +1 -1
  18. package/scaffold/en/commands/dw-functional-doc.md +2 -2
  19. package/scaffold/en/commands/dw-help.md +2 -2
  20. package/scaffold/en/commands/dw-redesign-ui.md +7 -7
  21. package/scaffold/en/commands/dw-run-qa.md +5 -4
  22. package/scaffold/en/commands/dw-run-task.md +2 -2
  23. package/scaffold/en/templates/constitution-template.md +1 -1
  24. package/scaffold/pt-br/agent-instructions.md +68 -0
  25. package/scaffold/pt-br/commands/dw-autopilot.md +1 -1
  26. package/scaffold/pt-br/commands/dw-brainstorm.md +1 -1
  27. package/scaffold/pt-br/commands/dw-bugfix.md +4 -3
  28. package/scaffold/pt-br/commands/dw-code-review.md +1 -0
  29. package/scaffold/pt-br/commands/dw-create-tasks.md +6 -0
  30. package/scaffold/pt-br/commands/dw-create-techspec.md +1 -1
  31. package/scaffold/pt-br/commands/dw-deps-audit.md +1 -1
  32. package/scaffold/pt-br/commands/dw-fix-qa.md +1 -1
  33. package/scaffold/pt-br/commands/dw-functional-doc.md +2 -2
  34. package/scaffold/pt-br/commands/dw-help.md +2 -2
  35. package/scaffold/pt-br/commands/dw-redesign-ui.md +7 -7
  36. package/scaffold/pt-br/commands/dw-run-qa.md +5 -4
  37. package/scaffold/pt-br/commands/dw-run-task.md +2 -2
  38. package/scaffold/pt-br/templates/constitution-template.md +1 -1
  39. package/scaffold/skills/dw-council/SKILL.md +1 -1
  40. package/scaffold/skills/dw-incident-response/SKILL.md +164 -0
  41. package/scaffold/skills/dw-incident-response/references/blameless-discipline.md +126 -0
  42. package/scaffold/skills/dw-incident-response/references/communication-templates.md +107 -0
  43. package/scaffold/skills/dw-incident-response/references/postmortem-template.md +133 -0
  44. package/scaffold/skills/dw-incident-response/references/runbook-templates.md +169 -0
  45. package/scaffold/skills/dw-incident-response/references/severity-and-triage.md +186 -0
  46. package/scaffold/skills/dw-llm-eval/SKILL.md +148 -0
  47. package/scaffold/skills/dw-llm-eval/references/agent-eval.md +252 -0
  48. package/scaffold/skills/dw-llm-eval/references/judge-calibration.md +169 -0
  49. package/scaffold/skills/dw-llm-eval/references/oracle-ladder.md +171 -0
  50. package/scaffold/skills/dw-llm-eval/references/rag-metrics.md +186 -0
  51. package/scaffold/skills/dw-llm-eval/references/reference-dataset.md +190 -0
  52. package/scaffold/skills/dw-testing-discipline/SKILL.md +171 -0
  53. package/scaffold/skills/dw-testing-discipline/references/agent-guardrails.md +170 -0
  54. package/scaffold/skills/dw-testing-discipline/references/anti-patterns.md +336 -0
  55. package/scaffold/skills/dw-testing-discipline/references/core-rules.md +128 -0
  56. package/scaffold/skills/dw-testing-discipline/references/flaky-discipline.md +163 -0
  57. package/scaffold/skills/dw-testing-discipline/references/patterns.md +241 -0
  58. package/scaffold/skills/dw-testing-discipline/references/playwright-recipes.md +282 -0
  59. package/scaffold/skills/{webapp-testing → dw-testing-discipline}/references/security-boundary.md +1 -1
  60. package/scaffold/skills/dw-ui-discipline/SKILL.md +150 -0
  61. package/scaffold/skills/dw-ui-discipline/references/accessibility-floor.md +225 -0
  62. package/scaffold/skills/dw-ui-discipline/references/curated-defaults.md +195 -0
  63. package/scaffold/skills/dw-ui-discipline/references/hard-gate.md +162 -0
  64. package/scaffold/skills/dw-ui-discipline/references/state-matrix.md +101 -0
  65. package/scaffold/skills/dw-ui-discipline/references/visual-slop.md +152 -0
  66. package/scaffold/skills/ui-ux-pro-max/LICENSE +0 -21
  67. package/scaffold/skills/ui-ux-pro-max/SKILL.md +0 -659
  68. package/scaffold/skills/ui-ux-pro-max/data/_sync_all.py +0 -414
  69. package/scaffold/skills/ui-ux-pro-max/data/app-interface.csv +0 -31
  70. package/scaffold/skills/ui-ux-pro-max/data/charts.csv +0 -26
  71. package/scaffold/skills/ui-ux-pro-max/data/colors.csv +0 -162
  72. package/scaffold/skills/ui-ux-pro-max/data/design.csv +0 -1776
  73. package/scaffold/skills/ui-ux-pro-max/data/draft.csv +0 -1779
  74. package/scaffold/skills/ui-ux-pro-max/data/google-fonts.csv +0 -1924
  75. package/scaffold/skills/ui-ux-pro-max/data/icons.csv +0 -106
  76. package/scaffold/skills/ui-ux-pro-max/data/landing.csv +0 -35
  77. package/scaffold/skills/ui-ux-pro-max/data/products.csv +0 -162
  78. package/scaffold/skills/ui-ux-pro-max/data/react-performance.csv +0 -45
  79. package/scaffold/skills/ui-ux-pro-max/data/stacks/angular.csv +0 -51
  80. package/scaffold/skills/ui-ux-pro-max/data/stacks/astro.csv +0 -54
  81. package/scaffold/skills/ui-ux-pro-max/data/stacks/flutter.csv +0 -53
  82. package/scaffold/skills/ui-ux-pro-max/data/stacks/html-tailwind.csv +0 -56
  83. package/scaffold/skills/ui-ux-pro-max/data/stacks/jetpack-compose.csv +0 -53
  84. package/scaffold/skills/ui-ux-pro-max/data/stacks/laravel.csv +0 -51
  85. package/scaffold/skills/ui-ux-pro-max/data/stacks/nextjs.csv +0 -53
  86. package/scaffold/skills/ui-ux-pro-max/data/stacks/nuxt-ui.csv +0 -51
  87. package/scaffold/skills/ui-ux-pro-max/data/stacks/nuxtjs.csv +0 -59
  88. package/scaffold/skills/ui-ux-pro-max/data/stacks/react-native.csv +0 -52
  89. package/scaffold/skills/ui-ux-pro-max/data/stacks/react.csv +0 -54
  90. package/scaffold/skills/ui-ux-pro-max/data/stacks/shadcn.csv +0 -61
  91. package/scaffold/skills/ui-ux-pro-max/data/stacks/svelte.csv +0 -54
  92. package/scaffold/skills/ui-ux-pro-max/data/stacks/swiftui.csv +0 -51
  93. package/scaffold/skills/ui-ux-pro-max/data/stacks/threejs.csv +0 -54
  94. package/scaffold/skills/ui-ux-pro-max/data/stacks/vue.csv +0 -50
  95. package/scaffold/skills/ui-ux-pro-max/data/styles.csv +0 -85
  96. package/scaffold/skills/ui-ux-pro-max/data/typography.csv +0 -74
  97. package/scaffold/skills/ui-ux-pro-max/data/ui-reasoning.csv +0 -162
  98. package/scaffold/skills/ui-ux-pro-max/data/ux-guidelines.csv +0 -100
  99. package/scaffold/skills/ui-ux-pro-max/scripts/core.py +0 -262
  100. package/scaffold/skills/ui-ux-pro-max/scripts/design_system.py +0 -1148
  101. package/scaffold/skills/ui-ux-pro-max/scripts/search.py +0 -114
  102. package/scaffold/skills/ui-ux-pro-max/skills/brand/SKILL.md +0 -97
  103. package/scaffold/skills/ui-ux-pro-max/skills/design/SKILL.md +0 -302
  104. package/scaffold/skills/ui-ux-pro-max/skills/design-system/SKILL.md +0 -244
  105. package/scaffold/skills/ui-ux-pro-max/templates/base/quick-reference.md +0 -297
  106. package/scaffold/skills/ui-ux-pro-max/templates/base/skill-content.md +0 -358
  107. package/scaffold/skills/ui-ux-pro-max/templates/platforms/agent.json +0 -21
  108. package/scaffold/skills/ui-ux-pro-max/templates/platforms/augment.json +0 -18
  109. package/scaffold/skills/ui-ux-pro-max/templates/platforms/claude.json +0 -21
  110. package/scaffold/skills/ui-ux-pro-max/templates/platforms/codebuddy.json +0 -21
  111. package/scaffold/skills/ui-ux-pro-max/templates/platforms/codex.json +0 -21
  112. package/scaffold/skills/ui-ux-pro-max/templates/platforms/continue.json +0 -21
  113. package/scaffold/skills/ui-ux-pro-max/templates/platforms/copilot.json +0 -21
  114. package/scaffold/skills/ui-ux-pro-max/templates/platforms/cursor.json +0 -21
  115. package/scaffold/skills/ui-ux-pro-max/templates/platforms/droid.json +0 -21
  116. package/scaffold/skills/ui-ux-pro-max/templates/platforms/gemini.json +0 -21
  117. package/scaffold/skills/ui-ux-pro-max/templates/platforms/kilocode.json +0 -21
  118. package/scaffold/skills/ui-ux-pro-max/templates/platforms/kiro.json +0 -21
  119. package/scaffold/skills/ui-ux-pro-max/templates/platforms/opencode.json +0 -21
  120. package/scaffold/skills/ui-ux-pro-max/templates/platforms/qoder.json +0 -21
  121. package/scaffold/skills/ui-ux-pro-max/templates/platforms/roocode.json +0 -21
  122. package/scaffold/skills/ui-ux-pro-max/templates/platforms/trae.json +0 -21
  123. package/scaffold/skills/ui-ux-pro-max/templates/platforms/warp.json +0 -18
  124. package/scaffold/skills/ui-ux-pro-max/templates/platforms/windsurf.json +0 -21
  125. package/scaffold/skills/webapp-testing/SKILL.md +0 -138
  126. package/scaffold/skills/webapp-testing/assets/test-helper.js +0 -56
  127. /package/scaffold/skills/{webapp-testing → dw-testing-discipline}/references/three-workflow-patterns.md +0 -0
@@ -0,0 +1,170 @@
1
+ # Six guardrails — mandatory when an agent writes tests
2
+
3
+ LLMs have characteristic failure modes when authoring tests. Six guardrails are forcing functions for the most common ones.
4
+
5
+ Every test produced by an agent (via `/dw-run-task`, `/dw-bugfix`, `/dw-autopilot`, or any code-generating flow) must clear all six BEFORE the diff goes to review.
6
+
7
+ ## Guardrail 1 — State the invariant, layer, and host suite first
8
+
9
+ **Failure mode it blocks:** agent writes 200 lines of test code without articulating what the test is supposed to prove or where it belongs.
10
+
11
+ **What the agent must do:**
12
+
13
+ Before any test code, print:
14
+
15
+ ```
16
+ INVARIANT: <one sentence — what behavior the test verifies>
17
+ OWNING_LAYER: <unit | integration | contract | e2e>
18
+ EXISTING_SUITE: <path to the existing test file the new test joins, or "NEW: <reason>">
19
+ ```
20
+
21
+ If the agent can't fill any line, it stops and asks the user — it does NOT invent an invariant.
22
+
23
+ **Why it works:**
24
+ - "Invariant" forces specific behavior naming.
25
+ - "Owning layer" forces Rule 2 (lowest detectable layer).
26
+ - "Existing suite" forces extending coverage rather than spawning orphan files.
27
+
28
+ **Verification:** `/dw-code-review` looks for this 3-line preamble in the PR description or commit body. Missing = REJECTED.
29
+
30
+ ## Guardrail 2 — Real execution somewhere
31
+
32
+ **Failure mode it blocks:** agent writes tests that mock everything. They pass green forever and validate nothing.
33
+
34
+ **What the agent must do:**
35
+
36
+ At SOME layer, the test path must run against real systems before merge:
37
+
38
+ - Pure logic: unit only is sufficient.
39
+ - Code touching DB: at least one integration test with real DB (testcontainers, ephemeral container, dedicated test DB).
40
+ - Code calling external services: a contract test OR a sandbox-account smoke test.
41
+ - UI interactions: at least one E2E run on a real preview environment.
42
+
43
+ **Verification:** PR description lists the real-system runs covering the touched code. If no real-system path covers the change → REJECTED.
44
+
45
+ ## Guardrail 3 — On red, read production first
46
+
47
+ **Failure mode it blocks:** agent sees a test go red and modifies the test until green. Bug ships.
48
+
49
+ **What the agent must do:**
50
+
51
+ When a test fails (its own or pre-existing):
52
+
53
+ 1. Print: `INVESTIGATING FAILURE: <test name>`.
54
+ 2. Read production code in the path that produced the observed value.
55
+ 3. Print: `ANALYSIS: <2-3 sentences — is production wrong, the test wrong, or has the invariant changed?>`.
56
+ 4. Decide:
57
+ - Production wrong → fix production.
58
+ - Test wrong → fix the test AND document the change in the commit body.
59
+ - Invariant changed → update the test AND open an ADR if it's a public-contract change.
60
+
61
+ **Verification:** every commit changing a previously-green test must have an `ANALYSIS:` line. Missing = REJECTED.
62
+
63
+ ## Guardrail 4 — No self-confirming assertions
64
+
65
+ **Failure mode it blocks:** two related shortcuts —
66
+ - Agent writes `mockFn.mockReturnValue('X')` then asserts `expect(mockFn()).toBe('X')`. Proves nothing — the test asserts what the test set up.
67
+ - Agent reaches for `toMatchSnapshot()` whenever unsure what to assert. The snapshot becomes the assertion; drift goes unnoticed.
68
+
69
+ **What the agent must do:**
70
+
71
+ **For mocks:** never assert on a value the test body fed into a mock. Assert on:
72
+ - The OUTPUT of production code that consumed the mock.
73
+ - The SIDE EFFECTS (DB state, network calls, event emissions) caused by production code.
74
+ - The VISIBLE behavior (UI change, log line, response) the user/caller observes.
75
+
76
+ **For snapshots:** before adding `toMatchSnapshot()`, classify the artifact:
77
+ - `PRODUCT_CONTRACT` — a stable contract worth pinning (serialized API output, stored-record schema). Snapshot OK; document the classification in a comment.
78
+ - `IMPLEMENTATION_DETAIL` — HTML structure, internal representation, component tree shape. Snapshot is FORBIDDEN; write specific assertions instead.
79
+
80
+ **Verification:**
81
+ - Mock value flowing directly from setup to assertion without passing through production code → REJECTED.
82
+ - Snapshot in diff without classification comment → REJECTED.
83
+ - Snapshot classified `IMPLEMENTATION_DETAIL` → REJECTED.
84
+
85
+ ## Guardrail 5 — Negative companion
86
+
87
+ **Failure mode it blocks:** agent writes happy-path-only tests. Edge cases, error paths, boundary inputs uncovered.
88
+
89
+ **What the agent must do:**
90
+
91
+ Every positive assertion ships WITH at least one negative companion:
92
+
93
+ - Asserting `createUser(validInput)` succeeds → also assert `createUser(invalidInput)` fails with a specific error.
94
+ - Asserting `parseDate(validString)` returns a Date → also assert `parseDate(invalidString)` throws or returns null.
95
+ - Asserting `transferFunds(...)` succeeds with sufficient balance → also assert it fails with insufficient balance.
96
+
97
+ **Verification:** a PR adding N positive assertions to a public path must add ≥1 negative assertion. Imbalance >3:1 (positive:negative) on a public path → REJECTED.
98
+
99
+ ## Guardrail 6 — Don't expand the surface to test it
100
+
101
+ **Failure mode it blocks:** agent exports internals, adds `*ForTesting` methods, or introduces `process.env.NODE_ENV === 'test'` branches in production code to make the test possible.
102
+
103
+ **What the agent must do:**
104
+
105
+ If the test needs access the production API doesn't grant:
106
+ - **Refactor production for testability** via dependency injection or interface seams.
107
+ - **Emit an observable side effect** the test can verify (event, log line, metric) that production also benefits from.
108
+ - **Use a dedicated test environment** with test credentials, not a backdoor flag.
109
+
110
+ The agent does NOT:
111
+ - Export `_internal` symbols just for tests.
112
+ - Add `// for testing only` methods on classes.
113
+ - Wrap production logic in `if (process.env.NODE_ENV !== 'test')` branches.
114
+
115
+ **Verification:** diff includes new production exports, env checks, or `*ForTesting`-style symbols → REJECTED. Refactor the surface or change the test approach.
116
+
117
+ ## How the six guardrails compose
118
+
119
+ A test that passes all six:
120
+ 1. States the invariant, layer, and host suite up front (Guardrail 1).
121
+ 2. Exercises real systems somewhere in the pipeline (Guardrail 2).
122
+ 3. When red, reads production first and documents the analysis (Guardrail 3).
123
+ 4. Asserts on production's observable output, not its own setup (Guardrail 4).
124
+ 5. Covers failures alongside successes (Guardrail 5).
125
+ 6. Lives inside the production API's existing surface (Guardrail 6).
126
+
127
+ Tests passing all six are worth running. Tests missing any one are more likely to mislead than to help.
128
+
129
+ ## Override procedure
130
+
131
+ To skip a guardrail explicitly:
132
+ 1. State which guardrail is skipped.
133
+ 2. State why in one sentence.
134
+ 3. Add a `// SKIP-GUARDRAIL-N: <reason>` comment in the test.
135
+ 4. Open a follow-up issue tracking the gap.
136
+
137
+ Without all four, the guardrail is enforced.
138
+
139
+ ## Prompt block injected when an agent writes tests
140
+
141
+ ```
142
+ You are about to write tests. Before producing test code, complete the
143
+ 6-guardrail preamble:
144
+
145
+ INVARIANT: ___
146
+ OWNING_LAYER: ___
147
+ EXISTING_SUITE: ___
148
+
149
+ If you cannot complete these three lines, STOP and ask the user for
150
+ the requirement. Do not invent an invariant.
151
+
152
+ Then, while writing tests:
153
+ - Real execution: name the real-system path covering this code.
154
+ - On red: read production first; print ANALYSIS: before changing the test.
155
+ - Mocks: never assert on values fed into a mock.
156
+ - Snapshots: classify PRODUCT_CONTRACT or IMPLEMENTATION_DETAIL — the latter is forbidden.
157
+ - Coverage: every positive assertion needs a negative companion.
158
+ - Production surface: don't export internals or add test-only branches.
159
+
160
+ Tests violating guardrails without explicit SKIP-GUARDRAIL-N comments
161
+ will be REJECTED at review.
162
+ ```
163
+
164
+ `/dw-run-task` and `/dw-bugfix` inject this prompt block before generating test code.
165
+
166
+ ## Why six and not more
167
+
168
+ These are the highest-frequency LLM failure modes observed across multiple projects. Other tendencies exist but are either covered by the positive patterns (e.g., wall-clock waits) or are lower-frequency than these six.
169
+
170
+ If a new failure mode appears that none of the six catches, add a guardrail AND document the failure that motivated it. Don't add guardrails speculatively.
@@ -0,0 +1,336 @@
1
+ # Anti-patterns — 25 smells across 4 families
2
+
3
+ Each anti-pattern names the smell, shows the violation in pseudo-code, gives the fix, and notes how `/dw-code-review` detects it. Agent-specific failure modes are covered separately in `agent-guardrails.md`.
4
+
5
+ ---
6
+
7
+ ## Family A: Fragile to refactor (tests bound to internals, not behavior)
8
+
9
+ ### A1. Implementation-detail selectors
10
+
11
+ **Violation:**
12
+ ```javascript
13
+ await page.click('.btn.btn-primary.checkout-button');
14
+ ```
15
+
16
+ **Fix:** Use `getByRole('button', { name: 'Checkout' })`.
17
+
18
+ **Detection:** Grep for `.click(`, `.querySelector(`, `cy.get('.`, `getByTestId(` with a class-flavored argument.
19
+
20
+ ### A2. Testing internal structure vs observable behavior
21
+
22
+ **Violation:**
23
+ ```javascript
24
+ expect(component.state.cart.items.length).toBe(3);
25
+ ```
26
+
27
+ **Fix:** Assert what the user sees: `expect(await page.getByText('3 items in cart')).toBeVisible()`.
28
+
29
+ **Detection:** Tests that import/inspect internal state, refs, or private fields. Class-based component tests that read `.state` directly.
30
+
31
+ ### A3. Testing private methods directly
32
+
33
+ **Violation:**
34
+ ```javascript
35
+ expect(orderService._calculateTax(...)).toBe(8.5);
36
+ ```
37
+
38
+ **Fix:** Test the public method that uses tax calculation; verify the result. If the private method is independently complex, extract it to a module and test that module's public API.
39
+
40
+ **Detection:** Identifiers starting with `_` accessed from tests.
41
+
42
+ ### A4. Snapshot-as-test (snapshot replacing assertion)
43
+
44
+ **Violation:**
45
+ ```javascript
46
+ expect(rendered).toMatchSnapshot(); // ← only assertion in test
47
+ ```
48
+
49
+ **Fix:** Either:
50
+ 1. Write specific assertions about what the component renders, OR
51
+ 2. Use a snapshot AS A SECONDARY check after specific assertions, with a comment classifying the snapshot as `PRODUCT_CONTRACT` (UI contract worth pinning) — never `IMPLEMENTATION_DETAIL`.
52
+
53
+ **Detection:** Tests where the only assertion is `toMatchSnapshot` or equivalent.
54
+
55
+ ### A5. Vague existence assertions
56
+
57
+ **Violation:**
58
+ ```javascript
59
+ expect(result).toBeTruthy();
60
+ expect(element).toBeDefined();
61
+ expect(button).should('exist');
62
+ ```
63
+
64
+ **Fix:** Assert what you actually want: `expect(result.status).toBe('success')`, `expect(button).toBeEnabled()`, `expect(button).toHaveText('Continue')`.
65
+
66
+ **Detection:** Tests asserting only existence/truthiness without follow-up semantic check.
67
+
68
+ ### A6. Action without assertion
69
+
70
+ **Violation:**
71
+ ```javascript
72
+ test('clicking save works', async () => {
73
+ await page.getByRole('button', { name: 'Save' }).click();
74
+ // ← no assertion. What did "works" mean?
75
+ });
76
+ ```
77
+
78
+ **Fix:** Define what "works" means. Assert the observable outcome: URL changed, modal closed, success message visible, data persisted.
79
+
80
+ **Detection:** Tests with `await x.click()` or `await x.type()` followed by no `expect(...)`.
81
+
82
+ ---
83
+
84
+ ## Family B: Non-deterministic outcomes (tests that flip verdict on the same code)
85
+
86
+ ### A7. Static sleeps / fixed-timeout waits
87
+
88
+ **Violation:**
89
+ ```javascript
90
+ await page.waitForTimeout(2000);
91
+ ```
92
+
93
+ **Fix:** `await expect(page.getByText(/order confirmed/i)).toBeVisible({ timeout: 5000 })` — wait on the actual condition.
94
+
95
+ **Detection:** Grep for `waitForTimeout`, `cy.wait(<number>)`, `sleep(`, `Thread.sleep`, `time.sleep` in test files.
96
+
97
+ ### A8. Test order dependency / hidden shared state
98
+
99
+ **Violation:** Test B passes only after Test A has run because A populates a shared cache or DB row.
100
+
101
+ **Fix:** Each test sets up its own state in `beforeEach`. Verify by running tests with `--shuffle` or `--randomize`.
102
+
103
+ **Detection:** Tests fail when run with `.only`. Tests fail with `--shuffle`. Setup in `beforeAll` instead of `beforeEach`.
104
+
105
+ ### A9. Non-deterministic inputs (clock, RNG, locale)
106
+
107
+ **Violation:**
108
+ ```javascript
109
+ test('today is Monday', () => {
110
+ expect(new Date().getDay()).toBe(1); // fails 6 days a week
111
+ });
112
+ ```
113
+
114
+ **Fix:** Mock the clock (`vi.useFakeTimers()`, `jest.useFakeTimers()`, `freezegun` in Python). Seed RNG. Pin locale.
115
+
116
+ **Detection:** Tests using `new Date()`, `Date.now()`, `Math.random()`, `Intl.DateTimeFormat` without fakes.
117
+
118
+ ---
119
+
120
+ ## Family C: Mock-driven false confidence (tests asserting on their own setup)
121
+
122
+ ### A10. Asserting the mock exists
123
+
124
+ **Violation:**
125
+ ```javascript
126
+ expect(mockFn).toBeDefined();
127
+ ```
128
+
129
+ **Fix:** Don't assert on mock setup. If the mock is wrong, the behavior assertion downstream will fail naturally.
130
+
131
+ **Detection:** Mock functions referenced in assertions without `toHaveBeenCalled` semantics.
132
+
133
+ ### A11. Mock drift
134
+
135
+ **Violation:** Mocked API response set up 6 months ago still returns `{ status: 'OK' }` while the real API now returns `{ ok: true }`.
136
+
137
+ **Fix:** Contract testing (Pact, schemathesis) or periodic recording (msw + real-traffic capture). Re-validate mocks against real APIs quarterly.
138
+
139
+ **Detection:** Tests with mocks that haven't been touched in >90 days against APIs that have changed. Hard to detect in CI; needs explicit contract checks.
140
+
141
+ ### A12. Over-mocking child components
142
+
143
+ **Violation:**
144
+ ```javascript
145
+ vi.mock('./UserAvatar');
146
+ vi.mock('./UserMenu');
147
+ vi.mock('./UserBanner');
148
+ // ... testing nothing real
149
+ ```
150
+
151
+ **Fix:** Mock at boundaries (HTTP, DB, third-party SDKs). Render real children unless they're genuinely expensive or test-irrelevant.
152
+
153
+ **Detection:** Test files with >3 module mocks of internal modules.
154
+
155
+ ### A13. Incomplete mocks (missing fields the code reads)
156
+
157
+ **Violation:**
158
+ ```javascript
159
+ const mockUser = { id: 1 }; // missing email, but code reads user.email
160
+ ```
161
+
162
+ **Fix:** Use a factory that supplies sensible defaults for ALL fields the type/contract declares.
163
+
164
+ **Detection:** Runtime errors like `Cannot read property 'X' of undefined` inside production code under test.
165
+
166
+ ### A14. Mocking wrong level (mocking methods the logic depends on)
167
+
168
+ **Violation:**
169
+ ```javascript
170
+ // Testing OrderService, but mocking its private calculate() method
171
+ const service = new OrderService();
172
+ vi.spyOn(service, 'calculate').mockReturnValue(100);
173
+ expect(service.processOrder(...)).toBe(/* uses mocked 100 */);
174
+ ```
175
+
176
+ You've tested the SCAFFOLD, not the logic.
177
+
178
+ **Fix:** Mock at the EDGE (DB call, HTTP call, time). Let internal logic run.
179
+
180
+ **Detection:** Spies on methods of the System Under Test itself.
181
+
182
+ ---
183
+
184
+ ## Family D: Suite hygiene problems (team and suite-level pathologies)
185
+
186
+ ### A15. Coverage as vanity metric
187
+
188
+ **Violation:** PR comments demanding "you need to hit 90% coverage" with no discussion of what the coverage means.
189
+
190
+ **Fix:** Coverage is a flashlight. Use it to FIND blind spots. Don't optimize for the number.
191
+
192
+ **Detection:** Cultural; visible in PR templates that gate on coverage percentage.
193
+
194
+ ### A16. Happy-path-only coverage
195
+
196
+ **Violation:** Every test exercises the success case. Edge cases, error paths, boundary values uncovered.
197
+
198
+ **Fix:** For each unit, write at minimum: happy path + 1 boundary + 1 invalid input + 1 failure path.
199
+
200
+ **Detection:** Tests where every assertion is positive (`toBe`, `toEqual`) and none is negative (`toThrow`, `toReject`).
201
+
202
+ ### A17. Eternal `beforeAll` / shared setup hiding dependencies
203
+
204
+ **Violation:**
205
+ ```javascript
206
+ beforeAll(async () => {
207
+ await db.users.create([100 users]);
208
+ await db.orders.create([500 orders]);
209
+ });
210
+ ```
211
+
212
+ Tests now SHARE state. Order matters. Cleanup is fragile.
213
+
214
+ **Fix:** `beforeEach` with minimal setup specific to each test.
215
+
216
+ **Detection:** `beforeAll` blocks creating data (vs `beforeAll` blocks doing one-time framework setup like spinning testcontainers).
217
+
218
+ ### A18. Cleanup in `afterEach` (use `beforeEach` instead)
219
+
220
+ **Violation:**
221
+ ```javascript
222
+ afterEach(async () => {
223
+ await db.users.deleteAll();
224
+ });
225
+ ```
226
+
227
+ If a test fails mid-run, cleanup might not run; next test starts dirty.
228
+
229
+ **Fix:** `beforeEach` with explicit setup-from-clean (truncate + seed). Reliable regardless of previous test outcome.
230
+
231
+ **Detection:** `afterEach` blocks doing state reset.
232
+
233
+ ### A19. Magic strings and logic in tests
234
+
235
+ **Violation:**
236
+ ```javascript
237
+ const TIMESTAMP = '2024-01-15T10:30:00Z'; // why?
238
+ expect(formatted).toBe('a long string with embedded specifics');
239
+ ```
240
+
241
+ When the test fails, what was the test's INTENT?
242
+
243
+ **Fix:** Use factories with named defaults. Extract magic values to constants with documenting names. Use snapshot testing for legitimate snapshot cases (with classification).
244
+
245
+ **Detection:** Test files with ≥10 string literals not bound to a named variable.
246
+
247
+ ### A20. Testing against third-party sites you don't control
248
+
249
+ **Violation:**
250
+ ```javascript
251
+ test('Google homepage loads', async ({ page }) => {
252
+ await page.goto('https://google.com');
253
+ expect(await page.title()).toContain('Google');
254
+ });
255
+ ```
256
+
257
+ You're testing Google's availability, not your code.
258
+
259
+ **Fix:** Mock the third party. Use a wiremock or msw to fake their responses. If you must call them, do it in a separate "external dependencies up" smoke test, not unit/integration.
260
+
261
+ **Detection:** External URLs in test code outside designated smoke tests.
262
+
263
+ ### A21. Quarantine-as-cemetery
264
+
265
+ **Violation:**
266
+ ```javascript
267
+ test.skip('flaky on CI sometimes', () => { /* ... */ });
268
+ // commented 8 months ago, no owner, no fix-by date
269
+ ```
270
+
271
+ **Fix:** Every skip/quarantine has a NAMED OWNER and a FIX-BY DATE. Tracking issue exists. PR that introduces the skip says exactly when the test gets fixed.
272
+
273
+ **Detection:** Skipped tests without comments/labels naming owner and date.
274
+
275
+ ### A22. Retry-as-fix (auto-retry hiding real bugs)
276
+
277
+ **Violation:**
278
+ ```javascript
279
+ // jest.config or playwright.config
280
+ retries: 3,
281
+ ```
282
+
283
+ A flaky test is a SIGNAL. Retrying until green hides it.
284
+
285
+ **Fix:** When a test is flaky, FIX IT (probably a race condition or non-deterministic input). Quarantine if you can't fix immediately. Never just retry.
286
+
287
+ **Detection:** CI config with retry counts. Test runners showing "1 retry succeeded" badges.
288
+
289
+ ### A23. Duplicate tests across pyramid layers
290
+
291
+ **Violation:** Same scenario tested at unit, integration, AND E2E. Triple maintenance, no triple value.
292
+
293
+ **Fix:** Apply Law 2 — lowest layer wins. Drop higher-layer duplicates.
294
+
295
+ **Detection:** Search for the same scenario name across `tests/unit`, `tests/integration`, `tests/e2e`.
296
+
297
+ ### A24. Weakening tests to make them pass
298
+
299
+ **Violation:**
300
+ ```diff
301
+ - expect(orders.length).toBe(5);
302
+ + expect(orders.length).toBeGreaterThan(0);
303
+ ```
304
+
305
+ The "fix" makes the test useless.
306
+
307
+ **Fix:** Read Law 3. Fix production OR document WHY the assertion is weaker.
308
+
309
+ **Detection:** PR diff shows assertion relaxation without commit body explanation.
310
+
311
+ ### A25. Mock-driven confidence (test asserts on its own setup)
312
+
313
+ **Violation:**
314
+ ```javascript
315
+ const mock = vi.fn().mockReturnValue('hello');
316
+ expect(mock()).toBe('hello');
317
+ ```
318
+
319
+ You wrote `hello` in the mock. You asserted `hello`. You proved nothing.
320
+
321
+ **Fix:** Assert on the OUTPUT of the production code that consumed the mock — not on the mock itself.
322
+
323
+ **Detection:** Tests asserting equality between a value the test body created and a value the test body retrieved.
324
+
325
+ ---
326
+
327
+ ## How `/dw-code-review` uses this catalog
328
+
329
+ For each diff hunk under a test path:
330
+ 1. Run regex scans for the patterns flagged "Detection" above.
331
+ 2. Each hit becomes a finding with severity from this skill's `dw-review-rigor` integration.
332
+ 3. Hits classified as Brittleness/Flakiness/Mock-misuse → severity `high`.
333
+ 4. Hits classified as Process → severity `medium`.
334
+ 5. Hits where the SAME test has multiple patterns → severity `critical` (suite-health smell, not just one test).
335
+
336
+ A PR with ≥1 `high` test anti-pattern that lacks ADR justification gets REJECTED.
@@ -0,0 +1,128 @@
1
+ # Six core rules — expanded with examples
2
+
3
+ The rules are short for memorization. Each carries nuance that matters in practice.
4
+
5
+ ## Rule 1: Test the behavior, never the mock
6
+
7
+ **What it means:** the test asserts what the system DOES from the caller's perspective. It does not assert that internal call X was made with internal argument Y.
8
+
9
+ **Why it matters:** a test bound to internal calls breaks the day you refactor — even when behavior didn't change. The "test is red, behavior is fine" experience erodes trust. Soon no one runs the suite.
10
+
11
+ **Violation:**
12
+
13
+ ```javascript
14
+ // BAD — asserting on mock internals
15
+ test('createOrder calls inventory.reserve', () => {
16
+ const inventory = { reserve: vi.fn() };
17
+ createOrder({ items: [...] }, inventory);
18
+ expect(inventory.reserve).toHaveBeenCalledWith(items, 'reserve');
19
+ });
20
+ ```
21
+
22
+ You've asserted that `createOrder` USES the inventory adapter a specific way. The refactor that consolidates `reserve` into `commit-with-reservation` breaks this even though the order still gets created.
23
+
24
+ **Correct version:**
25
+
26
+ ```javascript
27
+ // GOOD — asserting behavior
28
+ test('createOrder reserves inventory before confirming', async () => {
29
+ const result = await createOrder({ items: [...] });
30
+ expect(result.status).toBe('confirmed');
31
+ expect(await getInventoryFor(items[0].sku)).toBe(originalStock - 1);
32
+ });
33
+ ```
34
+
35
+ Now the test cares about the OUTCOME (inventory decremented, order confirmed), not the path.
36
+
37
+ ## Rule 2: Push each test to the lowest layer that can detect the defect
38
+
39
+ **What it means:** if a unit test can catch the bug, use a unit. If only an integration test can, integration. If only an end-to-end run can, E2E.
40
+
41
+ **Why it matters:** lower layers run faster, fail more precisely, isolate the cause better. A bug in pure logic caught at unit takes 50ms and points at the exact function. The same bug at E2E takes 30 seconds and tells you "checkout failed."
42
+
43
+ **The pyramid resolved:**
44
+
45
+ | Layer | Catches | Speed | Cost |
46
+ |-------|---------|-------|------|
47
+ | Unit | Pure logic, math, parsing, formatters | <100ms | low |
48
+ | Integration | Module composition, DB queries, HTTP handlers | 500ms–5s | medium |
49
+ | Contract | Producer/consumer agreement at API boundary | 1–10s | medium |
50
+ | E2E | User journey across multiple services | 10s–60s | high |
51
+
52
+ **Rule of thumb:**
53
+ - If you can write a unit test, do so.
54
+ - If unit can't reach it (needs DB, queue, real HTTP), write integration.
55
+ - E2E only for journeys NO lower layer can detect: browser-renders-correctly, third-party-callback-arrives, multi-step session state.
56
+
57
+ ## Rule 3: When a test fails, read production first — change the test only with documented justification
58
+
59
+ **What it means:** a red test is a signal. The first question is "what's wrong with production?" — not "why is the test wrong?"
60
+
61
+ **Why it matters:** tests are weakened to pass far more often than they should be. "The behavior is fine; the test is too strict" is the slope that leaves a green suite full of meaningless assertions.
62
+
63
+ **Process when a test goes red:**
64
+
65
+ 1. **Read the failure message.** What invariant did the test claim? What did it observe?
66
+ 2. **Read production code** in the path that produces the observation.
67
+ 3. **Decide which is wrong.** If production violates the invariant, fix production. If the test mis-states the invariant, document WHY before relaxing.
68
+ 4. **Commit the analysis** in the test's commit message or PR body. "Relaxed assertion from X to Y because `<reason>`" is auditable; "fix test" is not.
69
+
70
+ **Anti-pattern:** re-run the test until green. Auto-retry on flake. Add `.only` to skip the rest.
71
+
72
+ ## Rule 4: Real systems gate the merge. Mocks isolate; they do not validate.
73
+
74
+ **What it means:** before code merges to main, at least ONE test path exercised real systems — real DB, real route, real external integration in a sandbox or test account. Mocks are fine for fast unit feedback; they cannot decide "safe to ship."
75
+
76
+ **Why it matters:** mock drift is real. The mocked HTTP response from 3 months ago no longer matches the actual API. Tests pass; production fails on first real call.
77
+
78
+ **Practical pattern:**
79
+
80
+ - Unit tests: mock the world; run on every keystroke / commit.
81
+ - Integration tests: real local DB (testcontainers, in-memory if equivalent); run on every PR.
82
+ - Contract tests: real producer/consumer agreement check; run on every PR.
83
+ - E2E: real preview environment with real services; run on PRs before merge to main.
84
+
85
+ The discipline: no merge without a green E2E (or equivalent real-system check) for the touched path.
86
+
87
+ ## Rule 5: Coverage is a flashlight; mutation score is a quality probe. Neither is a target.
88
+
89
+ **What it means:**
90
+ - **Coverage** tells you what lines executed. Useful as a NEGATIVE signal — 30% coverage means lots of dark code. Useless as a positive signal — 95% coverage with weak assertions is decorative.
91
+ - **Mutation score** introduces small bugs (mutations) and measures whether tests catch them. A high mutation score means tests actually probe behavior, not just execute lines.
92
+ - Neither should be a number you optimize for. They're diagnostics.
93
+
94
+ **Anti-pattern:** "We need 90% coverage to merge." Coverage as a gate produces tests written to pass the gate, not to find bugs.
95
+
96
+ **Healthier framing:** "What lines in the touched diff are NOT covered? Why?" Sometimes the answer is "we don't care, it's logging." Sometimes it's "actually that's a critical branch — add a test."
97
+
98
+ ## Rule 6: No test-only methods, branches, or flags leak into production code
99
+
100
+ **What it means:** production code should not have `if (process.env.NODE_ENV === 'test') { ... }` branches, `// for testing only` methods exposed on classes, or internals exported just for assertions.
101
+
102
+ **Why it matters:** production code carrying test-only logic is test decorations leaking into the artifact users run. Bug surface grows; the test environment diverges from production.
103
+
104
+ **Correct patterns:**
105
+
106
+ - Need to inject a dependency for testing? Use constructor injection / dependency injection.
107
+ - Need to assert on internal state? Add a logging hook or event emission that production also benefits from.
108
+ - Need to bypass auth in tests? Use a dedicated test environment with test credentials, not a backdoor flag.
109
+
110
+ **Tells:**
111
+ - `// only used in tests` comments.
112
+ - `*ForTesting` suffix on methods.
113
+ - `vi.spyOn(module, '_internal')` accessing underscore-prefixed members.
114
+ - `process.env.E2E_MODE` reaching into production runtime decisions.
115
+
116
+ If you see these, the test design is wrong. Refactor production to be testable; don't add backdoors.
117
+
118
+ ## Putting the rules together
119
+
120
+ A healthy test:
121
+ 1. Asserts behavior visible to a caller (Rule 1).
122
+ 2. Sits at the lowest layer that can prove that behavior (Rule 2).
123
+ 3. When red, sends you to read production code (Rule 3).
124
+ 4. Has a sibling exercising real systems somewhere in the pipeline (Rule 4).
125
+ 5. Survives a mutation in the code it claims to cover (Rule 5).
126
+ 6. Has zero footprint in production code (Rule 6).
127
+
128
+ Any test failing ≥2 of these is technical debt accumulating. `/dw-code-review` flags them.