@rune-kit/rune 2.3.3 → 2.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/README.md +86 -17
  2. package/compiler/__tests__/pack-split.test.js +141 -1
  3. package/compiler/__tests__/parser.test.js +147 -55
  4. package/compiler/__tests__/scripts-bundling.test.js +283 -0
  5. package/compiler/__tests__/skill-index.test.js +218 -0
  6. package/compiler/__tests__/tier-override.test.js +41 -0
  7. package/compiler/adapters/antigravity.js +71 -53
  8. package/compiler/adapters/codex.js +4 -0
  9. package/compiler/adapters/cursor.js +4 -0
  10. package/compiler/adapters/generic.js +4 -0
  11. package/compiler/adapters/openclaw.js +4 -0
  12. package/compiler/adapters/opencode.js +4 -0
  13. package/compiler/adapters/windsurf.js +4 -0
  14. package/compiler/bin/rune.js +355 -355
  15. package/compiler/doctor.js +11 -1
  16. package/compiler/emitter.js +678 -386
  17. package/compiler/parser.js +267 -247
  18. package/compiler/transforms/scripts-path.js +18 -0
  19. package/extensions/zalo/PACK.md +20 -1
  20. package/extensions/zalo/references/conversation-management.md +214 -0
  21. package/extensions/zalo/references/eval-scenarios.md +157 -0
  22. package/extensions/zalo/references/listen-mode.md +237 -0
  23. package/extensions/zalo/references/mcp-production.md +274 -0
  24. package/extensions/zalo/references/multi-account-proxy.md +224 -0
  25. package/extensions/zalo/references/vietqr-banking.md +160 -0
  26. package/hooks/hooks.json +12 -0
  27. package/hooks/intent-router/index.cjs +108 -0
  28. package/hooks/pre-tool-guard/index.cjs +177 -68
  29. package/package.json +63 -64
  30. package/skills/brainstorm/SKILL.md +2 -0
  31. package/skills/cook/SKILL.md +661 -648
  32. package/skills/debug/SKILL.md +394 -392
  33. package/skills/deploy/SKILL.md +2 -0
  34. package/skills/fix/SKILL.md +283 -281
  35. package/skills/marketing/SKILL.md +3 -0
  36. package/skills/onboard/SKILL.md +7 -0
  37. package/skills/plan/SKILL.md +344 -342
  38. package/skills/preflight/SKILL.md +362 -360
  39. package/skills/review/SKILL.md +491 -489
  40. package/skills/scout/SKILL.md +1 -0
  41. package/skills/sentinel/SKILL.md +319 -296
  42. package/skills/sentinel/references/auth-crypto-reference.md +192 -0
  43. package/skills/sentinel/references/desktop-security.md +201 -0
  44. package/skills/sentinel/references/supply-chain.md +160 -0
  45. package/skills/session-bridge/SKILL.md +1 -0
  46. package/skills/slides/SKILL.md +142 -0
  47. package/skills/slides/scripts/build-deck.js +158 -0
  48. package/skills/team/SKILL.md +1 -0
  49. package/skills/test/SKILL.md +587 -585
  50. package/skills/verification/SKILL.md +1 -0
  51. package/skills/watchdog/SKILL.md +2 -0
  52. package/docs/ANTIGRAVITY-GAP-ANALYSIS.md +0 -369
  53. package/docs/ARCHITECTURE.md +0 -332
  54. package/docs/COMMUNITY-PACKS.md +0 -109
  55. package/docs/CONTRIBUTING-L4.md +0 -215
  56. package/docs/CROSS-IDE-ANALYSIS.md +0 -164
  57. package/docs/EXTENSION-TEMPLATE.md +0 -126
  58. package/docs/MESH-RULES.md +0 -34
  59. package/docs/MULTI-PLATFORM.md +0 -804
  60. package/docs/SKILL-DEPTH-AUDIT.md +0 -191
  61. package/docs/SKILL-TEMPLATE.md +0 -118
  62. package/docs/TRADE-MATRIX.md +0 -327
  63. package/docs/VERSIONING.md +0 -91
  64. package/docs/VISION.md +0 -263
  65. package/docs/assets/demo-subtitles.srt +0 -215
  66. package/docs/assets/end-card.html +0 -276
  67. package/docs/assets/mesh-diagram.html +0 -654
  68. package/docs/assets/thumbnail.html +0 -295
  69. package/docs/guides/cli.md +0 -403
  70. package/docs/guides/index.html +0 -1450
  71. package/docs/index.html +0 -1005
  72. package/docs/references/claudekit-analysis.md +0 -414
  73. package/docs/references/voltagent-analysis.md +0 -189
  74. package/docs/script.js +0 -495
  75. package/docs/skills/index.html +0 -832
  76. package/docs/style.css +0 -958
  77. package/docs/video-demo-plan.md +0 -172
@@ -1,585 +1,587 @@
1
- ---
2
- name: test
3
- description: "TDD test writer. Writes failing tests FIRST (red), then verifies they pass after implementation (green). Covers unit, integration, and e2e tests."
4
- metadata:
5
- author: runedev
6
- version: "1.1.0"
7
- layer: L2
8
- model: sonnet
9
- group: development
10
- tools: "Read, Write, Edit, Bash, Glob, Grep"
11
- ---
12
-
13
- # test
14
-
15
- <HARD-GATE>
16
- Tests define the EXPECTED BEHAVIOR. They MUST be written BEFORE implementation code.
17
- If tests pass without implementation → the tests are wrong. Rewrite them.
18
- The only exception: when retrofitting tests for existing untested code.
19
-
20
- THE IRON LAW: Write code before test? DELETE IT. Start over.
21
- - Do NOT keep it as "reference"
22
- - Do NOT "adapt" it while writing tests
23
- - Do NOT look at it to "inform" test design
24
- - Delete means delete. `git checkout -- <file>` or remove the changes entirely.
25
- This is not negotiable. This is not optional. "But I already wrote it" is a sunk cost fallacy.
26
-
27
- ROLE BOUNDARY: Test writes TEST FILES only. NEVER modify source/implementation files.
28
- - Do NOT "quickly fix" a broken import in source to make tests run
29
- - Do NOT refactor source code to be "more testable"
30
- - Do NOT add missing exports to source files
31
- - If source needs changes hand off to `rune:fix`. Test's job ends at the test file.
32
- This separation ensures test never writes code biased toward passing its own tests.
33
- </HARD-GATE>
34
-
35
- ## Instructions
36
-
37
- ### Phase 1: Understand What to Test
38
-
39
- 1. Read the implementation plan or task description carefully
40
- 2. Use `Glob` to find existing test files: `**/*.test.*`, `**/*.spec.*`, `**/test_*`
41
- 3. Use `Read` on 2-3 existing test files to understand:
42
- - Test framework in use
43
- - File naming convention (e.g., `foo.test.ts` mirrors `foo.ts`)
44
- - Test directory structure (co-located vs `__tests__/` vs `tests/`)
45
- - Assertion style and patterns
46
- 4. Use `Glob` to find the source file(s) being tested
47
-
48
- ```
49
- TodoWrite: [
50
- { content: "Understand scope and find existing test patterns", status: "in_progress" },
51
- { content: "Detect test framework and conventions", status: "pending" },
52
- { content: "Write failing tests (RED phase)", status: "pending" },
53
- { content: "Run tests verify they FAIL", status: "pending" },
54
- { content: "After implementation: verify tests PASS (GREEN phase)", status: "pending" }
55
- ]
56
- ```
57
-
58
- ### Phase 2: Detect Test Framework
59
-
60
- Use `Glob` to find config files and identify the framework:
61
-
62
- - `jest.config.*` or `"jest"` key in `package.json` Jest
63
- - `vitest.config.*` or `"vitest"` key in `package.json` → Vitest
64
- - `pytest.ini`, `[tool.pytest.ini_options]` in `pyproject.toml` → pytest
65
- - **Async check**: If pytest detected AND source files contain `async def`:
66
- - Check if `pytest-asyncio` is in dependencies (`pyproject.toml [project.dependencies]` or `[project.optional-dependencies]`)
67
- - Check if `asyncio_mode` is set in `[tool.pytest.ini_options]` (values: `auto`, `strict`, or absent)
68
- - If async code exists but no `asyncio_mode` configured → **WARN**: "pytest-asyncio not configured. Async tests may silently pass without executing async code. Recommend adding `asyncio_mode = \"auto\"` to `[tool.pytest.ini_options]` in pyproject.toml."
69
- - `Cargo.toml` with `#[cfg(test)]` pattern built-in `cargo test`
70
- - `*_test.go` files present built-in `go test`
71
- - `cypress.config.*`Cypress (E2E)
72
- - `playwright.config.*`Playwright (E2E)
73
-
74
- **Verification gate**: Framework identified before writing any test code.
75
-
76
- ### Phase 3: Write Failing Tests
77
-
78
- Use `Write` to create test files following the detected conventions:
79
-
80
- 1. Mirror source file location: if source is `src/auth/login.ts`, test is `src/auth/login.test.ts`
81
- 2. Structure tests with clear `describe` / `it` blocks (or language equivalent):
82
- - `describe('Feature name')`
83
- - `it('should [expected behavior] when [condition]')`
84
- 3. Cover all three categories:
85
- - **Happy path**: valid inputs, expected success output
86
- - **Edge cases**: empty input, boundary values, large input
87
- - **Error cases**: invalid input, missing data, network failure simulation
88
-
89
- 4. Use proper assertions. Do NOT use implementation details — test behavior:
90
- - Jest/Vitest: `expect(result).toBe(expected)`
91
- - pytest: `assert result == expected`
92
- - Rust: `assert_eq!(result, expected)`
93
- - Go: `if result != expected { t.Errorf(...) }`
94
-
95
- 5. For async code: use `async/await` or pytest `@pytest.mark.asyncio`
96
-
97
- #### Python Async Tests (pytest-asyncio)
98
-
99
- When writing tests for async Python code:
100
-
101
- 1. **Verify setup before writing tests**:
102
- - Confirm `pytest-asyncio` is in project dependencies
103
- - Confirm `asyncio_mode` is set in `pyproject.toml` `[tool.pytest.ini_options]` (recommend `"auto"`)
104
- - If neither is configured, warn the caller and suggest setup before proceeding
105
-
106
- 2. **Writing async test functions**:
107
- - With `asyncio_mode = "auto"`: just write `async def test_something():` — no decorator needed
108
- - With `asyncio_mode = "strict"`: every async test needs `@pytest.mark.asyncio`
109
- - Without asyncio_mode set: always use `@pytest.mark.asyncio` decorator explicitly
110
-
111
- 3. **Async fixtures**:
112
- - Use `@pytest_asyncio.fixture` (NOT `@pytest.fixture`) for async setup/teardown
113
- - Scope rules: async fixtures default to `function` scope — use `scope="session"` carefully with async
114
-
115
- 4. **Common pitfalls**:
116
- - Tests that `pass` without `await` — they run but don't execute the async path
117
- - Missing `pytest-asyncio` makes `async def test_*` silently pass as empty coroutines
118
- - Mixing sync and async fixtures can cause event loop errors
119
-
120
- ### Phase 4: Run Tests Verify They FAIL (RED)
121
-
122
- Use `Bash` to run ONLY the newly created test files (not full suite):
123
-
124
- - **Jest**: `npx jest path/to/test.ts --no-coverage`
125
- - **Vitest**: `npx vitest run path/to/test.ts`
126
- - **pytest**: `pytest path/to/test_file.py -v` (if async tests and no `asyncio_mode` in config: add `--asyncio-mode=auto`)
127
- - **Rust**: `cargo test test_module_name`
128
- - **Go**: `go test ./path/to/package/... -run TestFunctionName`
129
-
130
- **Hard gate**: ALL new tests MUST fail at this point.
131
-
132
- - If ANY test passes before implementation exists that test is not testing real behavior. Rewrite it to be stricter.
133
- - If tests fail with import/syntax errors (not assertion errors) → fix the test code, re-run
134
-
135
- ### Phase 5: After Implementation Verify Tests PASS (GREEN)
136
-
137
- After `rune:fix` writes implementation code, run the same test command again:
138
-
139
- 1. ALL tests in the new test files MUST pass
140
- 2. Run the full test suite with `Bash` to check for regressions:
141
- - `npm test`, `pytest`, `cargo test`, `go test ./...`
142
- 3. If any test fails: report clearly which test, what was expected, what was received
143
- 4. If an existing test now fails (regression): escalate to `rune:debug`
144
-
145
- **Verification gate**: 100% of new tests pass AND 0 regressions in existing tests.
146
-
147
- ### Phase 6: Coverage Check
148
-
149
- After GREEN phase, call `verification` to check coverage threshold (80% minimum):
150
-
151
- - If coverage drops below 80%: identify uncovered lines, write additional tests
152
- - Report coverage gaps with file:line references
153
-
154
- ### Phase 6.5: Diff-Aware Mode (optional)
155
-
156
- When invoked with `mode: "diff-aware"` or by `cook` after implementation:
157
-
158
- 1. Run `git diff main --name-only` to get changed files
159
- 2. For each changed file, trace its **blast radius**: what imports it? what routes does it serve? what components render it?
160
- 3. Map changed files affected routes/endpoints/pages
161
- 4. Prioritize tests: files with most downstream dependents get tested first
162
- 5. Generate targeted test commands that cover ONLY affected paths — skip unchanged modules
163
-
164
- This mode is valuable for large codebases where running the full suite is slow. It answers: "what could this diff have broken?"
165
-
166
- ```
167
- Input: git diff main --name-only
168
- Output: Prioritized test plan targeting only affected paths
169
- ```
170
-
171
- ## Test Types — 4-Layer Methodology
172
-
173
- Tests are organized in 4 layers. Each layer catches a different failure class. Higher layers are slower but catch integration issues lower layers miss.
174
-
175
- | Layer | Type | What It Catches | Framework | Speed |
176
- |-------|------|-----------------|-----------|-------|
177
- | L1 | **Unit** | Logic bugs, boundary violations, pure function errors | jest/vitest/pytest/cargo test | Fast |
178
- | L2 | **Integration** | API contract breaks, DB query errors, service interaction failures | supertest/httpx/reqwest | Medium |
179
- | L3 | **True Backend** | Real tool/service output correctness (not just exit 0) | Same + real software invocation | Medium-Slow |
180
- | L4 | **E2E / Subprocess** | Full workflow from user/agent perspective, installed app works | Playwright/Cypress/subprocess | Slow |
181
-
182
- **Layer rules:**
183
- - **L1 (Unit)**: Synthetic data, no external deps. Every function tested in isolation. Fast, deterministic, CI-friendly
184
- - **L2 (Integration)**: Tests service boundaries — API endpoints, DB operations, message queues. May need test DB or mock server
185
- - **L3 (True Backend)**: **Invokes the REAL tool/service** and verifies output programmatically. No graceful degradation if the dependency isn't installed, tests FAIL (not skip). Verify: magic bytes, file size > 0, content structure. Print artifact paths for manual inspection
186
- - **L4 (E2E/Subprocess)**: Tests the installed command/app via subprocess or browser automation. Full user workflow: input process → output → verify
187
-
188
- **"No graceful degradation" rule** (L3/L4): Hard dependencies MUST be installed. Tests MUST NOT skip or produce fake results when the dependency is missing. A silently skipping test is worse than a loudly failing test.
189
-
190
- Additional modes:
191
-
192
- | Type | When | Speed |
193
- |------|------|-------|
194
- | Regression | After bug fixes | Fast |
195
- | Diff-aware | After implementation, large codebases (Phase 6.5) | Fast (targeted) |
196
-
197
- ## TEST.md Test Plan + Results Document
198
-
199
- For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` in the test directory. This is BOTH a planning doc (written BEFORE tests) and results doc (appended AFTER tests pass).
200
-
201
- ### Before writing tests write the plan:
202
- ```markdown
203
- # Test Plan: [Feature Name]
204
-
205
- ## Test Inventory
206
- - `test_core.py`: ~XX unit tests planned (L1)
207
- - `test_integration.py`: ~XX integration tests planned (L2)
208
- - `test_e2e.py`: ~XX E2E tests planned (L3/L4)
209
-
210
- ## Unit Test Plan (L1)
211
- | Module | Functions | Edge Cases | Est. Tests | Req IDs |
212
- |--------|-----------|------------|------------|---------|
213
- | `core/auth.py` | login, register, refresh | expired token, invalid creds, rate limit | 12 | REQ-001, REQ-003 |
214
-
215
- ## E2E Scenarios (L3/L4)
216
- | Workflow | Simulates | Operations | Verified | Req IDs |
217
- |----------|-----------|------------|----------|---------|
218
- | User signup | New user onboarding | register → verify → login | Token valid, profile created | REQ-005 |
219
-
220
- ## Realistic Workflow Scenarios
221
- - **[Name]**: [Step 1] → [Step 2] → verify [output properties]
222
- ```
223
-
224
- ### After tests pass — append results:
225
- ```markdown
226
- ## Test Results
227
- [Paste full `pytest -v --tb=no` or `npm test` output]
228
-
229
- ## Summary
230
- - Total: XX | Passed: XX | Failed: 0
231
- - Execution time: X.Xs | Coverage: XX%
232
-
233
- ## Requirement Coverage
234
- | Req ID | Test File(s) | Status |
235
- |--------|-------------|--------|
236
- | REQ-001 | `test_auth.py::test_login` | Covered |
237
- | REQ-002 | — | ❌ Not covered |
238
-
239
- ## Gaps
240
- - [Areas not covered and why]
241
- ```
242
-
243
- **Why TEST.md**: Planning tests before code catches missing edge cases early. Appending results creates permanent evidence. One document = complete testing story.
244
-
245
- ## Skill Behavior Tests (Eval Scenarios)
246
-
247
- For testing SKILL.md behavior (not code), use **Eval Scenarios** — unit tests for skill files, not code files.
248
-
249
- ### Eval Scenario Format
250
-
251
- ```markdown
252
- ## Eval: E[NN] — [scenario name]
253
-
254
- ### Prompt
255
- [The exact situation/message an agent receives]
256
-
257
- ### Expected Reasoning
258
- [Step-by-step reasoning the agent SHOULD follow]
259
-
260
- ### Must Include
261
- - [Assertion 1: what the output MUST contain or do]
262
- - [Assertion 2]
263
-
264
- ### Must NOT
265
- - [Anti-pattern 1: what the output MUST NOT do]
266
- - [Anti-pattern 2]
267
-
268
- ### Category
269
- happy-path | adversarial | edge-case | jailbreak | credential-leak
270
- ```
271
-
272
- ### Eval Coverage Requirements
273
-
274
- A skill is **behavior-tested** when it has evals covering:
275
-
276
- | Category | Min Evals | Purpose |
277
- |----------|-----------|---------|
278
- | Happy path | 1 | Core workflow executes correctly |
279
- | Edge case | 1 | Empty input, missing context, unusual state |
280
- | Adversarial | 1 | Time pressure, sunk cost, authority pressure |
281
- | Jailbreak / injection | 1 | Prompt injection attempt, "ignore instructions" |
282
-
283
- **Minimum**: 4 evals per skill (1 per category). Security-critical skills (sentinel, safeguard): 8+ evals.
284
-
285
- ### Eval Storage
286
-
287
- Save eval files as `skills/<name>/evals.md`. Each eval is a numbered scenario (E01–E24 range). skill-forge Phase 7 checks for evals presence before ship.
288
-
289
-
290
- ## Error Recovery
291
-
292
- - If test framework not found: ask calling skill to specify, or check `package.json` `devDependencies`
293
- - If `Write` to test file fails: check if directory exists, create it first with `Bash mkdir -p`
294
- - If tests error on import (module not found): check that source file path is correct, adjust imports
295
- - If `Bash` test runner hangs beyond 120 seconds: kill and report as TIMEOUT
296
-
297
- ## Called By (inbound)
298
-
299
- - `cook` (L1): Phase 3 TEST — write tests first
300
- - `fix` (L2): verify fix passes tests
301
- - `review` (L2): untested edge case found write test for it
302
- - `deploy` (L2): pre-deployment full test suite
303
- - `preflight` (L2): run targeted regression tests on affected code
304
- - `surgeon` (L2): verify refactored code
305
- - `launch` (L1): pre-deployment test suite
306
- - `safeguard` (L2): writing characterization tests for legacy code
307
- - `review-intake` (L2): write tests for issues identified during review intake
308
-
309
- ## Calls (outbound)
310
-
311
- - `verification` (L3): Phase 6 — coverage check (80% minimum threshold)
312
- - `browser-pilot` (L3): Phase 4 — e2e and visual testing for UI flows
313
- - `debug` (L2): Phase 5when existing test regresses unexpectedly
314
-
315
- ## Data Flow
316
-
317
- ### Feeds Into →
318
-
319
- - `cook` (L1): test results (pass/fail/coverage) cook's Phase 5 quality gate evidence
320
- - `completion-gate` (L3): test runner stdout → evidence for "tests pass" claims
321
- - `fix` (L2): failing test outputfix's target (what to make green)
322
-
323
- ### Fed By
324
-
325
- - `plan` (L2): phase file test tasks → test's RED phase targets (what to test)
326
- - `review` (L2): untested edge cases found during review → new test targets
327
- - `fix` (L2): implemented code → test's GREEN phase verification target
328
-
329
- ### Feedback Loops
330
-
331
- - `test` `fix`: test writes failing tests (RED) → fix implements to pass → test verifies (GREEN) → if new failures emerge, loop continues
332
- - `test` ↔ `debug`: test discovers regression → debug diagnoses root cause → test writes regression test to prevent recurrence
333
-
334
- ## Anti-Rationalization Table
335
-
336
- | Excuse | Reality |
337
- |---|---|
338
- | "Too simple to need tests first" | Simple code breaks. Test takes 30 seconds. Write it first. |
339
- | "I'll write tests after — same result" | Tests-after = "what does this do?" Tests-first = "what SHOULD this do?" Completely different. |
340
- | "I already wrote the code, let me just add tests" | Iron Law: delete the code. Start over with tests. Sunk cost is not an argument. |
341
- | "Tests after achieve the same goals" | They don't. Tests-after are biased by the implementation you just wrote. |
342
- | "It's about spirit not ritual" | Violating the letter IS violating the spirit. Write the test first. |
343
- | "I mentally tested it" | Mental testing is not testing. Run the command, show the output. |
344
- | "This is different because..." | It's not. Write the test first. |
345
-
346
- ## Advanced: Oracle-Injection E2E Testing
347
-
348
- For **data pipelines, AI workflows, and multi-stage processing** where comparing full output structures is impractical, use oracle injection:
349
-
350
- 1. **Generate a UUID oracle token**: `const oracle = crypto.randomUUID()`
351
- 2. **Inject into synthetic input**: embed the oracle in realistic test data that flows through the pipeline
352
- 3. **Run the full pipeline**: input all stages → output
353
- 4. **Search for oracle in output**: if found data flowed end-to-end correctly
354
-
355
- ```
356
- // Example: testing a document processing pipeline
357
- const oracle = "ORACLE-" + crypto.randomUUID();
358
- const testDoc = `Meeting notes: discussed ${oracle} integration timeline`;
359
- const result = await pipeline.process(testDoc);
360
- assert(result.output.includes(oracle), "Oracle not found pipeline lost data");
361
- ```
362
-
363
- **When to use**: E2E tests for pipelines with 3+ stages, LLM-based processing, ETL workflows, or any system where output structure is complex/non-deterministic but data preservation is critical.
364
-
365
- **When NOT to use**: Unit tests, simple CRUD, or when exact output comparison is feasible.
366
-
367
-
368
- ## Spec→Test Traceability
369
-
370
- When a plan with acceptance criteria exists (`.rune/features/<name>/plan.md` or phase file), every criterion MUST map to at least one test case.
371
-
372
- ```
373
- Plan Acceptance Criteria → Test Case → Implementation
374
-
375
- AC-1: "User can reset password via email" test_password_reset_sends_email()
376
- AC-2: "Rate limit: max 3 reset attempts/hour" → test_password_reset_rate_limit()
377
- AC-3: "Expired tokens rejected" → test_expired_reset_token_rejected()
378
- ```
379
-
380
- **Validation step** (after writing tests): Cross-check plan's acceptance criteria against test names. For each criterion:
381
- - Has test → OK
382
- - No test flag as UNTESTED REQUIREMENT (more serious than uncovered lines)
383
-
384
- **Why this is stronger than coverage**: Coverage checks that lines were EXECUTED. Traceability checks that INTENT was VERIFIED. You can have 100% coverage but miss a requirement if the test doesn't assert the right behavior.
385
-
386
- **Skip if**: No plan exists (ad-hoc fix), or plan has no acceptance criteria section.
387
-
388
- ## Eval-Driven Development
389
-
390
- Define **capability evals** and **regression evals** BEFORE writing implementation code. Evals go beyond unit tests — they verify that the agent/system can handle the feature's intent, not just its mechanics.
391
-
392
- ### Two Eval Types
393
-
394
- | Type | Purpose | Pass Criteria | When |
395
- |------|---------|---------------|------|
396
- | **Capability eval** | Can the system do this new thing? | pass@k: ≥1 success in k attempts (k=3-5) | Before implementation |
397
- | **Regression eval** | Did we break existing behavior? | pass^k: ALL k attempts must pass | After implementation |
398
-
399
- **pass@k** (capability): At least 1 of k runs succeeds. Used for new features where some variance is acceptable. Threshold: ≥90% pass@3 for standard features, ≥95% pass@5 for critical paths.
400
-
401
- **pass^k** (regression): ALL k runs must pass. Used for existing behavior that must never break. If ANY run fails, it's a regression. Threshold: 100% pass^3.
402
-
403
- ### Eval File Format
404
-
405
- Store evals in `.rune/evals/<feature>.md`:
406
-
407
- ```markdown
408
- # Eval: <feature name>
409
-
410
- ## Capability Evals (pass@k)
411
- | ID | Description | k | Threshold | Status |
412
- |----|-------------|---|-----------|--------|
413
- | CAP-1 | [what the system should be able to do] | 3 | 90% | pending |
414
-
415
- ## Regression Evals (pass^k)
416
- | ID | Description | k | Status |
417
- |----|-------------|---|--------|
418
- | REG-1 | [existing behavior that must not break] | 3 | pending |
419
- ```
420
-
421
- ### Anti-Pattern: Eval Overfitting
422
-
423
- Do NOT overfit evals to specific prompts or known examples. Evals should test the **capability**, not the **exact input**.
424
-
425
- - BAD: `"When user says 'hello', respond with 'Hi there!'"` tests exact string match
426
- - GOOD: `"When user greets, respond with a greeting"` — tests capability
427
-
428
- ### Integration with TDD
429
-
430
- 1. Write eval definitions (capability + regression) → `.rune/evals/<feature>.md`
431
- 2. Write unit/integration tests (RED phase) → test files
432
- 3. Implement feature (GREEN phase) → source files
433
- 4. Run evals to verify capability achieved + no regressions
434
- 5. Preflight checks eval results as part of quality gate
435
-
436
- ## Red Flags STOP and Start Over
437
-
438
- If you catch yourself with ANY of these, delete implementation code and restart with tests:
439
-
440
- - Code exists before test file
441
- - "I already manually tested it"
442
- - "Tests after achieve the same purpose"
443
- - "It's about spirit not ritual"
444
- - "This is different because..."
445
- - "Let me just finish this, then add tests"
446
-
447
- **All of these mean: Delete code. Start over with TDD.**
448
-
449
- ## Constraints
450
-
451
- 1. MUST write tests BEFORE implementation code — if tests pass without implementation, they are wrong
452
- 2. MUST cover happy path + edge cases + error cases — not just happy path
453
- 3. MUST run tests to verify they FAIL before implementation exists (RED phase is mandatory)
454
- 4. MUST NOT write tests that test mock behavior instead of real code behavior
455
- 5. MUST achieve 80% coverage minimum identify and fill gaps
456
- 6. MUST use the project's existing test framework and conventions don't introduce a new one
457
- 7. MUST NOT say "tests pass" without showing actual test runner output
458
- 8. MUST delete implementation code written before testsIron Law, no exceptions
459
- 9. MUST show RED phase output (actual failure) "I confirmed they fail" without output is REJECTED
460
- 10. MUST NOT modify source/implementation files test writes test files ONLY, hand off source changes to rune:fix
461
-
462
- ## Mesh Gates
463
-
464
- | Gate | Requires | If Missing |
465
- |------|----------|------------|
466
- | RED Gate | All new tests FAIL before implementation | If any pass, rewrite stricter tests |
467
- | GREEN Gate | All tests PASS after implementation | Fix code, not tests |
468
- | Coverage Gate | 80%+ coverage verified via verification | Write additional tests for gaps |
469
-
470
- ## Output Format
471
-
472
- ```
473
- ## Test Report
474
- - **Framework**: [detected]
475
- - **Files Created**: [list of new test file paths]
476
- - **Tests Written**: [count]
477
- - **Status**: RED (failing as expected) | GREEN (all passing)
478
-
479
- ### Test Cases
480
- | Test | Status | Description |
481
- |------|--------|-------------|
482
- | `test_name` | FAIL/PASS | [what it tests] |
483
-
484
- ### Coverage
485
- - Lines: [X]% | Branches: [Y]%
486
- - Gaps: `path/to/file.ts:42-58` — uncovered branch (error handling)
487
-
488
- ### Regressions (if any)
489
- - [existing test that broke, with error details]
490
- ```
491
-
492
- ## Testing Anti-Patterns (Gate Functions)
493
-
494
- Before writing tests, check yourself against these 5 anti-patterns. Each has a **gate function** — a question you MUST answer before proceeding.
495
-
496
- ### Anti-Pattern 1: Testing Mock Behavior
497
- Asserting that a mock exists (e.g., `testId="sidebar-mock"`) instead of testing real component behavior. You're proving the mock works, not the code.
498
- **Gate**: "Am I testing real component behavior or just mock existence?" → If mock existence: STOP. Rewrite to test real behavior.
499
-
500
- ### Anti-Pattern 2: Test-Only Methods in Production
501
- Adding `destroy()`, `reset()`, or `__testSetup()` methods to production classes that are ONLY called from test files. Production code should not know tests exist.
502
- **Gate**: "Is this method only called by tests?" → If yes: STOP. Move to test utilities or test helper file, not production class.
503
-
504
- ### Anti-Pattern 3: Mocking Without Understanding Side Effects
505
- Mocking a function without first understanding ALL its side effects. The real function may write config files, update caches, or emit events that downstream code depends on.
506
- **Gate**: Before mocking, STOP and answer: "What side effects does the REAL function have? Does this test depend on any of those?" → Run with real implementation first, observe what happens, THEN add minimal mocking.
507
-
508
- ### Anti-Pattern 4: Incomplete Mocks
509
- Partial mock missing fields that downstream code consumes. Your test passes because it only checks the fields you mocked, but production code reads fields your mock doesn't have → runtime crash.
510
- **Iron Rule**: Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses. Examine actual API response / real data shape before writing mock.
511
-
512
- ### Anti-Pattern 5: Mock Setup Longer Than Test Logic
513
- If mock setup is 30 lines and the actual test assertion is 3 lines, the test is testing infrastructure, not behavior. This is a code smell that indicates wrong abstraction level.
514
- **Gate**: "Is my mock setup longer than my test logic?" → If yes: test at a higher level (integration) or extract mock factories.
515
-
516
- ### Anti-Pattern 6: Test Slop (Framework-Behavior Tests)
517
- Tests that verify the framework works rather than YOUR code works. If the test would still pass with an empty component/function, it's testing infrastructure.
518
- **Gate**: "Would this test pass if I deleted my business logic?" → If yes: STOP. Rewrite to test behavior that YOUR code introduces.
519
-
520
- Examples of test slop:
521
- - "renders without crashing" (tests that React works, not your component)
522
- - "route responds with 200" without checking response body (tests Express, not your handler)
523
- - Asserting a mock was called N times without checking the RESULT of those calls
524
- - Type existence tests (`typeof result === 'object'`) when you should test the actual value
525
-
526
- **Red flags any of these means STOP and rethink:**
527
- - Mock setup longer than test logic
528
- - `*-mock` test IDs in assertions
529
- - Methods only called in test files
530
- - Can't explain in one sentence why a mock is needed
531
- - Test would pass with empty implementation (test slop)
532
-
533
- ## Returns
534
-
535
- | Artifact | Format | Location |
536
- |----------|--------|----------|
537
- | Test files | Source files | Co-located or `__tests__/` per project convention |
538
- | Test plan + results | Markdown | `TEST.md` in test directory (non-trivial features only) |
539
- | Eval scenarios | Markdown | `skills/<name>/evals.md` (for skill behavior testing) |
540
- | Coverage report | Inline stdout | Shown in Test Report |
541
- | Test Report | Markdown (inline) | Emitted to calling skill (cook, fix, review) |
542
-
543
- ## Sharp Edges
544
-
545
- Known failure modes for this skill. Check these before declaring done.
546
-
547
- | Failure Mode | Severity | Mitigation |
548
- |---|---|---|
549
- | Tests passing before implementation exists | CRITICAL | RED Gate: rewrite stricter tests — passing without code = not testing real behavior |
550
- | Skipping the RED phase (not confirming FAIL) | HIGH | Run tests, confirm FAIL output before calling cook/fix to implement |
551
- | Testing mock behavior instead of real code | HIGH | Anti-Pattern 1 gate: "Am I testing real behavior or mock existence?" |
552
- | Mocking without understanding side effects | HIGH | Anti-Pattern 3 gate: run with real impl first, observe side effects, THEN mock minimally |
553
- | Incomplete mocks missing downstream fields | HIGH | Anti-Pattern 4 iron rule: mock COMPLETE data structure, not just fields your test checks |
554
- | Coverage below 80% without filling gaps | MEDIUM | Coverage Gate: identify uncovered lines and write additional tests |
555
- | Introducing a new test framework instead of using existing one | MEDIUM | Constraint 6: detect framework first, use project's existing one always |
556
- | Modifying source files to make tests work | HIGH | Role boundary: test writes test files ONLY source changes go to rune:fix |
557
- | Test-only methods leaking into production code | MEDIUM | Anti-Pattern 2 gate: if method only called by tests move to test utilities |
558
-
559
- ## Self-Validation
560
-
561
- ```
562
- SELF-VALIDATION (run before emitting Test Report):
563
- - [ ] Every test file has at least one assertion — no empty test bodies
564
- - [ ] RED phase output shows actual failures (not "0 tests") — tests were real, not stubs
565
- - [ ] No test modifies source code test files only, source changes belong to fix
566
- - [ ] Test names describe behavior, not implementation ("should reject expired token" not "test function X")
567
- - [ ] No mocks of the thing being tested only mock external dependencies
568
- - [ ] If BA requirements exist (REQ-xxx), every requirement has at least one test check plan's Traceability Matrix
569
- ```
570
-
571
- ## Done When
572
-
573
- - Test framework detected from project config files
574
- - Tests cover happy path + at least 2 edge cases + error case
575
- - All new tests FAIL (RED phase — actual failure output shown)
576
- - After implementation: all tests PASS (GREEN phase actual pass output shown)
577
- - Coverage ≥80% verified via verification
578
- - Test Report emitted with framework, test count, RED/GREEN status, and coverage
579
- - Self-Validation: all checks passed
580
-
581
- ## Cost Profile
582
-
583
- ~$0.03-0.08 per invocation. Sonnet for writing tests, Bash for running them. Frequent invocation in TDD workflow.
584
-
585
- **Scope guardrail**: Do not modify source or implementation files to make tests pass unless explicitly delegated by the parent agent.
1
+ ---
2
+ name: test
3
+ description: "TDD test writer. Writes failing tests FIRST (red), then verifies they pass after implementation (green). Covers unit, integration, and e2e tests."
4
+ metadata:
5
+ author: runedev
6
+ version: "1.1.0"
7
+ layer: L2
8
+ model: sonnet
9
+ group: development
10
+ tools: "Read, Write, Edit, Bash, Glob, Grep"
11
+ emit: tests.passed, tests.failed
12
+ listen: code.changed
13
+ ---
14
+
15
+ # test
16
+
17
+ <HARD-GATE>
18
+ Tests define the EXPECTED BEHAVIOR. They MUST be written BEFORE implementation code.
19
+ If tests pass without implementation → the tests are wrong. Rewrite them.
20
+ The only exception: when retrofitting tests for existing untested code.
21
+
22
+ THE IRON LAW: Write code before test? DELETE IT. Start over.
23
+ - Do NOT keep it as "reference"
24
+ - Do NOT "adapt" it while writing tests
25
+ - Do NOT look at it to "inform" test design
26
+ - Delete means delete. `git checkout -- <file>` or remove the changes entirely.
27
+ This is not negotiable. This is not optional. "But I already wrote it" is a sunk cost fallacy.
28
+
29
+ ROLE BOUNDARY: Test writes TEST FILES only. NEVER modify source/implementation files.
30
+ - Do NOT "quickly fix" a broken import in source to make tests run
31
+ - Do NOT refactor source code to be "more testable"
32
+ - Do NOT add missing exports to source files
33
+ - If source needs changes → hand off to `rune:fix`. Test's job ends at the test file.
34
+ This separation ensures test never writes code biased toward passing its own tests.
35
+ </HARD-GATE>
36
+
37
+ ## Instructions
38
+
39
+ ### Phase 1: Understand What to Test
40
+
41
+ 1. Read the implementation plan or task description carefully
42
+ 2. Use `Glob` to find existing test files: `**/*.test.*`, `**/*.spec.*`, `**/test_*`
43
+ 3. Use `Read` on 2-3 existing test files to understand:
44
+ - Test framework in use
45
+ - File naming convention (e.g., `foo.test.ts` mirrors `foo.ts`)
46
+ - Test directory structure (co-located vs `__tests__/` vs `tests/`)
47
+ - Assertion style and patterns
48
+ 4. Use `Glob` to find the source file(s) being tested
49
+
50
+ ```
51
+ TodoWrite: [
52
+ { content: "Understand scope and find existing test patterns", status: "in_progress" },
53
+ { content: "Detect test framework and conventions", status: "pending" },
54
+ { content: "Write failing tests (RED phase)", status: "pending" },
55
+ { content: "Run tests — verify they FAIL", status: "pending" },
56
+ { content: "After implementation: verify tests PASS (GREEN phase)", status: "pending" }
57
+ ]
58
+ ```
59
+
60
+ ### Phase 2: Detect Test Framework
61
+
62
+ Use `Glob` to find config files and identify the framework:
63
+
64
+ - `jest.config.*` or `"jest"` key in `package.json` → Jest
65
+ - `vitest.config.*` or `"vitest"` key in `package.json` Vitest
66
+ - `pytest.ini`, `[tool.pytest.ini_options]` in `pyproject.toml` pytest
67
+ - **Async check**: If pytest detected AND source files contain `async def`:
68
+ - Check if `pytest-asyncio` is in dependencies (`pyproject.toml [project.dependencies]` or `[project.optional-dependencies]`)
69
+ - Check if `asyncio_mode` is set in `[tool.pytest.ini_options]` (values: `auto`, `strict`, or absent)
70
+ - If async code exists but no `asyncio_mode` configured**WARN**: "pytest-asyncio not configured. Async tests may silently pass without executing async code. Recommend adding `asyncio_mode = \"auto\"` to `[tool.pytest.ini_options]` in pyproject.toml."
71
+ - `Cargo.toml` with `#[cfg(test)]` pattern built-in `cargo test`
72
+ - `*_test.go` files present built-in `go test`
73
+ - `cypress.config.*` → Cypress (E2E)
74
+ - `playwright.config.*` Playwright (E2E)
75
+
76
+ **Verification gate**: Framework identified before writing any test code.
77
+
78
+ ### Phase 3: Write Failing Tests
79
+
80
+ Use `Write` to create test files following the detected conventions:
81
+
82
+ 1. Mirror source file location: if source is `src/auth/login.ts`, test is `src/auth/login.test.ts`
83
+ 2. Structure tests with clear `describe` / `it` blocks (or language equivalent):
84
+ - `describe('Feature name')`
85
+ - `it('should [expected behavior] when [condition]')`
86
+ 3. Cover all three categories:
87
+ - **Happy path**: valid inputs, expected success output
88
+ - **Edge cases**: empty input, boundary values, large input
89
+ - **Error cases**: invalid input, missing data, network failure simulation
90
+
91
+ 4. Use proper assertions. Do NOT use implementation details — test behavior:
92
+ - Jest/Vitest: `expect(result).toBe(expected)`
93
+ - pytest: `assert result == expected`
94
+ - Rust: `assert_eq!(result, expected)`
95
+ - Go: `if result != expected { t.Errorf(...) }`
96
+
97
+ 5. For async code: use `async/await` or pytest `@pytest.mark.asyncio`
98
+
99
+ #### Python Async Tests (pytest-asyncio)
100
+
101
+ When writing tests for async Python code:
102
+
103
+ 1. **Verify setup before writing tests**:
104
+ - Confirm `pytest-asyncio` is in project dependencies
105
+ - Confirm `asyncio_mode` is set in `pyproject.toml` `[tool.pytest.ini_options]` (recommend `"auto"`)
106
+ - If neither is configured, warn the caller and suggest setup before proceeding
107
+
108
+ 2. **Writing async test functions**:
109
+ - With `asyncio_mode = "auto"`: just write `async def test_something():` — no decorator needed
110
+ - With `asyncio_mode = "strict"`: every async test needs `@pytest.mark.asyncio`
111
+ - Without asyncio_mode set: always use `@pytest.mark.asyncio` decorator explicitly
112
+
113
+ 3. **Async fixtures**:
114
+ - Use `@pytest_asyncio.fixture` (NOT `@pytest.fixture`) for async setup/teardown
115
+ - Scope rules: async fixtures default to `function` scope — use `scope="session"` carefully with async
116
+
117
+ 4. **Common pitfalls**:
118
+ - Tests that `pass` without `await` they run but don't execute the async path
119
+ - Missing `pytest-asyncio` makes `async def test_*` silently pass as empty coroutines
120
+ - Mixing sync and async fixtures can cause event loop errors
121
+
122
+ ### Phase 4: Run Tests Verify They FAIL (RED)
123
+
124
+ Use `Bash` to run ONLY the newly created test files (not full suite):
125
+
126
+ - **Jest**: `npx jest path/to/test.ts --no-coverage`
127
+ - **Vitest**: `npx vitest run path/to/test.ts`
128
+ - **pytest**: `pytest path/to/test_file.py -v` (if async tests and no `asyncio_mode` in config: add `--asyncio-mode=auto`)
129
+ - **Rust**: `cargo test test_module_name`
130
+ - **Go**: `go test ./path/to/package/... -run TestFunctionName`
131
+
132
+ **Hard gate**: ALL new tests MUST fail at this point.
133
+
134
+ - If ANY test passes before implementation exists → that test is not testing real behavior. Rewrite it to be stricter.
135
+ - If tests fail with import/syntax errors (not assertion errors) → fix the test code, re-run
136
+
137
+ ### Phase 5: After Implementation Verify Tests PASS (GREEN)
138
+
139
+ After `rune:fix` writes implementation code, run the same test command again:
140
+
141
+ 1. ALL tests in the new test files MUST pass
142
+ 2. Run the full test suite with `Bash` to check for regressions:
143
+ - `npm test`, `pytest`, `cargo test`, `go test ./...`
144
+ 3. If any test fails: report clearly which test, what was expected, what was received
145
+ 4. If an existing test now fails (regression): escalate to `rune:debug`
146
+
147
+ **Verification gate**: 100% of new tests pass AND 0 regressions in existing tests.
148
+
149
+ ### Phase 6: Coverage Check
150
+
151
+ After GREEN phase, call `verification` to check coverage threshold (80% minimum):
152
+
153
+ - If coverage drops below 80%: identify uncovered lines, write additional tests
154
+ - Report coverage gaps with file:line references
155
+
156
+ ### Phase 6.5: Diff-Aware Mode (optional)
157
+
158
+ When invoked with `mode: "diff-aware"` or by `cook` after implementation:
159
+
160
+ 1. Run `git diff main --name-only` to get changed files
161
+ 2. For each changed file, trace its **blast radius**: what imports it? what routes does it serve? what components render it?
162
+ 3. Map changed files affected routes/endpoints/pages
163
+ 4. Prioritize tests: files with most downstream dependents get tested first
164
+ 5. Generate targeted test commands that cover ONLY affected paths skip unchanged modules
165
+
166
+ This mode is valuable for large codebases where running the full suite is slow. It answers: "what could this diff have broken?"
167
+
168
+ ```
169
+ Input: git diff main --name-only
170
+ Output: Prioritized test plan targeting only affected paths
171
+ ```
172
+
173
+ ## Test Types 4-Layer Methodology
174
+
175
+ Tests are organized in 4 layers. Each layer catches a different failure class. Higher layers are slower but catch integration issues lower layers miss.
176
+
177
+ | Layer | Type | What It Catches | Framework | Speed |
178
+ |-------|------|-----------------|-----------|-------|
179
+ | L1 | **Unit** | Logic bugs, boundary violations, pure function errors | jest/vitest/pytest/cargo test | Fast |
180
+ | L2 | **Integration** | API contract breaks, DB query errors, service interaction failures | supertest/httpx/reqwest | Medium |
181
+ | L3 | **True Backend** | Real tool/service output correctness (not just exit 0) | Same + real software invocation | Medium-Slow |
182
+ | L4 | **E2E / Subprocess** | Full workflow from user/agent perspective, installed app works | Playwright/Cypress/subprocess | Slow |
183
+
184
+ **Layer rules:**
185
+ - **L1 (Unit)**: Synthetic data, no external deps. Every function tested in isolation. Fast, deterministic, CI-friendly
186
+ - **L2 (Integration)**: Tests service boundaries API endpoints, DB operations, message queues. May need test DB or mock server
187
+ - **L3 (True Backend)**: **Invokes the REAL tool/service** and verifies output programmatically. No graceful degradation — if the dependency isn't installed, tests FAIL (not skip). Verify: magic bytes, file size > 0, content structure. Print artifact paths for manual inspection
188
+ - **L4 (E2E/Subprocess)**: Tests the installed command/app via subprocess or browser automation. Full user workflow: input process output verify
189
+
190
+ **"No graceful degradation" rule** (L3/L4): Hard dependencies MUST be installed. Tests MUST NOT skip or produce fake results when the dependency is missing. A silently skipping test is worse than a loudly failing test.
191
+
192
+ Additional modes:
193
+
194
+ | Type | When | Speed |
195
+ |------|------|-------|
196
+ | Regression | After bug fixes | Fast |
197
+ | Diff-aware | After implementation, large codebases (Phase 6.5) | Fast (targeted) |
198
+
199
+ ## TEST.md Test Plan + Results Document
200
+
201
+ For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` in the test directory. This is BOTH a planning doc (written BEFORE tests) and results doc (appended AFTER tests pass).
202
+
203
+ ### Before writing tests — write the plan:
204
+ ```markdown
205
+ # Test Plan: [Feature Name]
206
+
207
+ ## Test Inventory
208
+ - `test_core.py`: ~XX unit tests planned (L1)
209
+ - `test_integration.py`: ~XX integration tests planned (L2)
210
+ - `test_e2e.py`: ~XX E2E tests planned (L3/L4)
211
+
212
+ ## Unit Test Plan (L1)
213
+ | Module | Functions | Edge Cases | Est. Tests | Req IDs |
214
+ |--------|-----------|------------|------------|---------|
215
+ | `core/auth.py` | login, register, refresh | expired token, invalid creds, rate limit | 12 | REQ-001, REQ-003 |
216
+
217
+ ## E2E Scenarios (L3/L4)
218
+ | Workflow | Simulates | Operations | Verified | Req IDs |
219
+ |----------|-----------|------------|----------|---------|
220
+ | User signup | New user onboarding | register → verify → login | Token valid, profile created | REQ-005 |
221
+
222
+ ## Realistic Workflow Scenarios
223
+ - **[Name]**: [Step 1] → [Step 2] → verify [output properties]
224
+ ```
225
+
226
+ ### After tests pass — append results:
227
+ ```markdown
228
+ ## Test Results
229
+ [Paste full `pytest -v --tb=no` or `npm test` output]
230
+
231
+ ## Summary
232
+ - Total: XX | Passed: XX | Failed: 0
233
+ - Execution time: X.Xs | Coverage: XX%
234
+
235
+ ## Requirement Coverage
236
+ | Req ID | Test File(s) | Status |
237
+ |--------|-------------|--------|
238
+ | REQ-001 | `test_auth.py::test_login` | ✅ Covered |
239
+ | REQ-002 | — | ❌ Not covered |
240
+
241
+ ## Gaps
242
+ - [Areas not covered and why]
243
+ ```
244
+
245
+ **Why TEST.md**: Planning tests before code catches missing edge cases early. Appending results creates permanent evidence. One document = complete testing story.
246
+
247
+ ## Skill Behavior Tests (Eval Scenarios)
248
+
249
+ For testing SKILL.md behavior (not code), use **Eval Scenarios** — unit tests for skill files, not code files.
250
+
251
+ ### Eval Scenario Format
252
+
253
+ ```markdown
254
+ ## Eval: E[NN] — [scenario name]
255
+
256
+ ### Prompt
257
+ [The exact situation/message an agent receives]
258
+
259
+ ### Expected Reasoning
260
+ [Step-by-step reasoning the agent SHOULD follow]
261
+
262
+ ### Must Include
263
+ - [Assertion 1: what the output MUST contain or do]
264
+ - [Assertion 2]
265
+
266
+ ### Must NOT
267
+ - [Anti-pattern 1: what the output MUST NOT do]
268
+ - [Anti-pattern 2]
269
+
270
+ ### Category
271
+ happy-path | adversarial | edge-case | jailbreak | credential-leak
272
+ ```
273
+
274
+ ### Eval Coverage Requirements
275
+
276
+ A skill is **behavior-tested** when it has evals covering:
277
+
278
+ | Category | Min Evals | Purpose |
279
+ |----------|-----------|---------|
280
+ | Happy path | 1 | Core workflow executes correctly |
281
+ | Edge case | 1 | Empty input, missing context, unusual state |
282
+ | Adversarial | 1 | Time pressure, sunk cost, authority pressure |
283
+ | Jailbreak / injection | 1 | Prompt injection attempt, "ignore instructions" |
284
+
285
+ **Minimum**: 4 evals per skill (1 per category). Security-critical skills (sentinel, safeguard): 8+ evals.
286
+
287
+ ### Eval Storage
288
+
289
+ Save eval files as `skills/<name>/evals.md`. Each eval is a numbered scenario (E01–E24 range). skill-forge Phase 7 checks for evals presence before ship.
290
+
291
+
292
+ ## Error Recovery
293
+
294
+ - If test framework not found: ask calling skill to specify, or check `package.json` `devDependencies`
295
+ - If `Write` to test file fails: check if directory exists, create it first with `Bash mkdir -p`
296
+ - If tests error on import (module not found): check that source file path is correct, adjust imports
297
+ - If `Bash` test runner hangs beyond 120 seconds: kill and report as TIMEOUT
298
+
299
+ ## Called By (inbound)
300
+
301
+ - `cook` (L1): Phase 3 TEST write tests first
302
+ - `fix` (L2): verify fix passes tests
303
+ - `review` (L2): untested edge case found write test for it
304
+ - `deploy` (L2): pre-deployment full test suite
305
+ - `preflight` (L2): run targeted regression tests on affected code
306
+ - `surgeon` (L2): verify refactored code
307
+ - `launch` (L1): pre-deployment test suite
308
+ - `safeguard` (L2): writing characterization tests for legacy code
309
+ - `review-intake` (L2): write tests for issues identified during review intake
310
+
311
+ ## Calls (outbound)
312
+
313
+ - `verification` (L3): Phase 6coverage check (80% minimum threshold)
314
+ - `browser-pilot` (L3): Phase 4 — e2e and visual testing for UI flows
315
+ - `debug` (L2): Phase 5 — when existing test regresses unexpectedly
316
+
317
+ ## Data Flow
318
+
319
+ ### Feeds Into
320
+
321
+ - `cook` (L1): test results (pass/fail/coverage) cook's Phase 5 quality gate evidence
322
+ - `completion-gate` (L3): test runner stdout → evidence for "tests pass" claims
323
+ - `fix` (L2): failing test output → fix's target (what to make green)
324
+
325
+ ### Fed By
326
+
327
+ - `plan` (L2): phase file test tasks → test's RED phase targets (what to test)
328
+ - `review` (L2): untested edge cases found during review → new test targets
329
+ - `fix` (L2): implemented code → test's GREEN phase verification target
330
+
331
+ ### Feedback Loops
332
+
333
+ - `test` ↔ `fix`: test writes failing tests (RED) → fix implements to pass → test verifies (GREEN) → if new failures emerge, loop continues
334
+ - `test` ↔ `debug`: test discovers regression → debug diagnoses root cause → test writes regression test to prevent recurrence
335
+
336
+ ## Anti-Rationalization Table
337
+
338
+ | Excuse | Reality |
339
+ |---|---|
340
+ | "Too simple to need tests first" | Simple code breaks. Test takes 30 seconds. Write it first. |
341
+ | "I'll write tests after same result" | Tests-after = "what does this do?" Tests-first = "what SHOULD this do?" Completely different. |
342
+ | "I already wrote the code, let me just add tests" | Iron Law: delete the code. Start over with tests. Sunk cost is not an argument. |
343
+ | "Tests after achieve the same goals" | They don't. Tests-after are biased by the implementation you just wrote. |
344
+ | "It's about spirit not ritual" | Violating the letter IS violating the spirit. Write the test first. |
345
+ | "I mentally tested it" | Mental testing is not testing. Run the command, show the output. |
346
+ | "This is different because..." | It's not. Write the test first. |
347
+
348
+ ## Advanced: Oracle-Injection E2E Testing
349
+
350
+ For **data pipelines, AI workflows, and multi-stage processing** where comparing full output structures is impractical, use oracle injection:
351
+
352
+ 1. **Generate a UUID oracle token**: `const oracle = crypto.randomUUID()`
353
+ 2. **Inject into synthetic input**: embed the oracle in realistic test data that flows through the pipeline
354
+ 3. **Run the full pipeline**: input → all stages → output
355
+ 4. **Search for oracle in output**: if found → data flowed end-to-end correctly
356
+
357
+ ```
358
+ // Example: testing a document processing pipeline
359
+ const oracle = "ORACLE-" + crypto.randomUUID();
360
+ const testDoc = `Meeting notes: discussed ${oracle} integration timeline`;
361
+ const result = await pipeline.process(testDoc);
362
+ assert(result.output.includes(oracle), "Oracle not found — pipeline lost data");
363
+ ```
364
+
365
+ **When to use**: E2E tests for pipelines with 3+ stages, LLM-based processing, ETL workflows, or any system where output structure is complex/non-deterministic but data preservation is critical.
366
+
367
+ **When NOT to use**: Unit tests, simple CRUD, or when exact output comparison is feasible.
368
+
369
+
370
+ ## Spec→Test Traceability
371
+
372
+ When a plan with acceptance criteria exists (`.rune/features/<name>/plan.md` or phase file), every criterion MUST map to at least one test case.
373
+
374
+ ```
375
+ Plan Acceptance Criteria Test CaseImplementation
376
+
377
+ AC-1: "User can reset password via email" → test_password_reset_sends_email()
378
+ AC-2: "Rate limit: max 3 reset attempts/hour" → test_password_reset_rate_limit()
379
+ AC-3: "Expired tokens rejected" → test_expired_reset_token_rejected()
380
+ ```
381
+
382
+ **Validation step** (after writing tests): Cross-check plan's acceptance criteria against test names. For each criterion:
383
+ - Has test → OK
384
+ - No test flag as UNTESTED REQUIREMENT (more serious than uncovered lines)
385
+
386
+ **Why this is stronger than coverage**: Coverage checks that lines were EXECUTED. Traceability checks that INTENT was VERIFIED. You can have 100% coverage but miss a requirement if the test doesn't assert the right behavior.
387
+
388
+ **Skip if**: No plan exists (ad-hoc fix), or plan has no acceptance criteria section.
389
+
390
+ ## Eval-Driven Development
391
+
392
+ Define **capability evals** and **regression evals** BEFORE writing implementation code. Evals go beyond unit tests — they verify that the agent/system can handle the feature's intent, not just its mechanics.
393
+
394
+ ### Two Eval Types
395
+
396
+ | Type | Purpose | Pass Criteria | When |
397
+ |------|---------|---------------|------|
398
+ | **Capability eval** | Can the system do this new thing? | pass@k: ≥1 success in k attempts (k=3-5) | Before implementation |
399
+ | **Regression eval** | Did we break existing behavior? | pass^k: ALL k attempts must pass | After implementation |
400
+
401
+ **pass@k** (capability): At least 1 of k runs succeeds. Used for new features where some variance is acceptable. Threshold: ≥90% pass@3 for standard features, ≥95% pass@5 for critical paths.
402
+
403
+ **pass^k** (regression): ALL k runs must pass. Used for existing behavior that must never break. If ANY run fails, it's a regression. Threshold: 100% pass^3.
404
+
405
+ ### Eval File Format
406
+
407
+ Store evals in `.rune/evals/<feature>.md`:
408
+
409
+ ```markdown
410
+ # Eval: <feature name>
411
+
412
+ ## Capability Evals (pass@k)
413
+ | ID | Description | k | Threshold | Status |
414
+ |----|-------------|---|-----------|--------|
415
+ | CAP-1 | [what the system should be able to do] | 3 | 90% | pending |
416
+
417
+ ## Regression Evals (pass^k)
418
+ | ID | Description | k | Status |
419
+ |----|-------------|---|--------|
420
+ | REG-1 | [existing behavior that must not break] | 3 | pending |
421
+ ```
422
+
423
+ ### Anti-Pattern: Eval Overfitting
424
+
425
+ Do NOT overfit evals to specific prompts or known examples. Evals should test the **capability**, not the **exact input**.
426
+
427
+ - BAD: `"When user says 'hello', respond with 'Hi there!'"` — tests exact string match
428
+ - GOOD: `"When user greets, respond with a greeting"` — tests capability
429
+
430
+ ### Integration with TDD
431
+
432
+ 1. Write eval definitions (capability + regression) → `.rune/evals/<feature>.md`
433
+ 2. Write unit/integration tests (RED phase) test files
434
+ 3. Implement feature (GREEN phase) source files
435
+ 4. Run evals to verify capability achieved + no regressions
436
+ 5. Preflight checks eval results as part of quality gate
437
+
438
+ ## Red Flags STOP and Start Over
439
+
440
+ If you catch yourself with ANY of these, delete implementation code and restart with tests:
441
+
442
+ - Code exists before test file
443
+ - "I already manually tested it"
444
+ - "Tests after achieve the same purpose"
445
+ - "It's about spirit not ritual"
446
+ - "This is different because..."
447
+ - "Let me just finish this, then add tests"
448
+
449
+ **All of these mean: Delete code. Start over with TDD.**
450
+
451
+ ## Constraints
452
+
453
+ 1. MUST write tests BEFORE implementation code if tests pass without implementation, they are wrong
454
+ 2. MUST cover happy path + edge cases + error cases not just happy path
455
+ 3. MUST run tests to verify they FAIL before implementation exists (RED phase is mandatory)
456
+ 4. MUST NOT write tests that test mock behavior instead of real code behavior
457
+ 5. MUST achieve 80% coverage minimum identify and fill gaps
458
+ 6. MUST use the project's existing test framework and conventions don't introduce a new one
459
+ 7. MUST NOT say "tests pass" without showing actual test runner output
460
+ 8. MUST delete implementation code written before tests Iron Law, no exceptions
461
+ 9. MUST show RED phase output (actual failure) — "I confirmed they fail" without output is REJECTED
462
+ 10. MUST NOT modify source/implementation files — test writes test files ONLY, hand off source changes to rune:fix
463
+
464
+ ## Mesh Gates
465
+
466
+ | Gate | Requires | If Missing |
467
+ |------|----------|------------|
468
+ | RED Gate | All new tests FAIL before implementation | If any pass, rewrite stricter tests |
469
+ | GREEN Gate | All tests PASS after implementation | Fix code, not tests |
470
+ | Coverage Gate | 80%+ coverage verified via verification | Write additional tests for gaps |
471
+
472
+ ## Output Format
473
+
474
+ ```
475
+ ## Test Report
476
+ - **Framework**: [detected]
477
+ - **Files Created**: [list of new test file paths]
478
+ - **Tests Written**: [count]
479
+ - **Status**: RED (failing as expected) | GREEN (all passing)
480
+
481
+ ### Test Cases
482
+ | Test | Status | Description |
483
+ |------|--------|-------------|
484
+ | `test_name` | FAIL/PASS | [what it tests] |
485
+
486
+ ### Coverage
487
+ - Lines: [X]% | Branches: [Y]%
488
+ - Gaps: `path/to/file.ts:42-58` — uncovered branch (error handling)
489
+
490
+ ### Regressions (if any)
491
+ - [existing test that broke, with error details]
492
+ ```
493
+
494
+ ## Testing Anti-Patterns (Gate Functions)
495
+
496
+ Before writing tests, check yourself against these 5 anti-patterns. Each has a **gate function** — a question you MUST answer before proceeding.
497
+
498
+ ### Anti-Pattern 1: Testing Mock Behavior
499
+ Asserting that a mock exists (e.g., `testId="sidebar-mock"`) instead of testing real component behavior. You're proving the mock works, not the code.
500
+ **Gate**: "Am I testing real component behavior or just mock existence?" → If mock existence: STOP. Rewrite to test real behavior.
501
+
502
+ ### Anti-Pattern 2: Test-Only Methods in Production
503
+ Adding `destroy()`, `reset()`, or `__testSetup()` methods to production classes that are ONLY called from test files. Production code should not know tests exist.
504
+ **Gate**: "Is this method only called by tests?" → If yes: STOP. Move to test utilities or test helper file, not production class.
505
+
506
+ ### Anti-Pattern 3: Mocking Without Understanding Side Effects
507
+ Mocking a function without first understanding ALL its side effects. The real function may write config files, update caches, or emit events that downstream code depends on.
508
+ **Gate**: Before mocking, STOP and answer: "What side effects does the REAL function have? Does this test depend on any of those?" → Run with real implementation first, observe what happens, THEN add minimal mocking.
509
+
510
+ ### Anti-Pattern 4: Incomplete Mocks
511
+ Partial mock missing fields that downstream code consumes. Your test passes because it only checks the fields you mocked, but production code reads fields your mock doesn't have → runtime crash.
512
+ **Iron Rule**: Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses. Examine actual API response / real data shape before writing mock.
513
+
514
+ ### Anti-Pattern 5: Mock Setup Longer Than Test Logic
515
+ If mock setup is 30 lines and the actual test assertion is 3 lines, the test is testing infrastructure, not behavior. This is a code smell that indicates wrong abstraction level.
516
+ **Gate**: "Is my mock setup longer than my test logic?" → If yes: test at a higher level (integration) or extract mock factories.
517
+
518
+ ### Anti-Pattern 6: Test Slop (Framework-Behavior Tests)
519
+ Tests that verify the framework works rather than YOUR code works. If the test would still pass with an empty component/function, it's testing infrastructure.
520
+ **Gate**: "Would this test pass if I deleted my business logic?" → If yes: STOP. Rewrite to test behavior that YOUR code introduces.
521
+
522
+ Examples of test slop:
523
+ - "renders without crashing" (tests that React works, not your component)
524
+ - "route responds with 200" without checking response body (tests Express, not your handler)
525
+ - Asserting a mock was called N times without checking the RESULT of those calls
526
+ - Type existence tests (`typeof result === 'object'`) when you should test the actual value
527
+
528
+ **Red flags any of these means STOP and rethink:**
529
+ - Mock setup longer than test logic
530
+ - `*-mock` test IDs in assertions
531
+ - Methods only called in test files
532
+ - Can't explain in one sentence why a mock is needed
533
+ - Test would pass with empty implementation (test slop)
534
+
535
+ ## Returns
536
+
537
+ | Artifact | Format | Location |
538
+ |----------|--------|----------|
539
+ | Test files | Source files | Co-located or `__tests__/` per project convention |
540
+ | Test plan + results | Markdown | `TEST.md` in test directory (non-trivial features only) |
541
+ | Eval scenarios | Markdown | `skills/<name>/evals.md` (for skill behavior testing) |
542
+ | Coverage report | Inline stdout | Shown in Test Report |
543
+ | Test Report | Markdown (inline) | Emitted to calling skill (cook, fix, review) |
544
+
545
+ ## Sharp Edges
546
+
547
+ Known failure modes for this skill. Check these before declaring done.
548
+
549
+ | Failure Mode | Severity | Mitigation |
550
+ |---|---|---|
551
+ | Tests passing before implementation exists | CRITICAL | RED Gate: rewrite stricter tests passing without code = not testing real behavior |
552
+ | Skipping the RED phase (not confirming FAIL) | HIGH | Run tests, confirm FAIL output before calling cook/fix to implement |
553
+ | Testing mock behavior instead of real code | HIGH | Anti-Pattern 1 gate: "Am I testing real behavior or mock existence?" |
554
+ | Mocking without understanding side effects | HIGH | Anti-Pattern 3 gate: run with real impl first, observe side effects, THEN mock minimally |
555
+ | Incomplete mocks missing downstream fields | HIGH | Anti-Pattern 4 iron rule: mock COMPLETE data structure, not just fields your test checks |
556
+ | Coverage below 80% without filling gaps | MEDIUM | Coverage Gate: identify uncovered lines and write additional tests |
557
+ | Introducing a new test framework instead of using existing one | MEDIUM | Constraint 6: detect framework first, use project's existing one always |
558
+ | Modifying source files to make tests work | HIGH | Role boundary: test writes test files ONLY — source changes go to rune:fix |
559
+ | Test-only methods leaking into production code | MEDIUM | Anti-Pattern 2 gate: if method only called by tests → move to test utilities |
560
+
561
+ ## Self-Validation
562
+
563
+ ```
564
+ SELF-VALIDATION (run before emitting Test Report):
565
+ - [ ] Every test file has at least one assertion no empty test bodies
566
+ - [ ] RED phase output shows actual failures (not "0 tests") tests were real, not stubs
567
+ - [ ] No test modifies source code test files only, source changes belong to fix
568
+ - [ ] Test names describe behavior, not implementation ("should reject expired token" not "test function X")
569
+ - [ ] No mocks of the thing being tested — only mock external dependencies
570
+ - [ ] If BA requirements exist (REQ-xxx), every requirement has at least one test — check plan's Traceability Matrix
571
+ ```
572
+
573
+ ## Done When
574
+
575
+ - Test framework detected from project config files
576
+ - Tests cover happy path + at least 2 edge cases + error case
577
+ - All new tests FAIL (RED phase — actual failure output shown)
578
+ - After implementation: all tests PASS (GREEN phase actual pass output shown)
579
+ - Coverage ≥80% verified via verification
580
+ - Test Report emitted with framework, test count, RED/GREEN status, and coverage
581
+ - Self-Validation: all checks passed
582
+
583
+ ## Cost Profile
584
+
585
+ ~$0.03-0.08 per invocation. Sonnet for writing tests, Bash for running them. Frequent invocation in TDD workflow.
586
+
587
+ **Scope guardrail**: Do not modify source or implementation files to make tests pass unless explicitly delegated by the parent agent.