@rune-kit/rune 2.3.0 → 2.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +395 -389
- package/compiler/__tests__/tier-override.test.js +158 -0
- package/compiler/adapters/antigravity.js +3 -8
- package/compiler/adapters/codex.js +3 -8
- package/compiler/adapters/cursor.js +3 -8
- package/compiler/adapters/generic.js +3 -8
- package/compiler/adapters/openclaw.js +4 -9
- package/compiler/adapters/opencode.js +3 -8
- package/compiler/adapters/windsurf.js +3 -8
- package/compiler/bin/rune.js +34 -1
- package/compiler/emitter.js +94 -5
- package/compiler/transforms/branding.js +10 -3
- package/docs/ARCHITECTURE.md +3 -3
- package/docs/VISION.md +3 -3
- package/docs/guides/index.html +14 -14
- package/docs/index.html +7 -7
- package/docs/skills/index.html +832 -832
- package/extensions/ai-ml/PACK.md +7 -0
- package/extensions/content/PACK.md +7 -0
- package/extensions/mobile/PACK.md +9 -9
- package/extensions/zalo/PACK.md +9 -0
- package/package.json +2 -2
- package/skills/audit/SKILL.md +526 -529
- package/skills/ba/SKILL.md +349 -351
- package/skills/completion-gate/SKILL.md +260 -263
- package/skills/context-engine/SKILL.md +0 -6
- package/skills/cook/SKILL.md +2 -11
- package/skills/debug/SKILL.md +392 -394
- package/skills/deploy/references/post-deploy-integration.md +192 -0
- package/skills/fix/SKILL.md +281 -282
- package/skills/onboard/SKILL.md +0 -4
- package/skills/plan/references/completeness-scoring.md +0 -1
- package/skills/plan/references/outcome-block.md +0 -1
- package/skills/plan/references/workflow-registry.md +0 -1
- package/skills/preflight/SKILL.md +360 -365
- package/skills/rescue/SKILL.md +1 -0
- package/skills/research/SKILL.md +149 -150
- package/skills/review/SKILL.md +489 -495
- package/skills/sentinel/SKILL.md +0 -11
- package/skills/sentinel/references/destructive-commands.md +0 -1
- package/skills/sentinel/references/skill-content-guard.md +0 -1
- package/skills/session-bridge/SKILL.md +0 -4
- package/skills/skill-router/{SKILL.md → skill.md} +446 -397
- package/skills/team/SKILL.md +1 -2
- package/skills/test/SKILL.md +585 -593
- package/skills/watchdog/references/webhook-health-checks.md +243 -0
package/skills/test/SKILL.md
CHANGED
|
@@ -1,593 +1,585 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: test
|
|
3
|
-
description: "TDD test writer. Writes failing tests FIRST (red), then verifies they pass after implementation (green). Covers unit, integration, and e2e tests."
|
|
4
|
-
metadata:
|
|
5
|
-
author: runedev
|
|
6
|
-
version: "1.1.0"
|
|
7
|
-
layer: L2
|
|
8
|
-
model: sonnet
|
|
9
|
-
group: development
|
|
10
|
-
tools: "Read, Write, Edit, Bash, Glob, Grep"
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
# test
|
|
14
|
-
|
|
15
|
-
<HARD-GATE>
|
|
16
|
-
Tests define the EXPECTED BEHAVIOR. They MUST be written BEFORE implementation code.
|
|
17
|
-
If tests pass without implementation → the tests are wrong. Rewrite them.
|
|
18
|
-
The only exception: when retrofitting tests for existing untested code.
|
|
19
|
-
|
|
20
|
-
THE IRON LAW: Write code before test? DELETE IT. Start over.
|
|
21
|
-
- Do NOT keep it as "reference"
|
|
22
|
-
- Do NOT "adapt" it while writing tests
|
|
23
|
-
- Do NOT look at it to "inform" test design
|
|
24
|
-
- Delete means delete. `git checkout -- <file>` or remove the changes entirely.
|
|
25
|
-
This is not negotiable. This is not optional. "But I already wrote it" is a sunk cost fallacy.
|
|
26
|
-
|
|
27
|
-
ROLE BOUNDARY: Test writes TEST FILES only. NEVER modify source/implementation files.
|
|
28
|
-
- Do NOT "quickly fix" a broken import in source to make tests run
|
|
29
|
-
- Do NOT refactor source code to be "more testable"
|
|
30
|
-
- Do NOT add missing exports to source files
|
|
31
|
-
- If source needs changes → hand off to `rune:fix`. Test's job ends at the test file.
|
|
32
|
-
This separation ensures test never writes code biased toward passing its own tests.
|
|
33
|
-
</HARD-GATE>
|
|
34
|
-
|
|
35
|
-
## Instructions
|
|
36
|
-
|
|
37
|
-
### Phase 1: Understand What to Test
|
|
38
|
-
|
|
39
|
-
1. Read the implementation plan or task description carefully
|
|
40
|
-
2. Use `Glob` to find existing test files: `**/*.test.*`, `**/*.spec.*`, `**/test_*`
|
|
41
|
-
3. Use `Read` on 2-3 existing test files to understand:
|
|
42
|
-
- Test framework in use
|
|
43
|
-
- File naming convention (e.g., `foo.test.ts` mirrors `foo.ts`)
|
|
44
|
-
- Test directory structure (co-located vs `__tests__/` vs `tests/`)
|
|
45
|
-
- Assertion style and patterns
|
|
46
|
-
4. Use `Glob` to find the source file(s) being tested
|
|
47
|
-
|
|
48
|
-
```
|
|
49
|
-
TodoWrite: [
|
|
50
|
-
{ content: "Understand scope and find existing test patterns", status: "in_progress" },
|
|
51
|
-
{ content: "Detect test framework and conventions", status: "pending" },
|
|
52
|
-
{ content: "Write failing tests (RED phase)", status: "pending" },
|
|
53
|
-
{ content: "Run tests — verify they FAIL", status: "pending" },
|
|
54
|
-
{ content: "After implementation: verify tests PASS (GREEN phase)", status: "pending" }
|
|
55
|
-
]
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
### Phase 2: Detect Test Framework
|
|
59
|
-
|
|
60
|
-
Use `Glob` to find config files and identify the framework:
|
|
61
|
-
|
|
62
|
-
- `jest.config.*` or `"jest"` key in `package.json` → Jest
|
|
63
|
-
- `vitest.config.*` or `"vitest"` key in `package.json` → Vitest
|
|
64
|
-
- `pytest.ini`, `[tool.pytest.ini_options]` in `pyproject.toml` → pytest
|
|
65
|
-
- **Async check**: If pytest detected AND source files contain `async def`:
|
|
66
|
-
- Check if `pytest-asyncio` is in dependencies (`pyproject.toml [project.dependencies]` or `[project.optional-dependencies]`)
|
|
67
|
-
- Check if `asyncio_mode` is set in `[tool.pytest.ini_options]` (values: `auto`, `strict`, or absent)
|
|
68
|
-
- If async code exists but no `asyncio_mode` configured → **WARN**: "pytest-asyncio not configured. Async tests may silently pass without executing async code. Recommend adding `asyncio_mode = \"auto\"` to `[tool.pytest.ini_options]` in pyproject.toml."
|
|
69
|
-
- `Cargo.toml` with `#[cfg(test)]` pattern → built-in `cargo test`
|
|
70
|
-
- `*_test.go` files present → built-in `go test`
|
|
71
|
-
- `cypress.config.*` → Cypress (E2E)
|
|
72
|
-
- `playwright.config.*` → Playwright (E2E)
|
|
73
|
-
|
|
74
|
-
**Verification gate**: Framework identified before writing any test code.
|
|
75
|
-
|
|
76
|
-
### Phase 3: Write Failing Tests
|
|
77
|
-
|
|
78
|
-
Use `Write` to create test files following the detected conventions:
|
|
79
|
-
|
|
80
|
-
1. Mirror source file location: if source is `src/auth/login.ts`, test is `src/auth/login.test.ts`
|
|
81
|
-
2. Structure tests with clear `describe` / `it` blocks (or language equivalent):
|
|
82
|
-
- `describe('Feature name')`
|
|
83
|
-
- `it('should [expected behavior] when [condition]')`
|
|
84
|
-
3. Cover all three categories:
|
|
85
|
-
- **Happy path**: valid inputs, expected success output
|
|
86
|
-
- **Edge cases**: empty input, boundary values, large input
|
|
87
|
-
- **Error cases**: invalid input, missing data, network failure simulation
|
|
88
|
-
|
|
89
|
-
4. Use proper assertions. Do NOT use implementation details — test behavior:
|
|
90
|
-
- Jest/Vitest: `expect(result).toBe(expected)`
|
|
91
|
-
- pytest: `assert result == expected`
|
|
92
|
-
- Rust: `assert_eq!(result, expected)`
|
|
93
|
-
- Go: `if result != expected { t.Errorf(...) }`
|
|
94
|
-
|
|
95
|
-
5. For async code: use `async/await` or pytest `@pytest.mark.asyncio`
|
|
96
|
-
|
|
97
|
-
#### Python Async Tests (pytest-asyncio)
|
|
98
|
-
|
|
99
|
-
When writing tests for async Python code:
|
|
100
|
-
|
|
101
|
-
1. **Verify setup before writing tests**:
|
|
102
|
-
- Confirm `pytest-asyncio` is in project dependencies
|
|
103
|
-
- Confirm `asyncio_mode` is set in `pyproject.toml` `[tool.pytest.ini_options]` (recommend `"auto"`)
|
|
104
|
-
- If neither is configured, warn the caller and suggest setup before proceeding
|
|
105
|
-
|
|
106
|
-
2. **Writing async test functions**:
|
|
107
|
-
- With `asyncio_mode = "auto"`: just write `async def test_something():` — no decorator needed
|
|
108
|
-
- With `asyncio_mode = "strict"`: every async test needs `@pytest.mark.asyncio`
|
|
109
|
-
- Without asyncio_mode set: always use `@pytest.mark.asyncio` decorator explicitly
|
|
110
|
-
|
|
111
|
-
3. **Async fixtures**:
|
|
112
|
-
- Use `@pytest_asyncio.fixture` (NOT `@pytest.fixture`) for async setup/teardown
|
|
113
|
-
- Scope rules: async fixtures default to `function` scope — use `scope="session"` carefully with async
|
|
114
|
-
|
|
115
|
-
4. **Common pitfalls**:
|
|
116
|
-
- Tests that `pass` without `await` — they run but don't execute the async path
|
|
117
|
-
- Missing `pytest-asyncio` makes `async def test_*` silently pass as empty coroutines
|
|
118
|
-
- Mixing sync and async fixtures can cause event loop errors
|
|
119
|
-
|
|
120
|
-
### Phase 4: Run Tests — Verify They FAIL (RED)
|
|
121
|
-
|
|
122
|
-
Use `Bash` to run ONLY the newly created test files (not full suite):
|
|
123
|
-
|
|
124
|
-
- **Jest**: `npx jest path/to/test.ts --no-coverage`
|
|
125
|
-
- **Vitest**: `npx vitest run path/to/test.ts`
|
|
126
|
-
- **pytest**: `pytest path/to/test_file.py -v` (if async tests and no `asyncio_mode` in config: add `--asyncio-mode=auto`)
|
|
127
|
-
- **Rust**: `cargo test test_module_name`
|
|
128
|
-
- **Go**: `go test ./path/to/package/... -run TestFunctionName`
|
|
129
|
-
|
|
130
|
-
**Hard gate**: ALL new tests MUST fail at this point.
|
|
131
|
-
|
|
132
|
-
- If ANY test passes before implementation exists → that test is not testing real behavior. Rewrite it to be stricter.
|
|
133
|
-
- If tests fail with import/syntax errors (not assertion errors) → fix the test code, re-run
|
|
134
|
-
|
|
135
|
-
### Phase 5: After Implementation — Verify Tests PASS (GREEN)
|
|
136
|
-
|
|
137
|
-
After `rune:fix` writes implementation code, run the same test command again:
|
|
138
|
-
|
|
139
|
-
1. ALL tests in the new test files MUST pass
|
|
140
|
-
2. Run the full test suite with `Bash` to check for regressions:
|
|
141
|
-
- `npm test`, `pytest`, `cargo test`, `go test ./...`
|
|
142
|
-
3. If any test fails: report clearly which test, what was expected, what was received
|
|
143
|
-
4. If an existing test now fails (regression): escalate to `rune:debug`
|
|
144
|
-
|
|
145
|
-
**Verification gate**: 100% of new tests pass AND 0 regressions in existing tests.
|
|
146
|
-
|
|
147
|
-
### Phase 6: Coverage Check
|
|
148
|
-
|
|
149
|
-
After GREEN phase, call `verification` to check coverage threshold (80% minimum):
|
|
150
|
-
|
|
151
|
-
- If coverage drops below 80%: identify uncovered lines, write additional tests
|
|
152
|
-
- Report coverage gaps with file:line references
|
|
153
|
-
|
|
154
|
-
### Phase 6.5: Diff-Aware Mode (optional)
|
|
155
|
-
|
|
156
|
-
When invoked with `mode: "diff-aware"` or by `cook` after implementation:
|
|
157
|
-
|
|
158
|
-
1. Run `git diff main --name-only` to get changed files
|
|
159
|
-
2. For each changed file, trace its **blast radius**: what imports it? what routes does it serve? what components render it?
|
|
160
|
-
3. Map changed files → affected routes/endpoints/pages
|
|
161
|
-
4. Prioritize tests: files with most downstream dependents get tested first
|
|
162
|
-
5. Generate targeted test commands that cover ONLY affected paths — skip unchanged modules
|
|
163
|
-
|
|
164
|
-
This mode is valuable for large codebases where running the full suite is slow. It answers: "what could this diff have broken?"
|
|
165
|
-
|
|
166
|
-
```
|
|
167
|
-
Input: git diff main --name-only
|
|
168
|
-
Output: Prioritized test plan targeting only affected paths
|
|
169
|
-
```
|
|
170
|
-
|
|
171
|
-
## Test Types — 4-Layer Methodology
|
|
172
|
-
|
|
173
|
-
Tests are organized in 4 layers. Each layer catches a different failure class. Higher layers are slower but catch integration issues lower layers miss.
|
|
174
|
-
|
|
175
|
-
| Layer | Type | What It Catches | Framework | Speed |
|
|
176
|
-
|-------|------|-----------------|-----------|-------|
|
|
177
|
-
| L1 | **Unit** | Logic bugs, boundary violations, pure function errors | jest/vitest/pytest/cargo test | Fast |
|
|
178
|
-
| L2 | **Integration** | API contract breaks, DB query errors, service interaction failures | supertest/httpx/reqwest | Medium |
|
|
179
|
-
| L3 | **True Backend** | Real tool/service output correctness (not just exit 0) | Same + real software invocation | Medium-Slow |
|
|
180
|
-
| L4 | **E2E / Subprocess** | Full workflow from user/agent perspective, installed app works | Playwright/Cypress/subprocess | Slow |
|
|
181
|
-
|
|
182
|
-
**Layer rules:**
|
|
183
|
-
- **L1 (Unit)**: Synthetic data, no external deps. Every function tested in isolation. Fast, deterministic, CI-friendly
|
|
184
|
-
- **L2 (Integration)**: Tests service boundaries — API endpoints, DB operations, message queues. May need test DB or mock server
|
|
185
|
-
- **L3 (True Backend)**: **Invokes the REAL tool/service** and verifies output programmatically. No graceful degradation — if the dependency isn't installed, tests FAIL (not skip). Verify: magic bytes, file size > 0, content structure. Print artifact paths for manual inspection
|
|
186
|
-
- **L4 (E2E/Subprocess)**: Tests the installed command/app via subprocess or browser automation. Full user workflow: input → process → output → verify
|
|
187
|
-
|
|
188
|
-
**"No graceful degradation" rule** (L3/L4): Hard dependencies MUST be installed. Tests MUST NOT skip or produce fake results when the dependency is missing. A silently skipping test is worse than a loudly failing test.
|
|
189
|
-
|
|
190
|
-
Additional modes:
|
|
191
|
-
|
|
192
|
-
| Type | When | Speed |
|
|
193
|
-
|------|------|-------|
|
|
194
|
-
| Regression | After bug fixes | Fast |
|
|
195
|
-
| Diff-aware | After implementation, large codebases (Phase 6.5) | Fast (targeted) |
|
|
196
|
-
|
|
197
|
-
## TEST.md — Test Plan + Results Document
|
|
198
|
-
|
|
199
|
-
For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` in the test directory. This is BOTH a planning doc (written BEFORE tests) and results doc (appended AFTER tests pass).
|
|
200
|
-
|
|
201
|
-
### Before writing tests — write the plan:
|
|
202
|
-
```markdown
|
|
203
|
-
# Test Plan: [Feature Name]
|
|
204
|
-
|
|
205
|
-
## Test Inventory
|
|
206
|
-
- `test_core.py`: ~XX unit tests planned (L1)
|
|
207
|
-
- `test_integration.py`: ~XX integration tests planned (L2)
|
|
208
|
-
- `test_e2e.py`: ~XX E2E tests planned (L3/L4)
|
|
209
|
-
|
|
210
|
-
## Unit Test Plan (L1)
|
|
211
|
-
| Module | Functions | Edge Cases | Est. Tests | Req IDs |
|
|
212
|
-
|--------|-----------|------------|------------|---------|
|
|
213
|
-
| `core/auth.py` | login, register, refresh | expired token, invalid creds, rate limit | 12 | REQ-001, REQ-003 |
|
|
214
|
-
|
|
215
|
-
## E2E Scenarios (L3/L4)
|
|
216
|
-
| Workflow | Simulates | Operations | Verified | Req IDs |
|
|
217
|
-
|----------|-----------|------------|----------|---------|
|
|
218
|
-
| User signup | New user onboarding | register → verify → login | Token valid, profile created | REQ-005 |
|
|
219
|
-
|
|
220
|
-
## Realistic Workflow Scenarios
|
|
221
|
-
- **[Name]**: [Step 1] → [Step 2] → verify [output properties]
|
|
222
|
-
```
|
|
223
|
-
|
|
224
|
-
### After tests pass — append results:
|
|
225
|
-
```markdown
|
|
226
|
-
## Test Results
|
|
227
|
-
[Paste full `pytest -v --tb=no` or `npm test` output]
|
|
228
|
-
|
|
229
|
-
## Summary
|
|
230
|
-
- Total: XX | Passed: XX | Failed: 0
|
|
231
|
-
- Execution time: X.Xs | Coverage: XX%
|
|
232
|
-
|
|
233
|
-
## Requirement Coverage
|
|
234
|
-
| Req ID | Test File(s) | Status |
|
|
235
|
-
|--------|-------------|--------|
|
|
236
|
-
| REQ-001 | `test_auth.py::test_login` | ✅ Covered |
|
|
237
|
-
| REQ-002 | — | ❌ Not covered |
|
|
238
|
-
|
|
239
|
-
## Gaps
|
|
240
|
-
- [Areas not covered and why]
|
|
241
|
-
```
|
|
242
|
-
|
|
243
|
-
**Why TEST.md**: Planning tests before code catches missing edge cases early. Appending results creates permanent evidence. One document = complete testing story.
|
|
244
|
-
|
|
245
|
-
## Skill Behavior Tests (Eval Scenarios)
|
|
246
|
-
|
|
247
|
-
For testing SKILL.md behavior (not code), use **Eval Scenarios** — unit tests for skill files, not code files.
|
|
248
|
-
|
|
249
|
-
### Eval Scenario Format
|
|
250
|
-
|
|
251
|
-
```markdown
|
|
252
|
-
## Eval: E[NN] — [scenario name]
|
|
253
|
-
|
|
254
|
-
### Prompt
|
|
255
|
-
[The exact situation/message an agent receives]
|
|
256
|
-
|
|
257
|
-
### Expected Reasoning
|
|
258
|
-
[Step-by-step reasoning the agent SHOULD follow]
|
|
259
|
-
|
|
260
|
-
### Must Include
|
|
261
|
-
- [Assertion 1: what the output MUST contain or do]
|
|
262
|
-
- [Assertion 2]
|
|
263
|
-
|
|
264
|
-
### Must NOT
|
|
265
|
-
- [Anti-pattern 1: what the output MUST NOT do]
|
|
266
|
-
- [Anti-pattern 2]
|
|
267
|
-
|
|
268
|
-
### Category
|
|
269
|
-
happy-path | adversarial | edge-case | jailbreak | credential-leak
|
|
270
|
-
```
|
|
271
|
-
|
|
272
|
-
### Eval Coverage Requirements
|
|
273
|
-
|
|
274
|
-
A skill is **behavior-tested** when it has evals covering:
|
|
275
|
-
|
|
276
|
-
| Category | Min Evals | Purpose |
|
|
277
|
-
|----------|-----------|---------|
|
|
278
|
-
| Happy path | 1 | Core workflow executes correctly |
|
|
279
|
-
| Edge case | 1 | Empty input, missing context, unusual state |
|
|
280
|
-
| Adversarial | 1 | Time pressure, sunk cost, authority pressure |
|
|
281
|
-
| Jailbreak / injection | 1 | Prompt injection attempt, "ignore instructions" |
|
|
282
|
-
|
|
283
|
-
**Minimum**: 4 evals per skill (1 per category). Security-critical skills (sentinel, safeguard): 8+ evals.
|
|
284
|
-
|
|
285
|
-
### Eval Storage
|
|
286
|
-
|
|
287
|
-
Save eval files as `skills/<name>/evals.md`. Each eval is a numbered scenario (E01–E24 range). skill-forge Phase 7 checks for evals presence before ship.
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
- If test
|
|
294
|
-
- If
|
|
295
|
-
- If
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
- `
|
|
301
|
-
- `
|
|
302
|
-
- `
|
|
303
|
-
- `
|
|
304
|
-
- `
|
|
305
|
-
- `
|
|
306
|
-
- `
|
|
307
|
-
- `
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
- `
|
|
313
|
-
- `
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
- `
|
|
321
|
-
- `
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
- `
|
|
327
|
-
- `
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
- `test` ↔ `
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
| "
|
|
340
|
-
| "I
|
|
341
|
-
| "
|
|
342
|
-
| "
|
|
343
|
-
| "
|
|
344
|
-
| "
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
const
|
|
359
|
-
const
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
AC-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
**
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
-
|
|
528
|
-
-
|
|
529
|
-
-
|
|
530
|
-
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
|
548
|
-
|
|
549
|
-
|
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
|
556
|
-
|
|
557
|
-
|
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
```
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
-
|
|
574
|
-
-
|
|
575
|
-
-
|
|
576
|
-
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
- Test Report emitted with framework, test count, RED/GREEN status, and coverage
|
|
587
|
-
- Self-Validation: all checks passed
|
|
588
|
-
|
|
589
|
-
## Cost Profile
|
|
590
|
-
|
|
591
|
-
~$0.03-0.08 per invocation. Sonnet for writing tests, Bash for running them. Frequent invocation in TDD workflow.
|
|
592
|
-
|
|
593
|
-
**Scope guardrail**: Do not modify source or implementation files to make tests pass unless explicitly delegated by the parent agent.
|
|
1
|
+
---
|
|
2
|
+
name: test
|
|
3
|
+
description: "TDD test writer. Writes failing tests FIRST (red), then verifies they pass after implementation (green). Covers unit, integration, and e2e tests."
|
|
4
|
+
metadata:
|
|
5
|
+
author: runedev
|
|
6
|
+
version: "1.1.0"
|
|
7
|
+
layer: L2
|
|
8
|
+
model: sonnet
|
|
9
|
+
group: development
|
|
10
|
+
tools: "Read, Write, Edit, Bash, Glob, Grep"
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
# test
|
|
14
|
+
|
|
15
|
+
<HARD-GATE>
|
|
16
|
+
Tests define the EXPECTED BEHAVIOR. They MUST be written BEFORE implementation code.
|
|
17
|
+
If tests pass without implementation → the tests are wrong. Rewrite them.
|
|
18
|
+
The only exception: when retrofitting tests for existing untested code.
|
|
19
|
+
|
|
20
|
+
THE IRON LAW: Write code before test? DELETE IT. Start over.
|
|
21
|
+
- Do NOT keep it as "reference"
|
|
22
|
+
- Do NOT "adapt" it while writing tests
|
|
23
|
+
- Do NOT look at it to "inform" test design
|
|
24
|
+
- Delete means delete. `git checkout -- <file>` or remove the changes entirely.
|
|
25
|
+
This is not negotiable. This is not optional. "But I already wrote it" is a sunk cost fallacy.
|
|
26
|
+
|
|
27
|
+
ROLE BOUNDARY: Test writes TEST FILES only. NEVER modify source/implementation files.
|
|
28
|
+
- Do NOT "quickly fix" a broken import in source to make tests run
|
|
29
|
+
- Do NOT refactor source code to be "more testable"
|
|
30
|
+
- Do NOT add missing exports to source files
|
|
31
|
+
- If source needs changes → hand off to `rune:fix`. Test's job ends at the test file.
|
|
32
|
+
This separation ensures test never writes code biased toward passing its own tests.
|
|
33
|
+
</HARD-GATE>
|
|
34
|
+
|
|
35
|
+
## Instructions
|
|
36
|
+
|
|
37
|
+
### Phase 1: Understand What to Test
|
|
38
|
+
|
|
39
|
+
1. Read the implementation plan or task description carefully
|
|
40
|
+
2. Use `Glob` to find existing test files: `**/*.test.*`, `**/*.spec.*`, `**/test_*`
|
|
41
|
+
3. Use `Read` on 2-3 existing test files to understand:
|
|
42
|
+
- Test framework in use
|
|
43
|
+
- File naming convention (e.g., `foo.test.ts` mirrors `foo.ts`)
|
|
44
|
+
- Test directory structure (co-located vs `__tests__/` vs `tests/`)
|
|
45
|
+
- Assertion style and patterns
|
|
46
|
+
4. Use `Glob` to find the source file(s) being tested
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
TodoWrite: [
|
|
50
|
+
{ content: "Understand scope and find existing test patterns", status: "in_progress" },
|
|
51
|
+
{ content: "Detect test framework and conventions", status: "pending" },
|
|
52
|
+
{ content: "Write failing tests (RED phase)", status: "pending" },
|
|
53
|
+
{ content: "Run tests — verify they FAIL", status: "pending" },
|
|
54
|
+
{ content: "After implementation: verify tests PASS (GREEN phase)", status: "pending" }
|
|
55
|
+
]
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
### Phase 2: Detect Test Framework
|
|
59
|
+
|
|
60
|
+
Use `Glob` to find config files and identify the framework:
|
|
61
|
+
|
|
62
|
+
- `jest.config.*` or `"jest"` key in `package.json` → Jest
|
|
63
|
+
- `vitest.config.*` or `"vitest"` key in `package.json` → Vitest
|
|
64
|
+
- `pytest.ini`, `[tool.pytest.ini_options]` in `pyproject.toml` → pytest
|
|
65
|
+
- **Async check**: If pytest detected AND source files contain `async def`:
|
|
66
|
+
- Check if `pytest-asyncio` is in dependencies (`pyproject.toml [project.dependencies]` or `[project.optional-dependencies]`)
|
|
67
|
+
- Check if `asyncio_mode` is set in `[tool.pytest.ini_options]` (values: `auto`, `strict`, or absent)
|
|
68
|
+
- If async code exists but no `asyncio_mode` configured → **WARN**: "pytest-asyncio not configured. Async tests may silently pass without executing async code. Recommend adding `asyncio_mode = \"auto\"` to `[tool.pytest.ini_options]` in pyproject.toml."
|
|
69
|
+
- `Cargo.toml` with `#[cfg(test)]` pattern → built-in `cargo test`
|
|
70
|
+
- `*_test.go` files present → built-in `go test`
|
|
71
|
+
- `cypress.config.*` → Cypress (E2E)
|
|
72
|
+
- `playwright.config.*` → Playwright (E2E)
|
|
73
|
+
|
|
74
|
+
**Verification gate**: Framework identified before writing any test code.
|
|
75
|
+
|
|
76
|
+
### Phase 3: Write Failing Tests
|
|
77
|
+
|
|
78
|
+
Use `Write` to create test files following the detected conventions:
|
|
79
|
+
|
|
80
|
+
1. Mirror source file location: if source is `src/auth/login.ts`, test is `src/auth/login.test.ts`
|
|
81
|
+
2. Structure tests with clear `describe` / `it` blocks (or language equivalent):
|
|
82
|
+
- `describe('Feature name')`
|
|
83
|
+
- `it('should [expected behavior] when [condition]')`
|
|
84
|
+
3. Cover all three categories:
|
|
85
|
+
- **Happy path**: valid inputs, expected success output
|
|
86
|
+
- **Edge cases**: empty input, boundary values, large input
|
|
87
|
+
- **Error cases**: invalid input, missing data, network failure simulation
|
|
88
|
+
|
|
89
|
+
4. Use proper assertions. Do NOT use implementation details — test behavior:
|
|
90
|
+
- Jest/Vitest: `expect(result).toBe(expected)`
|
|
91
|
+
- pytest: `assert result == expected`
|
|
92
|
+
- Rust: `assert_eq!(result, expected)`
|
|
93
|
+
- Go: `if result != expected { t.Errorf(...) }`
|
|
94
|
+
|
|
95
|
+
5. For async code: use `async/await` or pytest `@pytest.mark.asyncio`
|
|
96
|
+
|
|
97
|
+
#### Python Async Tests (pytest-asyncio)
|
|
98
|
+
|
|
99
|
+
When writing tests for async Python code:
|
|
100
|
+
|
|
101
|
+
1. **Verify setup before writing tests**:
|
|
102
|
+
- Confirm `pytest-asyncio` is in project dependencies
|
|
103
|
+
- Confirm `asyncio_mode` is set in `pyproject.toml` `[tool.pytest.ini_options]` (recommend `"auto"`)
|
|
104
|
+
- If neither is configured, warn the caller and suggest setup before proceeding
|
|
105
|
+
|
|
106
|
+
2. **Writing async test functions**:
|
|
107
|
+
- With `asyncio_mode = "auto"`: just write `async def test_something():` — no decorator needed
|
|
108
|
+
- With `asyncio_mode = "strict"`: every async test needs `@pytest.mark.asyncio`
|
|
109
|
+
- Without asyncio_mode set: always use `@pytest.mark.asyncio` decorator explicitly
|
|
110
|
+
|
|
111
|
+
3. **Async fixtures**:
|
|
112
|
+
- Use `@pytest_asyncio.fixture` (NOT `@pytest.fixture`) for async setup/teardown
|
|
113
|
+
- Scope rules: async fixtures default to `function` scope — use `scope="session"` carefully with async
|
|
114
|
+
|
|
115
|
+
4. **Common pitfalls**:
|
|
116
|
+
- Tests that `pass` without `await` — they run but don't execute the async path
|
|
117
|
+
- Missing `pytest-asyncio` makes `async def test_*` silently pass as empty coroutines
|
|
118
|
+
- Mixing sync and async fixtures can cause event loop errors
|
|
119
|
+
|
|
120
|
+
### Phase 4: Run Tests — Verify They FAIL (RED)
|
|
121
|
+
|
|
122
|
+
Use `Bash` to run ONLY the newly created test files (not full suite):
|
|
123
|
+
|
|
124
|
+
- **Jest**: `npx jest path/to/test.ts --no-coverage`
|
|
125
|
+
- **Vitest**: `npx vitest run path/to/test.ts`
|
|
126
|
+
- **pytest**: `pytest path/to/test_file.py -v` (if async tests and no `asyncio_mode` in config: add `--asyncio-mode=auto`)
|
|
127
|
+
- **Rust**: `cargo test test_module_name`
|
|
128
|
+
- **Go**: `go test ./path/to/package/... -run TestFunctionName`
|
|
129
|
+
|
|
130
|
+
**Hard gate**: ALL new tests MUST fail at this point.
|
|
131
|
+
|
|
132
|
+
- If ANY test passes before implementation exists → that test is not testing real behavior. Rewrite it to be stricter.
|
|
133
|
+
- If tests fail with import/syntax errors (not assertion errors) → fix the test code, re-run
|
|
134
|
+
|
|
135
|
+
### Phase 5: After Implementation — Verify Tests PASS (GREEN)
|
|
136
|
+
|
|
137
|
+
After `rune:fix` writes implementation code, run the same test command again:
|
|
138
|
+
|
|
139
|
+
1. ALL tests in the new test files MUST pass
|
|
140
|
+
2. Run the full test suite with `Bash` to check for regressions:
|
|
141
|
+
- `npm test`, `pytest`, `cargo test`, `go test ./...`
|
|
142
|
+
3. If any test fails: report clearly which test, what was expected, what was received
|
|
143
|
+
4. If an existing test now fails (regression): escalate to `rune:debug`
|
|
144
|
+
|
|
145
|
+
**Verification gate**: 100% of new tests pass AND 0 regressions in existing tests.
|
|
146
|
+
|
|
147
|
+
### Phase 6: Coverage Check
|
|
148
|
+
|
|
149
|
+
After GREEN phase, call `verification` to check coverage threshold (80% minimum):
|
|
150
|
+
|
|
151
|
+
- If coverage drops below 80%: identify uncovered lines, write additional tests
|
|
152
|
+
- Report coverage gaps with file:line references
|
|
153
|
+
|
|
154
|
+
### Phase 6.5: Diff-Aware Mode (optional)
|
|
155
|
+
|
|
156
|
+
When invoked with `mode: "diff-aware"` or by `cook` after implementation:
|
|
157
|
+
|
|
158
|
+
1. Run `git diff main --name-only` to get changed files
|
|
159
|
+
2. For each changed file, trace its **blast radius**: what imports it? what routes does it serve? what components render it?
|
|
160
|
+
3. Map changed files → affected routes/endpoints/pages
|
|
161
|
+
4. Prioritize tests: files with most downstream dependents get tested first
|
|
162
|
+
5. Generate targeted test commands that cover ONLY affected paths — skip unchanged modules
|
|
163
|
+
|
|
164
|
+
This mode is valuable for large codebases where running the full suite is slow. It answers: "what could this diff have broken?"
|
|
165
|
+
|
|
166
|
+
```
|
|
167
|
+
Input: git diff main --name-only
|
|
168
|
+
Output: Prioritized test plan targeting only affected paths
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
## Test Types — 4-Layer Methodology
|
|
172
|
+
|
|
173
|
+
Tests are organized in 4 layers. Each layer catches a different failure class. Higher layers are slower but catch integration issues lower layers miss.
|
|
174
|
+
|
|
175
|
+
| Layer | Type | What It Catches | Framework | Speed |
|
|
176
|
+
|-------|------|-----------------|-----------|-------|
|
|
177
|
+
| L1 | **Unit** | Logic bugs, boundary violations, pure function errors | jest/vitest/pytest/cargo test | Fast |
|
|
178
|
+
| L2 | **Integration** | API contract breaks, DB query errors, service interaction failures | supertest/httpx/reqwest | Medium |
|
|
179
|
+
| L3 | **True Backend** | Real tool/service output correctness (not just exit 0) | Same + real software invocation | Medium-Slow |
|
|
180
|
+
| L4 | **E2E / Subprocess** | Full workflow from user/agent perspective, installed app works | Playwright/Cypress/subprocess | Slow |
|
|
181
|
+
|
|
182
|
+
**Layer rules:**
|
|
183
|
+
- **L1 (Unit)**: Synthetic data, no external deps. Every function tested in isolation. Fast, deterministic, CI-friendly
|
|
184
|
+
- **L2 (Integration)**: Tests service boundaries — API endpoints, DB operations, message queues. May need test DB or mock server
|
|
185
|
+
- **L3 (True Backend)**: **Invokes the REAL tool/service** and verifies output programmatically. No graceful degradation — if the dependency isn't installed, tests FAIL (not skip). Verify: magic bytes, file size > 0, content structure. Print artifact paths for manual inspection
|
|
186
|
+
- **L4 (E2E/Subprocess)**: Tests the installed command/app via subprocess or browser automation. Full user workflow: input → process → output → verify
|
|
187
|
+
|
|
188
|
+
**"No graceful degradation" rule** (L3/L4): Hard dependencies MUST be installed. Tests MUST NOT skip or produce fake results when the dependency is missing. A silently skipping test is worse than a loudly failing test.
|
|
189
|
+
|
|
190
|
+
Additional modes:
|
|
191
|
+
|
|
192
|
+
| Type | When | Speed |
|
|
193
|
+
|------|------|-------|
|
|
194
|
+
| Regression | After bug fixes | Fast |
|
|
195
|
+
| Diff-aware | After implementation, large codebases (Phase 6.5) | Fast (targeted) |
|
|
196
|
+
|
|
197
|
+
## TEST.md — Test Plan + Results Document
|
|
198
|
+
|
|
199
|
+
For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` in the test directory. This is BOTH a planning doc (written BEFORE tests) and results doc (appended AFTER tests pass).
|
|
200
|
+
|
|
201
|
+
### Before writing tests — write the plan:
|
|
202
|
+
```markdown
|
|
203
|
+
# Test Plan: [Feature Name]
|
|
204
|
+
|
|
205
|
+
## Test Inventory
|
|
206
|
+
- `test_core.py`: ~XX unit tests planned (L1)
|
|
207
|
+
- `test_integration.py`: ~XX integration tests planned (L2)
|
|
208
|
+
- `test_e2e.py`: ~XX E2E tests planned (L3/L4)
|
|
209
|
+
|
|
210
|
+
## Unit Test Plan (L1)
|
|
211
|
+
| Module | Functions | Edge Cases | Est. Tests | Req IDs |
|
|
212
|
+
|--------|-----------|------------|------------|---------|
|
|
213
|
+
| `core/auth.py` | login, register, refresh | expired token, invalid creds, rate limit | 12 | REQ-001, REQ-003 |
|
|
214
|
+
|
|
215
|
+
## E2E Scenarios (L3/L4)
|
|
216
|
+
| Workflow | Simulates | Operations | Verified | Req IDs |
|
|
217
|
+
|----------|-----------|------------|----------|---------|
|
|
218
|
+
| User signup | New user onboarding | register → verify → login | Token valid, profile created | REQ-005 |
|
|
219
|
+
|
|
220
|
+
## Realistic Workflow Scenarios
|
|
221
|
+
- **[Name]**: [Step 1] → [Step 2] → verify [output properties]
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
### After tests pass — append results:
|
|
225
|
+
```markdown
|
|
226
|
+
## Test Results
|
|
227
|
+
[Paste full `pytest -v --tb=no` or `npm test` output]
|
|
228
|
+
|
|
229
|
+
## Summary
|
|
230
|
+
- Total: XX | Passed: XX | Failed: 0
|
|
231
|
+
- Execution time: X.Xs | Coverage: XX%
|
|
232
|
+
|
|
233
|
+
## Requirement Coverage
|
|
234
|
+
| Req ID | Test File(s) | Status |
|
|
235
|
+
|--------|-------------|--------|
|
|
236
|
+
| REQ-001 | `test_auth.py::test_login` | ✅ Covered |
|
|
237
|
+
| REQ-002 | — | ❌ Not covered |
|
|
238
|
+
|
|
239
|
+
## Gaps
|
|
240
|
+
- [Areas not covered and why]
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
**Why TEST.md**: Planning tests before code catches missing edge cases early. Appending results creates permanent evidence. One document = complete testing story.
|
|
244
|
+
|
|
245
|
+
## Skill Behavior Tests (Eval Scenarios)
|
|
246
|
+
|
|
247
|
+
For testing SKILL.md behavior (not code), use **Eval Scenarios** — unit tests for skill files, not code files.
|
|
248
|
+
|
|
249
|
+
### Eval Scenario Format
|
|
250
|
+
|
|
251
|
+
```markdown
|
|
252
|
+
## Eval: E[NN] — [scenario name]
|
|
253
|
+
|
|
254
|
+
### Prompt
|
|
255
|
+
[The exact situation/message an agent receives]
|
|
256
|
+
|
|
257
|
+
### Expected Reasoning
|
|
258
|
+
[Step-by-step reasoning the agent SHOULD follow]
|
|
259
|
+
|
|
260
|
+
### Must Include
|
|
261
|
+
- [Assertion 1: what the output MUST contain or do]
|
|
262
|
+
- [Assertion 2]
|
|
263
|
+
|
|
264
|
+
### Must NOT
|
|
265
|
+
- [Anti-pattern 1: what the output MUST NOT do]
|
|
266
|
+
- [Anti-pattern 2]
|
|
267
|
+
|
|
268
|
+
### Category
|
|
269
|
+
happy-path | adversarial | edge-case | jailbreak | credential-leak
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
### Eval Coverage Requirements
|
|
273
|
+
|
|
274
|
+
A skill is **behavior-tested** when it has evals covering:
|
|
275
|
+
|
|
276
|
+
| Category | Min Evals | Purpose |
|
|
277
|
+
|----------|-----------|---------|
|
|
278
|
+
| Happy path | 1 | Core workflow executes correctly |
|
|
279
|
+
| Edge case | 1 | Empty input, missing context, unusual state |
|
|
280
|
+
| Adversarial | 1 | Time pressure, sunk cost, authority pressure |
|
|
281
|
+
| Jailbreak / injection | 1 | Prompt injection attempt, "ignore instructions" |
|
|
282
|
+
|
|
283
|
+
**Minimum**: 4 evals per skill (1 per category). Security-critical skills (sentinel, safeguard): 8+ evals.
|
|
284
|
+
|
|
285
|
+
### Eval Storage
|
|
286
|
+
|
|
287
|
+
Save eval files as `skills/<name>/evals.md`. Each eval is a numbered scenario (E01–E24 range). skill-forge Phase 7 checks for evals presence before ship.
|
|
288
|
+
|
|
289
|
+
|
|
290
|
+
## Error Recovery
|
|
291
|
+
|
|
292
|
+
- If test framework not found: ask calling skill to specify, or check `package.json` `devDependencies`
|
|
293
|
+
- If `Write` to test file fails: check if directory exists, create it first with `Bash mkdir -p`
|
|
294
|
+
- If tests error on import (module not found): check that source file path is correct, adjust imports
|
|
295
|
+
- If `Bash` test runner hangs beyond 120 seconds: kill and report as TIMEOUT
|
|
296
|
+
|
|
297
|
+
## Called By (inbound)
|
|
298
|
+
|
|
299
|
+
- `cook` (L1): Phase 3 TEST — write tests first
|
|
300
|
+
- `fix` (L2): verify fix passes tests
|
|
301
|
+
- `review` (L2): untested edge case found → write test for it
|
|
302
|
+
- `deploy` (L2): pre-deployment full test suite
|
|
303
|
+
- `preflight` (L2): run targeted regression tests on affected code
|
|
304
|
+
- `surgeon` (L2): verify refactored code
|
|
305
|
+
- `launch` (L1): pre-deployment test suite
|
|
306
|
+
- `safeguard` (L2): writing characterization tests for legacy code
|
|
307
|
+
- `review-intake` (L2): write tests for issues identified during review intake
|
|
308
|
+
|
|
309
|
+
## Calls (outbound)
|
|
310
|
+
|
|
311
|
+
- `verification` (L3): Phase 6 — coverage check (80% minimum threshold)
|
|
312
|
+
- `browser-pilot` (L3): Phase 4 — e2e and visual testing for UI flows
|
|
313
|
+
- `debug` (L2): Phase 5 — when existing test regresses unexpectedly
|
|
314
|
+
|
|
315
|
+
## Data Flow
|
|
316
|
+
|
|
317
|
+
### Feeds Into →
|
|
318
|
+
|
|
319
|
+
- `cook` (L1): test results (pass/fail/coverage) → cook's Phase 5 quality gate evidence
|
|
320
|
+
- `completion-gate` (L3): test runner stdout → evidence for "tests pass" claims
|
|
321
|
+
- `fix` (L2): failing test output → fix's target (what to make green)
|
|
322
|
+
|
|
323
|
+
### Fed By ←
|
|
324
|
+
|
|
325
|
+
- `plan` (L2): phase file test tasks → test's RED phase targets (what to test)
|
|
326
|
+
- `review` (L2): untested edge cases found during review → new test targets
|
|
327
|
+
- `fix` (L2): implemented code → test's GREEN phase verification target
|
|
328
|
+
|
|
329
|
+
### Feedback Loops ↻
|
|
330
|
+
|
|
331
|
+
- `test` ↔ `fix`: test writes failing tests (RED) → fix implements to pass → test verifies (GREEN) → if new failures emerge, loop continues
|
|
332
|
+
- `test` ↔ `debug`: test discovers regression → debug diagnoses root cause → test writes regression test to prevent recurrence
|
|
333
|
+
|
|
334
|
+
## Anti-Rationalization Table
|
|
335
|
+
|
|
336
|
+
| Excuse | Reality |
|
|
337
|
+
|---|---|
|
|
338
|
+
| "Too simple to need tests first" | Simple code breaks. Test takes 30 seconds. Write it first. |
|
|
339
|
+
| "I'll write tests after — same result" | Tests-after = "what does this do?" Tests-first = "what SHOULD this do?" Completely different. |
|
|
340
|
+
| "I already wrote the code, let me just add tests" | Iron Law: delete the code. Start over with tests. Sunk cost is not an argument. |
|
|
341
|
+
| "Tests after achieve the same goals" | They don't. Tests-after are biased by the implementation you just wrote. |
|
|
342
|
+
| "It's about spirit not ritual" | Violating the letter IS violating the spirit. Write the test first. |
|
|
343
|
+
| "I mentally tested it" | Mental testing is not testing. Run the command, show the output. |
|
|
344
|
+
| "This is different because..." | It's not. Write the test first. |
|
|
345
|
+
|
|
346
|
+
## Advanced: Oracle-Injection E2E Testing
|
|
347
|
+
|
|
348
|
+
For **data pipelines, AI workflows, and multi-stage processing** where comparing full output structures is impractical, use oracle injection:
|
|
349
|
+
|
|
350
|
+
1. **Generate a UUID oracle token**: `const oracle = crypto.randomUUID()`
|
|
351
|
+
2. **Inject into synthetic input**: embed the oracle in realistic test data that flows through the pipeline
|
|
352
|
+
3. **Run the full pipeline**: input → all stages → output
|
|
353
|
+
4. **Search for oracle in output**: if found → data flowed end-to-end correctly
|
|
354
|
+
|
|
355
|
+
```
|
|
356
|
+
// Example: testing a document processing pipeline
|
|
357
|
+
const oracle = "ORACLE-" + crypto.randomUUID();
|
|
358
|
+
const testDoc = `Meeting notes: discussed ${oracle} integration timeline`;
|
|
359
|
+
const result = await pipeline.process(testDoc);
|
|
360
|
+
assert(result.output.includes(oracle), "Oracle not found — pipeline lost data");
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
**When to use**: E2E tests for pipelines with 3+ stages, LLM-based processing, ETL workflows, or any system where output structure is complex/non-deterministic but data preservation is critical.
|
|
364
|
+
|
|
365
|
+
**When NOT to use**: Unit tests, simple CRUD, or when exact output comparison is feasible.
|
|
366
|
+
|
|
367
|
+
|
|
368
|
+
## Spec→Test Traceability
|
|
369
|
+
|
|
370
|
+
When a plan with acceptance criteria exists (`.rune/features/<name>/plan.md` or phase file), every criterion MUST map to at least one test case.
|
|
371
|
+
|
|
372
|
+
```
|
|
373
|
+
Plan Acceptance Criteria → Test Case → Implementation
|
|
374
|
+
|
|
375
|
+
AC-1: "User can reset password via email" → test_password_reset_sends_email()
|
|
376
|
+
AC-2: "Rate limit: max 3 reset attempts/hour" → test_password_reset_rate_limit()
|
|
377
|
+
AC-3: "Expired tokens rejected" → test_expired_reset_token_rejected()
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
**Validation step** (after writing tests): Cross-check plan's acceptance criteria against test names. For each criterion:
|
|
381
|
+
- Has test → OK
|
|
382
|
+
- No test → flag as UNTESTED REQUIREMENT (more serious than uncovered lines)
|
|
383
|
+
|
|
384
|
+
**Why this is stronger than coverage**: Coverage checks that lines were EXECUTED. Traceability checks that INTENT was VERIFIED. You can have 100% coverage but miss a requirement if the test doesn't assert the right behavior.
|
|
385
|
+
|
|
386
|
+
**Skip if**: No plan exists (ad-hoc fix), or plan has no acceptance criteria section.
|
|
387
|
+
|
|
388
|
+
## Eval-Driven Development
|
|
389
|
+
|
|
390
|
+
Define **capability evals** and **regression evals** BEFORE writing implementation code. Evals go beyond unit tests — they verify that the agent/system can handle the feature's intent, not just its mechanics.
|
|
391
|
+
|
|
392
|
+
### Two Eval Types
|
|
393
|
+
|
|
394
|
+
| Type | Purpose | Pass Criteria | When |
|
|
395
|
+
|------|---------|---------------|------|
|
|
396
|
+
| **Capability eval** | Can the system do this new thing? | pass@k: ≥1 success in k attempts (k=3-5) | Before implementation |
|
|
397
|
+
| **Regression eval** | Did we break existing behavior? | pass^k: ALL k attempts must pass | After implementation |
|
|
398
|
+
|
|
399
|
+
**pass@k** (capability): At least 1 of k runs succeeds. Used for new features where some variance is acceptable. Threshold: ≥90% pass@3 for standard features, ≥95% pass@5 for critical paths.
|
|
400
|
+
|
|
401
|
+
**pass^k** (regression): ALL k runs must pass. Used for existing behavior that must never break. If ANY run fails, it's a regression. Threshold: 100% pass^3.
|
|
402
|
+
|
|
403
|
+
### Eval File Format
|
|
404
|
+
|
|
405
|
+
Store evals in `.rune/evals/<feature>.md`:
|
|
406
|
+
|
|
407
|
+
```markdown
|
|
408
|
+
# Eval: <feature name>
|
|
409
|
+
|
|
410
|
+
## Capability Evals (pass@k)
|
|
411
|
+
| ID | Description | k | Threshold | Status |
|
|
412
|
+
|----|-------------|---|-----------|--------|
|
|
413
|
+
| CAP-1 | [what the system should be able to do] | 3 | 90% | pending |
|
|
414
|
+
|
|
415
|
+
## Regression Evals (pass^k)
|
|
416
|
+
| ID | Description | k | Status |
|
|
417
|
+
|----|-------------|---|--------|
|
|
418
|
+
| REG-1 | [existing behavior that must not break] | 3 | pending |
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
### Anti-Pattern: Eval Overfitting
|
|
422
|
+
|
|
423
|
+
Do NOT overfit evals to specific prompts or known examples. Evals should test the **capability**, not the **exact input**.
|
|
424
|
+
|
|
425
|
+
- BAD: `"When user says 'hello', respond with 'Hi there!'"` — tests exact string match
|
|
426
|
+
- GOOD: `"When user greets, respond with a greeting"` — tests capability
|
|
427
|
+
|
|
428
|
+
### Integration with TDD
|
|
429
|
+
|
|
430
|
+
1. Write eval definitions (capability + regression) → `.rune/evals/<feature>.md`
|
|
431
|
+
2. Write unit/integration tests (RED phase) → test files
|
|
432
|
+
3. Implement feature (GREEN phase) → source files
|
|
433
|
+
4. Run evals to verify capability achieved + no regressions
|
|
434
|
+
5. Preflight checks eval results as part of quality gate
|
|
435
|
+
|
|
436
|
+
## Red Flags — STOP and Start Over
|
|
437
|
+
|
|
438
|
+
If you catch yourself with ANY of these, delete implementation code and restart with tests:
|
|
439
|
+
|
|
440
|
+
- Code exists before test file
|
|
441
|
+
- "I already manually tested it"
|
|
442
|
+
- "Tests after achieve the same purpose"
|
|
443
|
+
- "It's about spirit not ritual"
|
|
444
|
+
- "This is different because..."
|
|
445
|
+
- "Let me just finish this, then add tests"
|
|
446
|
+
|
|
447
|
+
**All of these mean: Delete code. Start over with TDD.**
|
|
448
|
+
|
|
449
|
+
## Constraints
|
|
450
|
+
|
|
451
|
+
1. MUST write tests BEFORE implementation code — if tests pass without implementation, they are wrong
|
|
452
|
+
2. MUST cover happy path + edge cases + error cases — not just happy path
|
|
453
|
+
3. MUST run tests to verify they FAIL before implementation exists (RED phase is mandatory)
|
|
454
|
+
4. MUST NOT write tests that test mock behavior instead of real code behavior
|
|
455
|
+
5. MUST achieve 80% coverage minimum — identify and fill gaps
|
|
456
|
+
6. MUST use the project's existing test framework and conventions — don't introduce a new one
|
|
457
|
+
7. MUST NOT say "tests pass" without showing actual test runner output
|
|
458
|
+
8. MUST delete implementation code written before tests — Iron Law, no exceptions
|
|
459
|
+
9. MUST show RED phase output (actual failure) — "I confirmed they fail" without output is REJECTED
|
|
460
|
+
10. MUST NOT modify source/implementation files — test writes test files ONLY, hand off source changes to rune:fix
|
|
461
|
+
|
|
462
|
+
## Mesh Gates
|
|
463
|
+
|
|
464
|
+
| Gate | Requires | If Missing |
|
|
465
|
+
|------|----------|------------|
|
|
466
|
+
| RED Gate | All new tests FAIL before implementation | If any pass, rewrite stricter tests |
|
|
467
|
+
| GREEN Gate | All tests PASS after implementation | Fix code, not tests |
|
|
468
|
+
| Coverage Gate | 80%+ coverage verified via verification | Write additional tests for gaps |
|
|
469
|
+
|
|
470
|
+
## Output Format
|
|
471
|
+
|
|
472
|
+
```
|
|
473
|
+
## Test Report
|
|
474
|
+
- **Framework**: [detected]
|
|
475
|
+
- **Files Created**: [list of new test file paths]
|
|
476
|
+
- **Tests Written**: [count]
|
|
477
|
+
- **Status**: RED (failing as expected) | GREEN (all passing)
|
|
478
|
+
|
|
479
|
+
### Test Cases
|
|
480
|
+
| Test | Status | Description |
|
|
481
|
+
|------|--------|-------------|
|
|
482
|
+
| `test_name` | FAIL/PASS | [what it tests] |
|
|
483
|
+
|
|
484
|
+
### Coverage
|
|
485
|
+
- Lines: [X]% | Branches: [Y]%
|
|
486
|
+
- Gaps: `path/to/file.ts:42-58` — uncovered branch (error handling)
|
|
487
|
+
|
|
488
|
+
### Regressions (if any)
|
|
489
|
+
- [existing test that broke, with error details]
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
## Testing Anti-Patterns (Gate Functions)
|
|
493
|
+
|
|
494
|
+
Before writing tests, check yourself against these 5 anti-patterns. Each has a **gate function** — a question you MUST answer before proceeding.
|
|
495
|
+
|
|
496
|
+
### Anti-Pattern 1: Testing Mock Behavior
|
|
497
|
+
Asserting that a mock exists (e.g., `testId="sidebar-mock"`) instead of testing real component behavior. You're proving the mock works, not the code.
|
|
498
|
+
**Gate**: "Am I testing real component behavior or just mock existence?" → If mock existence: STOP. Rewrite to test real behavior.
|
|
499
|
+
|
|
500
|
+
### Anti-Pattern 2: Test-Only Methods in Production
|
|
501
|
+
Adding `destroy()`, `reset()`, or `__testSetup()` methods to production classes that are ONLY called from test files. Production code should not know tests exist.
|
|
502
|
+
**Gate**: "Is this method only called by tests?" → If yes: STOP. Move to test utilities or test helper file, not production class.
|
|
503
|
+
|
|
504
|
+
### Anti-Pattern 3: Mocking Without Understanding Side Effects
|
|
505
|
+
Mocking a function without first understanding ALL its side effects. The real function may write config files, update caches, or emit events that downstream code depends on.
|
|
506
|
+
**Gate**: Before mocking, STOP and answer: "What side effects does the REAL function have? Does this test depend on any of those?" → Run with real implementation first, observe what happens, THEN add minimal mocking.
|
|
507
|
+
|
|
508
|
+
### Anti-Pattern 4: Incomplete Mocks
|
|
509
|
+
Partial mock missing fields that downstream code consumes. Your test passes because it only checks the fields you mocked, but production code reads fields your mock doesn't have → runtime crash.
|
|
510
|
+
**Iron Rule**: Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses. Examine actual API response / real data shape before writing mock.
|
|
511
|
+
|
|
512
|
+
### Anti-Pattern 5: Mock Setup Longer Than Test Logic
|
|
513
|
+
If mock setup is 30 lines and the actual test assertion is 3 lines, the test is testing infrastructure, not behavior. This is a code smell that indicates wrong abstraction level.
|
|
514
|
+
**Gate**: "Is my mock setup longer than my test logic?" → If yes: test at a higher level (integration) or extract mock factories.
|
|
515
|
+
|
|
516
|
+
### Anti-Pattern 6: Test Slop (Framework-Behavior Tests)
|
|
517
|
+
Tests that verify the framework works rather than YOUR code works. If the test would still pass with an empty component/function, it's testing infrastructure.
|
|
518
|
+
**Gate**: "Would this test pass if I deleted my business logic?" → If yes: STOP. Rewrite to test behavior that YOUR code introduces.
|
|
519
|
+
|
|
520
|
+
Examples of test slop:
|
|
521
|
+
- "renders without crashing" (tests that React works, not your component)
|
|
522
|
+
- "route responds with 200" without checking response body (tests Express, not your handler)
|
|
523
|
+
- Asserting a mock was called N times without checking the RESULT of those calls
|
|
524
|
+
- Type existence tests (`typeof result === 'object'`) when you should test the actual value
|
|
525
|
+
|
|
526
|
+
**Red flags — any of these means STOP and rethink:**
|
|
527
|
+
- Mock setup longer than test logic
|
|
528
|
+
- `*-mock` test IDs in assertions
|
|
529
|
+
- Methods only called in test files
|
|
530
|
+
- Can't explain in one sentence why a mock is needed
|
|
531
|
+
- Test would pass with empty implementation (test slop)
|
|
532
|
+
|
|
533
|
+
## Returns
|
|
534
|
+
|
|
535
|
+
| Artifact | Format | Location |
|
|
536
|
+
|----------|--------|----------|
|
|
537
|
+
| Test files | Source files | Co-located or `__tests__/` per project convention |
|
|
538
|
+
| Test plan + results | Markdown | `TEST.md` in test directory (non-trivial features only) |
|
|
539
|
+
| Eval scenarios | Markdown | `skills/<name>/evals.md` (for skill behavior testing) |
|
|
540
|
+
| Coverage report | Inline stdout | Shown in Test Report |
|
|
541
|
+
| Test Report | Markdown (inline) | Emitted to calling skill (cook, fix, review) |
|
|
542
|
+
|
|
543
|
+
## Sharp Edges
|
|
544
|
+
|
|
545
|
+
Known failure modes for this skill. Check these before declaring done.
|
|
546
|
+
|
|
547
|
+
| Failure Mode | Severity | Mitigation |
|
|
548
|
+
|---|---|---|
|
|
549
|
+
| Tests passing before implementation exists | CRITICAL | RED Gate: rewrite stricter tests — passing without code = not testing real behavior |
|
|
550
|
+
| Skipping the RED phase (not confirming FAIL) | HIGH | Run tests, confirm FAIL output before calling cook/fix to implement |
|
|
551
|
+
| Testing mock behavior instead of real code | HIGH | Anti-Pattern 1 gate: "Am I testing real behavior or mock existence?" |
|
|
552
|
+
| Mocking without understanding side effects | HIGH | Anti-Pattern 3 gate: run with real impl first, observe side effects, THEN mock minimally |
|
|
553
|
+
| Incomplete mocks missing downstream fields | HIGH | Anti-Pattern 4 iron rule: mock COMPLETE data structure, not just fields your test checks |
|
|
554
|
+
| Coverage below 80% without filling gaps | MEDIUM | Coverage Gate: identify uncovered lines and write additional tests |
|
|
555
|
+
| Introducing a new test framework instead of using existing one | MEDIUM | Constraint 6: detect framework first, use project's existing one always |
|
|
556
|
+
| Modifying source files to make tests work | HIGH | Role boundary: test writes test files ONLY — source changes go to rune:fix |
|
|
557
|
+
| Test-only methods leaking into production code | MEDIUM | Anti-Pattern 2 gate: if method only called by tests → move to test utilities |
|
|
558
|
+
|
|
559
|
+
## Self-Validation
|
|
560
|
+
|
|
561
|
+
```
|
|
562
|
+
SELF-VALIDATION (run before emitting Test Report):
|
|
563
|
+
- [ ] Every test file has at least one assertion — no empty test bodies
|
|
564
|
+
- [ ] RED phase output shows actual failures (not "0 tests") — tests were real, not stubs
|
|
565
|
+
- [ ] No test modifies source code — test files only, source changes belong to fix
|
|
566
|
+
- [ ] Test names describe behavior, not implementation ("should reject expired token" not "test function X")
|
|
567
|
+
- [ ] No mocks of the thing being tested — only mock external dependencies
|
|
568
|
+
- [ ] If BA requirements exist (REQ-xxx), every requirement has at least one test — check plan's Traceability Matrix
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+
## Done When
|
|
572
|
+
|
|
573
|
+
- Test framework detected from project config files
|
|
574
|
+
- Tests cover happy path + at least 2 edge cases + error case
|
|
575
|
+
- All new tests FAIL (RED phase — actual failure output shown)
|
|
576
|
+
- After implementation: all tests PASS (GREEN phase — actual pass output shown)
|
|
577
|
+
- Coverage ≥80% verified via verification
|
|
578
|
+
- Test Report emitted with framework, test count, RED/GREEN status, and coverage
|
|
579
|
+
- Self-Validation: all checks passed
|
|
580
|
+
|
|
581
|
+
## Cost Profile
|
|
582
|
+
|
|
583
|
+
~$0.03-0.08 per invocation. Sonnet for writing tests, Bash for running them. Frequent invocation in TDD workflow.
|
|
584
|
+
|
|
585
|
+
**Scope guardrail**: Do not modify source or implementation files to make tests pass unless explicitly delegated by the parent agent.
|