@rune-kit/rune 2.2.4 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/compiler/adapters/openclaw.js +63 -0
- package/compiler/emitter.js +10 -0
- package/docs/EXTENSION-TEMPLATE.md +18 -0
- package/docs/SKILL-TEMPLATE.md +46 -0
- package/docs/guides/index.html +84 -21
- package/docs/index.html +303 -37
- package/docs/script.js +236 -18
- package/docs/style.css +417 -59
- package/extensions/saas/PACK.md +13 -8
- package/extensions/saas/skills/billing-integration.md +82 -3
- package/package.json +7 -5
- package/skills/adversary/SKILL.md +12 -0
- package/skills/audit/SKILL.md +64 -1
- package/skills/autopsy/SKILL.md +12 -0
- package/skills/ba/SKILL.md +63 -1
- package/skills/brainstorm/SKILL.md +11 -0
- package/skills/completion-gate/SKILL.md +51 -2
- package/skills/context-engine/SKILL.md +83 -1
- package/skills/context-pack/SKILL.md +160 -0
- package/skills/cook/SKILL.md +657 -808
- package/skills/cook/references/deviation-rules.md +19 -0
- package/skills/cook/references/error-recovery.md +37 -0
- package/skills/cook/references/exit-conditions.md +31 -0
- package/skills/cook/references/loop-detection.md +39 -0
- package/skills/cook/references/mid-run-signals.md +31 -0
- package/skills/cook/references/output-format.md +40 -0
- package/skills/cook/references/pack-detection.md +82 -0
- package/skills/cook/references/pause-resume-template.md +38 -0
- package/skills/cook/references/rfc-template.md +52 -0
- package/skills/cook/references/sharp-edges.md +24 -0
- package/skills/cook/references/subagent-status.md +38 -0
- package/skills/db/SKILL.md +12 -0
- package/skills/debug/SKILL.md +81 -2
- package/skills/deploy/SKILL.md +56 -1
- package/skills/design/SKILL.md +9 -0
- package/skills/docs/SKILL.md +12 -0
- package/skills/docs-seeker/SKILL.md +11 -0
- package/skills/fix/SKILL.md +63 -3
- package/skills/git/SKILL.md +55 -1
- package/skills/incident/SKILL.md +10 -0
- package/skills/journal/SKILL.md +51 -3
- package/skills/launch/SKILL.md +12 -0
- package/skills/logic-guardian/SKILL.md +11 -0
- package/skills/marketing/SKILL.md +13 -0
- package/skills/mcp-builder/SKILL.md +13 -0
- package/skills/onboard/SKILL.md +54 -2
- package/skills/perf/SKILL.md +11 -0
- package/skills/plan/SKILL.md +342 -629
- package/skills/plan/references/completeness-scoring.md +37 -0
- package/skills/plan/references/outcome-block.md +41 -0
- package/skills/plan/references/plan-templates.md +193 -0
- package/skills/plan/references/wave-planning.md +44 -0
- package/skills/plan/references/workflow-registry.md +53 -0
- package/skills/preflight/SKILL.md +135 -1
- package/skills/rescue/SKILL.md +10 -0
- package/skills/research/SKILL.md +36 -8
- package/skills/retro/SKILL.md +325 -0
- package/skills/review/SKILL.md +100 -1
- package/skills/review-intake/SKILL.md +11 -0
- package/skills/safeguard/SKILL.md +12 -0
- package/skills/scaffold/SKILL.md +10 -0
- package/skills/scope-guard/SKILL.md +11 -0
- package/skills/scout/SKILL.md +9 -0
- package/skills/sentinel/SKILL.md +307 -320
- package/skills/sentinel/references/config-protection.md +52 -0
- package/skills/sentinel/references/destructive-commands.md +40 -0
- package/skills/sentinel/references/domain-hooks.md +73 -0
- package/skills/sentinel/references/framework-patterns.md +46 -0
- package/skills/sentinel/references/owasp-patterns.md +69 -0
- package/skills/sentinel/references/secret-patterns.md +40 -0
- package/skills/sentinel/references/skill-content-guard.md +55 -0
- package/skills/session-bridge/SKILL.md +60 -2
- package/skills/skill-forge/SKILL.md +131 -4
- package/skills/skill-router/SKILL.md +33 -1
- package/skills/surgeon/SKILL.md +12 -0
- package/skills/team/SKILL.md +35 -1
- package/skills/test/SKILL.md +211 -7
|
@@ -4,7 +4,7 @@ description: "Meta-enforcement layer that routes EVERY agent action through the
|
|
|
4
4
|
user-invocable: false
|
|
5
5
|
metadata:
|
|
6
6
|
author: runedev
|
|
7
|
-
version: "1.
|
|
7
|
+
version: "1.3.0"
|
|
8
8
|
layer: L0
|
|
9
9
|
model: haiku
|
|
10
10
|
group: orchestrator
|
|
@@ -64,6 +64,17 @@ Before standard routing, check if adaptive routing rules exist:
|
|
|
64
64
|
- Model hints are written by cook Phase 8 when debug-fix loops hit max retries on the same error pattern
|
|
65
65
|
- Model hints do NOT override explicit user model preferences
|
|
66
66
|
|
|
67
|
+
### Context Efficiency (Trigger-Table Pattern)
|
|
68
|
+
|
|
69
|
+
Skill-router's routing table above IS the trigger table — it maps keywords to skill paths without loading any skill content. Skills are loaded on-demand via the Skill tool only when routed. This keeps baseline context usage minimal.
|
|
70
|
+
|
|
71
|
+
**Rules for context efficiency:**
|
|
72
|
+
- NEVER read a SKILL.md to decide routing — use the routing table keywords
|
|
73
|
+
- NEVER load multiple skills speculatively — route to ONE, let it chain if needed
|
|
74
|
+
- Skill content is loaded by the Skill tool, not by skill-router reading files
|
|
75
|
+
|
|
76
|
+
> Source: affaan-m/everything-claude-code (91.9k★) — trigger-table lazy loading for 50%+ context reduction.
|
|
77
|
+
|
|
67
78
|
### Step 0.25 — Request Classifier (Fast-Path Filter)
|
|
68
79
|
|
|
69
80
|
Before intent classification, categorize the request into one of 5 types. This determines the **enforcement level** — how strictly routing must be followed.
|
|
@@ -284,6 +295,27 @@ These DO NOT need skill routing:
|
|
|
284
295
|
- Single-line factual answers with no code impact
|
|
285
296
|
- Resuming an already-active skill workflow
|
|
286
297
|
|
|
298
|
+
## Proactive Skill Recommendations (One-Hop Max)
|
|
299
|
+
|
|
300
|
+
At the end of a skill's workflow, skill-router MAY suggest a **complementary skill** — limited to ONE recommendation to prevent infinite referral chains.
|
|
301
|
+
|
|
302
|
+
| After This Skill | Suggest | Rationale |
|
|
303
|
+
|-----------------|---------|-----------|
|
|
304
|
+
| `debug` | `review` | Root cause found — review the fix area for broader issues |
|
|
305
|
+
| `fix` | `test` | Code changed — verify with tests |
|
|
306
|
+
| `plan` | `adversary` | Plan created — stress-test before implementation |
|
|
307
|
+
| `test` (GREEN) | `preflight` | Tests pass — check for edge cases and completeness |
|
|
308
|
+
| `review` (issues found) | `fix` | Issues identified — apply fixes |
|
|
309
|
+
| `sentinel` (findings) | `fix` | Security issues — remediate |
|
|
310
|
+
|
|
311
|
+
**Rules:**
|
|
312
|
+
- Hard limit: 1 hop. NEVER chain recommendations (fix→test→preflight→...). Suggest ONE, let the user decide.
|
|
313
|
+
- Announcement format: "Suggested next: `rune:<skill>` — [1-line reason]. Run it? (skip to continue)"
|
|
314
|
+
- User can disable with "no suggestions" or "just do what I asked"
|
|
315
|
+
- Inside `cook` orchestration: skip recommendations — cook already manages transitions
|
|
316
|
+
|
|
317
|
+
> Source: garrytan/gstack v0.9.1 (50.9k★) — one-hop skill chaining with proactive: false escape hatch.
|
|
318
|
+
|
|
287
319
|
## Output Format
|
|
288
320
|
|
|
289
321
|
### Routing Proof (Required in Every Code Response)
|
package/skills/surgeon/SKILL.md
CHANGED
|
@@ -198,6 +198,18 @@ Known failure modes for this skill. Check these before declaring done.
|
|
|
198
198
|
- Surgery committed with message format `refactor(<module>): <pattern> — <description>`
|
|
199
199
|
- journal updated with module health delta and remaining work
|
|
200
200
|
|
|
201
|
+
## Returns
|
|
202
|
+
|
|
203
|
+
| Artifact | Format | Location |
|
|
204
|
+
|----------|--------|----------|
|
|
205
|
+
| Refactored module | Edited source files (max 5) | in-place |
|
|
206
|
+
| Before/after diff | Git diff | via `git diff` |
|
|
207
|
+
| Surgery Report | Markdown | inline |
|
|
208
|
+
| Git commit(s) | Conventional commits | git history |
|
|
209
|
+
| Journal entry | Text | via `journal` L3 |
|
|
210
|
+
|
|
201
211
|
## Cost Profile
|
|
202
212
|
|
|
203
213
|
~3000-6000 tokens input, ~1000-2000 tokens output. Sonnet. One module per session.
|
|
214
|
+
|
|
215
|
+
**Scope guardrail:** surgeon operates on ONE module per session (max 5 files). Any work beyond that scope must be deferred to a separate surgeon session.
|
package/skills/team/SKILL.md
CHANGED
|
@@ -5,7 +5,7 @@ context: fork
|
|
|
5
5
|
agent: general-purpose
|
|
6
6
|
metadata:
|
|
7
7
|
author: runedev
|
|
8
|
-
version: "0.
|
|
8
|
+
version: "0.6.0"
|
|
9
9
|
layer: L1
|
|
10
10
|
model: opus
|
|
11
11
|
group: orchestrator
|
|
@@ -66,6 +66,28 @@ Lite Mode Phases:
|
|
|
66
66
|
|
|
67
67
|
Standard team workflow with worktree isolation (Phases 1-5 as documented below).
|
|
68
68
|
|
|
69
|
+
### Complexity Tiers (DAG Stage Selection)
|
|
70
|
+
|
|
71
|
+
Before decomposing, classify the task into a complexity tier. Each tier defines a different DAG (directed acyclic graph) of stages, ensuring the right amount of process for the task's complexity.
|
|
72
|
+
|
|
73
|
+
| Tier | Signals | DAG Stages | Context Windows |
|
|
74
|
+
|------|---------|------------|-----------------|
|
|
75
|
+
| **Trivial** | ≤3 files, single module, no shared contracts | impl → test | 1 (single cook) |
|
|
76
|
+
| **Medium** | 4-10 files, 2-3 modules, shared interfaces | research → plan → impl → test → review → fix | 3 (plan, impl+test, review+fix) |
|
|
77
|
+
| **Large** | 10+ files, 3+ modules, breaking changes or RFC | research → plan → impl → test → review₁ → fix → review₂ → final merge | 4+ (plan, impl+test, review₁+fix, review₂+merge) |
|
|
78
|
+
|
|
79
|
+
**Key principle — reviewer isolation**: The agent that writes code MUST NOT review its own code. Each review stage uses a **separate context window** (separate Task invocation) that has never seen the implementation reasoning. This prevents author bias from contaminating the review.
|
|
80
|
+
|
|
81
|
+
**Stage → Context Window mapping**:
|
|
82
|
+
- `research + plan` = Context Window 1 (opus — architectural reasoning)
|
|
83
|
+
- `impl + test` = Context Window 2 (sonnet — code writing)
|
|
84
|
+
- `review₁ + fix` = Context Window 3 (sonnet — fresh eyes, no impl context)
|
|
85
|
+
- `review₂ + merge` = Context Window 4 (sonnet — final verification, Large tier only)
|
|
86
|
+
|
|
87
|
+
**Merge queue**: When multiple streams complete at different times, use dependency order for merging. If a later stream's merge creates conflicts with an already-merged stream, provide the conflicting stream's cook report as **conflict context** to the resolution agent — never resolve blindly.
|
|
88
|
+
|
|
89
|
+
> Source: affaan-m/everything-claude-code (91.9k★) — RFC-driven DAG orchestration with reviewer isolation.
|
|
90
|
+
|
|
69
91
|
## Calls (outbound)
|
|
70
92
|
|
|
71
93
|
- `plan` (L2): high-level task decomposition into independent workstreams
|
|
@@ -433,6 +455,16 @@ Dependent streams → SEQUENTIAL (respecting dependency order)
|
|
|
433
455
|
All streams done → MERGE sequentially (avoid conflicts)
|
|
434
456
|
```
|
|
435
457
|
|
|
458
|
+
## Returns
|
|
459
|
+
|
|
460
|
+
| Artifact | Format | Location |
|
|
461
|
+
|----------|--------|----------|
|
|
462
|
+
| Workstream assignments | Markdown (inline) | NEXUS Handoff Templates emitted per stream |
|
|
463
|
+
| Cook Reports (per stream) | Markdown (inline) | Collected from each parallel cook instance |
|
|
464
|
+
| Merged implementation | Source files | `main` branch after Phase 4 merge |
|
|
465
|
+
| Integration test results | Inline stdout | Captured in Phase 5 verify |
|
|
466
|
+
| Team Report | Markdown (inline) | Emitted at end of session |
|
|
467
|
+
|
|
436
468
|
## Sharp Edges
|
|
437
469
|
|
|
438
470
|
Known failure modes for this skill. Check these before declaring done.
|
|
@@ -464,3 +496,5 @@ Known failure modes for this skill. Check these before declaring done.
|
|
|
464
496
|
## Cost Profile
|
|
465
497
|
|
|
466
498
|
~$0.20-0.50 per session. Opus for coordination. Most expensive orchestrator but handles largest tasks.
|
|
499
|
+
|
|
500
|
+
**Scope guardrail**: Do not invoke launch, rescue, or scaffold autonomously unless explicitly delegated by the parent agent.
|
package/skills/test/SKILL.md
CHANGED
|
@@ -3,7 +3,7 @@ name: test
|
|
|
3
3
|
description: "TDD test writer. Writes failing tests FIRST (red), then verifies they pass after implementation (green). Covers unit, integration, and e2e tests."
|
|
4
4
|
metadata:
|
|
5
5
|
author: runedev
|
|
6
|
-
version: "
|
|
6
|
+
version: "1.1.0"
|
|
7
7
|
layer: L2
|
|
8
8
|
model: sonnet
|
|
9
9
|
group: development
|
|
@@ -208,14 +208,14 @@ For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` i
|
|
|
208
208
|
- `test_e2e.py`: ~XX E2E tests planned (L3/L4)
|
|
209
209
|
|
|
210
210
|
## Unit Test Plan (L1)
|
|
211
|
-
| Module | Functions | Edge Cases | Est. Tests |
|
|
212
|
-
|
|
213
|
-
| `core/auth.py` | login, register, refresh | expired token, invalid creds, rate limit | 12 |
|
|
211
|
+
| Module | Functions | Edge Cases | Est. Tests | Req IDs |
|
|
212
|
+
|--------|-----------|------------|------------|---------|
|
|
213
|
+
| `core/auth.py` | login, register, refresh | expired token, invalid creds, rate limit | 12 | REQ-001, REQ-003 |
|
|
214
214
|
|
|
215
215
|
## E2E Scenarios (L3/L4)
|
|
216
|
-
| Workflow | Simulates | Operations | Verified |
|
|
217
|
-
|
|
218
|
-
| User signup | New user onboarding | register → verify → login | Token valid, profile created |
|
|
216
|
+
| Workflow | Simulates | Operations | Verified | Req IDs |
|
|
217
|
+
|----------|-----------|------------|----------|---------|
|
|
218
|
+
| User signup | New user onboarding | register → verify → login | Token valid, profile created | REQ-005 |
|
|
219
219
|
|
|
220
220
|
## Realistic Workflow Scenarios
|
|
221
221
|
- **[Name]**: [Step 1] → [Step 2] → verify [output properties]
|
|
@@ -230,12 +230,64 @@ For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` i
|
|
|
230
230
|
- Total: XX | Passed: XX | Failed: 0
|
|
231
231
|
- Execution time: X.Xs | Coverage: XX%
|
|
232
232
|
|
|
233
|
+
## Requirement Coverage
|
|
234
|
+
| Req ID | Test File(s) | Status |
|
|
235
|
+
|--------|-------------|--------|
|
|
236
|
+
| REQ-001 | `test_auth.py::test_login` | ✅ Covered |
|
|
237
|
+
| REQ-002 | — | ❌ Not covered |
|
|
238
|
+
|
|
233
239
|
## Gaps
|
|
234
240
|
- [Areas not covered and why]
|
|
235
241
|
```
|
|
236
242
|
|
|
237
243
|
**Why TEST.md**: Planning tests before code catches missing edge cases early. Appending results creates permanent evidence. One document = complete testing story.
|
|
238
244
|
|
|
245
|
+
## Skill Behavior Tests (Eval Scenarios)
|
|
246
|
+
|
|
247
|
+
For testing SKILL.md behavior (not code), use **Eval Scenarios** — unit tests for skill files, not code files.
|
|
248
|
+
|
|
249
|
+
### Eval Scenario Format
|
|
250
|
+
|
|
251
|
+
```markdown
|
|
252
|
+
## Eval: E[NN] — [scenario name]
|
|
253
|
+
|
|
254
|
+
### Prompt
|
|
255
|
+
[The exact situation/message an agent receives]
|
|
256
|
+
|
|
257
|
+
### Expected Reasoning
|
|
258
|
+
[Step-by-step reasoning the agent SHOULD follow]
|
|
259
|
+
|
|
260
|
+
### Must Include
|
|
261
|
+
- [Assertion 1: what the output MUST contain or do]
|
|
262
|
+
- [Assertion 2]
|
|
263
|
+
|
|
264
|
+
### Must NOT
|
|
265
|
+
- [Anti-pattern 1: what the output MUST NOT do]
|
|
266
|
+
- [Anti-pattern 2]
|
|
267
|
+
|
|
268
|
+
### Category
|
|
269
|
+
happy-path | adversarial | edge-case | jailbreak | credential-leak
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
### Eval Coverage Requirements
|
|
273
|
+
|
|
274
|
+
A skill is **behavior-tested** when it has evals covering:
|
|
275
|
+
|
|
276
|
+
| Category | Min Evals | Purpose |
|
|
277
|
+
|----------|-----------|---------|
|
|
278
|
+
| Happy path | 1 | Core workflow executes correctly |
|
|
279
|
+
| Edge case | 1 | Empty input, missing context, unusual state |
|
|
280
|
+
| Adversarial | 1 | Time pressure, sunk cost, authority pressure |
|
|
281
|
+
| Jailbreak / injection | 1 | Prompt injection attempt, "ignore instructions" |
|
|
282
|
+
|
|
283
|
+
**Minimum**: 4 evals per skill (1 per category). Security-critical skills (sentinel, safeguard): 8+ evals.
|
|
284
|
+
|
|
285
|
+
### Eval Storage
|
|
286
|
+
|
|
287
|
+
Save eval files as `skills/<name>/evals.md`. Each eval is a numbered scenario (E01–E24 range). skill-forge Phase 7 checks for evals presence before ship.
|
|
288
|
+
|
|
289
|
+
> Source: PhucMPham/zalo-agent-cli (35★) — 24 eval scenario format (E01-E24) with Must Include / Must NOT assertions.
|
|
290
|
+
|
|
239
291
|
## Error Recovery
|
|
240
292
|
|
|
241
293
|
- If test framework not found: ask calling skill to specify, or check `package.json` `devDependencies`
|
|
@@ -261,6 +313,25 @@ For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` i
|
|
|
261
313
|
- `browser-pilot` (L3): Phase 4 — e2e and visual testing for UI flows
|
|
262
314
|
- `debug` (L2): Phase 5 — when existing test regresses unexpectedly
|
|
263
315
|
|
|
316
|
+
## Data Flow
|
|
317
|
+
|
|
318
|
+
### Feeds Into →
|
|
319
|
+
|
|
320
|
+
- `cook` (L1): test results (pass/fail/coverage) → cook's Phase 5 quality gate evidence
|
|
321
|
+
- `completion-gate` (L3): test runner stdout → evidence for "tests pass" claims
|
|
322
|
+
- `fix` (L2): failing test output → fix's target (what to make green)
|
|
323
|
+
|
|
324
|
+
### Fed By ←
|
|
325
|
+
|
|
326
|
+
- `plan` (L2): phase file test tasks → test's RED phase targets (what to test)
|
|
327
|
+
- `review` (L2): untested edge cases found during review → new test targets
|
|
328
|
+
- `fix` (L2): implemented code → test's GREEN phase verification target
|
|
329
|
+
|
|
330
|
+
### Feedback Loops ↻
|
|
331
|
+
|
|
332
|
+
- `test` ↔ `fix`: test writes failing tests (RED) → fix implements to pass → test verifies (GREEN) → if new failures emerge, loop continues
|
|
333
|
+
- `test` ↔ `debug`: test discovers regression → debug diagnoses root cause → test writes regression test to prevent recurrence
|
|
334
|
+
|
|
264
335
|
## Anti-Rationalization Table
|
|
265
336
|
|
|
266
337
|
| Excuse | Reality |
|
|
@@ -273,6 +344,101 @@ For non-trivial features (3+ test files or 20+ test cases), create a `TEST.md` i
|
|
|
273
344
|
| "I mentally tested it" | Mental testing is not testing. Run the command, show the output. |
|
|
274
345
|
| "This is different because..." | It's not. Write the test first. |
|
|
275
346
|
|
|
347
|
+
## Advanced: Oracle-Injection E2E Testing
|
|
348
|
+
|
|
349
|
+
For **data pipelines, AI workflows, and multi-stage processing** where comparing full output structures is impractical, use oracle injection:
|
|
350
|
+
|
|
351
|
+
1. **Generate a UUID oracle token**: `const oracle = crypto.randomUUID()`
|
|
352
|
+
2. **Inject into synthetic input**: embed the oracle in realistic test data that flows through the pipeline
|
|
353
|
+
3. **Run the full pipeline**: input → all stages → output
|
|
354
|
+
4. **Search for oracle in output**: if found → data flowed end-to-end correctly
|
|
355
|
+
|
|
356
|
+
```
|
|
357
|
+
// Example: testing a document processing pipeline
|
|
358
|
+
const oracle = "ORACLE-" + crypto.randomUUID();
|
|
359
|
+
const testDoc = `Meeting notes: discussed ${oracle} integration timeline`;
|
|
360
|
+
const result = await pipeline.process(testDoc);
|
|
361
|
+
assert(result.output.includes(oracle), "Oracle not found — pipeline lost data");
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
**When to use**: E2E tests for pipelines with 3+ stages, LLM-based processing, ETL workflows, or any system where output structure is complex/non-deterministic but data preservation is critical.
|
|
365
|
+
|
|
366
|
+
**When NOT to use**: Unit tests, simple CRUD, or when exact output comparison is feasible.
|
|
367
|
+
|
|
368
|
+
> Source: airweave-ai/airweave (6k★) — Bongo oracle-injection pattern for pipeline E2E testing.
|
|
369
|
+
|
|
370
|
+
## Spec→Test Traceability
|
|
371
|
+
|
|
372
|
+
When a plan with acceptance criteria exists (`.rune/features/<name>/plan.md` or phase file), every criterion MUST map to at least one test case.
|
|
373
|
+
|
|
374
|
+
```
|
|
375
|
+
Plan Acceptance Criteria → Test Case → Implementation
|
|
376
|
+
|
|
377
|
+
AC-1: "User can reset password via email" → test_password_reset_sends_email()
|
|
378
|
+
AC-2: "Rate limit: max 3 reset attempts/hour" → test_password_reset_rate_limit()
|
|
379
|
+
AC-3: "Expired tokens rejected" → test_expired_reset_token_rejected()
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
**Validation step** (after writing tests): Cross-check plan's acceptance criteria against test names. For each criterion:
|
|
383
|
+
- Has test → OK
|
|
384
|
+
- No test → flag as UNTESTED REQUIREMENT (more serious than uncovered lines)
|
|
385
|
+
|
|
386
|
+
**Why this is stronger than coverage**: Coverage checks that lines were EXECUTED. Traceability checks that INTENT was VERIFIED. You can have 100% coverage but miss a requirement if the test doesn't assert the right behavior.
|
|
387
|
+
|
|
388
|
+
**Skip if**: No plan exists (ad-hoc fix), or plan has no acceptance criteria section.
|
|
389
|
+
|
|
390
|
+
> Source: Fission-AI/OpenSpec (32.8k★) — spec→test traceability validates intent, not just lines.
|
|
391
|
+
|
|
392
|
+
## Eval-Driven Development
|
|
393
|
+
|
|
394
|
+
Define **capability evals** and **regression evals** BEFORE writing implementation code. Evals go beyond unit tests — they verify that the agent/system can handle the feature's intent, not just its mechanics.
|
|
395
|
+
|
|
396
|
+
### Two Eval Types
|
|
397
|
+
|
|
398
|
+
| Type | Purpose | Pass Criteria | When |
|
|
399
|
+
|------|---------|---------------|------|
|
|
400
|
+
| **Capability eval** | Can the system do this new thing? | pass@k: ≥1 success in k attempts (k=3-5) | Before implementation |
|
|
401
|
+
| **Regression eval** | Did we break existing behavior? | pass^k: ALL k attempts must pass | After implementation |
|
|
402
|
+
|
|
403
|
+
**pass@k** (capability): At least 1 of k runs succeeds. Used for new features where some variance is acceptable. Threshold: ≥90% pass@3 for standard features, ≥95% pass@5 for critical paths.
|
|
404
|
+
|
|
405
|
+
**pass^k** (regression): ALL k runs must pass. Used for existing behavior that must never break. If ANY run fails, it's a regression. Threshold: 100% pass^3.
|
|
406
|
+
|
|
407
|
+
### Eval File Format
|
|
408
|
+
|
|
409
|
+
Store evals in `.rune/evals/<feature>.md`:
|
|
410
|
+
|
|
411
|
+
```markdown
|
|
412
|
+
# Eval: <feature name>
|
|
413
|
+
|
|
414
|
+
## Capability Evals (pass@k)
|
|
415
|
+
| ID | Description | k | Threshold | Status |
|
|
416
|
+
|----|-------------|---|-----------|--------|
|
|
417
|
+
| CAP-1 | [what the system should be able to do] | 3 | 90% | pending |
|
|
418
|
+
|
|
419
|
+
## Regression Evals (pass^k)
|
|
420
|
+
| ID | Description | k | Status |
|
|
421
|
+
|----|-------------|---|--------|
|
|
422
|
+
| REG-1 | [existing behavior that must not break] | 3 | pending |
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
### Anti-Pattern: Eval Overfitting
|
|
426
|
+
|
|
427
|
+
Do NOT overfit evals to specific prompts or known examples. Evals should test the **capability**, not the **exact input**.
|
|
428
|
+
|
|
429
|
+
- BAD: `"When user says 'hello', respond with 'Hi there!'"` — tests exact string match
|
|
430
|
+
- GOOD: `"When user greets, respond with a greeting"` — tests capability
|
|
431
|
+
|
|
432
|
+
### Integration with TDD
|
|
433
|
+
|
|
434
|
+
1. Write eval definitions (capability + regression) → `.rune/evals/<feature>.md`
|
|
435
|
+
2. Write unit/integration tests (RED phase) → test files
|
|
436
|
+
3. Implement feature (GREEN phase) → source files
|
|
437
|
+
4. Run evals to verify capability achieved + no regressions
|
|
438
|
+
5. Preflight checks eval results as part of quality gate
|
|
439
|
+
|
|
440
|
+
> Source: affaan-m/everything-claude-code (91.9k★) — eval-driven development with pass@k/pass^k distinction.
|
|
441
|
+
|
|
276
442
|
## Red Flags — STOP and Start Over
|
|
277
443
|
|
|
278
444
|
If you catch yourself with ANY of these, delete implementation code and restart with tests:
|
|
@@ -353,11 +519,34 @@ Partial mock missing fields that downstream code consumes. Your test passes beca
|
|
|
353
519
|
If mock setup is 30 lines and the actual test assertion is 3 lines, the test is testing infrastructure, not behavior. This is a code smell that indicates wrong abstraction level.
|
|
354
520
|
**Gate**: "Is my mock setup longer than my test logic?" → If yes: test at a higher level (integration) or extract mock factories.
|
|
355
521
|
|
|
522
|
+
### Anti-Pattern 6: Test Slop (Framework-Behavior Tests)
|
|
523
|
+
Tests that verify the framework works rather than YOUR code works. If the test would still pass with an empty component/function, it's testing infrastructure.
|
|
524
|
+
**Gate**: "Would this test pass if I deleted my business logic?" → If yes: STOP. Rewrite to test behavior that YOUR code introduces.
|
|
525
|
+
|
|
526
|
+
Examples of test slop:
|
|
527
|
+
- "renders without crashing" (tests that React works, not your component)
|
|
528
|
+
- "route responds with 200" without checking response body (tests Express, not your handler)
|
|
529
|
+
- Asserting a mock was called N times without checking the RESULT of those calls
|
|
530
|
+
- Type existence tests (`typeof result === 'object'`) when you should test the actual value
|
|
531
|
+
|
|
532
|
+
> Source: affaan-m/everything-claude-code (91.9k★) — de-sloppify pattern applied to test writing.
|
|
533
|
+
|
|
356
534
|
**Red flags — any of these means STOP and rethink:**
|
|
357
535
|
- Mock setup longer than test logic
|
|
358
536
|
- `*-mock` test IDs in assertions
|
|
359
537
|
- Methods only called in test files
|
|
360
538
|
- Can't explain in one sentence why a mock is needed
|
|
539
|
+
- Test would pass with empty implementation (test slop)
|
|
540
|
+
|
|
541
|
+
## Returns
|
|
542
|
+
|
|
543
|
+
| Artifact | Format | Location |
|
|
544
|
+
|----------|--------|----------|
|
|
545
|
+
| Test files | Source files | Co-located or `__tests__/` per project convention |
|
|
546
|
+
| Test plan + results | Markdown | `TEST.md` in test directory (non-trivial features only) |
|
|
547
|
+
| Eval scenarios | Markdown | `skills/<name>/evals.md` (for skill behavior testing) |
|
|
548
|
+
| Coverage report | Inline stdout | Shown in Test Report |
|
|
549
|
+
| Test Report | Markdown (inline) | Emitted to calling skill (cook, fix, review) |
|
|
361
550
|
|
|
362
551
|
## Sharp Edges
|
|
363
552
|
|
|
@@ -375,6 +564,18 @@ Known failure modes for this skill. Check these before declaring done.
|
|
|
375
564
|
| Modifying source files to make tests work | HIGH | Role boundary: test writes test files ONLY — source changes go to rune:fix |
|
|
376
565
|
| Test-only methods leaking into production code | MEDIUM | Anti-Pattern 2 gate: if method only called by tests → move to test utilities |
|
|
377
566
|
|
|
567
|
+
## Self-Validation
|
|
568
|
+
|
|
569
|
+
```
|
|
570
|
+
SELF-VALIDATION (run before emitting Test Report):
|
|
571
|
+
- [ ] Every test file has at least one assertion — no empty test bodies
|
|
572
|
+
- [ ] RED phase output shows actual failures (not "0 tests") — tests were real, not stubs
|
|
573
|
+
- [ ] No test modifies source code — test files only, source changes belong to fix
|
|
574
|
+
- [ ] Test names describe behavior, not implementation ("should reject expired token" not "test function X")
|
|
575
|
+
- [ ] No mocks of the thing being tested — only mock external dependencies
|
|
576
|
+
- [ ] If BA requirements exist (REQ-xxx), every requirement has at least one test — check plan's Traceability Matrix
|
|
577
|
+
```
|
|
578
|
+
|
|
378
579
|
## Done When
|
|
379
580
|
|
|
380
581
|
- Test framework detected from project config files
|
|
@@ -383,7 +584,10 @@ Known failure modes for this skill. Check these before declaring done.
|
|
|
383
584
|
- After implementation: all tests PASS (GREEN phase — actual pass output shown)
|
|
384
585
|
- Coverage ≥80% verified via verification
|
|
385
586
|
- Test Report emitted with framework, test count, RED/GREEN status, and coverage
|
|
587
|
+
- Self-Validation: all checks passed
|
|
386
588
|
|
|
387
589
|
## Cost Profile
|
|
388
590
|
|
|
389
591
|
~$0.03-0.08 per invocation. Sonnet for writing tests, Bash for running them. Frequent invocation in TDD workflow.
|
|
592
|
+
|
|
593
|
+
**Scope guardrail**: Do not modify source or implementation files to make tests pass unless explicitly delegated by the parent agent.
|