opencode-swarm-plugin 0.37.0 → 0.39.1

package/CHANGELOG.md

@@ -1,5 +1,263 @@
 # opencode-swarm-plugin
 
+## 0.39.1
+
+### Patch Changes
+
+- [`19a6557`](https://github.com/joelhooks/swarm-tools/commit/19a6557cee9878858e7f61e2aba86b37a3ec10ad) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🐝 Eval Quality Gates: Signal Over Noise
+
+  The eval system now filters coordinator sessions to focus on high-quality data.
+
+  **Problem:** 67 of 82 captured sessions had <3 events - noise from aborted runs, test pokes, and incomplete swarms. This diluted eval scores and made metrics unreliable.
+
+  **Solution:** Quality filters applied BEFORE sampling:
+
+  | Filter               | Default | Purpose                           |
+  | -------------------- | ------- | --------------------------------- |
+  | `minEvents`          | 3       | Skip incomplete/aborted sessions  |
+  | `requireWorkerSpawn` | true    | Ensure coordinator delegated work |
+  | `requireReview`      | true    | Ensure full swarm lifecycle       |
+
+  **Impact:**
+
+  - Filters 93 noisy sessions automatically
+  - Overall eval score: 63% → 71% (true signal, not diluted)
+  - Coordinator discipline: 47% → 57% (accurate measurement)
+
+  **Usage:**
+
+  ```typescript
+  // Default: high-quality sessions only
+  const sessions = await loadCapturedSessions();
+
+  // Override for specific analysis
+  const allSessions = await loadCapturedSessions({
+    minEvents: 1,
+    requireWorkerSpawn: false,
+    requireReview: false,
+  });
+  ```
+
+  Includes 7 unit tests covering filter logic and edge cases.
+
+## 0.39.0
+
+### Minor Changes
+
+- [`aa12943`](https://github.com/joelhooks/swarm-tools/commit/aa12943f3edc8d5e23878b22f44073e4c71367c5) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🐝 Eval-Driven Development: The System That Scores Itself
+
+  > "What gets measured gets managed." — Peter Drucker
+  > "What gets scored gets improved." — The Swarm
+
+  The plugin now evaluates its own output quality through a progressive gate system. Every compaction prompt gets scored, tracked, and learned from. Regressions become impossible to ignore.
+
+  ### The Pipeline
+
+  ```
+  CAPTURE → SCORE → STORE → GATE → LEARN → IMPROVE
+     ↑                                      ↓
+     └──────────────────────────────────────┘
+  ```
+
+  ### What's New
+
+  **Event Capture** (5 integration points)
+
+  - `detection_triggered` - When compaction is detected
+  - `prompt_generated` - Full LLM prompt captured
+  - `context_injected` - Final content before injection
+  - All events stored to `~/.config/swarm-tools/sessions/{session_id}.jsonl`
+
+  **5 Compaction Prompt Scorers**
+
+  - `epicIdSpecificity` - Real IDs, not placeholders (20%)
+  - `actionability` - Specific tool calls with values (20%)
+  - `coordinatorIdentity` - ASCII header + mandates (25%)
+  - `forbiddenToolsPresent` - Lists what NOT to do (15%)
+  - `postCompactionDiscipline` - First tool is correct (20%)
+
+  **Progressive Gates**
+  | Phase | Threshold | Behavior |
+  |-------|-----------|----------|
+  | Bootstrap | N/A | Always pass, building baseline |
+  | Stabilization | 0.6 | Warn but pass |
+  | Production | 0.7 | Fail CI on regression |
+
+  **CLI Commands**
+
+  ```bash
+  swarm eval status          # Current phase, thresholds, scores
+  swarm eval history         # Trends with sparklines ▁▂▃▄▅▆▇█
+  swarm eval run [--ci]      # Execute evals, gate check
+  ```
+
+  **CI Integration**
+
+  - Runs after tests pass
+  - Posts results as PR comment with emoji status
+  - Only fails in production phase with actual regression
+
+  **Learning Feedback Loop**
+
+  - Significant score drops auto-stored to semantic memory
+  - Future agents learn from past failures
+  - Pattern maturity tracking
+
+  ### Breaking Changes
+
+  None. All new functionality is additive.
+
+  ### Files Changed
+
+  - `src/eval-capture.ts` - Event capture with Zod schemas
+  - `src/eval-gates.ts` - Progressive gate logic
+  - `src/eval-history.ts` - Score tracking over time
+  - `src/eval-learning.ts` - Failure-to-learning extraction
+  - `src/compaction-prompt-scoring.ts` - 5 pure scoring functions
+  - `evals/compaction-prompt.eval.ts` - Evalite integration
+  - `bin/swarm.ts` - CLI commands
+  - `.github/workflows/ci.yml` - CI integration
+
+  ### Test Coverage
+
+  - 422 new tests for eval-capture
+  - 48 CLI tests
+  - 7 integration tests for capture wiring
+  - All existing tests still passing
+
+### Patch Changes
+
+- Updated dependencies [[`aa12943`](https://github.com/joelhooks/swarm-tools/commit/aa12943f3edc8d5e23878b22f44073e4c71367c5)]:
+  - swarm-mail@1.5.2
+
+## 0.38.0
+
+### Minor Changes
+
+- [`41a1965`](https://github.com/joelhooks/swarm-tools/commit/41a19657b252eb1c7a7dc82bc59ab13589e8758f) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🐝 Coordinators Now Delegate Research to Workers
+
+  Coordinators finally know their place. They orchestrate, they don't fetch.
+
+  **The Problem:**
+  Coordinators were calling `repo-crawl_file`, `webfetch`, `context7_*` directly, burning expensive Sonnet context on raw file contents instead of spawning researcher workers.
+
+  **The Fix:**
+
+  ### Forbidden Tools Section
+
+  COORDINATOR_PROMPT now explicitly lists tools coordinators must NEVER call:
+
+  - `repo-crawl_*`, `repo-autopsy_*` - repository fetching
+  - `webfetch`, `fetch_fetch` - web fetching
+  - `context7_*` - library documentation
+  - `pdf-brain_search`, `pdf-brain_read` - knowledge base
+
+  ### Phase 1.5: Research Phase
+
+  New workflow phase between Initialize and Knowledge Gathering:
+
+  ```
+  swarm_spawn_researcher(
+    research_id="research-nextjs-cache",
+    tech_stack=["Next.js 16 Cache Components"],
+    project_path="/path/to/project"
+  )
+  ```
+
+  ### Strong Coordinator Identity Post-Compaction
+
+  When context compacts, the resuming agent now sees:
+
+  ```
+  ┌─────────────────────────────────────────────────────────────┐
+  │             🐝  YOU ARE THE COORDINATOR  🐝                 │
+  │             NOT A WORKER. NOT AN IMPLEMENTER.               │
+  │                  YOU ORCHESTRATE.                           │
+  └─────────────────────────────────────────────────────────────┘
+  ```
+
+  ### runResearchPhase Returns Spawn Instructions
+
+  ```typescript
+  const result = await runResearchPhase(task, projectPath);
+  // result.spawn_instructions = [
+  //   { research_id, tech, prompt, subagent_type: "swarm/researcher" }
+  // ]
+  ```
+
+  **32+ new tests, all 425 passing.**
+
+- [`b06f69b`](https://github.com/joelhooks/swarm-tools/commit/b06f69bc3db099c14f712585d88b42c801123d01) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🔬 Eval Capture Pipeline: Complete
+
+  > "The purpose of computing is insight, not numbers." — Richard Hamming
+
+  Wire all eval-capture functions into the swarm execution path, enabling ground-truth collection from real swarm executions.
+
+  **What changed:**
+
+  | Function                  | Wired Into                     | Purpose                            |
+  | ------------------------- | ------------------------------ | ---------------------------------- |
+  | `captureDecomposition()`  | `swarm_validate_decomposition` | Records task → subtasks mapping    |
+  | `captureSubtaskOutcome()` | `swarm_complete`               | Records per-subtask execution data |
+  | `finalizeEvalRecord()`    | `swarm_record_outcome`         | Computes aggregate metrics         |
+
+  **New npm scripts:**
+
+  ```bash
+  bun run eval:run           # Run all evals
+  bun run eval:decomposition # Decomposition quality
+  bun run eval:coordinator   # Coordinator discipline
+  ```
+
+  **Data flow:**
+
+  ```
+  swarm_decompose → captureDecomposition → .opencode/eval-data.jsonl
+         ↓
+  swarm_complete → captureSubtaskOutcome → updates record with outcomes
+         ↓
+  swarm_record_outcome → finalizeEvalRecord → computes scope_accuracy, time_balance
+         ↓
+  evalite → reads JSONL → scores decomposition quality
+  ```
+
+  **Why it matters:**
+
+  - Enables data-driven decomposition strategy selection
+  - Tracks which strategies work for which task types
+  - Provides ground truth for Evalite evals
+  - Foundation for learning from swarm outcomes
+
+  **Key discovery:** New cell ID format doesn't follow `epicId.subtaskNum` pattern. Must use `cell.parent_id` to get epic ID for subtasks.
+
+### Patch Changes
+
+- [`56e5d4c`](https://github.com/joelhooks/swarm-tools/commit/56e5d4c5ac96ddd2184d12c63e163bb9c291fb69) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🔬 Eval Capture Pipeline: Phase 1
+
+  > "The first step toward wisdom is getting things right. The second step is getting them wrong in interesting ways." — Marvin Minsky
+
+  Wire `captureDecomposition()` into `swarm_validate_decomposition` to record decomposition inputs/outputs for evaluation.
+
+  **What changed:**
+
+  - `swarm_validate_decomposition` now calls `captureDecomposition()` after successful validation
+  - Captures: epicId, projectPath, task, context, strategy, epicTitle, subtasks
+  - Data persisted to `.opencode/eval-data.jsonl` for Evalite consumption
+
+  **Why it matters:**
+
+  - Enables ground-truth collection from real swarm executions
+  - Foundation for decomposition quality evals
+  - Tracks what strategies work for which task types
+
+  **Tests added:**
+
+  - Verifies `captureDecomposition` called with correct params on success
+  - Verifies NOT called on validation failure
+  - Handles optional context/description fields
+
+  **Next:** Wire `captureSubtaskOutcome()` and `finalizeEvalRecord()` to complete the pipeline.
+
 ## 0.37.0
 
 ### Minor Changes

package/README.md

@@ -231,6 +231,56 @@ bun test
 bun run typecheck
 ```
 
+### Evaluation Pipeline
+
+Test decomposition quality and coordinator discipline with **Evalite** (TypeScript-native eval framework):
+
+```bash
+# Run all evals
+bun run eval:run
+
+# Run specific suites
+bun run eval:decomposition    # Task decomposition quality
+bun run eval:coordinator      # Coordinator protocol compliance
+bun run eval:compaction       # Compaction prompt quality
+
+# Check eval status (progressive gates)
+swarm eval status [eval-name]
+
+# View history with trends
+swarm eval history
+```
+
+**Progressive Gates:**
+
+```
+Phase             Runs    Gate Behavior
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Bootstrap         <10     ✅ Always pass (collect data)
+Stabilization     10-50   ⚠️  Warn on >10% regression
+Production        >50     ❌ Fail on >5% regression
+```
+
+**What gets evaluated:**
+
+| Eval Suite            | Measures                                                      | Data Source                                      |
+| --------------------- | ------------------------------------------------------------- | ------------------------------------------------ |
+| `swarm-decomposition` | Subtask independence, complexity balance, coverage, clarity   | Fixtures + `.opencode/eval-data.jsonl`           |
+| `coordinator-session` | Violation count, spawn efficiency, review thoroughness        | `~/.config/swarm-tools/sessions/*.jsonl`         |
+| `compaction-prompt`   | ID specificity, actionability, identity, forbidden tools      | Session compaction events                        |
+
+**Learning Feedback Loop:**
+
+When eval scores drop >15% from baseline, failure context is automatically stored to semantic memory. Future prompts query these learnings for context.
+
+**Data capture locations:**
+- Decomposition inputs/outputs: `.opencode/eval-data.jsonl`
+- Eval history: `.opencode/eval-history.jsonl`
+- Coordinator sessions: `~/.config/swarm-tools/sessions/*.jsonl`
+- Subtask outcomes: swarm-mail database
+
+See **[evals/README.md](./evals/README.md)** for full architecture, scorer details, CI integration, and how to write new evals.
+
 ---
 
 ## CLI