opencode-swarm-plugin 0.38.0 → 0.39.1

package/.env

@@ -0,0 +1,2 @@
+NPM_1P_ITEM=yeu4tbknx5crxmudtu3pfg3eba
+AI_GATEWAY_API_KEY=vck_2w2KCfF5YskBaxnsIaOqnr87kAOIyL6HpPwtLCTWtn7DFyKXEP4IJsKA

package/.hive/eval-results.json

@@ -0,0 +1,26 @@
+{
+  "compaction-prompt": {
+    "passed": true,
+    "phase": "bootstrap",
+    "message": "Bootstrap phase (1/10 runs) - collecting data",
+    "currentScore": 0.85
+  },
+  "coordinator-behavior": {
+    "passed": true,
+    "phase": "bootstrap",
+    "message": "Bootstrap phase (1/10 runs) - collecting data",
+    "currentScore": 0.85
+  },
+  "coordinator-session": {
+    "passed": true,
+    "phase": "bootstrap",
+    "message": "Bootstrap phase (1/10 runs) - collecting data",
+    "currentScore": 0.85
+  },
+  "swarm-decomposition": {
+    "passed": true,
+    "phase": "bootstrap",
+    "message": "Bootstrap phase (1/10 runs) - collecting data",
+    "currentScore": 0.85
+  }
+}

package/.hive/issues.jsonl

@@ -26,3 +26,14 @@
 {"id":"opencode-swarm-plugin--ys7z8-mjkn5xp1blq","title":"Wire captureSubtaskOutcome() into swarm_complete","status":"closed","priority":0,"issue_type":"task","created_at":"2025-12-24T23:25:06.133Z","updated_at":"2025-12-24T23:52:01.496Z","closed_at":"2025-12-24T23:52:01.496Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkn5xocowf","dependencies":[],"labels":[],"comments":[]}
 {"id":"opencode-swarm-plugin--ys7z8-mjkn5xp41f2","title":"Wire finalizeEvalRecord() into swarm_record_outcome","status":"closed","priority":0,"issue_type":"task","created_at":"2025-12-24T23:25:06.136Z","updated_at":"2025-12-24T23:52:02.719Z","closed_at":"2025-12-24T23:52:02.719Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkn5xocowf","dependencies":[],"labels":[],"comments":[]}
 {"id":"opencode-swarm-plugin--ys7z8-mjkn5xp793w","title":"Add eval scripts to package.json and update README","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-24T23:25:06.139Z","updated_at":"2025-12-24T23:52:04.385Z","closed_at":"2025-12-24T23:52:04.385Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkn5xocowf","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkvvysb1bk","title":"Fix Eval Pipeline: Database Table + Scorer API","description":"Two fixes: 1) Ensure eval_records table is created when swarm-mail database initializes, 2) Fix composite scorer API usage in evalite tests","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-12-25T03:29:17.531Z","updated_at":"2025-12-25T03:42:14.497Z","closed_at":"2025-12-25T03:42:14.497Z","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkvvysjyrv","title":"Verify eval_records table creation in swarm-mail","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-25T03:29:17.539Z","updated_at":"2025-12-25T03:40:16.396Z","closed_at":"2025-12-25T03:40:16.396Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkvvysb1bk","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkvvysl8ye","title":"Fix composite scorer API in coordinator-discipline.ts","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-25T03:29:17.541Z","updated_at":"2025-12-25T03:40:17.922Z","closed_at":"2025-12-25T03:40:17.922Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkvvysb1bk","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkvvysnzae","title":"Fix composite scorer API in compaction-scorers.ts","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-25T03:29:17.543Z","updated_at":"2025-12-25T03:40:19.200Z","closed_at":"2025-12-25T03:40:19.200Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkvvysb1bk","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkvvysrwgk","title":"Fix composite scorer API in coordinator-behavior.eval.ts","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-25T03:29:17.547Z","updated_at":"2025-12-25T03:42:04.249Z","closed_at":"2025-12-25T03:42:04.249Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkvvysb1bk","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkwt9rqf2s","title":"Fix session ID propagation in eval capture","description":"Session IDs not flowing to captureCoordinatorEvent - 82% of events orphaned in unknown.jsonl. Root cause: swarm tools use process.env.OPENCODE_SESSION_ID which is not set, instead of ctx.sessionID which IS available.","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-12-25T03:55:11.414Z","updated_at":"2025-12-25T04:14:23.283Z","closed_at":"2025-12-25T04:14:23.283Z","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkwt9rzlw3","title":"Add sessionId parameter to captureCoordinatorEvent and update call sites","status":"closed","priority":0,"issue_type":"task","created_at":"2025-12-25T03:55:11.423Z","updated_at":"2025-12-25T04:05:28.792Z","closed_at":"2025-12-25T04:05:28.792Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkwt9rqf2s","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkwt9s2boa","title":"Create migration script to re-attribute unknown.jsonl events to proper sessions","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-25T03:55:11.426Z","updated_at":"2025-12-25T04:05:29.764Z","closed_at":"2025-12-25T04:05:29.764Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkwt9rqf2s","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkwt9s6xoa","title":"Run migration and verify data integrity","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-25T03:55:11.430Z","updated_at":"2025-12-25T04:14:16.676Z","closed_at":"2025-12-25T04:14:16.676Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkwt9rqf2s","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkyhrqmecc","title":"Add quality gate filters to eval session loader","description":"Filter eval sessions by quality signals: minEvents >= 3, hasWorkerSpawn, hasReviewCompleted. Currently 67 of 82 sessions are noise (<3 events). Quality gate will keep ~15 high-signal sessions.","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-12-25T04:42:14.062Z","updated_at":"2025-12-25T04:49:40.809Z","closed_at":"2025-12-25T04:49:40.809Z","dependencies":[],"labels":[],"comments":[]}
+{"id":"opencode-swarm-plugin--ys7z8-mjkyhrr2qm7","title":"Add quality filter options to loadCapturedSessions with TDD","status":"closed","priority":0,"issue_type":"task","created_at":"2025-12-25T04:42:14.078Z","updated_at":"2025-12-25T04:49:39.904Z","closed_at":"2025-12-25T04:49:39.904Z","parent_id":"opencode-swarm-plugin--ys7z8-mjkyhrqmecc","dependencies":[],"labels":[],"comments":[]}

package/.hive/memories.jsonl

@@ -546,4 +546,26 @@
 {"id":"mem_mjkifog0_kyrf1i8","information":"Findable test memory with unique keyword xyztest123","created_at":"2025-12-24T21:12:42.624Z"}
 {"id":"mem_mjkifrmb_cfzpsbl","information":"Test memory for adapter wiring verification","created_at":"2025-12-24T21:12:46.739Z","tags":"test,memory"}
 {"id":"mem_mjkifrp8_6p3hyc0","information":"OAuth refresh tokens need 5min buffer before expiry","created_at":"2025-12-24T21:12:46.844Z","metadata":"{\"raw\":\"auth,tokens,oauth\"}","tags":"auth,integration-test"}
-{"id":"mem_mjkifrty_n2obcci","information":"Smoke test verified full tool adapter wiring works end-to-end","created_at":"2025-12-24T21:12:47.014Z","tags":"test,verification"}
+{"id":"mem_mjkifrty_n2obcci","information":"Smoke test verified full tool adapter wiring works end-to-end","created_at":"2025-12-24T21:12:47.014Z","tags":"test,verification"}
+{"id":"mem_mjkvzysv_sc2t9vz","information":"Test memory for tools integration","created_at":"2025-12-25T03:32:24.175Z","tags":"test"}
+{"id":"mem_mjkvzzi6_1p6e6a9","information":"Findable test memory with unique keyword xyztest123","created_at":"2025-12-25T03:32:25.086Z"}
+{"id":"mem_mjkw8n77_qjdsp7f","information":"Test memory for tools integration","created_at":"2025-12-25T03:39:09.043Z","tags":"test"}
+{"id":"mem_mjkw8njx_i8h8cyh","information":"Findable test memory with unique keyword xyztest123","created_at":"2025-12-25T03:39:09.501Z"}
+{"id":"mem_mjkw8rmk_f6hitx1","information":"Test memory for adapter wiring verification","created_at":"2025-12-25T03:39:14.780Z","tags":"test,memory"}
+{"id":"mem_mjkw8rpm_lje9arh","information":"OAuth refresh tokens need 5min buffer before expiry","created_at":"2025-12-25T03:39:14.890Z","metadata":"{\"raw\":\"auth,tokens,oauth\"}","tags":"auth,integration-test"}
+{"id":"mem_mjkw8rtm_adjnpml","information":"Smoke test verified full tool adapter wiring works end-to-end","created_at":"2025-12-25T03:39:15.034Z","tags":"test,verification"}
+{"id":"mem_mjkwmbkm_33rhosw","information":"Test memory for tools integration","created_at":"2025-12-25T03:49:47.158Z","tags":"test"}
+{"id":"mem_mjkwmc55_9oi3pyz","information":"Findable test memory with unique keyword xyztest123","created_at":"2025-12-25T03:49:47.897Z"}
+{"id":"mem_mjkwmg5h_07q5cqq","information":"Test memory for adapter wiring verification","created_at":"2025-12-25T03:49:53.093Z","tags":"test,memory"}
+{"id":"mem_mjkwmg9a_evvx6t6","information":"OAuth refresh tokens need 5min buffer before expiry","created_at":"2025-12-25T03:49:53.230Z","metadata":"{\"raw\":\"auth,tokens,oauth\"}","tags":"auth,integration-test"}
+{"id":"mem_mjkwmge4_2pkurm7","information":"Smoke test verified full tool adapter wiring works end-to-end","created_at":"2025-12-25T03:49:53.404Z","tags":"test,verification"}
+{"id":"mem_mjkx05sw_izlcsfs","information":"Test memory for tools integration","created_at":"2025-12-25T04:00:32.864Z","tags":"test"}
+{"id":"mem_mjkx067y_b9hn5qi","information":"Findable test memory with unique keyword xyztest123","created_at":"2025-12-25T04:00:33.406Z"}
+{"id":"mem_mjkx09hf_ygskd44","information":"Test memory for adapter wiring verification","created_at":"2025-12-25T04:00:37.635Z","tags":"test,memory"}
+{"id":"mem_mjkx09lg_hwd8wid","information":"OAuth refresh tokens need 5min buffer before expiry","created_at":"2025-12-25T04:00:37.780Z","metadata":"{\"raw\":\"auth,tokens,oauth\"}","tags":"auth,integration-test"}
+{"id":"mem_mjkx09p9_lc3whf6","information":"Smoke test verified full tool adapter wiring works end-to-end","created_at":"2025-12-25T04:00:37.917Z","tags":"test,verification"}
+{"id":"mem_mjkxgljy_xvyprn1","information":"Test memory for tools integration","created_at":"2025-12-25T04:13:19.774Z","tags":"test"}
+{"id":"mem_mjkxglqg_5ojok3n","information":"Findable test memory with unique keyword xyztest123","created_at":"2025-12-25T04:13:20.008Z"}
+{"id":"mem_mjkxgogk_48pml1f","information":"Test memory for adapter wiring verification","created_at":"2025-12-25T04:13:23.540Z","tags":"test,memory"}
+{"id":"mem_mjkxgomk_mm0hvqg","information":"OAuth refresh tokens need 5min buffer before expiry","created_at":"2025-12-25T04:13:23.756Z","metadata":"{\"raw\":\"auth,tokens,oauth\"}","tags":"auth,integration-test"}
+{"id":"mem_mjkxgopz_mqvrw0z","information":"Smoke test verified full tool adapter wiring works end-to-end","created_at":"2025-12-25T04:13:23.879Z","tags":"test,verification"}

package/.opencode/eval-history.jsonl

@@ -0,0 +1,12 @@
+{"timestamp":"2025-12-25T04:28:42.041Z","eval_name":"compaction-prompt","score":0.85,"run_count":1}
+{"timestamp":"2025-12-25T04:28:42.041Z","eval_name":"coordinator-behavior","score":0.85,"run_count":1}
+{"timestamp":"2025-12-25T04:28:42.042Z","eval_name":"coordinator-session","score":0.85,"run_count":1}
+{"timestamp":"2025-12-25T04:28:42.042Z","eval_name":"swarm-decomposition","score":0.85,"run_count":1}
+{"timestamp":"2025-12-25T04:28:52.405Z","eval_name":"compaction-prompt","score":0.85,"run_count":2}
+{"timestamp":"2025-12-25T04:28:52.405Z","eval_name":"coordinator-behavior","score":0.85,"run_count":2}
+{"timestamp":"2025-12-25T04:28:52.405Z","eval_name":"coordinator-session","score":0.85,"run_count":2}
+{"timestamp":"2025-12-25T04:28:52.405Z","eval_name":"swarm-decomposition","score":0.85,"run_count":2}
+{"timestamp":"2025-12-25T05:11:18.469Z","eval_name":"compaction-prompt","score":0.85,"run_count":3}
+{"timestamp":"2025-12-25T05:11:18.469Z","eval_name":"coordinator-behavior","score":0.85,"run_count":3}
+{"timestamp":"2025-12-25T05:11:18.469Z","eval_name":"coordinator-session","score":0.85,"run_count":3}
+{"timestamp":"2025-12-25T05:11:18.469Z","eval_name":"swarm-decomposition","score":0.85,"run_count":3}

package/CHANGELOG.md

@@ -1,5 +1,135 @@
 # opencode-swarm-plugin
 
+## 0.39.1
+
+### Patch Changes
+
+- [`19a6557`](https://github.com/joelhooks/swarm-tools/commit/19a6557cee9878858e7f61e2aba86b37a3ec10ad) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🐝 Eval Quality Gates: Signal Over Noise
+
+  The eval system now filters coordinator sessions to focus on high-quality data.
+
+  **Problem:** 67 of 82 captured sessions had <3 events - noise from aborted runs, test pokes, and incomplete swarms. This diluted eval scores and made metrics unreliable.
+
+  **Solution:** Quality filters applied BEFORE sampling:
+
+  | Filter               | Default | Purpose                           |
+  | -------------------- | ------- | --------------------------------- |
+  | `minEvents`          | 3       | Skip incomplete/aborted sessions  |
+  | `requireWorkerSpawn` | true    | Ensure coordinator delegated work |
+  | `requireReview`      | true    | Ensure full swarm lifecycle       |
+
+  **Impact:**
+
+  - Filters 93 noisy sessions automatically
+  - Overall eval score: 63% → 71% (true signal, not diluted)
+  - Coordinator discipline: 47% → 57% (accurate measurement)
+
+  **Usage:**
+
+  ```typescript
+  // Default: high-quality sessions only
+  const sessions = await loadCapturedSessions();
+
+  // Override for specific analysis
+  const allSessions = await loadCapturedSessions({
+    minEvents: 1,
+    requireWorkerSpawn: false,
+    requireReview: false,
+  });
+  ```
+
+  Includes 7 unit tests covering filter logic and edge cases.
+
+## 0.39.0
+
+### Minor Changes
+
+- [`aa12943`](https://github.com/joelhooks/swarm-tools/commit/aa12943f3edc8d5e23878b22f44073e4c71367c5) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🐝 Eval-Driven Development: The System That Scores Itself
+
+  > "What gets measured gets managed." — Peter Drucker
+  > "What gets scored gets improved." — The Swarm
+
+  The plugin now evaluates its own output quality through a progressive gate system. Every compaction prompt gets scored, tracked, and learned from. Regressions become impossible to ignore.
+
+  ### The Pipeline
+
+  ```
+  CAPTURE → SCORE → STORE → GATE → LEARN → IMPROVE
+     ↑                                      ↓
+     └──────────────────────────────────────┘
+  ```
+
+  ### What's New
+
+  **Event Capture** (5 integration points)
+
+  - `detection_triggered` - When compaction is detected
+  - `prompt_generated` - Full LLM prompt captured
+  - `context_injected` - Final content before injection
+  - All events stored to `~/.config/swarm-tools/sessions/{session_id}.jsonl`
+
+  **5 Compaction Prompt Scorers**
+
+  - `epicIdSpecificity` - Real IDs, not placeholders (20%)
+  - `actionability` - Specific tool calls with values (20%)
+  - `coordinatorIdentity` - ASCII header + mandates (25%)
+  - `forbiddenToolsPresent` - Lists what NOT to do (15%)
+  - `postCompactionDiscipline` - First tool is correct (20%)
+
+  **Progressive Gates**
+  | Phase | Threshold | Behavior |
+  |-------|-----------|----------|
+  | Bootstrap | N/A | Always pass, building baseline |
+  | Stabilization | 0.6 | Warn but pass |
+  | Production | 0.7 | Fail CI on regression |
+
+  **CLI Commands**
+
+  ```bash
+  swarm eval status          # Current phase, thresholds, scores
+  swarm eval history         # Trends with sparklines ▁▂▃▄▅▆▇█
+  swarm eval run [--ci]      # Execute evals, gate check
+  ```
+
+  **CI Integration**
+
+  - Runs after tests pass
+  - Posts results as PR comment with emoji status
+  - Only fails in production phase with actual regression
+
+  **Learning Feedback Loop**
+
+  - Significant score drops auto-stored to semantic memory
+  - Future agents learn from past failures
+  - Pattern maturity tracking
+
+  ### Breaking Changes
+
+  None. All new functionality is additive.
+
+  ### Files Changed
+
+  - `src/eval-capture.ts` - Event capture with Zod schemas
+  - `src/eval-gates.ts` - Progressive gate logic
+  - `src/eval-history.ts` - Score tracking over time
+  - `src/eval-learning.ts` - Failure-to-learning extraction
+  - `src/compaction-prompt-scoring.ts` - 5 pure scoring functions
+  - `evals/compaction-prompt.eval.ts` - Evalite integration
+  - `bin/swarm.ts` - CLI commands
+  - `.github/workflows/ci.yml` - CI integration
+
+  ### Test Coverage
+
+  - 422 new tests for eval-capture
+  - 48 CLI tests
+  - 7 integration tests for capture wiring
+  - All existing tests still passing
+
+### Patch Changes
+
+- Updated dependencies [[`aa12943`](https://github.com/joelhooks/swarm-tools/commit/aa12943f3edc8d5e23878b22f44073e4c71367c5)]:
+  - swarm-mail@1.5.2
+
 ## 0.38.0
 
 ### Minor Changes

package/README.md

@@ -242,27 +242,44 @@ bun run eval:run
 # Run specific suites
 bun run eval:decomposition    # Task decomposition quality
 bun run eval:coordinator      # Coordinator protocol compliance
+bun run eval:compaction       # Compaction prompt quality
+
+# Check eval status (progressive gates)
+swarm eval status [eval-name]
+
+# View history with trends
+swarm eval history
+```
+
+**Progressive Gates:**
+
+```
+Phase             Runs    Gate Behavior
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Bootstrap         <10     ✅ Always pass (collect data)
+Stabilization     10-50   ⚠️  Warn on >10% regression
+Production        >50     ❌ Fail on >5% regression
 ```
 
 **What gets evaluated:**
 
-| Eval Suite | Measures | Data Source |
-|------------|----------|-------------|
-| `swarm-decomposition` | Subtask independence, complexity balance, coverage, clarity | Fixtures + captured real decompositions |
-| `coordinator-session` | Violation count, spawn efficiency, review thoroughness | Real sessions from `~/.config/swarm-tools/sessions/` |
+| Eval Suite            | Measures                                                      | Data Source                                      |
+| --------------------- | ------------------------------------------------------------- | ------------------------------------------------ |
+| `swarm-decomposition` | Subtask independence, complexity balance, coverage, clarity   | Fixtures + `.opencode/eval-data.jsonl`           |
+| `coordinator-session` | Violation count, spawn efficiency, review thoroughness        | `~/.config/swarm-tools/sessions/*.jsonl`         |
+| `compaction-prompt`   | ID specificity, actionability, identity, forbidden tools      | Session compaction events                        |
+
+**Learning Feedback Loop:**
+
+When eval scores drop >15% from baseline, failure context is automatically stored to semantic memory. Future prompts query these learnings for context.
 
 **Data capture locations:**
 - Decomposition inputs/outputs: `.opencode/eval-data.jsonl`
+- Eval history: `.opencode/eval-history.jsonl`
 - Coordinator sessions: `~/.config/swarm-tools/sessions/*.jsonl`
-- Subtask outcomes: swarm-mail database (used for pattern learning)
-
-**Custom scorers:**
-- Subtask independence (0-1): Files don't overlap between subtasks
-- Complexity balance (0-1): Subtasks have similar estimated complexity
-- Coverage completeness (0-1): Required files are covered
-- Instruction clarity (0-1): Descriptions are specific and actionable
+- Subtask outcomes: swarm-mail database
 
-See [evals/README.md](./evals/README.md) for scorer details and how to write new evals.
+See **[evals/README.md](./evals/README.md)** for full architecture, scorer details, CI integration, and how to write new evals.
 
 ---