opencode-swarm-plugin 0.39.1 → 0.42.0

package/.hive/analysis/eval-failure-analysis-2025-12-25.md

@@ -0,0 +1,331 @@
+# Eval Failure Analysis Report
+**Date:** 2025-12-25
+**Analyst:** BrightStar
+**Cell:** opencode-swarm-plugin--ys7z8-mjlk7jsl4tt
+**Epic:** opencode-swarm-plugin--ys7z8-mjlk7js9bt1
+
+## Executive Summary
+
+Two eval failures analyzed:
+- **example.eval.ts**: 0% score - structural bug in eval setup
+- **compaction-prompt.eval.ts**: 53% score - case sensitivity + missing forbidden tools
+
+Both are fixable with code changes. No test data quality issues.
+
+---
+
+## example.eval.ts - 0% Score
+
+### Status
+❌ **CRITICAL** - Complete failure (0%)
+
+### Root Cause
+**Eval structure mismatch** between data provider and task function.
+
+### Technical Details
+
+**File:** `evals/example.eval.ts`
+**Lines:** 14-30
+
+The eval has a fundamental flow error:
+
+```typescript
+// Line 14-26: data() provides BOTH input AND expected output
+data: async () => {
+  return [
+    {
+      input: "Test task",                    // ← String for task function
+      output: JSON.stringify({               // ← Expected output (ignored!)
+        epic: { title: "Test Epic", ... },
+        subtasks: [...]
+      }),
+    },
+  ];
+},
+
+// Line 28-30: task() does passthrough
+task: async (input) => {
+  return input;  // ← Returns "Test task" string, NOT the CellTree
+},
+
+// Line 31: Scorer expects CellTree JSON
+scorers: [subtaskIndependence],
+```
+
+**What happens:**
+1. Evalite passes `input` ("Test task") to task function
+2. Task returns "Test task" string unchanged
+3. Scorer `subtaskIndependence` receives "Test task"
+4. Scorer tries to parse as CellTree JSON → **FAILS**
+5. Score: 0%
+
+The `output` field in `data()` is ignored by Evalite - it's the `task()` return value that gets scored.
+
+### Impact
+- Example eval is useless for validation
+- False signal that scorer infrastructure is broken (it's not)
+- Wastes CI time
+
+### Proposed Fix
+
+**Option 1: Remove output from data (recommended)**
+```typescript
+data: async () => {
+  return [
+    {
+      input: {
+        epic: { title: "Test Epic", description: "Test" },
+        subtasks: [
+          { title: "Subtask 1", files: ["a.ts"], estimated_complexity: 1 },
+          { title: "Subtask 2", files: ["b.ts"], estimated_complexity: 1 },
+        ],
+      },
+    },
+  ];
+},
+
+task: async (input) => {
+  return JSON.stringify(input);  // Stringify the CellTree
+},
+```
+
+**Option 2: Fix task to use output**
+```typescript
+// Keep data() as-is, but fix task:
+task: async (input, context) => {
+  return context.expected.output;  // Use the output from data()
+},
+```
+
+Option 1 is cleaner - task functions should generate output, not just pass through.
+
+---
+
+## compaction-prompt.eval.ts - 53% Score
+
+### Status
+⚠️ **DEGRADED** - Below target (53% vs 100% historical)
+
+### Root Causes
+
+#### RC1: Case-Sensitive Forbidden Tool Patterns (15% weight)
+
+**File:** `src/compaction-prompt-scoring.ts`
+**Lines:** 213-218
+
+```typescript
+const forbiddenTools = [
+  /\bEdit\b/,     // ← Requires capital E
+  /\bWrite\b/,    // ← Requires capital W
+  /swarmmail_reserve/,
+  /git commit/,
+];
+```
+
+**File:** `evals/fixtures/compaction-prompt-cases.ts`
+**Lines:** 76-83 (perfect fixture)
+
+```
+- edit      // ← lowercase e
+- write     // ← lowercase w
+- bash (for file modifications)
+```
+
+**Evidence:**
+```javascript
+/\bEdit\b/.test("- Edit")  // ✅ true
+/\bEdit\b/.test("- edit")  // ❌ false (word boundary + case)
+```
+
+**Impact:**
+- Perfect fixture: 0/4 forbidden tools matched
+- Forbidden tools scorer: 0% (should be 75-100%)
+- Overall impact: 15% of total score lost
+
+#### RC2: Missing Forbidden Tools (15% weight)
+
+Scorer expects **4 tools**:
+1. Edit (or edit)
+2. Write (or write)
+3. swarmmail_reserve
+4. git commit
+
+Perfect fixture has **3 tools** (and case mismatch):
+1. edit ❌ (lowercase)
+2. write ❌ (lowercase)
+3. bash ❌ (not in scorer's list)
+
+Missing: swarmmail_reserve, git commit
+
+**Impact:**
+- Even if case fixed, still only 2/4 tools = 50% on this scorer
+- Weighted: 50% × 15% = 7.5% contribution (should be 15%)
+
+#### RC3: "bash" Not in Scorer's List
+
+Fixtures mention "bash (for file modifications)" as forbidden, but scorer doesn't check for it.
+This creates a 3-way mismatch:
+- Fixture lists: edit, write, bash
+- Scorer checks: Edit, Write, swarmmail_reserve, git commit
+- Overlap: 0 tools (due to case)
+
+### Score Breakdown - Perfect Fixture
+
+Expected (if 100%):
+```
+epicIdSpecificity:        20% × 1.0 = 20%
+actionability:            20% × 1.0 = 20%
+coordinatorIdentity:      25% × 1.0 = 25%
+forbiddenToolsPresent:    15% × 1.0 = 15%
+postCompactionDiscipline: 20% × 1.0 = 20%
+                                 ─────
+TOTAL:                              100%
+```
+
+Actual (current):
+```
+epicIdSpecificity:        20% × 1.0 = 20% ✅
+actionability:            20% × 1.0 = 20% ✅
+coordinatorIdentity:      25% × 1.0 = 25% ✅
+forbiddenToolsPresent:    15% × 0.0 =  0% ❌ (0/4 matched)
+postCompactionDiscipline: 20% × 1.0 = 20% ✅
+                                 ─────
+TOTAL:                               85%
+```
+
+Perfect fixture alone should score 85%, but overall eval is 53%.
+This means the 5 "bad" fixtures are pulling average down further (expected behavior).
+
+### Historical Context
+
+Semantic memory claims 100% score previously. Likely scenarios:
+1. **Never actually ran** - aspiration documented before implementation
+2. **Ran with different fixtures** - fixtures were updated after scorer was written
+3. **Scorer was case-insensitive before** - regression in recent commit aa12943
+
+Commit aa12943 (2025-12-24) added the eval infrastructure. This is brand new code.
+
+### Proposed Fixes
+
+#### Fix 1: Make Scorer Case-Insensitive (Recommended)
+
+**File:** `src/compaction-prompt-scoring.ts`
+**Lines:** 213-218
+
+```typescript
+const forbiddenTools = [
+  /\bedit\b/i,              // Case insensitive with 'i' flag
+  /\bwrite\b/i,             // Case insensitive
+  /\bbash\b/i,              // Add bash (was missing)
+  /swarmmail_reserve/i,     // Keep, add 'i' for safety
+  /git commit/i,            // Keep, add 'i' for safety
+];
+```
+
+**Rationale:**
+- Coordinators might capitalize differently in prompts
+- Real prompts won't always match exact case
+- More robust matching
+
+#### Fix 2: Update Fixtures to Match Scorer (Alternative)
+
+**File:** `evals/fixtures/compaction-prompt-cases.ts`
+**Lines:** 76-83 (and all other fixtures)
+
+```
+- Edit                           // Capital E
+- Write                          // Capital W
+- bash (for file modifications)  // Keep or remove
+- swarmmail_reserve              // ADD
+- git commit                     // ADD
+```
+
+**Rationale:**
+- Keeps scorer strict (may catch real case issues)
+- Makes fixtures comprehensive (all 5 tools)
+- More explicit about what's forbidden
+
+#### Fix 3: Hybrid (Best of Both)
+
+1. Make scorer case-insensitive (Fix 1)
+2. Update fixtures to include all 5 tools (Fix 2)
+3. Remove "bash" from fixtures if not in coordinator forbidden list
+
+```typescript
+// Scorer (5 tools, case-insensitive):
+const forbiddenTools = [
+  /\bedit\b/i,
+  /\bwrite\b/i,
+  /swarmmail_reserve/i,
+  /git\s+commit/i,
+  /\bread\b/i,  // Consider adding - coordinators shouldn't read, should check status
+];
+```
+
+```
+// Fixture:
+- Edit
+- Write
+- swarmmail_reserve (only workers reserve files)
+- git commit (workers commit their changes)
+```
+
+### Risk Assessment
+
+**If we fix this, will scores jump to 100%?**
+
+**Perfect fixture:** 85% → 100% (if all 4 tools matched)
+**Other fixtures:** Depends on their issues
+
+Looking at fixture expected values:
+- Fixture 0 (perfect): Should be 100%
+- Fixture 1 (placeholder): Should fail (expected)
+- Fixture 2 (generic): Should fail (expected)
+- Fixture 3 (weak identity): Should partially fail (expected)
+- Fixture 4 (missing forbidden): Should fail on forbidden tools only
+- Fixture 5 (wrong first tool): Should fail on discipline only
+
+Average across 6 fixtures: ~66% expected (not 100%)
+
+**So 53% → ~70-80%** is realistic after fixes (not 100%).
+
+To get higher scores, need to fix issues in bad fixtures too, but those are SUPPOSED to fail.
+The scorer is working correctly on those.
+
+---
+
+## Recommendations
+
+### Immediate Actions (P0)
+
+1. **Fix example.eval.ts structure** - 5 min fix, unblocks that eval
+2. **Make forbidden tools case-insensitive** - 5 min fix, +15-20% score boost
+3. **Add missing tools to fixtures** - 10 min, comprehensive coverage
+
+### Medium-term Actions (P1)
+
+4. **Verify 100% claim in semantic memory** - Check if historical data exists
+5. **Document scorer expectations** - Add comments to fixtures explaining weights
+6. **Add unit tests for scorers** - Test edge cases independently
+
+### Long-term Actions (P2)
+
+7. **Consider LLM-as-judge for semantic checks** - Case-insensitive by nature
+8. **Add visual diff in eval output** - Show what's missing from prompts
+9. **Create eval dashboard** - Track scores over time, detect regressions
+
+---
+
+## Conclusion
+
+Both evals have **code bugs, not test data issues**:
+- example.eval.ts: Structural bug (task/data mismatch)
+- compaction-prompt.eval.ts: Case sensitivity + incomplete tool list
+
+Fixes are straightforward and low-risk. After fixes, expect:
+- example.eval.ts: 0% → 100%
+- compaction-prompt.eval.ts: 53% → 70-80%
+
+The 100% historical score in semantic memory is likely aspirational - these evals are brand new (commit aa12943, Dec 24).
+
+**Ready to implement fixes or escalate for review?**

package/.hive/analysis/session-data-quality-audit.md

@@ -0,0 +1,320 @@
+# Session Data Quality Audit Report
+
+**Date:** 2025-12-25  
+**Cell:** opencode-swarm-plugin--ys7z8-mjlk7jspacf  
+**Agent:** WildDawn  
+
+## Executive Summary
+
+Investigation of why only 3 of 102 sessions (2.9%) pass the coordinator-session eval filter reveals:
+
+1. **Filter is working as designed** - correctly isolating high-quality complete coordinator sessions
+2. **Data quality is actually GOOD** - the 3 passing sessions are gold-standard examples
+3. **97% filtered out is EXPECTED** - most sessions are worker completions, not coordinator sessions
+4. **Filter may be too strict** for broad coordinator behavior analysis (needs tuning)
+
+## Data Breakdown
+
+### Total Sessions: 102
+
+| Category | Count | % | Description |
+|----------|-------|---|-------------|
+| **Single-event sessions** | 70 | 68.6% | Worker completions (subtask_success), isolated reviews |
+| **Multi-event incomplete** | 29 | 28.4% | Coordinator sessions that didn't complete full cycle |
+| **Passing sessions** | 3 | 2.9% | Complete coordinator cycles with spawn + review |
+
+### Single-Event Sessions (70 sessions - 68.6%)
+
+**Event Type Breakdown:**
+- `OUTCOME/subtask_success`: 56 (80.0%) - **Worker completions, not coordinator sessions**
+- `DECISION/review_completed`: 12 (17.1%) - Isolated review events
+- `DECISION/worker_spawned`: 2 (2.9%) - Isolated spawn events
+
+**Analysis:** These are **NOT coordinator sessions**. They're worker agents reporting completion or isolated coordinator actions captured in separate session files.
+
+### Multi-Event Failures (29 sessions - 28.4%)
+
+**Failure Breakdown:**
+- **No worker_spawned event**: 20 sessions
+  - Review-only sessions (3-22 events, all `review_completed`)
+  - Appears to be test data or session capture split across files
+- **Has worker_spawned but no review_completed**: 5 sessions
+  - Incomplete coordinator sessions (4-24 events)
+  - Coordinator spawned workers but reviews weren't captured (yet)
+- **Too few events (<3)**: 4 sessions
+  - Aborted early
+
+**Key Finding:** None of these 29 sessions have `decomposition_complete` events. This suggests:
+1. Session capture may not be recording decomposition events
+2. OR coordinator sessions span multiple session files
+3. OR these are partial captures from long-running coordinators
+
+### Passing Sessions (3 sessions - 2.9%)
+
+#### ses_4b86f0867ffeXKv95ktf31igfD
+- **Events:** 33
+- **Worker spawns:** 20
+- **Reviews completed:** 13
+- **Violations:** 0
+- **Duration:** 437 minutes (7.3 hours)
+- **Quality:** GOLD STANDARD
+
+#### ses_4ac0f508dffeEcwSQ6OSMWrmWF
+- **Events:** 21
+- **Worker spawns:** 17
+- **Reviews completed:** 4
+- **Duration:** 540 minutes (9.0 hours)
+- **Quality:** GOLD STANDARD
+
+#### ses_4ae8c2f66ffecyfyre7ZQ7y5LW
+- **Events:** 31
+- **Worker spawns:** 24
+- **Reviews completed:** 7
+- **Violations:** 0
+- **Duration:** 368 minutes (6.1 hours)
+- **Quality:** GOLD STANDARD
+
+**Analysis:** These are FULL multi-hour coordinator sessions with extensive worker coordination. They represent the ideal coordinator behavior the eval is designed to measure.
+
+## Current Filter Criteria
+
+```typescript
+{
+  minEvents: 3,              // Default
+  requireWorkerSpawn: true,  // Default
+  requireReview: true,       // Default
+}
+```
+
+### Filter Performance
+
+| Check | Impact |
+|-------|--------|
+| `minEvents >= 3` | Filters out 74 sessions (72.5%) |
+| `requireWorkerSpawn: true` | Filters out 20 additional sessions (19.6%) |
+| `requireReview: true` | Filters out 5 additional sessions (4.9%) |
+
+**Cascade effect:** Each filter compounds, resulting in 2.9% passing rate.
+
+## Root Cause Analysis
+
+### Is the Filter Too Strict?
+
+**YES and NO:**
+
+✅ **Working as designed:**
+- Correctly excludes worker-only sessions (80% of single-event data)
+- Correctly excludes incomplete coordinator sessions
+- Isolates high-quality complete coordinator cycles
+
+❌ **Too strict for real-world analysis:**
+- 2.9% passing rate means most coordinator behavior is invisible to the eval
+- Filter assumes coordinators ALWAYS complete full spawn+review cycles
+- Doesn't account for:
+  - Long-running multi-session coordinators
+  - Coordinators that spawn workers but reviews aren't captured yet
+  - Early-stage coordinator sessions (before first spawn)
+
+### Is the Data Quality Low?
+
+**NO.** The data quality is actually GOOD:
+
+- The 3 passing sessions are excellent gold-standard examples
+- They contain rich coordinator behavior (20-24 worker spawns, 4-13 reviews)
+- Zero violations in all 3 sessions
+- Multi-hour timelines showing sustained coordination
+
+The "low passing rate" is a **filter strictness issue**, not a data quality issue.
+
+### Why Only 3/102 Pass?
+
+**Theory 1: Session Capture Splits Long Coordinators**
+- The 3 passing sessions are 6-9 hour marathons
+- Most coordinator work may be happening in shorter bursts
+- Session files might be split by epic_id or time windows
+
+**Evidence:**
+- Some sessions have 20+ `review_completed` events with no `worker_spawned`
+- This suggests reviews from previous spawns in a different session file
+
+**Theory 2: Review Capture Is Incomplete**
+- 5 sessions have `worker_spawned` but no `review_completed`
+- Reviews may be captured in separate session files
+- OR review capture isn't working consistently
+
+**Theory 3: Most Coordinator Sessions Are Short**
+- Only 32/102 sessions (31.4%) have ANY `review_completed` event
+- Only 10/102 sessions (9.8%) have ANY `worker_spawned` event
+- This suggests most captured activity is worker completions, not coordinator cycles
+
+## Recommendations
+
+### 1. Make Filter Parameters Optional (IMMEDIATE)
+
+**Current default:**
+```typescript
+{
+  minEvents: 3,
+  requireWorkerSpawn: true,
+  requireReview: true,
+}
+```
+
+**Recommended default:**
+```typescript
+{
+  minEvents: 3,              // Keep - filters out noise
+  requireWorkerSpawn: false, // CHANGE - allow early-stage sessions
+  requireReview: false,      // CHANGE - allow incomplete sessions
+}
+```
+
+**Impact:** This would increase passing rate from 3 to ~28 sessions (from 2.9% to 27.5%).
+
+**Rationale:**
+- Captures more coordinator behavior (spawns without reviews)
+- Allows evaluation of early-stage coordination patterns
+- Still filters out single-event worker completions
+- Users can opt-in to stricter filters if needed
+
+### 2. Add Session Type Detection (ENHANCEMENT)
+
+Add a filter to exclude worker-only sessions automatically:
+
+```typescript
+function isCoordinatorSession(session: CoordinatorSession): boolean {
+  return session.events.some(e => 
+    e.event_type === "DECISION" && 
+    (e.decision_type === "decomposition_complete" || 
+     e.decision_type === "worker_spawned" ||
+     e.decision_type === "strategy_selected")
+  );
+}
+```
+
+**Impact:** Filters out 70+ worker-only sessions before applying other criteria.
+
+### 3. Investigate Session Capture Splitting (BUG FIX?)
+
+**Symptoms:**
+- Sessions with 22 `review_completed` events but no `worker_spawned`
+- Sessions with 24 `worker_spawned` events but no reviews
+- No `decomposition_complete` events in ANY session (including the 3 passing)
+
+**Hypothesis:** Long-running coordinator sessions may be split across multiple session files.
+
+**Action:** Investigate `eval-capture.ts` to understand:
+- How `session_id` is generated
+- Whether sessions are split by epic_id
+- Whether there's a session timeout that creates new files
+
+### 4. Add Filter Reporting to Data Loader (OBSERVABILITY)
+
+The data loader logs filtered-out count, but doesn't break down WHY sessions failed.
+
+**Enhancement:**
+```typescript
+console.log(`Filtered out ${filteredOutCount} sessions:`);
+console.log(`  - Too few events (<${minEvents}): ${stats.tooFewEvents}`);
+console.log(`  - No worker_spawned: ${stats.noWorkerSpawn}`);
+console.log(`  - No review_completed: ${stats.noReview}`);
+console.log(`  - Worker-only sessions: ${stats.workerOnly}`);
+```
+
+This helps users understand filter impact.
+
+### 5. Consider Separate Evals for Different Session Types
+
+Instead of one eval with strict filters, consider:
+
+**Eval 1: Full Coordinator Cycles** (current behavior)
+- Filters: `minEvents=3, requireWorkerSpawn=true, requireReview=true`
+- Focus: End-to-end coordinator discipline
+- Expected passing rate: ~3% (gold standard only)
+
+**Eval 2: Coordinator Spawning Behavior**
+- Filters: `minEvents=3, requireWorkerSpawn=true, requireReview=false`
+- Focus: How coordinators delegate work
+- Expected passing rate: ~10%
+
+**Eval 3: Coordinator Review Behavior**
+- Filters: `minEvents=3, requireWorkerSpawn=false, requireReview=true`
+- Focus: How coordinators review worker output
+- Expected passing rate: ~31%
+
+**Eval 4: All Coordinator Activity**
+- Filters: `minEvents=3, requireWorkerSpawn=false, requireReview=false, isCoordinatorSession=true`
+- Focus: Broad coordinator behavior patterns
+- Expected passing rate: ~27%
+
+## Conclusion
+
+The coordinator-session eval filter is **working as designed**. It successfully isolates high-quality complete coordinator sessions for evaluation.
+
+However, the **2.9% passing rate is too strict** for comprehensive coordinator behavior analysis. The filter should:
+
+1. **Default to more lenient settings** (requireWorkerSpawn=false, requireReview=false)
+2. **Allow users to opt-in** to stricter filters for gold-standard analysis
+3. **Automatically exclude worker-only sessions** via session type detection
+4. **Provide visibility** into why sessions are filtered out
+
+The data quality itself is GOOD. The 3 passing sessions are excellent examples of sustained multi-hour coordinator behavior with extensive worker coordination and zero violations.
+
+---
+
+## Appendix: Raw Data
+
+### Event Count Distribution
+
+```
+  1 event:  70 sessions (68.6%) - Mostly worker completions
+  2 events:  4 sessions (3.9%)
+  3 events:  6 sessions (5.9%)
+  4 events:  3 sessions (2.9%)
+  5 events:  3 sessions (2.9%)
+  6 events:  2 sessions (2.0%)
+  7 events:  1 session  (1.0%)
+  9 events:  1 session  (1.0%)
+ 21 events:  1 session  (1.0%) ✓ PASSING
+ 22 events:  5 sessions (4.9%)
+ 24 events:  1 session  (1.0%)
+ 27 events:  1 session  (1.0%)
+ 30 events:  2 sessions (2.0%)
+ 31 events:  1 session  (1.0%) ✓ PASSING
+ 33 events:  1 session  (1.0%) ✓ PASSING
+```
+
+### Sample Worker-Only Sessions
+
+```
+ses_6EraEW6LTRswygMPQa2voC.jsonl (1 event):
+  OUTCOME/subtask_success
+
+ses_xyJ85H9SaA5FSnJvDL7ktJ.jsonl (1 event):
+  OUTCOME/subtask_success
+
+ses_BiqTpFyafkbpt3tvZbh29R.jsonl (1 event):
+  DECISION/review_completed
+```
+
+### Sample Incomplete Coordinator Sessions
+
+```
+ses_4aa1d6e57ffeGfXIoIMNhTQ9JI.jsonl (7 events):
+  DECISION/worker_spawned (x7)
+  → Missing reviews
+
+ses_3t9CP2ZG54wF3D982kZgps.jsonl (3 events):
+  DECISION/review_completed (x3)
+  → Missing spawns
+
+test-review-1766636012605.jsonl (22 events):
+  DECISION/review_completed (x22)
+  → Missing spawns (likely test data)
+```
+
+---
+
+**Generated by:** WildDawn (swarm worker agent)  
+**Date:** 2025-12-25  
+**Files analyzed:** 102 session files from `~/.config/swarm-tools/sessions/`