opencode-swarm-plugin 0.39.1 → 0.42.0

package/evals/README.md

@@ -167,6 +167,119 @@ coordinator-behavior
    → overallDiscipline: 0.89 ✅ PASS (bootstrap phase, collecting data)
 ```
 
+#### Coordinator Session Capture (Deep Dive)
+
+**How it works:** Session capture is fully automatic when coordinator tools are used. No manual instrumentation needed.
+
+**Capture flow:**
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                  SESSION CAPTURE FLOW                       │
+│                                                             │
+│  1. Coordinator tool call detected                          │
+│     ├─ swarm_decompose, hive_create_epic, etc.              │
+│     └─ Tool name + args inspected in real-time              │
+│                                                             │
+│  2. Violation detection (planning-guardrails.ts)            │
+│     ├─ detectCoordinatorViolation() checks patterns         │
+│     ├─ Edit/Write tools → coordinator_edited_file           │
+│     ├─ bash with test patterns → coordinator_ran_tests      │
+│     └─ swarmmail_reserve → coordinator_reserved_files       │
+│                                                             │
+│  3. Event emission (eval-capture.ts)                        │
+│     ├─ captureCoordinatorEvent() validates via Zod          │
+│     ├─ Appends JSONL line to session file                   │
+│     └─ ~/.config/swarm-tools/sessions/{session_id}.jsonl    │
+│                                                             │
+│  4. Eval consumption (coordinator-session.eval.ts)          │
+│     ├─ loadCapturedSessions() reads all *.jsonl files       │
+│     ├─ Parses events, reconstructs sessions                 │
+│     └─ Scorers analyze event sequences                      │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
+
+**Event types:**
+
+| Event Type     | Subtypes                                                              | When Captured                        |
+| -------------- | --------------------------------------------------------------------- | ------------------------------------ |
+| `DECISION`     | strategy_selected, worker_spawned, review_completed, decomposition_complete | Coordinator makes decision           |
+| `VIOLATION`    | coordinator_edited_file, coordinator_ran_tests, coordinator_reserved_files, no_worker_spawned | Protocol violation detected          |
+| `OUTCOME`      | subtask_success, subtask_retry, subtask_failed, epic_complete        | Worker completes or epic finishes    |
+| `COMPACTION`   | detection_complete, prompt_generated, context_injected, resumption_started, tool_call_tracked | Compaction lifecycle events          |
+
+**Violation detection patterns** (from `planning-guardrails.ts`):
+
+```typescript
+// File modification detection
+VIOLATION_PATTERNS.FILE_MODIFICATION_TOOLS = ["edit", "write"];
+
+// Test execution detection (regex patterns in bash commands)
+VIOLATION_PATTERNS.TEST_EXECUTION_PATTERNS = [
+  /\bbun\s+test\b/i,
+  /\bnpm\s+(run\s+)?test/i,
+  /\bjest\b/i,
+  /\bvitest\b/i,
+  // ... and 6 more patterns
+];
+
+// File reservation detection
+VIOLATION_PATTERNS.RESERVATION_TOOLS = ["swarmmail_reserve", "agentmail_reserve"];
+```
+
+**Example session file** (`~/.config/swarm-tools/sessions/session-abc123.jsonl`):
+
+```jsonl
+{"session_id":"session-abc123","epic_id":"mjkw81rkq4c","timestamp":"2025-01-01T12:00:00Z","event_type":"DECISION","decision_type":"strategy_selected","payload":{"strategy":"feature-based"}}
+{"session_id":"session-abc123","epic_id":"mjkw81rkq4c","timestamp":"2025-01-01T12:01:00Z","event_type":"DECISION","decision_type":"decomposition_complete","payload":{"subtask_count":3}}
+{"session_id":"session-abc123","epic_id":"mjkw81rkq4c","timestamp":"2025-01-01T12:02:00Z","event_type":"DECISION","decision_type":"worker_spawned","payload":{"worker_id":"SwiftFire","bead_id":"mjkw81rkq4c.1"}}
+{"session_id":"session-abc123","epic_id":"mjkw81rkq4c","timestamp":"2025-01-01T12:05:00Z","event_type":"VIOLATION","violation_type":"coordinator_edited_file","payload":{"tool":"edit","file":"src/auth.ts"}}
+{"session_id":"session-abc123","epic_id":"mjkw81rkq4c","timestamp":"2025-01-01T12:10:00Z","event_type":"OUTCOME","outcome_type":"subtask_success","payload":{"bead_id":"mjkw81rkq4c.1","duration_ms":480000}}
+```
+
+**Viewing sessions:**
+
+```bash
+# List all captured sessions (coming soon)
+swarm log sessions
+
+# View specific session events
+cat ~/.config/swarm-tools/sessions/session-abc123.jsonl | jq .
+
+# Filter to violations only
+cat ~/.config/swarm-tools/sessions/*.jsonl | jq 'select(.event_type == "VIOLATION")'
+
+# Count violations by type
+cat ~/.config/swarm-tools/sessions/*.jsonl | jq -r 'select(.event_type == "VIOLATION") | .violation_type' | sort | uniq -c
+```
+
+**Why JSONL format?**
+
+- **Append-only**: No file locking, safe for concurrent writes
+- **Streamable**: Process events one-by-one without loading full file
+- **Line-oriented**: Easy to `grep`, `jq`, `tail -f` for live monitoring
+- **Fault-tolerant**: Corrupted line doesn't break entire file
+
+**Integration points:**
+
+| Where                      | What Gets Captured                        | File                    |
+| -------------------------- | ----------------------------------------- | ----------------------- |
+| `swarm_decompose`          | DECISION: strategy_selected, decomposition_complete | sessions/*.jsonl        |
+| `swarm_spawn_subtask`      | DECISION: worker_spawned                  | sessions/*.jsonl        |
+| `swarm_review`             | DECISION: review_completed                | sessions/*.jsonl        |
+| `swarm_complete`           | OUTCOME: subtask_success/failed           | sessions/*.jsonl        |
+| Tool call inspection       | VIOLATION: (real-time pattern matching)   | sessions/*.jsonl        |
+| Compaction hook            | COMPACTION: (all lifecycle stages)        | sessions/*.jsonl        |
+
+**Source files:**
+
+- **Schema**: `src/eval-capture.ts` - CoordinatorEventSchema (Zod discriminated union)
+- **Violation detection**: `src/planning-guardrails.ts` - detectCoordinatorViolation()
+- **Capture**: `src/eval-capture.ts` - captureCoordinatorEvent()
+- **Scorers**: `evals/scorers/coordinator-discipline.ts` - violationCount, spawnEfficiency, etc.
+- **Eval**: `evals/coordinator-session.eval.ts` - Real sessions + fixtures
+
 ### Compaction Prompt (`compaction-prompt.eval.ts`)
 
 **What it measures:** Quality of continuation prompts after context compaction

package/evals/example.eval.ts

@@ -14,19 +14,18 @@ evalite("Example: Basic scorer test", {
   data: async () => {
     return [
       {
-        input: "Test task",
-        output: JSON.stringify({
+        input: {
           epic: { title: "Test Epic", description: "Test" },
           subtasks: [
             { title: "Subtask 1", files: ["a.ts"], estimated_complexity: 1 },
             { title: "Subtask 2", files: ["b.ts"], estimated_complexity: 1 },
           ],
-        }),
+        },
       },
     ];
   },
   task: async (input) => {
-    return input; // passthrough for testing
+    return JSON.stringify(input);
   },
   scorers: [subtaskIndependence],
 });

package/evals/fixtures/compaction-prompt-cases.ts

@@ -78,6 +78,8 @@ Coordinators do NOT edit code directly. These tools are FORBIDDEN:
 - edit
 - write
 - bash (for file modifications)
+- swarmmail_reserve (only workers reserve)
+- git commit (workers commit)
 
 Use swarm_spawn_subtask to delegate work to workers.
 
@@ -249,6 +251,8 @@ You are the COORDINATOR of epic mjkweh7q9n4.
 - edit
 - write
 - bash (for file mods)
+- swarmmail_reserve (only workers)
+- git commit (workers only)
 
 NEVER edit files yourself.
 ALWAYS delegate to workers.
@@ -289,6 +293,8 @@ You are coordinating epics:
 - edit
 - write  
 - bash
+- swarmmail_reserve
+- git commit
 
 ALWAYS check status first.
 NEVER edit files directly.

package/evals/scorers/coordinator-discipline.evalite-test.ts

@@ -5,6 +5,7 @@ import { describe, expect, it } from "bun:test";
 import type { CoordinatorSession } from "../../src/eval-capture.js";
 import {
 	overallDiscipline,
+	reviewEfficiency,
 	reviewThoroughness,
 	spawnEfficiency,
 	timeToFirstSpawn,
@@ -535,3 +536,165 @@ describe("overallDiscipline", () => {
 		expect(result.message).toContain("Speed:");
 	});
 });
+
+describe("reviewEfficiency", () => {
+	it("scores 1.0 for ideal 1:1 ratio (one review per spawn)", async () => {
+		const session: CoordinatorSession = {
+			session_id: "test-session",
+			epic_id: "test-epic",
+			start_time: "2025-01-01T00:00:00Z",
+			events: [
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:00:10Z",
+					event_type: "DECISION",
+					decision_type: "worker_spawned",
+					payload: { bead_id: "bd-1" },
+				},
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:00:20Z",
+					event_type: "DECISION",
+					decision_type: "worker_spawned",
+					payload: { bead_id: "bd-2" },
+				},
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:10:00Z",
+					event_type: "DECISION",
+					decision_type: "review_completed",
+					payload: { bead_id: "bd-1" },
+				},
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:10:10Z",
+					event_type: "DECISION",
+					decision_type: "review_completed",
+					payload: { bead_id: "bd-2" },
+				},
+			],
+		};
+
+		const result = await reviewEfficiency({
+			output: JSON.stringify(session),
+			expected: {},
+			input: undefined,
+		});
+
+		expect(result.score).toBe(1.0);
+		expect(result.message).toContain("2 reviews / 2 spawns");
+	});
+
+	it("penalizes over-reviewing (>2:1 ratio)", async () => {
+		// 6 reviews for 2 spawns = 3:1 ratio (over-reviewing)
+		const session: CoordinatorSession = {
+			session_id: "test-session",
+			epic_id: "test-epic",
+			start_time: "2025-01-01T00:00:00Z",
+			events: [
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:00:10Z",
+					event_type: "DECISION",
+					decision_type: "worker_spawned",
+					payload: { bead_id: "bd-1" },
+				},
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:00:20Z",
+					event_type: "DECISION",
+					decision_type: "worker_spawned",
+					payload: { bead_id: "bd-2" },
+				},
+				...Array.from({ length: 6 }, (_, i) => ({
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: `2025-01-01T00:10:${String(i * 10).padStart(2, "0")}Z`,
+					event_type: "DECISION" as const,
+					decision_type: "review_completed" as const,
+					payload: { bead_id: `bd-${(i % 2) + 1}` },
+				})),
+			],
+		};
+
+		const result = await reviewEfficiency({
+			output: JSON.stringify(session),
+			expected: {},
+			input: undefined,
+		});
+
+		// 3:1 ratio should be penalized (score < 0.5)
+		expect(result.score).toBeLessThan(0.5);
+		expect(result.message).toContain("6 reviews / 2 spawns");
+	});
+
+	it("handles no spawns gracefully", async () => {
+		const session: CoordinatorSession = {
+			session_id: "test-session",
+			epic_id: "test-epic",
+			start_time: "2025-01-01T00:00:00Z",
+			events: [
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:00:00Z",
+					event_type: "DECISION",
+					decision_type: "strategy_selected",
+					payload: { strategy: "file-based" },
+				},
+			],
+		};
+
+		const result = await reviewEfficiency({
+			output: JSON.stringify(session),
+			expected: {},
+			input: undefined,
+		});
+
+		expect(result.score).toBe(1.0);
+		expect(result.message).toContain("No workers spawned");
+	});
+
+	it("handles no reviews gracefully (0:N ratio)", async () => {
+		const session: CoordinatorSession = {
+			session_id: "test-session",
+			epic_id: "test-epic",
+			start_time: "2025-01-01T00:00:00Z",
+			events: [
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:00:10Z",
+					event_type: "DECISION",
+					decision_type: "worker_spawned",
+					payload: { bead_id: "bd-1" },
+				},
+				{
+					session_id: "test-session",
+					epic_id: "test-epic",
+					timestamp: "2025-01-01T00:00:20Z",
+					event_type: "DECISION",
+					decision_type: "worker_spawned",
+					payload: { bead_id: "bd-2" },
+				},
+			],
+		};
+
+		const result = await reviewEfficiency({
+			output: JSON.stringify(session),
+			expected: {},
+			input: undefined,
+		});
+
+		// No reviews is bad (should use reviewThoroughness for this)
+		// But this scorer focuses on over-reviewing, so no reviews = 1.0 (not over-reviewing)
+		expect(result.score).toBe(1.0);
+		expect(result.message).toContain("0 reviews / 2 spawns");
+	});
+});

package/evals/scorers/coordinator-discipline.ts

@@ -70,6 +70,9 @@ export const violationCount = createScorer({
  * Coordinators should delegate work, not do it themselves.
  *
  * Score: workers_spawned / subtasks_planned
+ *
+ * If no decomposition_complete event exists, falls back to counting spawns
+ * and returns 1.0 if any workers were spawned (better than nothing).
  */
 export const spawnEfficiency = createScorer({
   name: "Spawn Efficiency",
@@ -85,7 +88,20 @@ export const spawnEfficiency = createScorer({
           e.decision_type === "decomposition_complete"
       );
 
+      // Count worker_spawned events
+      const spawned = session.events.filter(
+        (e) =>
+          e.event_type === "DECISION" && e.decision_type === "worker_spawned"
+      ).length;
+
       if (!decomp) {
+        // Fallback: if workers were spawned but no decomp event, assume they're doing work
+        if (spawned > 0) {
+          return {
+            score: 1.0,
+            message: `${spawned} workers spawned (no decomposition event)`,
+          };
+        }
         return {
           score: 0,
           message: "No decomposition event found",
@@ -101,17 +117,81 @@ export const spawnEfficiency = createScorer({
         };
       }
 
+      const score = spawned / subtaskCount;
+
+      return {
+        score,
+        message: `${spawned}/${subtaskCount} workers spawned (${(score * 100).toFixed(0)}%)`,
+      };
+    } catch (error) {
+      return {
+        score: 0,
+        message: `Failed to parse CoordinatorSession: ${error}`,
+      };
+    }
+  },
+});
+
+/**
+ * Review Efficiency Scorer
+ *
+ * Measures review-to-spawn ratio to detect over-reviewing.
+ * Ideal ratio is 1:1 (one review per spawned worker).
+ * Penalizes >2:1 ratio (over-reviewing wastes context).
+ *
+ * Scoring:
+ * - 0:N or 1:1 ratio = 1.0 (perfect)
+ * - 2:1 ratio = 0.5 (threshold)
+ * - >2:1 ratio = linear penalty toward 0.0
+ *
+ * Score: normalized to 0-1 (lower ratio is better)
+ */
+export const reviewEfficiency = createScorer({
+  name: "Review Efficiency",
+  description: "Review-to-spawn ratio (penalize over-reviewing >2:1)",
+  scorer: ({ output }) => {
+    try {
+      const session = JSON.parse(String(output)) as CoordinatorSession;
+
       // Count worker_spawned events
       const spawned = session.events.filter(
         (e) =>
           e.event_type === "DECISION" && e.decision_type === "worker_spawned"
       ).length;
 
-      const score = spawned / subtaskCount;
+      if (spawned === 0) {
+        return {
+          score: 1.0,
+          message: "No workers spawned",
+        };
+      }
+
+      // Count review_completed events
+      const reviewed = session.events.filter(
+        (e) =>
+          e.event_type === "DECISION" && e.decision_type === "review_completed"
+      ).length;
+
+      const ratio = reviewed / spawned;
+
+      // Scoring:
+      // - ratio <= 1.0: perfect (1.0)
+      // - ratio <= 2.0: linear decay from 1.0 to 0.5
+      // - ratio > 2.0: linear penalty from 0.5 toward 0.0
+      let score: number;
+      if (ratio <= 1.0) {
+        score = 1.0;
+      } else if (ratio <= 2.0) {
+        // Linear decay: 1.0 at ratio=1.0, 0.5 at ratio=2.0
+        score = 1.0 - (ratio - 1.0) * 0.5;
+      } else {
+        // Penalty for extreme over-reviewing: 0.5 at ratio=2.0, 0.0 at ratio=4.0
+        score = Math.max(0, 0.5 - (ratio - 2.0) * 0.25);
+      }
 
       return {
         score,
-        message: `${spawned}/${subtaskCount} workers spawned (${(score * 100).toFixed(0)}%)`,
+        message: `${reviewed} reviews / ${spawned} spawns (${ratio.toFixed(1)}:1 ratio)`,
       };
     } catch (error) {
       return {

package/evals/scorers/index.test.ts

@@ -0,0 +1,146 @@
+/**
+ * Tests for decomposition scorers
+ *
+ * Uses Vitest (evalite's test runner), not Bun's test runner.
+ *
+ * Note: evalite's Score type only exposes `score`, not `message`.
+ * We test scores only - message testing requires accessing internal scorer.
+ */
+import { describe, expect, test } from "vitest";
+import {
+  coverageCompleteness,
+  decompositionCoherence,
+  instructionClarity,
+  subtaskIndependence,
+} from "./index.js";
+
+describe("Heuristic Scorers", () => {
+  const goodDecomposition = JSON.stringify({
+    epic: { title: "Add auth", description: "Add authentication" },
+    subtasks: [
+      {
+        title: "Add login form component",
+        description: "Create React component for login with email/password",
+        files: ["src/components/LoginForm.tsx"],
+      },
+      {
+        title: "Add auth API routes",
+        description: "Create API endpoints for login/logout/session",
+        files: ["src/api/auth.ts"],
+      },
+      {
+        title: "Add auth middleware",
+        description: "Create middleware to protect routes",
+        files: ["src/middleware/auth.ts"],
+      },
+    ],
+  });
+
+  const conflictingDecomposition = JSON.stringify({
+    epic: { title: "Add auth", description: "Add authentication" },
+    subtasks: [
+      {
+        title: "Add login",
+        files: ["src/auth.ts"],
+      },
+      {
+        title: "Add logout",
+        files: ["src/auth.ts"], // Same file - conflict!
+      },
+    ],
+  });
+
+  test("subtaskIndependence scores 1.0 for no conflicts", async () => {
+    const result = await subtaskIndependence({
+      output: goodDecomposition,
+      expected: undefined,
+      input: {},
+    });
+    expect(result.score).toBe(1);
+  });
+
+  test("subtaskIndependence scores 0 for file conflicts", async () => {
+    const result = await subtaskIndependence({
+      output: conflictingDecomposition,
+      expected: undefined,
+      input: {},
+    });
+    expect(result.score).toBe(0);
+  });
+
+  test("instructionClarity scores higher for detailed subtasks", async () => {
+    const result = await instructionClarity({
+      output: goodDecomposition,
+      expected: undefined,
+      input: {},
+    });
+    expect(result.score).toBeGreaterThan(0.7);
+  });
+
+  test("coverageCompleteness checks subtask count", async () => {
+    const result = await coverageCompleteness({
+      output: goodDecomposition,
+      expected: { minSubtasks: 2, maxSubtasks: 5 },
+      input: {},
+    });
+    expect(result.score).toBe(1);
+  });
+});
+
+describe("LLM-as-Judge Scorer", () => {
+  // Skip LLM test in CI - requires API key
+  const hasApiKey = !!process.env.AI_GATEWAY_API_KEY;
+
+  test(
+    "decompositionCoherence returns valid score",
+    async () => {
+      if (!hasApiKey) {
+        console.log("Skipping LLM test - no AI_GATEWAY_API_KEY");
+        return;
+      }
+
+      const decomposition = JSON.stringify({
+        epic: { title: "Add auth", description: "Add authentication" },
+        subtasks: [
+          {
+            title: "Add login form",
+            description: "Create login UI",
+            files: ["src/LoginForm.tsx"],
+          },
+          {
+            title: "Add auth API",
+            description: "Create auth endpoints",
+            files: ["src/api/auth.ts"],
+          },
+        ],
+      });
+
+      const result = await decompositionCoherence({
+        output: decomposition,
+        expected: undefined,
+        input: { task: "Add user authentication with login/logout" },
+      });
+
+      expect(result.score).toBeGreaterThanOrEqual(0);
+      expect(result.score).toBeLessThanOrEqual(1);
+    },
+    30000,
+  );
+
+  test("decompositionCoherence scores invalid decomposition low", async () => {
+    if (!process.env.AI_GATEWAY_API_KEY) {
+      console.log("Skipping LLM test - no AI_GATEWAY_API_KEY");
+      return;
+    }
+
+    const result = await decompositionCoherence({
+      output: "not valid json at all {{{",
+      expected: undefined,
+      input: {},
+    });
+
+    // LLM should recognize garbage input and score it very low
+    // (0 or close to 0, not 0.5 fallback)
+    expect(result.score).toBeLessThanOrEqual(0.2);
+  }, 30000);
+});