opencode-swarm-plugin 0.33.0 → 0.35.0

package/CHANGELOG.md

@@ -1,5 +1,138 @@
 # opencode-swarm-plugin
 
+## 0.35.0
+
+### Minor Changes
+
+- [`084f888`](https://github.com/joelhooks/swarm-tools/commit/084f888fcac4912f594428b1ac7148c8a8aaa422) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 👁️ Watch Your Swarm in Real-Time
+
+  `swarm log` now has a `--watch` mode for continuous log monitoring. No more running the command repeatedly - just sit back and watch the bees work.
+
+  ```bash
+  # Watch all logs
+  swarm log --watch
+
+  # Watch with filters
+  swarm log compaction -w --level error
+
+  # Faster polling (500ms instead of default 1s)
+  swarm log --watch --interval 500
+  ```
+
+  **New flags:**
+
+  - `--watch`, `-w` - Enable continuous monitoring mode
+  - `--interval <ms>` - Poll interval in milliseconds (default: 1000, min: 100)
+
+  **How it works:**
+
+  - Shows initial logs (last N lines based on `--limit`)
+  - Polls log files for new entries at the specified interval
+  - Tracks file positions for efficient incremental reads
+  - Handles log rotation gracefully (detects file truncation)
+  - All existing filters work: `--level`, `--since`, module name
+  - Clean shutdown on Ctrl+C
+
+  _"The hive that watches itself, debugs itself."_
+
+## 0.34.0
+
+### Minor Changes
+
+- [`704c366`](https://github.com/joelhooks/swarm-tools/commit/704c36690fb6fd52cfb9222ddeef3b663dfdb9ed) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🪵 Pino Logging Infrastructure
+
+  > "You can't improve what you can't measure." — Peter Drucker
+
+  Finally, visibility into what the swarm is actually doing.
+
+  ### What's New
+
+  **Structured Logging with Pino**
+
+  - Daily log rotation via `pino-roll` (14-day retention)
+  - Logs to `~/.config/swarm-tools/logs/`
+  - Module-specific log files (e.g., `compaction.1log`, `swarm.1log`)
+  - Pretty mode for development: `SWARM_LOG_PRETTY=1`
+
+  **Compaction Hook Instrumented**
+
+  - 14 strategic log points across all phases
+  - START: session context, trigger reason
+  - GATHER: per-source timing (hive, swarm-mail, skills)
+  - DETECT/INJECT: confidence scores, context decisions
+  - COMPLETE: duration, success, what was injected
+
+  **New CLI: `swarm log`**
+
+  ```bash
+  swarm log                    # Tail recent logs
+  swarm log compaction         # Filter by module
+  swarm log --level warn       # Filter by severity
+  swarm log --since 1h         # Last hour only
+  swarm log --json | jq        # Pipe to jq for analysis
+  ```
+
+  ### Why This Matters
+
+  The compaction hook does a LOT of work with zero visibility:
+
+  - Context injection decisions
+  - Data gathering from multiple sources
+  - Template rendering and size calculations
+
+  Now you can answer: "What did compaction do on the last run?"
+
+  ### Technical Details
+
+  - Pino + pino-roll for async, non-blocking file writes
+  - Child loggers for module namespacing
+  - Lazy initialization pattern for test isolation
+  - 56 new tests (10 logger + 18 compaction + 28 CLI)
+
+  Complements existing `DEBUG=swarm:*` env var approach — Pino for structured file logs, debug for stderr filtering.
+
+### Patch Changes
+
+- [`b5792bd`](https://github.com/joelhooks/swarm-tools/commit/b5792bd5f6aa4bf3ad9757fe351bc144e84f09af) Thanks [@joelhooks](https://github.com/joelhooks)! - ## 🎯 Coordinators Remember Who They Are
+
+  Fixed the compaction bug where coordinators lost their identity after context compression.
+
+  **The Problem:**
+  After compaction, coordinators would wake up and start doing worker tasks directly (running tests, editing files) instead of spawning workers. The injected context said "you are a coordinator" but gave worker-style resume commands.
+
+  **The Fix:**
+  `buildDynamicSwarmState()` now generates coordinator-focused context:
+
+  ```
+  ## 🎯 YOU ARE THE COORDINATOR
+
+  **Primary role:** Orchestrate workers, review their output, unblock dependencies.
+  **Spawn workers** for implementation tasks - don't do them yourself.
+
+  **RESUME STEPS:**
+  1. Check swarm status: `swarm_status(epic_id="bd-actual-id", ...)`
+  2. Check inbox: `swarmmail_inbox(limit=5)`
+  3. For in_progress subtasks: Review with `swarm_review`
+  4. For open subtasks: Spawn workers with `swarm_spawn_subtask`
+  5. For blocked subtasks: Investigate and unblock
+  ```
+
+  Also captures specific swarm state during detection:
+
+  - Epic ID and title (not placeholders)
+  - Subtask counts by status
+  - Actual project path
+
+  **New eval infrastructure:**
+
+  - `coordinator-behavior.eval.ts` - LLM-as-judge eval testing whether Claude actually behaves like a coordinator given the injected context
+  - Scorers for coordinator tools, avoiding worker behaviors, and coordinator mindset
+
+  > "The coordinator's job is to keep the swarm cooking, not to cook themselves."
+
+- Updated dependencies [[`a78a40d`](https://github.com/joelhooks/swarm-tools/commit/a78a40de32eb34d1738b208f2a36929a4ab6cb81), [`5a7c084`](https://github.com/joelhooks/swarm-tools/commit/5a7c084514297b5b9ca5df9459a74f18eb805b8a)]:
+  - swarm-mail@1.5.0
+
 ## 0.33.0
 
 ### Minor Changes

package/README.md

@@ -179,7 +179,14 @@ Auto-saves progress at milestones. Survives context death or crashes. Data store
 
 ---
 
-## What's New in v0.32
+## What's New in v0.33
+
+- **Pino logging infrastructure** - Structured JSON logs with daily rotation to `~/.config/swarm-tools/logs/`
+- **Compaction hook instrumented** - 14 log points across all phases (START, GATHER, RENDER, DECIDE, COMPLETE)
+- **`swarm log` CLI** - Query/tail logs with module, level, and time filters
+- **Analytics queries** - 5 pre-built queries based on Four Golden Signals (latency, traffic, errors, saturation, conflicts)
+
+### v0.32
 
 - **libSQL storage** (embedded SQLite) replaced PGLite - no external DB needed
 - **95% integration test coverage** - checkpoint/recovery proven with 9 tests
@@ -235,6 +242,27 @@ swarm init      # Initialize hive in project
 swarm config    # Show config file paths
 ```
 
+### Logging & Observability
+
+Structured Pino logging with daily rotation:
+
+```bash
+# Enable pretty logging during development
+SWARM_LOG_PRETTY=1 opencode
+
+# Query logs
+swarm log                      # Tail recent logs
+swarm log compaction           # Filter by module
+swarm log --level warn         # Filter by level (warn+)
+swarm log --since 1h           # Last hour
+swarm log --json | jq          # Pipe to jq for analysis
+```
+
+**Log files:** `~/.config/swarm-tools/logs/`
+- `swarm.1log`, `swarm.2log`, ... (main logs)
+- `compaction.1log`, ... (module-specific)
+- Daily rotation, 14-day retention
+
 ---
 
 ## Further Reading

package/bin/swarm.test.ts

@@ -8,7 +8,7 @@
  * - rmWithStatus: logs file removal
  */
 import { describe, test, expect, beforeEach, afterEach } from "bun:test";
-import { mkdirSync, rmSync, writeFileSync, existsSync, readFileSync } from "fs";
+import { mkdirSync, rmSync, writeFileSync, existsSync, readFileSync, readdirSync } from "fs";
 import { join } from "path";
 import { tmpdir } from "os";
 
@@ -192,3 +192,344 @@ READ-ONLY research agent. Never modifies code - only gathers intel and stores fi
     });
   });
 });
+
+// ============================================================================
+// Log Command Tests (TDD)
+// ============================================================================
+
+describe("Log command helpers", () => {
+  let testDir: string;
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `swarm-log-test-${Date.now()}`);
+    mkdirSync(testDir, { recursive: true });
+  });
+
+  afterEach(() => {
+    if (existsSync(testDir)) {
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+
+  describe("parseLogLine", () => {
+    function parseLogLine(line: string): { level: number; time: string; module: string; msg: string } | null {
+      try {
+        const parsed = JSON.parse(line);
+        if (typeof parsed.level === "number" && parsed.time && parsed.msg) {
+          return {
+            level: parsed.level,
+            time: parsed.time,
+            module: parsed.module || "unknown",
+            msg: parsed.msg,
+          };
+        }
+      } catch {
+        // Invalid JSON
+      }
+      return null;
+    }
+
+    test("parses valid log line", () => {
+      const line = '{"level":30,"time":"2024-12-24T16:00:00.000Z","module":"compaction","msg":"started"}';
+      const result = parseLogLine(line);
+      
+      expect(result).not.toBeNull();
+      expect(result?.level).toBe(30);
+      expect(result?.module).toBe("compaction");
+      expect(result?.msg).toBe("started");
+    });
+
+    test("returns null for invalid JSON", () => {
+      const line = "not json";
+      expect(parseLogLine(line)).toBeNull();
+    });
+
+    test("defaults module to 'unknown' if missing", () => {
+      const line = '{"level":30,"time":"2024-12-24T16:00:00.000Z","msg":"test"}';
+      const result = parseLogLine(line);
+      
+      expect(result?.module).toBe("unknown");
+    });
+  });
+
+  describe("filterLogsByLevel", () => {
+    function filterLogsByLevel(logs: Array<{ level: number }>, minLevel: number): Array<{ level: number }> {
+      return logs.filter((log) => log.level >= minLevel);
+    }
+
+    test("filters logs by minimum level", () => {
+      const logs = [
+        { level: 10 }, // trace
+        { level: 30 }, // info
+        { level: 50 }, // error
+      ];
+      
+      const result = filterLogsByLevel(logs, 30);
+      expect(result).toHaveLength(2);
+      expect(result[0].level).toBe(30);
+      expect(result[1].level).toBe(50);
+    });
+
+    test("includes all logs when minLevel is 0", () => {
+      const logs = [
+        { level: 10 },
+        { level: 20 },
+        { level: 30 },
+      ];
+      
+      const result = filterLogsByLevel(logs, 0);
+      expect(result).toHaveLength(3);
+    });
+  });
+
+  describe("filterLogsByModule", () => {
+    function filterLogsByModule(logs: Array<{ module: string }>, module: string): Array<{ module: string }> {
+      return logs.filter((log) => log.module === module);
+    }
+
+    test("filters logs by exact module name", () => {
+      const logs = [
+        { module: "compaction" },
+        { module: "swarm" },
+        { module: "compaction" },
+      ];
+      
+      const result = filterLogsByModule(logs, "compaction");
+      expect(result).toHaveLength(2);
+    });
+
+    test("returns empty array when no match", () => {
+      const logs = [
+        { module: "compaction" },
+      ];
+      
+      const result = filterLogsByModule(logs, "swarm");
+      expect(result).toHaveLength(0);
+    });
+  });
+
+  describe("filterLogsBySince", () => {
+    function parseDuration(duration: string): number | null {
+      const match = duration.match(/^(\d+)([smhd])$/);
+      if (!match) return null;
+      
+      const [, num, unit] = match;
+      const value = parseInt(num, 10);
+      
+      const multipliers: Record<string, number> = {
+        s: 1000,
+        m: 60 * 1000,
+        h: 60 * 60 * 1000,
+        d: 24 * 60 * 60 * 1000,
+      };
+      
+      return value * multipliers[unit];
+    }
+
+    function filterLogsBySince(logs: Array<{ time: string }>, sinceMs: number): Array<{ time: string }> {
+      const cutoffTime = Date.now() - sinceMs;
+      return logs.filter((log) => new Date(log.time).getTime() >= cutoffTime);
+    }
+
+    test("parseDuration handles seconds", () => {
+      expect(parseDuration("30s")).toBe(30 * 1000);
+    });
+
+    test("parseDuration handles minutes", () => {
+      expect(parseDuration("5m")).toBe(5 * 60 * 1000);
+    });
+
+    test("parseDuration handles hours", () => {
+      expect(parseDuration("2h")).toBe(2 * 60 * 60 * 1000);
+    });
+
+    test("parseDuration handles days", () => {
+      expect(parseDuration("1d")).toBe(24 * 60 * 60 * 1000);
+    });
+
+    test("parseDuration returns null for invalid format", () => {
+      expect(parseDuration("invalid")).toBeNull();
+      expect(parseDuration("30x")).toBeNull();
+      expect(parseDuration("30")).toBeNull();
+    });
+
+    test("filterLogsBySince filters old logs", () => {
+      const now = Date.now();
+      const logs = [
+        { time: new Date(now - 10000).toISOString() }, // 10s ago
+        { time: new Date(now - 120000).toISOString() }, // 2m ago
+        { time: new Date(now - 1000).toISOString() }, // 1s ago
+      ];
+      
+      const result = filterLogsBySince(logs, 60000); // Last 1m
+      expect(result).toHaveLength(2); // Only logs within last minute
+    });
+  });
+
+  describe("formatLogLine", () => {
+    function levelToName(level: number): string {
+      if (level >= 60) return "FATAL";
+      if (level >= 50) return "ERROR";
+      if (level >= 40) return "WARN ";
+      if (level >= 30) return "INFO ";
+      if (level >= 20) return "DEBUG";
+      return "TRACE";
+    }
+
+    function formatLogLine(log: { level: number; time: string; module: string; msg: string }): string {
+      const timestamp = new Date(log.time).toLocaleTimeString();
+      const levelName = levelToName(log.level);
+      const module = log.module.padEnd(12);
+      return `${timestamp} ${levelName} ${module} ${log.msg}`;
+    }
+
+    test("formats log line with timestamp and level", () => {
+      const log = {
+        level: 30,
+        time: "2024-12-24T16:00:00.000Z",
+        module: "compaction",
+        msg: "started",
+      };
+      
+      const result = formatLogLine(log);
+      expect(result).toContain("INFO");
+      expect(result).toContain("compaction");
+      expect(result).toContain("started");
+    });
+
+    test("pads module name for alignment", () => {
+      const log1 = formatLogLine({ level: 30, time: "2024-12-24T16:00:00.000Z", module: "a", msg: "test" });
+      const log2 = formatLogLine({ level: 30, time: "2024-12-24T16:00:00.000Z", module: "compaction", msg: "test" });
+      
+      // Module names should be padded to 12 chars
+      expect(log1).toContain("a            test"); // 'a' + 11 spaces
+      expect(log2).toContain("compaction   test"); // 'compaction' + 3 spaces (10 chars + 2)
+    });
+
+    test("levelToName maps all levels correctly", () => {
+      expect(levelToName(10)).toBe("TRACE");
+      expect(levelToName(20)).toBe("DEBUG");
+      expect(levelToName(30)).toBe("INFO ");
+      expect(levelToName(40)).toBe("WARN ");
+      expect(levelToName(50)).toBe("ERROR");
+      expect(levelToName(60)).toBe("FATAL");
+    });
+  });
+
+  describe("readLogFiles", () => {
+    test("reads multiple .1log files", () => {
+      // Create test log files
+      const log1 = join(testDir, "swarm.1log");
+      const log2 = join(testDir, "swarm.2log");
+      const log3 = join(testDir, "compaction.1log");
+      
+      writeFileSync(log1, '{"level":30,"time":"2024-12-24T16:00:00.000Z","msg":"line1"}\n');
+      writeFileSync(log2, '{"level":30,"time":"2024-12-24T16:00:01.000Z","msg":"line2"}\n');
+      writeFileSync(log3, '{"level":30,"time":"2024-12-24T16:00:02.000Z","module":"compaction","msg":"line3"}\n');
+      
+      function readLogFiles(dir: string): string[] {
+        if (!existsSync(dir)) return [];
+        
+        const files = readdirSync(dir)
+          .filter((f) => /\.\d+log$/.test(f))
+          .sort() // Sort by filename
+          .map((f) => join(dir, f));
+        
+        const lines: string[] = [];
+        for (const file of files) {
+          const content = readFileSync(file, "utf-8");
+          lines.push(...content.split("\n").filter((line) => line.trim()));
+        }
+        
+        return lines;
+      }
+      
+      const lines = readLogFiles(testDir);
+      expect(lines).toHaveLength(3);
+      // Files are sorted alphabetically: compaction.1log, swarm.1log, swarm.2log
+      expect(lines.some((l) => l.includes("line1"))).toBe(true);
+      expect(lines.some((l) => l.includes("line2"))).toBe(true);
+      expect(lines.some((l) => l.includes("line3"))).toBe(true);
+    });
+
+    test("returns empty array for non-existent directory", () => {
+      function readLogFiles(dir: string): string[] {
+        if (!existsSync(dir)) return [];
+        return [];
+      }
+      
+      const lines = readLogFiles(join(testDir, "nonexistent"));
+      expect(lines).toHaveLength(0);
+    });
+  });
+
+  describe("watchLogs", () => {
+    test("detects new log lines appended to file", async () => {
+      const logFile = join(testDir, "swarm.1log");
+      const collectedLines: string[] = [];
+      
+      // Create initial log file
+      writeFileSync(logFile, '{"level":30,"time":"2024-12-24T16:00:00.000Z","msg":"initial"}\n');
+      
+      // Import watch utilities
+      const { watch } = await import("fs");
+      const { appendFileSync } = await import("fs");
+      
+      // Track file position for incremental reads
+      let lastSize = 0;
+      
+      function readNewLines(filePath: string): string[] {
+        const content = readFileSync(filePath, "utf-8");
+        const newContent = content.slice(lastSize);
+        lastSize = content.length;
+        return newContent.split("\n").filter((line) => line.trim());
+      }
+      
+      // Simulate watch behavior
+      const watcher = watch(testDir, (eventType, filename) => {
+        if (filename && /\.\d+log$/.test(filename)) {
+          const newLines = readNewLines(join(testDir, filename));
+          collectedLines.push(...newLines);
+        }
+      });
+      
+      // Wait for watcher to be ready
+      await new Promise((resolve) => setTimeout(resolve, 100));
+      
+      // Append new log line
+      appendFileSync(logFile, '{"level":30,"time":"2024-12-24T16:00:01.000Z","msg":"appended"}\n');
+      
+      // Wait for event to fire
+      await new Promise((resolve) => setTimeout(resolve, 200));
+      
+      watcher.close();
+      
+      // Should have detected the new line
+      expect(collectedLines.some((l) => l.includes("appended"))).toBe(true);
+    });
+
+    test("parseWatchArgs extracts --watch flag", () => {
+      function parseWatchArgs(args: string[]): { watch: boolean; interval: number } {
+        let watch = false;
+        let interval = 1000; // default 1 second
+        
+        for (let i = 0; i < args.length; i++) {
+          const arg = args[i];
+          if (arg === "--watch" || arg === "-w") {
+            watch = true;
+          } else if (arg === "--interval" && i + 1 < args.length) {
+            interval = parseInt(args[++i], 10);
+          }
+        }
+        
+        return { watch, interval };
+      }
+      
+      expect(parseWatchArgs(["--watch"])).toEqual({ watch: true, interval: 1000 });
+      expect(parseWatchArgs(["-w"])).toEqual({ watch: true, interval: 1000 });
+      expect(parseWatchArgs(["--watch", "--interval", "500"])).toEqual({ watch: true, interval: 500 });
+      expect(parseWatchArgs(["compaction", "--watch"])).toEqual({ watch: true, interval: 1000 });
+      expect(parseWatchArgs(["--level", "error"])).toEqual({ watch: false, interval: 1000 });
+    });
+  });
+});