@nathapp/nax 0.18.1

package/test/integration/US-005-TEST-SUMMARY.md

@@ -0,0 +1,138 @@
+# US-005: Plugin Routing Integration - Test Summary
+
+**Story:** Plugin routing strategies integrate into router chain
+
+**Test File:** `test/integration/plugin-routing.test.ts`
+
+**Status:** ✅ Tests written and failing (as expected)
+
+---
+
+## Test Coverage
+
+### AC1: Plugin routers are tried before the built-in routing strategy
+
+**Tests (4):**
+- ✅ Plugin routers execute before built-in keyword strategy
+- ✅ Multiple plugin routers maintain load order
+- ✅ Plugin routers are inserted before manual strategy
+- ✅ Plugin routers are inserted before llm strategy
+
+**Coverage:** Validates that plugin routers are prepended to the strategy chain and execute in load order before any built-in strategies.
+
+---
+
+### AC2: First plugin router that returns a non-null result wins
+
+**Tests (5):**
+- ✅ First plugin router decision is used
+- ✅ Second plugin router is used when first returns null
+- ✅ Plugin router overrides built-in keyword strategy
+- ✅ Third plugin router is used when first two return null
+- ✅ Plugin router can delegate based on conditional logic (integration test)
+
+**Coverage:** Validates the chain precedence rules where the first non-null decision wins, and tests realistic scenarios where plugins selectively handle certain stories.
+
+---
+
+### AC3: If all plugin routers return null, built-in strategy is used as fallback
+
+**Tests (4):**
+- ✅ Keyword strategy is used when all plugin routers return null
+- ✅ Keyword strategy handles complex story when plugins return null
+- ✅ Manual strategy is used as fallback when plugins return null
+- ✅ Empty plugin registry falls back to keyword strategy
+
+**Coverage:** Validates that the chain properly falls through to built-in strategies (keyword, manual) when all plugin routers return null or when no plugins are loaded.
+
+---
+
+### AC4: Plugin routers receive the same story context as built-in routers
+
+**Tests (5):**
+- ✅ Plugin router receives story object
+- ✅ Plugin router receives routing context with config
+- ✅ Plugin router receives codebase context when available
+- ✅ Plugin router receives metrics when available
+- ✅ Multiple plugin routers receive same context
+
+**Coverage:** Validates that plugin routers receive the complete `UserStory` and `RoutingContext` objects, including optional fields like `codebaseContext` and `metrics`.
+
+---
+
+### AC5: Router errors are caught and logged; fallback to next router in chain
+
+**Tests (6):**
+- ❌ Error in plugin router is caught and next router is tried
+- ❌ Error in plugin router is logged
+- ❌ Multiple router errors are caught and keyword fallback succeeds
+- ❌ Async error in plugin router is caught
+- ❌ Error in last plugin router falls back to keyword strategy
+- ❌ Error message includes plugin name for debugging
+
+**Coverage:** Validates that errors thrown by plugin routers don't crash the routing system and that proper error logging occurs with router names for debugging.
+
+**Status:** Currently failing (expected) - error handling not yet implemented in `StrategyChain.route()`
+
+---
+
+## Integration Tests
+
+**Real-world scenarios (6):**
+- ✅ Premium plugin forces security stories to expert tier
+- ✅ Cost-optimization plugin downgrades simple docs to fast tier
+- ✅ Domain-specific plugin routes database migrations to expert tier
+- ✅ Multiple plugins: first matching plugin wins
+- ✅ Plugin router can delegate based on conditional logic (duplicate coverage)
+
+**Coverage:** Tests realistic plugin use cases that demonstrate the value of the plugin router system.
+
+---
+
+## Test Results
+
+```
+ 22 pass
+ 6 fail
+ 54 expect() calls
+Ran 28 tests across 1 file.
+```
+
+**Passing Tests (22):** Plugin router chain integration, precedence, fallback, and context passing all work correctly with the current implementation.
+
+**Failing Tests (6):** All failures are in AC5 (error handling). The errors are propagating instead of being caught, logged, and triggering fallback to the next router.
+
+---
+
+## Implementation Gaps
+
+The tests reveal that the following needs to be implemented:
+
+1. **Error handling in `StrategyChain.route()`** (`src/routing/chain.ts:38-44`)
+   - Wrap each `strategy.route()` call in try-catch
+   - Log errors with strategy name
+   - Continue to next strategy on error
+
+2. **Error logging with plugin context**
+   - Include plugin router name in error logs
+   - Use `getSafeLogger()` to log routing errors
+   - Log at error level with category "routing"
+
+---
+
+## Next Steps for Implementer
+
+1. Modify `src/routing/chain.ts` to wrap `strategy.route()` in try-catch
+2. Add error logging that includes strategy name
+3. Continue chain iteration on error (same as null return)
+4. Run tests: `bun test ./test/integration/plugin-routing.test.ts`
+5. All 28 tests should pass after implementation
+
+---
+
+## Notes
+
+- Tests use mock plugins with custom routing logic
+- Tests verify both sync and async router error handling
+- Tests validate error log messages include router names for debugging
+- Integration tests demonstrate realistic plugin use cases (security enforcement, cost optimization, domain-specific routing)

package/test/integration/US-007-TEST-SUMMARY.md

@@ -0,0 +1,100 @@
+# US-007 Test Coverage Summary
+
+## Story
+Read plugin config entries from nax config.json
+
+## Acceptance Criteria Coverage
+
+### AC1: plugins[] from config.json are passed to loadPlugins() as configPlugins parameter
+**Tests:**
+- `test/integration/runner-config-plugins.test.ts`: "config.plugins[] entries are passed to loadPlugins() when runner initializes"
+  - Verifies config.plugins is loaded via loadConfig()
+  - Verifies config.plugins is passed to loadPlugins() with correct projectRoot
+  - Verifies plugin is initialized with correct config
+
+### AC2: Relative module paths in plugins[].module are resolved relative to project root
+**Tests:**
+- `test/integration/runner-config-plugins.test.ts`: "relative plugin paths in config.plugins[] are resolved relative to project root"
+  - Tests `./lib/plugins/plugin.ts` resolves correctly from project root
+  - Verifies plugin is loaded and initialized successfully
+
+### AC3: Absolute module paths and npm package names work as-is
+**Tests:**
+- `test/integration/runner-config-plugins.test.ts`: "absolute plugin paths in config.plugins[] work without project root resolution"
+  - Tests absolute paths are passed through without modification
+  - Verifies plugin is loaded correctly
+
+### AC4: If a plugin module cannot be found, a clear error message is logged with the path tried
+**Tests:**
+- `test/integration/runner-config-plugins.test.ts`: "missing plugin module from config.plugins[] logs clear error (does not crash runner)"
+  - Verifies error message contains original path
+  - Verifies error message contains attempted resolved path
+  - Verifies runner doesn't crash (returns empty registry)
+
+### AC5: Plugin-specific config (plugins[].config) is passed to the plugin's setup() function
+**Tests:**
+- `test/integration/runner-config-plugins.test.ts`: "config.plugins[] entries are passed to loadPlugins() when runner initializes"
+  - Verifies config object is passed to plugin's setup()
+  - Verifies config values match what was in config.json
+
+## Additional Test Coverage
+
+### Edge Cases
+1. **Empty plugins[] array**: Verifies no plugins loaded
+2. **Undefined plugins field**: Verifies runner's `config.plugins || []` fallback works
+3. **Plugin name collision**: Verifies config plugins override auto-discovered plugins
+
+## Test Files
+
+### New Test File
+- `test/integration/runner-config-plugins.test.ts` (7 tests, 32 assertions)
+  - Focuses on integration between loadConfig() and loadPlugins()
+  - Simulates runner.ts initialization flow
+  - All tests passing
+
+### Existing Test Files (Already Exist)
+- `test/integration/config-loader.test.ts` (8 tests for plugin config loading)
+  - Tests schema validation
+  - Tests config merging (global + project)
+- `test/integration/plugins/config-resolution.test.ts` (20+ tests)
+  - Comprehensive AC coverage at plugin loader level
+- `test/integration/plugins/config-integration.test.ts` (1 E2E test)
+  - Realistic scenario with relative paths
+
+## Test Strategy
+
+1. **Unit level**: Plugin loader path resolution logic (existing tests)
+2. **Integration level**: Config loader + plugin loader (NEW tests in runner-config-plugins.test.ts)
+3. **E2E level**: Full runner initialization flow (existing test in config-integration.test.ts)
+
+## Coverage Summary
+
+- ✅ All 5 acceptance criteria covered with multiple test cases
+- ✅ Edge cases covered (empty array, undefined, collisions)
+- ✅ Error handling verified (missing modules, invalid paths)
+- ✅ Integration verified (config → runner → plugins)
+
+## Running Tests
+
+```bash
+# Run new integration tests only
+bun test ./test/integration/runner-config-plugins.test.ts
+
+# Run all plugin-related tests
+bun test ./test/integration/plugins/
+
+# Run all config-related tests
+bun test ./test/integration/config-loader.test.ts
+```
+
+## Implementation Status
+
+**Status**: Tests written and passing ✅
+
+The implementation already exists in:
+- `src/config/loader.ts`: Loads plugins[] from config.json
+- `src/config/schema.ts`: Validates plugin config entries
+- `src/plugins/loader.ts`: Resolves paths and loads plugins
+- `src/execution/runner.ts:201-202`: Passes config.plugins to loadPlugins()
+
+These tests verify the integration works correctly end-to-end.

package/test/integration/agent-validation.test.ts

@@ -0,0 +1,439 @@
+import { describe, expect, mock, test } from "bun:test";
+import type { AgentAdapter, AgentRunOptions } from "../../src/agents";
+import { ClaudeCodeAdapter } from "../../src/agents/claude";
+import { describeAgentCapabilities, validateAgentFeature, validateAgentForTier } from "../../src/agents/validation";
+
+describe("Agent Validation and Retry Logic", () => {
+  describe("ClaudeCodeAdapter.isInstalled", () => {
+    test("returns true when binary exists in PATH", async () => {
+      const adapter = new ClaudeCodeAdapter();
+      // Mock successful which command
+      const originalSpawn = Bun.spawn;
+      (Bun as any).spawn = mock((cmd: string[]) => {
+        if (cmd[0] === "which" && cmd[1] === "claude") {
+          return {
+            exited: Promise.resolve(0),
+            stdout: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
+            stderr: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
+          };
+        }
+        return originalSpawn(cmd);
+      });
+
+      const installed = await adapter.isInstalled();
+      expect(installed).toBe(true);
+
+      Bun.spawn = originalSpawn;
+    });
+
+    test("returns false when binary does not exist", async () => {
+      const adapter = new ClaudeCodeAdapter();
+      // Mock failed which command
+      const originalSpawn = Bun.spawn;
+      (Bun as any).spawn = mock((cmd: string[]) => {
+        if (cmd[0] === "which" && cmd[1] === "claude") {
+          return {
+            exited: Promise.resolve(1),
+            stdout: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
+            stderr: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
+          };
+        }
+        return originalSpawn(cmd);
+      });
+
+      const installed = await adapter.isInstalled();
+      expect(installed).toBe(false);
+
+      Bun.spawn = originalSpawn;
+    });
+
+    test("returns false on exception", async () => {
+      const adapter = new ClaudeCodeAdapter();
+      const originalSpawn = Bun.spawn;
+      (Bun as any).spawn = mock(() => {
+        throw new Error("Command not found");
+      });
+
+      const installed = await adapter.isInstalled();
+      expect(installed).toBe(false);
+
+      Bun.spawn = originalSpawn;
+    });
+  });
+
+  describe("ClaudeCodeAdapter timeout handling", () => {
+    test("distinguishes timeout from normal failure", async () => {
+      const adapter = new ClaudeCodeAdapter();
+      const originalSpawn = Bun.spawn;
+
+      // Mock process that times out
+      (Bun as any).spawn = mock(() => {
+        let killed = false;
+        return {
+          exited: new Promise((resolve) => {
+            setTimeout(() => resolve(killed ? 143 : 0), 100);
+          }),
+          kill: (signal: string) => {
+            if (signal === "SIGTERM") killed = true;
+          },
+          stdout: new Response("").body,
+          stderr: new Response("").body,
+        };
+      });
+
+      const options: AgentRunOptions = {
+        prompt: "test",
+        workdir: "/tmp",
+        modelTier: "balanced",
+        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
+        timeoutSeconds: 0.05, // 50ms timeout
+      };
+
+      const result = await adapter.run(options);
+
+      // Should be marked as timeout (exit code 124)
+      expect(result.exitCode).toBe(124);
+      expect(result.success).toBe(false);
+
+      Bun.spawn = originalSpawn;
+    });
+  });
+
+  describe("ClaudeCodeAdapter retry logic", () => {
+    test(
+      "retries on rate limit with exponential backoff",
+      async () => {
+        const adapter = new ClaudeCodeAdapter();
+        const originalSpawn = Bun.spawn;
+        let attemptCount = 0;
+
+        // Mock rate-limited response that succeeds on 3rd try
+        (Bun as any).spawn = mock(() => {
+          attemptCount++;
+          const isRateLimited = attemptCount < 3;
+
+          return {
+            exited: Promise.resolve(isRateLimited ? 1 : 0),
+            kill: () => {},
+            stdout: new Response(isRateLimited ? "" : "success").body,
+            stderr: new Response(isRateLimited ? "rate limit exceeded" : "").body,
+          };
+        });
+
+        const options: AgentRunOptions = {
+          prompt: "test",
+          workdir: "/tmp",
+          modelTier: "balanced",
+          modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
+          timeoutSeconds: 60,
+        };
+
+        const startTime = Date.now();
+        const result = await adapter.run(options);
+        const duration = Date.now() - startTime;
+
+        // Should succeed after retries
+        expect(result.success).toBe(true);
+        expect(attemptCount).toBe(3);
+
+        // Should have backoff delays (2s + 4s = 6s, but we'll check for at least 3s)
+        // Note: In real implementation, backoff is 2^attempt * 1000 = 2s, 4s
+        expect(duration).toBeGreaterThanOrEqual(3000);
+
+        Bun.spawn = originalSpawn;
+      },
+      { timeout: 15000 },
+    );
+
+    test(
+      "fails immediately on agent execution errors (no retry)",
+      async () => {
+        const adapter = new ClaudeCodeAdapter();
+        const originalSpawn = Bun.spawn;
+        let attemptCount = 0;
+
+        // Mock agent execution failure (exit code 1)
+        // These are not retried because they're likely legitimate agent failures
+        (Bun as any).spawn = mock(() => {
+          attemptCount++;
+          return {
+            exited: Promise.resolve(1),
+            kill: () => {},
+            stdout: new Response("").body,
+            stderr: new Response("agent error").body,
+          };
+        });
+
+        const options: AgentRunOptions = {
+          prompt: "test",
+          workdir: "/tmp",
+          modelTier: "balanced",
+          modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
+          timeoutSeconds: 60,
+        };
+
+        const result = await adapter.run(options);
+
+        // Should fail after 1 attempt (no retry for agent errors)
+        expect(result.success).toBe(false);
+        expect(attemptCount).toBe(1);
+
+        Bun.spawn = originalSpawn;
+      },
+      { timeout: 15000 },
+    );
+
+    test("succeeds immediately on first attempt if no error", async () => {
+      const adapter = new ClaudeCodeAdapter();
+      const originalSpawn = Bun.spawn;
+      let attemptCount = 0;
+
+      // Mock successful execution
+      (Bun as any).spawn = mock(() => {
+        attemptCount++;
+        return {
+          exited: Promise.resolve(0),
+          kill: () => {},
+          stdout: new Response("success").body,
+          stderr: new Response("").body,
+        };
+      });
+
+      const options: AgentRunOptions = {
+        prompt: "test",
+        workdir: "/tmp",
+        modelTier: "balanced",
+        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
+        timeoutSeconds: 60,
+      };
+
+      const result = await adapter.run(options);
+
+      // Should succeed on first try
+      expect(result.success).toBe(true);
+      expect(attemptCount).toBe(1);
+
+      Bun.spawn = originalSpawn;
+    });
+
+    test("does not retry on timeout (exit code 124)", async () => {
+      const adapter = new ClaudeCodeAdapter();
+      const originalSpawn = Bun.spawn;
+      let attemptCount = 0;
+
+      // Mock timeout
+      (Bun as any).spawn = mock(() => {
+        attemptCount++;
+        let killed = false;
+        return {
+          exited: new Promise((resolve) => {
+            setTimeout(() => resolve(killed ? 143 : 0), 100);
+          }),
+          kill: (signal: string) => {
+            if (signal === "SIGTERM") killed = true;
+          },
+          stdout: new Response("").body,
+          stderr: new Response("").body,
+        };
+      });
+
+      const options: AgentRunOptions = {
+        prompt: "test",
+        workdir: "/tmp",
+        modelTier: "balanced",
+        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
+        timeoutSeconds: 0.05, // 50ms timeout
+      };
+
+      const result = await adapter.run(options);
+
+      // Should not retry on timeout
+      expect(result.exitCode).toBe(124);
+      expect(attemptCount).toBe(1);
+
+      Bun.spawn = originalSpawn;
+    });
+  });
+
+  describe("ClaudeCodeAdapter command building", () => {
+    test("builds correct command with model and prompt", () => {
+      const adapter = new ClaudeCodeAdapter();
+      const options: AgentRunOptions = {
+        prompt: "test prompt",
+        workdir: "/tmp",
+        modelTier: "balanced",
+        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
+        timeoutSeconds: 60,
+      };
+
+      const cmd = adapter.buildCommand(options);
+
+      expect(cmd).toEqual([
+        "claude",
+        "--model",
+        "claude-sonnet-4.5",
+        "--dangerously-skip-permissions",
+        "-p",
+        "test prompt",
+      ]);
+    });
+  });
+
+  describe("Agent Capability Metadata", () => {
+    const claudeAdapter = new ClaudeCodeAdapter();
+
+    describe("ClaudeCodeAdapter capabilities", () => {
+      test("declares all expected tiers", () => {
+        const caps = claudeAdapter.capabilities;
+        expect(caps.supportedTiers).toContain("fast");
+        expect(caps.supportedTiers).toContain("balanced");
+        expect(caps.supportedTiers).toContain("powerful");
+        expect(caps.supportedTiers.length).toBe(3);
+      });
+
+      test("declares all expected features", () => {
+        const caps = claudeAdapter.capabilities;
+        expect(caps.features.has("tdd")).toBe(true);
+        expect(caps.features.has("review")).toBe(true);
+        expect(caps.features.has("refactor")).toBe(true);
+        expect(caps.features.has("batch")).toBe(true);
+        expect(caps.features.size).toBe(4);
+      });
+
+      test("declares 200k token context window", () => {
+        expect(claudeAdapter.capabilities.maxContextTokens).toBe(200_000);
+      });
+    });
+
+    describe("validateAgentForTier", () => {
+      test("returns true for supported tiers", () => {
+        expect(validateAgentForTier(claudeAdapter, "fast")).toBe(true);
+        expect(validateAgentForTier(claudeAdapter, "balanced")).toBe(true);
+        expect(validateAgentForTier(claudeAdapter, "powerful")).toBe(true);
+      });
+
+      test("returns false for unsupported tiers (custom agent)", () => {
+        // Create a mock agent that only supports fast tier
+        const limitedAgent: AgentAdapter = {
+          name: "limited",
+          displayName: "Limited Agent",
+          binary: "limited",
+          capabilities: {
+            supportedTiers: ["fast"],
+            maxContextTokens: 50_000,
+            features: new Set(["review"]),
+          },
+          async isInstalled() {
+            return true;
+          },
+          async run() {
+            return {
+              success: true,
+              exitCode: 0,
+              output: "",
+              rateLimited: false,
+              durationMs: 1000,
+              estimatedCost: 0.01,
+            };
+          },
+          buildCommand() {
+            return ["limited"];
+          },
+        };
+
+        expect(validateAgentForTier(limitedAgent, "fast")).toBe(true);
+        expect(validateAgentForTier(limitedAgent, "balanced")).toBe(false);
+        expect(validateAgentForTier(limitedAgent, "powerful")).toBe(false);
+      });
+    });
+
+    describe("validateAgentFeature", () => {
+      test("returns true for supported features", () => {
+        expect(validateAgentFeature(claudeAdapter, "tdd")).toBe(true);
+        expect(validateAgentFeature(claudeAdapter, "review")).toBe(true);
+        expect(validateAgentFeature(claudeAdapter, "refactor")).toBe(true);
+        expect(validateAgentFeature(claudeAdapter, "batch")).toBe(true);
+      });
+
+      test("returns false for unsupported features (custom agent)", () => {
+        const reviewOnlyAgent: AgentAdapter = {
+          name: "reviewer",
+          displayName: "Review Agent",
+          binary: "reviewer",
+          capabilities: {
+            supportedTiers: ["fast", "balanced"],
+            maxContextTokens: 100_000,
+            features: new Set(["review"]),
+          },
+          async isInstalled() {
+            return true;
+          },
+          async run() {
+            return {
+              success: true,
+              exitCode: 0,
+              output: "",
+              rateLimited: false,
+              durationMs: 1000,
+              estimatedCost: 0.01,
+            };
+          },
+          buildCommand() {
+            return ["reviewer"];
+          },
+        };
+
+        expect(validateAgentFeature(reviewOnlyAgent, "review")).toBe(true);
+        expect(validateAgentFeature(reviewOnlyAgent, "tdd")).toBe(false);
+        expect(validateAgentFeature(reviewOnlyAgent, "refactor")).toBe(false);
+        expect(validateAgentFeature(reviewOnlyAgent, "batch")).toBe(false);
+      });
+    });
+
+    describe("describeAgentCapabilities", () => {
+      test("formats Claude Code capabilities correctly", () => {
+        const description = describeAgentCapabilities(claudeAdapter);
+        expect(description).toContain("claude:");
+        expect(description).toContain("tiers=[fast,balanced,powerful]");
+        expect(description).toContain("maxTokens=200000");
+        expect(description).toContain("features=");
+        expect(description).toContain("tdd");
+        expect(description).toContain("review");
+        expect(description).toContain("refactor");
+        expect(description).toContain("batch");
+      });
+
+      test("formats limited agent capabilities correctly", () => {
+        const limitedAgent: AgentAdapter = {
+          name: "tiny",
+          displayName: "Tiny Agent",
+          binary: "tiny",
+          capabilities: {
+            supportedTiers: ["fast"],
+            maxContextTokens: 10_000,
+            features: new Set(["review"]),
+          },
+          async isInstalled() {
+            return true;
+          },
+          async run() {
+            return {
+              success: true,
+              exitCode: 0,
+              output: "",
+              rateLimited: false,
+              durationMs: 1000,
+              estimatedCost: 0.01,
+            };
+          },
+          buildCommand() {
+            return ["tiny"];
+          },
+        };
+
+        const description = describeAgentCapabilities(limitedAgent);
+        expect(description).toBe("tiny: tiers=[fast], maxTokens=10000, features=[review]");
+      });
+    });
+  });
+});