@nathapp/nax 0.27.1 → 0.29.0

package/test/integration/US-005-TEST-SUMMARY.md

@@ -1,138 +0,0 @@
-# US-005: Plugin Routing Integration - Test Summary
-
-**Story:** Plugin routing strategies integrate into router chain
-
-**Test File:** `test/integration/plugin-routing.test.ts`
-
-**Status:** ✅ Tests written and failing (as expected)
-
----
-
-## Test Coverage
-
-### AC1: Plugin routers are tried before the built-in routing strategy
-
-**Tests (4):**
-- ✅ Plugin routers execute before built-in keyword strategy
-- ✅ Multiple plugin routers maintain load order
-- ✅ Plugin routers are inserted before manual strategy
-- ✅ Plugin routers are inserted before llm strategy
-
-**Coverage:** Validates that plugin routers are prepended to the strategy chain and execute in load order before any built-in strategies.
-
----
-
-### AC2: First plugin router that returns a non-null result wins
-
-**Tests (5):**
-- ✅ First plugin router decision is used
-- ✅ Second plugin router is used when first returns null
-- ✅ Plugin router overrides built-in keyword strategy
-- ✅ Third plugin router is used when first two return null
-- ✅ Plugin router can delegate based on conditional logic (integration test)
-
-**Coverage:** Validates the chain precedence rules where the first non-null decision wins, and tests realistic scenarios where plugins selectively handle certain stories.
-
----
-
-### AC3: If all plugin routers return null, built-in strategy is used as fallback
-
-**Tests (4):**
-- ✅ Keyword strategy is used when all plugin routers return null
-- ✅ Keyword strategy handles complex story when plugins return null
-- ✅ Manual strategy is used as fallback when plugins return null
-- ✅ Empty plugin registry falls back to keyword strategy
-
-**Coverage:** Validates that the chain properly falls through to built-in strategies (keyword, manual) when all plugin routers return null or when no plugins are loaded.
-
----
-
-### AC4: Plugin routers receive the same story context as built-in routers
-
-**Tests (5):**
-- ✅ Plugin router receives story object
-- ✅ Plugin router receives routing context with config
-- ✅ Plugin router receives codebase context when available
-- ✅ Plugin router receives metrics when available
-- ✅ Multiple plugin routers receive same context
-
-**Coverage:** Validates that plugin routers receive the complete `UserStory` and `RoutingContext` objects, including optional fields like `codebaseContext` and `metrics`.
-
----
-
-### AC5: Router errors are caught and logged; fallback to next router in chain
-
-**Tests (6):**
-- ❌ Error in plugin router is caught and next router is tried
-- ❌ Error in plugin router is logged
-- ❌ Multiple router errors are caught and keyword fallback succeeds
-- ❌ Async error in plugin router is caught
-- ❌ Error in last plugin router falls back to keyword strategy
-- ❌ Error message includes plugin name for debugging
-
-**Coverage:** Validates that errors thrown by plugin routers don't crash the routing system and that proper error logging occurs with router names for debugging.
-
-**Status:** Currently failing (expected) - error handling not yet implemented in `StrategyChain.route()`
-
----
-
-## Integration Tests
-
-**Real-world scenarios (6):**
-- ✅ Premium plugin forces security stories to expert tier
-- ✅ Cost-optimization plugin downgrades simple docs to fast tier
-- ✅ Domain-specific plugin routes database migrations to expert tier
-- ✅ Multiple plugins: first matching plugin wins
-- ✅ Plugin router can delegate based on conditional logic (duplicate coverage)
-
-**Coverage:** Tests realistic plugin use cases that demonstrate the value of the plugin router system.
-
----
-
-## Test Results
-
-```
- 22 pass
- 6 fail
- 54 expect() calls
-Ran 28 tests across 1 file.
-```
-
-**Passing Tests (22):** Plugin router chain integration, precedence, fallback, and context passing all work correctly with the current implementation.
-
-**Failing Tests (6):** All failures are in AC5 (error handling). The errors are propagating instead of being caught, logged, and triggering fallback to the next router.
-
----
-
-## Implementation Gaps
-
-The tests reveal that the following needs to be implemented:
-
-1. **Error handling in `StrategyChain.route()`** (`src/routing/chain.ts:38-44`)
-   - Wrap each `strategy.route()` call in try-catch
-   - Log errors with strategy name
-   - Continue to next strategy on error
-
-2. **Error logging with plugin context**
-   - Include plugin router name in error logs
-   - Use `getSafeLogger()` to log routing errors
-   - Log at error level with category "routing"
-
----
-
-## Next Steps for Implementer
-
-1. Modify `src/routing/chain.ts` to wrap `strategy.route()` in try-catch
-2. Add error logging that includes strategy name
-3. Continue chain iteration on error (same as null return)
-4. Run tests: `bun test ./test/integration/plugin-routing.test.ts`
-5. All 28 tests should pass after implementation
-
----
-
-## Notes
-
-- Tests use mock plugins with custom routing logic
-- Tests verify both sync and async router error handling
-- Tests validate error log messages include router names for debugging
-- Integration tests demonstrate realistic plugin use cases (security enforcement, cost optimization, domain-specific routing)

package/test/integration/US-007-TEST-SUMMARY.md

@@ -1,100 +0,0 @@
-# US-007 Test Coverage Summary
-
-## Story
-Read plugin config entries from nax config.json
-
-## Acceptance Criteria Coverage
-
-### AC1: plugins[] from config.json are passed to loadPlugins() as configPlugins parameter
-**Tests:**
-- `test/integration/runner-config-plugins.test.ts`: "config.plugins[] entries are passed to loadPlugins() when runner initializes"
-  - Verifies config.plugins is loaded via loadConfig()
-  - Verifies config.plugins is passed to loadPlugins() with correct projectRoot
-  - Verifies plugin is initialized with correct config
-
-### AC2: Relative module paths in plugins[].module are resolved relative to project root
-**Tests:**
-- `test/integration/runner-config-plugins.test.ts`: "relative plugin paths in config.plugins[] are resolved relative to project root"
-  - Tests `./lib/plugins/plugin.ts` resolves correctly from project root
-  - Verifies plugin is loaded and initialized successfully
-
-### AC3: Absolute module paths and npm package names work as-is
-**Tests:**
-- `test/integration/runner-config-plugins.test.ts`: "absolute plugin paths in config.plugins[] work without project root resolution"
-  - Tests absolute paths are passed through without modification
-  - Verifies plugin is loaded correctly
-
-### AC4: If a plugin module cannot be found, a clear error message is logged with the path tried
-**Tests:**
-- `test/integration/runner-config-plugins.test.ts`: "missing plugin module from config.plugins[] logs clear error (does not crash runner)"
-  - Verifies error message contains original path
-  - Verifies error message contains attempted resolved path
-  - Verifies runner doesn't crash (returns empty registry)
-
-### AC5: Plugin-specific config (plugins[].config) is passed to the plugin's setup() function
-**Tests:**
-- `test/integration/runner-config-plugins.test.ts`: "config.plugins[] entries are passed to loadPlugins() when runner initializes"
-  - Verifies config object is passed to plugin's setup()
-  - Verifies config values match what was in config.json
-
-## Additional Test Coverage
-
-### Edge Cases
-1. **Empty plugins[] array**: Verifies no plugins loaded
-2. **Undefined plugins field**: Verifies runner's `config.plugins || []` fallback works
-3. **Plugin name collision**: Verifies config plugins override auto-discovered plugins
-
-## Test Files
-
-### New Test File
-- `test/integration/runner-config-plugins.test.ts` (7 tests, 32 assertions)
-  - Focuses on integration between loadConfig() and loadPlugins()
-  - Simulates runner.ts initialization flow
-  - All tests passing
-
-### Existing Test Files (Already Exist)
-- `test/integration/config-loader.test.ts` (8 tests for plugin config loading)
-  - Tests schema validation
-  - Tests config merging (global + project)
-- `test/integration/plugins/config-resolution.test.ts` (20+ tests)
-  - Comprehensive AC coverage at plugin loader level
-- `test/integration/plugins/config-integration.test.ts` (1 E2E test)
-  - Realistic scenario with relative paths
-
-## Test Strategy
-
-1. **Unit level**: Plugin loader path resolution logic (existing tests)
-2. **Integration level**: Config loader + plugin loader (NEW tests in runner-config-plugins.test.ts)
-3. **E2E level**: Full runner initialization flow (existing test in config-integration.test.ts)
-
-## Coverage Summary
-
-- ✅ All 5 acceptance criteria covered with multiple test cases
-- ✅ Edge cases covered (empty array, undefined, collisions)
-- ✅ Error handling verified (missing modules, invalid paths)
-- ✅ Integration verified (config → runner → plugins)
-
-## Running Tests
-
-```bash
-# Run new integration tests only
-bun test ./test/integration/runner-config-plugins.test.ts
-
-# Run all plugin-related tests
-bun test ./test/integration/plugins/
-
-# Run all config-related tests
-bun test ./test/integration/config-loader.test.ts
-```
-
-## Implementation Status
-
-**Status**: Tests written and passing ✅
-
-The implementation already exists in:
-- `src/config/loader.ts`: Loads plugins[] from config.json
-- `src/config/schema.ts`: Validates plugin config entries
-- `src/plugins/loader.ts`: Resolves paths and loads plugins
-- `src/execution/runner.ts:201-202`: Passes config.plugins to loadPlugins()
-
-These tests verify the integration works correctly end-to-end.

package/test/integration/cli/agent-validation.test.ts

@@ -1,439 +0,0 @@
-import { describe, expect, mock, test } from "bun:test";
-import type { AgentAdapter, AgentRunOptions } from "../../../src/agents";
-import { ClaudeCodeAdapter } from "../../../src/agents/claude";
-import { describeAgentCapabilities, validateAgentFeature, validateAgentForTier } from "../../../src/agents/validation";
-
-describe("Agent Validation and Retry Logic", () => {
-  describe("ClaudeCodeAdapter.isInstalled", () => {
-    test("returns true when binary exists in PATH", async () => {
-      const adapter = new ClaudeCodeAdapter();
-      // Mock successful which command
-      const originalSpawn = Bun.spawn;
-      (Bun as any).spawn = mock((cmd: string[]) => {
-        if (cmd[0] === "which" && cmd[1] === "claude") {
-          return {
-            exited: Promise.resolve(0),
-            stdout: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
-            stderr: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
-          };
-        }
-        return originalSpawn(cmd);
-      });
-
-      const installed = await adapter.isInstalled();
-      expect(installed).toBe(true);
-
-      Bun.spawn = originalSpawn;
-    });
-
-    test("returns false when binary does not exist", async () => {
-      const adapter = new ClaudeCodeAdapter();
-      // Mock failed which command
-      const originalSpawn = Bun.spawn;
-      (Bun as any).spawn = mock((cmd: string[]) => {
-        if (cmd[0] === "which" && cmd[1] === "claude") {
-          return {
-            exited: Promise.resolve(1),
-            stdout: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
-            stderr: { getReader: () => ({ read: () => Promise.resolve({ done: true }) }) },
-          };
-        }
-        return originalSpawn(cmd);
-      });
-
-      const installed = await adapter.isInstalled();
-      expect(installed).toBe(false);
-
-      Bun.spawn = originalSpawn;
-    });
-
-    test("returns false on exception", async () => {
-      const adapter = new ClaudeCodeAdapter();
-      const originalSpawn = Bun.spawn;
-      (Bun as any).spawn = mock(() => {
-        throw new Error("Command not found");
-      });
-
-      const installed = await adapter.isInstalled();
-      expect(installed).toBe(false);
-
-      Bun.spawn = originalSpawn;
-    });
-  });
-
-  describe("ClaudeCodeAdapter timeout handling", () => {
-    test("distinguishes timeout from normal failure", async () => {
-      const adapter = new ClaudeCodeAdapter();
-      const originalSpawn = Bun.spawn;
-
-      // Mock process that times out
-      (Bun as any).spawn = mock(() => {
-        let killed = false;
-        return {
-          exited: new Promise((resolve) => {
-            setTimeout(() => resolve(killed ? 143 : 0), 100);
-          }),
-          kill: (signal: string) => {
-            if (signal === "SIGTERM") killed = true;
-          },
-          stdout: new Response("").body,
-          stderr: new Response("").body,
-        };
-      });
-
-      const options: AgentRunOptions = {
-        prompt: "test",
-        workdir: "/tmp",
-        modelTier: "balanced",
-        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
-        timeoutSeconds: 0.05, // 50ms timeout
-      };
-
-      const result = await adapter.run(options);
-
-      // Should be marked as timeout (exit code 124)
-      expect(result.exitCode).toBe(124);
-      expect(result.success).toBe(false);
-
-      Bun.spawn = originalSpawn;
-    });
-  });
-
-  describe("ClaudeCodeAdapter retry logic", () => {
-    test(
-      "retries on rate limit with exponential backoff",
-      async () => {
-        const adapter = new ClaudeCodeAdapter();
-        const originalSpawn = Bun.spawn;
-        let attemptCount = 0;
-
-        // Mock rate-limited response that succeeds on 3rd try
-        (Bun as any).spawn = mock(() => {
-          attemptCount++;
-          const isRateLimited = attemptCount < 3;
-
-          return {
-            exited: Promise.resolve(isRateLimited ? 1 : 0),
-            kill: () => {},
-            stdout: new Response(isRateLimited ? "" : "success").body,
-            stderr: new Response(isRateLimited ? "rate limit exceeded" : "").body,
-          };
-        });
-
-        const options: AgentRunOptions = {
-          prompt: "test",
-          workdir: "/tmp",
-          modelTier: "balanced",
-          modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
-          timeoutSeconds: 60,
-        };
-
-        const startTime = Date.now();
-        const result = await adapter.run(options);
-        const duration = Date.now() - startTime;
-
-        // Should succeed after retries
-        expect(result.success).toBe(true);
-        expect(attemptCount).toBe(3);
-
-        // Should have backoff delays (2s + 4s = 6s, but we'll check for at least 3s)
-        // Note: In real implementation, backoff is 2^attempt * 1000 = 2s, 4s
-        expect(duration).toBeGreaterThanOrEqual(3000);
-
-        Bun.spawn = originalSpawn;
-      },
-      { timeout: 15000 },
-    );
-
-    test(
-      "fails immediately on agent execution errors (no retry)",
-      async () => {
-        const adapter = new ClaudeCodeAdapter();
-        const originalSpawn = Bun.spawn;
-        let attemptCount = 0;
-
-        // Mock agent execution failure (exit code 1)
-        // These are not retried because they're likely legitimate agent failures
-        (Bun as any).spawn = mock(() => {
-          attemptCount++;
-          return {
-            exited: Promise.resolve(1),
-            kill: () => {},
-            stdout: new Response("").body,
-            stderr: new Response("agent error").body,
-          };
-        });
-
-        const options: AgentRunOptions = {
-          prompt: "test",
-          workdir: "/tmp",
-          modelTier: "balanced",
-          modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
-          timeoutSeconds: 60,
-        };
-
-        const result = await adapter.run(options);
-
-        // Should fail after 1 attempt (no retry for agent errors)
-        expect(result.success).toBe(false);
-        expect(attemptCount).toBe(1);
-
-        Bun.spawn = originalSpawn;
-      },
-      { timeout: 15000 },
-    );
-
-    test("succeeds immediately on first attempt if no error", async () => {
-      const adapter = new ClaudeCodeAdapter();
-      const originalSpawn = Bun.spawn;
-      let attemptCount = 0;
-
-      // Mock successful execution
-      (Bun as any).spawn = mock(() => {
-        attemptCount++;
-        return {
-          exited: Promise.resolve(0),
-          kill: () => {},
-          stdout: new Response("success").body,
-          stderr: new Response("").body,
-        };
-      });
-
-      const options: AgentRunOptions = {
-        prompt: "test",
-        workdir: "/tmp",
-        modelTier: "balanced",
-        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
-        timeoutSeconds: 60,
-      };
-
-      const result = await adapter.run(options);
-
-      // Should succeed on first try
-      expect(result.success).toBe(true);
-      expect(attemptCount).toBe(1);
-
-      Bun.spawn = originalSpawn;
-    });
-
-    test("does not retry on timeout (exit code 124)", async () => {
-      const adapter = new ClaudeCodeAdapter();
-      const originalSpawn = Bun.spawn;
-      let attemptCount = 0;
-
-      // Mock timeout
-      (Bun as any).spawn = mock(() => {
-        attemptCount++;
-        let killed = false;
-        return {
-          exited: new Promise((resolve) => {
-            setTimeout(() => resolve(killed ? 143 : 0), 100);
-          }),
-          kill: (signal: string) => {
-            if (signal === "SIGTERM") killed = true;
-          },
-          stdout: new Response("").body,
-          stderr: new Response("").body,
-        };
-      });
-
-      const options: AgentRunOptions = {
-        prompt: "test",
-        workdir: "/tmp",
-        modelTier: "balanced",
-        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
-        timeoutSeconds: 0.05, // 50ms timeout
-      };
-
-      const result = await adapter.run(options);
-
-      // Should not retry on timeout
-      expect(result.exitCode).toBe(124);
-      expect(attemptCount).toBe(1);
-
-      Bun.spawn = originalSpawn;
-    });
-  });
-
-  describe("ClaudeCodeAdapter command building", () => {
-    test("builds correct command with model and prompt", () => {
-      const adapter = new ClaudeCodeAdapter();
-      const options: AgentRunOptions = {
-        prompt: "test prompt",
-        workdir: "/tmp",
-        modelTier: "balanced",
-        modelDef: { provider: "anthropic", model: "claude-sonnet-4.5", env: {} },
-        timeoutSeconds: 60,
-      };
-
-      const cmd = adapter.buildCommand(options);
-
-      expect(cmd).toEqual([
-        "claude",
-        "--model",
-        "claude-sonnet-4.5",
-        "--dangerously-skip-permissions",
-        "-p",
-        "test prompt",
-      ]);
-    });
-  });
-
-  describe("Agent Capability Metadata", () => {
-    const claudeAdapter = new ClaudeCodeAdapter();
-
-    describe("ClaudeCodeAdapter capabilities", () => {
-      test("declares all expected tiers", () => {
-        const caps = claudeAdapter.capabilities;
-        expect(caps.supportedTiers).toContain("fast");
-        expect(caps.supportedTiers).toContain("balanced");
-        expect(caps.supportedTiers).toContain("powerful");
-        expect(caps.supportedTiers.length).toBe(3);
-      });
-
-      test("declares all expected features", () => {
-        const caps = claudeAdapter.capabilities;
-        expect(caps.features.has("tdd")).toBe(true);
-        expect(caps.features.has("review")).toBe(true);
-        expect(caps.features.has("refactor")).toBe(true);
-        expect(caps.features.has("batch")).toBe(true);
-        expect(caps.features.size).toBe(4);
-      });
-
-      test("declares 200k token context window", () => {
-        expect(claudeAdapter.capabilities.maxContextTokens).toBe(200_000);
-      });
-    });
-
-    describe("validateAgentForTier", () => {
-      test("returns true for supported tiers", () => {
-        expect(validateAgentForTier(claudeAdapter, "fast")).toBe(true);
-        expect(validateAgentForTier(claudeAdapter, "balanced")).toBe(true);
-        expect(validateAgentForTier(claudeAdapter, "powerful")).toBe(true);
-      });
-
-      test("returns false for unsupported tiers (custom agent)", () => {
-        // Create a mock agent that only supports fast tier
-        const limitedAgent: AgentAdapter = {
-          name: "limited",
-          displayName: "Limited Agent",
-          binary: "limited",
-          capabilities: {
-            supportedTiers: ["fast"],
-            maxContextTokens: 50_000,
-            features: new Set(["review"]),
-          },
-          async isInstalled() {
-            return true;
-          },
-          async run() {
-            return {
-              success: true,
-              exitCode: 0,
-              output: "",
-              rateLimited: false,
-              durationMs: 1000,
-              estimatedCost: 0.01,
-            };
-          },
-          buildCommand() {
-            return ["limited"];
-          },
-        };
-
-        expect(validateAgentForTier(limitedAgent, "fast")).toBe(true);
-        expect(validateAgentForTier(limitedAgent, "balanced")).toBe(false);
-        expect(validateAgentForTier(limitedAgent, "powerful")).toBe(false);
-      });
-    });
-
-    describe("validateAgentFeature", () => {
-      test("returns true for supported features", () => {
-        expect(validateAgentFeature(claudeAdapter, "tdd")).toBe(true);
-        expect(validateAgentFeature(claudeAdapter, "review")).toBe(true);
-        expect(validateAgentFeature(claudeAdapter, "refactor")).toBe(true);
-        expect(validateAgentFeature(claudeAdapter, "batch")).toBe(true);
-      });
-
-      test("returns false for unsupported features (custom agent)", () => {
-        const reviewOnlyAgent: AgentAdapter = {
-          name: "reviewer",
-          displayName: "Review Agent",
-          binary: "reviewer",
-          capabilities: {
-            supportedTiers: ["fast", "balanced"],
-            maxContextTokens: 100_000,
-            features: new Set(["review"]),
-          },
-          async isInstalled() {
-            return true;
-          },
-          async run() {
-            return {
-              success: true,
-              exitCode: 0,
-              output: "",
-              rateLimited: false,
-              durationMs: 1000,
-              estimatedCost: 0.01,
-            };
-          },
-          buildCommand() {
-            return ["reviewer"];
-          },
-        };
-
-        expect(validateAgentFeature(reviewOnlyAgent, "review")).toBe(true);
-        expect(validateAgentFeature(reviewOnlyAgent, "tdd")).toBe(false);
-        expect(validateAgentFeature(reviewOnlyAgent, "refactor")).toBe(false);
-        expect(validateAgentFeature(reviewOnlyAgent, "batch")).toBe(false);
-      });
-    });
-
-    describe("describeAgentCapabilities", () => {
-      test("formats Claude Code capabilities correctly", () => {
-        const description = describeAgentCapabilities(claudeAdapter);
-        expect(description).toContain("claude:");
-        expect(description).toContain("tiers=[fast,balanced,powerful]");
-        expect(description).toContain("maxTokens=200000");
-        expect(description).toContain("features=");
-        expect(description).toContain("tdd");
-        expect(description).toContain("review");
-        expect(description).toContain("refactor");
-        expect(description).toContain("batch");
-      });
-
-      test("formats limited agent capabilities correctly", () => {
-        const limitedAgent: AgentAdapter = {
-          name: "tiny",
-          displayName: "Tiny Agent",
-          binary: "tiny",
-          capabilities: {
-            supportedTiers: ["fast"],
-            maxContextTokens: 10_000,
-            features: new Set(["review"]),
-          },
-          async isInstalled() {
-            return true;
-          },
-          async run() {
-            return {
-              success: true,
-              exitCode: 0,
-              output: "",
-              rateLimited: false,
-              durationMs: 1000,
-              estimatedCost: 0.01,
-            };
-          },
-          buildCommand() {
-            return ["tiny"];
-          },
-        };
-
-        const description = describeAgentCapabilities(limitedAgent);
-        expect(description).toBe("tiny: tiers=[fast], maxTokens=10000, features=[review]");
-      });
-    });
-  });
-});