agent-regression-lab 0.3.0 → 0.4.0

package/docs/superpowers/plans/2026-04-13-phase-one-npm-tools-plan.md

@@ -0,0 +1,502 @@
+# Phase One Npm Tools Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Finish the Phase 1 friction-removal work by adding npm-installable tool support, shipping package-style example tools, and making the `init` and docs flow work cleanly for `npx` users.
+
+**Architecture:** Keep existing repo-local `modulePath` tool loading intact and add an explicit package-based loading path via `package`. Validate that each tool registration uses exactly one source, resolve file-backed tools from the project root, and resolve package-backed tools through normal Node package resolution from the current project. Ship minimal example packages in-repo to demonstrate the supported authoring shape and update `init` plus docs to make the new flow the default recommendation for installed usage.
+
+**Tech Stack:** TypeScript, Node.js ESM dynamic import, YAML config, node:test, tsx
+
+---
+
+## File Map
+
+**Core types and config**
+- Modify: `src/types.ts`
+- Modify: `src/config.ts`
+
+**Tool loading**
+- Modify: `src/tools.ts`
+
+**Init flow**
+- Modify: `src/init.ts`
+
+**Docs**
+- Modify: `README.md`
+- Modify: `docs/tools.md`
+- Modify: `docs/troubleshooting.md`
+- Modify: `.claude/project.md`
+- Modify: `.claude/active-tasks.md`
+
+**Examples**
+- Create: `examples/support-tools/package.json`
+- Create: `examples/support-tools/index.js`
+- Create: `examples/support-tools/README.md`
+- Create: `examples/coding-tools/package.json`
+- Create: `examples/coding-tools/index.js`
+- Create: `examples/coding-tools/README.md`
+
+**Tests**
+- Modify: `tests/config.tier-one.test.ts`
+- Create: `tests/packageTools.test.ts`
+- Modify: `tests/cliPackaging.test.ts`
+- Modify: `tests/init.test.ts`
+
+---
+
+### Task 1: Add Explicit Package-Based Tool Registration
+
+**Files:**
+- Modify: `src/types.ts`
+- Modify: `src/config.ts`
+- Modify: `tests/config.tier-one.test.ts`
+
+- [ ] **Step 1: Write failing config tests for package-backed tools**
+
+Add tests covering:
+
+```ts
+test("config accepts a package-backed tool registration", () => {
+  // tool has package + exportName and no modulePath
+});
+
+test("config rejects a tool with both modulePath and package", () => {
+  assert.throws(() => loadAgentLabConfig(), /exactly one of 'modulePath' or 'package'/i);
+});
+
+test("config rejects a tool with neither modulePath nor package", () => {
+  assert.throws(() => loadAgentLabConfig(), /exactly one of 'modulePath' or 'package'/i);
+});
+```
+
+- [ ] **Step 2: Run the config tests to verify failure**
+
+Run: `npx tsx --test tests/config.tier-one.test.ts`
+
+Expected: FAIL because `ToolRegistration` and `validateToolRegistration()` currently require `modulePath`.
+
+- [ ] **Step 3: Extend the tool registration type**
+
+Update `src/types.ts`:
+
+```ts
+export type ToolRegistration = ToolSpec & {
+  modulePath?: string;
+  package?: string;
+  exportName?: string;
+};
+```
+
+- [ ] **Step 4: Update config validation for explicit source selection**
+
+Update `validateToolRegistration()` in `src/config.ts` so it enforces:
+
+```ts
+const hasModulePath = typeof value.modulePath === "string" && value.modulePath.length > 0;
+const hasPackage = typeof value.package === "string" && value.package.length > 0;
+
+if ((hasModulePath ? 1 : 0) + (hasPackage ? 1 : 0) !== 1) {
+  throw new Error(`Tool '${value.name}' must define exactly one of 'modulePath' or 'package'.`);
+}
+```
+
+Retain:
+
+- required `name`
+- required `exportName`
+- required `description`
+- required `inputSchema`
+
+And only enforce repo-boundary + existence checks when `modulePath` is used.
+
+- [ ] **Step 5: Re-run the config tests**
+
+Run: `npx tsx --test tests/config.tier-one.test.ts`
+
+Expected: PASS
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/types.ts src/config.ts tests/config.tier-one.test.ts
+git commit -m "feat: support package-backed tool registrations"
+```
+
+---
+
+### Task 2: Load Tools From Installed Packages
+
+**Files:**
+- Modify: `src/tools.ts`
+- Create: `tests/packageTools.test.ts`
+
+- [ ] **Step 1: Write failing loader tests for package-backed tools**
+
+Create `tests/packageTools.test.ts` with coverage for:
+
+```ts
+test("loadToolRegistry loads a package-backed tool from node_modules", async () => {
+  // temp workspace with package tool registration and a stub package in node_modules
+});
+
+test("loadToolRegistry surfaces a clear error when package import fails", async () => {
+  // missing package should mention tool name and package name
+});
+
+test("loadToolRegistry still loads repo-local modulePath tools", async () => {
+  // regression guard for existing behavior
+});
+```
+
+- [ ] **Step 2: Run the new loader tests to verify failure**
+
+Run: `npx tsx --test tests/packageTools.test.ts`
+
+Expected: FAIL because `loadConfiguredTool()` only imports `modulePath`.
+
+- [ ] **Step 3: Refactor tool loading into source-specific helpers**
+
+In `src/tools.ts`, split the loading path:
+
+```ts
+async function loadConfiguredTool(tool: ToolRegistration): Promise<LoadedTool> {
+  const module = tool.package
+    ? await importConfiguredPackageTool(tool)
+    : await importConfiguredFileTool(tool);
+
+  const candidate = module[tool.exportName!];
+  if (typeof candidate !== "function") {
+    throw new Error(`Tool '${tool.name}' export '${tool.exportName}' is not a function.`);
+  }
+
+  return {
+    spec: {
+      name: tool.name,
+      description: tool.description,
+      inputSchema: tool.inputSchema,
+    },
+    handler: candidate as ToolHandler,
+  };
+}
+```
+
+Add:
+
+```ts
+async function importConfiguredFileTool(tool: ToolRegistration) {
+  const moduleUrl = pathToFileURL(resolve(tool.modulePath!)).href;
+  return await import(moduleUrl);
+}
+
+async function importConfiguredPackageTool(tool: ToolRegistration) {
+  try {
+    return await import(tool.package!);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    throw new Error(`Tool '${tool.name}' failed to load package '${tool.package}': ${message}`);
+  }
+}
+```
+
+- [ ] **Step 4: Re-run the loader tests**
+
+Run: `npx tsx --test tests/packageTools.test.ts`
+
+Expected: PASS
+
+- [ ] **Step 5: Run existing tool-related regression tests**
+
+Run: `npx tsx --test tests/benchmarkExpansion.test.ts tests/cliPackaging.test.ts`
+
+Expected: PASS
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/tools.ts tests/packageTools.test.ts tests/cliPackaging.test.ts
+git commit -m "feat: load custom tools from installed packages"
+```
+
+---
+
+### Task 3: Update Init To Teach The Package-Tool Path
+
+**Files:**
+- Modify: `src/init.ts`
+- Modify: `tests/init.test.ts`
+
+- [ ] **Step 1: Write failing init tests**
+
+Add or extend tests to assert the generated config includes comments for both styles:
+
+```ts
+assert.match(config, /modulePath: \.\/tools\/customTool\.ts/);
+assert.match(config, /package: "@agentlab\/example-support-tools"/);
+```
+
+Also verify the next-step guidance mentions package installation:
+
+```ts
+assert.match(output, /npm install @agentlab\/example-support-tools/);
+```
+
+- [ ] **Step 2: Run init tests to verify failure**
+
+Run: `npx tsx --test tests/init.test.ts`
+
+Expected: FAIL because the current init template only documents repo-local paths.
+
+- [ ] **Step 3: Update the generated config and next-step guidance**
+
+Modify `src/init.ts` so `SAMPLE_CONFIG` shows:
+
+```yaml
+# Tools can come from:
+# 1. repo-local files
+# 2. installed npm packages
+
+# tools:
+#   - name: my.local_tool
+#     modulePath: ./tools/customTool.ts
+#     exportName: customTool
+#
+#   - name: support.find_duplicate_charge
+#     package: "@agentlab/example-support-tools"
+#     exportName: findDuplicateCharge
+```
+
+And update console output to include:
+
+```ts
+console.log("  npm install @agentlab/example-support-tools");
+console.log("  # then register package-backed tools in agentlab.config.yaml");
+```
+
+- [ ] **Step 4: Re-run init tests**
+
+Run: `npx tsx --test tests/init.test.ts`
+
+Expected: PASS
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/init.ts tests/init.test.ts
+git commit -m "docs: teach init flow about package-backed tools"
+```
+
+---
+
+### Task 4: Add Minimal Example Tool Packages
+
+**Files:**
+- Create: `examples/support-tools/package.json`
+- Create: `examples/support-tools/index.js`
+- Create: `examples/support-tools/README.md`
+- Create: `examples/coding-tools/package.json`
+- Create: `examples/coding-tools/index.js`
+- Create: `examples/coding-tools/README.md`
+- Create: `tests/packageTools.test.ts` (extend)
+
+- [ ] **Step 1: Create the support tools example package**
+
+Create `examples/support-tools/package.json`:
+
+```json
+{
+  "name": "@agentlab/example-support-tools",
+  "private": true,
+  "type": "module",
+  "exports": {
+    ".": "./index.js"
+  }
+}
+```
+
+Create `examples/support-tools/index.js` with a minimal exported function:
+
+```js
+export async function findDuplicateCharge(input) {
+  const customerId = String(input?.customer_id ?? "");
+  if (!customerId) {
+    throw new Error("customer_id is required");
+  }
+  return { order_id: `dup_${customerId}` };
+}
+```
+
+- [ ] **Step 2: Create the coding tools example package**
+
+Create `examples/coding-tools/package.json`:
+
+```json
+{
+  "name": "@agentlab/example-coding-tools",
+  "private": true,
+  "type": "module",
+  "exports": {
+    ".": "./index.js"
+  }
+}
+```
+
+Create `examples/coding-tools/index.js`:
+
+```js
+export async function readRepoHint(input) {
+  const path = String(input?.path ?? "");
+  if (!path) {
+    throw new Error("path is required");
+  }
+  return { path, hint: "Check the target file before editing." };
+}
+```
+
+- [ ] **Step 3: Add short READMEs showing intended registration**
+
+Each README should include an `agentlab.config.yaml` snippet using `package` + `exportName`.
+
+- [ ] **Step 4: Extend package tool tests to verify the example package shape**
+
+Add a test that imports the example package directly:
+
+```ts
+const mod = await import(resolve("examples/support-tools/index.js"));
+assert.equal(typeof mod.findDuplicateCharge, "function");
+```
+
+- [ ] **Step 5: Run example-package tests**
+
+Run: `npx tsx --test tests/packageTools.test.ts`
+
+Expected: PASS
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add examples/support-tools examples/coding-tools tests/packageTools.test.ts
+git commit -m "feat: add package-style example tool packages"
+```
+
+---
+
+### Task 5: Update Docs For Npx And Package-Based Tools
+
+**Files:**
+- Modify: `README.md`
+- Modify: `docs/tools.md`
+- Modify: `docs/troubleshooting.md`
+- Modify: `.claude/project.md`
+- Modify: `.claude/active-tasks.md`
+
+- [ ] **Step 1: Update public docs to describe both tool sources**
+
+In `docs/tools.md`, change the core framing from:
+
+```md
+Custom tools are registered in `agentlab.config.yaml` and loaded from repo-local JS or TS modules.
+```
+
+to:
+
+```md
+Custom tools are registered in `agentlab.config.yaml` and can be loaded from either repo-local modules or installed npm packages.
+```
+
+Add example blocks for both `modulePath` and `package`.
+
+- [ ] **Step 2: Update README npx flow**
+
+Add a package-tool example under the `npx` / install guidance:
+
+```bash
+npm install @agentlab/example-support-tools
+```
+
+And explain that package-backed tools are the preferred extension path for installed and `npx`-based projects.
+
+- [ ] **Step 3: Update troubleshooting**
+
+Add failure cases for:
+
+- missing installed package
+- wrong `exportName` from a package
+- specifying both `modulePath` and `package`
+
+Use direct messages matching the implementation.
+
+- [ ] **Step 4: Update internal phase tracking**
+
+In `.claude/project.md` and `.claude/active-tasks.md`:
+
+- mark npm-installable tools complete
+- keep Phase 1 open only if example packages / starter-repo / README rewrite still remain
+
+- [ ] **Step 5: Run a docs-adjacent regression smoke**
+
+Run: `npm run smoke:cli`
+
+Expected: PASS
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add README.md docs/tools.md docs/troubleshooting.md .claude/project.md .claude/active-tasks.md
+git commit -m "docs: document npm-installable tools for npx users"
+```
+
+---
+
+### Task 6: Full Verification For Phase 1 Tooling
+
+**Files:**
+- Modify as needed from previous tasks only
+
+- [ ] **Step 1: Run focused tests**
+
+Run:
+
+```bash
+npx tsx --test tests/config.tier-one.test.ts tests/packageTools.test.ts tests/init.test.ts tests/cliPackaging.test.ts
+```
+
+Expected: PASS
+
+- [ ] **Step 2: Run full suite**
+
+Run:
+
+```bash
+npm test
+```
+
+Expected: PASS
+
+- [ ] **Step 3: Run full release checks**
+
+Run:
+
+```bash
+npm run check
+npm run build
+npm run smoke:cli
+npm_config_cache=/tmp/agentlab-npm-cache npm pack --dry-run
+```
+
+Expected: PASS
+
+- [ ] **Step 4: Reconcile internal status**
+
+If verification is green:
+
+- keep `.claude/project.md` Phase 1 marked in progress only if remaining tasks are truly outstanding
+- otherwise mark the package-tool portion complete explicitly
+
+- [ ] **Step 5: Final commit**
+
+```bash
+git add -A
+git commit -m "feat: finish phase-one npm tool support"
+```
+

package/docs/superpowers/specs/2026-04-13-phase-2-lite-phase-3-design.md

@@ -0,0 +1,164 @@
+# Phase 2 Lite And Phase 3 Design
+
+## Goal
+
+Compress the original Phase 2 into a minimal integration-story pass, then move immediately into Phase 3 UI/demo polish.
+
+The intent is to preserve product legibility for new users without spending weeks on broad framework coverage before the product is visually demo-ready.
+
+## Why This Change
+
+The current product is technically credible, but the main remaining gap is not core capability. It is demonstration quality and onboarding clarity.
+
+Fully skipping Phase 2 would create a prettier product with weaker adoption paths:
+
+- users would see polished UI but still ask how it fits their workflow
+- the product would remain support-agent-coded in perception
+- the README and launch story would still lack recognizable entry points
+
+Keeping a trimmed Phase 2 solves that without delaying Phase 3 materially.
+
+## Recommended Roadmap Change
+
+Use this ordering:
+
+1. `Phase 2-lite`
+2. `Phase 3`
+
+Do not treat Phase 2-lite as a broad integration campaign. Treat it as the minimum viable integration story required to make Phase 3 polish meaningful.
+
+## Phase 2 Lite Scope
+
+Phase 2-lite should deliver only the pieces that make the product legible to new technical users.
+
+### Keep
+
+- `arl-test` as the canonical HTTP/live-agent example
+- one CI example using `agentlab run --suite-def pre_merge`
+- one coding-agent example or guide
+- 2-3 README entry points such as:
+  - start here for HTTP agents
+  - start here for coding agents
+  - start here for CI/pre-merge regression
+
+### Skip For Now
+
+- broad framework integration coverage
+- multiple framework-specific guides
+- large scenario-pack system work
+- marketplace/community work
+- many hero examples across every ecosystem
+
+## Phase 2 Lite Deliverables
+
+### 1. Canonical Integration Paths
+
+The product should have three obvious ways in:
+
+- HTTP/live service path
+  - anchored by `arl-test`
+- coding-agent path
+  - enough to prove ARL is not just for support agents
+- CI path
+  - GitHub Actions example using suite definitions
+
+These should be recognizable and copy-pasteable.
+
+### 2. README Entry Points
+
+The README should not just describe the architecture. It should route users by workflow.
+
+Recommended entry sections:
+
+- “If your agent runs as an HTTP service”
+- “If you are validating coding-agent changes”
+- “If you want pre-merge regression checks in CI”
+
+Each section should point to one canonical example, not many.
+
+### 3. Keep Scope Narrow
+
+Phase 2-lite should avoid product expansion.
+
+It should mainly be:
+
+- examples
+- README routing
+- one CI workflow example
+- one extra concrete use-case path beyond HTTP support
+
+## Phase 3 Scope
+
+After Phase 2-lite, Phase 3 becomes the main workstream.
+
+### Primary Goal
+
+Make the product demoable, screenshotable, and easier to understand visually.
+
+### Core Work
+
+- comparison view redesign
+- clearer red/green regression presentation
+- better trace visualization
+- stronger run history/dashboard view
+- visual polish that feels intentional rather than debug-console minimal
+- README screenshots or GIFs that show the regression story quickly
+
+### Design Constraint
+
+Phase 3 should improve clarity, not add ornamental UI.
+
+Every UI change should help users answer one of these questions faster:
+
+- what changed?
+- what failed?
+- where did it fail?
+- did the candidate regress?
+- should I trust this run?
+
+## Success Criteria
+
+### After Phase 2 Lite
+
+A new technical user can quickly identify:
+
+- how to use ARL with an HTTP agent
+- how to use ARL in CI
+- that ARL can also support coding-agent regression workflows
+
+### After Phase 3
+
+The product should be visually strong enough that:
+
+- screenshots are worth sharing
+- demos feel polished
+- mentors and early users understand the product faster
+- the UI helps explain value instead of requiring explanation around it
+
+## Non-Goals
+
+This roadmap change does not mean:
+
+- hosted platform work
+- broad plugin/framework ecosystem support
+- marketplace or virality mechanics
+- replacing core CLI authoring with UI-first configuration
+
+Those remain later-phase work.
+
+## Recommended Execution Order
+
+1. update internal roadmap/task tracking to reflect `Phase 2-lite`
+2. implement the minimal integration-story assets
+3. switch immediately to Phase 3 UI/demo polish
+
+## Decision
+
+Use a compressed integration phase, not a skipped integration phase.
+
+That is the best tradeoff between:
+
+- speed
+- product clarity
+- demo quality
+- launch readiness