agent-regression-lab 0.2.0 → 0.4.0

package/docs/tools.md

@@ -1,6 +1,6 @@
 # Custom Tools
 
-Custom tools are registered in `agentlab.config.yaml` and loaded from repo-local JS or TS modules.
+Custom tools are registered in `agentlab.config.yaml` and can be loaded from repo-local JS/TS modules or installed npm packages.
 
 This is the main extension point when built-in tools are not enough.
 
@@ -9,12 +9,14 @@ This is the main extension point when built-in tools are not enough.
 Each tool entry must define:
 
 - `name`
-- `modulePath`
+- exactly one source:
+  - `modulePath`, or
+  - `package`
 - `exportName`
 - `description`
 - `inputSchema`
 
-Example:
+Repo-local example:
 
 ```yaml
 tools:
@@ -33,6 +35,25 @@ tools:
         - customer_id
 ```
 
+Installed package example:
+
+```yaml
+tools:
+  - name: support.find_duplicate_charge
+    package: "@agentlab/example-support-tools"
+    exportName: findDuplicateCharge
+    description: Find the duplicated charge order id for a given customer.
+    inputSchema:
+      type: object
+      additionalProperties: false
+      properties:
+        customer_id:
+          type: string
+          description: Customer id to inspect for duplicated charges.
+      required:
+        - customer_id
+```
+
 ## Tool Module Shape
 
 The exported function should be async and should return JSON-serializable output.
@@ -48,11 +69,15 @@ export async function myTool(input: unknown): Promise<{ ok: boolean }> {
 The existing working example is:
 
 - `user_tools/findDuplicateCharge.ts`
+- `examples/support-tools`
+- `examples/coding-tools`
 
 ## Important Constraints
 
+- each tool must define exactly one of `modulePath` or `package`
 - `modulePath` must stay within the repo
 - the module must exist at load time
+- installed packages must be resolvable from the current project
 - the named export must exist
 - tool input should be validated defensively inside the tool
 - tool output should be deterministic and JSON-serializable
@@ -100,3 +125,9 @@ Typical config failures:
 - invalid `inputSchema` shape
 
 See [troubleshooting.md](troubleshooting.md) for failure examples and fixes.
+
+For installed-package workflows, a good local path is:
+
+```bash
+npm install @agentlab/example-support-tools
+```

package/docs/troubleshooting.md

@@ -25,6 +25,8 @@ Or skip linking and use:
 npm run start -- --help
 ```
 
+---
+
 ## `OPENAI_API_KEY is required`
 
 You used an OpenAI-backed agent without exporting the API key.
@@ -36,6 +38,8 @@ export OPENAI_API_KEY=...
 agentlab run support.refund-correct-order --agent openai-cheap
 ```
 
+---
+
 ## `No scenarios found for suite ...`
 
 The suite id must match a suite under `scenarios/`.
@@ -52,6 +56,9 @@ Current built-in suites in this repo include:
 - `coding`
 - `research`
 - `ops`
+- `internal-teams`
+
+---
 
 ## `Run '<id>' not found`
 
@@ -70,6 +77,8 @@ agentlab show <run-id>
 agentlab compare <baseline-run-id> <candidate-run-id>
 ```
 
+---
+
 ## `Missing baseline or candidate suite batch id`
 
 `compare --suite` does not use run ids. It uses suite batch ids printed by `run --suite`.
@@ -82,6 +91,8 @@ agentlab run --suite support --agent mock-default
 agentlab compare --suite <baseline-batch-id> <candidate-batch-id>
 ```
 
+---
+
 ## Cross-suite suite comparison errors
 
 Suite batch comparison is strict. Compare batches from the same suite only.
@@ -99,6 +110,8 @@ This is not valid:
 
 If you are unsure which batch came from which suite, rerun the suite and record the printed batch ids.
 
+---
+
 ## `agentlab ui` fails to load assets
 
 Installed packages should already include the built UI assets.
@@ -116,6 +129,8 @@ If the problem persists, verify that these files exist:
 - `dist/ui-assets/client.js`
 - `dist/ui-assets/client.css`
 
+---
+
 ## Config tool or agent not found
 
 Typical reasons:
@@ -123,14 +138,190 @@ Typical reasons:
 - `agentlab.config.yaml` is missing
 - the configured `name` does not match the CLI `--agent` value
 - `modulePath` points outside the repo
+- both `modulePath` and `package` were provided for the same tool
+- the configured npm package is not installed
 - the configured export or command does not exist
 
 Working references in this repo:
 
 - tool config: `agentlab.config.yaml`
 - custom tool: `user_tools/findDuplicateCharge.ts`
+- package-style tools: `examples/support-tools`, `examples/coding-tools`
 - external agents: `custom_agents/node_agent.mjs`, `custom_agents/python_agent.py`
 
+### `Tool '<name>' must define exactly one of 'modulePath' or 'package'`
+
+Your tool registration is ambiguous or incomplete.
+
+Valid:
+
+```yaml
+tools:
+  - name: support.find_duplicate_charge
+    modulePath: ./user_tools/findDuplicateCharge.ts
+    exportName: findDuplicateCharge
+```
+
+Also valid:
+
+```yaml
+tools:
+  - name: support.find_duplicate_charge
+    package: "@agentlab/example-support-tools"
+    exportName: findDuplicateCharge
+```
+
+Invalid:
+
+- setting both `modulePath` and `package`
+- setting neither of them
+
+### `Tool '<name>' failed to load package '<pkg>'`
+
+The package-backed tool could not be resolved from the current project.
+
+Check:
+
+- the package is installed in the current project
+- the package name is correct
+- the package exports the named function you configured
+
+Typical fix:
+
+```bash
+npm install @agentlab/example-support-tools
+```
+
+### `Tool '<name>' export '<export>' is not a function`
+
+The module loaded successfully, but the named export does not exist or is not callable.
+
+Check:
+
+- `exportName` matches the actual exported function name
+- the package or local module uses ESM exports as expected
+
+---
+
+## HTTP agent errors
+
+### `HTTP agents require a configured url`
+
+You ran a conversation scenario with `--provider http` but no HTTP agent config was found.
+
+Fix: define a named http agent in `agentlab.config.yaml`:
+
+```yaml
+agents:
+  - name: my-agent
+    provider: http
+    url: http://localhost:3000/api/chat
+```
+
+Then run with:
+
+```bash
+agentlab run internal-teams.memory-followup-recall --agent my-agent
+```
+
+### `termination_reason: http_connection_failed`
+
+agentlab could not connect to your agent's URL. The most common cause is that the agent service is not running.
+
+Check:
+
+- is the service running on the configured port?
+- is the URL in `agentlab.config.yaml` correct?
+- is there a firewall or proxy blocking the connection?
+
+### `termination_reason: http_error`
+
+Your agent returned an HTTP 4xx or 5xx response.
+
+Check:
+
+- is the route path correct?
+- does your agent expect a different request shape? Use `request_template` if so.
+- are there auth errors? Check `headers` config.
+
+### `termination_reason: timeout_exceeded`
+
+Your agent did not respond within `timeout_ms` (default 30 seconds).
+
+Fix options:
+
+- increase `timeout_ms` in the agent config
+- investigate why the agent is slow for the given input
+
+### `termination_reason: invalid_response_format`
+
+Your agent either returned non-JSON or did not include the expected field.
+
+Defaults: agentlab reads the `message` field from the JSON response. Override with `response_field` if your agent uses a different name:
+
+```yaml
+agents:
+  - name: my-agent
+    provider: http
+    url: http://localhost:3000/api/chat
+    response_field: reply
+```
+
+---
+
+## `database is locked`
+
+You hit SQLite write contention on the local artifacts DB.
+
+Most common cause:
+
+- multiple `agentlab` runs writing to the same `artifacts/agentlab.db` at the same time
+
+Fix:
+
+- wait for the current run to finish
+- rerun sequentially instead of in parallel
+- keep live HTTP fixture verification serialized when using the same local project directory
+
+The product now uses a busy timeout, but sequential execution is still the safest path for local live verification.
+
+---
+
+## Conversation scenario errors
+
+### `Scenario '...' is a conversation scenario and requires provider: http`
+
+You tried to run a `type: conversation` scenario with a non-HTTP agent (`mock`, `openai`, or `external_process`).
+
+Conversation scenarios only work with `provider: http`. Configure an HTTP agent in `agentlab.config.yaml` and use `--agent <name>`.
+
+### `Conversation scenario '...' must not define 'tools'`
+
+Your conversation scenario YAML has a `tools:` field. HTTP agents manage their own tools internally — remove the `tools:` block.
+
+### `Conversation scenario '...' must define at least one step`
+
+The `steps:` list is empty or missing. Add at least one step:
+
+```yaml
+steps:
+  - role: user
+    message: "Hello"
+```
+
+### Per-step evaluator type rejected
+
+Only these evaluator types are valid inside `steps[].evaluators`:
+
+- `response_contains`
+- `response_not_contains`
+- `response_matches_regex`
+- `response_latency_max`
+
+End-of-run types (`step_count_max`, `final_answer_contains`, `exact_final_answer`) belong at the top-level `evaluators:` block, not inside individual steps.
+
+---
+
 ## Global install behaves differently from repo mode
 
 That usually means the current working directory is wrong.
@@ -143,6 +334,8 @@ The CLI operates on the current working directory and expects:
 
 Run it from the project root you want to evaluate.
 
+---
+
 ## Release Verification
 
 Before publishing or cutting a release, run:

package/docs/variant-sets.md

@@ -0,0 +1,63 @@
+# Variant Sets
+
+Variant sets are named comparison groups defined in `agentlab.config.yaml`.
+
+They are the Tier 1 mechanism for prompt, model, tool-schema, and config experiments without turning every comparison into manual CLI bookkeeping.
+
+## Why They Exist
+
+Named agents remain the executable unit.
+
+Variant sets sit on top of named agents so you can run the same scenario or suite against multiple variants and compare the results intentionally.
+
+## Config Shape
+
+```yaml
+variant_sets:
+  - name: refund-agent-model-comparison
+    variants:
+      - agent: mock-default
+        label: baseline
+        prompt_version: prompt-v3
+        model_version: mock-model
+        tool_schema_version: support-tools-v1
+        config_label: baseline-refund-flow
+      - agent: mock-compact
+        label: concise
+        prompt_version: prompt-v4
+        model_version: mock-model
+        tool_schema_version: support-tools-v1
+        config_label: concise-refund-flow
+```
+
+## CLI Usage
+
+Run one scenario against all variants:
+
+```bash
+agentlab run support.refund-correct-order --variant-set refund-agent-model-comparison
+```
+
+Run one suite definition against all variants:
+
+```bash
+agentlab run --suite-def pre_merge --variant-set refund-agent-model-comparison
+```
+
+## Stored Identity
+
+Each resulting run stores and surfaces:
+
+- `variant_set_name`
+- `variant_label`
+- `prompt_version`
+- `model_version`
+- `tool_schema_version`
+- `config_label`
+- `config_hash`
+
+Those fields appear in CLI run summaries, `agentlab show`, run history, comparisons, and the UI.
+
+## Design Rule
+
+Use variant sets for intentional experiments. Keep named agents stable, and treat the variant set as the comparison layer.

package/examples/coding-tools/README.md

@@ -0,0 +1,21 @@
+# Example Coding Tools
+
+Minimal package-style coding-tool example for Agent Regression Lab.
+
+Register it in `agentlab.config.yaml` like this:
+
+```yaml
+tools:
+  - name: coding.read_repo_hint
+    package: "@agentlab/example-coding-tools"
+    exportName: readRepoHint
+    description: Return a small repo hint for the target path.
+    inputSchema:
+      type: object
+      additionalProperties: false
+      properties:
+        path:
+          type: string
+      required:
+        - path
+```

package/examples/coding-tools/index.js

@@ -0,0 +1,11 @@
+export async function readRepoHint(input) {
+  const path = String(input?.path ?? "");
+  if (!path) {
+    throw new Error("path is required");
+  }
+
+  return {
+    path,
+    hint: "Check the target file before editing.",
+  };
+}

package/examples/coding-tools/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "@agentlab/example-coding-tools",
+  "private": true,
+  "type": "module",
+  "exports": {
+    ".": "./index.js"
+  }
+}

package/examples/support-tools/README.md

@@ -0,0 +1,21 @@
+# Example Support Tools
+
+Minimal package-style tool example for Agent Regression Lab.
+
+Register it in `agentlab.config.yaml` like this:
+
+```yaml
+tools:
+  - name: support.find_duplicate_charge
+    package: "@agentlab/example-support-tools"
+    exportName: findDuplicateCharge
+    description: Find the duplicated charge order id for a given customer.
+    inputSchema:
+      type: object
+      additionalProperties: false
+      properties:
+        customer_id:
+          type: string
+      required:
+        - customer_id
+```

package/examples/support-tools/index.js

@@ -0,0 +1,8 @@
+export async function findDuplicateCharge(input) {
+  const customerId = String(input?.customer_id ?? "");
+  if (!customerId) {
+    throw new Error("customer_id is required");
+  }
+
+  return { order_id: `dup_${customerId}` };
+}

package/examples/support-tools/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "@agentlab/example-support-tools",
+  "private": true,
+  "type": "module",
+  "exports": {
+    ".": "./index.js"
+  }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-regression-lab",
-  "version": "0.2.0",
+  "version": "0.4.0",
   "private": false,
   "description": "Local-first scenario-based evaluation harness for AI agents.",
   "license": "MIT",
@@ -21,22 +21,24 @@
   ],
   "type": "module",
   "bin": {
-    "agentlab": "./dist/index.js"
+    "agentlab": "bin/agentlab.js"
   },
   "files": [
+    "bin",
     "dist",
     "dist/ui-assets",
     "README.md",
-    "docs"
+    "docs",
+    "examples"
   ],
   "engines": {
-    "node": ">=22"
+    "node": ">=18"
   },
   "scripts": {
     "build": "tsc -p tsconfig.json && npm run build:ui",
     "build:ui": "esbuild src/ui/client.tsx --bundle --format=esm --platform=browser --outdir=dist/ui-assets --loader:.css=css --log-level=warning",
     "check": "tsc -p tsconfig.json --noEmit",
-    "test": "tsx --test tests/**/*.test.ts",
+    "test": "tsx --test tests/*.test.ts tests/**/*.test.ts",
     "smoke:cli": "npm run build && node dist/index.js --help && node dist/index.js version",
     "start": "tsx src/index.ts",
     "run": "tsx src/index.ts"