agent-regression-lab 0.1.1 → 0.3.0

package/docs/tools.md

@@ -0,0 +1,102 @@
+# Custom Tools
+
+Custom tools are registered in `agentlab.config.yaml` and loaded from repo-local JS or TS modules.
+
+This is the main extension point when built-in tools are not enough.
+
+## What A Tool Registration Needs
+
+Each tool entry must define:
+
+- `name`
+- `modulePath`
+- `exportName`
+- `description`
+- `inputSchema`
+
+Example:
+
+```yaml
+tools:
+  - name: support.find_duplicate_charge
+    modulePath: user_tools/findDuplicateCharge.ts
+    exportName: findDuplicateCharge
+    description: Find the duplicated charge order id for a given customer.
+    inputSchema:
+      type: object
+      additionalProperties: false
+      properties:
+        customer_id:
+          type: string
+          description: Customer id to inspect for duplicated charges.
+      required:
+        - customer_id
+```
+
+## Tool Module Shape
+
+The exported function should be async and should return JSON-serializable output.
+
+Minimal example:
+
+```ts
+export async function myTool(input: unknown): Promise<{ ok: boolean }> {
+  return { ok: true };
+}
+```
+
+The existing working example is:
+
+- `user_tools/findDuplicateCharge.ts`
+
+## Important Constraints
+
+- `modulePath` must stay within the repo
+- the module must exist at load time
+- the named export must exist
+- tool input should be validated defensively inside the tool
+- tool output should be deterministic and JSON-serializable
+
+For launch usage, treat tools as fixture-backed local functions, not live integrations.
+
+## Recommended Pattern
+
+Use this approach:
+
+1. read fixture data from `fixtures/`
+2. validate the input shape
+3. return a small structured result
+4. throw a clear error for missing fixture state or invalid input
+
+The current `findDuplicateCharge` tool shows that pattern.
+
+## Wiring A Tool Into A Scenario
+
+1. register the tool in `agentlab.config.yaml`
+2. add the tool name to the scenario allowlist
+3. add an evaluator that confirms the tool was used correctly if the behavior is important
+
+Example scenario:
+
+- `scenarios/support/refund-via-config-tool.yaml`
+
+## Best Practices
+
+- keep tool names stable and descriptive
+- keep tools scenario-agnostic where possible
+- prefer read-only or sandboxed behavior
+- do not mutate global machine state
+- do not call live external systems in benchmark paths
+- keep schemas narrow so agent tool calls are easy to validate and compare
+
+## Common Errors
+
+Typical config failures:
+
+- duplicate tool names
+- repo-external module paths
+- missing module files
+- missing exports
+- invalid `inputSchema` shape
+
+See [troubleshooting.md](troubleshooting.md) for failure examples and fixes.

package/docs/troubleshooting.md

@@ -0,0 +1,296 @@
+# Troubleshooting
+
+This page covers the main failure modes users hit during install, first run, and comparison.
+
+## `agentlab: command not found`
+
+You are probably in one of these states:
+
+- the package is not installed globally
+- you have not run `npm link` from the repo
+- your shell path does not include npm global bins
+
+Fast fixes:
+
+```bash
+npm install
+npm run build
+npm link
+agentlab --help
+```
+
+Or skip linking and use:
+
+```bash
+npm run start -- --help
+```
+
+---
+
+## `OPENAI_API_KEY is required`
+
+You used an OpenAI-backed agent without exporting the API key.
+
+Fix:
+
+```bash
+export OPENAI_API_KEY=...
+agentlab run support.refund-correct-order --agent openai-cheap
+```
+
+---
+
+## `No scenarios found for suite ...`
+
+The suite id must match a suite under `scenarios/`.
+
+List valid options:
+
+```bash
+agentlab list scenarios
+```
+
+Current built-in suites in this repo include:
+
+- `support`
+- `coding`
+- `research`
+- `ops`
+- `internal-teams`
+
+---
+
+## `Run '<id>' not found`
+
+`show` and run-to-run `compare` require run ids from completed runs.
+
+Get a fresh run id by executing a scenario:
+
+```bash
+agentlab run support.refund-correct-order --agent mock-default
+```
+
+Then use:
+
+```bash
+agentlab show <run-id>
+agentlab compare <baseline-run-id> <candidate-run-id>
+```
+
+---
+
+## `Missing baseline or candidate suite batch id`
+
+`compare --suite` does not use run ids. It uses suite batch ids printed by `run --suite`.
+
+Example:
+
+```bash
+agentlab run --suite support --agent mock-default
+agentlab run --suite support --agent mock-default
+agentlab compare --suite <baseline-batch-id> <candidate-batch-id>
+```
+
+---
+
+## Cross-suite suite comparison errors
+
+Suite batch comparison is strict. Compare batches from the same suite only.
+
+This is valid:
+
+```bash
+agentlab compare --suite suite_...support_batch_a suite_...support_batch_b
+```
+
+This is not valid:
+
+- a `support` batch compared against an `ops` batch
+- mixed or malformed suite batch selections
+
+If you are unsure which batch came from which suite, rerun the suite and record the printed batch ids.
+
+---
+
+## `agentlab ui` fails to load assets
+
+Installed packages should already include the built UI assets.
+
+If you are running from a repo checkout, build first:
+
+```bash
+npm install
+npm run build
+agentlab ui
+```
+
+If the problem persists, verify that these files exist:
+
+- `dist/ui-assets/client.js`
+- `dist/ui-assets/client.css`
+
+---
+
+## Config tool or agent not found
+
+Typical reasons:
+
+- `agentlab.config.yaml` is missing
+- the configured `name` does not match the CLI `--agent` value
+- `modulePath` points outside the repo
+- the configured export or command does not exist
+
+Working references in this repo:
+
+- tool config: `agentlab.config.yaml`
+- custom tool: `user_tools/findDuplicateCharge.ts`
+- external agents: `custom_agents/node_agent.mjs`, `custom_agents/python_agent.py`
+
+---
+
+## HTTP agent errors
+
+### `HTTP agents require a configured url`
+
+You ran a conversation scenario with `--provider http` but no HTTP agent config was found.
+
+Fix: define a named http agent in `agentlab.config.yaml`:
+
+```yaml
+agents:
+  - name: my-agent
+    provider: http
+    url: http://localhost:3000/api/chat
+```
+
+Then run with:
+
+```bash
+agentlab run internal-teams.memory-followup-recall --agent my-agent
+```
+
+### `termination_reason: http_connection_failed`
+
+agentlab could not connect to your agent's URL. The most common cause is that the agent service is not running.
+
+Check:
+
+- is the service running on the configured port?
+- is the URL in `agentlab.config.yaml` correct?
+- is there a firewall or proxy blocking the connection?
+
+### `termination_reason: http_error`
+
+Your agent returned an HTTP 4xx or 5xx response.
+
+Check:
+
+- is the route path correct?
+- does your agent expect a different request shape? Use `request_template` if so.
+- are there auth errors? Check `headers` config.
+
+### `termination_reason: timeout_exceeded`
+
+Your agent did not respond within `timeout_ms` (default 30 seconds).
+
+Fix options:
+
+- increase `timeout_ms` in the agent config
+- investigate why the agent is slow for the given input
+
+### `termination_reason: invalid_response_format`
+
+Your agent either returned non-JSON or did not include the expected field.
+
+Defaults: agentlab reads the `message` field from the JSON response. Override with `response_field` if your agent uses a different name:
+
+```yaml
+agents:
+  - name: my-agent
+    provider: http
+    url: http://localhost:3000/api/chat
+    response_field: reply
+```
+
+---
+
+## `database is locked`
+
+You hit SQLite write contention on the local artifacts DB.
+
+Most common cause:
+
+- multiple `agentlab` runs writing to the same `artifacts/agentlab.db` at the same time
+
+Fix:
+
+- wait for the current run to finish
+- rerun sequentially instead of in parallel
+- keep live HTTP fixture verification serialized when using the same local project directory
+
+The product now uses a busy timeout, but sequential execution is still the safest path for local live verification.
+
+---
+
+## Conversation scenario errors
+
+### `Scenario '...' is a conversation scenario and requires provider: http`
+
+You tried to run a `type: conversation` scenario with a non-HTTP agent (`mock`, `openai`, or `external_process`).
+
+Conversation scenarios only work with `provider: http`. Configure an HTTP agent in `agentlab.config.yaml` and use `--agent <name>`.
+
+### `Conversation scenario '...' must not define 'tools'`
+
+Your conversation scenario YAML has a `tools:` field. HTTP agents manage their own tools internally — remove the `tools:` block.
+
+### `Conversation scenario '...' must define at least one step`
+
+The `steps:` list is empty or missing. Add at least one step:
+
+```yaml
+steps:
+  - role: user
+    message: "Hello"
+```
+
+### Per-step evaluator type rejected
+
+Only these evaluator types are valid inside `steps[].evaluators`:
+
+- `response_contains`
+- `response_not_contains`
+- `response_matches_regex`
+- `response_latency_max`
+
+End-of-run types (`step_count_max`, `final_answer_contains`, `exact_final_answer`) belong at the top-level `evaluators:` block, not inside individual steps.
+
+---
+
+## Global install behaves differently from repo mode
+
+That usually means the current working directory is wrong.
+
+The CLI operates on the current working directory and expects:
+
+- `scenarios/`
+- `fixtures/`
+- optional `agentlab.config.yaml`
+
+Run it from the project root you want to evaluate.
+
+---
+
+## Release Verification
+
+Before publishing or cutting a release, run:
+
+```bash
+npm run check
+npm test
+npm run build
+npm run smoke:cli
+npm pack --dry-run
+```
+
+For the full pre-launch checklist, see [release-checklist.md](release-checklist.md).

package/docs/variant-sets.md

@@ -0,0 +1,63 @@
+# Variant Sets
+
+Variant sets are named comparison groups defined in `agentlab.config.yaml`.
+
+They are the Tier 1 mechanism for prompt, model, tool-schema, and config experiments without turning every comparison into manual CLI bookkeeping.
+
+## Why They Exist
+
+Named agents remain the executable unit.
+
+Variant sets sit on top of named agents so you can run the same scenario or suite against multiple variants and compare the results intentionally.
+
+## Config Shape
+
+```yaml
+variant_sets:
+  - name: refund-agent-model-comparison
+    variants:
+      - agent: mock-default
+        label: baseline
+        prompt_version: prompt-v3
+        model_version: mock-model
+        tool_schema_version: support-tools-v1
+        config_label: baseline-refund-flow
+      - agent: mock-compact
+        label: concise
+        prompt_version: prompt-v4
+        model_version: mock-model
+        tool_schema_version: support-tools-v1
+        config_label: concise-refund-flow
+```
+
+## CLI Usage
+
+Run one scenario against all variants:
+
+```bash
+agentlab run support.refund-correct-order --variant-set refund-agent-model-comparison
+```
+
+Run one suite definition against all variants:
+
+```bash
+agentlab run --suite-def pre_merge --variant-set refund-agent-model-comparison
+```
+
+## Stored Identity
+
+Each resulting run stores and surfaces:
+
+- `variant_set_name`
+- `variant_label`
+- `prompt_version`
+- `model_version`
+- `tool_schema_version`
+- `config_label`
+- `config_hash`
+
+Those fields appear in CLI run summaries, `agentlab show`, run history, comparisons, and the UI.
+
+## Design Rule
+
+Use variant sets for intentional experiments. Keep named agents stable, and treat the variant set as the comparison layer.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-regression-lab",
-  "version": "0.1.1",
+  "version": "0.3.0",
   "private": false,
   "description": "Local-first scenario-based evaluation harness for AI agents.",
   "license": "MIT",
@@ -26,7 +26,8 @@
   "files": [
     "dist",
     "dist/ui-assets",
-    "README.md"
+    "README.md",
+    "docs"
   ],
   "engines": {
     "node": ">=22"
@@ -35,7 +36,7 @@
     "build": "tsc -p tsconfig.json && npm run build:ui",
     "build:ui": "esbuild src/ui/client.tsx --bundle --format=esm --platform=browser --outdir=dist/ui-assets --loader:.css=css --log-level=warning",
     "check": "tsc -p tsconfig.json --noEmit",
-    "test": "tsx --test tests/**/*.test.ts",
+    "test": "tsx --test tests/*.test.ts tests/**/*.test.ts",
     "smoke:cli": "npm run build && node dist/index.js --help && node dist/index.js version",
     "start": "tsx src/index.ts",
     "run": "tsx src/index.ts"