agent-regression-lab 0.2.0 → 0.3.0

package/docs/scenarios.md

@@ -2,28 +2,38 @@
 
 Scenarios are YAML files under `scenarios/`. They are the core authoring interface for the product.
 
-Each scenario should describe one narrow job for the agent, not a vague capability test.
+agentlab supports two scenario types:
 
-## Required Shape
+- `task` — a single-instruction job for a tool-using agent (default, no `type` field needed)
+- `conversation` — a multi-turn dialog with an HTTP agent
 
-Each scenario should define:
+---
+
+## Task Scenarios
+
+Task scenarios are the default format. They describe a single job for an agent that uses tools to complete it.
+
+### Required Shape
+
+Each task scenario should define:
 
 - `id`
 - `name`
 - `suite`
 - `task`
 - `tools`
-- `runtime`
 - `evaluators`
 
-Common optional fields already used in this repo:
+Common optional fields:
 
 - `description`
 - `difficulty`
 - `tags`
+- `runtime_profile`
+- `runtime`
 - task `context`
 
-## Example
+### Example
 
 ```yaml
 id: support.refund-correct-order
@@ -65,32 +75,52 @@ evaluators:
         - ord_1024
 ```
 
-## Suites In This Repo
+### Runtime Profiles
 
-Current benchmark domains:
+Task scenarios can reference a named `runtime_profile` from `agentlab.config.yaml`.
 
-- `support`
-- `coding`
-- `research`
-- `ops`
+```yaml
+runtime_profile: timeout-orders-tool
+```
 
-Use a suite when scenarios belong to one behavior family and should be runnable together with:
+Runtime profiles let you apply reusable degraded-tool conditions without duplicating them across scenarios. Current shipped behavior:
 
-```bash
-agentlab run --suite support --agent mock-default
-```
+- task scenarios: tool fault injection is active
+- conversation scenarios: config reference is allowed for shared authoring, but ARL does not yet inject faults into the HTTP agent's internal tools
 
-`run --suite` creates a suite batch id. That id is later used for:
+### Evaluators
 
-```bash
-agentlab compare --suite <baseline-batch-id> <candidate-batch-id>
+Use deterministic evaluators only.
+
+| Type | Description |
+|------|-------------|
+| `tool_call_assertion` | Assert a specific tool was called with specific input |
+| `forbidden_tool` | Fail if a tool was called that should not have been |
+| `final_answer_contains` | Check that the final output contains required substrings |
+| `exact_final_answer` | Require an exact match on the final output |
+| `step_count_max` | Fail if the agent used more steps than allowed |
+| `tool_call_count_max` | Fail if the total number of tool calls exceeds a budget |
+| `tool_repeat_max` | Fail if one tool is overused |
+| `cost_max` | Fail if the run cost exceeds a configured USD budget |
+
+Evaluator modes:
+
+- `hard_gate` — failure immediately fails the run, regardless of other evaluators
+- `weighted` — contributes to the weighted score (0–100)
+
+### Runtime Limits
+
+```yaml
+runtime:
+  max_steps: 8
+  timeout_seconds: 60
 ```
 
-Suite comparison is strict. Only compare batches from the same suite.
+Both are optional. `max_steps` defaults to 8. `timeout_seconds` is uncapped if not set.
 
-## Tools
+### Tools
 
-Each scenario declares its allowed tools:
+Each task scenario declares its allowed tools:
 
 ```yaml
 tools:
@@ -98,72 +128,289 @@ tools:
     - crm.search_customer
     - orders.list
     - orders.refund
+  forbidden:
+    - orders.delete
 ```
 
-Keep the tool allowlist as narrow as possible. A broad allowlist weakens the benchmark and makes regressions harder to interpret.
+Keep the allowlist as narrow as possible. Broad allowlists weaken the benchmark.
 
-This repo supports both:
+### Budget And Governance Checks
 
-- built-in deterministic tools
-- repo-local custom tools registered in `agentlab.config.yaml`
+Operational regressions are often just as important as correctness regressions. Use budget evaluators to encode "technically worked, but unacceptable in production":
 
-The launch benchmark now includes built-in tools for:
+```yaml
+evaluators:
+  - id: total-tool-budget
+    type: tool_call_count_max
+    mode: hard_gate
+    config:
+      max: 2
+  - id: no-repeat-order-list
+    type: tool_repeat_max
+    mode: hard_gate
+    config:
+      tool: orders.list
+      max: 1
+```
 
-- support
-- coding
-- research
-- ops
+Use `cost_max` only where the run records cost metadata.
 
-See [tools.md](tools.md) for custom tool registration.
+---
+
+## Conversation Scenarios
+
+Conversation scenarios test HTTP agents through multi-turn dialogs. They require `type: conversation` and work exclusively with `provider: http` agents. The agent is responsible for maintaining its own conversation history.
+
+### Required Shape
+
+```yaml
+type: conversation
+id: internal-teams.memory-followup-recall
+name: Follow-Up Recall Within Conversation
+suite: internal-teams
+steps:
+  - role: user
+    message: "I'm traveling next Tuesday and I prefer aisle seats. Please remember that."
+  - role: user
+    message: "What seat preference did I mention earlier?"
+```
+
+Each step must have:
+
+- `role: user`
+- `message` — the message sent to the agent this turn
+
+### Per-Step Evaluators
+
+Evaluators can be attached to individual steps. They run immediately after the agent replies to that step.
+
+```yaml
+steps:
+  - role: user
+    message: "Where's my order #ORD-001?"
+    evaluators:
+      - type: response_contains
+        mode: hard_gate
+        config:
+          keywords: [shipped, tracking]
+      - type: response_latency_max
+        mode: hard_gate
+        config:
+          ms: 3000
+  - role: user
+    message: "What's the tracking number?"
+    evaluators:
+      - type: response_not_contains
+        mode: weighted
+        weight: 1
+        config:
+          keywords: ["don't know", error]
+```
+
+If a `hard_gate` per-step evaluator fails, the run stops immediately and remaining steps are skipped.
+
+### Per-Step Evaluator Types
+
+| Type | Config | Behavior |
+|------|--------|----------|
+| `response_contains` | `keywords: string[]` | Passes if ALL keywords appear in the reply (case-insensitive) |
+| `response_not_contains` | `keywords: string[]` | Passes if NONE of the keywords appear in the reply (case-insensitive) |
+| `response_matches_regex` | `pattern: string` | Passes if the reply matches the regex pattern (case-insensitive) |
+| `response_latency_max` | `ms: number` | Passes if the HTTP response arrived within the time limit |
+
+### Scenario Quality Rules
+
+- prefer `hard_gate` for business-critical assertions
+- use `weighted` checks for quality gradients, not for the single condition that makes the scenario trustworthy
+- conversation scenarios must use `config.keywords` for `response_contains` and `response_not_contains`
+- stale `config.text` authoring is rejected
+- use conversation scenarios when the agent owns memory, tool execution, or conversation history internally
+- keep golden suites focused on repeatable workflows, historical regressions, and ugly edge cases rather than one-off demos
+
+### End-of-Run Evaluators
+
+End-of-run evaluators run after all steps complete. They apply to the final reply.
+
+```yaml
+evaluators:
+  - type: step_count_max
+    mode: hard_gate
+    config:
+      max: 10
+  - type: final_answer_contains
+    mode: weighted
+    weight: 1
+    config:
+      keywords: [resolved, confirmed]
+```
 
-## Runtime Limits
+End-of-run evaluator types:
 
-Scenarios can enforce:
+| Type | Config | Behavior |
+|------|--------|----------|
+| `step_count_max` | `max: number` | Passes if the number of completed turns is within the limit |
+| `final_answer_contains` | `keywords: string[]` | Passes if ALL keywords appear in the final reply |
+| `exact_final_answer` | `expected: string` | Passes if the final reply exactly matches the expected string |
 
-- `max_steps`
-- `timeout_seconds`
+### Conversation State
+
+agentlab auto-generates a UUID `conversation_id` for each run. It is sent in every step request. The agent uses it to look up and maintain its own conversation history.
+
+The `state` block is optional:
+
+```yaml
+state:
+  conversation_id: auto
+```
+
+`auto` is the only supported value. The UUID is always generated regardless of whether the `state` block is present.
+
+### Restrictions
+
+Conversation scenarios must not define a `tools:` field. HTTP agents manage their own tools internally. If `tools:` is present, validation will fail with a clear error.
+
+Conversation scenarios may define `runtime_profile`, but today that is for shared scenario organization and future stateful hooks. ARL does not inject tool faults into HTTP agents.
+
+---
+
+## Suite Definitions
+
+Scenario `suite` still groups related files, but operational launch workflows should use config-level `suite_definitions`.
 
 Example:
 
 ```yaml
-runtime:
-  max_steps: 8
-  timeout_seconds: 60
+suite_definitions:
+  - name: pre_merge
+    include:
+      tags:
+        - smoke
+        - regression
 ```
 
-These limits are enforced by the runner. Use them to keep runs bounded and comparisons meaningful.
+Run one with:
 
-## Evaluators
+```bash
+agentlab run --suite-def pre_merge --agent mock-default
+agentlab run --suite-def pre_merge --variant-set refund-agent-model-comparison
+```
 
-Use deterministic evaluators only.
+Use suite definitions for stable workflow units like:
+
+- `smoke`
+- `pre_merge`
+- `release`
+- `incident_regressions`
 
-The current evaluator set includes:
+### Full Example
+
+```yaml
+type: conversation
+id: internal-teams.memory-followup-recall
+name: Follow-Up Recall Within Conversation
+suite: internal-teams
+description: Memoryful agent should recall a user-provided fact later in the same conversation.
+difficulty: medium
+tags:
+  - internal-teams
+  - conversation
+
+steps:
+  - role: user
+    message: "I'm traveling next Tuesday and I prefer aisle seats. Please remember that."
+    evaluators:
+      - type: response_contains
+        mode: weighted
+        config:
+          keywords:
+            - aisle
+
+  - role: user
+    message: "What seat preference did I mention earlier?"
+    evaluators:
+      - type: response_contains
+        mode: hard_gate
+        config:
+          keywords:
+            - aisle
 
-- `tool_call_assertion`
-- `forbidden_tool`
-- `final_answer_contains`
-- `exact_final_answer`
-- `step_count_max`
+evaluators:
+  - type: step_count_max
+    mode: hard_gate
+    config:
+      max: 2
+```
 
-Guidance:
+Run it with:
 
-- use hard gates for non-negotiable behavior
-- use weighted evaluators for softer quality checks
-- prefer tool assertions or exact output checks over vague answer checks when possible
+```bash
+agentlab run internal-teams.memory-followup-recall --agent my-production-agent
+```
 
-## Authoring Conventions
+Where `my-production-agent` is a named `http` agent in `agentlab.config.yaml`. See [agents.md](agents.md) for HTTP agent config.
 
-Use these defaults:
+### CLI Output
+
+Conversation runs print a different output format from task runs:
+
+```
+run internal-teams.memory-followup-recall — PASS
+  agent: my-production-agent (http://localhost:3000/api/chat)
+  turns completed: 2/2
+  step 1: pass (response_contains ✓)
+  step 2: pass (response_contains ✓)
+  run id: run_20260407_001234
+```
+
+If a hard-gate fails mid-run:
+
+```
+run internal-teams.memory-followup-recall — FAIL
+  agent: my-production-agent (http://localhost:3000/api/chat)
+  turns completed: 1/2
+  step 1: FAIL (response_contains ✗)
+  run stopped (evaluator_failed)
+  run id: run_20260407_001235
+```
+
+---
+
+## Suites
+
+Both task and conversation scenarios can belong to a suite.
+
+```yaml
+suite: support
+```
+
+Run an entire suite:
+
+```bash
+agentlab run --suite support --agent mock-default
+```
+
+`run --suite` skips conversation scenarios when using non-HTTP agents (conversation scenarios require `provider: http`). Task scenarios and conversation scenarios can coexist in the same suite directory.
+
+`run --suite` prints a suite batch id at the end. That id is used for suite comparison:
+
+```bash
+agentlab compare --suite <baseline-batch-id> <candidate-batch-id>
+```
+
+---
+
+## Authoring Conventions
 
 - `id` format: `<suite>.<short-name>`
 - keep scenario jobs narrow and concrete
-- keep fixture-backed context in `task.context`
+- keep fixture-backed context in `task.context` (task scenarios)
 - prefer deterministic fixture references over open-ended prompts
 - include `difficulty`, `description`, and `tags` for every launch scenario
+- for conversation scenarios, keep step count low (2–5) and evaluators specific
 
 ## Current Examples
 
-Useful scenario references in this repo:
+Task scenario references in this repo:
 
 - support: `scenarios/support/refund-correct-order.yaml`
 - support with config tool: `scenarios/support/refund-via-config-tool.yaml`

package/docs/troubleshooting.md

@@ -25,6 +25,8 @@ Or skip linking and use:
 npm run start -- --help
 ```
 
+---
+
 ## `OPENAI_API_KEY is required`
 
 You used an OpenAI-backed agent without exporting the API key.
@@ -36,6 +38,8 @@ export OPENAI_API_KEY=...
 agentlab run support.refund-correct-order --agent openai-cheap
 ```
 
+---
+
 ## `No scenarios found for suite ...`
 
 The suite id must match a suite under `scenarios/`.
@@ -52,6 +56,9 @@ Current built-in suites in this repo include:
 - `coding`
 - `research`
 - `ops`
+- `internal-teams`
+
+---
 
 ## `Run '<id>' not found`
 
@@ -70,6 +77,8 @@ agentlab show <run-id>
 agentlab compare <baseline-run-id> <candidate-run-id>
 ```
 
+---
+
 ## `Missing baseline or candidate suite batch id`
 
 `compare --suite` does not use run ids. It uses suite batch ids printed by `run --suite`.
@@ -82,6 +91,8 @@ agentlab run --suite support --agent mock-default
 agentlab compare --suite <baseline-batch-id> <candidate-batch-id>
 ```
 
+---
+
 ## Cross-suite suite comparison errors
 
 Suite batch comparison is strict. Compare batches from the same suite only.
@@ -99,6 +110,8 @@ This is not valid:
 
 If you are unsure which batch came from which suite, rerun the suite and record the printed batch ids.
 
+---
+
 ## `agentlab ui` fails to load assets
 
 Installed packages should already include the built UI assets.
@@ -116,6 +129,8 @@ If the problem persists, verify that these files exist:
 - `dist/ui-assets/client.js`
 - `dist/ui-assets/client.css`
 
+---
+
 ## Config tool or agent not found
 
 Typical reasons:
@@ -131,6 +146,127 @@ Working references in this repo:
 - custom tool: `user_tools/findDuplicateCharge.ts`
 - external agents: `custom_agents/node_agent.mjs`, `custom_agents/python_agent.py`
 
+---
+
+## HTTP agent errors
+
+### `HTTP agents require a configured url`
+
+You ran a conversation scenario with `--provider http` but no HTTP agent config was found.
+
+Fix: define a named http agent in `agentlab.config.yaml`:
+
+```yaml
+agents:
+  - name: my-agent
+    provider: http
+    url: http://localhost:3000/api/chat
+```
+
+Then run with:
+
+```bash
+agentlab run internal-teams.memory-followup-recall --agent my-agent
+```
+
+### `termination_reason: http_connection_failed`
+
+agentlab could not connect to your agent's URL. The most common cause is that the agent service is not running.
+
+Check:
+
+- is the service running on the configured port?
+- is the URL in `agentlab.config.yaml` correct?
+- is there a firewall or proxy blocking the connection?
+
+### `termination_reason: http_error`
+
+Your agent returned an HTTP 4xx or 5xx response.
+
+Check:
+
+- is the route path correct?
+- does your agent expect a different request shape? Use `request_template` if so.
+- are there auth errors? Check `headers` config.
+
+### `termination_reason: timeout_exceeded`
+
+Your agent did not respond within `timeout_ms` (default 30 seconds).
+
+Fix options:
+
+- increase `timeout_ms` in the agent config
+- investigate why the agent is slow for the given input
+
+### `termination_reason: invalid_response_format`
+
+Your agent either returned non-JSON or did not include the expected field.
+
+Defaults: agentlab reads the `message` field from the JSON response. Override with `response_field` if your agent uses a different name:
+
+```yaml
+agents:
+  - name: my-agent
+    provider: http
+    url: http://localhost:3000/api/chat
+    response_field: reply
+```
+
+---
+
+## `database is locked`
+
+You hit SQLite write contention on the local artifacts DB.
+
+Most common cause:
+
+- multiple `agentlab` runs writing to the same `artifacts/agentlab.db` at the same time
+
+Fix:
+
+- wait for the current run to finish
+- rerun sequentially instead of in parallel
+- keep live HTTP fixture verification serialized when using the same local project directory
+
+The product now uses a busy timeout, but sequential execution is still the safest path for local live verification.
+
+---
+
+## Conversation scenario errors
+
+### `Scenario '...' is a conversation scenario and requires provider: http`
+
+You tried to run a `type: conversation` scenario with a non-HTTP agent (`mock`, `openai`, or `external_process`).
+
+Conversation scenarios only work with `provider: http`. Configure an HTTP agent in `agentlab.config.yaml` and use `--agent <name>`.
+
+### `Conversation scenario '...' must not define 'tools'`
+
+Your conversation scenario YAML has a `tools:` field. HTTP agents manage their own tools internally — remove the `tools:` block.
+
+### `Conversation scenario '...' must define at least one step`
+
+The `steps:` list is empty or missing. Add at least one step:
+
+```yaml
+steps:
+  - role: user
+    message: "Hello"
+```
+
+### Per-step evaluator type rejected
+
+Only these evaluator types are valid inside `steps[].evaluators`:
+
+- `response_contains`
+- `response_not_contains`
+- `response_matches_regex`
+- `response_latency_max`
+
+End-of-run types (`step_count_max`, `final_answer_contains`, `exact_final_answer`) belong at the top-level `evaluators:` block, not inside individual steps.
+
+---
+
 ## Global install behaves differently from repo mode
 
 That usually means the current working directory is wrong.
@@ -143,6 +279,8 @@ The CLI operates on the current working directory and expects:
 
 Run it from the project root you want to evaluate.
 
+---
+
 ## Release Verification
 
 Before publishing or cutting a release, run:

package/docs/variant-sets.md

@@ -0,0 +1,63 @@
+# Variant Sets
+
+Variant sets are named comparison groups defined in `agentlab.config.yaml`.
+
+They are the Tier 1 mechanism for prompt, model, tool-schema, and config experiments without turning every comparison into manual CLI bookkeeping.
+
+## Why They Exist
+
+Named agents remain the executable unit.
+
+Variant sets sit on top of named agents so you can run the same scenario or suite against multiple variants and compare the results intentionally.
+
+## Config Shape
+
+```yaml
+variant_sets:
+  - name: refund-agent-model-comparison
+    variants:
+      - agent: mock-default
+        label: baseline
+        prompt_version: prompt-v3
+        model_version: mock-model
+        tool_schema_version: support-tools-v1
+        config_label: baseline-refund-flow
+      - agent: mock-compact
+        label: concise
+        prompt_version: prompt-v4
+        model_version: mock-model
+        tool_schema_version: support-tools-v1
+        config_label: concise-refund-flow
+```
+
+## CLI Usage
+
+Run one scenario against all variants:
+
+```bash
+agentlab run support.refund-correct-order --variant-set refund-agent-model-comparison
+```
+
+Run one suite definition against all variants:
+
+```bash
+agentlab run --suite-def pre_merge --variant-set refund-agent-model-comparison
+```
+
+## Stored Identity
+
+Each resulting run stores and surfaces:
+
+- `variant_set_name`
+- `variant_label`
+- `prompt_version`
+- `model_version`
+- `tool_schema_version`
+- `config_label`
+- `config_hash`
+
+Those fields appear in CLI run summaries, `agentlab show`, run history, comparisons, and the UI.
+
+## Design Rule
+
+Use variant sets for intentional experiments. Keep named agents stable, and treat the variant set as the comparison layer.