agent-regression-lab 0.2.0 → 0.3.0

package/docs/golden-suites.md

@@ -0,0 +1,74 @@
+# Golden Suites
+
+Golden suites are the scenario portfolio internal engineering teams should keep as long-lived regression assets.
+
+They are not just demos. They are engineering memory for the behaviors that matter before merge and before release.
+
+## Required Launch Categories
+
+- coding agent regressions
+- support and policy agents
+- incident / ops agents
+- memoryful multi-turn agents
+- tool-failure recovery
+- ambiguity and escalation
+- adversarial or malformed tool output
+- cost / latency / step-discipline checks
+
+## Recommended Portfolio Composition
+
+- 5 golden workflows
+- 5 historical regressions
+- 5 ugly edge failures
+- 3 degraded-tool scenarios
+- 2 policy or escalation scenarios
+
+## How To Use Golden Suites
+
+1. Keep one or two scenarios for the happy path that must always work.
+2. Add scenarios from real incidents as soon as a failure is understood.
+3. Add edge-case scenarios for ambiguity, degraded tools, malformed outputs, and multi-turn drift.
+4. Group launch-critical workflows into config-level `suite_definitions`.
+5. Run one scenario while debugging locally.
+6. Run a `pre_merge` suite definition before merge.
+7. Run curated `release` and `incident_regressions` suite definitions before release.
+
+## Suggested Initial Internal-Team Scenarios
+
+- coding destructive edit guardrails
+- incident triage under noisy alerts
+- escalation on ambiguity instead of guessing
+- malformed tool output or partial tool output
+- cross-session memory leakage
+- follow-up recall across turns
+
+## Design Rule
+
+Treat suite composition as a product artifact.
+
+The suite is part of the system design, not a disposable test folder.
+
+## Recommended Suite Definitions
+
+Use first-class `suite_definitions` instead of ad hoc tags alone:
+
+```yaml
+suite_definitions:
+  - name: smoke
+    include:
+      tags: [smoke]
+
+  - name: pre_merge
+    include:
+      tags: [smoke, regression]
+
+  - name: release
+    include:
+      suites: [support, internal-teams]
+
+  - name: incident_regressions
+    include:
+      tags: [incident, regression]
+```
+
+These become the operational units you wire into local verification, pre-merge checks, and release readiness.

package/docs/integrations-and-live-services.md

@@ -0,0 +1,58 @@
+# Integrations And Live Services
+
+Use this guide to choose the right ARL provider path for the engineering question you are trying to answer.
+
+## Provider Matrix
+
+### `mock`
+
+Use when you want:
+
+- deterministic smoke tests
+- stable docs examples
+- baseline verification while changing the harness itself
+
+### `openai`
+
+Use when you want:
+
+- real model behavior against deterministic tool surfaces
+- prompt and model validation before merge
+- quick local comparisons where the model is the variable
+
+### `external_process`
+
+Use when you want:
+
+- a local Node or Python agent to participate in the runner-controlled tool loop
+- the runner to remain authoritative for tools, step limits, and storage
+- a thin adapter around an existing local agent implementation
+
+### `http`
+
+Use when you want:
+
+- production-like multi-turn validation against a running service
+- the agent to own memory, conversation history, and internal tool execution
+- live verification of a real app instead of a deterministic wrapper
+
+`arl-test/` is the canonical example of this path in this repo.
+
+## Live-Service Verification
+
+Default workflow:
+
+1. start the service
+2. run `agentlab` from the project containing the relevant scenarios and `agentlab.config.yaml`
+3. run one scenario while debugging
+4. run a suite before merge
+5. compare candidate runs or suite batches against a known baseline
+
+## Integration Design Rule
+
+Choose the simplest provider that answers the engineering question you have.
+
+- If you only need deterministic regression evidence, prefer `mock`.
+- If you need real model behavior but deterministic tools, prefer `openai`.
+- If you need a local agent implementation but still want runner-owned tools, prefer `external_process`.
+- If you need the real running service with its own memory and orchestration, use `http`.

package/docs/memory-and-stateful-agents.md

@@ -0,0 +1,51 @@
+# Memory And Stateful Agents
+
+Memoryful agents are a distinct category in ARL.
+
+Use `type: conversation` scenarios when the agent owns:
+
+- conversation history
+- internal memory
+- internal tool execution
+- session or conversation identifiers
+
+## What ARL Owns
+
+For conversation scenarios, ARL owns:
+
+- the ordered user steps
+- the generated `conversation_id`
+- per-step and end-of-run evaluation
+- trace capture
+- run storage and comparison
+
+## What The Agent Owns
+
+For conversation scenarios, the agent owns:
+
+- how it stores conversation state
+- how it interprets `conversation_id`
+- what internal tools it calls
+- how it handles memory and recall across turns
+
+## How To Test Memoryful Agents
+
+Good memory-focused scenarios should cover:
+
+- follow-up recall within one conversation
+- refusal to leak identity or state across sessions
+- correct handling of repeated turns
+- graceful behavior when earlier turns are ambiguous or incomplete
+
+## Recommended Stateful Regression Cases
+
+- follow-up recall after two or more turns
+- cross-session contamination
+- stale memory overriding fresh input
+- memory surviving the right turns but not the wrong sessions
+
+## Design Rule
+
+Use task scenarios when the runner should stay authoritative for tools and turn control.
+
+Use conversation scenarios when the agent itself is being tested for memory, session behavior, or internal orchestration.

package/docs/release-checklist.md

@@ -37,6 +37,36 @@ Verify at least one extension path:
 - run `support.refund-via-config-tool` with `custom-node-agent`, or
 - verify a repo-local custom tool still loads from `agentlab.config.yaml`
 
+## HTTP Provider Smoke
+
+Verify the HTTP provider path for conversation scenarios:
+
+1. Start a minimal echo server (or any running HTTP agent service)
+2. Add a named `http` agent to `agentlab.config.yaml`:
+
+```yaml
+agents:
+  - name: my-agent
+    provider: http
+    url: http://localhost:3000/api/chat
+```
+
+3. Run a conversation scenario:
+
+```bash
+agentlab run internal-teams.memory-followup-recall --agent my-agent
+```
+
+4. Confirm the run produces a pass/fail result and the CLI output shows turn-by-turn step status
+
+If no live HTTP service is available, confirm the HTTP error paths work correctly:
+
+```bash
+agentlab run internal-teams.memory-followup-recall --agent my-agent
+# (with no service running)
+# Expected: status: error, terminationReason: http_connection_failed
+```
+
 ## Docs Verification
 
 Confirm these files match current behavior:

package/docs/runtime-profiles.md

@@ -0,0 +1,67 @@
+# Runtime Profiles
+
+Runtime profiles are reusable test-environment overlays defined in `agentlab.config.yaml`.
+
+They let you keep degraded-tool conditions and state-related authoring metadata out of individual scenarios.
+
+## Why They Exist
+
+Use a runtime profile when multiple scenarios should run under the same bad condition or seeded state instead of repeating that setup inline.
+
+Typical uses:
+
+- force one tool to time out
+- return malformed or partial tool output
+- keep a named profile for memory-related scenario setup
+
+## Config Shape
+
+```yaml
+runtime_profiles:
+  - name: timeout-orders-tool
+    tool_faults:
+      - tool: orders.list
+        mode: timeout
+        timeout_ms: 1500
+
+  - name: malformed-docs-read
+    tool_faults:
+      - tool: docs.read
+        mode: malformed_output
+```
+
+Supported tool fault modes:
+
+- `timeout`
+- `error`
+- `malformed_output`
+- `partial_output`
+
+## Scenario Usage
+
+Reference the profile from the scenario:
+
+```yaml
+runtime_profile: timeout-orders-tool
+```
+
+Example command:
+
+```bash
+agentlab run internal-teams.tool-timeout-profile --agent mock-default
+```
+
+## Current Execution Scope
+
+Today, runtime-profile fault injection is active only for task scenarios where ARL owns the tool loop.
+
+That means:
+
+- task scenarios: tool faults are injected deterministically by the runner
+- conversation scenarios: the reference is allowed, but ARL does not intercept the HTTP agent's internal tools
+
+The `state` block is available in config for reusable authoring metadata, but automatic seeded-state execution is not yet applied by the runner.
+
+## Design Rule
+
+Use runtime profiles for reusable conditions, not one-off scenario-specific quirks.