agent-regression-lab 0.3.0 → 0.5.0

package/docs/coding-agents.md

@@ -0,0 +1,74 @@
+# Coding Agents
+
+ARL supports coding-agent regression workflows through deterministic task scenarios.
+
+Use this path when the runner should remain authoritative for:
+
+- file inspection tools
+- patch application tools
+- step limits
+- regression scoring
+
+## Start With The Built-In Coding Scenarios
+
+This repo already includes two coding scenarios:
+
+- `coding.fix-add-function`
+- `coding.update-greeting`
+
+Run one directly:
+
+```bash
+agentlab run coding.fix-add-function --agent mock-default
+```
+
+These scenarios use fixture-backed repo tools, which makes them useful for:
+
+- prompt changes
+- model comparisons
+- patch-discipline checks
+- pre-merge behavioral regression checks
+
+## Why This Matters
+
+Coding agents often regress in subtle ways:
+
+- they inspect too much of the repo
+- they patch the wrong file
+- they over-edit instead of making a narrow change
+- they stop naming the changed file clearly
+
+ARL helps by making those expectations explicit in scenario evaluators.
+
+## Minimal Workflow
+
+1. run one coding scenario locally
+2. inspect the run output and trace
+3. run it again against a changed prompt/model/agent variant
+4. compare the two runs
+
+Example:
+
+```bash
+agentlab run coding.fix-add-function --agent mock-default
+agentlab run coding.fix-add-function --agent mock-default
+agentlab compare <baseline-run-id> <candidate-run-id>
+```
+
+## When To Use Task Scenarios Versus HTTP
+
+Use task scenarios for coding agents when:
+
+- you want deterministic fixture-backed tools
+- you want ARL to own the tool loop
+- you want reproducible patch-evaluator behavior
+
+Use HTTP/conversation scenarios only when the coding agent already exists as a running service and owns its own orchestration internally.
+
+## Next Step
+
+If you want coding-agent checks in team workflows, pair these scenarios with suite definitions and CI:
+
+```bash
+agentlab run --suite-def pre_merge --agent mock-default
+```

package/docs/superpowers/plans/2026-04-13-phase-2-lite-phase-3-plan.md

@@ -0,0 +1,160 @@
+# Phase 2 Lite And Phase 3 Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Deliver a minimal integration story for new users, then improve the UI enough that ARL is easier to demo, screenshot, and understand visually.
+
+**Architecture:** Keep Phase 2-lite focused on assets that clarify adoption: README routing, one coding-agent path, and one CI path. Keep Phase 3 focused on UI clarity instead of new product surface area by improving the runs dashboard, comparison screens, and trace presentation inside the existing React UI.
+
+**Tech Stack:** TypeScript, React, node:test, esbuild, Markdown, GitHub Actions YAML
+
+---
+
+## File Map
+
+**Roadmap and product docs**
+- Modify: `.claude/active-tasks.md`
+- Modify: `.claude/project.md`
+- Modify: `README.md`
+
+**Phase 2-lite assets**
+- Create: `docs/coding-agents.md`
+- Create: `.github/workflows/agentlab-pre-merge.yml`
+
+**UI**
+- Modify: `src/ui/App.tsx`
+- Modify: `src/ui/styles.css`
+
+**Tests**
+- Modify: `tests/launch/ui-smoke.test.ts`
+
+---
+
+### Task 1: Reframe Roadmap To Phase 2-lite Then Phase 3
+
+**Files:**
+- Modify: `.claude/active-tasks.md`
+- Modify: `.claude/project.md`
+
+- [ ] Update active task tracking so the current next phase is `Phase 2-lite`, not the original full Phase 2.
+- [ ] Update project memory so Phase 2-lite is the minimal integration-story pass and Phase 3 is the main visual/demo workstream.
+- [ ] Keep the scope explicit:
+  - HTTP example via `arl-test`
+  - CI example
+  - coding-agent example
+  - then UI polish
+
+Verification:
+
+```bash
+rg -n "Phase 2-lite|Phase 3" .claude/active-tasks.md .claude/project.md
+```
+
+---
+
+### Task 2: Add Phase 2-lite Integration Assets
+
+**Files:**
+- Modify: `README.md`
+- Create: `docs/coding-agents.md`
+- Create: `.github/workflows/agentlab-pre-merge.yml`
+
+- [ ] Add README routing sections:
+  - if your agent runs as an HTTP service
+  - if you are validating coding-agent changes
+  - if you want pre-merge regression checks in CI
+- [ ] Add one coding-agent guide using the existing coding scenarios and current tool-loop model.
+- [ ] Add one GitHub Actions example that runs:
+
+```bash
+npm ci
+npm run build
+node dist/index.js run --suite-def pre_merge --agent mock-default
+```
+
+- [ ] Keep this section narrow and copy-pasteable. No broad framework matrix.
+
+Verification:
+
+```bash
+rg -n "HTTP service|coding-agent|pre-merge|GitHub Actions" README.md docs/coding-agents.md .github/workflows/agentlab-pre-merge.yml
+```
+
+---
+
+### Task 3: Improve Runs Dashboard And Comparison UX
+
+**Files:**
+- Modify: `src/ui/App.tsx`
+- Modify: `src/ui/styles.css`
+- Modify: `tests/launch/ui-smoke.test.ts`
+
+- [ ] Add a stronger runs dashboard summary at the top of the list page:
+  - total runs shown
+  - pass/fail/error counts
+  - most recent suite/context hint
+- [ ] Redesign the compare page to make regressions visually obvious:
+  - top classification banner
+  - clearer delta cards
+  - evaluator/tool diff blocks with stronger hierarchy
+  - more obvious baseline vs candidate sections
+- [ ] Make the suite compare page easier to scan:
+  - headline regression/improvement counts
+  - clearer scenario groupings
+
+Verification:
+
+```bash
+npx tsx --test tests/launch/ui-smoke.test.ts
+```
+
+---
+
+### Task 4: Improve Trace And Detail Presentation
+
+**Files:**
+- Modify: `src/ui/App.tsx`
+- Modify: `src/ui/styles.css`
+- Modify: `tests/launch/ui-smoke.test.ts`
+
+- [ ] Replace the plain trace list with a more intentional timeline treatment:
+  - event badges or type labels
+  - stronger step grouping
+  - clearer source metadata
+- [ ] Keep failure-first behavior intact.
+- [ ] Preserve readability on narrow screens.
+
+Verification:
+
+```bash
+npx tsx --test tests/launch/ui-smoke.test.ts
+```
+
+---
+
+### Task 5: Full Verification
+
+**Files:**
+- Modify only if verification exposes issues
+
+- [ ] Run focused UI/docs-related verification:
+
+```bash
+npx tsx --test tests/launch/ui-smoke.test.ts tests/cliPackaging.test.ts
+```
+
+- [ ] Run full suite:
+
+```bash
+npm test
+```
+
+- [ ] Run release gates:
+
+```bash
+npm run check
+npm run build
+npm run smoke:cli
+npm_config_cache=/tmp/agentlab-npm-cache npm pack --dry-run
+```
+