ur-agent 1.21.0 → 1.22.0

package/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## 1.22.0
+
+### Added
+- **Agent execution metrics in `ur eval`.** `ur eval run` now captures
+  cost, input/output tokens, model used, API duration, files changed,
+  insertions/deletions, command failures, human-edit heuristics, and optional
+  per-case `testCommand` results. Metrics are written by the headless child
+  via `UR_EVAL_METRICS_FILE`, so future parallel eval runs stay safe.
+- **Richer eval dashboard.** `ur eval dashboard` and
+  `ur eval report <suite> --dashboard` generate local HTML dashboards with
+  summary cards (pass rate, test pass rate, cost, tokens, files changed, command
+  failures, human edits, duration) and a per-case timeline showing model, time,
+  cost, tokens, diffs, test result, and output preview.
+- **Per-case run metrics persistence.** `ur eval run <suite> --metrics` writes
+  each case's metrics to `.ur/evals/.runs/<suite>/<case>.json` for downstream
+  analysis.
+- **Benchmark-style reporting.** `formatEvalReport` prints aggregate cost,
+  tokens, files changed, command failures, human edits, duration, and test pass
+  rate alongside the pass-rate summary.
+
+### Changed
+- **`makeCliEvalRunner` resets cost state per case**, reads child metrics,
+  gathers `git diff --stat` totals, runs `expect.testCommand` when present, and
+  returns an `EvalRunMetrics` object with best-effort command-failure and
+  human-edit counts.
+- **Version bump.** Updated from 1.21.0 to 1.22.0 across `package.json`,
+  `bunfig.toml`, and the VS Code extension manifest.
+
+### Verified
+- Added `test/evalMetrics.test.ts` and `test/evalDashboard.test.ts` covering
+  child metrics serialization, report aggregation, dashboard HTML rendering,
+  HTML escaping, and run-metrics persistence.
+
 ## 1.21.0
 
 ### Added

package/README.md

@@ -144,6 +144,9 @@ as first-class subcommands in the shipped CLI.
 | `ur artifacts` | Capture diffs, test runs, notes, and review feedback under `.ur/artifacts/`. |
 | `ur claim-ledger` | Map generated claims to file, web, MCP, tool, or user sources. |
 | `ur browser-qa` | Validate and smoke-run browser QA replay fixtures. |
+| `ur eval run` | Run an eval suite and grade outputs; optionally capture cost/tokens/files/test metrics. |
+| `ur eval report` | Show a saved eval report; `--dashboard` writes a single-suite HTML timeline. |
+| `ur eval dashboard` | Generate the local-first HTML dashboard across all saved reports. |
 | `ur eval bench` | Import local SWE-bench, Terminal-Bench, or Aider Polyglot exports. |
 | `ur model-doctor` | Inspect Ollama models and report likely agent capabilities. |
 | `ur model-route` | Recommend a local model for a task by capability fit. |
@@ -187,6 +190,9 @@ ur artifacts capture-tests --command "bun test"
 ur agent-task pr --create --dry-run
 ur acp serve --port 8123
 ur exec "add tests for the parser" --concurrency 4 --json
+ur eval run starter --metrics --json
+ur eval report starter --dashboard
+ur eval dashboard
 ```
 
 ## Project Context
@@ -296,6 +302,7 @@ CLI. Rebuild it after source, version, or macro changes.
 - [Configuration](docs/CONFIGURATION.md)
 - [Agent Feature Expansion](docs/AGENT_FEATURES.md)
 - [Agent Trend Coverage](docs/AGENT_TRENDS.md)
+- [1.22.0 Upgrade Notes](docs/AGENT_UPGRADE_1.22.0.md)
 - [1.20.0 Upgrade Notes](docs/AGENT_UPGRADE_1.20.0.md)
 - [1.19.0 Upgrade Notes](docs/AGENT_UPGRADE_1.19.0.md)
 - [1.18.0 Upgrade Notes](docs/AGENT_UPGRADE_1.18.0.md)