ur-agent 1.21.0 → 1.22.0

package/docs/AGENT_FEATURES.md

@@ -51,8 +51,19 @@ ur --discover-ollama
 ur --ollama-host http://192.168.1.50:11434
 ur worktree list
 ur worktree clean --dry-run
+ur eval run starter --metrics --json
+ur eval report starter --dashboard
+ur eval dashboard
 ```
 
+## v1.22.0 Additions
+
+| Addition | Surface | What it adds |
+| --- | --- | --- |
+| Eval execution metrics | `ur eval run <suite> --metrics`, `UR_EVAL_METRICS_FILE` | Child-serialized cost, tokens, model, duration, files changed, insertions/deletions, command failures, human-edit heuristics, and per-case `testCommand` pass/fail. Safe for parallel runs because each child writes its own metrics file. |
+| Richer eval dashboard | `ur eval dashboard`, `ur eval report <suite> --dashboard` | Local-first HTML dashboard with summary cards and a per-case timeline showing model, time, cost, tokens, diffs, test result, command failures, and human edits. |
+| Per-case run metrics persistence | `.ur/evals/.runs/<suite>/<case>.json` | `ur eval run <suite> --metrics` writes each case's metrics to a JSON file for downstream analysis. |
+
 ## v1.21.0 Additions
 
 | Addition | Surface | What it adds |

package/docs/AGENT_UPGRADE_1.22.0.md

@@ -0,0 +1,48 @@
+# Upgrading to UR Agent v1.22.0
+
+## What's new
+
+v1.22.0 turns `ur eval` into a real agent benchmark tool.
+
+- **Execution metrics** — headless `ur -p` child runs now write a metrics JSON file
+  via `UR_EVAL_METRICS_FILE`. The eval runner reads it and also runs
+  `git diff --stat` and an optional per-case `expect.testCommand`. Captured
+  metrics: cost USD, input/output tokens, model, API duration, files changed,
+  insertions/deletions, command failures, human-edit heuristics, and test
+  pass/fail.
+- **Parallel-safe by design** — each child writes its own metrics file, so
+  future parallel eval runs will not corrupt shared cost state.
+- **Richer dashboard** — `ur eval dashboard` and
+  `ur eval report <suite> --dashboard` generate local HTML dashboards with
+  summary cards (pass rate, test pass rate, cost, tokens, files changed, command
+  failures, human edits, duration) and a per-case timeline showing model, time,
+  cost, tokens, diffs, test result, and output preview.
+- **Per-case metrics persistence** — `ur eval run <suite> --metrics` writes
+  each case's metrics to `.ur/evals/.runs/<suite>/<case>.json`.
+- **Aggregate reporting** — `ur eval report <suite>` now prints totals for cost,
+  tokens, files changed, command failures, human edits, duration, and test pass
+  rate.
+
+## Quick examples
+
+```sh
+# Run the starter suite and print JSON with metrics
+ur eval run starter --metrics --json
+
+# Write a single-suite HTML timeline dashboard
+ur eval report starter --dashboard
+
+# Generate the combined dashboard across all saved reports
+ur eval dashboard
+
+# Import a SWE-bench-style export and run it
+ur eval bench swe-bench --file issues.jsonl --name my-bench
+ur eval run my-bench --metrics
+```
+
+## Upgrade steps
+
+1. Pull the release and run `bun install`.
+2. Run `bun run typecheck` and `bun test` to confirm the local state.
+3. Rebuild the bundled CLI: `bun run bundle`.
+4. No settings or project file migration is required.

package/docs/USAGE.md

@@ -125,6 +125,9 @@ UR includes slash commands and CLI subcommands for common workflows:
 - `ur ide diff ...` to capture editor-readable inline diff bundles
 - `ur acp ...` to start/stop/status the Agent Communication Protocol server for IDE extensions
 - `ur exec ...` to run prompts in non-interactive mode with optional concurrency
+- `ur eval run ...` to run a suite, grade results, and capture execution metrics
+- `ur eval report ...` to show a saved report or write a single-suite dashboard
+- `ur eval dashboard` to generate the local HTML dashboard across all reports
 - `ur eval bench ...` to import local SWE-bench, Terminal-Bench, or Aider Polyglot exports
 - `ur doctor` to inspect CLI health
 - `ur update` or `ur upgrade` to check for updates
@@ -162,6 +165,10 @@ ur repo-edit preview rename oldName --to newName
 ur repo-edit apply rename oldName --to newName --check "bun test"
 ur ide diff capture --title "Working tree review"
 ur eval bench list
+ur eval run starter --dry-run --json
+ur eval run starter --metrics --json
+ur eval report starter --dashboard
+ur eval dashboard
 ```
 
 ## Permissions

package/documentation/app.js

@@ -241,8 +241,8 @@ const commands = [
     name: 'eval',
     category: 'Verification',
     aliases: ['evals'],
-    summary: 'Create, validate, run, and report public agent eval suites with deterministic grading.',
-    examples: ['ur eval init', 'ur eval list', 'ur eval validate starter', 'ur eval run starter --dry-run', 'ur eval report starter'],
+    summary: 'Create, validate, run, and report public agent eval suites with deterministic grading plus execution metrics and a local HTML dashboard.',
+    examples: ['ur eval init', 'ur eval list', 'ur eval validate starter', 'ur eval run starter --dry-run', 'ur eval run starter --metrics --json', 'ur eval report starter --dashboard', 'ur eval dashboard'],
   },
   {
     name: 'goal',
@@ -630,8 +630,8 @@ const examples = [
   },
   {
     title: 'Eval suite',
-    text: 'Create a suite, validate, run offline, then run live.',
-    code: 'ur eval init\nur eval validate starter\nur eval run starter --dry-run\nur eval report starter',
+    text: 'Create a suite, validate, run offline with metrics, then open the dashboard.',
+    code: 'ur eval init\nur eval validate starter\nur eval run starter --dry-run\nur eval run starter --metrics\nur eval report starter --dashboard',
   },
   {
     title: 'Safe automation',

package/documentation/index.html

@@ -43,7 +43,7 @@
     <main id="content" class="content">
       <header class="topbar">
         <div>
-          <p class="eyebrow">Version 1.21.0</p>
+          <p class="eyebrow">Version 1.22.0</p>
           <h1>UR Agent Documentation</h1>
           <p class="lead">A practical, tutorial-style reference for installing, configuring, automating, extending, and operating UR Agent.</p>
         </div>

package/extensions/vscode-ur-inline-diffs/package.json

@@ -2,7 +2,7 @@
   "name": "ur-inline-diffs",
   "displayName": "UR Inline Diffs",
   "description": "Review UR inline diff bundles from .ur/ide/diffs inside VS Code.",
-  "version": "1.21.0",
+  "version": "1.22.0",
   "publisher": "ur-agent",
   "engines": {
     "vscode": "^1.92.0"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ur-agent",
-  "version": "1.21.0",
+  "version": "1.22.0",
   "description": "UR terminal coding agent CLI",
   "type": "module",
   "packageManager": "bun@1.3.14",