ur-agent 1.20.0 → 1.22.0

package/CHANGELOG.md

@@ -1,5 +1,67 @@
 # Changelog
 
+## 1.22.0
+
+### Added
+- **Agent execution metrics in `ur eval`.** `ur eval run` now captures
+  cost, input/output tokens, model used, API duration, files changed,
+  insertions/deletions, command failures, human-edit heuristics, and optional
+  per-case `testCommand` results. Metrics are written by the headless child
+  via `UR_EVAL_METRICS_FILE`, so future parallel eval runs stay safe.
+- **Richer eval dashboard.** `ur eval dashboard` and
+  `ur eval report <suite> --dashboard` generate local HTML dashboards with
+  summary cards (pass rate, test pass rate, cost, tokens, files changed, command
+  failures, human edits, duration) and a per-case timeline showing model, time,
+  cost, tokens, diffs, test result, and output preview.
+- **Per-case run metrics persistence.** `ur eval run <suite> --metrics` writes
+  each case's metrics to `.ur/evals/.runs/<suite>/<case>.json` for downstream
+  analysis.
+- **Benchmark-style reporting.** `formatEvalReport` prints aggregate cost,
+  tokens, files changed, command failures, human edits, duration, and test pass
+  rate alongside the pass-rate summary.
+
+### Changed
+- **`makeCliEvalRunner` resets cost state per case**, reads child metrics,
+  gathers `git diff --stat` totals, runs `expect.testCommand` when present, and
+  returns an `EvalRunMetrics` object with best-effort command-failure and
+  human-edit counts.
+- **Version bump.** Updated from 1.21.0 to 1.22.0 across `package.json`,
+  `bunfig.toml`, and the VS Code extension manifest.
+
+### Verified
+- Added `test/evalMetrics.test.ts` and `test/evalDashboard.test.ts` covering
+  child metrics serialization, report aggregation, dashboard HTML rendering,
+  HTML escaping, and run-metrics persistence.
+
+## 1.21.0
+
+### Added
+- **Agent skill runner (`agentSkillRunner.ts`).** Reusable helper that wraps
+  `startBackgroundTask({ worktree: true, pr: true })`, polls the background
+  manifest to completion, and returns a PR-style summary with branch, commits,
+  PR URL, and diff summary.
+- **New slash skills for agent worktrees.** Added `/debug-v2`, `/refactor`,
+  `/paper-implementation`, `/benchmark`, `/security-review`, `/dockerize`, and
+  `/latex-paper` bundled slash skills. Each expands into a prompt that instructs
+  the model to work in an isolated git worktree and produce a clean branch,
+  commits, and PR.
+- **Matching agent templates.** Added `debug-v2`, `refactor`, `paper-implementation`,
+  `benchmark`, `security-review`, `dockerize`, and `latex-paper` templates to
+  `AGENT_TEMPLATES`; install them with `ur agent-templates install`.
+- **`ur worktree` command.** Added `ur worktree list|status|clean` to inspect
+  and clean up UR agent worktrees created by background runs.
+
+### Changed
+- **Background runner exports.** Exported `commitIfNeeded` and `createPullRequest`
+  from `backgroundRunner.ts` so the agent skill runner can inspect and finalize
+  PR state.
+- **Version bump.** Updated from 1.20.0 to 1.21.0 across `package.json`,
+  `bunfig.toml`, the VS Code extension manifest, and the bundled CLI.
+
+### Verified
+- Added tests for the agent skill runner, each new bundled skill, and the new
+  `ur worktree` command.
+
 ## 1.20.0
 
 ### Added

package/README.md

@@ -132,6 +132,7 @@ as first-class subcommands in the shipped CLI.
 | `ur safety` | Inspect or initialize project shell safety policy and evaluate command risk before execution. |
 | `ur context-pack` | Write project architecture context, task memory, and compressed context under `.ur/`. |
 | `ur bg` | Run and manage detached local background agents with optional worktrees and PR creation. |
+| `ur worktree` | List, inspect, and clean up UR agent worktrees. |
 | `ur automation` | Store and run project-local scheduled automation specs under `.ur/automations/`. |
 | `ur workflow` | Define, validate, graph, run, and resume declarative agent workflows. |
 | `ur crew` | Run a lead and worker subagent crew over a shared task board. |
@@ -143,6 +144,9 @@ as first-class subcommands in the shipped CLI.
 | `ur artifacts` | Capture diffs, test runs, notes, and review feedback under `.ur/artifacts/`. |
 | `ur claim-ledger` | Map generated claims to file, web, MCP, tool, or user sources. |
 | `ur browser-qa` | Validate and smoke-run browser QA replay fixtures. |
+| `ur eval run` | Run an eval suite and grade outputs; optionally capture cost/tokens/files/test metrics. |
+| `ur eval report` | Show a saved eval report; `--dashboard` writes a single-suite HTML timeline. |
+| `ur eval dashboard` | Generate the local-first HTML dashboard across all saved reports. |
 | `ur eval bench` | Import local SWE-bench, Terminal-Bench, or Aider Polyglot exports. |
 | `ur model-doctor` | Inspect Ollama models and report likely agent capabilities. |
 | `ur model-route` | Recommend a local model for a task by capability fit. |
@@ -156,6 +160,10 @@ as first-class subcommands in the shipped CLI.
 | `ur a2a serve` | Start an opt-in local A2A task server with bearer or delegation auth. |
 | `ur sdk` | Show programmatic headless usage and scaffold SDK examples. |
 
+New slash skills run agentic work in isolated git worktrees with clean commits and PR output:
+`/debug-v2`, `/refactor`, `/paper-implementation`, `/benchmark`, `/security-review`, `/dockerize`, `/latex-paper`.
+Install matching agent templates with `ur agent-templates install`.
+
 New built-in tools (exposed through MCP and the ACP server): GitHub, API, Browser, Docker, TestRunner, Database. File-system and terminal tools are already built in (FileRead, FileEdit, FileWrite, Glob, Grep, Bash, PowerShell).
 
 Examples:
@@ -182,6 +190,9 @@ ur artifacts capture-tests --command "bun test"
 ur agent-task pr --create --dry-run
 ur acp serve --port 8123
 ur exec "add tests for the parser" --concurrency 4 --json
+ur eval run starter --metrics --json
+ur eval report starter --dashboard
+ur eval dashboard
 ```
 
 ## Project Context
@@ -291,6 +302,7 @@ CLI. Rebuild it after source, version, or macro changes.
 - [Configuration](docs/CONFIGURATION.md)
 - [Agent Feature Expansion](docs/AGENT_FEATURES.md)
 - [Agent Trend Coverage](docs/AGENT_TRENDS.md)
+- [1.22.0 Upgrade Notes](docs/AGENT_UPGRADE_1.22.0.md)
 - [1.20.0 Upgrade Notes](docs/AGENT_UPGRADE_1.20.0.md)
 - [1.19.0 Upgrade Notes](docs/AGENT_UPGRADE_1.19.0.md)
 - [1.18.0 Upgrade Notes](docs/AGENT_UPGRADE_1.18.0.md)