autopilot-ai 2.0.0 → 2.2.0

package/README.md

@@ -236,6 +236,20 @@ autopilot -p "..." --dangerously-allow-all          # auto-approve mutating tool
 autopilot -p "..." --reasoning                      # include chain-of-thought in stderr
 ```
 
+### Run tasks on your own server
+
+Send a task to an Autopilot server you run. It works in its own sandbox and opens a pull request, and you can close your laptop while it does.
+
+```sh
+autopilot remote add https://acme.example.com      # approve the code in your browser; no keys copied
+autopilot --on acme "fix the flaky checkout test"  # runs on the server; uncommitted changes go along
+autopilot runs                                     # what's running or done, with PR links
+autopilot runs watch <run>                         # follow a run (Ctrl-C detaches, the run keeps going)
+autopilot pull <run>                               # check out the run's branch here
+```
+
+The run starts from the last commit your branch shares with `origin`. Unpushed commits, staged and unstaged edits, and untracked files that aren't ignored are sent along as a patch, up to 3 MiB. Add `--detach` to return right away and `-m <model>` to pick one of the server's models. Each task gets its own sandbox and branch, so several can run at once. The server keeps the model key and GitHub credentials; your laptop holds only a revocable, 90-day pairing credential in `~/.config/autopilot/remotes.json`. Setting up the server side is covered in [docs/aster-api.md](docs/aster-api.md).
+
 ### Headless SDK
 
 Use autopilot programmatically from your own application — no TUI required.
@@ -390,7 +404,6 @@ autopilot
 | `/mode edit\|plan\|auto` | Switch permission mode |
 | `/shell auto\|bash\|cmd\|powershell` | Show or set the shell for the bash tool |
 | `/thinking low\|medium\|high` | Reasoning effort (persists) |
-| `/theme` | Interactive theme picker (`Ctrl+T`) |
 | `/resume` | Pick a past conversation to restore |
 | `/compact` | Summarize older turns to free context |
 | `/init` | Scan repo and write `KIMI.md` project context |
@@ -401,9 +414,10 @@ autopilot
 | `/provider` | List providers, or switch with `/provider <id> [key]` (see [Providers](#providers)) |
 | `/jev` | Ask Jev for a typed, scoped one-shot decision |
 | `/subagents` | List running subagents; `/subagents cancel <n>\|all` stops them; `/subagents off\|suggest\|auto` sets the delegation policy (default: `auto`) |
-| `/now` | Run your latest queued message now instead of after the current turn (also `Ctrl+G` in Ink) |
+| `/now` | Run your latest queued message now instead of after the current turn |
 | `/key` | Show the active provider's key — on OpenRouter with spend and credit (`/key set <key>`, `/key clear`) |
 | `/cost` | Show cost by session, day, month and all time (OpenRouter-confirmed, or estimated from the provider's prices) |
+| `/good` / `/bad` `[note]` | Rate the last turn; recorded in the session trace when [tracing](#session-traces-and-evals) is on |
 | `/update` | Check for updates |
 | `/help` | List all commands |
 
@@ -428,10 +442,8 @@ Type `!` followed by a command (for example `! gh auth login`) to run it in your
 | `Ctrl+C` / `Esc` | Interrupt current turn when busy; exit when idle |
 | `Ctrl+R` | Toggle reasoning display |
 | `Ctrl+O` | Toggle verbose tool output |
-| `Ctrl+T` | Open theme picker |
 | `Shift+Tab` | Cycle mode (edit → plan → auto) |
 | `↑` / `↓` | Walk prompt history |
-| `Ctrl+G` | While the agent works: run your latest queued message now instead of after the turn |
 
 ### Messages sent while the agent is working
 
@@ -439,7 +451,7 @@ A message you send mid-turn is triaged instead of always waiting in line:
 
 - **interrupt** ("stop, that's the wrong file"): the turn stops at the next safe point and your message runs next. Running subagents are never cancelled by an interrupt; while they run, the message is delivered as soon as they finish.
 - **steer** ("also handle the empty case"): folded into the current task at the agent's next step.
-- **queue** ("after this, update the changelog", or anything unclear): runs as its own turn afterwards. The queue shows `Ctrl+G run this now` so you can promote it.
+- **queue** ("after this, update the changelog", or anything unclear): runs as its own turn afterwards. Send `/now` to promote it.
 - **aside** ("how's it going?"): answered from the current state without disturbing the turn.
 
 Clear phrasing is decided instantly; ambiguous messages get a quick model check (OpenRouter only), and anything uncertain is queued. Use `/subagents cancel <n>` to stop a single subagent.
@@ -498,6 +510,33 @@ Each log entry maps to one OTel `LogRecord`. Correlation IDs
 encoding, and a `service.name=kimiflare` + `service.version` pair sits
 on the resource.
 
+## Session traces and evals
+
+Autopilot can record its own sessions so you can see how it performs on your work, and compare setups on your own tasks. Everything stays on your machine.
+
+### Session traces
+
+Tracing is off by default. Turn it on for one run with `KIMIFLARE_TRACE=1`, or always with `"trace": { "enabled": true }` in `~/.config/kimiflare/config.json`. Each session then writes a `trace.jsonl` under `~/.local/share/kimiflare/traces/` (`KIMIFLARE_TRACE_DIR` or `trace.root` to change it), one event per line:
+
+- every model request and response, with tokens, cost and latency
+- tool calls and their results, and each permission decision
+- compactions, completion checks, subagents and turn ends
+- **outcome signals:** the git changes a session made, test, lint, typecheck and build commands and whether they passed, your `/good` and `/bad` ratings (with an optional note), and whether your next message corrected the agent or told it to carry on
+
+Large bodies are stored once, by hash, next to the trace. Likely secrets are redacted before anything is written, nothing is uploaded, and old traces are pruned automatically: after 90 days, above 200 MB for one session, or when all traces pass 2 GB.
+
+### Evals
+
+`autopilot eval` runs Autopilot headlessly on a set of tasks and compares two configurations: a model, a setting, or a harness change.
+
+```bash
+autopilot eval run --tasks evals/tasks/autopilot-bench --config evals/configs/baseline.json --model <id> --out /tmp/eval-a
+autopilot eval run --tasks evals/tasks/autopilot-bench --config evals/configs/my-change.json --model <id> --out /tmp/eval-b
+autopilot eval compare /tmp/eval-a/results.jsonl /tmp/eval-b/results.jsonl
+```
+
+Each task is a JSON file naming a repo, a base commit, a prompt and a test command. Every run gets its own git worktree with tracing forced on, and the task passes if the test exits 0. `eval compare` prints success rates with a paired significance test, cost, turns and the tasks that flipped. This repository ships `autopilot-bench`, twenty tasks mined from its own merged pull requests; see [evals/README.md](evals/README.md) for the task format and how to build a set from your own repo.
+
 ## Hooks
 
 autopilot can fire shell commands at these points, configured
@@ -507,12 +546,12 @@ still read (see [Settings location](#settings-location)):
 
 | Event              | Fires when                                      | Veto? |
 |--------------------|-------------------------------------------------|-------|
-| `SessionStart`     | A session's first turn (`source`: startup/resume) | No  |
+| `SessionStart`     | A session's first prompt, before `UserPromptSubmit` (`source`: startup/resume) | No  |
 | `PreToolUse`       | A tool call is about to run                     | Yes   |
 | `PostToolUse`      | A tool call just finished (success or failure)  | No    |
 | `PostToolUseFailure` | A tool call failed (also fires `PostToolUse`) | No    |
 | `PermissionRequest` | You're about to be asked to approve a call     | Allow / deny |
-| `UserPromptSubmit` | You hit Enter on a prompt                       | Yes   |
+| `UserPromptSubmit` | A prompt is submitted, in every mode (TUI, `-p`, `--emit-events`, `serve`, SDK) | Yes   |
 | `Stop`             | A turn ended cleanly                            | No    |
 | `PreCompact`       | Auto-compaction is about to run                 | No    |
 | `SessionEnd`       | autopilot exits, or `/clear` (`reason`)         | No    |
@@ -528,7 +567,17 @@ prefer reading stdin. Payloads
 also carry Claude Code's field names (`hook_event_name`, `tool_name`,
 `tool_input`, `tool_response`, `tool_call_id`, `transcript_path`), so
 Claude Code hook scripts — and tools built for them, like
-[Agent Beacon](docs/beacon.md) — work unchanged.
+[Agent Beacon](docs/beacon.md) — work unchanged. A successful
+`write`/`edit`'s `tool_response` carries Claude Code's `structuredPatch`
+and `originalFile` (and `type: "create" | "update"` for `Write`); see
+[docs/beacon.md](docs/beacon.md#tool-names-and-result-shapes) for the
+size limits.
+
+A blocked `UserPromptSubmit` never reaches the model: the TUI shows the
+reason, `-p` and one-shot `--emit-events` exit with code **44**,
+`serve` answers `403 prompt_blocked`, and the SDK's `prompt()` rejects
+with `PromptBlockedError` (details in
+[docs/beacon.md](docs/beacon.md#session-and-prompt-events)).
 #### Blocking and exit codes
 
 Veto events (`PreToolUse`, `UserPromptSubmit`) follow Claude Code's rules: