amness 0.0.0-stage → 0.1.2

package/docs/bench.md

@@ -0,0 +1,292 @@
+# Compare coding agents with `amness bench`
+
+`amness bench` runs the same tasks through several coding-agent tools (Amness, Claude Code, Codex, or any command) and grades every result with the same hidden check.
+
+In this page you will learn:
+
+- what a bake-off can and cannot tell you
+- how to set it up on Windows and run it
+- how to read the result table
+- how to add any other tool as a contestant
+
+A *contestant* is one tool in the comparison. A *task* is one job with a start state and a check. A *trial* is one attempt at one task. A *hidden check* is a command or function that decides if the task is solved. The tool never sees it.
+
+## What a bake-off can and cannot tell you
+
+**What it is.** A bake-off starts each tool on each task in a fresh folder. Then the same check grades the folder.
+
+**Why you would use it.** You get a pass rate with a confidence range, tokens, cost and time for each tool, on tasks that you choose.
+
+**What is the same for every contestant:**
+
+- the tasks and the hidden check
+- the prompt text (`promptTemplate`)
+- the timeout
+- a fresh temporary folder with a new git repository for each run
+
+**What is not the same:**
+
+- The model, unless you set one for each contestant. Each tool picks its own default model.
+- The tool's own system prompt, tools and permission settings.
+- How tokens and cost are reported. See "Read the table" below.
+
+So the result tells you how the tools behave as a whole, with the settings that you used. It does not tell you which tool is "smarter". Different models can cause a difference, not only the tool. The report prints a list of the settings that were used. Publish that list with your numbers.
+
+## Set up on Windows
+
+Do these steps in PowerShell.
+
+1. Install Node.js 22.18 or newer from nodejs.org. Check it:
+
+   ```powershell
+   node --version
+   ```
+
+2. Install [Git for Windows](https://git-scm.com/download/win). The bench needs `git` on your PATH. The hidden checks of the built-in tasks use bash tools such as `test`. Amness looks for the `bash.exe` that comes with Git (in `Program Files\Git\bin`) and for any `bash.exe` on PATH. It skips `C:\Windows\System32\bash.exe`, because that is the WSL launcher and it sees other files (see `src/shell.ts`).
+
+3. Install Amness from npm:
+
+   ```powershell
+   npm install -g amness
+   amness --version
+   ```
+
+   To bench the latest `main` instead of the release, install from a clone: `git clone https://github.com/amirsdream/Amness.git`, then `cd Amness`, `npm install` and `npm link`.
+
+4. Install and log in to the other tools yourself. Follow their own documentation. This page does not repeat install commands, because they change.
+   - Claude Code: <https://docs.claude.com/en/docs/claude-code>
+   - Codex: <https://github.com/openai/codex>
+
+5. Open a new terminal, so that the new PATH is read. Make sure `claude` and `codex` run from any folder. Then check:
+
+   ```powershell
+   amness bench --check
+   ```
+
+   You see one line per contestant, `found` or `missing`. The exit code is 1 if one is missing. The check calls no model.
+
+6. Set the API keys that you need in the same terminal. For Amness, see [Getting started](getting-started.md). For `claude-bare` you need `ANTHROPIC_API_KEY`.
+
+   ```powershell
+   $env:ANTHROPIC_API_KEY = "sk-ant-..."
+   ```
+
+**What was tested on Windows.**
+
+- Tested: the bench code and the stdin path (`amness run -`) run in the automated tests on a Windows machine in CI. That CI job is marked informational. It is allowed to fail.
+- Not tested: the `claude` and `codex` presets have not been run on any machine by the author. Their flags come from the vendors' documentation and may differ in your version.
+
+**If a flag differs on your version,** run `claude --help` or `codex exec --help`. Then edit `args` for that contestant in `bench.json`. No code change is needed.
+
+On Windows, Amness starts each tool through the shell, so `.cmd` shims work. It quotes the arguments for you. The presets send the prompt on standard input, so the prompt text never needs quoting.
+
+## Run your first bake-off
+
+1. Write a starter config. It has four contestants: `amness`, `amness-gate`, `claude` and `codex`.
+
+   ```bash
+   amness bench --init
+   ```
+
+   The command writes `bench.json`. It will not overwrite a file that exists.
+
+2. Edit `bench.json`:
+
+   | Setting | Meaning |
+   |---|---|
+   | `contestants` | The list of tools. Delete the ones that you do not have. |
+   | `model` (per contestant) | The model for that tool. Set it for all, in the same family if you can. |
+   | `trials` | Attempts per task and contestant (starter file: 3). |
+   | `jobs` | Runs at the same time (starter file: 2). Use 1 if you see rate-limit errors. |
+   | `timeoutSec` | Seconds before a run is killed (starter file: 900). |
+   | `promptTemplate` | The prompt for every tool. `{task}` is the task text. `{verify}` is the check command that the task offers. |
+   | `gitInit` | Make a git repository in each folder (default true). |
+   | `logDir` | Where the raw output of every run is kept (default `.amness/bench/logs`). |
+
+3. Look at the plan. This runs nothing and costs nothing:
+
+   ```bash
+   amness bench --config bench.json --dry-run
+   ```
+
+   You see the number of runs, the prompt for the first task, and the exact command line of every contestant. Models you did not set show as "the tool's default". A line `! codex: "codex" is not on PATH` warns you about a missing tool.
+
+4. Run the probe. It gives one tiny real task to each contestant. It confirms that the tool starts, that the grading works, and that tokens and cost are read. It costs a few cents.
+
+   ```bash
+   amness bench --config bench.json --probe
+   ```
+
+   Read the `PASS` or `FAIL` lines and the notes. Fix problems here, before the big run.
+
+5. Set a spending limit on every account. Then run:
+
+   ```bash
+   amness bench --config bench.json --builtin code
+   ```
+
+   Amness asks you to confirm the cost. Add `--yes` to skip the question. Amness prints one `PASS` or `FAIL` line for each run while it works. At the end it prints the table and saves the raw results in `.amness/bench/<time>.json`. Use `--out <file>` to choose another file.
+
+### Read the table
+
+Without `--config`, `amness bench --contestants amness,claude --model <name>` uses presets. `--models amness=<a>,claude=<b>` sets one model for each preset.
+
+The first table looks like this. The numbers are an example. They are not a measurement.
+
+```text
+13 tasks × 3 trials; intervals are 95% bootstrap over tasks; baseline = amness
+
+variant                pass [95% CI]        pass^3  tok/solved  cache  steps  secs   pareto
+amness                 77% [59-92]          62%     41k         58%    9.0    74.2   ●
+claude                 82% [64-95]          69%     55k         71%    7.5    61.0
+codex                  69% [49-87]          51%     n/a         n/a    0.0    88.5
+
+vs amness (paired by task)
+variant                Δ pass [95% CI]          W/L/T     p (Holm)  Δ tok/solved  verdict
+claude                 +5 [-10, +21]            3/2/8     0.600     +14k          inconclusive
+
+custom metrics (mean per run; Δ against amness with 95% interval over tasks)
+variant                cost-usd                       secs
+amness                 n/a                            74.20
+claude                 0.31  Δ n/a                    61.00  Δ -13.20 [-30.10, +3.00]
+```
+
+| Column | Meaning |
+|---|---|
+| `pass [95% CI]` | Share of runs that passed, and the range where the true rate probably lies. |
+| `pass^3` | Chance that 3 attempts at a task all pass. It is low for a tool that only wins sometimes. |
+| `tok/solved` | Tokens used for each solved task. A *token* is a small piece of text that a model reads. It shows `n/a` when the tool does not report tokens. |
+| `cache` | Share of input tokens that came from the cache. |
+| `steps` | Mean turns per run. It is 0 when the tool does not report turns. |
+| `secs` | Mean wall-clock seconds per run. |
+| `pareto` | A dot marks a tool that no other tool beats on both pass rate and cost. |
+| `Δ pass` | Difference in pass rate against the baseline, paired by task. The baseline is the first contestant. |
+| `W/L/T` | Tasks where this tool won, lost or tied against the baseline. |
+| `p (Holm)` | The p-value, adjusted because you compare several tools at once. |
+| `verdict` | `better`, `worse` or `inconclusive`. |
+
+The `cost-usd` and `secs` metrics are in the custom metrics table. `cost-usd` is the money that the tool reports for a run. `secs` is the run time in seconds, measured by Amness. Only tools that report a cost have `cost-usd`.
+
+**What "inconclusive" means.** The data cannot tell the tools apart. It does not mean that the tools are equal. In the example, the interval for the difference goes from -10 to +21 points. That includes zero.
+
+**Why 13 tasks and 3 trials is a minimum.** The report prints a line like "only pass-rate differences above ~55 points can be detected reliably". It comes from `minDetectable` in `src/stats.ts`. It is a rough rule: 80% power, 5% error, worst-case variance. With 13 tasks, it is about 55 points. With 50 tasks, it is about 28. More trials reduce the noise from model sampling. They do not add tasks. Only more tasks make small differences visible. With fewer than 3 trials, the report warns you.
+
+If a run cannot be set up, it goes to the "exclusion ledger" and is not counted. After the run, read the logs in `logDir` for any surprise.
+
+To read the saved results again:
+
+```bash
+amness eval --report .amness/bench/<run>.json
+```
+
+## Choose the tasks
+
+| Suite | Command | What is in it |
+|---|---|---|
+| `code` (default) | `amness bench --builtin code` | 13 easy tasks |
+| `hard` | `amness bench --builtin hard` | 6 harder tasks |
+| `adversarial` | `amness bench --builtin adversarial` | Tasks that try to trick the tool |
+| Your own | `amness bench --cases ./my-tasks` | A folder of `.json` task files |
+| Your own, with functions | `amness bench --cases my-tasks.ts` | A `.ts` or `.js` module that exports a list of cases |
+| SWE-bench format | see below | A `.jsonl` file |
+
+The built-in suites are small and made by us. They are not a public benchmark. Use `--only a,b` to run only some tasks.
+
+For your own tasks, use the same file format as `amness eval`. See [Evaluating the harness](evaluation.md) for the fields `files`, `task`, `verify` and `check`, and for a function check. Test them first with `amness eval --cases ./my-tasks --selftest`.
+
+For a SWE-bench-format file:
+
+```bash
+amness bench --suite instances.jsonl --test-cmd "python -m pytest -x -q {tests}" --install "pip install -e ." --limit 20
+```
+
+`--test-cmd` is required with `--suite`. Python and the repositories' needs must be installed on your machine. Not verified on Windows.
+
+## Use a subscription or an API key
+
+| Preset | Login | What it loads | Note |
+|---|---|---|---|
+| `claude` | Your Claude login | Your own settings, `CLAUDE.md` and hooks | Your configuration mixes into the result. |
+| `claude-bare` | `ANTHROPIC_API_KEY` only | Nothing from your user setup | A clean room. The `--bare` flag never reads OAuth or the keychain, so a subscription login does not work. |
+| `codex` | Your Codex login | Ignores your user config and rules | Runs in a `workspace-write` sandbox. |
+| `amness`, `amness-gate` | Your Amness provider key | Your Amness settings | `amness-gate` also uses `--verify`, so it cannot finish before the check passes. The others only see the check in the prompt. |
+
+Cost and tokens:
+
+- Claude's JSON output includes `total_cost_usd`, so you see `cost-usd` and tokens.
+- Codex shows tokens only. The token event format is not documented, so the parser reads what is commonly there. The `--probe` run tells you if it worked. Not verified on a real Codex.
+- Amness shows total tokens only (no split of input and output, no cost).
+- To get cost for tools without it, multiply the tokens by the price yourself. The bench does not add prices.
+
+## Add any other tool as a contestant
+
+Add an object to `contestants` in `bench.json`:
+
+```json
+{
+  "name": "my-tool",
+  "command": "my-agent",
+  "args": ["--headless", "--model", "{model}"],
+  "stdin": true,
+  "model": "my-model-1",
+  "parse": "none",
+  "env": { "MY_AGENT_KEY": "..." },
+  "timeoutSec": 600,
+  "note": "my agent, default settings"
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `name` | A unique name. It is the row name in the report. |
+| `command` | The program. It is found through PATH. |
+| `args` | Arguments. These placeholders are replaced: `{task}` (task text), `{cwd}` (the run folder), `{model}`, `{verify}` (the task's check command). |
+| `stdin` | `true` sends the prompt on standard input. This is the best choice, because there is no quoting problem. `false` sends no input. Then put `{task}` in `args`. |
+| `model` | The value for `{model}`. Shown in the report. |
+| `parse` | How to read tokens and cost: `claude-json`, `codex-jsonl`, `amness-json` or `none`. With `none`, the report has no tokens or cost for this tool. |
+| `env` | Extra environment variables for this tool only. |
+| `timeoutSec` | A timeout for this tool. The default is the bench timeout. |
+| `note` | A text shown in the report. Say what is special, for example "uses my login". |
+
+Names must be different. Every contestant needs a `name` and a `command`. Check the new entry with `--dry-run` and `--probe`.
+
+## Fair-play checklist
+
+- Use the same model family for every tool where you can, and set `model` for each.
+- Use the same permission level. The presets allow file edits and shell commands without asking.
+- The run order is task by task, then contestant, then trial. A slow period of the API can still affect one tool more than another. Repeat the bake-off on another day.
+- Do not run it while the machine is busy. Time is measured in wall-clock seconds.
+- `jobs` above 1 runs tools at the same time. Your API rate limits may then cause failures that are not the tool's fault. Use `jobs` 1 for timing.
+- Set a spending limit on every account before you start.
+- Run it in a VM or container. The tools are not sandboxed from your machine, and they can run commands. This matters most for tasks that you did not write.
+- Publish `bench.json` and the raw results file together with your numbers.
+- Do not use `claude` for a fair test unless you accept that your own `CLAUDE.md` and hooks are part of it. Use `claude-bare` or say so in the `note`.
+
+## Limits and common problems
+
+| Problem | What to do |
+|---|---|
+| `missing ... (not on PATH)` | Open a new terminal. Run `where claude` or `where codex` (Windows) or `which claude`. Add the folder to PATH. |
+| `could not start <command>: ...` in the note | The program could not run. Run `--check`, and run the command by hand in a terminal. |
+| `timed out after Ns` | Raise `timeoutSec` or `--timeout`. A timed-out run counts as a fail. |
+| `n/a` in `tok/solved` or `cost-usd` | The tool did not report it, or `parse` is wrong. Run `--probe` and open the `.out.txt` file in `logDir`. |
+| Every run of one tool fails with the same note | The flags or the login are wrong. Read the `.err.txt` and `.cmd.json` files in `logDir/<name>/`. Fix `args`. |
+| A tool says an option is unknown | Your version differs. Check `--help` of the tool and edit `args`. |
+| Quoting problems on Windows | Keep `stdin: true`. Avoid `{task}` in `args`. |
+| The check fails on `test` or `bash` not found (Windows) | Install Git for Windows. Make sure its `bin` folder is not blocked, or put a `bash.exe` on PATH. |
+| `none of the contestants can start` | No tool was found. Run `--check`. |
+| `the tool reported an error but the check passed` | Only a note. The run counts as a pass, because the check decides. |
+
+Other limits: the bench does not stop a tool from reading your files. Tokens and cost depend on what each tool reports. The `claude` and `codex` presets are not verified on a real machine.
+
+## Related pages
+
+- [Evaluating the harness](evaluation.md)
+- [Headless and CI](headless.md)
+- [Getting started](getting-started.md)
+
+## Next steps
+
+- Read [Evaluating the harness](evaluation.md) to write your own tasks and understand the statistics.
+- Read [Plugins](plugins.md) to change Amness before you compare it.
+- Read the [Cookbook](cookbook.md) for a short recipe.

package/docs/budgets.md

@@ -0,0 +1,185 @@
+# Budgets, exploration and subagents
+
+This page explains why runs used to end with "token budget exhausted" and no answer, what the vendors and papers advise, and what Amness does now.
+
+In this page you will learn:
+
+- how to choose a token budget and what you see when a run hits it
+- what the budget notes, the final report and the subagent slices do
+- how Amness stays under the provider's rate limits
+- which sources were read and which were only search snippets
+
+A *token* is a small piece of text that a model reads or writes. One token is about four letters. A *budget* is the most tokens a run may use.
+
+## Choose a budget
+
+**What it is.** The budget is a cap on the tokens for one turn of the chat, or for one headless run. The default is 300,000 tokens (`maxTokens` in `cli/config.ts`).
+
+**Why you would change it.** A small budget stops a run early. A large budget lets a lost agent spend more before it stops.
+
+**How the cap counts.** The cap counts cached input at one tenth of its size, because cached input costs about that much and a long session reads its whole history again on every turn (`Budget.effective` in `src/harness.ts`). The token count shown on screen is not changed. Where a server reports no token usage, Amness estimates about 4 characters per token and marks the figure `estimated`.
+
+**How to set it.**
+
+1. In the chat, type `/budget` to see the cap and what the last turn used.
+2. Type `/budget 500k` to change it. You can write `500k`, `1m` or `400000`. The minimum is 10,000.
+3. In headless mode, add `--max-tokens 500000` to `amness run`.
+4. To keep the value, set `"maxTokens"` in `~/.amness/config.json`.
+
+**Advice.** These values are starting points from how the code behaves. They are not measured on real tasks.
+
+| Task | Starting budget | Why |
+|---|---|---|
+| A question about one file or one function | 100,000 to 150,000 | The prompt rules ask for 3 to 10 tool calls for this kind of question. |
+| A bug fix with tests | 300,000 (the default) | One logged run of a broad question on a 300,000 cap used 132,000 (see below). Fixes with test runs cost more per step. |
+| A survey of a large code base, or a refactor over many files | 500,000 to 1,000,000 | Subagents use part of the budget. Their cost counts against yours. |
+| An unattended run in CI | The value you can afford, not more | The agent stops at the cap, so the cap is your worst-case cost. |
+
+Rules of thumb:
+
+- Start with the default. Look at the budget line in the sidebar or in `/budget` after the first turn. Then raise or lower the cap.
+- Do not set the cap very low. At 50% the model is told to stop broad reading, so a small cap makes the agent give up exploration early.
+- If a big task keeps hitting the cap, split it. Ask for the plan in one turn and for each part in a later turn. Each chat turn gets a fresh budget.
+- Use `/loop` or `/tokens` to see where the tokens went before you raise the cap.
+
+**Time limits.** The chat also has a time cap of 60 minutes per turn. `amness run` has no time cap. The classic menu mode uses 30 minutes.
+
+### Example: what you see when the budget stops a run
+
+When the cap is reached, or when one more step plus a summary would not fit, the loop makes one last call without tools. It asks the model what it found, the state of the work and what remains. You get that text instead of a bare error.
+
+In the chat, a warning line appears with the first line of the result. The result text looks like this (the numbers and the report are an example built from the format in `src/harness.ts`, not a captured run):
+
+```
+stopped: token budget exhausted (301204/300000)
+
+I read the router and the rate-limit gate. The retry logic is in
+src/ratelimit.ts. I did not read the OpenAI adapter yet.
+Left to do: check how src/openai.ts handles a 429.
+```
+
+If the loop sees that the next step would not fit, the first line is `stopped: budget nearly exhausted` instead.
+
+In headless mode with `--json`, the last line is a result object with `"finished": false` and `"stopped": "budget"`. The exit code is 1. Here `tokens` is the raw count, before the cached-input discount.
+
+**What to do next.** Raise the cap with `/budget`, or send a message such as "continue with the OpenAI adapter". The conversation keeps its history, so the agent continues from the report.
+
+**Limits and common problems.**
+
+- The final report is one more model call. If that call fails, the result has only the `stopped:` line.
+- The run counts as not finished (`finished: false`), even if the report is a good answer.
+- Time budgets give a similar message: `stopped: time budget exhausted (...)`.
+
+**Related pages.** [The agent loop](agent-loop.md), [Using the chat](usage.md), [Headless and CI](headless.md).
+
+## Understand why runs ran out of budget
+
+A question like "explore the codebase" had no natural end. The model kept reading. It started several deep explorers at once, and one used a wrong agent name, which failed and was retried. Every request sent the whole history again. The 300,000 cap then cut the run off with no answer.
+
+## Read what the sources say
+
+Sources marked *read* were fetched and read. *Snippet* means only a search summary was seen.
+
+| Source | Finding | Used here |
+|---|---|---|
+| [Claude task budgets](https://platform.claude.com/docs/en/build-with-claude/task-budgets) (*read*) | An advisory token budget for the whole agentic loop. The model sees a running countdown and "finishes gracefully". It counts what the model *sees* (new content), not the history that is sent again. Set it once: changing it on each request breaks the prompt cache. A budget that is too small for the task can make the model refuse or stop early. Minimum 20,000. Beta. Not Claude Code. | Opt-in `taskBudget` for Claude (below). |
+| [BATS, arXiv 2511.17006](https://arxiv.org/abs/2511.17006) (*read*) | A per-turn budget tracker, with behaviour tiers (high, medium, low, critical remaining), improved the accuracy of search agents and cut cost. A tracker at budget 10 matched plain ReAct at budget 100 with about 31% lower cost. The tasks were web search, not coding. | Budget notes at 50%, 75% and 90%. |
+| [Anthropic, multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system) (*read*) | Models misjudge effort, so put scaling rules in the prompt (simple fact: 1 agent, 3-10 tool calls; comparison: 2-4 agents, 10-15 calls each). Vague subagent briefs cause duplicated work: give an objective, an output format, tool guidance and boundaries. Token use explains about 80% of performance variance. Multi-agent runs use about 15 times the tokens of chat. They suit parallel, high-value work, and "domains like most coding" less. | Effort rules in the system prompts. Brief requirements in the `task` tool. Subagents are limited. |
+| [Anthropic, context engineering](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) (*read*) | Fetch what you need when you need it (paths, grep) instead of loading everything. Clear old tool results. Subagents may use tens of thousands of tokens but should return a short result of 1,000 to 2,000 tokens. Accuracy falls as the context grows ("context rot"). | Subagent reports are clipped to 8,000 characters. Old tool results are elided. |
+| [OpenAI Codex subagents](https://learn.chatgpt.com/docs/agent-configuration/subagents) (*read*) | Subagents are for read-heavy parallel work. They cost more tokens than one agent. They return summaries, not raw output. There is a cap on concurrent threads. Be careful with parallel writers. | At most 3 `task` calls run at once. |
+| [OpenAI compaction guide](https://developers.openai.com/api/docs/guides/compaction) (*snippet*) | Server-side compaction through `context_management` with a `compact_threshold`. | Optional `compactAt` (see "Not done, and why"). |
+| Blogs on "exploration burn", for example [First-Touch Tool Burn](https://tianpan.co/blog/2026/04/23/first-touch-tool-burn-agent-exploration-budget) (*snippet*) | A claim that reading uses most of an agent's spend (figures from 47% to 87%), and a proposal to cap reads before the agent must state a plan. The numbers are not verified. | The check for 12 read-only turns is the same idea. |
+
+## Know what Amness does now
+
+1. **Prompt rules, not only limits** (Anthropic). The system prompts say how many tool calls a question deserves and when a subagent is worth its cost.
+2. **The model sees its budget** (BATS). At 50%, 75% and 90% of the cap, a note is added at the end of the conversation. The cache stays valid, because nothing earlier changes. The 50% note says to stop broad exploration. The 75% note says to produce the result. The 90% note says the next reply must be the result.
+3. **Never end empty-handed.** At 95% of the cap, when the cap is reached, or when another step plus a summary would not fit, one last call without tools asks what was found, the state of the work and what remains.
+4. **Exploration check.** After 12 turns in a row that only read, the model is told to answer (if the task is a question) or start (if it is a change), or to name exactly what is still missing. It repeats after another 12.
+5. **Subagents are bounded** (next section).
+6. **The cap measures cost.** Cached input counts at a tenth.
+7. **Claude's own countdown (opt-in).** `"taskBudget": 150000` in the config (at least 20,000) sends `output_config.task_budget` with the `task-budgets-2026-03-13` beta header. It works for the Claude models that the adapter lists as supported: Fable, Mythos, Opus 4.7, 4.8 and 5, Sonnet 5.5 and Haiku 5.5. It does not work for Sonnet 5, Opus 4.6, Sonnet 4.6 or Haiku 4.5 (`src/anthropic.ts`). It is set once and never changed per request. It is advice for the model only: Amness's own cap still applies. Choose the size from your real usage. A budget that is too small makes Claude narrow the task or stop early.
+
+All of this is tested with scripted models and fake SDK clients (for example `test/features.test.ts`). None of it is measured on a real model for fewer exhausted runs. The eval runner can measure this once there are exploration-style tasks in the suite.
+
+### Subagent slices
+
+**What it is.** A subagent gets a slice of the budget, not the whole budget. `Budget.slice(share, max)` in `src/harness.ts` gives a subagent at most `share` of what is left, and no more than `max` tokens. Everything the subagent spends also counts against the main budget.
+
+| Subagent | Share of what is left | Token limit | Other limits |
+|---|---|---|---|
+| Read-only explorer (built in) | 12% | 60,000 | 14 steps. The prompt says it has about 12 tool calls and should answer in under 300 words. |
+| Custom agent (from `.amness/agents/`) | 25% | 120,000 | Its own `maxSteps`. |
+
+Other limits (`src/code.ts`):
+
+- A request may start at most 6 subagents (`MAX_SUBAGENTS_PER_REQUEST`). More return an error that tells the lead to work with what it has.
+- At most 3 `task` calls run at the same time.
+- Reports are clipped to 8,000 characters.
+- Every report ends with a line such as `[subagent used about 3k tokens; the budget for this request is 40% used]`, so the lead can decide to stop delegating.
+- A `task` call with an agent name that does not exist uses the read-only explorer and says so, instead of failing.
+
+Inside its slice, a subagent gets the same budget notes and final report. An explorer that runs out returns what it found, not an error. Tested in `test/agents.test.ts` and `test/features.test.ts`.
+
+**Limits.** An older version of this page said 20% and 80,000 tokens for the explorer, and 30% and 150,000 for custom agents. Those numbers were out of date. The values in the table match the code.
+
+## What went wrong in the second report, and the fix
+
+A "survey the whole codebase" run stopped at "budget nearly exhausted" while the display said 46% used. There were three bugs.
+
+1. The step-cost estimate behind the early final report was the change in the shared budget. That also contains what the subagents spent in the meantime, so a step that ran three explorers looked very expensive. It now uses the cost of the lead's own last request.
+2. Subagent budget notes were drawn as red errors, and every nudge was labelled "nudged the model to act instead of describing". Now nudges say what happened, subagent notes are not shown, and the final report is shown as a normal answer.
+3. Nothing limited how many subagents a request could start. Now a request may start at most 6 subagents, explorers get a smaller slice, and every subagent report ends with what it cost.
+
+After the fix, the same kind of request on `gpt-6-luna` (300,000 cap) finished in 83 seconds with a full answer. It used 132,000 of the 300,000 (cached input counted at a tenth; 234,000 raw). Two subagents wrapped up inside their slices. This was one run. It is not a measurement.
+
+## Rate limits (requests and tokens per minute)
+
+Several agents that share one API key (the lead, up to three explorers, the reviewer) can go over the per-minute limits. Failed requests count toward the limits too. A *429 error* is the answer a provider sends when you are over a limit.
+
+What the vendors say ([OpenAI rate limits](https://developers.openai.com/api/docs/guides/rate-limits), [Claude rate limits](https://platform.claude.com/docs/en/api/rate-limits), both *read*):
+
+- Wait at least the time in the `Retry-After` header, plus a small random delay (jitter). When the header is missing, use exponential backoff with jitter. Cap the number of attempts and the total wait. Do not retry quota or spend-cap errors, because they also arrive as 429. If you retry yourself, do not also let an SDK retry.
+- Limits are enforced over short windows. A limit of 60 requests per minute can mean one request per second. A sharp rise in traffic can trigger "acceleration" 429 errors, so ramp up step by step.
+- Claude counts only uncached input toward its input-tokens-per-minute limit, and `max_tokens` does not count toward its output limit. OpenAI counts the larger of `max_tokens` and the estimated request size, so keep `max_tokens` near the output you expect.
+
+What Amness does (`src/ratelimit.ts`, for every provider, shared by all agents in the process):
+
+1. One gate for each endpoint and key limits the requests in flight. The ceiling is 4 by default (`maxConcurrent` in the config). The gate starts at 2 and adds one after three successes, so traffic ramps up step by step.
+2. On a 429 that waiting can fix, every caller pauses for the server's `Retry-After` time (or an exponential backoff of up to 60 seconds, with jitter). The gate halves its concurrency, like TCP congestion control. The chat tells you: `rate limited by the provider: waiting 12s, then retrying (attempt 1); 2 requests at a time for now`.
+3. These cases are not retried, and the error reaches you: quota, billing and spend-cap errors; an abort; a requested wait longer than two minutes; and more than six retries.
+4. The OpenAI adapter does not retry 429 by itself when the gate is on (`externalRetry`). It still retries timeouts and 5xx errors, and it honours `Retry-After`.
+
+**How to use it.** If you see rate-limit messages often, lower the ceiling in `~/.amness/config.json`:
+
+```json
+{ "maxConcurrent": 2 }
+```
+
+**Limits and common problems.** Tested with fake models and clocks, and end to end against a local HTTP server that returns 429 with `Retry-After` (`test/ratelimit.test.ts`). The limit of 3 subagents at once is tested too (`test/agents.test.ts`). Not yet seen with a real provider 429. Evals ran slower than expected when many sessions ran at once. That is the situation this gate exists for.
+
+## See subagents in the UI
+
+A subagent's steps used to be drawn as loose lines in the transcript, mixed together when explorers ran in parallel. Now every event carries the id of the `task` call that started it (`parent`).
+
+- **Full-screen view.** The steps live inside their card. While the subagent runs, the card shows the newest three steps. When it is done, the card shows a one-line summary such as "12 steps · 7 files read · 4 searches". Open the card to see all steps.
+- **Inline view.** One live line for each subagent shows its newest step and step count.
+- An interrupted or failed turn closes every card that was still running.
+
+## Know that this works with any model
+
+Everything except item 7 lives in the harness and works the same for OpenAI, Claude and local models. The notes and the final report are ordinary messages. The subagent limits are Amness's own. The prompt rules are plain text. Where a server reports no token usage, the harness estimates it, so the cap, the notes and the final report still work. Only the opt-in `taskBudget` is specific to Claude. Other providers and unsupported Claude models ignore it.
+
+## Not done, and why
+
+- **OpenAI server-side compaction** is an option: set `"compactAt": 150000` in the config. It works on the Responses API on official hosts only. The request carries `context_management` with that threshold, and input before the newest compaction item is no longer sent again. Tested at the wire level only. Not measured: it is unknown whether it beats Amness's own staged trimming. It is off by default.
+- **A hard cap on reads in the tool layer** (stop the read tool after N reads until the agent states a plan): proposed in blogs, not verified, and risky for legitimate large reads. The nudge comes first. Measure before making it a hard stop.
+- **Exploration that returns only file paths and line ranges** (a paper named FastContext is cited in a blog, *snippet*, not verified): not implemented.
+
+## Next steps
+
+- [The agent loop](agent-loop.md): see where the budget check sits in each turn.
+- [Using the chat](usage.md): the `/budget`, `/loop` and `/tokens` commands.
+- [Headless and CI](headless.md): `--max-tokens` and exit codes.
+- [Providers and models](providers.md): OpenAI, Claude and local models.
+- [Evaluating the harness](evaluation.md): measure what a guard is worth.