amness 0.0.0-stage → 0.1.2

package/docs/cookbook.md

@@ -0,0 +1,375 @@
+# Cookbook: short recipes
+
+Each recipe shows one task you may want to do, with the exact commands and what you will see.
+
+In this page you will learn how to:
+
+- run Amness in CI, with several attempts or with parts in parallel
+- control cost, and use a local model
+- add your own rules, and measure them
+- replay, review and schedule work
+
+Commands marked "calls your model" cost tokens. The others call no model.
+
+## 1. Run a fix in CI, with several attempts
+
+*Calls your model.* Use `amness run` for a one-shot run with no terminal. `--verify` is a command that must exit 0 before the agent may finish.
+
+```bash
+amness run --verify "npm test" --json "fix the failing test"
+```
+
+You will see one JSON event per line, then a last line with `"kind":"result"`. The exit code is `0` if the agent finished, `1` if it did not, and `2` for a usage or configuration error. Risky commands are denied unless you add `--yes`.
+
+Project hooks and plugins are skipped in headless runs unless you add `--trust-project`. See [Headless and CI](headless.md).
+
+Want more than one try? Add `--best-of 3`. Amness then runs the task up to three times in separate git worktrees and stops at the first attempt that passes `--verify`. It applies that result to your folder.
+
+```bash
+amness run --best-of 3 --verify "npm test" "fix the failing test"
+```
+
+While it runs, you will see lines such as `attempt 1: failed verify (41200 tokens)`. At the end you will see `attempt 2 won and was applied`. If no attempt passes, you see `no attempt passed npm test (3 tried, N tokens)`. `--verify` is required, and the folder must be a git repository with at least one commit. The token cap is shared between the attempts. Whether best-of-n improves results is not measured yet.
+
+## 2. Split work into parts
+
+*Calls your model.* `--parts` runs independent tasks at the same time, each in its own worktree. Amness merges them one at a time and runs `--verify` after each merge.
+
+1. Write `parts.json`:
+
+   ```json
+   [
+     { "name": "docs", "task": "Fix the typos in README.md" },
+     { "name": "tests", "task": "Add a test for the parse function in src/parse.ts" }
+   ]
+   ```
+
+2. Run it:
+
+   ```bash
+   amness run --parts parts.json --verify "npm test"
+   ```
+
+You will see one line per part, with a status such as `merged`, `conflict`, `failed-verify`, `broke-verify`, `no-change`, `error` or `skipped-budget`. A conflict is reported and not applied. The exit code is `0` only if every part is `merged`. Use parts only for work that does not touch the same files.
+
+## 3. Limit cost
+
+The cap is the most tokens one turn may use. Cached input counts as a tenth, because it costs less.
+
+In the chat:
+
+```text
+/budget
+/budget 500k
+```
+
+The first line shows the cap, and the tokens used in the last turn. The second sets a new cap. The cap must be at least 10000 tokens. You can write `400000`, `400k` or `1m`.
+
+At 50%, 75% and 90% of the cap, Amness tells the model to wrap up. Just before the cap, the model writes a short final report. In a headless run, use a flag:
+
+```bash
+amness run --max-tokens 200000 --verify "npm test" "fix the failing test"
+```
+
+The default for headless runs is 300000. A very small cap can stop a task before it is done. See [Budgets and subagents](budgets.md).
+
+## 4. Use a local model
+
+Any server with an OpenAI-compatible `/v1` address works. Ollama is one example.
+
+```bash
+ollama pull qwen2.5-coder:7b
+amness run --provider ollama --model qwen2.5-coder:7b "list the TODO comments in src"
+```
+
+In the chat, pick the provider and model with `/provider` and `/model` instead: the interactive `amness` command does not read `--provider` or `--model`.
+
+The address defaults to `http://localhost:11434/v1`. For another server, use `amness run --provider custom --base-url http://localhost:1234/v1 --model <model-name>`. You need no API key for a local server.
+
+Limits: the model must support tool calling. Small models often work better with `/mode plan` and a `--max-tokens` limit. How well any local model performs is not measured here. To measure it, run `amness eval --variants ab` with that model. See [Providers and models](providers.md).
+
+## 5. Add a house rule as a plugin
+
+A plugin can block an action in code. This one blocks hand edits of `package-lock.json`.
+
+1. Save `.amness/plugins/no-lockfiles.ts`:
+
+   ```ts
+   export default {
+     name: "no-lockfiles",
+     hooks: {
+       PreToolUse: ({ tool, args }: { tool: string; args: any }) =>
+         (tool === "write_file" || tool === "edit_file") && /package-lock\.json$/.test(String(args.path))
+           ? { block: "lockfiles are generated: run the package manager instead" }
+           : undefined,
+     },
+   };
+   ```
+
+2. Start `amness` and choose "Yes, trust this version" when it asks.
+3. Type `/plugins`. You will see `no-lockfiles` and `hooks: PreToolUse`.
+
+When the agent tries to edit the lockfile, it gets `BLOCKED BY A HOOK: lockfiles are generated: run the package manager instead`. See [Plugins](plugins.md) for the full steps.
+
+## 6. Compare two harness variants
+
+*Calls your model.* First check the tasks. This is free.
+
+```bash
+amness eval --selftest
+```
+
+Then compare the full harness with the one that has the reviewer:
+
+```bash
+amness eval --variants full,full+review --trials 3 --jobs 2
+```
+
+Amness prints a plan and asks before it starts. Add `--yes` to skip the question. You will see a table with `pass [95% CI]`, `pass^3`, `tok/solved`, `cache`, `steps`, `secs` and `pareto`, and a second table with the change against the first variant. A `verdict` of `inconclusive` means the data cannot tell. See [Evaluating the harness](evaluation.md) for how to read each column.
+
+## 7. Write your own eval task
+
+Use a task from your own work. A task is a JSON file with files, a task text and a held-out check.
+
+1. Save `my-tasks/fix-add.json`:
+
+   ```json
+   {
+     "name": "fix-add",
+     "files": {
+       "add.js": "exports.add = (a, b) => a - b;\n",
+       "test.js": "const { add } = require('./add');\nif (add(2, 3) !== 5) process.exit(1);\n"
+     },
+     "task": "The add function is wrong. Fix it.",
+     "verify": "node test.js",
+     "check": "node test.js",
+     "solution": { "add.js": "exports.add = (a, b) => a + b;\n" }
+   }
+   ```
+
+2. Check it. This calls no model:
+
+   ```bash
+   amness eval --cases ./my-tasks --selftest
+   ```
+
+   You will see `ok  fix-add`.
+
+3. Run it (calls your model):
+
+   ```bash
+   amness eval --cases ./my-tasks --variants ab --trials 3
+   ```
+
+One task is too few to compare variants. Add more tasks first.
+
+## 8. Replay a saved trace after you change the harness
+
+Replay feeds a recorded session back through the current harness. It calls no model and runs no tool. Only the harness rules can differ, so you can test a change for free.
+
+1. Find a trace. Chat sessions write them to `.amness/traces/`. For a headless run, add `--trace`:
+
+   ```bash
+   ls .amness/traces
+   ```
+
+2. Replay one, here with the `nudge` guard off:
+
+   ```bash
+   amness replay .amness/traces/<session>.jsonl --off nudge
+   ```
+
+   Replace `<session>` with a file name from step 1.
+
+You will see a line such as `run 1: 12 model turn(s), identical harness behaviour`, or `3 difference(s)` followed by lines that start with `- recorded only:` and `+ replay only:`. The exit code is `0` if all runs match and `1` if not. Other flags are `--stagnation-turns n`, `--max-steps n` and `--context-budget chars`.
+
+Limits: after the first difference, the recording no longer matches what the model would have said, so later lines are only a hint. Old traces without model messages cannot be replayed.
+
+## 9. Find recurring failures
+
+Amness reads your session traces and groups what went wrong.
+
+```bash
+amness traces
+```
+
+You will see a summary line, then patterns with a count, for example `24×  ran out of budget: budget N% used  (10 sessions)`, and the sessions with the most trouble. In the chat, type `/report` for the same view.
+
+To keep a failure from coming back, turn it into an eval task. The `--check` command must exit 0 only when the task is really done:
+
+```bash
+amness traces --case my-bug --task "Fix the date parsing bug" --check "npm test"
+```
+
+Amness copies the tracked files at git `HEAD` to `.amness/evals/cases/my-bug.json`. Run it with `amness eval --cases .amness/evals/cases --variants ab --trials 3`.
+
+## 10. Schedule a nightly run
+
+*Calls your model when the job runs.* Schedules use cron. The five fields are minute, hour, day of month, month and day of week.
+
+1. Add a schedule for this project (3:00 on weekdays):
+
+   ```bash
+   amness schedule add nightly "0 3 * * 1-5" "run the tests and fix small failures" --verify "npm test"
+   ```
+
+   You will see the text: saved "nightly". Run `amness schedule install` to see the crontab line.
+
+2. Look at the crontab line. This writes nothing:
+
+   ```bash
+   amness schedule install
+   ```
+
+3. Put it in your crontab:
+
+   ```bash
+   amness schedule install --write
+   ```
+
+Other commands are `amness schedule list`, `amness schedule remove nightly` and `amness schedule uninstall`. Amness only changes a marked block for this project in your crontab.
+
+Each run is `amness run --json`, so risky commands are denied. The output goes to `.amness/schedule-logs/nightly.jsonl`. Cron must be able to find `amness` and your API key. Set them in the crontab environment or use a wrapper script. Cron names such as `MON` are not accepted. Use numbers.
+
+## 11. Use it from an editor
+
+Editors such as Zed and JetBrains can start Amness as their coding agent with the Agent Client Protocol (ACP). You do not run the command by hand.
+
+For Zed, add this to your settings:
+
+```json
+"agent_servers": { "Amness": { "command": "amness", "args": ["acp"] } }
+```
+
+Amness uses your configured provider and model. When the agent wants to do something risky, the editor shows a permission request with the choices Allow and Reject.
+
+Limits: Amness does not implement `session/load`, `authenticate`, `fs/*` or `terminal/*`. Only the Zed setting above comes from the built-in help. JetBrains setup is not verified here.
+
+## 12. Review your changes before a pull request
+
+In the chat, inside a git repository:
+
+```text
+/review
+```
+
+The agent reads your changes against the base branch and lists findings, or says `review: no problems found`. Use `/review main` to name the base branch. The review is read-only and costs model calls.
+
+To create the pull request:
+
+```text
+/pr
+```
+
+Amness drafts a title and a description from your diff and commits. It shows them and asks "Create this pull request on GitHub?" with "Create it", "Create it as a draft" and "Cancel". If the branch is not on the remote, it asks before it pushes. Use `/pr draft` for a draft.
+
+Limits: `/pr` needs the GitHub CLI (`gh`) and a branch that is not the base branch. It creates nothing without your choice.
+
+## 13. Write a finish gate
+
+A *finish gate* is a check that must pass before the agent may finish. Save `.amness/plugins/gate.ts`:
+
+```ts
+export default {
+  name: "gate",
+  gates: [{
+    name: "tests-after-edit",
+    check: (ctx: any) => ctx.changed.length && ctx.testsAfterLastEdit === false
+      ? "you changed files but did not run the tests after the last edit" : true,
+  }],
+};
+```
+
+Start `amness` and approve the plugin. If the agent edits files and tries to finish without a test run, it sees `REJECTED by a hook: gate "tests-after-edit" (plugin gate) found problems:`. After 2 rejections for the same task, the run finishes anyway with a warning. Details: [Plugins](plugins.md#add-a-finish-gate). Tested with scripted models only.
+
+## 14. Deny a tool with a policy and test it on past traces
+
+1. Save `.amness/plugins/nocurl.ts`:
+
+   ```ts
+   export default {
+     name: "nocurl",
+     policy: ({ tool, args }: any) =>
+       tool === "sh" && /\bcurl\b/.test(String(args.cmd)) ? { action: "deny", reason: "no curl" } : undefined,
+   };
+   ```
+
+2. Replay your old sessions through it. This calls no model:
+
+   ```bash
+   amness plugin policy .amness/traces --plugin .amness/plugins/nocurl.ts
+   ```
+
+   You will see a line such as `40 trace file(s): 2 denied, 0 would ask, 0 allowed by the policy, 900 without an opinion`, then the denied calls. Deny beats ask, and ask beats allow. A policy cannot loosen the safety floor. Details: [Plugins](plugins.md#write-policy-as-code).
+
+## 15. Try a loop strategy and measure it
+
+*Calls your model.* Try it in the chat first:
+
+```text
+/strategy plan-first
+```
+
+Then measure it against the default:
+
+```bash
+amness eval --variants full,strategy:plan-first --builtin hard --trials 3 --metrics diff-lines,tool-calls
+```
+
+Read the report as in recipe 6. On one run with `gpt-4.1-mini` (6 tasks), `plan-first` and `full` had the same pass rate and the result was inconclusive. See [Evaluating the harness](evaluation.md#loop-strategies-2026-10-30).
+
+## 16. Stop regressions in CI with `--gate`
+
+*Calls your model.* Save a baseline once and commit it:
+
+```bash
+amness eval --variants full --builtin hard --trials 3 --yes --out evals/baseline.json
+```
+
+In CI, run the same command with a gate:
+
+```bash
+amness eval --variants full --builtin hard --trials 3 --yes --gate evals/baseline.json
+```
+
+The exit code is 1 when the pass rate dropped by more than 5 points and the drop is clear, or when tokens per solved task grew by more than 25%. Change the limits with `--gate-tol` and `--gate-cost`. A drop that noise could explain is only a warning. See [Custom metrics and the regression gate](evaluation.md#custom-metrics-and-the-regression-gate).
+
+## 17. Make things fail on purpose with a chaos variant
+
+*Calls your model.* Check that the harness recovers when tools or the API fail:
+
+```bash
+amness eval --variants full,chaos:tools,chaos:429 --trials 3
+```
+
+`chaos:tools` makes about 1 in 4 tool calls fail. `chaos:429` makes about 3 in 10 model calls fail with a rate-limit error. Other kinds are `chaos:args` and `chaos:slow-tools`. A lower pass rate under chaos shows where the harness does not recover. Not measured on a real model yet.
+
+## 18. Compare Amness with Claude Code and Codex
+
+*Calls your models and costs money.* Install and log in to the other tools yourself. Then check, plan and probe before the real run:
+
+```bash
+amness bench --init
+amness bench --check
+amness bench --config bench.json --dry-run
+amness bench --config bench.json --probe
+amness bench --config bench.json --builtin code
+```
+
+Set a model for each contestant in `bench.json` and set a spending limit first. The result is a table with pass rate, tokens, cost and time. With 13 tasks, only large differences can be seen. Read [Compare coding agents with `amness bench`](bench.md), especially the Windows setup and the fair-play list. The `claude` and `codex` presets have not been run on a real machine by the author.
+
+## Related pages
+
+- [Compare coding agents](bench.md)
+- [Headless and CI](headless.md)
+- [Evaluating the harness](evaluation.md)
+- [Plugins](plugins.md)
+- [Providers and models](providers.md)
+- [Budgets and subagents](budgets.md)
+
+## Next steps
+
+- Start with recipe 6 or 7 to measure the harness on your own work.
+- Recipes 13 to 17 cover finish gates, policies, strategies, CI gates and chaos.
+- Read [Features in depth](features.md) for permissions, sandbox and memory.
+- Read [Using the chat](usage.md) for all slash commands.

package/docs/development.md

@@ -0,0 +1,150 @@
+# Development and limitations
+
+This page shows you how to run the tests, where the code lives, how to add a feature, and what Amness cannot do yet.
+
+In this page you will learn:
+
+- how to run the type check and the tests
+- what each folder contains
+- how to add a tool, a guard or a command
+- the known limits of the project
+
+You need Node.js 22.18 or newer (the `engines` field in `package.json`) and `git`. The pseudo-terminal tests also need `python3`.
+
+## Run the tests
+
+**What it is.** The project has a type check, unit tests, UI tests, and tests that run the real program in a pseudo-terminal. A *pseudo-terminal* is a fake terminal that a script can type into and read from.
+
+**Why you would use it.** Run them before every change. CI runs the same commands on every push to `main` and every pull request.
+
+**How to use it.**
+
+1. Install the packages:
+
+```bash
+npm install
+```
+
+2. Check the types. This runs `tsc --noEmit` over `src`, `cli`, `examples` and `test`:
+
+```bash
+npm run typecheck
+```
+
+3. Run the tests. This uses the Node.js built-in test runner on `test/**/*.test.ts`, with a 180 second limit per test:
+
+```bash
+npm test
+```
+
+4. Run the terminal tests (needs `python3`). This starts the real `amness` program in a pseudo-terminal against a mock server:
+
+```bash
+npm run test:pty
+```
+
+5. Run one test file:
+
+```bash
+node --test test/permissions.test.ts
+```
+
+**Example.** When all tests pass, the summary at the end of `npm test` shows `fail 0`, and the command exits with code 0. The exact format depends on your Node.js version.
+
+**Limits and common problems.**
+
+- The Linux sandbox tests skip themselves when Bubblewrap cannot create namespaces. CI has one extra step that sets `AMNESS_REQUIRE_SANDBOX=1`, so a skipped sandbox test shows up there. This step is informational and does not fail the build.
+- The tests do not call real model APIs. They use scripted models and mock HTTP servers (see below). Use `amness eval` against a real model to measure behavior. See [Evaluating the harness](evaluation.md).
+- Some tests set `AMNESS_HOME` to a temporary folder. If you run Amness by hand while testing, set `AMNESS_HOME` too, so you do not change your real `~/.amness`.
+
+## Know what CI runs
+
+The file `.github/workflows/ci.yml` has these jobs:
+
+| Job | What it does |
+|---|---|
+| `test` | `npm ci`, `npm run typecheck`, `npm test` on Node 22 and Node 24, with Bubblewrap installed |
+| `terminal` | `python3 test/pty/tui.py` on Node 24 |
+| `package` | `npm pack`, installs the tarball in a new folder, runs `npx amness --version`, and checks that `import("amness")` exports `createCodeAgent` |
+| `docker` | builds the `Dockerfile`, runs `--version` and `eval --selftest --only off-by-one` |
+| `windows` | `npm run typecheck` and `npm test` on Windows. It is informational (`continue-on-error`) |
+| `action` | runs the GitHub Action (`action.yml`) with no API key and checks that it reports instead of crashing |
+
+A second workflow, `.github/workflows/publish.yml`, runs type check and tests, then `npm stage publish --provenance`, when you push a tag that starts with `v`. It needs an `NPM_TOKEN` secret (a stage-only token is enough). It only stages the version; you release it with `npm stage approve <id>` and your 2FA code.
+
+## Find your way around the project
+
+| Path | What is in it |
+|---|---|
+| `src/` | The library: the loop (`harness.ts`), the coding agent (`code.ts`), the research agent (`deep.ts`), model adapters (`openai.ts`, `anthropic.ts`), permissions, sandbox, secrets, hooks, plugins, memory, evals. `src/index.ts` exports everything |
+| `cli/` | The program: `amness.ts` is the entry point. `headless.ts` is `amness run`. `evalcli.ts` is `amness eval`. `config.ts` reads `~/.amness/config.json`. `cli/tui/` is the chat interface (`app.ts`, `commands.ts` for slash commands, `full.ts` for the full-screen view) |
+| `test/` | Tests, one file per area, for example `permissions.test.ts`, `sandbox.test.ts`, `taint.test.ts`, `harness.test.ts`. UI tests start with `tui-`. |
+| `test/helpers/` | `vterm.ts` (a small VT100 emulator that reads the screen back) and `apprig.ts` |
+| `test/pty/tui.py` | The pseudo-terminal tests |
+| `test/fake-mcp.mjs` | A fake MCP server for tests |
+| `examples/` | Runnable examples: `code.ts`, `research.ts`, `eval.ts`, `cases.ts`, `plugins/house-rules.ts` |
+| `docs/` | These pages. Read [How we write the documentation](writing-guide.md) before you edit them |
+| `dist/` | Build output from `npm run build`. Do not edit it |
+
+For the layers and how the screen is drawn, see [Architecture](architecture.md).
+
+## How the tests work
+
+- The UI tests start the real `App` class, send fake key presses, and read the screen back through `test/helpers/vterm.ts`. They check layout, cursor place and redraw, not only text.
+- Model providers are tested with scripted models and mock HTTP servers. One mock reproduces the OpenAI error "Function tools with reasoning_effort are not supported" (see `test/pty/tui.py`).
+- Many tests use a scripted model. This is a function that returns prepared answers, so a test is fast and repeatable.
+
+## Add a feature
+
+**1. Add a tool for the coding agent.**
+
+1. Open `src/code.ts`. Tools are made with `defineTool` and listed in `codeTools`.
+2. Give the tool a `name`, a `description`, a JSON `schema`, a `run` function, and the flags that fit: `readOnly: true` for tools that change nothing, `risky` for tools that need approval, `untrusted: true` for tools that return text from outside.
+3. Write a test in `test/` that uses a scripted model.
+4. Run `npm run typecheck` and `npm test`.
+
+If you only need a tool for yourself, write a plugin instead. A plugin needs no change to the source. See [Plugins](plugins.md).
+
+**2. Add a guard to the loop.**
+
+1. Open `src/harness.ts`. The loop is the function `run` inside `createHarness`.
+2. Add the guard name to the `Guard` type, so `off` and `amness eval` can switch it off. Also add it to the `GUARDS` list in `src/profiles.ts`.
+3. Emit an event with `emit(...)` so the chat and traces can show it. Add the event to the `Event` type.
+4. Test it with a scripted model that makes the mistake. Add a case to `src/evalcases.ts` if you want to measure it with `amness eval`.
+
+**3. Add a slash command.**
+
+1. Open `cli/tui/commands.ts`. Each command is an object with `name`, `help` and `run`.
+2. Add yours. Run the program and type `/help` to see it.
+3. Add a test in `test/tui-app.test.ts` or a similar file.
+
+**4. Add a model provider.**
+
+A model is a function `(messages, tools, opts) => Promise<Message>`. Write one function in a new file under `src/`, export it from `src/index.ts`, and test it with a mock HTTP server. The providers in `cli/config.ts` list what the chat can choose.
+
+**Before you open a pull request.** Make sure `npm run typecheck` and `npm test` pass. Update the docs page that describes the feature, and keep to the [writing guide](writing-guide.md).
+
+## Known limits
+
+- **Platforms.** Amness is developed and tested on macOS and Linux terminals. The CI job for Windows is informational. There is no Windows sandbox.
+- **Real providers.** The automated tests use scripted models and mock servers. Behavior against live provider APIs was checked by hand only. Provider APIs change. If one breaks, open an issue with the error text.
+- **Cost figures.** They are estimates, not invoices. Cached input is assumed to cost about 10% of the input price. Some models have no price listed.
+- **New features.** Cache keep-alive, semantic memory and `amness eval` against a real model are covered by tests with scripted models. They were not run against live provider APIs in the tests. Numbers quoted from papers belong to those papers. They are not measurements of Amness.
+- **Semantic memory.** It sends note text and your messages to the embeddings endpoint of your provider. It needs an OpenAI-compatible provider. Claude has none.
+- **Local models.** Tool-calling quality with small local models varies a lot.
+- **Node.js.** Amness runs TypeScript directly with Node.js type stripping, so it needs Node 22.18 or newer. The package build in `dist/` is checked by the CI job `package`.
+- **Docker.** CI builds the image and runs two commands in it. It does not run a full task.
+- **Missing features.** The file [ROADMAP.md](../ROADMAP.md) lists what is done and what is planned.
+
+## Related pages
+
+- [Architecture](architecture.md)
+- [Plugins](plugins.md)
+- [Evaluating the harness](evaluation.md)
+- [Use it as a library](library.md)
+
+## Next steps
+
+- Read how the loop works: [The agent loop](agent-loop.md)
+- Add your own rules without changing the source: [Plugins](plugins.md)
+- Return to the [documentation index](index.md)